# The Independent Climate Change

E

- mail s Review

July 2010

Chair: Sir Muir Russell

Review team: Professor Geoffrey Boulton

Professor Peter Clarke David Eyton Professor James Norton2



## INDEX OF CONTENTS

GLOSSARY 8

Chapter 1 EXECUTIVE SUMMARY 10

1.1 Introduction 10

1.2 The Review Process 11

1.3 Findings 11

1. 3 .1 Land Station Temperatures 12 1.3.2 Temperature Reconstructions from Tree Ring Analysis 12

1.3

. 3 Peer Review and Editorial P olicy 13

1.3

. 4 Misuse of IPCC P rocess 13

1.3

. 5 Compliance with the Freedom of Information Act (FoIA) and the Environmental Information Regulations (EIR) 14

1.3 .6 Other Findings on Governance 14

1.4 Recommendations 14

1.5 Broader Issues 14

Chapter 2 INTRODUCTION 18

2.1 Background 18

2.2 The Review 19

Chapter 3 TER MS OF REFERENCE AND METHOD OF E NQUIRY 22

3.1 Terms of Reference 22

3.2 Method of E nquiry 22

Chapter 4 C ONTEXT OF THE E

- MAILS 26

4.1 Characterising the E

- mails 26

4.2 The T imeline 27

4.3 What the E

- mails Tell U s 32

Chapter 5 THE CHANGING CONTEXT OF MODERN SCIENCE 36

5.1 The Scientific P rocess 36

5.2 The Nature of Climate S cience 36

5.3 Data A ccessibility in the D igit al A ge 37

5.4 Handling U ncertainty 38

5.5 Scientific Journals and the Peer Review P rocess 39

5.6 The Responsibilities of Scientists in Communicating in the Public D omain 40

5.7 Communicating to P olicymakers 41

5.8 The C ha nging Forum for D ebate and the B logosphere 41

Chapter 6 LAND STATION INSTRUMENTAL TEMPERATURE DATA 44

6.1 Background 44 6.2 The Allegations 44 6.3 The Approach Adopted by the Review Team 45 6.4 The Results of the Analysis 46 6.5 Checking Specific Details in the CRUTEM Analysis 49

6.5.1 Identification of Data Sources 49 6.5.2 The Availability of Computer Codes 49 6.6 Use of Local temperature Data from China

6.7 Conclusions and Recommendations 51

5334 Chapter 7 TEMPERATURE RECONSTRUCTIONS FROM TREE RING ANALYSIS 54

7.1 Background 54

7.2 The Allegations 54 7.3 Findings 56 7.3.1 IPPC Reports 56 7.3.2 Divergence 59 7.3.3 Withholding Data 60 7.3.4 Mishandling Data 61

7.4 Conclusions and R ecommendations 62

Chapter 8 PEER REVIEW AND INFLUENCING EDITORIAL POLICY OF SCIENTIFIC

JOURNALS 64

8.1 Background : Peer Review, Testing and V erification 64

8.2 The Allegations 64

8.3 The Soon and Baliunas A ffair & Climate Research 64

8.4 The C onflict with Dr Boehmer

- Christiansen 66
8.5 Peer Review and Professor Briffa‘s E ditorship of Holocene 67

8.6 Conclusion s 68

Chapter 9 COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC 70

9.1 Background 70

9.2 The Allegations 70

9.3 The CRUTEM Temperature S eries 71

9.3.1 The Scientific C hallenge 71

9.3.2 The A llegations 71

9.3.3 Evidence in S u pport of the A llegations 71

9.3.4 Jones‘ R esponse 73

9.3.5 Eviden ce from IPCC Review Editor for C hapter 3 (Professor Sir Brian Hoskins) 74

9.3.6 Findings 75 9.4 The Tree Ring Proxy Temperature S eries 77

9.4.1 The Scientific C hallenge 77

9.4.2 The A llegations 77 9.4.3 Evidence in Support of the A llegations 78 9.4.4 Responses from Briffa 80

9.4.5 Eviden ce from IPCC Review Editor for C hapter 6 (Professor John Mitchell) 82

9.4.6 Findings 83

9.5 Conclusions 84

Chapter 10 COMPLIANCE WITH FoIA / ENVIRONMENTAL INFORMATION

REGULATIONS 86

10.1 Introduction and Method of Enquiry 86

10.2 The Allegations 86

10.3 General Context 86 10.4 Investigation 89 10.5 Findings 91

10.6 Recommendations 9456 Chapter 11 GOVERNANCE 98

11.1 Introduction and Method of E nquiry 98 11.2 Research Management Systems 98 11.2.1 Background 98 11.2.2 Funding Management 99
11.2.3 Funders‘ R equirements 99

11.2.4 Good Research Practice 99

11.2.5 Financial Controls 100 11.2.6 Risk Management 100 11.2.7 Findings on Research Management Systems 100 11.3 Software, Data Management and Data Security 101

11.3.1 General Context 101

11.3.2 Issues and the Investigation 102 11.3.3 Findings on Software, Data Management and Data Security 103

11.4 Recommendations 103

ACKNOWLEDGEMENTS 106

APPENDIX 1: REVIEW TEAM MEMBERS 108

APPENDIX 2: APPROACH AND WORK PLAN 110

APPENDIX 3: ISSUES FOR EXAMINATION 112

APPENDIX 4: INDEX OF MEETINGS, INTERVIEWS, SUBMISSIONS, 120

FOLLOW UP ENQUIRIES AND RESPONSES

APPENDIX 5: PEER REVIEW 126

APPENDIX 6: DATA MINING

– ACCESS TO THE ORIGINAL CRU E

- MAIL 146

ARCHIVE

APPENDIX 7: LAND STATION TEMPERATURE DATA 150

APPENDIX 8: SOFTWARE AND DATA STANDARDS 16078



## GLOSSARY

AR4 Fourth Assessment Report of the Intergovernmental Panel on

Climate Change in 2007

CLA Coordinating Lead Author

COP Conference of the Parties

COPE Committee on Publication Ethics

CRU Climatic Research Unit

CRUT EMX Land air temperature anomalies on a 5° by 5° grid

- box basis

,

version X

DEFRA UK Government Department for Environment, Food and Rural

Affairs

DPA Data Protection Act 1998

E&E Energy and Environment

EC European Community

EIR Environmental Information Regulations

ENSO El Niño

- Southern Oscillation

ENV U niversity of E ast A nglia School of Environmental Sciences

FOI Freedom of Information

FoIA Freedom of Information Act 2000

GHCN Global Historical Climatology Network

GISS Goddard Institute for Space Studies

GISTEMP Goddard Institute for Space Studies Surface Temperature Analysis

GWPF Global Warming Policy Foundation

HadCRUTX Combined land and marine temperature anomalies on a 5° by 5°

grid

- box basis , version X

IAC Inter Academy Council

ICCER Independent Climate Change E

- Mails Review

ICO Information Commissioner‘s Office

ICT Information Communications Technology

IDL Interactive Data Language

IEC International Electrotechnical Commission

IPCC Intergovernmental Panel on Climate Change

IPCM Information Policy and Compliance Manager

IS Information Systems

ISSC Information Systems Strategy Committee

IT Information Technology

JANET The United Kingdom‘s Education and Research Network

JISC Joint Information Systems Committee

LA Lead Authors

LIA Little Ice Age

MBH Mann, Bradley and Hughes

MM McKitrick and Michaels9

M&M McIntyre and McKitrick

MWP Medieval Warm Period

NAO North Atlantic Oscillation

NASA National Aeronautics and Space Administration

NCAR National Center for Atmospheric Research

NCDC National Climatic Data Center

NMO National Meteorological Office

NOAA National Oceanic and Atmospheric Administration

PERL Practical Extraction and Reporting Language

PI Principal Investigator

REE Research, Enterprise and Engagement

S&B Soon and Baliunas

SAP Scientific Assessment Panel

SPM Summary for Policy Makers

TAR Third Assessment Report of the IPCC in 2001

UEA University of East Anglia

UHI Urban Heat Island

UKRIO United Kingdom Research Integrity Office

UN United Nations

UNEP United Nations Environment Programme

WMO World Meteorological Organization

WWR World Weather RecordsCHAPTER 1: EXECUTIVE SUMMARY

10 

## CHAPTER 1: EXECUTIVE SUMMARY

1. The main findings of the Independent Climate Change E-mails Review (―the Review‖) are set out in Section 1.3 below, and the main recommendations in Section 1.4. We comment in Section 1.5 on some of the more general issues raised by the Review that we think are important about the context in which scientists operate and in which science contributes to public policy.



## 1.1 Introduction

2. In November 2009, approximately 1000 e-mails from the Climatic Research Unit (CRU) of the University of East Anglia (UEA) were made public without

authorisation.

3. CRU is a small research unit which over the last 30 years has played an important role in the development of climate science, in particular in their work on developing global temperature trends.

4. The e-mails fuelled challenges to the work of CRU, to the reliability of climate science generally, and to the conclusions of the Intergovernmental Panel on Climate Change (IPCC). All this happened shortly before the Copenhagen Summit, and was extensively referred to there.

5. In response, the UEA commissioned two inquiries. The first led by Lord Oxburgh, into the science being undertaken at CRU, has already reported. This document is the report of the second inquiry

– The Independent Climate Change E-mails

Review

–

which examines the conduct of the scientists involved and makes recommendations to the University of East Anglia. Our inquiry addresses a number of important allegations that were made following the e-mail release.

6. The allegations relate to aspects of the behaviour of the CRU scientists, such as their handling and release of data, their approach to peer review, and their role in the public presentation of results.

7. The allegations also include the assertion that actions were taken to promote a particular view of climate change by improperly influencing the process of advising policy makers. Therefore we have sought to understand the significance of the roles played by those involved from CRU and of the influence they had on

the relevant outcomes.

8. The Review examines the honesty, rigour and openness with which the CRU scientists have acted. It is important to note that we offer no opinion on the validity of their scientific work. Such an outcome could only come through the normal processes of scientific debate and not from the examination of e-mails or

from a series of interviews about conduct.CHAPTER 1: EXECUTIVE SUMMARY

11 

## 1.2 The Review Process

9. The approach taken by the Review was to identify and investigate the allegations to which the e-mails gave rise. This reflected our reading of the emails and the

comments made on them. An online consultation was undertaken to ensure that
the Team‘s initia l analysis of the allegations and concerns was sound. The method of investigation is explained in the relevant Chapters and Appendices to the report. The Review‘s evidence base is published on the website, which it intends

to archive.

10. In addressing the allegations about CRU‘s impact on climate science , we sought evidence to place these into perspective:

On handling global temperature data, we went to global primary sources and

tested how data was handled. On tree-ring temperature reconstructions, we looked at the overall picture painted in Chapter 6 of the Fourth Assessment Report of the IPCC in 2007 (AR4) and examined the influence of CRU. On peer review, we sought independent input (from the Editor of The Lancet ) on how the system works, to provide a context for our judgement. On influencing the IPCC process, we sought advice from the Review editors on the role individual contributors can play.

11. This work provided a context in which we considered the evidence about the specific allegations made in the submissions and identified in our interviews with

CRU and others.

12. Reflecting this approach, the report and conclusions are set out as follows. The heart of the report lies in Chapters 6 through 10 where the important allegations arising from the e-mail release are examined. Chapters 2 and 3 contain introductory material, 4 deals with the body of e-mails and 5 presents important contextual material. The report concludes with Chapter 11 on other governance

issues.



## 1.3 Findings

13. Climate science is a matter of such global importance, that the highest standards of honesty, rigour and openness are needed in its conduct. On the specific allegations made against the behaviour of CRU scientists, we find that their rigour and honesty as scientists are not in doubt.

14. In addition, we do not find that their behaviour has prejudiced the balance of advice given to policy makers. In particular, we did not find any evidence of behaviour that might undermine the conclusions of the IPCC assessments.

15. But we do find that there has been a consistent pattern of failing to display the proper degree of openness , both on the part of the CRU scientists and on the part of the UEA, who failed to recognise not only the significance of statutoryCHAPTER 1: EXECUTIVE SUMMARY

12 requirements but also the risk to the reputation of the University and, indeed, to the credibility of UK climate science.



## 1.3.1 Land Station Temperatures

16. On the allegation of withholding temperature data, we find that CRU was not in a position to withhold access to such data or tamper with it. We demonstrated that any independent researcher can download station data directly from primary sources and undertake their own temperature trend analysis.

17. On the allegation of biased station selection and analysis, we find no evidence

of bias. Our work indicates that analysis of global land temperature trends is robust to a range of station selections and to the use of adjusted or unadjusted data. The level of agreement between independent analyses is such that it is highly unlikely that CRU could have acted improperly to reach a predetermined outcome. Such action would have required collusion with multiple scientists in various independent organisations which we consider highly improbable.

18. On the allegation of withholding station identifiers we find that CRU should have made available an unambiguous list of the stations used in each of the versions of the Climatic Research Unit Land Temperature Record (CRUTEM) at the time of publication. We find that CRU‟s responses to reasonable requests for information were unhelpful and defensive.

19. The overall implication of the allegations was to cast doubt on the extent to which CRU‟s work in this area could be trusted and should be relied upon

and we find no evidence to support that implication.



## 1.3.2 Temperature Reconstructions from Tree Ring Analysis

20. The central implication of the allegations here is that in carrying out their work, both in the choices they made of data and the way in which it was handled, CRU scientists intended to bias the scientific conclusions towards a specific result and to set aside inconvenient evidence. More specifically, it was implied in the allegations that this should reduce the confidence ascribed to the conclusions i n Chapter 6 of the IPCC 4 th Report, Working Group 1 (WG1).

21. We do not find that the way that data derived from tree rings is described and presented in IPCC AR4 and shown in its Figure 6.10 is misleading. In particular, on the question of the composition of temperature reconstructions, we found no evidence of exclusion of other published temperature reconstructions

that would show a very different picture. The general discussion of sources of uncertainty in the text is extensive, including reference to divergence. In this respect it represented a significant advance on the IPCC Third Assessment Report (TAR).

22. On the allegation that the phenomenon of “divergence” may not have been properly taken into account when expressing the uncertainty associated with reconstructions, we are satisfied that it is not hidden and that theCHAPTER 1: EXECUTIVE SUMMARY

13 subject is openly and extensively discussed in the literature, including CRU

papers.

23. On the allegation that the references in a specific e- mail to a „trick‟ and to

„hide the decline‟ in respect of a 1999 WMO report figure show evidence of intent to paint a misleading picture, we find that, given its subsequent iconic significance (not least the use of a similar figure in the IPCC Third Assessment Report), the figure supplied for the WMO Report was misleading. We do not find that it is misleading to curtail reconstructions at some point per se , or to splice data, but we believe that both of these procedures should have been made plain

– ideally in the figure but certainly clearly

described in either the caption or the text.

24. On the allegations in relation to withholding data, in particular concerning the small sample size of the tree ring data from the Yamal peninsula, CRU did not withhold the underlying raw data (having correctly directed th e single request to the owners). But it is evidently true that access to the raw data was not simple until it was archived in 2009 and that this delay can rightly be criticized on general principles. In the interests of transparency, we believe that CRU should have ensured that the data they did not own, but on which their publications relied, was archived in a more timely way.



## 1.3.3 Peer Review and Editorial Policy

25. On the allegations that there was subversion of the peer review or editorial process we find no evidence to substantiate this in the three instances

examined in detail. On the basis of the independent work we commissioned (see Appendix 5) on the nature of peer review, we conclude that it is not uncommon for strongly opposed and robustly expressed positions to be taken up in heavily contested areas of science. We take the view that such behaviour does not in general threaten the integrity of peer review or publication.



## 1.3.4 Misuse of IPCC Process

26. On the allegations that in two specific cases there had been a misuse by CRU scientists of the IPCC process, in presenting AR4 to the public and policy makers, we find that the allegations cannot be upheld. In addition to taking evidence from them and checking the relevant records of the IPCC process, we have consulted the relevant IPCC review Editors. Both the CRU scientists were part of large groups of scientists taking joint responsibility for the relevant IPCC Working Group texts, and were not in a position to determine individually the final wording and content.CHAPTER 1: EXECUTIVE SUMMARY

14 

## 1.3.5 Compliance with the Freedom of Information Act (FoIA) and the Environmental Information Regulations (EIR)

27. On the allegation that CRU does not appear to have acted in a way consistent with the spirit and intent of the FoIA or EIR, we find that there was unhelpfulness in responding to requests and evidence that e-mails might have been deleted in order to make them unavailable should a subsequent request be made for them. University senior management should have accepted more responsibility for implementing the required processes for FoIA and EIR compliance.



## 1.3.6 Other Findings on Governance

28. Given the significance of the work of CRU, UEA management failed to recognise in their risk management the potential for damage to the University‟s reputation fuelled by the controversy over data access.



## 1.4 Recommendations

29. Our main recommendations for UEA are as follows:

Risk management processes should be directed to ensuring top management engagement in areas which have the potential to impact the reputation of the university. Compliance with FoIA/EIR is the responsibility of UEA faculty leadership and ultimately the Vice-Chancellor. Where there is an organisation and documented system in place to handle information requests, this needs to be owned, supported and reinforced by University leadership.
CRU should make available sufficient information, con current with any publications, to enable others to replicate their results.



## 1. 5 Broader Issues

30. Our work in conducting the Review has led us to identify a number of issues relevant not only to the climate science debate but also possibly more widely, on which we wish to comment briefly.

31. The nature of scientific challenge. We note that much of the challenge to CRU‘s work has not always followed the conventional scientific method of checking and seeking to falsify conclusions or offering alternative hypotheses for peer review and publication. We believe this is necessary if science is to move on, and we hope that all those involved on all sides of the climate science debate will adopt this approach.

32. Handling Uncertainty

– where policy meets science. Climate science is an area that exemplifies the importance of ensuring that policy makers

–CHAPTER 1: EXECUTIVE SUMMARY

15 particularly Governments and their advisers, N on -Governmental Organisation s and other lobbyists

– understand the limits on what scientists can say and with what degree of confidence. Statistical and other techniques for explaining uncertainty have developed greatly in recent years, and it is essential that they are properly deployed. But equally important is the need for alternative viewpoints to be recognized in policy presentations, with a robust assessment of their validity, and for the challenges to be rooted in science rather than rhetoric.

33. Peer review - what it can/cannot deliver. We believe that peer review is anessential part of the process of judging scientific work, but it should not be overrated as a guarantee of the validity of individual pieces of research, and the significance of challenge to individual publication decisions should be not exaggerated.

34. Openness and FoIA. We support the spirit of openness enshrined in the FoIA and the EIR. It is unfortunate that this was not embraced by UEA, and we make recommendations about that. A well thought through publication scheme would remove much potential for disruption by the submission of multiple requests for information. But at the level of public policy there is need for further thinking about the competing arguments for the timing of full disclosure of research data and associated computer codes etc, as against considerations of confidentiality

d uring the conduct of research. There is much scope for unintended

consequen ces that could hamper research: US experience is instructive. We

recommend that the ICO should initiate a debate on these wider issues.

35. Handling the blogosphere and non traditional scientific dialogue. One of the most obvious features of the climate change debate is the influence of the blogosphere. This provides an opportunity for unmoderated comment to stand alongside peer reviewed publications; for presentations or lectures at learned conferences to be challenged without inhibition; and for highly personalized critiques of individuals and their work to be promulgated without hindrance. This is a fact of life, and it would be foolish to challenge its existence. The Review team would simply urge all scientists to learn to communicate their work in ways that the public can access and understand. That said, a key issue is how scientists should be supported to explain their position, and how a public space can be created where these debates can be conducted on appropriate terms, where what is and is not uncertain can be recognised.

36. Openness and Reputation. An important feature of the blogosphere is the extent to which it demands openness and access to data. A failure to recognise this and to act appropriately, can lead to immense reputational damage by feeding allegations of cover up. Being part of a like minded group may provide no defence. Like it or not, this indicates a transformation in the way science has to be conducted in this century.

37. Role of Research Sponsors . One of the issues facing the Review was the release of data. At various points in the report we have commented on the formal requirements for this. We consider that it would make for clarity for researchers if funders were to be completely clear upfront in their requirements for the release of data (as well as its archiving, curation etc).CHAPTER 1: EXECUTIVE SUMMARY

16 38. The IPCC. We welcome the IPCC‘s decision to review its processes, and can only stress the importance of capturing the range of viewpoints and reflecting appropriately the statistical uncertainties surrounding the data it assesses. Our conclusions do not make a judgement on the work of IPCC, though we acknowledge the importance of its advice to policy makers.CHAPTER 1: EXECUTIVE SUMMARY

17CHAPTER 2: INTRODUCTION

18 

## CHAPTER 2: INTRODUCTION

2.1 Background

1. Prior to the 1960‘s there had been little investigation of past climatic changes and variability, except by geologists and botanists and on geological timescales. The Climatic Research Unit (CRU) in the UEA, established in 1972, was one of the first institutes in the world to address the science of climate change on more recent

timescales. Its objective was ―to establish the past record of climate over as much of the world as possible, as far back in time as was feasible, and in enough detail to recognis e the basic processes, interactions, and evolutions in the earth‘s fluid envelopes and those involving the earth‘s crust and its vegetation cover‖. A useful history of its work is available on its website: http://www.cru.uea.ac.uk/cru/about/history.

2. CRU is a small organisation: at present there are around 16 staff, with 3.5 established posts and the rest postgraduate students and post-doctoral researchers. Two of the main areas of its work are the focus of this review: the development of methodologies for calculating the extent to which the average temperature of the

earth ‘ s land masses is changing, using instrumental temperature measurements made over the past 160 years (the period for which reliable measurements have been available); and the estimation of the earth ‘ s temperature over the last millennium, using tree ring data as a proxy for temperature.

3. One of CRU‘s most important contributions to climate science is the production of a land based, gridded temperature data set showing how the temperature has varied year by year since 1850 relative to the 1961 to 1990 average. This work was started in 1978 and continues today. CRU is now one of a number of organisations working in this area. Others carrying out similar development of temperature records include the Goddard Institute for Space Studies (GISS) and the National Climatic Data Center (NCDC).

4. Over the period that CRU has been in existence and especially over the last 20 years, there has been a transformation in the importance attached to climate science. One clear indication of this was the creation of the Intergovernmental Panel on Climate Change (IPCC) in 1988, a body established by the United Nations Environment Programme (UNEP) and the World Meteorological Organization (WMO) ―to provide the world with a clear scientific view on thecurrent state of climate change and its potential environmental and socioeconomic consequences‖. It assesses the most recent scientific, technical and socio-economic information produced worldwide that is relevant to the understanding of climate change. Members of CRU have played several significant roles on the IPCC. In its successive assessment reports the IPCC has sought to achieve a scientific consensus, but many continue to challenge the basis

of its work and its conclusions.

5. The pioneering work conducted by CRU therefore began to assume a great deal of significance in the international debate that surrounded climate science. Not surprisingly, given the enormous level of public interest in the subject, the debateCHAPTER 2: INTRODUCTION

19 soon moved out of the confines of climate scientists and became highly polarized in websites, journals and conferences across the world. As a result, the work conducted by CRU became the focus of intense scrutiny and challenge with multiple demands from both fellow scientists and laymen for background

information and data.

6. In November 2009, the nature of the debate and challenge took on a whole new significance when approximately 1000 of CRU‘s e-mails were made public

without authorization. Th is material fuelled more challenges to their work, to the reliability of climate science generally, and to the conclusions of the IPCC. All this happened shortly before the Copenhagen Summit, and was extensively

referred to there.



## 2.2 The Review

7. The material in the e-mails led to a set of allegations against the leading members of CRU, focusing on whether CRU had operated in accordance with best scientific practice at the relevant time, and in particular: whether data had been manipulated or suppressed; whether peer review and the dissemination of findings had bee n properly handled; and whether CRU had complied with the Freedom of

Information Act.

8. In response to this, the Vice-Chancellor of the UEA established the Independent Climate Change E-mails Review, to be led by Sir Muir Russell. Sir Muir was given complete freedom to develop the terms of reference as necessary and to assemble an appropriate team and appropriate support; and was asked if possible to report in spring 2010. The terms of reference are in Chapter 3 and details of the team members are in Appendix 1.

9. Two other formal inquiries have addressed the matter. First, the House of Commons Science and Technology Committee held a hearing on 1 March 2010, reporting on 31 March 2010 1 . Secondly, the Vice-Chancellor commissioned

Lord Oxburgh to review CRU ‘ s scientific output. The Oxburgh Scientific Assessment Panel and this Review proceeded entirely independently, though the latter took steps to ensure, following the House of Commons Science and Technology Committee ‘ s Report, that Lord Oxburgh was aware of the approach it was taking to issues that might bear on his work. Lord Oxburgh ‘ s report was published on 14 April 2010 2

.

10. In addition investigations have been initiated by the police, looking at the

circumstances of the unauthoris ed disclosure of the e-mails; and by the ICO

, looking at compliance with the Data Protection Act, the FoIA and the EIR.

11. Responding to a joint request by the Chairman of the IPCC and the Secretary-

1 Science and Technology Committee - Eighth Report: The disclosure of climate data from the Climatic Research Unit at the University of East Anglia

2 Report by Lord Oxburgh‘s Science Assessment Panel http://www.uea.ac.uk/mac/comm/media/press/CRUstatements/S APCHAPTER 2: INTRODUCTION

20 General of the United Nations, an independent review of the IPCC‘s processes and procedures is now under way by the InterAcademy Council (IAC), chaired by economist Harold T. Shapiro, former president of Princeton University.

12. Although the focus of this Review is quite clearly on the behaviour of a number of scientists, it is also apparent that there are many wider issues raised by the events described above. These include issues such as the workings of the peer review system, the reporting of uncertainty in the translation of scientific findings into policy, the handling and release of data, and the role of research funders. As a

contribution to the wider scientific debate about the conduct of science in the 21 st
century, this Review also considers these questions.CHAPTER 3: TE RMS OF REFERENCE AND METHOD OF ENQUIRY

21CHAPTER 3: TE RMS OF REFERENCE AND METHOD OF ENQUIRY

22 

## CHAPTER 3: TERMS OF REFERENCE AND METHOD OF ENQUIRY

3.1 Terms of Reference

1. The Review‘s terms of reference are as follows:

― The Independent Review will investigate the key allegations that arose from a

series of hacked* e-mails from CRU. The Review will:

Examine the hacked e-mail exchanges, other relevant e-mail exchanges and any other information held at CRU to determine whether there is any evidence of the manipulation or suppression of data which is at odds with acceptable scientific practice and may therefore call into question any of the research

outcomes.

Review CRU ‘ s policies and practices for acquiring, assembling, subjecting to peer review and disseminating data and research findings, and their compliance or otherwise with best scientific practice.

Review CRU ‘ s compliance or otherwise with the University ‘ s policies and practices regarding requests under the Freedom of Information Act (  ̳ the

Fo IA ‘ ) and the Environmental Information Regulations (  ̳ the EIR ‘ ) for the

release of data. Review and make recommendations as to the appropriate management, governance and security structures for CRU and the security, integrity and

release of the data it holds. ‖

*Note: The word  ̳ hacked ‘ as contained in the Review ‘ s terms of reference has been challenged in submissions to the Review, on the basis that the means by which the unauthorized disclosure of the e-mails was made has not been established. This matter is subject to police enquiries and the Review has

made no judgment on the question.



## 3.2 Method of Enquiry

2. The Review sought to operate in an open and transparent way. This is described in

the  ̳ Approach and Work Plan ‟ paper (Appendix 2) made public at the launch at the Science Media Centre in February 2010. The Review called for submissions,which it undertook to publish on its website: http://www.ccereview.org/Evidence.php

3. To provide a focus for submissions the Review produced and placed on its

website an  ̳ Issues for Examination ‟ paper (Appendix 3) which addresses the main allegations made against the members of CRU. This paper set out our initial understanding based on a preliminary reading of the e-mails. Approximately 100 submissions were then received, including one from CRU itself . Some provided further background considered relevant by the authors, but there was no suggestion in the submissions that the original statement of the issues was fundamentally misdirected. As its work progressed, the team kept the issues paper constantly under review and gradually developed a sharper focus on the keyCHAPTER 3: TE RMS OF REFERENCE AND METHOD OF ENQUIRY

23 allegations.

4. Thus this report is structured to address the principal allegations. It is important torecognise that this is not a detailed inquiry into the precise meaning of every email.

5. The team proceeded to investigate the allegations by interviewing members of CRU and others from the University. We considered that the nature of our inquiry was such that holding public hearings to gather evidence, as some had urged, would be unlikely to add significant value over and above the written record. Nor have we produced transcripts of the interviews. This is because our conclusions are founded on information given in submissions and at interviews relating to facts that can be checked and referenced, rather than on interview testimony as such. The team found that this process helped it follow up key points, leading to supplementary submissions and references.

6. The team did not carry out interviews other than with CRU and other UEA staff (apart from preliminary discussions with ICO and the police and interviews with two relevant IPCC Review Editors). We recognise that natural justice requires that those in respect of whom findings will be made should have an opportunity to

be heard: this does not apply to the authors of submissions and other parties, in respect of whom the Review has made no findings.

7. Under our p ublication policy notes of the t eam‘s meetings and interviews have been placed on the website. Interview notes have been checked for factual accuracy with those involved. Only those that are potentially sensitive, such as notes of discussions with the police, have been withheld. The Team has sought to

pu blish all the submissions received unless they were explicitly described as confidential, subject only to the redaction, on legal advice, of content the publication of which could have led to legal action. In addition, on e submission has been withheld on legal advice. The Review has invited the author to make it available directly to enquirers. It is a matter for the author whether to proceed in this way. This submission has been given full consideration.

8. A full list of submissions and details of meetings can be found in Appendix 4.

9. It is important to note that the allegations relate to aspects of the behaviour of the

CRU scientists and not to the content of their work and hence this Review addressed scientific questions only to the extent necessary to place this behaviour

in context.

10. On the subject of peer review, in addition to investigating the specific allegations the team commissioned a contextual paper from a distinguished Journal editor

– Dr Richard Horton of The Lancet

– on which Elizabeth Wager, Chair of the Committee on Publication Ethics (COPE), was invited to comment in a personal capacity. This material forms Appendix 5 and is referred to Chapters 5 and 8.

11. The Review maintained contact with the police and the ICO. It was important to ensure that the work of the team did not compromise their statutory responsibilities, and also to avoid duplication of effort. The t eam ‘ s findings andCHAPTER 3: TE RMS OF REFERENCE AND METHOD OF ENQUIRY

24 recommendations respect this division of responsibilities and the fact that investigations are continuing.

12. The Review intends that the website and the submissions and evidence published on the website should be archived by the British Library Web Archive.CHAPTER 3: TE RMS OF REFERENCE AND METHOD OF ENQUIRY

25CHAPTER 4: CONTEXT OF THE E-MAILS

26 

## CHAPTER 4: CONTEXT OF THE E-MAILS

4.1 Characterising the E-mails

1. This Chapter gives an overview of the e-mails and seeks to relate their content to key events happening in the climate change world, thus giving context to the

criticism of CRU.

2. The information released comprises a very small (less than 0.3%) subset of files which were held on the back-up server at CRU, which include e-mails and other

documents

–

such as text files, Word documents, Excel spreadsheets, PDF documents, and computer code.

3. The focus of this Chapter is e-mails which spanned the period 7 th March 1996 to

12 th November 2009. The  ̳ primary ‘ e-mails number 1073 in total with 166

authors. There are more e-mails and authors if the associated e-mail chains are included. When printed on A4 paper the e-mails run to 3,375 pages and contain many embedded duplicates. Self -evidently each of the primary e-mails was either sent by or received by CRU members, but this is not the case for many of theassociated e-mail chains. Those who authored the largest numbers of primary emails are as follows.

Autho r Number Role

Philip Jones 174 Director, CRU, UEA and Coordinating Lead Author

IPCC 4 th Assessment Report

Michael

Mann 140 Director, Earth System Science Centre, Pennsylvania State University (from 2005), and Lead Author IPCC

3 rd Assessment Report

Keith Briffa 117 Professor, CRU, UEA and Lead Author IPCC 4 th
Assessment Report

Jonathan Overpeck 90 Institute Director, University of Arizona and Coordinating Lead Author IPCC 4 th Assessment Report

Tim Osborn 59 Academic Fellow, CRU, UEA and Contributing

Author IPCC 4 th Assessment Report

Ben Santer 51 Researcher, Lawr ence Livermore National Lab, US and Contributing Author IPCC 4 th Assessment Report

Tom Wigley 35 Scientist, University Corporation for Atmospheric Research, Contributing Author IPCC 4 th Assessment Report and a former Director of CRU

4. The e-mails relate to a number of the major developments in recent climate science from the Kyoto Summit in December 1997, through the 4 th Assessment Report of the IPCC in 2007, to the run-up to the 15 th Conference of the Parties (COP) in Copenhagen. Since CRU played a significant role in providing scientific input to these events, the release of the emails resulted in reduced globalCHAPTER 4: CONTEXT OF THE E-MAILS

27 confidence in climate science and more specifically in the findings of the IPCC.



## 4.2 The Timeline

5. The  ̳ story ‘ which underlies the selected e-mails, and hence is reflected in them, is

summarised in the table and text which follow.

Year Event

1997 Kyoto Summit

1998 MBH98 1 including what has become known as the ― Hockey Stick ‖

1999 MBH99 2 reconstruction extended to 1000 AD

WMO Statement on the Status of the Global Climate in 1999 3

2001 IPCC 3 rd Assessment Report 4

2002 Esper et al 2002 5

2003 Soon & Baliunas 2003 6

, McIntyre &McKitrick 2003 7 criticism of MBH

Mann et al 2003 8

, 9

2004 von Storch 2004 10 questioning the statistical methods used in MBH

Launch of RealClimate 11 website

1 M.E. Mann, R.S. Bradley and M.K. Hughes, ― Global-scale temperature patterns and climate forcing over the past six centuries ‖

, Nature 392, 779-787 23 April (1998)

2 M.E. Mann, R.S. Bradley and M.K. Hughes, ― Northern hemisphere temperatures during the past millennium: Inferences, uncertainties, and limitations ‖

, Geophysical Research Letters , Volume 26. No. 6, pp. 759-762 (1999)

3 http://www.wmo.ch/pages/prog/wcp/wcdmp/statemnt/wmo913.pdf

4 http://www.grida.no/publications/other/ipcc_tar/

5 J. Esper, E.R. Cook, F.H. Schweingruber, ― Low-Frequency Signals in Long Tree-Ring Chronologies for Reconstructing Past Temperature Variability ‖

, Science , Volume 295, 22 March (2002) pp. 2250-

2253

6 W. Soon, S. Baliunas, ― Proxy climatic and environmental changes of the past 1000 years ‖

, Climate

Research Volume 23: 89-110, January 2003

7 S. McIntyre, R. McKitrick, ― Corrections to the Mann et. al. (1998) proxy data base and Northern hemispheric average temperature series ‖

, Energy & Environment

, Volume 14, No. 6 (2003) pp. 751

- 771

8 M.E. Mann and P.D. Jones, ― Global surface temperatures over the past two millennia ‖

, Geophysical

Research Letters Volume 30, No. 15, 1820, August 2003

9 M.E Mann, C.M Ammann, R.S. Bradley, K.R. Briffa, T.J. Crowley, M.K. Hughes, P.D. Jones, M. Op penheimer, T.J. Osborn, J.T. Overpeck, S. Rutherford, K.E. Trenberth, T.M.L. Wigley, ― On past temperatures and anomalous late 20 th century warmth ‖

, Eos , 84, 256-258, 2003.

10 H. von Storch, E. Zorita, J.M. Jones, Y. Dimitriev, F. González-Rouco, S.F.B. Tett

, ― Reconstructing Past Climate from Noisy Data ‖

, Science , 22 October 2004, Volume 306. no. 5696, pp. 679

– 682

11 http://www.realclimate.org/CHAPTER 4: CONTEXT OF THE E-MAILS

28 Year Event (continued)

2005 UK Freedom of Information Act 12

Launch of ClimateAudit 13 website McIntyre &McKitrick 2005 14 criticism of MBH

Rutherford et al 2005 15 defence of MBH

2006 US National Research Council review 16
US Committee on Energy & Commerce (Wegman) statistical review 17

2007 IPCC 4 th Assessment Report 18

Wahl & Ammann 2007 19 defence of MBH

2009 15 th COP in Copenhagen

CRU‘s work assumed progressively greater significance as it became apparent that both the global temperature record and tree ring data it was collecting and analysing were being cited prominently in the work of the

IPCC.

Building in part on the work of CRU, Mann, Bradley & Hughes (MBH) published a paper in Nature in 1998 20 which sought to reconstruct historic temperatures back to 1400 AD, well before significant numbers of instrumental climate records began in around 1850. They usedmeasurements including those from tree rings, ice cores and corals, socalled proxy data, which can reflect local temperature changes. This paper was updated in 1999 21 with the reconstruction extended back to 1000 AD in the Northern Hemisphere. The resulting temperature profile

had a  ̳ hockey stick ‘ shape. In particular it reduced the significance of what is termed the Medieval Warm Period (MWP), that is believed to have reached its peak between about 950 and 1100 AD, and the Little Ice Ag e (LIA) (from about 1500 to 1850 AD, and centered around 1700 AD), relative to previous representations. Temperatures over the last 50 years appeared unprecedented in the past thousand years. These data were used to create an iconic representation of anthropogenic global warming, as supplied by Jones, in the 1999 World Meteorological Organisation for its

12 http://www.ico.gov.uk/what_we_cover/freedom_of_information.aspx

13 http://climateaudit.org/

14 S. McIntyre, R. McKitrick, ― Hockey sticks, principal components, and spurious significance ‖

, Geophysical Research Letters

, Volume 32, No. 3 (2005)

15 S. Rutherford, M.E. Mann, T.J. Osborn, R.S. Bradley, K.R. Briffa, M.K. Hughes, P.D. Jones,

― Proxy-based Northern Hemisphere surface temperature reconstructions: sensitivity to methodology, predictor network, target season and target domain ‖

, Journal of Climate (2005); 18: 2308- 2329

16 Surface Temperature Reconstructions for the Last 2,000 Years http://www.nap.edu/catalog.php?record_id=11676

17 Ad Hoc Committee Report on the  ̳Hockey Stick‘ Global Climate Reconstruction (2006) for the US Committee on Energy and Commerce

18 http://www.ipcc.ch/

19 E.R. Wahl, C.M. Ammann

, ― Robustness of the Mann, Bradley, Hughes reconstruction of Northern Hemisphere surface temperatures: examination of criticisms based on the nature and processing of proxy climate evidence ‖ Climatic Change , Volume 85, Numbers 1-2, pp.33-69, November 2007

20 Ibid. 1

21 Ibid. 2CHAPTER 4: CONTEXT OF THE E-MAILS

29 Statement on the Status of the Global Climate in 1999 22 . They were also

referenced in the Third Assessment Report (TAR) of the IPCC in 2001 23

.

The IPCC process occurs over a number of years, with drafts being produced periodically, leading up to the final report and summary for policy makers (SPM). The process is clearly documented 24 with extensive voluntary involvement of many eminent scientists in authoring and reviewing chapters of the report. Key players in CRU were involved inthe development of both the TAR and AR4 in 2007 and many of the emails relate to their involvement in the process.

Questions were increasingly asked about MBH. In 2002 Esper published

a paper 25 which suggested that MBH had underestimated the strength of the MWP, but it was Soon & Baliunas ‘ s (S&B) papers published in

Climate Research 26 and with minor amendments in Energy &

Environmen t (E&E) 27 which first challenged MBH ‘ s results directly. An e-mail by Osborn circulating S&B ‘ s 2003 paper 28 is one of the most duplicated in the released e-mails triggering nine of the subsequently

selected e-mail chains. S&B ‘ s papers were followed in 2003 by another paper by McIntyre and McKitrick (M & M) 29 also critical of MBH. At issue in the whole exchange of e-mails were the following key scientific

issues:-

(i) the selection of tree ring data (particularly those from Yamal

and the Polar Urals); (ii) the statistical methods for extracting information; (iii) the record of local and hence global temperatures (especially the impact of the urban heat island (UHI) effect); (iv) relating proxy records to instrumental temperature records; and (v) uncertainties in comparing recent warmth with that over the last 2000 years.

Prior to the publication of S&B ‘ s papers in 2003 those critical of MBH had not had a paper published in a mainstream journal. The publications

in Climate Research and Energy & Environment were significant not only because they challenged MBH but also because they had been peer reviewed. Not only were rebuttals published in 2003 by Mann, Jones, Briffa, Osborn et al 30

, 31

, but also the process of peer review at Climate

Research was questioned. The editor, de Freitas, sought initially to

22 Ibid. 3

23 Ibid. 4

24 http://www.ipcc.ch/organization/organization_procedures htm

25 Ibid. 5

26 Ibid. 6

27 W. Soon, S. Baliunas, C. Idso, S. Idso. D.R. Legates, ― Reconstructing climatic and environmental changes of the past 1000 years: a reappraisal ‖

, Energy & Environment , Volume 14: Nos. 2 & 3, 2003, pp. 233- 296

28 Ibid. 6

29 Ibid. 7

30 Ibid. 8

31 Ibid. 9CHAPTER 4: CONTEXT OF THE E-MAILS

30 defend himself as the e-mail extract below shows but he ultimately resigned as did members of the editorial board of Climate Research. The matter is discussed in detail in Section 8.3 of Chapter 8.

In an e-mail dated 18/6/03 (1057944829.txt) de Freitas wrote:

“I have spent a considerable amount of my time on this matter and had my integrity attacked in the process. I want to emphasize that the people leading this attack are hardly impartial observers. Mike himself refers to "politics" and political incitement involved. Both Hulme and Goodess are from the Climate Research Unit of UEA that is not particularly well known for impartial views on the climate change debate. The CRU has a large stake in climate change research funding as I understand it pays the salaries of most of its staff.”

In 2004 von Storch 32 questioned the statistical methods used in MBH and, at around the same time, the RealClimate 33 and Climate Audit 34 websites were launched, the former defending and the latter critical of majority climate science. This marked a new phase of more public climate science debate and criticism, and was seen as an unwelcome development by those in CRU, who wanted criticism to follow the normal channels of private peer review and publication. This view remained relatively unchanged for over a decade, as shown by the following:

In an e-mail dated 6/5/99 (926031061.txt) Jones wrote:

“I must admit to having little regard for the Web. Living over here makes that easier than in the US - but I would ignore the so-called skeptics until they get to the peer-review arena. I know this is harder for you in the US and it might become harder still at your new location. I guess it shows though that what we are doing in [sic] important. The skeptics are fighting a losing battle.”

In an e-mail dated 30/9/09 (1254323180.txt) Jones wrote:

“Another issue is science by blog sites - and the then immediate response mode. Science ought to work through the peer-review system..... sure you've said all these things before. We're getting a handful of nasty emails coming and requests for comments on other blog sites.”

Critics of CRU claimed that the y were unable to reproduce CRU‘s work

du e to a lack of access to data. The UK EIR and FoIA 35 should have encouraged the release of information, but this does not seem to have been the initial result. The volume of requests grew over time, more formally under the FoIA starting in 2007 and spectacularly in 2009, particularly related to details of the CRUTEM land temperature dataset s

32 Ibid. 10

33 Ibid. 11

34 Ibid. 13

35 Ibid. 12CHAPTER 4: CONTEXT OF THE E-MAILS

31 and the role CRU played in the development of IPCC AR4.

In an email dated 7/5/04 (1083962601.txt) Jones wrote:

“Many of us in the paleo field get requests from skeptics (mainly a guy called Steve McIntyre in Canada) asking us for series. Mike and I are not sending anything, partly because we don't have some of the series he wants, also partly as we've got the data through contacts like you, but mostly because he'll distort and misuse them. Despite this, Mike and I would like to make as many of the series we've used in the RoG [Reviews of Geophysics] plots available from the CRU web page.”

We do not suggest that the allegations made against McIntyre are correct.

In the run up to IPCC AR4 further papers were published criticising (McIntrye & McKitrick 2005 36 ) and defending MBH (Rutherford 2005 37

, Wahl and Ammann 2007 38 ). In parallel, political interest in AR4 grew and in the United States of America two reviews were commissioned, one by the National Research Council 39 into surface temperature reconstructions for the last 2000 years and the other by the Committee on Energy and Commerce (Wegman) 40 into the quality of the statistical analysis used in MBH. These reviews were critical of MBH; nevertheless

the AR4 SPM 41 contained the following statement.

“Average Northern Hemisphere temperatures during the second half of

the 20 th century were very likely higher than during any other 50-year period in the last 500 years and likely the highest in at least the past 1300 years.”

In the aftermath of AR4, attempts were made to demonstrate that the IPCC process had been abused by CRU members. Jones was accused of attempting to exclude reference to a 2004 paper by McKitrick and

Michaels 42 , and Briffa of including reference to Wahl 2007 43 before it was in publication, contrary to IPCC rules.

These accusations were accompanied by Freedom of Information FOI requests. In an email dated 10/12/08 (1228922050.txt) Jones wrote:

“Anyway requests have been of three types - observational data, paleo data and who made IPCC changes and why. Keith has got all the latter

–

36 Ibid. 14

37 Ibid. 15

38 Ibid. 19

39 Ibid. 16

40 Ibid. 17

41 AR4 WG1 Summary for policymakers: http://www.ipcc.ch/publications_and_data/ar4/wg1/en/spmsspm-a-palaeoclimatic.html

42 R. McKitrick and P.J Michaels, ― A test of corrections for extraneous signals in gridded surface temperature data ‖

, Climate Research 26, 159-173, 2004

43 Ibid. 19CHAPTER 4: CONTEXT OF THE E-MAILS

32 and there have been at least 4. We made Susan aware of these - all came from David Holland. According to the FOI Commissioner's Office, IPCC is an international organization, so is above any national FOI. Even if UEA holds anything about IPCC, we are not obliged to pass it on, unless it has anything to do with our core business - and it doesn't!”



## 4.3 What the E-mails Tell Us

6. Since the communication was assumed to be private, it was generally informal, using slang, jargon and acronyms. Now that the e-mails have become public, some are doubtless regretted by their authors. Indeed, some submissions have

charac terised them as  ̳unprofessional‘, or as evidence of CRU‘s contribution to a  ̳poisoned atmosphere‘ in climate science. The question is whether this was unusual, or simply characteristic of the normal mode of expression in e-mail

communication.

7. This is a well known issue in social psychology and communication. Early

authoritative works 44

, 45

, 46 , reinforced by later authors, explore normative language in electronic communication in comparison with other written and oral forms. They demonstrate that e-mail communication is less inhibited than other written and spoken forms, and suggest reasons why. Extreme forms of language are frequently applied to quite normal situations by people who would never use it in

other communication channels.

8. It is also clear from the submissions that it is possible to place different interpretations on the same phrase. In such circumstances, only the original author can really know what their intentions were. For example, on 16/11/99 (942777075.txt) Jones wrote his now infamous e-mai l including the following:

“ I've just completed Mike's Nature trick of adding in the real temps to each series for the last 20 years (ie. from 1981 onwards) amd [sic] from 1961 for Keith's to

hide the decline. ”

McIntyre in his submission 47 to the Review Team, states

“The IPCC “trick” was not a “clever” mathematical method – it was merely the

d eletion of inconvenient data after 1960.”

This compares with Jones‘s own commentary in the UEA submission 48 to the

Review Team.

“The email was written in haste and for a limited and “informed” audience (the people that had provided data). The word “trick” was not intended to imply any

44 Sproull, L. and Kiesler, Sara. 1986. Reducing Social Context Cues: Electronic Mail in Organisational

Communications. Management Science , 32, p.1492- 1512

45 Sproull, L. and Kiesler, S. 1991. Connections. MIT Press, Cambridge MA.

46 Shapiro, N and Anderson, R. 1985. Towards and Ethics and Etiquette for Electronic Mail. Rand Corporation.

47 McIntyre Submission (No.23), 1 March 2010, paragraph 22

48 Climatic Research Unit Submission (No.5), 1 March 2010, 3.1 on p.36CHAPTER 4: CONTEXT OF THE E-MAILS

33 deception, simply the “best way of doing or dealing with something”. The reconstruction from the tree-ring density network was not shown after 1960, and

thus in this sense it is “hidden” – but justifiably so: excluding the anomalous tree-ring density data is justified if the purpose is to illustrate the most likely course of temperature based on a combination of proxy and measured temperatures. Again, no deception was intended or practised.”

9. E-mails are rarely definitive evidence of what actually occurred. They are open to interpretation, but they are also indicative. Having identified specific allegations against CRU, based on the e-mails, the Review then sought to obtain evidence to substantiate or refute these allegations, as described in the subsequent Chapters of this report.

10. The presumption is that e-mails were selected to support a particular viewpoint. Recognising that they were a tiny fraction of those archived, the Review Team sought to learn more about the full contents of the back-up server. This attempt, summarised in Appendix 6, was largely unsuccessful due to the sheer scale of the task and ongoing police investigation.

11. In some instances the leaked e-mails contain statements which might be viewed as supporting the behavior of CRU and countering the allegations in paragraph 9 above. The following extracts from e-mails by Briffa are examples.

E-mail dated 22/9/99 (938018124.txt):

“ the early (pre-instrumental) data are much less reliable as indicators of global temperature than is apparent in modern calibrations that include them and when we don't know the precise role of particular proxies in the earlier portions of

recon struction it remains problematic to assign genuine confidence limits at multidecadal and longer timescales. ”

E-mail dated 3/2/06 (1138995069.txt):

“ we are having trouble to express the real message of the reconstructions - being scientifically sound in representing uncertainty , while still getting the crux of the information across clearly...We have settled on this version (attached) of the Figure which we hoe [sic] you will agree gets the message over but with the rigor required for such an important document. ”

12. On the other hand there are some e-mails which show that CRU members may have gone out of their way to make life difficult for their critics, just as they perceived the critics to be frustrating their work. Some examples follow:

Jones e-mail dated 27/4/05 (1114607213 .txt):

“ I got this email from McIntyre a few days ago. As far as I'm concerned he has the data - sent ages ago. I'll tell him this, but that's all - no code. If I can find it, it is likely to be hundreds of lines of uncommented fortran ! I recall the program did a lot more that just average the series. I know why he can't replicate the results early on - it is because there was a variance correction for fewer series.”CHAPTER 4: CONTEXT OF THE E-MAILS

34 Jones e-mail dated 3/12/08 (1228330629.txt ):

“ The inadvertent email I sent last month has led to a Data Protection Act request sent by a certain Canadian, saying that the email maligned his scientific credibility with his peers! If he pays 10 pounds (which he hasn't yet) I am supposed to go through my emails and he can get anything I've written about him. About 2 months ago I deleted loads of emails, so have very little - if anything at
all.”

Osborn e-mail dated 23/6/08 (1214228874.txt ):

“ I've just had a quick look at CA. They seem to think that somehow it is an advantage to send material outside the formal review process. But *anybody* could have emailed us directly. It is in fact a disadvantage! If it is outside the formal process then we could simply ignore it, whereas formal comments had to be formally considered. Strange that they don't realise this and instead argue for some secret conspiracy that they are excluded from! I'm not even sure if you sent me or Keith anything, despite McIntyre's conviction! But I'd ignore this guy's request anyway. If we aren't consistent in keeping our discussions out of the public domain, then it might be argued that none of them can be kept private. Apparently, consistency of our actions is important.”

13. During the work of the Review an additional concern surfaced, namely that of financial controls, for example see extract below. This is addressed in Chapter 11 (paragraph 18). In an e-mail dated 7/3/96 (826209667.txt) Shiyatov (A Russian tree ring researcher) wrote:

“It is important for us if you can transfer the ADVANCE money on the personal accounts which we gave you earlier and the sum for one occasion transfer (for example, during one day) will not be more than 10,000 USD. Only in this case we can avoid big taxes and use money for our work as much as possible. ”

14. Finding: The extreme modes of expression used in many e-mails are characteristic of the medium. Crucially, the e-mails cannot always be relied upon as evidence of what actually occurred, nor indicative of actual behaviour that is extreme, exceptional or unprofessional.

15. The Chapters which follow address specific behavioural allegations based in particular on a comparatively small number of e-mails. Despite the fact that all the e-mails released are unlikely to be representative of the larger set on the CRU

back -up server, there nevertheless would appear to be a pattern of behaviou r. There is little doubt about the polarisation of views in the world of climate science, which has overstepped the line dividing heated scientific debate from outright hostility. One camp comprises the main authors of the e-mails in paragraph 3, who are acknowledged leaders in majority climate science as indicated by the roles they all played in the IPCC. The other camp is their critics, for whom pejorative terms such as ―prat, dishonest, appalling, rubbish and crap‖CHAPTER 4: CONTEXT OF THE E-MAILS

35 were used by some CRU members to refer to them or their work. More generally the majority climate scientists appear to have been united in their defence against criticism. Whilst perhaps understandable, given the nature and methods of criticism, some of which impugns their personal integrity as well as challenging their work, this may have blinded some CRU members to the possibility of merit therein. Such denial then fuelled yet further antagonism. There needs to be better communication, as well as greater openness enabling more scientific debate. We comment on this in Chapters 5 and 10.CHAPTER 5: THE CHANGING CONTEXT OF MODERN SCIENCE

36 

## CHAPTER 5: THE CHANGING CONTEXT OF MODERN

SCIENCE

1. In this Chapter we review the changing context of modern science as a framework against which CRU‘s practices can be judged.

2. When CRU was created in the early 1970s, climate change was a relatively obscure area of science. But it has developed into an area of great political and public concern. CRU found itself ―in the eye of the storm‖ in generating much of the data necessary to address one of the most important issues of the century.

3. This has come at a time of transformation in the need for openness in the culture of publicly-funded science. It is being driven by a range of pressures including new recommendations from national academies addressing the timely release of data, increased demands for scientific input to public policy, important changes in the law and challenge from the blogosphere.



## 5.1 The Scientific Process

4. Scientific hypotheses and theories are presumed to be provisional: they can be refuted by testing but they cannot be verified as correct or true in an absolute sense. Verification is however possible of the results of the experiments upon which theories are built or that attempt to test a theory. If they can be repeated, and produce the same results, they are said to be validated. It is important to recognise that science progresses by substantive challenges based on rigorously logical, published arguments that present a different view of reality from that which they challenge. Criticising and attacking process and behaviour is not the same as an attack on a scientific hypothesis. Failure in one is not necessarily

failure in the other.

5. Given the nature of our remit, our concern is not with science, whether data has been validated or whether the hypotheses have survived testing, but with behaviour; whether attempts have been made to misrepresent, or ―cherry pick‖ data with the intention of supporting a particular hypothesis, or to withhold data so that it cannot be independently validated, or to suppress other hypotheses to prevent them being put to the test.



## 5.2 The Nature of Climate Science

6. Modern climate science is largely a creation of the last 30-40 years. It involves a wide range of disciplines, including geography, physics, chemistry, a wide variety of earth sciences, palaeo-biology and computer modelling. The synthesis of such diverse sources of evidence generated through wide variety of scientific approaches and traditions has been a major task, in which the IPCC has played an import ant role.

7. One of the main contributions of CRU has been in the laborious assembly of datasets of instrumental measures of climate change and of data relating to tree rings that are proxies for climatic variables.CHAPTER 5: THE CHANGING CONTEXT OF MODERN SCIENCE

37 

## 5.3 Data Accessibility in the Digital A ge

8. Modern digital technologies permit the acquisition and manipulation of very much larger datasets than formerly. To enable proper validation of the conclusions, such datasets must be made freely available, along with details of the associated computational manipulation. It is often difficult and time-consuming to describe
this  ̳meta

- data‘ in ways that permit others to verify experimental results and test the inferences drawn from them. The preservation of such data also presents a novel problem of archiving and stewardship compared with the archives of former

times.

9. An important shift in attitude is represented by a recent report by the US National

Academies 1 , which highlights these issues. It recommends a new approach to the integrity, accessibility, and stewardship of data in publicly-funded science, arguing that researchers should make all research data, methods, and other information underlying the results publicly accessible in a timely manner. These recommendations would require a substantial shift of behaviour amongst many scientists. The e-mails suggest that this would be true for CRU

– for example: Jones: “We have 25 or so years invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it ” 2

.

10. Digital datasets in specific areas of science are increasingly being placed in international archives. The World Data System, run by the International Council

for Scientific Unions 3 , for example, covers a very wide range of data types including the International Tree-Ring Data Bank. The majority of the source data used to create the CRUTEM gridded product is available through the Global Historical Climatology Network (GHCN) managed by the NCDC 4 and the US National Center for Atmospheric Research (NCAR) 5

.

11. Some funders of research require that applications for funding include plans for data archiving. The Natural Environment Research Council (NERC), the UK research council most relevant to the work of CRU, has a policy that requires recipients of grants to offer to deposit with NERC a copy of datasets resulting from the supported research 6

.

12. Over the period under review, data accessibility was also being driven by major changes in the law that ha d important implications for research scientists working in the UK. Whilst the precursors of the Environmental Information Regulations date back to the 1990s, the current EIR came into force on 1 st January 2005, on the same date as the general right of access under the Freedom of Information Act 2000 (FoIA). These are described in more detail in Chapter 10. The EIR applies exclusively within a rather broad definition of environmental and related

1 National Academy of Sciences. 2009. Ensuring the Integrity, Accessibility, and Stewardship of Research Data in the Digital Age . 188pp.

2 Referred to in Sherrington submission (No. 38), 17 February 2010, page 10)

3 http://www.ukssdc.ac.uk/wdcmain/guide/wdcguide html

4 http://www.ncdc noaa.gov/oa/ncdc.html

5 http://www.ncar.ucar.edu/

6 NERC Data Policy (Version February 2010)CHAPTER 5: THE CHANGING CONTEXT OF MODERN SCIENCE

38 information; this area is then excluded from the FoIA. The extent to which both regimes apply to access to information from publicly funded (or part-funded) research is still an evolving area. Whilst it is broadly recognised that these do apply to data and supporting metadata, algorithms, etc. at the point when a research paper is published, how to interpret the legislation and regulations in terms of data held for a long period prior to publication and also pre-publication drafts and other correspondence is work in progress. The  ̳public interest test‘, applicable to both EIR and FoIA, again described in Chapter 10, remains key to such interpretation.



## 5.4 Handling Uncertainty

13. There is a widespread misconception that science produces unequivocal and absolutely precise answers. It does not, and cannot. All scientific results contain uncertainties, and it is important that these are made clear to and are understood by those who use them. There are two fundamental sources of uncertainty in science: uncertainty in measuring a phenomenon and uncertainty in determining causes and causal relationships.

14. Over the period in question, the increasing amount of quantitative data available on climate change has made it easier to assess and represent uncertainty. This can clearly be seen in the representation of what is termed the Medieval Warm Period. Prior to the 1980s this was shown as a single line, without error bars 7 . Subsequent quantitative palaeo-temperature work on tree rings, in which CRU has been a leader, has permitted errors to be assessed as a consequence of the statistical correlations involved (see Chapter 7). Although the recent Oxburgh Scientific

Assessment Panel 8 has been cr itical of some of the statistical work of CRU in relation to the tree ring series, that issue lies beyond our remit. Our concern is whether tree ring series have been improperly selected and whether the uncertainties have been properly presented.

15. The processes of hypothesis, experiment, testing and refutation generally reduce uncertainties about causes and relationships but do not entirely remove them. Some of the conflicts involving CRU have been scientific disagreements about causes and relationships that in principle should be resolved by new data which exposes existing hypotheses to rigorous testing. One of the allegations against members of CRU has been that they have attempted improperly to prevent ideas with which they disagreed from being effectively discussed, for example in relation to the publication of papers (Chapter 8) and in parts of the IPCC process (Chapter 9).

7 e.g. Houghton, J.T., Jenkins, G.J. and Ephraums J.J. 1990. Climate Change: the IPCC Scientific

Assessment . Cambridge University Press.

8 Report of the International Panel set up by the University of East Anglia to examine the research of

the Climatic Research Unit . 2010. Available at: http://www.uea.ac.uk/mac/comm/media/press/CRUstatements/oxburghCHAPTER 5: THE CHANGING CONTEXT OF MODERN SCIENCE

39 

## 5.5 Scientific Journals and the Peer Review Process

16. Allegations against CRU include improper attempts to influence the editorial policy of scientific journals and the peer review process. To help it understand the context, the Review commissioned a contextual paper from a distinguished

Journal editor

–

Dr Richard Horton of The Lancet

– on which Elizabeth Wager,

Chair of CO PE , was invited to comment in a personal capacity. (Appendix 5).

17. Peer review is important because of the way that scientific journals provide the

frame of reference for most formal scientific debate. Scientific ideas that are not expressed in a scientific journal are rarely addressed. Edward Wilson, for example, commented that  ̳ a discovery does not exist until it is safely reviewed and in print ‟

. 9

18. Access to publication in scientific journals is therefore a crucial issue. Editors have used peer review by acknowledged experts of papers submitted for publication to advise them whether they contain errors, are important, trivial or simply repeat what has previously been published.

19. Richard Horton‘s essay explores typical patterns of behaviour relating to journals and peer review in contentious areas of science, and what behaviour should be regarded as improper. His conclusions are summarised as follows:

What an editor seeks from a reviewer is a powerful critique of the manuscript

– testing each assumption, probing every method, questioning all results, sceptically challenging interpretations and conclusions and ensuring that uncertainties are fully acknowledged, measured, and reported.

Armed with such a critique, the editors decide, and take full responsibility for deciding, whether to publish.

There is always the risk of group-think among experts which resists alternative perspectives. Editors try to reduce the risk of group-think by sending papers to different and widely dispersed reviewers, deliberately seeking or even provoking critical reviews.

Editors send manuscripts to reviewers based on a principle of confidentiality. The author expects the editor to maintain a covenant of trust between the two parties. Disclosure to a third party without the prior permission of the editor would be a serious violation of the peer review process and a breach of confidentiality.

Many who are far from the reality of the peer review process would like to believe that peer review is a firewall between truth on the one hand and error or dishonesty on the other. It is not. It is a means of sieving out evident error, currently unacceptable practices, repetition of previously published work without acknowledgement, and trivial contributions that add little to

9 Wilson, E.O. 1998. Consilience: The Unity of Knowledge . New York: Knopf.CHAPTER 5: THE CHANGING CONTEXT OF MODERN SCIENCE

40 knowledge.

It does not and cannot guarantee veracity. Many published papers have proved deeply flawed: many good ones have been rejected. Nor has it been efficient in identifying fraud, which has usually come to light by different routes.

However, journals, as the gatekeepers of scientific publication, have come to exert an increasing influence on the reputations of scientists, research units and universities. Many measures of academic success, promotion, tenure, grants, fame, and personal wealth depend upon journal publication. It is not surprising therefore that journals, and peer review, are the subject of constant tension and occasionally explosive controversy.

20. In conclusion, it is common for editors to have multiple, intense, and sometimes sharp and passionate interactions with authors and reviewers. The tone of their exchanges and communications with editors can be attacking, accusatory, aggressive, and even personal. If a research paper is especially controversial and word of it is circulating in a particular scientific community, third-party scientists or critics with an interest in the work may get to hear of it and decide to contact the journal. They might wish to warn or encourage editors. This kind of intervention is entirely normal. It is the task of editors to weigh up the passionate opinions of authors and reviewers, and to reflect on the comments (and motivations) of third parties. Occasionally, a line might be crossed into highly improper behaviour leading, for example, to censorship of ideas that might normally pass peer review. Defining that line is the crucial task when judging the role of CRU scientists, and determining whether, as has been alleged, they acted to subvert peer review by slowing or blocking the publication of research which disagreed with their own views. Was their activity part of the normal, robust hurly-burly surrounding publication in important highly contended fields, or was an important line crossed? We address this in Chapter 8.



## 5.6 The Responsibilities of Scientists in Communicating in

the Public Domain

21. The scientific literature is relatively opaque to non-specialists. Scientific understanding that is transmitted into the public domain must be comprehensible to non-specialists, make appropriate and not excessive claims, and include careful statements of the uncertainties surrounding that understanding. These principles apply universally to scientific advice, and are embedded in the codes of practice for scientific advice that have been adopted by many governments 10 . One of the allegations against CRU is that they have not been sufficiently frank in communicating uncertainties about their reconstructions into the public domain. This is particularly relevant to graphical presentation such as the  ̳hockey stick‘ which has taken on iconic significance. Images have a great power to persuade, and this is particularly true when complex issues are faced by lay audiences, who may often infer a level of certainty that does not in fact exist. The danger is

10 For example, the ― Code of Scientific Ethics ‖ published in 2007 by the UK Chief Scientific Advisor, and accepted by UK Government, includes the responsibility to: ―present and review scientific evidence, theory or interpretation honestly and accurately.‖CHAPTER 5: THE CHANGING CONTEXT OF MODERN SCIENCE

41 obviously heightened where an image is being used to support arguments for policy change. Therefore, if images are likely to be used in this way, it is essential that qualifications such as uncertainties are given a closely coupled prominence and explanation.



## 5.7 Communicating to Policymakers

22. The interface between science and public policy is a crucial one in matters of great public importance such as climate change. The IPCC was set up to provide just such an interface. Its job was to draw on and synthesise the diverse strands that contribute to modern climate science (see paragraph 6) and to make this accessible to the public, policymakers and other stakeholders in a way that is comprehensible and that does justice to underlying uncertainties. The importance of this process is underlined by the potential magnitude of the economic and social consequences of governmental decisions in the domain of global climate change. It has been alleged that CRU scientists subverted IPCC processes by minimising uncertainties and blocking ideas that disagreed with their established views.

23. The IPCC produces assessments of the current state of understanding of climate change, its causes and implications. Its approach is to produce the most probable account of these issues; together with their uncertainties, and to identify where there is insufficient evidence to discriminate between different interpretations of a phenomenon. Its purpose is to produce a  ̳best estimate‘ of what is currently understood, through the work of a group of scientists chosen for their expertise and experience to make reasoned assessments on the balance of evidence. It is not to produce a review of the scientific literature.

24. The IPCC assessment reports were published in 1990, 1996, 2001 and 2007. AR4, published in 2007, consisted of three components: The Physical Science Base (Working Group 1

–

WG1); Impacts, Adaptation and Vulnerability (WG2); and Mitigation of Climate Change (WG3). The issues that concern CRU are a consequence of their involvement in WG1. The core activities of these WGs are undertaken by writing teams consisting of Coordinating Lead Authors (CLAs) and Lead Authors (LAs). There are two drafting cycles prior to the production of the Final Draft. The first draft is submitted to Expert Reviewers for comment. The second draft is submitted for both expert and governmental review. Review Editors are appointed for each of the review cycles to ensure that critical comments are properly dealt with. The final draft is the source of a Summary for Policymakers, which is also reviewed by governments prior to publication.



## 5.8 The C ha nging Forum for Debate and the Blogosphere

25. The development in recent years of the internet as a vehicle for easy, instantaneous transmission of news and opinion has changed the nature of the debate about scientific issues. Prior to these developments, scientific debate largely took place in journals and conferences that effectively excluded the public from active engagement. Experts tended to introduce their conclusions to the public in ways that were difficult to challenge.CHAPTER 5: THE CHANGING CONTEXT OF MODERN SCIENCE

42 26. The mode has now changed and the field of climate change exemplifies this. There continues to be a scientific debate about the reality, causes and uncertaintiesof climate change that is conducted through the conventional mechanisms of peerreviewed publication of results, but this has been paralleled by a more vociferous, more polarised debate in the blogosphere and in popular books. In this the protagonists tend to be divided between those who believe that climate is changing and that human activities are contributing strongly to it, and those that are sceptical of this view. This strand of debate has been more passionate, more rhetorical, highly political and one in which each side frequently doubts the motives and impugns the honesty of the other, a conflict that has fuelled many of the views expressed in the released CRU emails, and one that has also been dramatically fuelled by them. It is difficult at the moment to predict whether and how the necessary cooler, rigorous scientific debate and the vital public policy interface will develop, or the effect that it will have on scientific publication or peer review.

27. Arguably the most significant change produced by the blogosphere is a transformation in the degree of openness now required of scientists whose work directly affects policy making. Without such openness, the credibility of their work will suffer because it will always be at risk of allegations of concealment and hence mal-practice. The extent to which this change was fully recognised by

both CRU and UEA administration is an important issue for the Review.

28. Therefore, the Review would urge all scientists to learn to communicate their work in ways that the public can access and understand; and to be open in providing the information that will enable the debate, wherever it occurs, to be conducted objectively. That said, a key issue is how scientists should be supported to explain their position, and how a public space can be created where these debates can be conducted on appropriate terms, where what is and is not uncertain can be recognised. The learned societies may have an expanded role to play here in encouraging debate. We would also commend the work of bodies such as the Science Media Centre at the Royal Institution for encouraging and helping scientists to take their work to lay audiences through the media, and advising them on how best to do this.CHAPTER 5: THE CHANGING CONTEXT OF MODERN SCIENCE

43CHAPTER 6: LAND STATION INSTRUMENTAL TEMPERATURE DATA

44 

## CHAPTER 6: LAND STATION INSTRUMENTAL

TEMPERATURE DATA

6.1 Background

1. Terrestrial temperature data is recorded at more than seven thousand land stations across the world. This is referred to as primary data.

2. The steps involved in producing CRU‘s analysis of global temperature trends involve assembling and quality checking the primary data from approximately four thousand stations, determining the monthly ―anomalies‖ 1 for each station, and then assembling averages based on a uniform 5x5 degree grid covering the globe.

3. This work, requiring considerable experience and man-years of effort, was started by CRU in 1978 and it continues today. The results were published in a series of

CRUTEM datasets over a period of 20 years between 1986 and 2006.

4. In assembling a gridded temperature dataset, some adjustment of primary data

ma y be needed in order to allow for what are referred to as  ̳non

- climatic effects‘. A simple example might involve adjusting temperatures at a particular station to allow for obvious recording errors, or the elimination of an obvious discontinuity in results if a station had been moved (referred to as homogenization). Such adjustments are both necessary and ubiquitous in scientific research.

5. This Chapter deals with the availability of these data to anyone wishing to carry out an equivalent study, the effect of adjustments, the comparison with independent studies and the availability of the information required to explicitly check the CRUTEM analysis. Finally, it addresses specific allegations about Chinese data used in a paper about the effect of urbanization on temperature.



## 6.2 The Allegations

6. The broad allegations which are prevalent in the public domain are:

That CRU prevented access to raw data. That CRU adjusted the data without scientific justification or adequate explanation. Some allegations imply that this was done to fabricate evidence for recent warming. That CRU inappropriately withheld data and computer code, thus inhibiting others from checking the conclusions of the CRUTEM analysis.

The overall implication of these allegations is to cast doubt on the extent to which CRU‘s work in this area can be trusted and should be relied upon.

1 A station  ̳normal‘ is the average recorded te mperature over a defined time period. In the case of

CRUTEM this is 1961- 1990. The temperature ―anomaly‖ is defined as the difference between the recorded temperature and the normal.CHAPTER 6: LAND STATION INSTRUMENTAL TEMPERATURE DATA

45 7. While very few of the submissions to this Inquiry make the allegations above explicitly, they are nevertheless implied. The full text of the submissions is contained on the team website. It is not the intent here to review each one, but only to highlight the most relevant and significant.

8. The most comprehensive and substantive submission critical of CRU is from

McKitrick 2 . He addresses the question of data adjustments and also clarifies that the issue is not generally one of data availability, but more specifically the availability of a list of additional stations used in the CRUTEM analysis since 1986. He draws attention to the series of requests to CRU for station identifiers over the period 2002-2007, eventually culminating in a series of FOI requests, also referred to by Matthews 3 and Cockroft 4 . Two submissions 5 allege that CRU has withheld access to primary data and by implication stopped others from repeating analyses, but present no evidence. A common theme in the submissions from the Global Warming Policy Foundation (GWPF) , McKitrick and McIntyre 6
is the lack of cooperation exhibited by CRU in their dealings with selected third parties.

9. In support of CRU, several submissions 7

, 8

, 9 point out that the data is in the public domain and can be accessed by anyone wishing to repeat a temperature trend analysis. Santer also defends the work done by CRU on regional adjustments, maintaining that it made very little difference to the global results. He also highlights that studies conducted independently from CRU, based on different choices and adjustment protocols, produce very similar results

– a point also mad e by Joos 10 and Tett.

10. In its own submission under issue 4, CRU 11 defends its position with regard to adjustments and openness. The website also contains the minutes of the meeting which took place on the subject of data sets on March 4 th , 2010 between members

of CRU and the Review Team.



## 6.3 The Approach Adopted by the Review Team

11. In order to test the principal allegations of withholding data and making inappropriate adjustments, the Review undertook its own trial analysis of land station temperature data. The goal was to determine whether it is possible for an independent researcher to (a) obtain primary data and (b) to analyse it in order to produce independent temperature trend results. This study was intended only to test the feasibility of conducting such a process, and not to generate scientific conclusions. The process followed is described in Appendix 7

.

2 McKitrick submission (no.15), 26 February

3 Matthews submission (no.16), 1 March

4 Cockroft submission (no. 6), 28 February

5 Global Warming Policy Foundation submission (no. 22), 28 February, and Cockroft (Ibid.)

6 McIntyre submission (no. 23), 2 March

7 Tett submission (no. 21), 1 March

8 Santer submission (no. 44), 28 February

9 Barnes submission (no. 1), 6 March

10 Joos submission (no. 12), 1 March

11 CRU submission (no. 5), 1 MarchCHAPTER 6: LAND STATION INSTRUMENTAL TEMPERATURE DATA

46 12. The Team then went on to address a further question, namely the extent to which an independent researcher could check the CRUTEM analysis. In particular, this

considered:

whether the data sources were properly explained the availability of software required in the analysis



## 6.4 The Results of the Analysis

13. To carry out the analysis we obtained raw primary instrumental temperature station data. This can be obtained either directly from the appropriate National Meteorological Office (NMO) or by consulting the World Weather Records (WWR) available through the National Centre for Atmospheric Research (NCAR), or the Global Historical Climatology Network (GHCN):

NCAR: http://www.ncar.ucar.edu/tools/datasets/ WWR: http://dss.ucar.edu/datasets/ds570.0/ GHCN: http://www.ncdc.noaa.gov/oa/climate/ghcn-monthly/index.php.

14. Anyone working in this area would have knowledge of the availability of data from these sources. There are also other sources but we have not investigated

these.

15. The Review extracted a comprehensive set of global temperature records from both GHCN and NCAR websites to test ease of availability from either. The stations included in these sources have much overlap as would be expected. In addition, as a test case, primary data was also obtained directly from the Japanese

NMO.

16. In a true analysis the researcher would at this point be able to make whatever quality control, station selection and adjustments they considered appropriate, which should be justified in the published record of research.

17. Figure 6.1 below gives the results of the Team‘s trial analysis to produce three different global temperature anomaly average series. These use a 5x5 degree grid and a 5-year smoothing (for details see Appendix 7). In each case we started with the full dataset, requiring only that a station had enough data to construct the relevant normals when constructing the gridded averages. Apart from this we made no selection or other adjustments of our own. We show the result obtained using the GHCN unadjusted data set (blue), the GHCN adjusted data set (yellow) and the NCAR dataset which is unadjusted (green). On the same figure we show as a black line the global average series obtained from the land air temperature anomalies on a 5x5 degree grid-box basis (CRUTEM3) gridded analysis (which uses adjusted data). The green and blue lines are both from unadjusted datasets. The yellow and black lines are from adjusted datasets.

18. All the lines, whether from adjusted or unadjusted datasets, agree very well.CHAPTER 6: LAND STATION INSTRUMENTAL TEMPERATURE DATA

47 Figure 6.1: Temperature anomaly time series created by the Review Team‘s own trial analysis using a 5x5 degree grid with 5 year smoothing. Shown are results obtained from GHCN (blue)

, GHCNadjusted (yellow) and NCAR (green) . Also shown is the CRUTEM3 line (black). The Y-axis is 10 x the anomaly in degrees. The X-axis is year.

19. In Figure 6.2 we show the results of independent scientific studies, where individual researchers have brought to bear their own judgment on station selection and adjustments to the data. Such studies have been carried out by NASA-GISS and NOAA-NCDC using the GHCN dataset. The figure from the

IPCC 4 th Repo rt Chapter 3 is reproduced below to show these in comparison to

CRUTEM3 12 . Once again there is good agreement.

12 The reader is referred to the comprehensive submission from the UK Met Office (CRU 54) to the House of Commons Committee on Science and Technology Inquiry which shows a similar unsmoothed figure.CHAPTER 6: LAND STATION INSTRUMENTAL TEMPERATURE DATA

48 Fig 6.2 Reproduced from the IPCC 4 th Report Chapter 3.

20. Finding: This simple analysis and the comparisons in figures 6.1 and 6.2 give rise to the following findings:

Any independent researcher may freely obtain the primary station data. It is impossible for a third party to withhold access to the data.

It is impossible for a third party to tamper improperly with the data unless they have also been able to corrupt the GHCN and NCAR sources. We do not consider this to be a credible possibility, and in any case this would be easily detectable by comparison to the original NMO records or other sources such as the Hadley Centre.

T he steps needed to create a global temperature series from the data ar e straightforward to implement.

The required computer code is straightforward and easily written by a competent researcher.

The shape of the temperature trends obtained in all cases is very similar: in other words following the same process with the same data obtained from different sources generates very similar results.CHAPTER 6: LAND STATION INSTRUMENTAL TEMPERATURE DATA

49 21. By performing this simple test one determines easily that the results of the CRUTEM analysis follow directly from the published description of the method, and that the resultant temperature trend is not significantly different from the other results regardless of stations used or adjustments made. The test is therefore sufficient to demonstrate that, with respect to the declared method, the CRUTEM analysis does not contain either error or adjustments which are responsible for the shape of the resultant temperature trend.

22. A researcher can evidently produce a study which would test the CRUTEM analysis quite precisely, without requiring any information from CRU to do so.

23. Finding: The high level trial analysis shows that adjustments make little difference to the overall conclusions on temperature trends. However, this has been a matter

about which much comment has been made and it is therefore discussed in more detail in Appendix 7. Adjustments were made by CRU to the data from 10% of stations. The Review Team is satisfied that this had no significant effect upon results. Very strong evidence for this is furnished by the level of agreement between CRUTEM and the other lines shown in Figure 6.1, as well as other studies detailed in the Appen dix, showing the effect of de-adjusting and of removing the adjusted stations.

24. It should be noted that in making these findings, the Review Team is making no statement regarding the correctness of any of these analyses in representing global temperature trends. We do not address any alleged deficiencies such as allowance for non climatic effects or the significant drop in station number post 1991. We do not address any possible deficiencies of the method. These are entirely matters for proper scientific study and debate and lie outside the scope of this Review.



## 6.5 Checking Specific Details in the CRUTEM Analysis

25. The goal here is to consider the requirements for carrying out an exact replication of the CRUTEM work to produce an identical gridded analysis. The work above demonstrates that this procedure is not necessary if the goal is simply to produce an alternative temperature trend analysis. As discussed in Appendix 7 , and as follows from the close agreement between the lines in Figures 6.1 and 6.2, it is also not necessary to test the robustness of the CRUTEM3 analysis. However, the Review upholds the important principle that any scientific result should be open to such scrutiny for whatever reason. It therefore considered whether all the necessary information was made available.



## 6.5.1 Identification of Data Sources

26. In order to reproduce exactly a CRUTEM study, the independent researcher requires the exact list of stations used in order to source the primary data

themselves.

27. The original version of the CRUTEM gridded averages was published in 1986 (in this report we call this CRUTEM1986) using 3276 stations (314 of which were adjusted). Subsequent revisions were produced in 1994, 2003 and 2006, referredCHAPTER 6: LAND STATION INSTRUMENTAL TEMPERATURE DATA

50 to as CRUTEM1, 2 and 3. CRUTEM3 used 4138 stations (298 were adjusted). Whereas the 3276 stations used in CRUTEM1986 are listed in the respective

references 13 , this is not the case for the additional approximately 1000 stations

included in CRUTEM3 14 . The additional stations are described only in general terms in the respective publication via references to the sources.

28. For some years prior to the coming into force of the general right of access to information under FoIA, CRU had received requests for data, including station identifiers. An example of the attitude to these requests is given in the following

e-mail extract:

Jones to Mann on 2 nd February 2005 (1107454306.txt):

“And don't leave stuff lying around on ftp sites - you never know who is trawling them. The two MMs have been after the CRU station data for years. If they ever hear there is a Freedom of Information Act now in the UK, I think I'll delete the file rather than send to anyone. Does your similar act in the US force you to respond to enquiries within 20 days? - our does ! The UK works on precedents, so the first request will test it. We also have a data protection act, which I will hide behind. Tom Wigley has sent me a worried email when he heard about it - thought people could ask him for his model code. He has retired officially from UEA so he can hide behind that. IPR should be relevant here, but I can see me getting into an argument with someone at UEA who'll say we must adhere to it !”

29. A request was made to CRU for both station identifiers and access to the raw data for these stations. This was formally logged by the UEA Information Policy & Compliance Manager (IPCM) in January 2007 and was dealt with under the FoIA

. The request was initially refused entirely on the grounds that raw station data was publicly available (from GHCN, NCAR and the NMOs), but without reference to providing station identifiers. Following further correspondence, an internal UEA appeal process and further prompting by the applicant, the list of the 4138 stations used in CRUTEM3 was finally released on 1st October. The issues that this raises

13 Bradley, R.S., Kelly, P.M., Jones, P.D., Goodess, C.M. and Diaz, H.F., 1985: A Climatic Data Bank for Northern Hemisphere Land Areas, 1851-1980, U.S. Dept. of Energy, Carbon Dioxide Research Division, Technical Report TRO17 , 335 pp. Jones, P.D., Raper, S.C.B., Santer, B.D., Cherry, B.S.G., Goodess, C.M., Kelly, P.M., Wigley, T.M.L., Bradley, R.S. and Diaz, H.F., 1985: A Grid Point Surface Air Temperature Data Set for the Northern Hemisphere, U.S. Dept. of Energy, Carbon Dioxide Research Division, Technical Report TRO22 , 251 Jones, P.D., Raper, S.C.B., Cherry, B.S.G., Goodess, C.M. and Wigley, T.M.L., 1986: A Grid Point Surface Air Temperature Data Set for the Southern Hemisphere, 1851-1984, U.S. Dept. of Energy, Carbon Dioxide Research Division, Technical Report TR027 , 73 pp. Jones, P.D., Raper, S.C.B., Bradley, R.S., Diaz, H.F., Kelly, P.M. and Wigley, T.M.L., 1986: Northern Hemisphere surface air temperature variations: 1851-1984. Journal of Climate and Applied Meteorology 25 , 161-179. Jones, P.D., Raper, S.C.B. and Wigley, T.M.L., 1986: Southern Hemisphere surface air temperature

variations: 1851-1984. Journal of Climate and Applied Meteorology 25 , 1213 1230

14 Brohan, P., Kennedy, J., Harris, I., Tett, S.F.B. and Jones, P.D., 2006: Uncertainty estimates in regional and global observed temperature changes: a new dataset from 1850. J. Geophys. Res. 111

, D12106, doi:10.1029/2005JD006548. Jones, P.D. and Moberg, A., 2003: Hemispheric and large-scale surface air temperature variations: An extensive revision and an update to 2001. J. Climate 16 , 206-223.CHAPTER 6: LAND STATION INSTRUMENTAL TEMPERATURE DATA

51 on compliance with the FoIA are dealt with in Chapter 10.

30. The Review Team verified that matching of stations to a simple list of identifiers is not a straightforward process. The identification numbers are not unique across data sets, station names differ, and in the case of GHCN several replicas of the same data exist. This is primarily a problem caused by the lack of standardisation of metadata within the global climate science community. We make a

recommendation on this below.

31. The Review Team was able to match 90% of stations given in the CRU list to GHCN (see Appendix 7). CRU has stated in a written submission 15 that that the remaining 10% can be obtained from other sources including the NMOs. Thus substantial work is required to take the CRU published list and assemble 100% of the primary station data from global repositories and NMOs. We make a

recommendation for the future below.

32. Finding: The Review finds that as a matter of good scientific practice, (and having established the precedent with CRUTEM1986) CRU should have made available an unambiguous list of the stations used in each of the versions of CRUTEM at the time of publication. In the absence of this, CRU was unhelpful and defensive and should have responded throughout to requests for this information in a more timely way.



## 6.5.2 The Availability of Computer Codes

33. Finding: The computer code required to read and analyse the instrumental temperature data is straightforward to write based upon the published literature. It amounts a few hundred lines of executable code (i.e. ignoring spaces and comments). Such code could be written by any research unit which is competent to reproduce or test the CRUTEM analysis. For the trial analysis of the Review Team, the code was written in less than two days and produced results similar to other independent analyses. No information was required from CRU to do this.

34. The Met Office Hadley centre has published Practical Extraction and Reporting Language (PERL) codes which are a few hundred lines long. The submission by

Barnes is also very helpful and illustrative in this context.



## 6. 6 Use of Local Temperature Data from China

35. The above account of our work has focused on global data, its handling and accessibility. We also considered the implication in a submission to us by Dr Benny Peizer of the Global Warming Policy Foundation 16 that Jones was complicit in malpractice in failing to respond appropriately to allegations of fraud made against a climate scientist at the State University of New York, Albany, Professor Wei-Chyung Wang in relation to more local data. An important paper

15 Follow up request (20 April) to Professor Jones and response in relation to raw instrument station availability for each CRUTEM set

16 Peizer/Global Warming Policy Foundation submission (no. 22), 28 FebruaryCHAPTER 6: LAND STATION INSTRUMENTAL TEMPERATURE DATA

52 on the effect of urbanization on temperature by Jones et al (1990) 17 included data from China (Wang et al, 1990) 18 that contained the statement that only instrument stations were selected where there had been few, if any, changes in instrumentation, location or observation times. It has been claimed by Keenan (2007) 19 that this latter statement was knowingly untrue (“fabricated”), that stations had been moved, thus that their record of temperature change would be unreliable, and that hard copies of details of station histories had been lost or withheld. A subsequent paper by Jones et al. (2008) 20 verified the original conclusions for the Chinese data for the period 1954

– 1983, showing that the precise location of weather stations was unimportant to the outcome

36. Peizer, as Editor of Energy and Environment , to which the Keenan paper had been submitted, had contacted Jones asking him to comment on the Keenan paper, which Jones did, by denying its criticisms and advising rejection of the

paper 21 . Jones was encouraged by colleagues to respond in writing to the journal, but chose not to do so "until the SUNY process has run its course." 22 (The State University of New York, Albany, investigated the accusation of fraud against Wang, which it rejected 23 ). Peizer has included some detail of

email traffic on this issue in his submission to us. 24

37. The allegation against Jones that is relevant to the Review 25 is that since significant doubts about the reliability of Chinese climate data were raised, Jones has taken no public steps to clear up the discrepancies regarding Wang's claims and data, and that “it is unacceptable that the scientist who disseminates a dat a product on which international treaties are based, as well as IPCC reports and countless government policies, should actively seek to suppress information that calls the quality of the data into question”.

38. We note that, in at least a Nature interview referring to the loss of records 26 , Jones is reported as acknowledging “It‟s not acceptable...not best practice ...the stations „probably did move‟...I thought it was the right way to get the data. I was specifically trying to get more rural station data that wasn‟ t routinely available in real time from [meteorological] services.” Although Jones chose not to respond in

17 Jones P.D., Groisman P.Y., Coughlan M., Plummer N., Wang W.- C., Karl T.R. (1990), ―Assessment of urbanization effects in time series of surface air temperature over land‖, Nature , 347: 169

– 172.

18 Wang W.- C., Zeng Z., Karl T.R. (1990), ―Urban heat islands in China‖, Geophysical Research

Letters , 17: 2377

– 2380.

19 Keenan, D.J. Fraud allegation against some climatic research of Wei-Chyung Wang. Energy and Environment, 18 (7- 8) , 985-995.

20 Jones, P.D., Lister, D.H. and Li, Q. 2008. Urbanization effects in large-scale temperature records, with an emphasis on China. Journal of Geophysical Research, 113 (D16122) , 12pp.

21 Ibid. 16

22 Ibid. 16

23 http://www.informath.org/apprise/a5620.htm. This site included a letter to Keenan from the State University of New York, Albany, that states that ―The Investigation Committee finds no evidence of the alleged fabrication of results and nothing that rises to the level of research misconduct having been committed by Dr. Wang.‖ We are not aware of a published version of this report.

24 Ibid. 16

25 Ibid. 16

26 Nature News, published online 15 February 2010 http://www.nature.com/news/2010/100215/full/news.2010.71.htmlCHAPTER 6: LAND STATION INSTRUMENTAL TEMPERATURE DATA

53 writing when Keenan‟s claim was made, he did respond through an analysis in a paper in a front rank peer-reviewed journal only one year later that verified the

conclusions of the earlier work 27 . By that time the SUNY investigation had concluded in Wang‟s favour

.


## 6.7 Conclusions and Recommendations

39. In summary, with regard to the allegations concerning the temperature dat a, the

conclusions of the Review Team are as follows:

Regarding data availability, there is no basis for the allegations that CRU prevented access to raw data. It was impossible for them to have done so.

Regarding data adjustments, there is no basis for the allegation that CRU made

ad justments to the data which had any significant effect upon global averages and through this fabricated evidence for recent warming.

We find that CRU was unhelpful in dealing with requests for information to enable detailed replication of the CRUTEM analysis.

Crucially, we find nothing in the behaviour on the part of CRU scientists that is the subject of the allegations dealt with in this Chapter to undermine the validity of their work.

40. Reflecting the analysis in Appendix 7, the Revi ew has the following

recommendations:

It would benefit the global climate research community if a standardised way of defining station metadata and station data could be agreed, preferably through a standards body, or perhaps the WMO. We understand that this is not straightforward and could be a lengthy process, but the process should start now. As example an xml based format would make the interpretation use, comparison, and exchange of data much more straightforward.

Without such standardisation there will always be problems in issuing unambiguous lists, and assembling primary data from them. It would be in the public interest if CRU and other such groups developed a standard process to capture and publish a snapshot of the data used for each important publication.

27 Ibid. 20CHAPTER 7: TEMPERATURE RECONSTRUCTIONS FROM TREE RING ANALYSIS

54 

## CHAPTER 7: TEMPERATURE RECONSTRUCTIONS

FROM TREE RING ANALYSIS

7.1 Background

1. This Chapter considers the criticisms that have been made of CRU scientists in relation to their work on obtaining and analysing tree ring data to reconstruct land temperature records back over thousands of years, and in relation to the use made

of that work in the IPCC.

2. Data is recorded from tree core samples from many sites around the globe. These data comprise measurements of width and density taken from the cores and do not in themselves represent temperature directly. The processes of assembling and standardising the tree data, into what are termed tree ring chronologies, are separate from the later stage of producing temperature reconstructions.

3. Temperature reconstructions are obtained through regression models which are calibrated against instrument temperature data during periods of overlap. A temperature reconstruction from any specific region does not in itself indicate global temperature patterns. Tree and other proxy reconstructions are combined to produce reconstructions pertaining to a wider area, and such reconstructions from many authors may be aggregated in order to provide an indication of global patterns. Even so the coverage is not as extensive as that of instruments.

4. The uncertainty associated with temperature reconstructions in the distant past is much larger than that of present day instrument records. There is a wide and healthy debate in the published literature on all aspects of this.

5. CRU scientists have developed temperature reconstructions based on a range of tree ring series. The current leader in CRU is Professor Briffa. Their work i s extensively described in the relevant scientific literature. It also contributes to the development of a global picture such as that produced by IPCC in its 4 th
Assessment Report (Working Group 1, Chapter 6, Palaeoclimate). In particular

this contained conclusions on the likelihood that the second half of the 20 th
Century was the warmest 50 year period in the last 500 or 1300 years in the Northern Hemisphere (different statements of probability are made for each

– see paragraph 20 ).



## 7.2 The Allegations

6. The release of the e-mails triggered the expression of a range of criticisms (many of long standing) of the work of CRU scientists. There was a clear implication that in carrying out their work, both the choices they made of data and the way it was handled, were intended to bias the scientific conclusions towards a specific result and to set aside inconvenient evidence. The overall implication is that this work is misleading and should not be trusted.CHAPTER 7: TEMPERATURE RECONSTRUCTIONS FROM TREE RING ANALYSIS

55 7. We address the following specific allegations which were explicit or can be

inferred from the comments made on the e-mails and from the submissions we

received

.

8. In relation to IPCC:

That the CRU work was ―flawed‖ and yet had a distinct influence on the balance of judgements made by IPCC Working Group 1 in Chapter 6 of their

4 th Assessment Report, and therefore that less confidence should be ascribed to the conclusions reached by the authors, mentioned in paragraph 5 above. The criticism here is often captured by the proposition that today‘s

temperatures are not unusual compared to the MWP.

That Yamal and other chronologies constructed by CRU are unrepresentative of temperature trends (in recent years), and had an undue influence on all of the lines appearing in Chapter 6 of the 4 th IPCC Report.

That a majority of the reconstructions would look significantly different if certain component series were replaced with others, and that if this were done then the conclusions reached in respect of the likelihood associated with ranking of recent warmth with respect to the past would be significantly

different.

9. In relation to divergence: That the phenomenon of ―divergence‖ (see discussion below) may not have been properly taken into account when expressing the uncertainty associated with reconstructions; and th at this was not addressed with sufficient openness in IPCC reports.

That the reference in a specific e- mail to a ―trick‖ and to ―hide the decline‖ in respect of a 1999 WMO report figure show evidence of intent to paint a misleading picture.

10. In relation to withholding data: That access to tree ring data from the Yamal peninsula in Siberia (used by Professor Briffa) was withheld, and that as a particular consequence of this the small sample size of the Yamal series in the most recent years was not acknowledged.

11. In relation to mishandling data: That improper manipulations have been performed

– specifically with respect

to the Yamal and Tornetrask tree series.CHAPTER 7: TEMPERATURE RECONSTRUCTIONS FROM TREE RING ANALYSIS

56 

## 7.3 Findings

7.3.1 IPCC Reports

12. The IPCC reports are at the centre of allegations of presenting misleading messages produced by CRU. We concentrate upon AR4 as it is the most recent and most comprehensive.

13. On the replacement of series in reconstructions, much of the critical material submitted presents several examples of differing results obtained from a tree series (A) compared to another tree series (B) in the same region. This is to be expected, and taken in isolation has no implication for the validity of either. To know whether replacing one with another would have a significant effect upon statements regarding Northern Hemisphere temperature series, would require the

fulfilment of several conditions:

That it has been proven that series B is more representative over the whole period than series A. That the difference between series A and B is important in a temperature

reconstruction in which it is included. That if A were replaced with B in each of the reconstructions in which A is included then the conclusions drawn in respect of the likely ranking of the past to the present would change significantly when considering all reconstructions together (including those in which it does not appear).

To support an allegation of improper conduct would require that the scientists involved, knowing the above, still use and promote non-representative series A in order to reach improperly a false conclusion.

14. Finding: We are unaware of any analysis to demonstrate that any of the above conditions are fulfilled for Yamal or any of the series cited in relation to CRU work (i.e. Tornetrask, Taymir). The Review is naturally aware that partial studies and comments referring to CRU‘s published work appear elsewhere . However these criticisms of CRU‘s work are not in peer reviewed journals, and we have not found that these are anywhere assembled into a coherent, comprehensive and scrutinised case which demonstrates the proposition in respect of any of the series

cited.

15. Finding: To make the case that replacing series in reconstructions calls in question the validity of the picture painted in Chapter 6 of the IPCC report (or to sustain a charge of impropriety on the part of its many authors) would require that all the conditions in paragraph 13 were met. No evidence of this has been presented to

the Review.

16. Finding: The influence of the Yamal series of Briffa 2000 1 is often called into question. The conditions in paragraph 13 above have not been demonstrated

1 Briffa, K. R. 2000. Annual climate variability in the Holocene: interpreting the message of ancient

trees. Quaternary Science Reviews 19:87- 105CHAPTER 7: TEMPERATURE RECONSTRUCTIONS FROM TREE RING ANALYSIS

57 anywhere which has been brought to our attention. This series is used in none of

the IPCC TAR and only 4 2 out of 12 reconstructions in AR4.

17. We now turn to the question of uncertainty. There are multiple sources of uncertainty in respect of proxy temperature reconstructions and they are the subject of an ongoing and open scientific debate.

18. Elements of that debate are touched on in the e-mails. For example, the following e-mails from Briffa show some concern to ensure that uncertainty was properly recognised.

E-mail dated 29/4/03 (1051638938.txt):

“Can I just say that I am not in the MBH camp - if that be characterized by an unshakable "belief" one way or the other , regarding the absolute magnitude of the global MWP. I certainly believe the " medieval" period was warmer than the 18th century - the equivalence of the warmth in the post 1900 period, and the post 1980s, compared to the circa Medieval times is very much still an area for much better resolution. I think that the geographic / seasonal biases and dating/response time issues still cloud the picture of when and how warm the Medieval period was . On present evidence , even with such uncertainties I would still come out favouring the "likely unprecedented recent warmth" opinion - but our motivation is to further explore the degree of certainty in this belief - based on the realistic interpretation of available data.”

E-mail dated 3/2/06 (1138995069.txt):

“we are having trouble to express the real message of the reconstructions

- being scientifically sound in representing uncertainty , while still getting the crux of the information across clearly...We have settled on this version (attached) of the Figure which we hoe [sic] you will agree gets the message over but with the rigor required for such an important document.”

19. What is clear is that the uncertainty associated with any estimate of past temperatures from reconstructions is much larger than that of recent instrument temperature data. This is demonstrated in the figure below taken from IPCC AR4 Figure 6.10. The middle plot shows the variability both within and between different reconstructions each using an aggregation of proxy data (i. e. several tree and non-tree proxies). The lower plot gives an indication of uncertainty bands from those reconstructions. Simply looking at any individual reconstruction line alone makes only a partial statement about Northern Hemisphere temperatures with a large uncertainty. It is obviously even less meaningful to look at an

individual tree series in isolation.

2 A fifth reconstruction is disputed.CHAPTER 7: TEMPERATURE RECONSTRUCTIONS FROM TREE RING ANALYSIS

58 Figure 7.1: From IPCC AR4 (Figure 6.10)

20. Understanding requires proper statistical interpretation, i.e. to determine the

confidence level associated with a statement such as “the present is likely warmer than the past”

. To do this as objectively as possible would require a complex (and difficult) study to perform hypothesis testing in a mathematically rigorous way, taking proper account all of the uncertainties and their correlations. We are not aware that this has been done in the production of IPCC reports to date, but instead qualitative statements have been made based on definitions of ―likely‖, ―very likely‖ etc according to criteria lai d down by the IPCC ( ̳Likely‘ mea ns a probability greater than 66 %, and  ̳Very Likely‘ means a probability greater than 90%).CHAPTER 7: TEMPERATURE RECONSTRUCTIONS FROM TREE RING ANALYSIS

59 21. Finding: We do not find that the data described in IPCC AR4 and shown in Figure 6.10 is misleading, and we do not find that the question marks placed over the CRU scientists‘ input casts doubt on th e conclusions. In particular:

The variation within and between lines, as well as the depiction of uncertainty is quite apparent to any reader.

It presents all relevant published reconstructions we are aware of, i.e. there has been no exclusion of other published temperature reconstructions which would show a very different picture.

The general discussion of sources of uncertainty in the text is extensive, including reference to divergence and it therefore cannot be said that anything has been suppressed. Presenting uncertainty in this way is a significant advance on the TAR.



## 7.3.2 Divergence

22. Divergence : The phenomenon of divergence is an ongoing research area which is discussed widely in the literature and the submissions. Tree core data yields both width and density measurements and both of these are used to reconstruct temperature records. In some cases the temperature reconstruction from one or other of these will track instrumental temperature measurements on a short timescale, but diverges on a longer timescale. The phenomenon of divergence is not observed in all tree series. There is as yet no complete understanding of the source of divergence although studies are suggesting possible contributory effects.

23. Finding: The question of whether divergence has been adequately taken into account when estimating uncertainties is a complex and ongoing question of science and is therefore outside the scope of this Review. We have however investigated this matter enough to be satisfied that it is not hidden and that the subject is openly and extensively discussed in the literature, including CRU

papers.

24. The WMO 1999 front cover figure : The allegation is that the figure appearing on the front cover of the WMO report of 1999 ( Statement on the Status of Global

Climate , 1999 3 ) did not disclose that it combined proxy and temperature records and failed to show a declining proxy line due to divergence; and that this was

done to deceive.

Jones‘ e-mail on 16/11/99 (942777075.txt)

“I've just completed Mike's Nature trick of adding in the real temps to each series for the last 20 years (ie from 1981 onwards) and from 1961 for Keith's to hide the

decline.”

is perhaps the prime example of the fact that opposing interpretations can be

obtained from the same statement. The word ―trick‖ has been widely taken to

3 http://www.wmo.ch/pages/prog/wcp/wcdmp/statement/wmo913.pdfCHAPTER 7: TEMPERATURE RECONSTRUCTIONS FROM TREE RING ANALYSIS

60 confirm the intention to deceive, but can equally well, when used by scientists, mean for example a mathematical approach brought to bear to solve a problem. It is the latter explanation that Jones has given the Review, quoted in paragraph 6 of Chapter 4.

25. The WMO report is a short document produced annually. It does not have the status or importance of the IPCC reports. The figure in question was a frontispiece and there is no major discussion or emphasis on it in the text. The caption of the figure states: ― Northern Hemisphere temperatures were reconstructed for the past 1000 years (up to 1999) using palaeoclimatic records (tree rings, corals, ice cores, lake sediments, etc.), along with historical and long instrumental records ”.

26. Finding: In relation to ―hide the decline‖ we find that, given its subsequent iconic significance (not least the use of a similar figure in the TAR ), the figure supplied for the WMO Report was misleading in not describing that one of the series was truncated post 1960 for the figure, and in not being clear on the fact that proxy and instrumental data were spliced together. We do not find that it is misleading to curtail reconstructions at some point per se , or to splice data, but we believe that both of these procedures should have been made plain

– ideally in the figure but certainly clearly described in either the caption or the text.



## 7.3.3 Withholding Data

27. Yamal

- withholding data and information on sample size : This issue concerns

the Briffa 2000 4 and Briffa 2008 5 publications. The underlying raw data were not archived at the time of publication. Since it was not owned by CRU, they had no right to archive it . The data were owned by others including Hantemirov and Shiyatov. CRU has stated that it directed the request it received for data to the owners. Whether it was a result of this, or otherwise, the requester was given a

copy 6 of the data in 2004 by Hantemirov. Later, following publication of Briffa

2008 7 , and again following requests for the data, CRU has stated that it asked the owners to archive the data, which they did.

28. Because the raw data was not archived, the information on the number of core counts used in each year was not easily available. As a result the reader could not know that the core count in the most recent years had dropped substantially. This fact grew in importance as the need to understand uncertainty grew. The information was actually published and in the public domain in 2002 8 but it is arguable that a user of the CRU Yamal chronology would not have known to look

4 Ibid. 1

5 Briffa, K. R., Shishov, V.V., Melvin, T. M., Vaganov, E. A., Grudd, H., Hantemirov, R. M., Eronen, M., and Naurzbaev, M. M. 2008. Trends in recent temperature and recent tree growth spanning 2000 years across Northwest Eurasia. Philosophical Transactions of the Royal Society B- Bi ological Sciences

363:2271-2284

6 It has also been pointed out to us by CRU that McIntyre acknowledged this in a comment (number 61) made underneath a Climate Audit post which states that McIntyre did not realize that this was the series used by Briffa when he received it. http://climateaudit.org/2009/10/05/yamal-and-ipcc-ar4-

review-comments/.

7 Ibid. 5

8 Hantemirov, R. M., and Shiyatov, S. G. 2002. A continuous multimillenial ring-width chronology in Yamal, northwestern Siberia. Holocene 12:717-726CHAPTER 7: TEMPERATURE RECONSTRUCTIONS FROM TREE RING ANALYSIS

61 there unless explicitly directed. Hantemirov and Shiyatov 2002 was referenced in

Briffa 2008 9

.

29. Finding: It is evidently true that access to the raw data was not simple until it was archived in 2009 and this can rightly be criticized on general principles of transparency, although it may have been common practice at the time of the original publication. We find that CRU has not withheld the data (having correctly directed the single request to the owners). However, in the interests of transparency, we believe that CRU should have ensured that the data it did not own, but had relied upon in publications, was archived in a more timely way. This is another example where a more open approach would have been helpful.



## 7.3.4 Mishandling Data

30. We have not focussed upon disagreements over comparisons of results using individual tree series for two reasons. First, because this is a science question and should be addressed through the normal channels of peer reviewed publication. Secondly, because as we have indicated above, it requires much more work to show that one is unequivocally more representative than the other, and even then requires the conjunction of several additional conditions to lead to an allegation of wrongdoing. We have addressed these conditions above, and we have found no evidence to substantiate them. Nevertheless we comment briefly upon Yamal as it has received so much attention and the Tornetrask series as it is subject of much misunderstanding.

31. Finding on Yamal: The Briffa 200 0 10 paper presents a collection of many chronologies (Yamal is simply one of them) assembled from many authors. It is quite clear that Briffa simply reprocessed data from other authors (Hantemirov and Shiyatov in this case) and no selection was performed. In their submission to the Review CRU included a copy of work they had made available on the web 11

. This was posted in response to criticisms published at Climate Audit 12 which described examples of variability between specific tree series in the region. This CRU response demonstrates the consistency of the original Yamal publication with that obtained using an updated analysis incorporating a comprehensive set of contemporary data. The Review is not aware of any equivalent comprehensive analysis which demonstrates that the conclusions of Briffa 2000 are inconsistent with the best analysis available today.

32. Finding on ―Bodging‖ in respect of Tornetrask

. Th e term ―bodging‖ has been used, including by Briffa himself, to refer to a procedure he adopted in 1992 13

. The  ̳bodge‘ refers to the upward adjustment of the low -frequency behaviour of the density signal after 1750, to make it agree with the width signal. This ad hoc process was based on the conjecture that the width signal was correct. There is nothing whatsoever underhand or unusual with this type of procedure, and it was

9 Ibid. 5

10 Ibid. 1

11 http://www.cru.uea.ac.uk/cru/people/briffa/yamal2009/

12 http://climateaudit.org/2009/09/27/yamal-a-divergence-problem/

13 Briffa, K. R., Jones, P. D., Bartholin, T. S., Eckstein, D., Schweingruber, F. H., Karlen, W., Zetterberg, P., and Eronen, M. 1992. Fennoscandian Summers from AD 500: temperature changes on long and short timescales. Climate Dynamics 7:111-119.CHAPTER 7: TEMPERATURE RECONSTRUCTIONS FROM TREE RING ANALYSIS

62 fully described in the paper. The interpretation of the results is simply subject to

this caveat. The conjecture was later validated 14 when it was shown to be an effect due to the standardisation technique adopted in 1992 . Briffa referred to it as a ―bodge‖ in a private e -mail in the way that many researchers might have done when corresponding with colleagues. We find it unreasonable that this issue, pertaining to a publication in 1992, should continue to be misrepresented widely to imply some sort of wrongdoing or sloppy science.



## 7.4 Conclusions and recommendations

33. We do not f

ind that the data described in AR4 and shown in Figure 6.10 is misleading, and in particular we do not find that the question marks placed over the CRU scientists’ input cast doubt on the conclusions.

34. The variation within and between lines, as well as the depiction of uncertainty is quite apparent to any reader. All relevant published reconstructions of which we are aware are presented, and we find no evidence of exclusion of other published temperature reconstructions which would show a very different picture. The general discussion of sources of uncertainty in the text is extensive, including reference to divergence and it therefore cannot be said that that anything has been suppressed. Presenting uncertainty in this way is a significant advance on the

TAR.

35. We have seen no evidence to sustain a charge of impropriety on the part of CRU staff (or the many other authors) in respect of selecting the reconstructions in AR4 Chapter 6. This would require that all the conditions in paragraph 13 were met in respect of tree chronologies either used by, or created by, CRU. No evidence of this has either been presented to the Review, nor has it been assembled as a scientific study published elsewhere and subjected to scrutiny. For the same reasons we found no evidence that there is anything wrong with the CRU publications using the Yamal or other tree series.

36. We find that divergence is well acknowledged in the literature, including CRU

papers.

37. In relation to ―hide the decline‖ we find that, given it s subsequent iconic significance (not least the use of a similar figure in the IPCC TAR), the figure supplied for the WMO Report was misleading in two regards. It did not make clear that in one case the data post 1960 was excluded, and it was not explicit on the fact that proxy and instrumental data were spliced together. We do not find that it is misleading to curtail reconstructions at some point per se, but that the reason for doing so should have been described.

38. We find that CRU has not withheld the underlying raw Yamal data (having correctly directed the single request to the owners). But it is evidently true that access to the raw data was not simple until it was archived in 2009 and this can rightly be criticised on general principles. In the interests of transparency, we believe CRU should have ensured that the data they did not own, but had relied upon in publications, was archived in a more timely way.

14 Briffa KR and Melvin TM (2010 in press)CHAPTER 7: TEMPERATURE RECONSTRUCTIONS FROM TREE RING ANALYSIS

63 39. It is a matter for the IPCC Review to determine whether the conclusions were in line with IPCC processes and guidelines for levels of likelihood. In respect of that Review we offer the suggestion that putting the combination of different reconstructions upon a more rigorous statistical footing would help in the future to make confidence levels more objective.CHAPTER 8: PEER REVIEW AND INFLUENCING EDITORIAL POLICY OF SCIENTIFIC JOURNALS

64 

## CHAPTER 8: PEER REVIEW AND INFLUENCING

EDITORIAL POLICY OF SCIENTIFIC JOURNALS

8.1 Background: Peer Review, Testing and Verification

1. The rigorous development of scientific ideas leans heavily on peer-reviewed publication in scientific journals that are subject to testing and challenge by other, published, scientifically derived evidence.

2. The processes whereby new scientific concepts are peer -reviewed and published can be complex, contentious and partial (see Chapter 5 and Appendix 5). Peer review is not a ―gold standard‖ that ensures validity, as some claim. The processes of testing, validation and replication are the means whereby error is discarded, uncertainty reduced and robust scientific progress made. Peer review attempts initial sieving-out of the demonstrably erroneous or trivial, thereby setting a high standard that encourages rigour. But it is not infallible. Many well-founded concepts are rejected and many erroneous ideas accepted.

3. If peer review is subverted to exclude apparently well-founded alternative views, or if journal editors are intimidated from considering their publication, progress on an important issue such as climate change can be seriously slowed or skewed.



## 8.2 The Allegations

4. The broad allegation is that CRU made improper attempts to influence the peer review system, pressuring journals to reject submitted articles that did not support a particular view of climate change.

5. This can only be properly assessed by analysis of the individual cases cited. Specific examples used to support this allegation are the events surrounding the publication of a paper by Soon and Baliunas (2003) 1 , the issues relating to Dr Boehmer-Christiansen, editor of the journal Energy and Environment , and certain

actions of Professor Briffa when he was editor of the Journal Holocene

.



## 8.3 The Soon and Baliunas Affair & Climate Research

6. The paper by Soon and Baliunas (2003) entitled Proxy climatic and environmental changes of the past 1000 years , published in the journal Climate Research

, reviewed 240 previously published papers on temperature trends during the last millennium. It challenged the conclusion of Mann et al (1998, 1999) 2 that the late

20 th century was the warmest period of the last millennium on a hemispheric scale, and claimed that recent temperatures were by no means unprecedented over

1 Soon, W. and Baliunas, S. 2003. Proxy climatic and environmental changes of the past 1000 years.

Climate Research 23 , 89- 110.

2 Mann ME, Bradley RS, Hughes MK (1998) Global-scale temperature patterns and climate forcing

ov er the past six centuries. Nature 392 :779

– 787. Mann ME, Bradley RS, Hughes MK (1999) Northern Hemisphere temperatures during the past millennium: inferences, uncertainties, and limitations. Geophys Res Lett , 26:759

– 762.CHAPTER 8: PEER REVIEW AND INFLUENCING EDITORIAL POLICY OF SCIENTIFIC JOURNALS

65 this period. It was greeted with enthusiasm by those sceptical of the hypothesis of anthropogenic global warming 3 . However it received a negative reception from many other climate scientists 4 on scientific grounds, viz. that it conflated qualitative data on temperature and precipitation from many sources that could not be combined into a consistent proxy record. That hostility is reflected in the released CRU e-mails (e.g. CRU 1051156418.txt, 1051202354.txt), and the following e-mail from Jones to Mann on 11.3.03 (1047388489.txt):

“I think the skeptics will use this paper to their own ends and it will set paleo back a number of years if it goes unchallenged. I will be emailing the journal to tell

them I'm having nothing more to do with it until they rid themselves of this troublesome editor (de Freitas), a well-known skeptic in NZ. A CRU person is on the board but papers get dealt with by the editor assigned by Hans von Storch."

7. The S&B paper had been seen by four reviewers, none of whom had recommended rejection 5 , and had been accepted by de Freitas (one of 10 Climate

Research review editors; papers could be submitted to any one of them). A number of review editors resigned as a reaction against the publication of what they regarded as a seriously flawed paper. The journal‘s publisher admitted that the journal should have requested appropriate revisions of the manuscript prior to publication 6 . The Editor in Chief resigned on being refused permission by the publisher to write an editorial about what he regarded as a failure of the peer review system. De Freitas was said to have described these events as ―a mix of a

witch- hunt and the Spanish Inquisition‖ 7

.

8. These events, and the e-mail quote in paragraph 6, have led to the allegation that normal procedures of publications were being improperly undermined by a group

that included Jones 8

.

9. Jones has responded in evidence 9 to us that the reaction to the S&B paper was not improper or disproportionate given what he saw as the self evident errors of the

paper. The arguments presented in the Eos article, if correct, are strongly put, and suggest that the reaction was based on a belief, for which evidence was adduced, that the science was poor. In light of the reaction of the Journa l‘s publisher , we do not believe that any criticism of Jones can be justified in this regard.

10. Finding: This was clearly a bruising experience for all concerned. But Richard Horton‘s paper (Appendix 5 ) and the comments on it in Chapter 5 suggest to the Team that this scale of reaction is not unusual in contested areas, and the peer review process does not provide insulation from it. The Review makes no judgement about the correctness or otherwise of the Soon and Baliunas paper, but we conclude that the strong reaction to it was understandable, and did not amount to undue pressure on Climate Research.

3 New York Times , 5 August 2003.

4 e.g. Eos , Vol. 84, No. 27, 8 July 2003.

5 http://www.sgr.org.uk/climate/StormyTimes_NL28 htm

6 A statement from Inter-Research, 2003. Climate Research, 24 , 197- 198

7 http://www.sgr.org.uk/climate/StormyTimes_NL28 htm

8 Montford, A.W. 2010. The Hockey Stick Illusion . Stacey International, London.

9 See the note of the CCER meetings held on 9 April 2010CHAPTER 8: PEER REVIEW AND INFLUENCING EDITORIAL POLICY OF SCIENTIFIC JOURNALS

66 

## 8.4 The Conflict with Dr Boehmer-Christiansen

11. Dr Boehmer-Christiansen is Reader Emeritus in Geography at the University of Hull and the editor of the scientific journal Energy and Environment . As part of her evidence to us, she submitted a copy of her evidence to the House of Commons Science and Technology Committee Select Committee for its inquiry
into ― The disclosure of climate data from CRU at the UEA ‖. In it she wrote 10 :

“ As editor of a journal which remained open to scientists who challenged the orthodoxy, I became the target of a number of CRU manoeuvres. The hacked emails revealed attempts to manipulate peer review to E&E‟s disadvantage, and showed that libel threats were considered against its editorial team. Dr Jones even tried to put pressure on my university department. The emailers expressed anger over my publication of several papers that questioned the „hockey stick‟ graph and the reliability of CRU temperature data. The desire to control the peer review process in their favour is expressed several times”.

12. Apart from the allegation of  ̳ att empts to manipulate peer review‘ for which Boehmer-Christiansen presents no evidence in relation to the journal that she edits, her statement 11 to us implies that pressure on her University was designed to

undermine her role as editor of Energy and Environment. She comments that ― a message was send to my head of department late last year by Phil Jones in relation to some other matter, which suggested that the University of Hull should dissociate itself from me as editor of Energy & Environment (Multi-Science) as I was causing difficulties for CRU .‖

13. This episode appears to have been precipitated by an email sent on 2 October 2009 (1254746802.txt) from Boehmer-Christiansen to Stephanie Ferguson at the UK Climate Impacts Programme, and copied to a number of others. Its title was : “Please take note of potetially [sic] serious scientific fraud by CRU and Met Office.” The email was brought to the attention of Jones by the ―help desk‖ of UKCP09 in the UK Department for Environment, Food and Rural Affairs (DEFRA). He then wrote an e-mail on 27 October 2009 (1256765544.txt) to Professor Graham Haughton, Boehmer-Christianse n‘s Head of Department, complaining about the e- mail as ― very malicious”. Haughton responded sympathetically, but commented that “I‟d want to protect another academic‟s freedom to be contrary and critical”, to which Jones replied: “I don‟t think there is anything more you can do. I have vented my frustration and have had a considered reply from you”.

14. Finding: We see nothing in these exchanges or in Boehmer- Christiansen‘s evidence that supports any allegation that CRU has directly and improperly attempted to influence the journal that she edits. Jones‘ response to her accusation of scientific fraud was appropriate, measured and restrained.

10 Memorandum submitted by Dr Sonja Boehmer-Christiansen (CRU 26), at 4.1

11 Dr Sonja Boehmer-Christiansen submission (no.43), 23 FebruaryCHAPTER 8: PEER REVIEW AND INFLUENCING EDITORIAL POLICY OF SCIENTIFIC JOURNALS

67 

## 8.5 Peer Review and Professor Briffa‟s E ditorship of

Holocene

15. An e-mail (1054748574.txt) was sent by Briffa, as Editor of the journal Holocene

, on 4.06.03 to Ed Cook, a reviewer, as follows:

“I am really sorry but I have to nag about that review – Confidentially I now need a hard and if required an extensive case for rejecting

– to support Dave Stahle‟s and really as soon as you can.”

This has been assumed to be an editor who wishes to have the paper rejected, for reasons that are not explicit, seeking help from a reviewer to do so. In general, reviewers are not, or should not be invited to reject, merely to review. This e-mail has been widely interpreted as Briffa perverting the purpose of peer review by asking a colleague for help in rejecting a paper that contained research findings contradictory to his own views 12

.

16. However, put in context, the e- mail‘s significance changes. The reality is r eflected in a series of emails provided by Briffa 13 and which have been redacted to remove names, apart from that of Stahle, as follows :

17.06.02

–

Briffa requests references from Dr Dave Stahle and referee B for a paper submitted to Holocene on the use of a specific tree ring record.

07.08.02

–

Stahle submits a referee report to Briffa. It is strongly critical of some of the underlying analysis, though it comments that the paper is well written. Advises rejection but suggests that if substantial additional work were done, it could be suitable for publication.

09.08.02

–

Briffa to Stahle, thanking him for the review, and committing himself to reading the paper very carefully in anticipation of receipt of a

second review.

28.05.03

– Briffa to referee B, pressing to send a second review.

0 4.06.0 3

– Briffa reiterates his request to referee B, sending the email copied in paragraph 15.

04.06.03

–

Referee B sends review to Briffa. In the event, the review is not as negative as that of Stahle: the paper is ―marginal at best, could justifiably be rejec ted‖. This is an immediate reply to the email copied in paragraph 15.

23.07.03

– One of the authors e-mails to Briffa enquiring about a decision on the paper.

24.07.03

–

Briffa to author. Apologis es for the delay (―the manuscript could

have been dealt wit h much better‖) and encloses referees‘ comments. Offers possible fast-track publication if the referees comments could be dealt with.

We can find no evidence that the article was subsequently published but the evidence above demonstrates that the possibility of publication was not rejected.

12 e.g. Channel 4 News 3.2.10

13 Copies of Communications relating to Keith Briffa‟s editorial treatment of a submitted manuscript

. Available at www.cce-review.org/Evidence.phpCHAPTER 8: PEER REVIEW AND INFLUENCING EDITORIAL POLICY OF SCIENTIFIC JOURNALS

68 17. Finding: Although much has been made of the e-mail in paragraph 15 as evidence of an unprincipled approach to the role of editor, we see nothing in these exchanges that supports the interpretations of subverting the peer review process that have been placed upon it. It appears to reflect an Editor with a strongly negative review in hand, and who presumably has read the paper, asking for confirmation that the paper should be rejected, possibly to reduce one of the many complications that assail an editor; and in view of the delay in communicating to authors, hoping for a strong decision from the referee. On receiving a second, more equivocal review, he offers the authors the opportunity to re-submit. These exchanges illustrate some of the complications of an E ditor‘s life as referred to in Chapter 5 and in Appendix 5. They do not provide evidence of subversion of process in rejecting contradictory ideas as has been alleged.



## 8.6 Conclusions

18. In our judgement none of the above instances represents subversio n of the peer review process nor unreasonable attempts to influence the editorial policy of journals. It might be thought that this reflects a pattern of behaviour that is partial and aggressive, but we think it more plausible that it reflects the rough and tumble of interaction in an area of science that has become heavily contested and where strongly opposed and aggressively expressed positions have been taken up on both sides. The evidence from an editor of a journal in an often strongly contested area such as medicine (Appendix 5) suggests that such instances are common and that they do not in general threaten the integrity of peer review or publication.CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC

69CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC

70 

## CHAPTER 9: COMMUNICATING INTO THE PUBLIC

DOMAIN THROUGH THE IPCC

9

. 1 Background

1. Where scientific results and concepts are relevant to issues of public or policy interest, their complexities are often such that they need to be communicated in simpler, comprehensible language that includes explicit statements about

uncertainties and errors.

2. The IPCC process and its assessment reports have been the prime routes through which the complexities of climate change are transmitted into the international governmental domain. Actions that undermined the rigour, honesty and expressions of scientific uncertainty in such communications would be serious infringements of good scientific practice.



## 9.2 The Allegations

3. The above principles are the frame for allegations levelled at CRU that relate to
the Review‘s remit . In broad terms the claim is that CRU attempted to prevent proper consideration of views which conflicted with their own through their roles

in the IPCC.

4. If this is the case it would represent a failure to represent current scientific understanding impartially at the vital interface between science and policy. If this has been done, it has the potential to understate uncertainty, and to undermine the rigour of risk-based evaluations by government, with the potential for severe social and economic consequences.

5. There are two specific, analogous allegations. They primarily relate to papers that challenge important elements of IPCC work; the first about the interpretation of the CRUTEM instrumental temperature series and the second about the reconstruction of tree ring proxy temperature series, discussed earlier in different contexts in Chapters 6 and 7 respectively. It is alleged that these papers were not properly considered by the relevant IPCC writing groups, and that members of CRU played the major role in ensuring that this was not done.

6. The following approach is adopted for each of these two sets of allegations:

summary of the scientific basis of the challenge;

allegations about the improper treatment of the papers by CRU members in their roles in IPCC writing groups;

evidence in support of the allegations;

the response of the CRU members alleged to have been primarily involved;

evidence from the IPCC Review Editors (see Chapter 5, section 5.7) about the procedures of the relevant IPCC writing groups;

findings of the Review.CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC

71 

## 9. 3 The CRUTEM Temperature Series

9. 3.1 The Scientific Challenge

7. A paper by McKitrick & Michaels (2004, hereafter MM2004) 1 argued that a large proportion of the measured late 20 th century warming was a consequence of increased economic activity and that many meteorological measurement sites had become increasingly influenced by the warming effect of urbanisation around them, the so- called ―heat island effect‖, and that the CRUTEM d ata set was not adequately adjusted for such non-climatic effects. They argued that removing the

effects of this socio-economic transition would reduce the trend of land surface temperatures from 0.27 deg C/decade to 0.11 deg C/decade and possibly to 0.06 deg C/decade. If this claim were correct it would, for many, fundamentally undermine the argument that the increasing concentration of atmospheric carbon dioxide as a consequence of human activity was a major driver of climate change.



## 9. 3.2 The Allegations

8. The allegation that CRU withheld their data from scrutiny and adjusted it without justification is dealt with in Chapter 6. Here we are concerned with the specific allegation that CRU improperly dismissed contrary views about the interpretation of the CRUTEM series and prevented its serious consideration by the IPCC.



## 9. 3.3 Evidence in Support of the Allegations

9. The reaction of Jones, who was then one of two CLAs for Chapter 3 ( Observations: Surface and Atmospheric Climate Change ) of the Working Group 1 report of IPCC AR4, to the MM2004 paper is revealed in an e-mail (1089318616.txt) sent on 8 th July 2004 to Mann, that included: “The other paper by MM (McKitrick & Michaels, 2004) is just garbage. ... I can‟t see either of these papers being in the next IPCC report. Kevin (Trenberth, the other coordinating lead author for ch. 3) and I will keep them out somehow

– even if we have to redefine what the peer- review literature is!” It indicates clearly his awareness of the paper prior to any drafts of IPCC AR4, and his determination to exclude it from consideration by IPCC.

10. It has been suggested 2 that this determination to exclude is reflected in the fact that MM2004 was not referred to in either the first order draft of Chapter 3 in August 2005 or in the second order draft of March 2006. Nor was a paper drawing similar conclusions by de Laat and Maurellis (2006) 3 referred to. It has been suggested to us by McIntyre 4 that it was Jones who kept reference to these papers out of the first and second order drafts, with the implication that this was done improperly to prevent incorporation of conclusions contrary to those held by the CRU group.

1 McKitrick, R. and Michaels, P.J. 2004. A test of corrections for extraneous signals in gridded surface temperature data. Climate Research, 26 , 159- 173.

2 McKitrick submission (no.15), 26 February

3 de Laat, A.T.J. and Maurellis, A.N. 2006. Evidence for influence of anthropogenic surface processes on lower tropospheric and surface temperature trends. International Journal of Climatology, 26, 897

- 913.

4 McIntyre submission (no. 23), 2 MarchCHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC

72 11. The omission of reference to MM2004 was criticised by expert reviewer Vincent Gray for Chapter 3, but this was rejected by the IPCC writing team for this Chapter (see Chapter 5 paragraph 24) with the argument that large scale patterns of atmospheric circulation over the continents and stronger warming of th e continents would produce patterns of warming such as those recorded by temperature stations 5

.

12. The final draft of Chapter 3 did however include reference both to McKitrick & Michaels (2004) and de Laat and Maurellis (2006), in the following paragraph:

“McKitrick and Michaels (2004) and De Laat and Maurellis (2006) attempted to demonstrate that geographical patterns of warming trends over land are strongly correlated with geographical patterns of industrial and socioeconomic development, implying that urbanisation and related land surface changes have caused much of the observed warming. However, the locations of greatest socioeconomic development are also those that have been most warmed byatmospheric circulation changes (Sections 3.2.2.7 and 3.6.4), which exhibit largescale coherence. Hence, the correlation of warming with industrial and socioeconomic development ceases to be statistically significant. In addition, observed warming has been, and transient greenhouse-induced warming is expected to be, greater over land than over the oceans (Chapter 10), owing to the smaller thermal capacity of the land.”

13. It has been surmised in a submission to the Review by McKitrick 6 that Jones wrote the above paragraph and bears responsibility for its inclusion. The same submission alleges that it represents a ―fabricated conclusion‖ or ―invented evidence‖ and that only the derivation of a p -value 7 from a statistical test that is compatible with the claim of statistical insignificance would rebut this allegation of fabrication. It alleges that this is evidence of bias, and that after attempts to exclude evidence that conflicted with the preferred CRU interpretation of the CRUTEM data series from drafts 1 and 2, reasons were contrived in the published draft for the specific purpose of rejecting this evidence. If this were correct, such actions would appear to violate the principles in Chapter 5, of the duty of scientists to ensure that uncertainties are clearly transmitted to those that have the responsibility for deciding on any contingent actions.

5 Gray: “The „corrections‟ to the surface temperature record have always been based on very poor evidence. The many references to studies on individual or regional stations which find the need for much higher corrections than are currently applied, are ignored. Now you have ignored the persuasive evidence of McKitrick and Michaels 2004 Climatic Research 26 156-173 who have shown a significant influence on your „corrected‟ figures of a series of socioeconomic factors. You cannot just ignore this paper.” Response of the IPCC writing team: “ References are plentiful. Those of value are cited. Rejected. The locations of socioeconomic development happen to have coincided with maximum warming, not for the reasons given by McKitrick and Michaels (2004) but because of the strengthening of the Arctic Oscillation and the greater sensitivity of land than ocean to greenhouse forcing owing to the smaller thermal cap acity of land. Parker (2005) demonstrates lack of urban influences.” http://pds.lib harvard.edu/pds/view/7795947?n=7&s=4&imagesize=1200&jp2Res=.25&rotation=0 (comment 3-34).

6 McKitrick submission (no.15), 26 February

7 The p -value in statistics is the probability of obtaining a statistic at least as extreme as the one actually observed, assuming that there is no relationship between the measured phenomena. The lower the p-value, the less likely that there is no relationship. McKitrick and Michaels (2004) obtained a pvalue of 0.002, suggesting a significant relationship between climatic and socio-economic trends.CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC

73 14. It has been further alleged 8 that the changing response to the McKitrick and Michaels (2004) paper between the response to Gray and the statement in paragraph 13 of AR4, both assumed to have been by Jones, is evidence of a preparedness to accept any argument, irrespective of how well founded, that would refute conclusions to which he was opposed.



## 9. 3.4 Jones‟ R esponse

15. These allegations were put to Jones. We summarise his oral and written responses

as follows 9

, 10 :

Jones comments that the e-mail of July 8, 2004 was sent on the spur of the moment and quickly forgotten. No pattern of behaviour with respect to his

IPCC work could be construed from this one email.

The reason for the strong response of the email and the justification for not including reference MM2004 in the early drafts is that it can be readily shown to be scientifically flawed.

The basis for this statement is that if the CRUTEM3 trend is reduced by the factor claimed by MM2004, the land-based record then becomes incompatible

with the ocean and the satellite record. MM2004 make no mention of this in their paper. In writing Chapter 3 of AR4 the author team were mindful of this. MM2004‘s analysis of the land surface temperature record is completely at odds with the rest of the surface and lower tropospheric temperature records. MM2004 also fails to take into account the effects of changes in the atmospheric circulation.

In summary, the atmospheric circulation has been shown, in numerous studies, to account for large scale patterns of temperature change. Before undertaking the kind of analysis in MM20 04, it is essential to account for known signals (i.e. the N orth Atlantic Oscillation (NAO) 11 and El Nino Southern Oscillation (ENSO) 12 and possibly others) and then examine the residuals. It does not

make sense to calculate the p -value without allowing first for the atmospheric effects and their spatial autocorrelation in that calculation.

In view of these arguments, it was reasonable to exclude MM2004, as the IPCC reports are assessments, not reviews of the literature. The author teams were chosen for their experience and expertise, and reliance was placed on this in determining which work should be included and discussed in the

8 McKitrick submission (no.15), 26 February

9 Response from Professor Jones to questions from Professor Geoffrey Boulton (20 April)

10 Clarification from Professor Jones to additional questions from Professor Geoffrey Boulton (3 May)

11 The North Atlantic Oscillation (NAO) and the associated Arctic Oscillation are parts of the global wave structure in the northern hemisphere atmosphere. The Icelandic low and the Azores high are parts of this wave structure, and their east-west oscillations control the strength and direction of westerly

winds and storm tracks across the North Atlantic.

12 ENSO (El Niño-Southern Oscillation) is a 5-year (on average) oscillation between warm and upwelling cold waters in the tropical eastern Pacific, which has a major impact on atmospheric pressure in the region and knock-on effects on weather in many other parts of the world.CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC

74 assessment.

Jones commented that the decision about which papers to include and which to exclude was a collective one of the author team of Chapter 3 of AR4. Jones stated that he did not write the relevant text in the final report as has been assumed: the lead on the relevant section (3.2) was another member of the writing team. The suggestion to make a response did not come from Jones, as he was not the responsible person for the section. He did, however, agree with its inclusion in the final draft as a part of the overall writing team and a CLA

fo r Chapter 3.

Jones explained that the decision to include MM20 04 (and de Laat and Maurellis, 2006) was made at the final plenary team meeting in Bergen, and as stated the text was seen by the whole writing team. It had not been possible to include de Laat and Maurellis (2006) until then, as it was not published until after the third Lead Author‘s meeting. Discussion of MM 20 04 can be seen in comments numbered 3-283 to 3-289 of the Second Order Draft of Chapter 3 13

. In two of these comments (3-284 and 3-285) it was stated that reference would

be made to MM 20 04 in Section 3.2.2.2 with some text, which would point out that the papers by MM 20 0 4 and de Laat and Maurellis were biased. The fact that the Chapter author team had now read de Laat and Maurellis is referred to in response to comment 3-289. The comments were signed off by the two Review Editors for the Chapter.

Jones also noted to us that there were three criticisms made of the draft of the Summary for Policy Makers (SPM) based on those of MM2004, but that these were rejected by the SPM writing team (which did not include any members of the Chapter 3 team). They regarded them as inconsistent with a large body of the climatological literature addressed in Chapter 3 (comments numbers 482, 864 and 1005) 14

.



## 9. 3.5 Evidence from IPCC Review Editor for Chapter 3 (Professor Sir Brian Hoskins)

16. The role of Review Editors was to ensure that all comments from expert and government reviewers were given appropriate consideration, to advise lead

authors how to handle contentious and controversial issues and to ensure that genuine controversies were reflected adequately in the text. Professor Hoskins was one of three Review Editors for Chapter 3. Telephone evidence was sought from Hoskins about the ways of working of the author team for Chapter 3. The full summary of the interview with Hoskins is available on the Review website 15

.

17. Hoskins confirmed that LAs, working individually and as small groups, were responsible for the collation and primary assessment of material relevant to the

13 Comments are available at http://hcl.harvard.edu/collections/ipcc/ by scrolling down to appropriate

comments.

14 Ibid. 13

15 Evidence from Review Editors for Chapters 3 and 6 of the IPCC Fourth Assessment Report on “The Physical Science Basis” (June 2010)CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC

75 topics for which they were responsible. The CLAs led the plenary meetings of the writing team prior to production of each of the drafts, led the process of overall collation of the Chapter material and the production of the initial drafts of the First and Second Draft Reports and the Final Draft Report of the Chapter. These drafts were discussed and agreed during plenary meetings of the whole writing team.

18. There were a very large number of comments from reviewers, of which a majority were from a relatively small group. The Review Editors made sure that they were all given proper consideration, and that they were either responded to by a change in the text or by an adequate reason for omission that was recorded in the author responses to expert and governmental review comments 16 . Hoskins, as a Review Editor, took part in the Chapter 3 plenary discussions and ensured that conflicting

views were addressed.

19. Led by the two CLAs, Jones and Trenberth, the writing team for Chapter 3 was assiduous in dealing with comments. Hoskins was very impressed by Jones‘ attention to detail, and the rigour of the Chapter 3 process.

20. The levels of confidence and uncertainty reflected in the drafts were based on the consensus of a group of CLAs and LAs who were chosen for their expertise and experience in relevant fields. Irrespective of whether a paper is published in a peer reviewed journal, it is the responsibility of the whole team to assess whether a paper‘s conclusions are robust and to justify whether its arguments should carry weight in the assessment. These decisions for each draft were taken in plenary sessions of the whole team. Hoskins said that it is inconceivable that a paper making significant claims relevant to the work of IPCC and the Chapter 3 team would not be considered by the team as a whole. The basis for rejecting one of the papers that is a focus of the allegation is included in IPCC records 17 . Decisions about the inclusion of the MM2004 paper would have been taken by the whole team. Jones‘ voice would have been one amongst many.



## 9. 3.6 Findings

21. The essential issues in determining whether there is substance to the allegation that Jones misused his position to exclude well-founded conclusions that conflicted with his prior views and ―invented‖ an explanation in order to reject

them are as follows:

i) Was the decision to exclude the MM2004 paper from the first and second drafts

an unreasonable one?

ii) Was the reason given for the rebuttal of MM2004 in the final draft ―invented‖?

iii) If the answer to either or both of these questions is ―yes‖, w as Jones the major

16 Available at: http://hcl.harvard.edu/collections/ipcc

17 The issue is discussed further in discussion of reviewers comments on the 2 nd order draft of Chapter 3 (footnote to comments 3-275 to 3-389), in part of which McKitrick objected to the exclusion of the MM2004 and de Laat and Michaelis 2004, to which the writing team responded by the addition of text to section 3.2.2.2 of the draft. This was included in the final draft of Chapter 3. The comments and responses can be found at: http://pds.lib harvard.edu/pds/view/7786376?n=40&s=4&imagesize=1200&jp2Res=.25&rotation=0CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC

76 influence in excluding early consideration and/or in providing an ―invented‖ explanation?

22. The answer to i) depends upon the scientific credibility of the article. The contemporary emails indicate a highly critical response to it, and Jones‘ email (paragraph 9) is very strongly negative. Was the absence of a discussion of MM2004 in the first and second drafts reasonable, or could it reflect suppression of a view merely because it conflicted with that of the writing team or of Jones? Was the analysis in MM2004, and later in de Laat and Maurellis (2006), so evidently flawed in an issue as important as the significance of the instrumental record of climate change (see Chapter 6), that it could be readily rejected in the AR4? Jones believes that it was (paragraph 15, bullets 2-4), although the persistence of a debate on the issue in peer reviewed journals, including on one side MM2004, de Laat and Maurellis (2006), McKitrick and Michaels (2007) 18

and McKitrick (2010) 19 , and on the other side Benestad (2004) 20 , Parker (2010) 21

, Jones et al (2008) 22 and Schmidt (2009) 23 , suggests at least a continuing margin of doubt. Those within the writing team took one view, and a group outside it took another. It is not in our remit to comment on the rights and wrongs of this debate, but those within the team had been entrusted with the responsibility of forming a view, and that is what they did. They initially rejected inclusion of reference to MM2004, and in the final draft included a commentary on it explaining why they disagreed with its conclusions. It may be that the conclusions of MM2004 conflicted so strongly with a generally held view among climate scientists tha t rejection was made too easily; but in the absence of better evidence, this is mere speculation. The mechanisms of the IPCC did however ensure that a reference to the article (and to de Laat and Maurellis, 2006) was contained in the final draft.

23. The answer to ii) depends upon the implication that the response to MM2004 in the published Chapter 3 was not scientifically credible. Having read most of the relevant papers however, we observe a consistence of view amongst those who disagree with MM2004 that has been sustained over the last 6 years, that the large scale organisation of atmospheric circulation produces a spatially integrated response to forcing. Although we do not comment on the relative merits of the two views, we see no justification of the view that that this response was ―invented‖, or even that its various expressions in the response to reviewer Gray 24
or the final text are fundamentally different.

24. Irrespective of the above comments on issues i) and ii), the evidence of the Review Editor underlines the team responsibility for the text, and the unlikelihood

18 McKitrick R and Michaels P. 2007. Quantifying the influence of anthropogenic surface processes and inhomogeneities on gridded global climate data. Journal of Geophysical Research 112 : D24S09.

19 McKitrick, R., 2010: Atmospheric circulations do not explain the temperature-industrtrialization

correlation. Statistics, Politics and Policy (in press

– summer 2010).

20 Benestad, R. 2004. Are temperature trends affected by economic activity? Comment on McKitrick

and Michaels. Climate Research, 27 , 171- 173.

21 Parker, D. E., 2010: Urban heat island effects on estimates of observed climate change, Wiley Interdisciplinary Reviews: Climate Change

, 1 (1), 123-133, doi:10.1002/wcc.21.

22 Jones, P.D., Lister, D.H. and Li, Q., 2008: Urbanization effects in large-scale temperature records, with an emphasis on China. J. Geophys. Res

. 113 , D16122, doi:10.1029/2008/JD009916.

23 Schmidt, G.A., 2009: Spurious correlations between recent warming and indices of local economic activity. Int. J. Climatol

. 29 , 2041-2048.

24 Ibid. 5CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC

77 that a single voice could dominate on an important issue. Jones ‘ evidence is that
McKitrick‘s ― surmise that Professor Jones ... wrote the paragraph (in Chapter 3) alone or in consultation with Trenberth, and bears responsibility for its
inclusion” 25 is false, and that the lead responsibility for the relevant section was another specified member of the writing team.

25. We conclude that there is evidence that the text was a team responsibility. It is clear that Jones (though not alone) had a strongly negative view of the paper but we do not find that he was biased, that there was any improper exclusion of material or that the comments on the MM2004 paper in the final draft were ―invented‖ given the (continuing) nature of the scientific debate on this issue.



## 9. 4 The Tree Ring Proxy Temperature Series

9. 4.1 The Scientific Challenge

26. A paper by McIntyre and McKitrick (2003; hereafter referred to as M&M2003) 26
argued that the so called ―hockey stick‖ plot (Mann, Bradley and Hughes, 1998; hereafter referred to as MBH98) 27 contained both simple errors and serious statistical errors. It suggested that the ―hockey stick‖ shape of the MBH98 reconstruction was largely an artefact of these errors and of the selection of specific tree ring series.

27. This work posed a significant challenge to interpretation of the assessment by the IPCC of the history of climate of the last millennium prior to the instrumental record of the last 150 years. It cast doubt on the validity of the claim of MBH98 and of Mann, Bradley and Hughes (1999; hereafter MBH99) 28 that the northern hemisphere was warmer in the late 20 th century than at any time during the last millennium, which has been assumed by many to be evidence in support of the argument for strong, human induced greenhouse warming over recent decades.



## 9. 4.2 The Allegations

28. It has been alleged that Briffa, in his role as lead author for Chapter 6 in Working Group 1 for AR4, and as the member of the writing team with the most relevant expertise, attempted to bias the scientific conclusions towards those of the MBH98/99 and to set aside the inconvenient evidence of M&M2003. It is alleged this behaviour was calculated to favour one particular view of climate change and its causes, and to discredit or ignore opposing views, without, at the time, an adequate scientific reason for doing so. It would therefore represent a failure to discharge a scientist‘s responsibility impartially to represent current scientific understanding at the vital interface between science and policy.

25 McKitrick submission (no. 15), 26 February

26 McIntyre, S. and McKitrick, R. 2003. Corrections to the Mann et al (1998) proxy database and northern hemisphere average temperature series. Energy and Environment, 14, 751- 771.

27 Mann, M.E., Bradley, R.S. and Hughes, M.K. 1998. Global-scale temperature patterns and climate forcing over the past six centuries. Nature, 392 , 779-787.

28 Mann, M.E., Bradley, R.S. and Hughes. M.K. 1999. Northern hemisphere temperatures during the last millenium: inferences, uncertainties and limitations. Geophysical Research Letters, 26(6), 759

- 762.CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC

78 

## 9. 4.3 Evidence in Support of the Allegations

29. The Chapter 6 writing team relied heavily on a paper that was in preparation by Wahl and Ammann (eventually published as Wahl and Ammann, 2007, hereafter referred to as WA2007) that purported to refute the arguments of M&M2003. I t has been alleged that Briffa played a central role in improperly using WA2007 to refute M&M2003, to discredit a paper that conflicted with his core hypothesis, and in doing so was willing to break IPCC rules:

The claim of refutation made in the second order and the published drafts of Chapter 6 was knowingly misleading, because the method used to support the conclusions of WA2007 was dubious, and in any case, relied upon material that was rejected for publication in Geophysical Research Letters , and was not made available until August 2008 in an online supplement to Ammann and Wahl 2007, hereafter referred to as AW2007, long after acceptance of the final draft of Chapter 6. Thus even if the paper did represent an effective refutation of the M&M2003 argument, it was not available prior to the acceptance of the final draft of Chapter 6 of AR4.

IPCC rules require papers that are to be referenced should be at least  ̳accepted‘ by journals by specific deadline dates. W A2007 missed these deadlines and should not have been quoted as evidence.

Briffa broke confidence by asking Wahl, who was not involved in the IPCC process, to comment on Chapter 6 text.

30. The second order draft text of IPCC 2007 WGI Chapter 6, assumed in the allegation to have been written by Briffa, and sent late in March 2006 to the Government and Expert Reviewers, included on page 29 the following text that

relied on WA2007 to rebut M&M2003:

“McIntyre and McKitrick (2003) reported that they were unable to replicate the results of Mann et al. (1998). Wahl and Ammann (accepted) demonstrated that this was due to the omission by McIntyre and McKitrick of several proxy series used by Mann et al. (1998). Wahl and Ammann (accepted) were able to reproduce the original reconstruction closely when all records were included”.

31. This text was criticised by the Reviewer for the Government of the United States of America, who wrote in comment 6-750 29 :

“The use of Wahl and Ammann (accepted) does not comply with WG1‟s deadlines and all text based on this reference should be deleted. WG1‟s rules require that all references be “published or in print” by De cember 16, 2005. Wahl and

Ammann was “provisionally accepted” on that date, and not fully accepted until February 28, 2006, at which time no final preprint was available. Substantial changes were made in the paper between December 16, 2005 and February 28,

29 Comments are available at http://hcl.harvard.edu/collections/ipcc/CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC

79 2006, including insertion of tables showing that the MBH98 reconstruction failed verification with r-squared statistics, as had been reported by McIntyre and McKitrick in 2003. These tables were not available in the draft considered by WG1 when developing the second- order draft.” The response to this was: “Rejected – the citation i s allowed under current rules” (Comment 6-1158) 30

.

32. It was alleged that the material derived from WA2007 that was the rationale for the text in the final version of Chapter 6 was based on material that was not published or openly available until after the last deadline for the final draft. Their

evidence should therefore not have been included.

33. In an email dated 18 July 2006 (1153470204), Briffa wrote to Wahl, who was not an official Expert Reviewer, as follows:

“Gene I am taking the liberty (confidentially) to send you a copy of the reviewers comments (please keep these to yourself) of the last IPCC draft chapter. I am concerned that I am not as objective as perhaps I should be and would appreciate your take on the comments from number 6-737 onwards, that relate to your reassessment of the Mann et al work. I have to consider whether the current text is fair or whether I should change things in the light of the sceptic comments. In practise this brief version has evolved and there is little scope for additional text, but I must put on record responses to these comments - any confidential help , opinions are appreciated . I have only days now to complete this revision and response. note that the sub heading 6.6 the last 2000 years is page 27 on the original (commented) draft. Cheers Keith”

34. It is alleged that this e- mail is an appeal to a strong proponent of the ―hockey stick‖ plot for assistance in coping with the comments of reviewers sceptical about it, that it hands confidential material to Wahl for him to help rebut the comments from Expert Reviewers critical of the Wahl and Ammann paper, and that it breaks rules of confidentiality at a stage when even official reviewers were denied ready access to review comments. It is implied that Briffa was prepared to go to exceptional and improper lengths to bolster a case that he supported and to defend it against alternative views.

35. Finally, it is alleged that the relevant paragraph on p.466 in Chapter 6 of the AR4 Final Report leaves the last word to Wahl and Ammann, and the reader is left with the clear impression that th e M&M2003 criticisms have been rebutted, although the work claimed to be the rigorous basis of this rebuttal had missed or was long

after IPCC deadlines.

30 Ibid. 29CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC

80 

## 9. 4.4 Responses from Briffa

36. These allegations were put to Briffa. We summarise his oral and written responses

as follows 31

, 32

, 33

, 34 :

a) Evaluation of the M&M2003 work by the Chapter 6 writing team

Briffa responded that the M&M2003 work was taken very seriously by the Chapter 6 writing group. There was genuine concern about whether its claim that the MBH98 reconstruction could not be replicated was correct. It was important to assess the merits of this, as a failure in replication would, if substantiated, be crucial in assessing confidence in the MBH99 reconstruction and the trend of past climate. Did M&M2003 provided sufficient grounds for dismissing the view of temperature change in MBH98?

The evaluation of this issue was very similar in both the second order and final, published versions of Chapter 6. The final version is:

“McIntyre and McKitrick (2003) reported that they were unable to replicate the results of Mann et al. (1998). Wahl and Ammann (2007) showed that this was a consequence of differences in the way McIntyre and McKitrick (2003) had implemented the method of Mann et al. (1998) and

th at the original reconstruction could be closely duplicated using the original proxy data. McIntyre and McKitrick (2005a,b) raised further concerns about the details of the Mann et al. (1998) method, principallyrelating to the independent verification of the reconstruction against 19thcentury instrumental temperature data and to the extraction of the dominant modes of variability present in a network of western North American tree ring chronologies, using Principal Components Analysis. The latter may have some theoretical foundation, but Wahl and Ammann (2006NOTE) also show that the impact on the amplitude of the final reconstruction is very small (~0.05°C; for further discussion of these issues see also Huybers, 2005; McIntyre and McKitrick, 2005c,d; von

S torch and Zorita, 2005).”

The first sentence above reflects the concern that the results of MBH98 could not be replicated.

Briffa noted however that M&M2003 had not followed the MBH98 method and

that Rutherford et al. (2005) 35 had pointed out that the M&M2003 approach to calculation resulted in the elimination of 77 out of the 95 pre-1500 tree ring

31 Response to specific questions raised by Professor Geoffrey Boulton, in his letter of 6 May 2010, in his role as a member of the Muir-Russell Review team (19 May)

32 Issues for discussion with Briffa and Jones on 9 April 2010 (April 10 )

33 Response to Additional Question regarding Keith Briffa‟s request to Eugene Wahl and his response (June 2010)

34 Copies of Communication s relating to Professor Briffa‟s editorial treatm ent of a submitted manuscript (June 20 10)

35 Rutherford, S., M.E. Mann, T.J. Osborn, R.S. Bradley, K.R. Briffa, M.K. Hughes and P.D. Jones, 2005: Proxy-based Northern Hemisphere surface temperature reconstructions: Sensitivity to methodology, predictor network, target season and target domain. Journal of Climate 18 , 2308-232.CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC

81 proxy series used by MBH98.

WA2007 had then shown that the results of MBH98 could be replicated very closely using their implementation of the MBH98 methods and using the same data. This is pointed out in the second sentence of the above paragraph from Chapter 6.

Briffa pointed out that the AR4 text did not state that WA2007 had disproved the concerns of M&M2003. Instead it considered their possible impact on the final reconstruction, citing papers that had assessed this impact including, but not exclusively, WA2007. These results indicated that the impact on the final reconstruction might be relatively small, leading to the view, contained in the AR4 text, that the criticisms raised against the MBH98 reconstruction were not sufficient to discount the evidence that it represented.

Briffa commented that he believed the above quoted paragraph from the published report represented a proper and balanced response to the issues raised by M&M2003, that the issues were important to the assessment and that the reference to WA2007 was a significant part of the debate.

Briffa and his colleague Osborn 36 commented that in any case the MBH98 was only one of 12 such reconstructions in figure 6.10 in Chapter 6, and does not therefore dominate the picture. The M&M2003 series is not presented in that figure as an alternative reconstruction, as McKitrick commented on the first order draft that, the draft ― trots out the straw man that we are selling an alternative climate history, despite our repeated and persistent statements that we are not trying to offer „our‟ climate history curve. ‖ 37

Briffa also rejected the implication that this text was his responsibility, asserting that it was the responsibility of the whole writing group, not of any one person.

b) Deadlines

Briffa rejected the allegation that IPCC rules on deadlines were broken because of a determination to include reference to WA2007 (which was claimed to have been in press at the time). Details of these allegations and the responses to them

from Briffa and Osborn are on the Review website 38

.

Briffa pointed out that the scientific content in AW2007, referred to by WA2007, is equivalent to the content in the manuscript rejected by Geophysical

Research Letters (this was not rejected for its scientific content, but for other editorial reasons).

c) Breaching confidentiality

36 Response to specific questions raised by Professor Geoffrey Boulton, in his letter of 6 May 2010, in his role as a member of the Muir-Russell Review team (May 2010)

37 Minute 6-1319 at http://hcl.harvard.edu/collections/ipcc/

38 Ibid. 33CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC

82 Briffa responded to the allegation of having broken confidentiality in sending draft text to Wahl to comment on, that there is no proscription in the IPCC rules to prevent the author team seeking expert advice when and where needed. The Technical Support Unit (TSU) and the CLAs of Chapter 6 agree that the author team was allowed to seek such advice. Copies of communications from both CLAs (Jansen and Overpeck) and the IPCC WG1 TSU are provided by Briffa (and published on the website) to provide support to Briffa‘s claim that his actions did not contravene IPCC procedures.

Briffa asserts that Wahl was asked for comment on text as a knowledgeable and objective arbiter and as such was a wholly reasonable judge of whether the responses were appropriate. Given his particular expertise on the details of the Mann et al. methodology and most importantly the implications for the character of the Mann et al. reconstruction, Briffa felt justified in seeking his advice and in using specific wording in a very few responses that were based on the text of a paper co-authored by Wahl (AW2007). Wahl did not write any of the main text, though he did make some suggestions for very minor edits. Briffa‘s evidence includes copies of the relevant email exchanges with Wahl 39

, which also confirm that both Jansen and Overpeck, as CLAs , were aware of th e approach to Wahl.



## 9. 4.5 Evidence from IPCC Review Editor for Chapter 6 (Professor John Mitchell)

37. Mitchell was one of two Review Editors for Chapter 6 of the AR4 Working Group 1 report. A telephone interview with him was conducted to establish how

unpu blished and ―in press‖ material was handled and how decisions about the text

of successive drafts were made. He commented as follows 40 :

“I was not aware of the debate about whether the Wahl and Ammann paper had or had not met the deadline for the 2 nd order draft for chapter 6, until after the event. The concentration on specific deadlines however misses the larger point. It must be recognised that if only published sources were used, the report would be two years old by the time of publication. In a fast-moving area such as climate change research, assessments could be significantly behind the times if important, but as yet unpublished, new results could not be used. The assessments for policymakers could also therefore be behind the times”.

“In earl ier assessments, there had been a relatively liberal regime in using unpublished material provided that there was a sound basis for regarding it as rigorous or reliable, although priority was always given to finding published sources. In AR4 however, the regime was tightened significantly, so that such material was only to be used under exceptional circumstances, but the use of unpublished material was not prohibited. „Hockey

- stick‟ issues were regarded at

39 Response to Additional Question regarding Keith Briffa‟s request to Eugene Wahl and his response (June 2010), available at: www.cce-review.org/Evidence.php

40 Evidence from Review Edit ors for Chapters 3 and 6 of the IPCC Fourth Assessment Report on “The Physical Science Basis”CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC

83 the time as sufficiently important to justify using new data. The dilemma between using only published material and being out of date, or using more recent unpublished material was increased in AR4 as the „latest publication date‟ was about 12 months earlier than in the process than in the previous assessm ent”.

“The email is problematic (para 33). On the one hand it appears to reflect an honest request to an expert for a comment about the extent to which the author is being balanced and fair. On the other hand, it stresses the need for confidentiality in three places, implying that the author realizes that the approach may be improper. There was also a leak of an early draft of the WG1 report to the press which led to IPCC emphasizing the need to maintain confidentiality in general which may have been at the back of the author‟s mind”.

“In principle however there is nothing in IPCC rules that prevents an author from seeking external help, comment or judgement on text through consulting his peers. It is questionable whether expert reviewers‟ comments, in some * ways analogous to the comments of a peer reviewer for a journal, should be shared with an third party without their consent. (*Note that unlike most peer-reviewed journals, IPCC reviewers names and comments are made available at the end of the process).”



## 9. 4.6 Findings

38. The essential issues to be resolved in relation to the allegations are:

Was the McIntyre and McKitrick (2003) paper dealt with in a reasonable fashion in the draft and final versions of Chapter 6?

The evidence and narrative provided by Briffa is persuasive that these issues were

dealt with in a careful and reasonable fashion that took into account the importance of the issues addressed in M&M2003. McIntyre or McKitrick may have wished to see them addressed in a different way, but they were addressed seriously and cogently.

Is there any evidence of any personal desire from Briffa to protect a fundamentalist line in defending the conclusions of MBH98/99?

The evidence of the Review Editor suggests that no one person in the writing team

cou ld have overridden the team responsibility for the text. Indeed, the evidence of a contemporary e-mail (1140039406; Feb 2006; to Overpeck) suggests that Briffa was unlikely to be an uncritical d efender of the MBH view of the  ̳ hockey stick ‘

, and wished to respect the view of the writing team as a whole:

― Peck, you have to consider that since the TAR, there has been a lot of argument re„hockey stick‟ and the real independence of the inputs to most subsequent analyses is minimal. True, there have been many different techniques used to aggregate and scale data - but the efficacy of these is still far from established. We should be careful not to push the conclusions beyond what we can securely justify - and this is not much other than a confirmation of the general conclusions of the TAR.....Just need to show the "most likely"course of temperatures over the last 1300 years - which we do well I think. Strong confirmation of TAR is a good result, given that we discuss uncertainty and base it on more data. Let us not try to over egg the pudding. For what it worth, the above comments are my (honestlyCHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC

84 long considered) views - and I would not be happy to go further. Of course this discussion now needs to go to the wider Chapter authorship, but do not let Susan [Solomon

– co -chair of IPCC WG1 for AR4] (or Mike) [Mann] push you (us) beyond where we know is right.”

Is the inclusion of references to WA2007 allegedly against the rules of IPCC

evidence of a determination to rebut M&M2003 at all costs?

Taking into account the evidence of the Review Editor, IPCC papers, the statement from the CLA, and the importance of the issues raised by M&M2003, we consider it to be reasonable that work that might throw further light on these issues and to ensure that assessments were as up to date as possible should be included. We do not consider therefore that these were exceptional unwarranted efforts to defend a particular position, but reasonable attempt so use up to date information that might resolve an issue. They appear to be consistent with IPCC principles and to reflect a concern for objectivity.

Was there breach of confidentiality in having Wahl comment on draft text of the
report and does this reflect determination to sustain a predetermined line ?

Altho ugh Briffa‘s e -mail 41 stressing confidentiality does imply an awareness of questionable conduct, the e-mail correspondence with Wahl 42 stresses in several places Briffa‘s concern to be fair to sceptical views. We see no evidence in the correspondence of anything other than a detailed determination to resolve a scientific issue. Nor do the IPCC Review Editor‘s comments to us suggest that what would normally be regarded in the research community as conventional requests for advice and help, were ruled out.

39. We conclude, in line with the comments made by Professor Mitchell, it was not unreasonable to include the WA2007 paper alongside the M&M2003 paper in presenting an up- to -date picture of the relevant scientific arguments. We do not find that Briffa acted improperly in the part he played in this, and note that in any case he did not have sole responsibility for the outcome.



## 9.5 Conclusions

40. In summary, we have not found any direct evidence to support the allegation that members of CRU misused their position on IPPC to seek to prevent th e publication of opposing ideas.

41. In addition to taking evidence from them and checking the relevant minutes of the IPCC process, we have consulted the relevant IPCC Review Editors. Both Jones and Briffa were part of large groups of scientists taking joint responsibility for the

41 Briffa to Wahl email: 1140039406.txt; Feb 2006 "I am taking the liberty (confidentially) to send you a copy of the reviewers comments (please keep these to yourself) of the last IPCC draft chapter. I am concerned that I am not as objective as perhaps I should be and would appreciate your take on the comments from number 6-737 onwards, that relate to your reassessment of the Mann et al work. I have to consider whether the current text is fair or whether I should change things in the light of the sceptic

comments. "

42 Response to Additional Question regarding Keith Briffa‟s request to Eugene Wahl and his response (June 2010)CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC

85 relevant IPCC Working Group texts and were not in a position to determine individually the final wording and content. We find that neither Jones nor Briffa behaved improperly by preventing or seeking to prevent proper consideration of views which conflicted with their own through their roles in the IPCC.CHAPTER 10 : COMPLIANCE WITH FoIA/ENVIRONMENTAL INFORMATION REGULATIONS

86 

## CHAPTER 10: COMPLIANCE WITH

FoIA/ENVIRONMENTAL INFORMATION

REGULATIONS

10.1 Introduction and Method of Enquiry

1. This Chapter addresses the third part of our remit: “ Review the Climatic Research Unit‘s compliance or otherwise with the University of East Anglia‘s policies and practices regarding requests under the Freedom of Information Act 2000 1 ( ̳the

FoIA ‘) and the Environmental Information Regulations 2 ( ̳the EIR‘) for the release of data‖. It also reviews subject access requests under the Data Protection Act

1998

– as amended 3

. Whilst compliance with FoIA, EIR and DPA fall within the

more gen eral topic of  ̳Governance‘ (covered in the following Chapter), they are extracted and covered in detail here, given their very specific relevance to the

Review.

2. Interviews were carried out with the: Information Policy & Compliance Manager

– IPCM; Science Faculty FoIA/EIR Contact;
Director of Information Services; and Key staff within CRU.

3. Discussions were also held with representatives of the ICO, both to co-ordinate the work of the Independent Review with that of the concurrent ICO investigation

and to seek advice.

4. Notes of all these interviews are available on the Review website.



## 10.2 The Allegations

5. It is alleged by a number of correspondents (for example the submission by

Matthews 4 ) and commentators that requests under the FoIA and the EIR were incorrectly denied. Other correspondents (for example the submission by Mann 5 ) have suggested that a number of these FoIA requests were inappropriate or frivolous. Similarly it is alleged that subject access requests under the DPA for access to e-mails specifically referencing the applicant were not fully complied

with.



## 10.3 General Context

6. The Freedom of Information Act 2000 created new statutory rights of access, subject to a number of defined exemptions, to information held by a wide range of public bodies. It replaced the earlier  ̳Code of Practice on Access to Government

Information‘ tha t was a non-statutory scheme. The general right of access under

1 See: http://www.opsi.gov.uk/Acts/acts2000/ukpga_20000036_en_1

2 See: http://opsi.gov.uk/si/si2004/uksi_20043391_en.pdf

3 See: http://www.ico.gov.uk/what_we_cover/data_protection/legislation_in_full.aspx

4 Matthews submission (no. 16), 1 March

5 Mann submission (no. 42), 28 FebruaryCHAPTER 10 : COMPLIANCE WITH FoIA/ENVIRONMENTAL INFORMATION REGULATIONS

87 this Act came into force on 1 st January 2005. In cases where access to information is refused, the Act requires the public authority to give reasons for that refusal, including detailing any exemptions the public authority believes may apply. There is also a duty under the Act to offer reasonable advice and assistance to applicants seeking information.

7. The current Environmental Information Regulations (EIR) also came into force on

1 st January 2005. These contain a similar general right of access to that defined under the FoIA. Their genesis however dates back to initial regulations based on the European Community (EC) Council Directive 90/313 published in 1992, to which UEA would have been subject. It is not clear whether this was widely understood either by the University or by those seeking information.

8. Both the FoIA and EIR:

encourage the use of formal publication policies (―Schemes‖) and pro -active dissemination of information; have initial 20 working day time limits for the public authority to respond to
requests; require public authorities to provide advice and assistance to applicants and to provide the information in the form or format requested wherever reasonably possible. use the  ̳Public Interest‘ test. Information can only be withheld if: ― in all the circumstances of the case, the public interest in maintaining the exception outweighs the public interest in disclosing the information ‖; have defined appeal processes (albeit with some subtle differences on time limits); and

have common recourse to the Information Commissioner and to the

Information Tribunal.

9. Key differences 6 between the FoIA and EIR regimes are that, under the EIR: requests can be made verbally and do not have to be submitted in writing (or by e-mail) with a name and address;

there is a clearer definition of  ̳information held‘ than under the FoIA : ― If the information is in the authority's possession and has been produced or received by the authority, or if it is held by another person on behalf of the authority ‖; a request cannot be rejected purely on grounds of cost; the allowed initial response deadline can be extended to 40 working days for particularly complex requests; withholding of information under exemptio ns remains subject to the  ̳public interest‘ test; there is an emphasis on improved decision making and participation; and there is no direct equivalent to the FoIA  ̳vexatious requests‘ provisions.

There are no guarantees of absolute confidentiality under the EIR.

10. The EIRs only cover environmental information, albeit this is subject to a very

broad definition i (see endnote after paragraph 35). The FoIA covers all other

6 See: http://www.defra.gov.uk/corporate/policy/opengov/eir/slides-leaflets htmCHAPTER 10 : COMPLIANCE WITH FoIA/ENVIRONMENTAL INFORMATION REGULATIONS

88 information held by public authorities, but with a specific exemption for

environmental information.

11. The DPA 1998 (as amended) gives individuals the right to know what information is held about them. It provides a framework to ensure that personal information is handled properly. The Act states that anyone who processes personal information must comply with eight principles, which make sure that personal information is: Fairly and lawfully processed Processed for limited purposes Adequate, relevant and not excessive Accurate and up to date Not kept for longer than is necessary Processed in line with your rights

Secure Not transferred to other countries without adequate protection

12. More specifically in the context of this Chapter, the Act provides individuals with important rights, including the right to find out what personal information is held on computer and most paper records.

13. UEA‘s (and thus CRU‘s) formal processes are described in 3 documents: ―The Code of Practice for Responding to Requests for Information under the

Freedom of Information Act 2000‖ 7 , which is based on the Lord Chancellor‘s Code and sets out the manner in which UEA processes requests.

―Guidance for Staff‖ 8 , which directs how UEA staff should respond to requests for information. ―Requests for Information‖ 9 , which explains how to make a request for information, and how the UEA expects to respond.

14. UEA‘s FoIA/EIR processes are based on concentric circles. The IPCM is considered the central point. He: should always be the co-ordinator for FoIA/EIR requests; maintains the formal FoIA/EIR logs on requests received; establishes a case file for each request; makes the initial determination as to whether to treat a request under the FoIA or EIR regime; determines in cooperation with the Faculty contact and relevant staff whether the information sought is indeed ―held‖ i n statutory terms; and identifies and approaches the key provider via the appropriate faculty or

central unit contact.

15. The IPCM is the key person within UEA with a detailed understanding of the FoIA and the EIR. He maintains liaison with the ICO. He is also the key training provider to others within UEA.

7 See: http://www.uea.ac.uk/is/strategies/infregs/FOIA+Code+of+Practice+for+Responding+to+Requests

8 See: http://www.uea.ac.uk/is/foi/guidance

9 See: http://www.uea.ac.uk/is/foi/foi_requestsCHAPTER 10 : COMPLIANCE WITH FoIA/ENVIRONMENTAL INFORMATION REGULATIONS

89 16. The next circle comprises FoIA/EIR contacts at the Faculty or Central Unit level. There are FoIA/EIR contacts in place for each of the four University Faculties and for each of the central university units (such as Information Systems and the Registrar‘s Department ). These receive initial training and then yearly updates. It is understood that participation in these training sessions has been good. Beyond the initial training, it is understood that much learning is ―on the job‖.

17. The final circle comprises individual members of staff/researchers. An initial brochure was sent to all units in the University when the FoIA/EIR regimes came into force in 2005. Awareness training is made available on a voluntary basis.



## 10.4 Investigation

18. As stated in this Review‘s  ̳ Issues for Examination ‟ document

(Appendix 3), it is alleged that requests under the FoIA, EIR and DPA were incorrectly denied by UEA on advice from CRU. This is the subject of a separate inquiry by the ICO,

but does fall within the terms of reference of the Review. The Review has remained in regular contact with the ICO to ensure that both investigations could continue in parallel (see paragraph 19). The Review is particularly concerned to

address: What formal processes were in place both centrally and within CRU to ensure fair and impartial assessment of FoIA requests? Were there any processes in place centrally to review recommendations from

CRU that information should not be released?

19. The improperly released e-mails contain a number of references that raise specific concerns with respect to FoIA, EIR and the DPA. In particular, since CRU uses data obtained from other bodies, guidance was sought from the ICO on the extent to which a public authority that was not a primary repository of data might be expected to act as a secondary source of that data for the purposes of FoIA/EIR.

The advice received was that : Neither FoIA nor EIR make a distinction as to whether a public body is a primary or secondary source of information or data. The point is simply whether they hold the information for the purposes of the legislation, i. e. section 3(2) of FoIA and Regulation 3(2) EIR. If the public authority holds the information for any reason or in any form they must provide it or rely on a provision in the legislation to refuse the

request. If however a request is received and the information requested is publicly available, and reasonably accessible, the public body can rely on either section 21 FoIA or Regulation 6 EIR and point the requester to these sources. “Reasonably accessible” will normally mean the information is published or available on demand. There is no obligation for the public body to apply section 21 or Regulation 6: the public body is free to choose to supply the data.CHAPTER 10 : COMPLIANCE WITH FoIA/ENVIRONMENTAL INFORMATION REGULATIONS

90 20. The Review recognises three time periods into which information requests should be grouped: before 2005, where the FoIA was not in operation but a precursor regime to the current EIR was operating; from 2005 until the start of 2007, when responses to requests were being handled informally by CRU; and

after 25 th January 2007 when the first request was formally logged by the

IPCM.

21. The Review had access to the complete formal log of FoIA/EIR requests, compiled by the IPCM, with respect to CRU since both current regimes came into force at the start of 2005. CRU only started to treat such requests formally via the IPCM in 2007. It should be noted however that there are extensive indications, from e-mails now in the public domain, of requests for information and some release of information prior to 2007. An example is the e-mail from Jones on 21 st
February 2005 (1109021312.txt), ― PS I'm getting hassled by a couple of people to release the CRU station temperature data. Don't any of you three tell anybody that the UK has a Freedom of Information Act! ‖. Indeed, thre e e-mails from 2004 also made various information requests.

22. Formal FoIA/EIR requests were initially quite limited. In 2007 four requests were received, of which two were given full release of the requested information but two, despite appeals, were rejected. In 2008 two requests were received, one was granted full release, but the other was rejected, both initially and upon appeal. In the first half of 2009 only one request was received and this was responded

to in full.

23. But in the third quarter of 2009 a wave of requests was received. In the five days starting on 24 th July, some 60 requests were logged by the IPCM. A further 10 requests were logged between the 31 st July and 14 th August. Some related to the raw station data underpinning the CRUTEM data sets and the vast majority sought details of any confidentiality agreements related to this data. The wordings bear the hallmarks of an organised campaign. One applicant (UEA Log 09/97) appears to have forgotten to customise the request before dispatch. The text reads: ― I hereby make a EIR/FOI request in respect to any confidentiality agreements) restricting transmission of CRUTEM data to non-academics involving the following countries: [insert 5 or so countries that are different from ones already requested] the date of any applicable confidentiality agreements; the parties to such confidentiality agreement, including the full name of any organization; a copy of the section of the confidentiality agreement that "prevents further

transmission to non-academics". a copy of the entire confidentiality agreement.”

24. In the final quarter of 2009 a further wave of 41 requests was received, starting on

20 th November. These were mostly related to the unauthorised release of e-mails and seeking information in the context of that release. However a number raised other issues such as: the extent of UEA staff training in FoIA/EIR issues; sourcesCHAPTER 10 : COMPLIANCE WITH FoIA/ENVIRONMENTAL INFORMATION REGULATIONS

91 of funding; and statistics on FoIA/EIR requests received. Still others harked back to previous themes of access to the CRUTEM data set and related confidentiality
agreements.



## 10.5 Findings

25. Education and culture. Whilst we did identify evidence that UEA had widely distributed initial guidance at the introduction of the FoIA/EIR regimes in 2005, we also found a lack of engagement by core CRU team in understanding EIR/FoIA legal requirements and how these might legitimately impact them. There was evident confusion within the CRU as to how these requirements might be applied for example to data, code, and personal correspondence. We found a tendency to assume that no action was required until precedents had been set. As an example, on 21 st January 2005 Jones wrote (1106338806.txt) : ― On the FOI Act there is a little leaflet we have all been sent. It doesn't really clarify what we might have to do re programs or data. Like all things in Britain we will only find out when the first person or organization asks. I wouldn't tell anybody about the FOI Act in Britain. I don't think UEA really knows what's involved. ‖ There was insufficient priority given from the UEA centre to motivating staff and to prompting continuing education. Various requests received by the CRU between 2005 and the start of 2007 had not been formally logged with the IPCM.

26. Foresight lacking . We found a lack of recognition by both CRU, and the University‘s senior management of the extent to which earlier action to release information or give full guidance to locate primary sources and to provide station identifiers might have minimized the problems. There are many references in the e-mails now in the public domain to ―hiding information‖, ―finding ways around releasing‖, or finding excuses not to release information. There was a fairly swift shift towards a lack of sympathy with the requesters, as seen in an e-mail from

Jones sent on 7 th May 2005 (1083962092.txt) : ― Mike and I are not sending anything, partly because we don't have some of the series he wants, also partly as we've got the data through contacts like you, but mostly because he'll distort and

misuse them

. ‖

We do not suggest that the allegations made against McIntyre are correct.

27. Unhelpful responses . We found a tendency to answer the wrong question or to give a partial answer. For example the very first formal FoIA request (UEA Log

07 -04) asked for the list of stations used in the preparation of HadCRUT3 and the raw data for these stations. The initial response refers simply to the availability of

the raw data from other sources and not to the station list. The requester then enters into an extended correspondence trying to extract the station identifiers. An extract from 14th April 2007 is given below:

“While it is good to know that the data is available at those two web sites, that information is useless without a list of stations used by Jones et al. to prepare the HadCRUT3 dataset. As I said in my request, I am asking for:

1) A list of the actual sites used by Dr. Jones in the preparation of the HadCRUT3 dataset, andCHAPTER 10 : COMPLIANCE WITH FoIA/ENVIRONMENTAL INFORMATION REGULATIONS

92 2) A clear indication of where the data for each site is available. This is quite important, as there are significant differences between the versions of each site's data at e.g. GHCN and NCAR."

Without knowing the name and WMO number of each site and the location of the source data (NCAR, GHCN, or National Met Service), it is not possible to access the information. Thus, Exemption 21 does not apply - I still cannot

access the data.

I don't understand why this is so hard. All I am asking for is a simple list of the sites and where each site's data is located. Pointing at two huge piles of data and saying, in effect, "The data is in there somewhere" does not help at all.

To clarify what I am requesting, I am only asking for a list of the stations used in HadCRUT3, a list that would look like this:

WMO# Name Source

58457 HangZhou NCAR

58659 WenZhou NCAR

59316 ShanTou GHCN

57516 ChongQing NMS

etc. for all of the stations used to prepare the HadCRUT3 temperature data.

That is the information requested, and it is not available "on non-UEA websites", or anywhere else that I have been able to find.

I appreciate all of your assistance in this matter, and I trust we can get it resolved satisfactorily.”

These station identifiers are finally promised on appeal, but not provided until some six months later following further prompting.

28. Deliberate actions to avoid release . There seems clear incitement to delete emails, although we have seen no evidence of any attempt to delete information in respect of a request already made. Two e-mails from Jones to Mann on 2 nd
February 2005 (1107454306.txt) and 29 th May 2008 (in 1212063122.txt) relate to

deletion :

2 nd February 2005: ― The two MMs have been after the CRU station data for years. If they ever hear there is a Freedom of Information Act now in the UK, I think I'll delete the file rather than send to anyone”

.

29 th May 2008: ― Can you delete any emails you may have had with Keith re AR4? Keith will do likewise. He's not in at the moment - minor family crisis. Can you also email Gene and get him to do the same? I don't have his new email address. We will be getting Caspar to do likewise ”

.CHAPTER 10 : COMPLIANCE WITH FoIA/ENVIRONMENTAL INFORMATION REGULATIONS

93 There is a clear statement that e-mails had been deleted

– for example, an e-mail

from Jones to Santer sent on 3 rd December 2008 (1228330629.txt ): ― About 2 months ago I deleted loads of emails, so have very little - if anything at all .‖

It seems likely that many of these  ̳deleted‘ e -mails subsequently became public following the unauthorised release from the backup server. There is evidence that the IPCM did try to warn Prof. Jones about deliberate deletion of information; for example, an email from Jones to Santer (1228922050.txt) 10 th December 2008

states : ― I did get an email from the FOI person here early yesterday to tell me I shouldn't be deleting emails - un less this was 'normal' deleting to keep emails manageable!”

.

29. Imbalance of authority . The current structure for handling FoIA/EIR requests within UEA focuses very much on the IPCM. At interview he

indicated that he felt: “ Very much the bull’s eye at the centre of the target ”. We believe that the UEA senior staff need to take more explicit responsibility for these processes, thus enhancing the authority and standing of the IPCM. We found that the IPCM may have lacked such standing within the University

s tructure and the authority to challenge the assertions of senior professors. In this context, the IPCM and the Faculty FoIA contact may not have been empowered to be sufficiently rigorous: Jones to Santer sent on 3 rd December 2008 (1228330629.txt): When the FOI requests began here, the FOI person said we had to abide by the requests. It took a couple of half hour sessions - one at a screen, to convince them otherwise showing them what CA was all about. Once they became aware of the types of people we were dealing with, everyone at UEA (in the registry and in the Environmental Sciences school - the head of school and a few others) became very supportive. ” However at interview the IPCM explicitly denied that he behaved in the way suggested in this e-mail.

30. Lack of constructive challenge . Whilst we found that efficient basic control, logging and progress chasing processes for FoIA/EIR requests were in place, we found a lack of independent working-level challenge in these systems. We found that the appeals mechanisms in place lacked the resources for effective challenge to basic assumptions. Similarly, the escalation processes failed to react sufficiently quickly to the dramatic change in the volume and character of requests and to provide timely high-level review and resources.

31. Limited internal communication . We found a lack of understanding within University central functions of the presence of extensive, and long duration, backups of e-mail and other materials despite these being on a server housed within the central Information Technology (IT) facilities. Awareness of these might have led to much greater challenge of assertions regarding non-availability of material by CRU, notably in the case of a subject access request made under the DPA for material naming the requesting individual.

32. The Review found an ethos of minimal compliance (and at times non-compliance) by the CRU with both the letter and the spirit of the FoIA and EIR. We believe that this must change and that leadership is required from the Un iversity‘s most senior staff in driving through a positive transformation of attitudes. Public trust in science depends on an inherent culture of honesty, rigour and transparency.CHAPTER 10 : COMPLIANCE WITH FoIA/ENVIRONMENTAL INFORMATION REGULATIONS

94 The requirements of FoIA and EIR must not be seen as impositions. They are a necessary part of the implicit contract between the scientist and broader society. Such an open culture will also lead to the best science.



## 10.6 Recommendations

33. The Review offers the following recommendations for action within the UEA: Change fundamentally the perception that responsibility for FoIA/EIR compliance lies with administrative staff . University senior staff need to make clear their commitment to a culture of honesty, rigour and transparency, plus the supporting processes and resources. Review the resourcing and standing of the FoIA/EIR/DPA compliance and request handling processes . Our findings have highlighted significant problems in the areas of: imbalance of authority; lack of effective challenge at appeal; over dependence on single individuals; inadequate escalation processes and limited strategic oversight. A concerted and sustained campaign to win hearts and minds . This should include: promotion of the University‘s formal publication policy; incorporating more information on Fo IA/EIR/DPA responsibilities in the induction processes for new staff members; developing a rolling awareness campaign to focus the attention of established staff, particularly in the context of the changing landscape e.g. Queens University judgment (see paragraph

3 4 ); and issuing annual reminders of the importance of transparency and of key FoIA/EIR/DPA responsibilities; Once the improved awareness measures and processes are in place, to run a programme of independent, external, tests with requests for information to verify the continuing effectiveness of these operations . This is a special case of the more general recommendation on  ̳Audit processes‘ given in the Governance Chapter.

34. The Review offers the following more general recommendations:

Definition of research data . There is extensive confusion and unease within the academic community as to exactly how FoIA/EIR should be applied in terms of the materials developed during a research process. The Review believes that all data, metadata and codes necessary to allow independent replication of results should be provident concurrent with peer-reviewed publication. However the situation regarding supporting materials such as early drafts, correspondence with research colleagues and working documents is widely regarded as unclear. The American experience is instructive here. The so called ―Shelby Amendment‖ in 1998 directed the US ―Office of Management & Budget (OMB)‖ to produce new standards requiring all data produced under Federally funded research to be made available under the US Freedom of Information Act. This resulted in great concern within the US Scientific community, expressed through Congressional testimony, that a very broad interpretation of this requirement could seriously impair scientific research and collaboration. In the final OMB guidelines 10 , recognising these concerns,

10 Federal Register: March 16, 2000 Volume 65, Number 52 Page 14406. http://frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=2000_register&docid=00-5674-filedCHAPTER 10 : COMPLIANCE WITH FoIA/ENVIRONMENTAL INFORMATION REGULATIONS

95 ―research data‖ is defined as: ― the recorded factual material commonly accepted in the scientific community as necessary to validate research findings, but not any of the following: preliminary analyses, drafts of scientific papers, plans for future research, peer reviews, or communications with colleagues ‖. The Review recommends that the ICO should hold consultations

on a similar distinction for the UK FoIA/EIR.

Orchestrated campaigns . As detailed in paragraph 23 , CRU was the subject of an orchestrated campaign of FoIA/EIR requests in late July and early August

2009. The Review believes that CRU helped create the conditions for this campaign by being unhelpful in its earlier responses to individual requests for station identifiers and the locations from which specific, detailed station raw

data could be downloaded. Similarly a clearer publication policy, reflecting the wishes of both the University and the research funders might have avoided these challenges. The Review team can however conceive of situations where such orchestrated campaigns might recur, with literally overwhelming impacts on small research units. We urge the ICO to give guidance on how best to respond to such organised campaigns, consistent with the underlying principles of openness.

Greater clarity in an evolving landscape

. Particularly in the light of the recent Queens University Belfast determination by the ICO in respect of the release of Irish Tree Ring data 11 , it would be helpful if the ICO could re-engage more generally with the Higher Education sector about their understanding of FoIA and EIR obligations and also consider what further guidance could be provided for that sector. It would be particularly useful if guidance were available as to how long it is reasonable to retain data without release, pending full publication as part of a peer reviewed paper. It is however recognised that

often such determinations have to be made on a case- by -case basis against a ―public interest‖ test.

35. As a final comment we find that a fundamental lack of engagement by the CRU team with their obligations under FoIA/EIR, both prior to 2005 and subsequently, led to an overly defensive approach that set the stage for the subsequent mass of FoIA/EIR requests in July and August 2009. We recognise that there was deep suspicion within CRU, as to the motives of those making detailed requests. Nonetheless, the requirements of the legislation for release of information are clear and early action would likely have prevented much subsequent grief.

11 ICO Case Ref: FS50163282

Date: 29/03/2010

Publ ic Authority: Queen‘s University Belfast Summary: The complainant requested electronic data relating to tree ring research (dendrochronology). The public authority confirmed that it held the requested information but refused to provide it citing section 12 of the Act. The Commissioner indicated to the public authority that the withheld information fell within the definition of environmental information under the EIR. The public authority subsequently cited the exceptions at regulations 12(4)(d), 12(4)(b), 12(5)(c) and 12(5)(e) to refuse the

13 cont information. The Commissioner finds that none of the exceptions is engaged and the withheld information should therefore be disclosed. The Commissioner also recorded a number of procedural breaches in the public authority‘s handling of the request.CHAPTER 10 : COMPLIANCE WITH FoIA/ENVIRONMENTAL INFORMATION REGULATIONS

96 i EIR definition of environmental information (see paragraph 10) :

Any information in written, visual, aural, electronic or any other material form on:

(a) the state of the elements of the environment, such as air and atmosphere, water, soil, land, landscape and natural sites including wetlands, coastal and marine areas, biological diversity and its components, including genetically modified organisms, and the interaction among these elements; (b) factors, such as substances, energy, noise, radiation or waste, including radioactive waste, emissions, discharges and other releases into the environment, affecting or likely to affect the elements of the environment referred to in (a); (c) measures (including administrative measures), such as policies, legislation, plans, programmes, environmental agreements, and activities affecting or likely to affect the elements and factors referred to in (a) and (b) as well as measures or activities designed to protect those elements;(d) reports on the implementation of environmental legislation; (d) reports on the implementation of environmental legislation; (e) cost-benefit and other economic analyses and assumptions used within the framework of the measures and activities referred to in (c); (f) the state of human health and safety, including the contamination of the food chain, where relevant, conditions of human life, cultural sites and built structures inasmuch as they are or may be affected by the state of the elements of the environment referred to in (a) or, through those elements, by any of the matters referred to in (b) and (c).CHAPTER 10 : COMPLIANCE WITH FoIA/ENVIRONMENTAL INFORMATION REGULATIONS

97CHAPTER 11 : GOVERNANCE

98 

## CHAPTER 11: GOVERNANCE

11.1 In troduction and Method of Enquiry

1. This C hapter addresses the fourth part of the Review‘s remit, ―Review and make recommendations as to the appropriate management, governance and security measures for CRU and the security, integrity and release of data it holds‖, to the extent that these issues are not addressed in previous Chapters. The previous Chapter deals extensively with the major governance issue of FoIA and EIR compliance. This Chapter considers other relevant aspects of the governance framework, first as it relates to Research Management Systems in Section 11.2 and then secondly to the specific issue of the Management of Software, Data Management and Security in Section 11.3. The Chapter concludes with a set of recommendations in Section 11.4 on governance matters relating to both issues.

2. The Review Team decided to focus on the efficacy of current control systems in UEA relevant to CRU‘s research, while developing a broad understanding of the evolution of UEA‘s policies and practices.

3. Interviews were carried out with the:

Vice-Chancellor PVC, Research Enterprise and Engagement Registrar and Secretary Associate Dean for Research, Faculty of Science
Director of Information Services; Information Communications Technology (ICT ) Systems Director; and CRU (part-time) IT Manager Director of Research, Enterprise and Engagement and Manager of Research

Services Research Finance Management Accountant and Faculty of Science Finance Manager

Director of Human Resources

4. Notes of the relevant meetings and supporting documentation are accessible on

the website.



## 11.2 Research Management Systems

11.2.1 Background

5. CRU is located within the School of Environmental Sciences (ENV) of UEA. Historically CRU had a high degree of autonomy, with separate reporting lines and distinct budgets. Apart from the Director, its staff were dependent on ―soft‖ money, that is on grants and other sources of funding mainly from sources such as UK Research Councils, EU programmes and the US Department of Energy. The position changed with the University‘s decision to invest strongly in environmental science. More recently up to 5 of the CRU staff have heldCHAPTER 11 : GOVERNANCE

99 established positions and the current full-time equivalent is 3.5.

6. The research policies and strategy of the University are set by a hierarchy of committees, with CRU being represented on the ENV Committee. The Director of CRU reports to the Head of ENV.



## 11.2.2 Funding Management

7. Responsibility for the University‘s funding management process rests with Research, Enterprise and Engagement (REE), through which major bids for funding are channelled. More routine funding bids are handled by Faculty Research Officers. The documentation supporting bids requires sign-off by the relevant principal investigator (PI) and Head of School, confirming their collective and individual responsibilities in delivering the research project in

accorda nce with the funder‘s terms and conditions and the University‘s and School‘s policies and procedures.



## 11.2.3 Funders‟ R equirements

8. Funders‘ requirements vary considerably and have developed over time. All seek publication of results from the work they support. The traditional expectation has been that this will be in peer reviewed journals. There is now some pressure for open access publication, which also involves peer review. The emphasis on protecting intellectual property varies. Regular progress reports are generally required, often as a condition of drawing down funds; and reports with specific formats and deadlines may be required on completion. There are rarely detailed requirements for the release of data or code.



## 11.2.4 Good Research Practice

9. The university has a well developed set of policies on good research practice, research ethics, misconduct in research (including whistle blowing and grievance procedures), which apply to all those working in research, as well as regular performance reviews. The system is continuously developing; the good practice guidelines were first introduced in 2003 (we saw the 2009 version). Details of these structures, policies and procedures are referenced with the records of our interviews with the relevant senior managers, on our website.

10. Prospective students and staff are sent weblinks before arrival. There is training for undergraduates carrying out research as part of their coursework, and for postgraduate students and for new members of staff there is probationary training. It is the responsibility of Principal Investigators to ensure that their people are fully trained

11. The University‘s statement of terms and conditions of appointment of academic, teaching and research staff cross-refers to the requirements of a wide range of university policies. There is an induction process that includes coverage of what the requirements are and includes the allocation of a ―mentor‖, to act as an advisory colleague to a new member of staff. There is a standard annual training programme for both supervisors and new recruits. Compliance with theCHAPTER 11 : GOVERNANCE

100 requirements under the terms and conditions is the responsibility of Heads of School, with support from Human Resources as necessary.

12. Outputs such as publication and citations are monitored on a yearly basis, research plans are prepared at institutional level and have to be signed off each year, and individuals receive appraisal with feedback every 2 years. UEA attaches importance to peer review as a check on people‘s work and publications, and encourages academics to seek outside involvement.

13. Whistle blowing has never been used for any research related issue (it has been used a couple of times in all, in relation to other matters). The research misconduct procedure has been used on only a few occasions, mainly in relation

to individuals‘ doctorate or master‘s theses.



## 11 .2.5 Financial Controls

14. Lead universities are normally responsible for administration of grants; payments are controlled by funders against a profile of expenditure and pass through central accounts only.

15. Where cash advances are required, there are specific, senior approval procedures.

I n exceptional circumstances advances could be made to an employee‘s bank

account.

16. It was reported by UEA Finance that audit procedures are robust, both in general and in relation to certain EU projects where these are mandatory. UK Research Councils are increasingly asking for specific audit checks.



## 11.2.6 Risk Management

17. The University maintains a risk register, regularly reviewed by Senior Management and annually by the Council and the Audit Committee. In relation to Research, the main concern has been the risk to reputation from slippage in league tables, with a range of mechanisms to drive performance in the desired direction. The register did not address the pressures on CRU as a result of the significance of its work for the climate change debate and the severity of the challenge to that work, and therefore did not consider the support CRU might need in handling its data and in making it available transparently. The upsurge of FoIA requests in 2009 was not brought to the attention of senior management soon enough; and, when it was, there was no prepared framework for addressing the implications. Action is now being taken to raise awareness of the implications of FoIA and EIR, with leadership from senior management.



## 11.2.7 Findings on Research Management Systems

18. The Review t eam‘s reading of the improperly rel eased e-mails together with Sherrington 1 suggested that cash advances to researchers in Russia might not have been adequately controlled, see also Chapter 4, paragraph 13 . Whereas nearly all

records from the mid- 1990s have now been destroyed, in keeping with UEA‘s policy, we have confirmed that CRU did not have its own bank account and any

1 Sherrington submission (no. 107), 4 MarchCHAPTER 11 : GOVERNANCE

101 such payment would have required authorisation in writing by the Director of Finance at that time. While we have been unable to deal exhaustively with the issue, we are advised that there is a payment of $5000 in November 1998 properly recorded in the UEA ledger which is consistent with e-mails 911405082.txt and

914256033.txt at that time.

19. Apart from those issues addressed in Chapters 6 to 10 , UEA‘s Research Management Systems appear adequate. Requirements have developed significantly over the period relevant to the Review, from fairly general expectations that terms and conditions set by funders would be complied with, to the highly developed system now in operation.

20. To summarise, the focus of risk management in research was on quality and standards. The critical pressure on CRU, with its attendant risk to the University‘s reputation, was not on the radar screen. Senior management were not sufficiently aware of the issue, were not alerted in time to its emergence, and had no preparation in place to respond adequately.



## 11.3 Software, Data Management and Data Security

11.3.1 General Context

21. In submissions, concerns were expressed to the Review over the quality of software development, testing and operational processes within CRU, though it was also asserted that much of such software is transient, and, though poorly structured and documented, may still be fit for the limited purposes envisaged. Submissions from Bratby 2 and Barnes 3 are relevant. We noted that a number of requests received by UEA under FoIA/EIR also sought access to code. Concerns have also been expressed about potential loss of data by CRU and the level of information security for CRU systems, which were vulnerable to unauthorised disclosure of the personal e-mails and other material subsequently published widely on the Internet.

22. IT Organisation . In common with other areas of the Science Faculty, CRU operates largely independently of the central IS functions of the UEA. Central IS has, in recent years, made significant efforts to better support the Science Faculty and some use of central facilities (such as the Storage Area Network) has been achieved. The University IS team does not provide desktop, remote access, hosting, database or software support to CRU, nor any quality control or

assessment. CRU has its own local architecture based on a mix of individual PC based and server based processing. In common with many other research groups across the university, this is distinct from the UEA preferred model of client

– server operation. Internet communications for CRU is however routed over the university network and through the university firewall. CRU has its own IT Manager for whom CRU is 40% of his workload. CRU originally had no central backup arrangements for the individual researchers‘ PCs however its IT Manager introduced automated backup (using open source software) to a simple server held securely within the Central IS machine room.

2 Bratby submission (no. 3), 22 February

3 Barnes submission (no. 1), 6 MarchCHAPTER 11 : GOVERNANCE

102 23. Policy

. A high level  ̳Information Systems Policy‘ and a related  ̳Information Security Policy‘ 4 were agreed and put in place in 2005 under the aegis of UEA‘s Information Systems Strategy Committee (ISSC), which includes representatives of all four Faculties. Low level, detailed, security policies had been developed and put in place by 2007 5

.

24. Standards

. The Review is aware of a number of industry standards relevant to this Chapter dealing with: software development, data stewardship and security. For further details see Appendix 8

. 

## 11.3.2 Issues and the Investigation

25. The Director of Information Services indicated that, whilst the central IT function were aware of the existence of the CRU Backup Server, they had no knowledge of the nature of the information held on the server as it was managed from CRU.

26. The CRU IT Manager indicated that researchers within CRU worked individually or in small groups. There was no master index of resources, be these data, algorithms or software. No systematic approach to the creation of metadata existed. There was no central database of underlying climate data; rather individual researchers assembled sets of data drawn from different primary sources outside CRU (for example the Met. Office Hadley Centre for Climate Change). These might arrive by network ( The United Kingdom‘s Educational and Research Network, or JANET), or on portable hard disks.

27. With the exception of a small amount of tree ring data, CRU does not generate new raw data. It relies instead on accessing existing primary data sources. Nonetheless the importance of maintaining clear records of what data has been accessed for what purpose

– for example in terms of station records processed

– is clear. The benefits of such a central data catalogue (or ―data dictionary‖) had long been recognised within CRU and past attempts had been made to create such a resource. These attempts had foundered on the lack of resources

– research grants made no provision for this and central UEA funding had not been available. Whilst there was no policy for the systematic archiving of data, many of CRU‘s processed results datasets are available on the Web 6

.

28. Individual researchers were responsible for acquiring or developing their own software applications (usually written in Fortran or Interactive Data Language, IDL). There was no formal quality control policy or review policy. However individual projects would comply with whatever quality control processes were specified as part of their funding arrangements.

29. At interview, the ICT Systems Director indicated that ―lessons had been learnt‖ and he expected (subject to the results of a security audit report) to bring forward proposals within the University for:

4 These can be downloaded from: http://www.uea.ac.uk/is/itregs/ictpolicies

5 A draft ―Security Manual‖ (not available for public download) was received by the Review on 8th February.

6 See: http://www.cru.uea.ac.uk/cru/data/hrg/CHAPTER 11 : GOVERNANCE

103 Greater compliance with centrally defined IS policies and architecture; An audit of research data held in digital storage across the University; and Clear data retention (and destruction) policies.



## 11.3.3 Findings on Software, Data Management and Data Security

30. The software development process . We found that, in common with many other small units across a range of universities and disciplines, CRU saw software development as a necessary part of a researcher‘s role, but not resourced in any professional sense. Small pieces of software were written as required, with whatever level of skill the specific researcher happened to possess. No formal standards were in place for:

Software specification and implementation;
Code reviews; and Software testing

31. Data management and archiving

. We found that there were no formal processes in place within CRU with respect to the systematic retention and archiving of data sets, or more particularly of metadata (data that allows the data set to be correctly interpreted). Individual researchers took whatever actions they deemed appropriate within the context of specific agreements with research funding

bodies.

32. Information security. We found that the basic security processes had been appropriately specified and documented by the UEA‘s Information Systems Strategy Committee. We are constrained in our detailed findings by the fact that a police investigation into the unauthorised release of information is ongoing.



## 11.4 Recommendations

33. Risk management

. UEA should be alert to the implications for their reputation of the sort of challenges we have seen in this case to the work of CRU and any other key groups. The risk register should reflect the range of external attitudes towards its key units and growing criticism or the attentions of pressure groups should be noted. Mitigation measures should be put in place, including increased security and a bias for openness and properly resourced policy on data management and availability. Reporting arrangements should ensure that key senior management are in touch with the issues and are informed quickly of problems; and response plans should be in place and rehearsed. These points are no doubt relevant to many other universities.

34. Training for researchers. We believe that Universities should develop formal approaches to the training of researchers in basic software development methodologies and best practice, as well as best practice in the handling and sharing of research data.

35. Provision of a formal metadata repository. Whilst we recognize and accept that CRU relies on other bodies both nationally and internationally to provide and toCHAPTER 11 : GOVERNANCE

104 archive basic weather station data, we believe that a formal approach to the storage and archiving of metadata is required. Such a repository would, for example, have made it far easier to respond quickly to requests for the list of station identifiers associated with particular CRUTEM datasets. Where a University is hosting a unit of such international significance, we believe that it should ensure funding is available for such a repository either through the research grant process or from central resources.

36. Role of research sponsors. We note the recent statement by the US National Science Foundation (NSF) 7 that, from October 2010, NSF plan to make inclusion of a ―Data Management Plan‖ a requirement for all research proposals. It will be important for such plans to recognize that in some areas of science huge volumes of data are created and a degree of processing and compression is inevitable before data suitable for storage is created. We agree that the way in which important research data (and the associated meta data to make that data useful) should be preserved, should be specified by those funding such research. Explicit budgetary and resource provision must be made. Sponsorship arrangements should include a clear statement of requirements on the extent to which such data should be placed in the public domain and any constraints on the timing of such release. The guidance from the UK Research Integrity Office (UKRIO) is helpful in this respect. 8

37. Making source code publicly available. We believe that, at the point of publication, enough information should be available to reconstruct the process of analysis. This may be a full description of algorithms and/or software programs where appropriate. We note the action of NASA‘s Goddard Institute for Space Science in making the source code used to generate the GISTEMP gridded dataset publically available. We also note the recommendation of the US National Academy of Sciences in its report ―Ensuring the Integrity, Accessibility and Stewardship of Research Data in the Digital Age‖ that: “...the default assumption should be that research data, methods (including the techniques, procedures and tools that have been used to collect, generate or analyze data, such as models, computer code and input data) and other information integral to a publically reported result will be publically accessible when results are reported ...‖. We commend this approach to CRU.

38. Audit processes. It is entirely acceptable that the central functions of a University should set, document and disseminate the standards expected across all governance areas, but without necessarily mandating the precise means by which these will be achieved. These standards will reflect the University‘s interpretation of applicable law (Data Protection, Computer Misuse, Health & Safety, Environmental Information Regulations) and best practice. In areas such as Information Systems, it may well be appropriate to allow a degree of local autonomy. However it is then essential that robust audit procedures are in place to ensure that, where local solutions are implemented, these do meet fully the standards specified.

7 NSF Press Release 10- 077 http://www.nsf.gov/news/news_summ.jsp?cntn_id=116928&org=NSF&from=news

8 See UKRIO  ̳Code of Practice for Research‘ Section 3.12 ―Collection and Retention of Data‖ at http://www.ukrio.orgCHAPTER 11 : GOVERNANCE

105106



## ACKNOWLEDGEMENTS

The Review Team would like to acknowledge the help and support they received during the conduct of the Review and in the preparation of this report. Our thanks go

to:

William Hardie who provided Secretariat support.

David Walker who acted as our Project Manager.

Kate Moffat, Adrian Dias

, Helen Davison and Mike Granatt who provided communications support and advice, set up and handled the Review website, and managed the production of the report.107APPENDIX 1

108 

## APPENDIX 1: REVIEW TEAM MEMBERS

TEAM CV‘s



## Sir Muir Russell, KCB, FRSE (Chair)

Sir Muir Russell was Principal and Vice-Chancellor of the University of Glasgow from 2003 to 2009. During that period he was Convener of Universities Scotland, a member of the Universities UK Main Board, a Trustee of the Universities Superannuation Scheme, and a member of the UCAS Board. He graduated from the University of Glasgow in 1970 with a First in Natural Philosophy and took up a career in the civil service. He was appointed Permanent Secretary to the Scottish Office in 1998, and was the first Permanent Secretary to the Scottish Executive following the establishment of the Scottish Parliament in 1999. He has honorary degrees from the Universities of Strathclyde, Glasgow and Edinburgh. He currently chairs the Judicial Appointments Board for Scotland. He is also a Trustee of the Glasgow School of Art, a Member of the Board of the Moredun Research Institute, and the Chairman of the

Dunedin Concert Trust.



## Professor Geoffrey Boulton, OBE, FRS, FRSE

Professor Geoffrey Boulton is Regius Professor Emeritus of Geology and former Vice Principal of the University of Edinburgh. His research is in the fields of glaciology, glacial geology and Quaternary science, and has been awarded several international awards and honorary degrees for his scientific work. He currently has research projects in Antarctica and Iceland. He has been the UK representative to the International Union of Geosciences and to the International Union of Quaternary
Sciences. He is a member of the UK Prime Minister‘s Council for Science and Technology, chairs the Advisory Board of the University of Heidelberg and is General Secretary of the Royal Society of Edinburgh, Scotland‘s national academy.

He has been a member of the Councils of the Natural Environment Research Council and the Royal Society, a member of the Royal Commission on Environmental Pollution, the Scottish Science Advisory Committee and the Scottish Higher Education Funding Council. He was formerly Head of the Department of Geology and Geophysics and Provost of Science and Engineering in the University of Edinburgh.



## Professor Peter Clarke, FInstP, CPhys, FIET, CEng

Peter Clarke is Professor of Physics at the University of Edinburgh. He has a B.Sc in Electronics Engineering (Southampton University, 1980) and a D.Phil in Particle Physics (Oxford 1985). He was a CERN Fellow before being appointed as a lecturer first at Brunel University in 1987 and then University College London in 1993. He was promoted to Reader and then Professor in 2001 and was Head of the Particle Physics Research Group between 2001-04. He moved to the University of Edinburgh in 2004 to take up the Chair of eScience and later become Director of the National

eScience Centre 2006-09.



## David Eyton MA, MIoM 3 

## , CEng.

David Eyton is Group Head of Research & Technology at BP, and was appointed in April 2008. He is accountable for technology strategy and its implementation across BP and conducting research and development (R&D) in areas of corporate renewal.APPENDIX 1

109 In this role, David oversees the R&D capability of the company and also sits on the UK Energy Technologies Institute and Science¦Business Innovation Boards. During his career he has held a number of Petroleum Engineering, Commercial and Business Management positions in the UK, Australia, Trinidad and USA.



## Professor James Norton, CDir, CEng, CITP, FIoD, FIET, FBCS, FRSA

Aged fifty-seven, Jim Norton is an independent director and policy adviser. He is an

external member of the Board of the UK Parliament's Office of Science & Technology (POST) and council member of the Parliamentary IT Committee (PITCOM). Jim is a Non-Executive Director of F&C Capital & Income Investment Trust plc, where he chairs the Audit & Management Engagement Committee. He is a Board Member and Trustee of the Foundation for Information Policy Research (FIPR), as well as a member of the 'Electronic Communications Expert Advisory Panel' for the Irish Commission for Communications Regulation (ComReg). Jim is a

Vice-President and Trustee of the BCS

– Chartered Institute for IT, an External Examiner for the Institute of Directors, and Chair of the IT Policy Panel for the Institution of Engineering & Technology. He also chairs the Steering Group for the Secure Software Development Partnership (SSDP) of the Technology Strategy Board.APPENDIX 2

110 

## APPENDIX 2 : APPROACH AND WORK PLAN

1. This document sets out how the Review Team has approached its terms of reference, and the way in which it will fulfil its remit.

2. The terms of reference are:

―The Independent Review will investigate the key allegations that arose from

a series of hacked e- mails from the University of East Anglia‘s Climatic Research Unit (CRU). The review will:

2.1. Examine the hacked e-mail exchanges, other relevant e-mail exchanges and any other information held at CRU to determine whether there is any evidence of the manipulation or suppression of data which is at odds with acceptable scientific practice and may therefore call into question any of the

research outcomes.

2.2. Review CRU‘s policies and practices for acquiring, assembling, subjecting to peer review and disseminating data and research findings, and their compliance or otherwise with best scientific practice.

2.3. Review CRU‘s compliance or otherwise with the University‘s policies and practices regarding requests under the Freedom of Information Act ( ̳the

FoIA ‘) and the Environmental Information Regulations ( ̳the EIR‘) for the

release of data.

2.4. Review and make recommendations as to the appropriate management, governance and security structures for CRU and the security, integrity and

release of the data it holds.‖

3. The remit requires the Review to address the specific allegations about the way in which CRU has handled its data, reflecting comments in the e-mail exchanges that have been made public. In a separate paper

– Issues for examination - the Team has set out its initial view of the questions that need to be addressed. It will seek written submissions from CRU and other appropriate parts of UEA. It will also invite interested parties to comment on what the Issues paper covers, and to propose any further matters that clearly fall within the Remit and should also be

examined.

4. The Review‘s remit does not invite it to re-appraise the scientific work of CRU. That re-appraisal is being separately commissioned by UEA, with the assistance of the Royal Society. The Review‘s conclusions will complemen t that re-appraisal by pointing to any steps that need to be taken in relation to data, its availability and its handling.

5. The Team wishes to focus on the honesty, rigour and openness with which CRU handled its data. It wishes to gain a proper understanding of: The range of data involved, and how it has been indexed and archived.APPENDIX 2

111 The procedures, processes and relevant protocols used to handle the data, recognizing that these may have changed over time as data-handling capacity has developed.

The associated metadata, algorithms and codes used for analysis. The extent to which other independent analysis produces similar

conclusions. The peer review process, examining how much was in common between

the work of the reviewers and the reviewed.

6. In making its analysis and conclusions, the Team will test the relevant work against pertinent standards at the time it was done, recognizing that such standards will have changed. It will also test them against current best practice, particularly statements of the ethics and norms such as those produced by the UK Government Office for Science and by the US National Academy of Sciences. These identify principles relating to rigour, respect and responsibility in scientific ethics and to integrity, accessibility and stewardship in relation to research data. This overall approach will allow the Team to establish a conceptual framework within which it can make judgements and comment about key issues such as the level of uncertainty inherent in all science, and the particular confidence limits associated

with the CRU work.

7. The police and the Information Commissioner are also considering issues in connection with the leaked e-mails, and the Team has established appropriate, continuing liaison with them.

8. The Team‘s an alysis and conclusions will include not only a view of what has happened in the past, but also comments and recommendations on best practice for the future. This will be done both at the level of CRU and the University as a whole, and may have broader implications for institutions undertaking scientific

work.

9. The Team will operate as openly and transparently as possible. It is establishing a website which will eventually display all of the submissions received, correspondence, analyses and conclusions. The aim will be to publish all received submissions quickly, unless there are wholly exceptional reasons to delay, for example legal issues.APPENDIX 3

112 

## APPENDIX 3

The questions, criticisms and allegations raised in the „Issues for Examination‟ paper below are not intended to be read in isolation. Readers are referred to the Review‟s findings on these points as set out in the body of the report.



## ISSUES FOR EXAMINATION

1. The Review Team’s remit is set out in its terms of reference as follows:

―The Independent Review will investigate the key allegations that arose from a

series of hacked e- mails from the University of East Anglia‘s Climatic Research Unit (CRU). The review will:

Examine the hacked e-mail exchanges, other relevant e-mail exchanges and any other information held at CRU to determine whether there is any evidence of the manipulation or suppression of data which is at odds with acceptable scientific practice and may therefore call into question any of the research outcomes.

Review CRU‘s policies and practices for acquiring, assembling, subjecting to peer review and disseminating data and research findings, and their compliance or otherwise with best scientific practice.

Review CRU‘s compliance or otherwise with the University‘s policies and practices regarding requests under the Freedom of Information Act ( ̳the FoIA‘) and the Environmental Information Regulations ( ̳the EIR‘) for the release of data.

Review and make recommendations as to the appropriate management, governance and security structures for CRU and the security, integrity and release

of the data it holds.‖

1. The remit reflects the reaction to the e-mails that became public, much of it questioning and critical. In following up those questions and criticisms the Team stresses that it has formed no view on whether they are fair or justified. In formulating in its own words an expression of the issues for examination the Team is not adopting those issues as its own criticisms.

2. The Team‘s approach is to distill the questions and criticisms into the broad questions set out below. Using its own enquiries and experience, it has added questions about the handling and dissemination of data, including the response to FOI requests. The issues addressed in the first three paragraphs of the terms of reference will inform recommendations for paragraph four, as to the appropriate management, governance and security structures for CRU and the release of data.

3. The Team w ill invite CRU and other parts of UEA to respond in writing to these questions, and will follow up those responses as required. The Team expect the CRU to provide original documentary evidence to support its responses.

4. The Team invites those with an interest in the matter to comment on this Issues paper. The Team stresses that its remit does not involve re-evaluation of theAPPENDIX 3

113 scientific conclusions of the CRU work, still less a reappraisal of the scientific debate about the existence and suggested causes of global warming. Please confine any comments on this paper to matters within the remit at paragraph 1 of this paper.

5. Written responses from CRU and others are requested by 1 March 2010 to the

address below:

Email: correspondence@cce-review.org

Or post to,

Climate Change E-Mails Review

Box 18

196 Rose Street Edinburgh

EH2 4AT



## ISSUES ARISING ON Para 1.1 OF THE TERMS OF

REFERENCE

6. The allegation of ignoring potential problems in deducing palaeotemperatures from tree ring data that might undermine the validity of

the so- called “hockey

- stick” curve.

7. In the late 20 th century, the correlation between the tree ring record and instrumental record of temperature change diverges from that for the earlier period. The cause of this divergence does not appear to be understood. If the method used to deduce temperatures from tree ring proxy metrics for the earlier tree ring record is applied to the late 20 th century tree ring series, then declining temperatures would be deduced for the late 20 th century. It is alleged that if the cause of divergence between the tree ring and instrumental temperature record is unknown, it may have existed in earlier periods. Therefore if tree rings had similarly failed to reflect the warming of the early Middle Ages, they may significantly under-estimate the warming during the Medieval Warm Period, thus falsely enhancing the contrast between the recent warming and that earlier period. (It is this contrast that has led to statements that the late 20 th century warming is unprecedented during at least the last 1000 years.)

QUESTIONS TO ADDRESS:

What method do you use to deduce palaeotemperatures from tree ring

data? Does not the problem of divergence for the late 20 th century record invalidate the deduction of tree ring palaeotemperatures for the period prior to the instrumental record? How open have you been about this issue? What attempts have you made to resolve it?APPENDIX 3

114 What is the evidence that the amplitude of warming during the Medieval Warm Period (MWP) is not underestimated by tree ring evidence? How does the tree ring evidence of the MWP compare with other proxy data? Have you showed how data from different sources compare or have you conflated them? If the latter, what is the justification? If tree ring proxies are removed from reconstructions, what evidence remains of the MWP? Have you been selective in utilizing tree ring evidence from Yamal in Siberia; and if so, what is the justification for selectivity and does the selection influence the deduced pattern of hemispheric climate change during the last millennium?

8. The allegation that CRU has colluded in attempting to diminish the significance of data that might appear to conflict with the 20 th century global warming hypothesis

9. The CRU group, in consultation with Professor Michael Mann, is alleged to have systematically attempted to diminish the significance of the Medieval Warm Period, evidenced by an email from Mann 4 th June 2003: ―I think that trying to adopt a timeframe of 2K, rather than the usual 1K, addresses a good earlier point that Peck made w/ regard to the memo, that it would be nice to try to "contain" the putative "MWP", even if we don't yet have a hemispheric mean reconstruction available that far back [Phil and I have one in review

-- not sure it is kosher to show that yet though

-- I've put in an inquiry to Judy Jacobs at AGU abou t this].‖ The use of the words ―contain‖ and ―putative‖ are alleged to imply an improper intention to diminish the magnitude and significance of the MWP so as to emphasise the

late 20 th century warming.

QUESTIONS TO ADDRESS

What does the word “contain” m ean in this context? What involvement have you had in “containing” the MWP? How important is the assertion of “unprecedented late 20 th century warming” in the argument for anthropogenic forcing of climate?

10. It is alleged that proxy temperature deductions and instrumental temperature data have been improperly combined to conceal mismatch

between the two data series

11. An attempt to hide the difficulty of combining these two data series and to mislead is alleged to be revealed in the following sentence in a November 1999 email from Professor Phillip Jones which is alleged to imply a conscious attempt to mislead: "I've just completed Mike's Nature trick of adding in the real temps to each series for the last 20 years (i.e. from 1981 onwards) and from 1961 for Keith's to hide
the decline‖.

QUESTIONS TO ADDRESS

What is the meaning of the quotation from the 1999 email?APPENDIX 3

115 How do you justify combining proxy and instrumental data in a single plotted line? What method do you use?

12. It is alleged that there has been an improper bias in selecting and adjusting data so as to favour the anthropogenic global warming hypothesis and details of sites and the data adjustments have not been made adequately available

13. It is alleged that instrumental data has been selected preferentially to include data from warmer, urban in contrast to rural sites; that the rationale for the choice of high/low latitude sites is poor; and that the processes by which data has been corrected, accepted and rejected are complex and unclear.

QUE STIONS TO ADDRESS

What is the rationale for the choice of data stations worldwide? How has this choice been tested as appropriate in generating a global or hemispheric mean temperature (both instrumental and proxy data)? Describe as clearly as possible the protocols you have followed in selecting, correcting and rejecting data and stations. Has this been an orderly and objective process applied to all datasets? To what extent have different procedures for data of different vintages and different sources been unified? What means do you use to test the coherence of the datasets?



## ISSUES ARISING ON Para 1.2 OF THE TERMS OF

REFERENCE

14. It is alleged that there have been improper attempts to influence the peer review system and a violation of IPCC procedures in attempting to prevent the publication of opposing ideas.

15. It is alleged that there has been an attempt to subvert the peer review process and exclude publication of scientific articles that do not support the Jones-Mann position on global climate change. A paper by Soon & Balunias was published in

the Journal Climate Research arguing that the 20 th century was abnormally warm.

An email from Professor Michael Mann on 11 th March 2003 contained the following: "I think we have to stop considering Climate Research as a legitimate peer-reviewed journal. Perhaps we should encourage our colleagues in the climate research community to no longer submit to, or cite papers in, this journal." The allegation is that journals might be pressured to reject submitted articles that do not support a particular view of climate change.

16. In an email to a fellow researcher in June 2003, Briffa wrote: ―Confidentially I now need a hard and if required extensive case for rejecting (an unnamed paper) to support Dave Stahle‘s and really as soon as you can.‖

17. In an email to Mann on 8 th July 2004, Jones wrote: "The other paper by MM is just garbage. [...] I can't see either of these papers being in the next IPCC report.APPENDIX 3

116 Kevin and I will keep them out somehow

— even if we have to redefine what the peer-review literature is!" The allegation is of an attempt to prevent ideas being published and the author being prepared to subvert the peer review process for a journal and to undermine the IPCC principle of accounting properly for

con tradictory views.



## QUESTIONS TO ADDRESS

Give full accounts of the issue in relation to the journal Climate Research, the June 2003 email, and the March 2004 email to Mann (“recently rejected two papers (one for Journal of Geophysical Research & one for Geophysical Research Letters) from people saying CRU has it wrong over Siberia. Went to town over both reviews, hopefully successfully. If either appears I will be very surprised”. Are the first two instances evidence of attempts to subvert the peer review process? In relation to the third, where do you draw the line between rejecting a paper on grounds of bad science etc, and attempting to suppress contrary

views? To what extent is your attitude to reviewing conditioned by the extent that a paper will set back the case for anthropogenic global warming and the political action that may be needed to mitigate it? What is the justification for an apparent attempt to exclude contrary views from the IPCC process?

18. The scrutiny and re-analysis of data by other scientists is a vital process if hypotheses are to rigorously tested and improved. It is alleged that there has been a failure to make important data available or the procedures used to adjust and analyse that data, thereby subverting a crucial scientific process.

19. It is alleged that there has been a systematic policy of denying access to data that has been used in publications, referring to an email from Jones to Mann on 2 nd
February 2005 which contains the following: "And don't leave stuff lying around on ftp sites - you never know who is trawling them. The two MMs have been after the CRU station data for years. If they ever hear there is a Freedom of Information Act now in the UK, I think I'll delete the

fi le rather than send to anyone. Does your similar act in the US force you to respond to enquiries within 20 days?

— our does! The UK works on precedents, so the first request will test it. We also have a data protection act, which I will hide
behind‖.



## QUESTIONS TO ADDRESS

Do you agree that releasing data for others to use and to test hypotheses is an important principle? If so, do you agree that this principle has been abused? If so, should not data be released for use by those with the intention to undermine your case, or is there a distinction you would wish to make between legitimate and illegitimate use?APPENDIX 3

117 If not, do others have reasonable access to the data at all levels and to the description of processing steps, in order to be able to carry out such a re

analysis? Can you describe clearly the data-sets and relevant meta-data that have been released; what has not been released and to what extent is it in useable form? Where has it been released? Where access is limited, or not possible, or not meaningful, for legitimate reasons please explain why?

20. The keeping of accurate records of datasets, algorithms and software used in the analysis of climate data.

21. A key concern expressed by a number of correspondents and commentators has been as to whether datasets, and analyses based thereon, were deleted.



## QUESTIONS TO ADDRESS

Were formal „data dictionaries‟ kept of the data sets acquired by the CRU at various times from other bodies such as the UK Meteorological Office Hadley Centre and its equivalents around the World? Were comprehensive records kept of the way these various data sets were used, the statistical and other algorithms used in processing them, and the various software programmes and modules used to carry out that processing? Does a formal library of algorithms and software used by the CRU exist? What quality control measures were used to test the various algorithms and software modules developed by the CRU? What techniques did members of the CRU employ to ensure the integrity of the various applications used to process climate data? What policies are in place to ensure the formal archiving of data sets and resultant analyses for future use and review.



## ISSUES ARISING ON Para 1.3 OF THE TERMS OF

REFERENCE

22. Response to Freedom of Information requests.

23. A number correspondents and commentators assert that requests under the Freedom of Information Act (FoIA) and the Environmental Information Regulations (EIR) were incorrectly denied by the University of East Anglia on

advice from the CRU. This is the subject of a separate inquiry by the Information Commissioner, but does fall within the terms of reference of the Review Team.



## QUESTIONS TO ADDRESS

What formal processes were in place both centrally and within the CRU to ensure fair and impartial assessment of FoIA requests?APPENDIX 3

118 Were there any processes in place centrally to review recommendations from the CRU that information should not be released? Over the five years to November 2009:

o how many requests were received?

o how many were rejected, and on what grounds?

o how many received full release of information?

o how many received partial release of information?

Independent Climate Change E-Mails Review, February 2010APPENDIX 3

119APPENDIX 4

120 

## APPENDIX 4: INDEX OF MEETINGS, INTERVIEWS, SUBMISSIONS, FOLLOW UP ENQUIRIES AND

RESPONSES

Team Meetings were held on the following dates in 2010. Confirmed notes of Team Meetings can be viewed at: http://www.cce-review.org/Meetings.php

12 January 4 February 25 February

20 March 1 April 13 April 22 April 28 April 11 May 26 May

7 June

11 June

15 June

Other Meetings and Interviews conducted. Notes of the meetings and supporting documentation can be accessed from: http://www.cce-review.org/Evidence.php

18 December 2009, meetings conducted by Sir Muir Russell at the UEA. Notes were taken by Lisa Williams (Senior Assistant Registrar), UEA: Brian Summers (Registrar and Secretary), Professor Trevor Davies (PVC
Research, Enterprise and Engagement), Professor David Richardson (Dean,
Faculty of Science), UEA

Stuart Holmes, (Chair, UEA Council), UEA

Jonathan Colam-French (Director of Information Services), Iain Reeman (ICT
Systems Director), Steve Mosley (ICT Policy Manager), UEA
Brian Summers (Registrar and Secretary), David Palmer (Information Policy
& Compliance Manager) and Jonathan Colam-French (Director of Information
Services), UEA
Alan Preece (Director of Marketing and Communications) and Annie Ogden
(Head of Communications), UEA

Professor Trevor Davies (PVC Research, Enterprise and Engagement) and
Professor Philip Jones (Climatic Research Unit), UEA
Professor Trevor Davies (PVC Research, Enterprise and Engagement) and
Professor Keith Briffa (Climatic Research Unit), UEA
Superintendent Julian Gregory, Norfolk Constabulary (meeting note withheld to avoid prejudicing ongoing police investigation).APPENDIX 4

121 27 January 2010, meetings conducted by Sir Muir Russell and Professor Jim Norton

at the UEA:

Mike Gorrill (Head of Enforcement) and David Clancy (Investigations Manager) of the Information Commissioner‘s Officer
Jonathan Colam-French (Director of Information Services) and Mike Salmon (IT Manager to the CRU - 40% time), UEA Professor Philip Jones and Professor Keith Briffa, Climatic Research Unit,

UEA Superintendent Julian Gregory and Andy Guy from Norfolk Constabulary (meeting note withheld to avoid prejudicing ongoing police investigation)

4 March 2010, interviews conducted by Professor Peter Clarke and Professor Jim

Norton at the UEA:

Professor Philip Jones, Dr Tim Osborn and Ian Harris, Climatic Research Unit, UEA

24 March 2010, meeting conducted by Professor Jim Norton at the Information
Commissioner‘s Office:

Mike Gorrill (Head of Enforcement) and Steve Wood (Assistant
Commissioner FOI)

26 March 2010, meetings conducted by Sir Muir Russell and David Eyton at the

UEA: Ian McCormick (Director of Research, Enterprise and Engagement) and Alan
Walker (Manager of Research Services), UEA
Rob Bell (Research Finance Management Accountant) and Laura McGonagle
(Faculty of Science Finance Manager), UEA
Professor Trevor Davies (PVC Research, Enterprise and Engagement), UEA
Professor Trevor Davies (PVC Research, Enterprise and Engagement) and
Professor David Russell (Associate Dean for Research, Faculty of Science),

UEA
Cecile Piper (Director of Human Resources), UEA
Brian Summers (Registrar and Secretary), UEA

Professor Edward Acton (Vice-Chancellor), UEA

30 March 2010, meetings conducted by Sir Muir Russell and Professor Jim Norton at

th e UEA: Michael McGarvie (Science Faculty FoIA contact), UEA
David Palmer (Information Policy & Compliance Manager) and Jonathan
Colam-French (Director of Information Services), UEA

9 April 2010, meetings conducted by Professor Geoffrey Boulton and Professor Peter

Clarke at the UEA: Professor Keith Briffa, Professor Philip Jones, Dr Tim Osborn and Dr Tom

Melvin (Climatic Research Unit), David Palmer (Information Policy &APPENDIX 4

122 Compliance Manager) and Jonathan Colam-French (Director of Information Services), UEA

15 June 2010, telephone interview conducted by Sir Muir Russell and David Eyton with Professor Trevor Davies, PVC Research, Enterprise and Engagement

Telephone interviews conducted by Professor Geoffrey Boulton: 1 May, Professor Sir Brian Hoskins, IPCC AR4 Review Editor Ch 3 (Surface and Atmospheric Climate Changes)
1 June, Professor John Mitchell, IPCC AR4 Review Editor Ch 6 (Palaeoclimate).

Follow- up enquiries, responses and related correspondence. This material can be accessed from: http://www.cce-review.org/Evidence.php

Presentations from 20 March Review Team Meeting
Independent forensic analyst report and commentary on e-mail examination
Redacted set of FOI and EIR requests received relating to the Climatic

Research Unit since 2005 Correspondence with Lord Oxburgh in respect of the separate reviews
Follow-up request to Professor Jones and response on issues relating to the

IPCC Clarification request to Professor Jones and response on issues relating to the

IPCC Follow-up request to Professor Jones and response in relation to raw instrument station availability for each CRUTEM data set Follow-up request to Professor Briffa and response in relation to best scientific practice in dissemination scientific evidence into the public domain Follow-up request and response on the scope of material that was backed up on the server that was compromised

Follow- up request and response in relation to the UEA‘s main university e

mail server domain Follow-up request and response in relation to governance and risk management issues Follow-up request and response in relation to financial controls Follow-up request to ICO and response in relation to the holding of

information or data Response to the Science and Technology Committee‘s Eight Report on The

Disclos ure of Climate Data from the Climatic Research Unit ( Note: at the time
of writing, the Science and Technology Commons Select Committee has not
given permission to publish this response)

Submissions The Review was launched on 11 February and submissions were invited from this date and accepted up to 16 April. The Review accepted a small number of submissions that it deemed to have significant relevance to its remit after this close

date.APPENDIX 4

123 Submissions to the Review can be viewed at: http://www.ccereview.org/Evidence.php

Submissions Published

Submission Name Date received

Number

1 Nicholas Barnes 06

- Mar

2 David Archer 25

- Feb

3 Dr Phillip Bratby 22

- Feb

4 Philip Brohan 24

- Feb

5 Climatic Research Unit 01

- Mar

6 David Cockroft 28

- Feb

7 Gill Chant 13

- Feb

8 Tor Berge S. Gjersvik 12

- Feb

9 Professor Gabriele Hegerl 02

- Mar

10 Michael Hughes 17

- Feb

11 Professor Mike Hulme 26

- Feb

12 Professor Dr Fortunat Joos 01

- Mar

13 Dr D.R. Keiller 14

- Mar

14 Andrew MacIntyre 12

- Feb

15 Dr Ross McKitrick 26

- Feb

16 Dr P.C. Matthews 01

- Mar

17 Forrest M.Mims III 18

- Feb

18 Andrew Montford 11

- Feb

19 Professor Raymond Bradley 01

- Mar

20 Jim Stathos 26

- Feb

21 Professor Simon Tett 01

- Mar

22 The Global Warming Policy Foundation 28

- Feb

23 Stephen McIntyre 02

- Mar

24 Professor Rob Wilby 26

- Feb

25 Dr Austin Woods 11

- Feb

26 Professor Malcolm Hughes 01

- Mar

27 W.F. Lenihan 15

- Feb

28 Richard Calhoun 14

- Feb

29 Dr Henry Barnard 27

- Feb

30 Ron Cram 15

- Feb

31 Charlie Kilgore 13

- Feb

32 Vincent Moran 14

- Feb

33 Dr Richard North 11

- Feb

34 Stephen Richards 15

- Feb

35 Cameron Rose 16

- Feb

36 Dr C.W. Schoneveld 15

- Feb

37 David Shepherd 13

- Feb

38 Geoffrey Sherrington 17

- Feb

39 Martin Vermeer 27

- Feb

40 Robert Wright 15

- FebAPPENDIX 4

124 41 Brent Hargreaves 16

- Feb

42 Professor Michael Mann 28

- Feb

43 Dr Sonja Boehmer

- Christiansen 23

- Feb

44 Dr Benjamin Santer 28

- Feb

45 Trevor Jones 28

- Feb

46 D R G Andrews 08

- Apr

47 Professor Joshua Halpern 08

- Apr

48 Mike Haseler 22

- Feb

49 John R. Smith 17

- Feb

50 Dr D.R. Keiller 17

- Feb

51 Dr D.R. Keiller 17

- Feb

52 Dr David Lehmiller 15

- Feb

53 Dr D.R. Keiller 24

- Mar

54 Bob Smith 19

- Feb

55 Robert Wright 19

- Feb

56 Ian MacDonald 28

- Feb

57 Dr David Lehmiller 15

- Feb

58 Stephen Richards 11

- Feb

59 Ray Soper 17

- Feb

60 David Sandeman 18

- Feb

61 Dr Josep Verges 15

- Feb

62 Dominic O'Kane 15

- Feb

63 Vid Stimac 17

- Feb

64 Chris Allen 12

- Feb

65 Ian MacDonald 20

- Feb

66 Tony Brown 13

- Feb

67 Gerry Morrow 12

- Feb

68 Stephen Graves 12

- Feb

69 Will Hawkes 12

- Feb

70 C. Barling 12

- Feb

71 Susan Ewens 13

- Feb

72 Malcolm McClure 13

- Feb

73 Robert Owen 13

- Feb

74 Bruce Garrick 13

- Feb

75 Stephen Johnson 13

- Feb

76 Jim Stathos 13

- Feb

77 Dr Barry Napier 14

- Feb

78 Joe Olson 16

- Feb

79 Roderick Campbell 16

- Feb

80 John R. Smith 16

- Feb

81 Ray Soper 18

- Feb

82 Julie Grace 17

- Feb

83 Doug Vickers 15

- Feb

84 David Shepherd 15

- Feb

87 Mike Haseler 12

- Feb

88 Dr D.R. Keiller 06

- MarAPPENDIX 4

125 89 Patrick Keane 17

- Feb

90 Patrick Keane 18

- Feb

91 Dr D.R. Keiller 20

- Feb

92 Richard Calhoun 11

- Feb

93 Robert Denton 18

- Feb

94 Robert Denton 27

- Feb

95 John Graham

- Cumming 17

- Feb

96 D.G. McBeath 22

- Feb

97 George Mihailides 28

- Feb

98 David Shepherd 11

- Feb

99 Patrick Keane 15

- Feb

100 Dr D.R. Keiller 23

- Feb

101 Dr D.R. Keiller 26

- Feb

102 Dr D.R. Keiller 28

- Feb

103 Climate Scientists Letter 28

- May

104 McIntyre Submission 9

- Jun

105 Dr McKitrick supplementary sub 1 13

- Apr

106 Dr McKitrick supplementary sub 2 13

- Apr

107 Geoffrey Sherrington 4

- Mar

108 Professor Tom Wigley 28

- Feb

109 Patrick Keane 12

- Feb

Submissions Not Published

David Holland Conor McMenemie (submission to be treated as confidential)

The Review reserved the right to withhold publication of submissions if they were abusive, potentially defamatory, anonymous, or there were other legal difficultie s which prevented publication.APPENDIX 5

126 

## APPENDIX 5 : PEER REVIEW

UNDERSTANDING UNCERTAINTY: A BRIEF HISTORY

OF PEER REVIEW

By Richard Horton, Editor of The Lancet

Amid the public and scientific furore over alleged events at the University of East Anglia‘s Climatic Research Unit (CRU), peer review has emerged as a central issue in
the dispute. In the Times Higher Education , for example, Andrew Montford, author of
The Hockey Stick Illusion: Climategate and the Corruption of Science (1), argued that
events at the CRU had far-reaching implications for the world of scientific publishing
(2). His charge sheet was extensive

– undermining the peer-review process,
threatening editors who published work contrary to orthodox scientific opinion,
organising mass resignations from editorial boards, and persuading colleagues to stop submitting papers to allegedly offending journals. Montford suggests that ―as many as

four different journals may have had their normal procedures interfered with‖. He

continues,

“What is an ethical way to deal with a journal?...
At what point does valid protest elide into something
more sinister?...If, as the [CRU] emails suggest, some

scientists are in fact putting illegitimate pressure on
journals, either to influence the peer-review process or
to prevent the release of data, it is easy to see how editors may find it difficult to respond.”

Implicit in Montford‘s argument is that peer review is critical to the process of – and
thereby public trust in

– science. Writing in The Guardian , George Monbiot put it this way: ―science happens to be [a] closed world with one of the most effective form s of

self- regulation: the peer review process.‖(3).

The importance of peer review has been invoked by climate sceptics in other domains
of the climate debate. Christopher Booker has challenged Dr Rajendra Pachauri, for
example, for claiming that his Intergovernmental Panel on Climate Change (IPCC)
included only peer-reviewed research (4). By contrast, Booker reports that a third of
IPCC sources were newspaper articles, student dissertations, even press releases.
Again, the suggestion is that the peer-rev iewed literature is something special and
sacred to science. One can make strong and reliable assertions if those statements are
underpinned by peer-reviewed science. If evidence has not been peer-reviewed, it is

next to worthless.

In the context of the CRU, there have been claims that peer reviewers censoredAPPENDIX 5

127 evidence deliberately to exclude those findings from scientific journals. Questions
have been raised about the line between rejecting a paper on grounds of bad science
and rejecting it as part of an attempt to suppress contrary opinions. Peer reviewers have been accused of letting their politics get in the way of science. Did reviewers‘
concerns about the potential adverse policy and political implications of supporting
perfectly good science at peer review override their scientific judgment? The
suggestion is that they did.

The House of Commons Science and Technology Committee report on the CRU (5) has emphasised that ―the question of the operation of peer review is going to be a
critical issue‖ in any evaluation of the CRU‘s work. In their report, in a section
headed ―Perverting the peer review process‖, MPs set out several specific allegations
against CRU scientists. First, that they colluded to subvert peer review, slowing or
blocking the publication of research which disagreed with their own views. Second, that climate scientists planned to ―redefine what the peer

- review literature is‖ in their
recommendations to the IPCC (6). Third, that they tried to suppress a paper on
research fraud. And finally, that CRU staff exerted improper pressure on the editor of
one journal, an editor who had been open to scientists challenging climate change
orthodoxy. The House of Commons Committee concluded that there was no evidence
to suggest that the CRU had tried to subvert the peer review process. They wrote that, ―Academics should not be criticised for making informal comments on academic
papers‖.

Much of the concern

–

and, indeed, confusion

– about what took place at the CRU in
relation to peer review may stem from misunderstandings about what peer review is
and what it can be expected to do.



## Peer review: firewall or the weakest link?

For scientific journals, peer review is the confidential evaluation of a submitted
manuscript by one or more individuals who are experts in an aspect of the work under
scrutiny.

Who invented peer review? It‘s hard to be sure, but possibly the prize goes to Ishaq
bin Ali Al Rahwi (AD 854-931) (7). In his book, Ethics of the Physician , Al Rahwi
apparently encouraged doctors to keep contemporaneous notes on their patients, later
to be reviewed by a jury of fellow physicians. But the serious business of journal peer
review had to wait another 800 years. Henry Oldenburg, editor of Philosophical
Transactions of the Royal Society , was the first modern editor to adopt peer review in
the seventeenth century. He used it to famous effect, provoking often fractious, but
illuminating, debates between scientists across Europe.

Technology

–

first the typewriter, then the photocopier, and now the Internet

– has
greatly facilitated peer review. Any scientific journal that lays claim to respectabilityAPPENDIX 5

128 must have a robust peer review process. At The Lancet

, the process goes like this. A
research paper is submitted electronically to a secure database and allocated by an
editor to a colleague. The first or second editor can reject the manuscript at that early stage if the paper is judged to be scientifically poor, unsuitable for the journal‘s
readership, unoriginal, or insufficiently topical. Journals differ here. For The Lancet
around three-quarters of manuscripts are rejected at this point.

If a paper survives preliminary editorial review, it is discussed at a pre-review
meeting to assess its suitability for external peer review. If judged a potential
candidate for publication, the manuscript is sent to three expert advisors, commonly
international and representing different methodological dimensions of the research, as
well as a statistician. There is always the risk of group-think among experts. That is,
there is an orthodox belief about a particular subject, strongly held, which resists
alternative perspectives. Editors try to reduce the risk of group-think by sending
papers to different and widely dispersed reviewers, deliberately seeking or even
provoking critical reviews (just like Henry Oldenburg). Reviewers are not referees in
the sense that they can blow a whistle and call time on the paper. We ask reviewers to
provide written comments for the authors, confidential comments to the editors, and a
detailed rating for each section of the paper. Those comments are collected, presented,

and discussed at a once- weekly manuscript meeting attended by all the journal‘s

editors.

At this stage, a paper can be rejected or we can open negotiations with authors. If we proceed, reviewers‘ questions and concerns are put to the authors, with appropriate
guidance from editors. The authors will reply by answering each question from
reviewers, submitting a revised manuscript that attempts to respond to all points
raised by editors and reviewers alike. The authors may also disagree with or challenge
reviewers with varying degrees of force. The revised paper is discussed again at a
manuscript meeting. The options at this stage are to reject, accept, go back to the
authors with further requests for clarification, or return to reviewers (old or new) for
additional opinions. We proceed with further revisions of the paper until a final
reject/accept decision is made. We know that with such a high rejection rate we may
get it wrong. To limit errors of omission, we have a formal appeals process where editors promise to look again at a paper, weigh up the authors‘ arguments, and

reconsider our decision.

Once the paper is provisionally accepted, the peer review process is not over. The paper is then passed to a scientifically qualified assistant editor who edits the paper‘s
technical content. Mistakes may still be found at this stage, leading to further editorial
or expert review, even (though rarely) rejection. A lesson learned from sometimes
bitter experience is that a paper is not fully accepted until it is published.

Here are some of the commonest questions asked about the peer review process (8).APPENDIX 5

129 Do reviewers make mistakes in their judgments ? Of course, and so do editors.
Sadly, the scientific literature is littered with retractions of papers that once passed the
test of peer review. Reviewers and editors are disappointingly human.

Are reviewers objective in their judgments ? Pure objectivity is impossible. For some
subjects, an editor can predict the judgment of the reviewer based on past experience
with that reviewer. But this misses the point of what an editor is seeking. It is not
simply the judgment of reject/accept that an editor wants from a reviewer. That
decision is the responsibility of the editor and the editor alone. What an editor really
seeks is a powerful critique of the manuscript

– testing each assumption, probing
every method, questioning all results, and sceptically challenging interpretations and

co nclusions. Armed with that critique, the editors decide

– and take full responsibility
for deciding.

Are reviewers willing to accept new ideas ? Certainly, they are, although they might
question those ideas to destruction. The vast majority of reviewers take their
responsibility as advisors very seriously indeed. They themselves are often on the
receiving end of peer review. Most try to be as open as possible to new findings,
although we encourage them to ask difficult and awkward questions. There are
occasional exceptions. For example, the world of complementary and alternative
medicine (CAM) divides the medical community. Orthodox medicine mostly rejects
papers about reflexology, iridology, and acupuncture treatment that invokes invisible
pathways (meridians) of qi. CAM is served by a separate class of journals that have
little overlap with the more mainstream medical literature. In this instance, ideas are

incommensurable.

Despite peer review, are authors able to get away with dishonest or dubious

research ? Yes, they are. Peer review does not replicate and so validate research. Peer
review does not prove that a piece of research is true. The best it can do is say that, on
the basis of a written account of what was done and some interrogation of the authors,
the research seems on the face of it to be acceptable for publication. This claim for
peer review is much softer than often portrayed to the general public. Experience
shows, for example, that peer review is an extremely unreliable way to detect research

misconduct.

Are peer reviewers accountable for what they do ? Yes, to the editor. But in a broader
sense, to the scientific community and to the public as well. To a large extent, the
trust society places in science depends on the scientific process, including peer review
and publication, getting it right most of the time.

Does peer review improve the quality of published research? In our everyday
practice, we see that it does. And research suggests that it does too (9). Peer review
improves discussion of the limitations of research. It emphasises uncertainty. It invites justification of generalisability. As one study of peer review concluded, ―peer reviewAPPENDIX 5

130 is a negotiation between authors and journal about the scope of the knowledge claims

that will ultima tely appear in print‖ (9).

Is there still a need for peer review, given the extraordinary ability of the Internet to
enable continuous open criticism of research once published (that is, surely a
thousand readers as reviewers after publication are better than 4 reviewers selected
by editors before publication)? There is no right answer to this question. Different
views have been expressed. Certainly, post-publication peer review adds greatly to the
understanding of a piece of research. But watching pre-pub lication peer review in
action - both at the macro level of external expert review and the micro level of
technical editing - and seeing the extent to which research papers change (mostly for
the better) after peer review, I think that pre-publication review still has an important
part to play in science. At its best, pre-publication peer review clarifies, introduces
uncertainty, insists on placing new work in the context of the totality of available
evidence, demands a careful explanation of limitations, and prevents flights of
fanciful over-interpretation.

Peer review has changed considerably during the past two decades (since I became an
editor). First, the stakes are higher. Individual and institutional success depends on
getting papers published in high -impact journals. Citation data are now a standard
metric for measuring research performance. This trend has increased competition and
rivalry for places in the best journals. Second, the globalisation of science has
expanded the geographic range of papers submitted to journals. Research originating
from China, for example, is now far more common than even five years ago. The
internationalisation of science has further intensified competition for publication.

Third, research papers are increasingly multi-disciplinary, requiring a much broader
range of expertise during peer review. Fourth, science is a stronger part of our public
culture now than it once was. What scientists used to write only for other scientists is
today available to

–

and sometimes read by

– non -scientists, policy makers, and the
media. Fifth, the importance of statistics has grown substantially. Whereas twenty
years ago The Lancet had no separate statistical peer review process, every paper we
now publish has been carefully scrutinised by an independent statistical advisor.
Editors are now far more aware of analytic errors in research. Sixth, to address the
often conflicting results of individual research studies that are trying to answer the
same (or a similar) question, a new type of research method has been devised

– the
systematic, as opposed to the narrative, review. Systematic reviews aim to search for
particular types of study (eg, the randomised trial), then select only the best according
to pre-specified criteria, and, if possible, to combine those findings in a statistically
meaningful way (which is called meta-analysis). Examples include the risk of cervical
cancer among women taking hormonal contraceptives (10) and the effects of a class
of medicines on heart disease (11). In biomedicine, the Cochrane Collaboration is the
most mature example of an effort to create a database of systematic reviews on

treatments.APPENDIX 5

131 Finally, editors have had to face an upsurge in the discovery of episodes of research
misconduct (fabrication, falsification, and plagiarism). The increasing awareness of
research fraud had led not only to greater vigilance (hopefully not suspicion) among

editors but also to the birth of institutional mechanisms to set standards and advise on
research practice (eg, the Committee on Publication Ethics).

Because of the faith journal editors have in peer review, together with the empirical
evidence they believe exists to support peer review, they take it very seriously indeed
(12). That said, editors are well aware that peer review is anything but
uncontroversial. Scientific discoveries that later turn out to be flagrant episodes of
dishonesty

–

from Woo- Suk Hwang‘s fabricated claims in Science about cloning embryonic stem cells, to Andrew Wakefield‘s falsifications in Th e Lancet

– are not uncommon. They raise troubling questions about the robustness of peer review.
Editors are only too well aware of the limitations of the peer-review system. Authors,
for example, can be deeply resistant to responding to questions from anonymous
critics (this fact at least partly drives the argument for fully transparent peer review,
where reviewers have to disclose their names to authors). The reluctance of some

authors

– and some very famous authors, at that

– to take the comments of their peers
seriously stems from the fact that they believe they have no peers. As one historian of peer review put it, somewhat poetically, ―anyone who possessed the MD degree had
no reason to defer to any colleague as an expert greater than he or she‖ (13).

So what is peer review in today‘s scientific culture? Various views have been more or
less vividly expressed. Peer review is a ―sacred academic cow‖, according to one
editor (14). She put it rather well:

“the „sacred‟ cow of peer review wanders the me adows
of scientific publishing because together, scholars and

editors, believe that it is the best mechanism we have to improve the quality of published papers...The one
component that we cannot control is that of competition

in the academic world, an issue that continues to circle
in all disciplines. We all have stories to tell about the viciousness of academic politics...although we might
not be able to eradicate the politics, we can at least
understand them for what they are ...”

Everyone

– scientists, the public, policymakers, politicians

– would like to believe
that peer review is a firewall between truth and error (or dishonesty) (15). But as the editor of one leading specialist medical journal has rightly pointed out, ―Ther e is no
question that, when it comes to peer review, the reviewers themselves are the weakest (or strongest) links‖ (16). This frustration among editors and scientists that peer
review cannot always live up to the claims sometimes made for it produces frequentAPPENDIX 5

132 expressions of dismay. Is peer review a castle built on sand or the bedrock of
scientific publishing (17)? Is peer review a landmark, landmine, or landfill (18)? Or, put bluntly, is peer review simply in crisis? (19). Is it ―a flawed process at the he art of science and journals‖ (20)?

Unfortunately, there is evidence of a lack of evidence for peer review‘s efficacy. In
2002, Tom Jefferson and colleagues published a startling systematic review of all the
evidence about editorial peer review in biomedical journals. Their exhaustive search yielded only a handful of studies. The conclusion? ―Editorial peer review, although
widely used, is largely untested and its effects are uncertain‖ (21). They went on,
―Given the widespread use of peer review and its im portance, it is surprising that so little is known of its effects.‖ Jefferson and his colleagues have confirmed their
observations more recently (22). Their findings have been replicated by others (23).
To be fair, there is some evidence that micro peer review

– technical editing

–

can improve papers in biomedical journals (24). But, once again, this evidence is not as
robust as one would either like or have expected.

Jefferson extended his investigation of peer review by arguing that the objectives of
the review process were also unclear (25). Without clear objectives, proving the value
of peer review (or not) would be impossible. After almost 350 years of journal peer
review, our zeal for and confidence in the peer review process seem inversely
proportional to our knowledge about what it actually does for science. Those who
make big claims for peer review need to face up to this disturbing absence of

evidence.

Worse still, what evidence is slowly accumulating should perhaps make scientists,
policymakers, and the public pause. Many who place great weight on the reliability of
the peer-reviewed scientific literature believe that it reflects the judgment of the
scientific community about the quality of research. But evidence suggests that
acceptance of research for publication may well depend on factors other than
scientific quality alone (26). Furthermore, peer reviewers will disagree greatly in their
recommendations to editors about a particular research paper. Yet editors seem to be
significantly influenced by reviewers who, when the quality of their advice is
measured independently, turn out to be extremely unreliable in their overall
judgments (27). Editors, some critics could reasonably argue, need to pay less, not
more, attention to the recommendations of their peer reviewers.

Scepticism about peer review is healthy. But every editor knows that peer review can
be an indispensible aid to his or her work. Peer review can rescue science from
embarrassment and error. An extreme example goes some way to showing why. Peter
Duesberg is a well-known molecular virologist who believes that HIV is not the cause
of AIDS. In 2009, the journal Medical Hypotheses published a paper by Duesberg
arguing that the deaths attributed to AIDS in South Africa were false. The editor of
Medical Hypotheses operated an editorial policy of no external peer review. TheAPPENDIX 5

133 justification was that peer review might suppress creative thinking. In the case of the
Duesberg paper, the idea that HIV does not cause AIDS was not new. More
imp ortantly, South Africa is only now reversing its disastrous denialist policies on

HIV- AIDS. To consider Duesberg‘s old (and discredited) idea at a critical moment for
the country he was writing about would, most reasonable editors would conclude,
require some kind of external peer review to assist decision-making. The editor did
not seek expert reviews. He accepted the paper within a few days of its submission.
Many scientists in the AIDS community were appalled. They wrote to the publishers
(Elsevier, also the publishers of The Lancet ) to complain. Elsevier removed the paper
from its online database pending the results of an independent investigation. The

Lancet was asked to review the paper. We did so and the reviews were uniformly and
deeply critical. No journal could have conceivably published the Duesberg paper
based on these reviews. The Duesberg paper remains retracted, excised from the
scientific literature. Here is an example of what can happen when peer review is excluded from a journal‘s processes, and why peer review can bring important
information to bear on judgments about the suitability of research for publication.
Thanks to these events, the journal will now implement peer review. The publishers
are seeking a new editor (28).



## Climate science and peer review

The events surrounding the peer review of certain climate science papers have raised
important questions about peer review. But some of these questions may be based on
a misinformed view of the peer review process. Here, I touch on general themes in
peer review that have emerged during the debate about the role of climate scientists in
research publication.

It is common for editors to have multiple, intense, and sometimes sharp interactions
with authors and reviewers. Publication matters. Authors and reviewers are frequently
passionate in their intellectual combat over a piece of research. The tone of their
exchanges and communications with editors can be attacking, accusatory, aggressive,
and even personal. If a research paper is especially controversial and word of it is
circulating in a particular scientific community, third-party scientists or critics with an
interest in the work may get to hear of it and decide to contact the journal. They might
wish to warn or encourage editors. This kind of intervention is entirely normal. It is
the task of editors to weigh up the passionate opinions of authors and reviewers, and
to reflect on the comments (and motivations) of third parties. To an onlooker, these
debates may appear as if improper pressure is being exerted on an editor. In fact, this
is the ordinary to and fro of scientific debate going on behind the public screen of
science. Occasionally, a line might be crossed. We experienced such a border crossing
recently, where several reviewers and third parties encouraged us to delay publication
of a paper for non-scientific reasons (29). Defining that line is the crucial task when
judging the role of CRU scientists.APPENDIX 5

134 One issue that is important to solve for the peer review process to work effectively is

the full disclosure of all financial and relevant non-financial conflicts of interest. If a
research paper about drug A for disease Y is sent to a reviewer who has shares in a
company that makes drug B, also for disease Y, there is a potential for the introduction of bias into that reviewer‘s advice to the journal – favouring drug B over drug A. The editor may still want and value that reviewer‘s advice, but s/he needs to know about the reviewer‘s financial conflict to judge the weight s/he gives to the
review. Non-financial conflicts may be even more important. If a scientist has devoted a life‘s work to theory A about disease Y, then clearly s/he might be biased if s/he is
sent a manuscript that criticises theory A and proposes an alternative and compelling
theory B for that same disease. Again, the editor would expect the reviewer to declare
any non-financial academic or intellectual conflicts that might have the potential to influence that reviewer‘s critique. In the field of climate research, con clusions about
the meaning of the science may have been strongly held. The fact that these
conclusions were strongly held is entirely to be expected and should simply have been
fully disclosed to editors during the peer review process.

It would be wrong for editors not to listen to advice about publication even after
acceptance of a paper. A paper is only fully accepted when it is published. New
information that informs the decision to publish a provisionally accepted paper before
publication can be very valuable. The Lancet has rejected papers in this twilight zone
of peer review. After publication, criticism is common and welcome, even lethal
criticism. This is the much vaunted self-regulation of science

– except that sometimes
editors and authors are reluctant to act when things go wrong after publication. In
climate science, disputatious research is unsurprising. It would be extremely
surprising if that research did not provoke heated exchanges both before and after
publication.

Much has been made of whether scientists should or should not take public positions
on the meaning of their data, especially if those data relate directly to policy or
practice. The reality is that they do, all the time. Science does not exist in a political
vacuum. The idea that scientists, including climate scientists, are neutral observers,
bereft of opinions, is naïve. In biomedical and public health research, scientists are
often quick to make statements applying their data to the real world. They will often
do so passionately and be well known for those passionate views. Indeed, the current
climate of science is such that scientists are encouraged at every stage of their
research to consider the impact

– economic or human

–

of what they do, and to
trumpet that impact. Research assessments in the future are likely to include a measure of impact when judging the quality of a scientist‘s work. In relation to peer
review, the scientific, policy, or political positions an author, reviewer, or editor may
hold could intervene to bias a review in one particular direction. There have been
many examples of such conflicts in other scientific disciplines

– eg, psychology (30)
and genetic epidemiology (31). These episodes are troubling, but an almost inevitable
consequence of the way peer review is ordinarily done.APPENDIX 5

135 The intersection of politics and science in climate change has arisen at least once

before. The Skeptical Environmentalist , written by Bjørn Lomborg and published by
Cambridge University Press, led to huge pressure on the publishers to withdraw the
book (32). Although the manuscript was reviewed by four experts who all
recommended publication, the scientific backlash was acute. Letters of protest were
written to newspapers. One scientist refused to work with Cambridge University Press
ever again. Lomborg was attacked physically.

Chris Harrison, in his thoughtful reflections as the editor at Cambridge University Press who dealt with Lomborg‘s book, points out that peer review ―offers no
guarantees of always ensuring the  ̳truth‘‖ (32). But in the case of The Skeptical

Environmentali st , the concerns were as much political as scientific. The publication of this book by a respected scholarly press might play ―to a particular political agenda
and can be used and abused by vested corporate and political interests.‖ Harrison
rejected the idea that he should have applied these kinds of value judgment in the
editorial process. But he notes, ruefully,

“While it may not be the responsibility of an editor to
second-guess how a publication will be received in political
circles, it is clear that politicians and advocacy groups have
always maintained a keen interest in what is published and that they have sought to influence editorial decisions.”

Harrison defended the scholarly publishing industry‘s commitment to pluralism. He

wrote,

“Given the scale of interest in the environment it is

perhaps peculiarly incumbent on the academy and
general interest intermediaries [Cambridge University Press] to host as full and as open a debate as possible...the
public and the academy can surely only be better served
by an opportunity to review and debate a wide range of perspectives.”

This commitment to pluralism would be the likely view of many scientific editors,
even when controversy follows. There are interesting parallels between the response

of t he scientific community to Lomborg‘s sceptical book and the reaction of some
scientists to journal articles on climate change which expressed an opinion contrary to
their own. One might conclude that these kinds of extreme debate, although difficult,
are part of the normal fabric of scientific discourse. The question to be answered is:
where is the line to be drawn between vigorous scientific exchange and improper
attempts to close down debate (these two positions can be remarkably close to one
another)? But one should also be conscious of what some observers have described asAPPENDIX 5

136 the ―chilling‖ effect of political controversy on science. A survey of US National
Institutes of Health scientists revealed that many engaged in self-censorship after they
found themselves the subject of political criticism for their work (33). Political
disagreement over science can shape not only the behaviour of scientists but also the
future of science itself. As Joanna Kempner has noted,

“There is a role for democratic public e ngagement in
science. The policy challenge will be to encourage this
public voice in scientific decision-making, while enabling

scientists to submit and conduct innovative studies, even when they may provoke controversy.”

Increasingly, commercial, as well as political, interests are also intervening to threaten
the integrity of peer review (34).

Peer review and publication can provoke important questions about access to data.
During the review process, reviewers may seek more information. Except in
allegations of fraud, it would be highly unusual to provide or request raw data (even
then, journals expect institutions to take responsibility for investigating the
authenticity and reliability of original data). But access to data may be sought after
publication. This is a highly contentious and unresolved issue. For example, Andrew
Montford argues that,

“the more important story in terms of the conduct of
science in this country concerns the repeated refusals of
CRU staff to release the data and code underlying their
global- temperature index...sceptics are universally of
the opinion that the scientific method requires all research material to be released to friend and foe alike...” (2)

In the field of medicine, these issues are currently the subject of much disagreement.
While many parties might like to see greater sharing of data, this practice remains
unusual. The Wellcome Trust is taking an especially strong interest in data access. It proposes a code of conduct calling for ―maximum public access t o data of public health importance.‖ The very fact that this proposal is being made illustrates the point

that routine access to data is not a settled issue or a universal norm in science, as some

claim.

The issue of retention of records and exclusion of data in climate research is also a
matter relevant to the peer review process and the ordinary working of journals.
Journals do expect records to be kept for limited periods (say, 5 years, although
journal practices vary). And they are comfortable with the exclusion of data provided

that those exclusions

–

and the reasons for exclusion

– are fully described, with
appropriate sensitivity analyses being completed where necessary.APPENDIX 5

137 Two additional dimensions of peer review have been raised in the debate over C RU‘s
work. One relates to confidentiality, the other to uncertainty. Editors send manuscripts
to reviewers based on a principle of confidentiality. The author expects the editor to
maintain a covenant of trust between the two parties. The editor will not misuse the author‘s work by circulating it outside of the confidential peer review process. The
editor expects that covenant of trust to be honoured by the peer reviewer. No
manuscript should be passed to a third party by a reviewer without the permission o f
the editor, usually on the grounds of improving the quality of the critique of the
manuscript by involving a colleague in the review process. A disclosure to a third
party without the prior permission of the editor would be a serious violation of the
peer review process

–

a breach of confidentiality. It is also of paramount importance
to report fully in all published scientific papers both quantitative and qualitative
measures of uncertainty. One of the main benefits of peer review is to focus on areas

o f potential uncertainty and to ensure that those uncertainties are fully acknowledged,
measured, and reported.



## The future of peer review

Peer review is a human process and so will always contain flaws, produce errors, and
occasionally mislead. Given that journals are the gatekeepers of scientific publication,
they have enormous

– probably too much

– influence over the reputations of
scientists, research units, and universities. Many measures of academic success
depend upon journal publication

– promotion, tenure, grants, fame, and personal
wealth. It is not surprising that journals, and the main decision aid used by journals
(peer review), are the subject of constant tension and occasionally explosive
controversy. At such moments, it is not only essential to be clear (and modest) about
what peer review can do, but also to look for opportunities to do better.

Journal articles are highly stylised reports of research. The linear and logical style of
the research they report rarely presents a true or accurate picture of how a piece of research was done. As the Nobel laureate, Peter Medawar, put it (35) in his essay ―Is
the scientific paper a fraud?‖ (to which he answered that it was),

“[the scientific paper] misrepresents the processes
of thought that accompanied or give rise to the work

that is described in the paper...The scientific paper in
its orthodox form does embody a totally mistaken
conception, even a travesty, of the nature of scientific thought.”

Medawar‘s point was that, ―There is no such thing as unprejudiced observations.‖ To
add insult to injury, research papers may not even fully represent the views of theAPPENDIX 5

138 authors who completed the work (36), and when faults are found after publication
those faults may be completely ignored in the subsequent use of that research (37).

There are actions that the scientific community could take to improve this far from
happy state of affairs surrounding one its foundational processes. First, there are new
opportunities and techniques available to search out, identify, and eliminate (or at
least reduce) unwanted bias in the peer review process (38, 39). Second, all young
scientists should receive formal training

– which they currently do not

– in the
standards and ethics expected in the peer review process (40). It is scandalous that
peer review is simply not taken as seriously as it should be in the training of scientists.
The result is that peer review is often idiosyncratic and sometimes unreliable, fuelling

sc ientific controversies, such as that over climate science, rather than defusing those
controversies. Strengthening the training, standards, and expectations around peer
review would do much to make the quality of peer reviewing part of the formal
appraisal of a scientist‘s contribution to his or her subject. There is a demand for
training in peer review (41). And the ethical dimensions of the review process are
now sufficiently concerning to scientists that they merit training as much as the more
formal methodological aspects of reviewing (42). Disappointingly, existing training
packages in peer review deliver little benefit to the quality of the peer review process
(43-45).

Third, the peer review process is enormously inefficient. Individual journals will
undertake peer review and reject manuscripts that will then cycle around other
journals until either the paper is accepted or the authors are sufficiently exhausted that
they abandon attempts at publication. In the face of such gross inefficiencies, some
scientific communities have tried to bring journals together to cooperate and make the
review process not only more efficient, but also less costly on the time and energy of
reviewers, authors, and editors (46). Alternatively, there may be intra-journal
procedures that can be introduced to deliver more efficient peer review (47).

Fourth, journal editors should adopt more effective methods to resolve disputes
between authors, reviewers, and readers. Within the journal, an ombudsperson
operating independently of the editors can be one useful way to resolve intractable
disagreements about journal processes (48). If a dispute remains impossible to
resolve, journal editors can take their concerns to the Committee on Publication
Ethics, a charity that aims to set standards for journal practices, including peer review.
Journal editors should consider using this facility more often than they currently do

– in some ways, it represents the collective wisdom of a wide range of journal editors, a

collective wisdom that any scientific editor can draw upon in times of crisis.

Lastly, peer review should be a subject for research in its own right. Although there is
a small group of scientists who study peer review (a biomedical peer review congress
is held every 4 years), that community is extraordinarily fragile when measured
against the size and importance of the contribution peer review makes to science (49).APPENDIX 5

139 Historically, science funding bodies have been reluctant to invest in research on peer
review. This reluctance is partly responsible for the present vacuum in our knowledge
about the way scientific knowledge is constructed, reported, and discussed. One
positive result of the debate over the role of CRU scientists in peer review might be to
encourage funding bodies

– such as the Medical Research Council and the National

Institute for Health Research

– to take the science of peer review far more seriously.

Journals have inevitable limitations. When a paper with important policy implications
is considered, editors can ask authors to balance their conclusions by putting the work
in the context of existing evidence. Or we can commission an editorial that does the
same. But a journal cannot adjudicate a public debate, and neither can conventional
peer review. For those occasions when science meets (or clashes with) policy, there
may be a case for referring the area of controversy to an independent body for a
public inquiry. In the US, the model used is the Institute of Medicine, which tackles
controversial aspects of health through its thorough and wide-ranging investigations
and reports. There may be a case for such a body in the UK.

The best one might hope for the future of peer review is to be able to foster an
environment of continuous critique of research papers before and after publication.
Many writers on peer review have made such a proposal, yet no journal has been able
to create the motivation or incentives among scientists to engage in permanent peer
review (50-52). Some observers might worry that extending opportunities for
criticism will only sustain maverick points-of-view. However, experience suggests
that the best science would survive such intensified peer review, while the worst
would find its deserved place at the margins of knowledge.

This process of weeding out weak research from the scientific literature can be
accelerated through more formal mechanisms, such as the systematic review. A
systematic approach to selecting evidence focuses on the quality of scientific methods
rather than the reputations of scientists and their institutions. This more rigorous
approach to gathering, appraising, and summing up the totality of available evidence
has been profoundly valuable to clinical medicine. There may be useful lessons here
for the IPCC. Climate sceptics and climate scientists, along with their colleagues in
other scientific disciplines, would likely welcome this greater rigour and scrutiny. It
would certainly promote quality and strengthen accountability to a more critical
public (and media) with higher expectations of science. More importantly, intensified
post as well as pre publication review would put uncertainty

– its extent and

boundaries

–

at the centre of the peer review and publication process. This new
emphasis on uncertainty would limit the rhetorical power of the scientific paper (53),
and offer an opportunity to make continuous but constructive public criticism of

research a new norm of science.

Finally, there are more general institutional lessons to be learned from the events that
took place at the CRU. Peer review cannot eliminate controversy. Leaders of researchAPPENDIX 5

140 organisations

–

from University Vice-Chancellors to Unit Directors

– might consider
a risk assessment of the research being done under their stewardship. Mapping
research in this way

–

who is doing what, where, and how?

– would provide the
institution with signals of potential controversy if the work addressed an issue of
substantive public, policy, or political interest. This kind of awareness might enable

scientists

–

and, indeed, the institution itself

– to prepare better for any likely public
disputes. Vigilance over the research being undertaken at a university could usefully
anticipate difficulties, promote best practices, protect reputations, and build wider
public trust and confidence in the research process. These outcomes, together with
more realistic expectations about peer review and the processes of science, would be
valuable consequences of the experiences at the University of East Anglia.

1. Montford A. The Hockey Stick Illusion : Climategate and the corruption of

science (Stacey International, 2010)

2 . Montford A. Heated discussions. Times Higher Education March 25, 2010:

43 -44.

3. Monbiot G. Our narrow, antiquated school system is at the root of the climate

email fiasco. The Guardian April 6, 2010: 25.

4. Booker C. Climategate: a scandal that won‘t go away. The Sunday Telegraph April 18, 2010: 29.

5. House of Commons Science and Technology Committee. The disclosure of climate data from the Climate Research Unit at the University of East Anglia (Stationery Office, 2010).

6. This quotation is taken from an email from Phil Jones to Michael Mann.

7. Spier R. The history of the peer review process. Trends in Biotechnology 2002; 20: 357-58.

8. Hernon P, Schwartz C. Peer review revisited. Library and Information Science

Research 2006; 28: 1-3.

9. Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of Internal Medicine

. Ann Intern

Med 1994; 121: 11 -21.

10. Smith JS, Green J, de Gonzalez AB, et al. Cervical cancer and use of hormonal contraceptives: a systematic review. Lancet 2003; 361 : 1159-67.

11. Jun M, Foote C, Lv J, et al. Effects of fibrates on cardiovascular outcomes: a systematic review and meta-analysis. Lancet 2010

12. Green SM, Callaham ML. Current status of peer review at Annals of Emergency Medicine. Ann Emerg Med 2006; 48: 304 -08.APPENDIX 5

141 13. Burnham JC. The evolution of editorial peer review. JAMA 1990; 263: 1323

- 9.

14. Fitzpatrick JJ. Peer review: a 2008 report on the sacred academic cow. Applied Nursing Research 2008; 21: 53.

15. Harms M. Peer review: the firewall of science. Physiotherapy 2006; 92: 193-

94.

16. DeMaria AN. Peer review: the weakest link. JACC 2010; 55: 1161-62.

17. Berger E. Peer review: a castle built on sand or the bedrock of scientific publishing? Ann Emerg Med 2006; 47: 157 -59.

18. Balistreri WF. Landmark, landmine, or landfill? The role of peer review in assessing manuscripts. J Pediatr 2007; 151: 107 -08.

19. Mulligan A. Is peer review in crisis? Oral Oncology 2005; 41: 135-41.

20. Smith R. Peer review: a flawed process at the heart of science and journals. J

R Soc Med 2006; 99 : 178-82.

21. Jefferson T, Alderson P, Wager E, Davidoff F. The effects of editorial peer

review. JAMA 2002; 287: 2784 -86.

22. Jefferson T, Rudin M, Brodney Folse S, Davidoff F. Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database Syst Rev 2007; 2 : MR16.

23. Richards D. Little evidence to support the use of editorial peer review to ensure quality of published research. Evid Based Dent 2007; 8 : 88-89.

24. Wager E, Middleton P. Technical editing of research reports in biomedical journals. Cochrane Database Syst Rev 2008; 4: MR2.

25. Jefferson T, Wager E, Davidoff F. Measuring the quality of editorial peer

review. JAMA 2002; 287: 2786 -90.

26. Aarssen LW, Lortie CJ, Budden AE, et al. Does publication in top-tier journals affect reviewer behaviour? PLoS ONE 2009; 4 : e6283.

27. Kravitz RL, Franks P, Feldman MD, et al. Editorial peer reviewers‘ recommendations at a general medical journal: are they reliable and do editors care? PLoS ONE 2010; 5 : e10072.

28. Enserink M. Elsevier to Editor: change controversial journal or resign. Science 2010; 327 : 1316.

29. Horton R. Maternal mortality: surprise, hope, and urgent action. Lancet 2010;APPENDIX 5

142 30. McCarty R. Science, po litics, and peer review: an editor‘s dilemma. American Psychologist 2002; 57: 198 -201.

31. Calnan M, Smith GD, Sterne JA. The publication process itself was the major cause of publication bias in genetic epidemiology. J Clin Epidemiol 2006; 59:

1312 -18.

32. Harrison C. Peer review, politics, and pluralism. Environmental Science and Policy 2004; 7: 357 -68.

33. Kempner J. The chilling effect: how do researchers react to controversy. PLoS Medicine 2008; 5 : e222.

34. Curfman GD, Morrissey S, Annas GJ, Drazen JM. Peer review in the balance.

N Engl J Med 2008; 358 : 2276-77.

35. Medawar PB. The Strange Case of the Spotted Mice (Oxford, 1996).

36. Horton R. The hidden research paper. JAMA 2002; 287 : 2775-78.

37. Horton R. Postpublication criticism and the shaping of knowledge. JAMA 2002; 287 : 2843-47.

38. Bornmann L, Mutz R, Daniel H-D. How to detect indications of potential sources of bias in peer review. J Informatics 2008; 2 : 280-87.

39. Ross JS, Gross CP, Desai MM, et al. Effect of blinded peer review on abstract
acceptance. JAMA 2006; 295 : 1675-80.

40. Walbot V. Are we training pit bulls to review our manuscripts? J Biol 2009; 8 : 24.

41. Snell L, Spencer J. Reviewers‘ perceptions of the peer review proces s for a medical education journal. Med Educ 2005; 39 : 90-97.

42. Resnik DB, Gutierrez-Ford C, Peddada S. Perceptions of ethical problems with scientific journal peer review. Sci Eng Ethics 2008; 14 : 305-10.

43. Schroter S, Black N, Evans S, et al. Effects of training on quality of peer

review. BMJ 2004 doi: 10.11136/bmj.38023.700775.AE.

44. Callaham ML, Tercier J. The relationship of previous training and experience of journal peer reviewers to subsequent review quality. PLoS Medicine 2007;

4 : e40.

45. Schroter S, Black N, Evans S, et al. What errors do peer reviewers detect, and does training improve their ability to detect them? J R Soc Med 2008: 101 : 507 -14.APPENDIX 5

143 46. Saper CB, Maunsell JHR. The Neuroscience Peer Review Consortium. Brain

Res 2009; 1272 : 1-2.

47. Johnston SC, Lowenstein DH, Ferriero DM, et al. Early editorial manuscript screening versus obligate peer review. Ann Neurol 2007; 61: A10-12.

48. Horton R. The journal ombudsperson: a step toward scientific press oversight.

JAMA 1998; 280 : 298-99.

49. Linkov F, Lovalekar M, LaPorte R. Scientific journals are ―faith based‖: is there science behind peer review? J R Soc Med 2006; 99 : 596-98.

50. von Segesser LK. Peer review versus public review

–new possibilities of online publishing! Interactive Cardiovascular and Thoracic Surgery 2002; 1:

61 -62.

51. Mandviwalla M, Patnayakuni R, Schuff D. Improving the peer review process with information technology. Decision Support Systems 2008; 46: 29 -40.

52. Liesegang TJ. Peer review should continue after publication. Am J Ophthalmol 2010; March; 149 : 359-60.

53. Horton R. The rhetoric of research. BMJ 1995; 310 : 985-88.



## Commentary on „ Understanding uncertainty:a brief history of peer review’

Elizabeth Wager, Publications Consultant Chair of the Committee on Publication Ethics (COPE)

Richard Horton suggests that  ̳ If evidence has not been peer-reviewed, it is next to worthless‘. Earlier commentators have gone even further. The evolutionary biologist

Edward Wilson commented that  ̳ a discovery does not exist until it is safely reviewed and in print‘. 1

Peer review has been used to select articles for publication since the 17 th century. It is likely that when Oldenburg was using peer review at London‘s Royal Society, his contemporaries at the Royal Society in Edinburgh and across the channel at the Académie Royale de Médicine were using similar systems. 2

Interestingly, the inability of peer review to guarantee scientific integrity was noted as early as 1785 when the Committee of Papers of the Literary and Philosophical Society of Manchester noted that  ̳Responsibility concerning the truth of facts, the soundness of reasoning ... [and] the accuracy of calculations is wholly disclaimed; and must rest alone, on the knowledge, judgement, or ability of th e authors‘. 3

Horton describes the peer review system used at The Lancet . This is typical of major medical journals that employ several full-time editors. Such journals reject aAPPENDIX 5

144 considerable proportion of submissions on the basis of in-house review alone. For example, at JAMA (the journal of the American Medical Association) only 36% of submissions are sent for external review and the remaining 64% are rejected after review by the journal editors alone. 4 At journals such as The Lancet and JAMA

, everything accepted for publication will have been reviewed by experts who are independent of the journal, but many articles will be rejected without review byexternal experts. In contrast, journals that do not have large editorial offices or fulltime editors, but whose editors are academics (generally fulfilling this role in addition to their regular job), generally send virtually all submissions for external review.

The Lancet generally sends papers to three reviewers. Other journals may use fewer, or occasionally more, reviewers. A survey of 200 journals from a range of disciplines found that 73% used two reviewers, 18% used three, 6% used one and 3% used more

than three. 5

Horton notes that  ̳ the scientific literature is littered with retractions of papers that

on ce passed the test of peer review‘. The biomedical database Medline (which includes over 19 million citations) currently contains nearly 1500 retractions. There have also been well-documented cases of journals failing to recognise important work. There is even a website devoted to accounts of journals that have rejected work that later led to their authors winning the Nobel prize. [ttp://www2.uah.es/jmc/nobel/nobel.html#reje].

Churchill‘s famous observation about democracy has often been applied to peer

re view, namely that it is the worst system for selecting papers except for all the other forms that have been tried. Horton mentions the possibility of alternative systems such as post-publication or public review. A few journals have experimented with these systems but found them unworkable. When Nature ran a trial of open peer review in 2006 (by posting submitted papers on its website), almost half the papers

received no comments. 6

While formal evidence for the effectiveness of peer review is lacking, there is

extensive anecdotal evidence from both editors and authors of its benefits. Editors and publishers should strive to optimise their peer review systems and to reduce potential harm or bias. Guidance is available from several organizations including the Committee on Publication Ethics (www.publicationethics.org).

While the internet has not produced dramatic new ways of assessing articles submitted to journals, information technology does offer possibilities which could increase the efficiency of disseminating and verifying scientific data. Gene sequence data is now routinely entered onto public databases, and such data posting is a requirement for publication in traditional journals in some cases. However, before such posting could be practical, even for such homogeneous data as gene sequences, agreements had to be developed (e.g. MIAME). 7 Such standardization may be possible for other types of data yet, as Horton notes, in many disciplines, data sharing remains either controversial, or limited by technological, practical or ethical issues. 8

Even with increased research and investment it seems unlikely that, despite its shortcomings, conventional peer review will be replaced or transformed in the next few years. In the meantime, researchers will probably continue to criticise the peerAPPENDIX 5

145 review system, sometimes with justification (because the system is flawed) and sometimes without (because researchers are human, feel strongly about their work and do not accept that some rejections are justified). The research community should listen to the criticisms of peer review and work towards fixing the remediable faults and correcting the inevitable errors. Editors and publishers should continue to work hard to prevent and detect misconduct by authors, reviewers and editors and should have systems in place for responding appropriately when misconduct occurs. However, judgement must be exercised to ensure that resources are wisely spent and care should be taken to avoid overzealous responses to individual cases or disproportionate reactions to rare problems.



## References

1. Wilson EO. Consilience: The Unity of Knowledge. Knopf, New York, 1998

2. Wager E. Peer review in science journals: past, present and future. In Holliman R et

al (eds) Practising Science Communication in the Information Age . Oxford University

Press, Oxford, 2009

3. Kronick DA. Peer review in 18 th -century scientific journalism. Journal of the

American Medical Association 1990;263:1321-2

4. Fontanarosa P, DeAngelis C. To JAMA peer reviewers and authors

– thank you. Journal of the American Medical Association 2010;303:780

5. Current Practice in Peer Review . ALPSP, EASE, ALSSS. www.alpsp.org

6. Anon Overview: Nature‘s peer revew trial, 2006 www.nature.com/nature/peerreview/debate/nature05535.html

7. Brazma A. Minimum information about a microarray experiment (MIAME)

-- toward standards for microarray data. Nature Genetics 2001;29:365-71

8. Wager E. Publishing clinical trial results: the future beckons.

PLoS Clinical Trials 2006;1:e31



## Competing interests statement

Elizabeth Wager is a freelance publications consultant who provides consulting, training, and editorial services mainly to pharmaceutical companies, publishers and academic institutions. She chairs the Committee on Publication Ethics (but has written this commentary in her individual capacity). She was part of the team that produced systematic reviews on the effects of peer review and technical editing cited by Richard Horton. She received a fee for producing this commentary.APPENDIX 6

146 

## APPENDIX 6 : DATA MINING - ACCESS TO THE

ORIGINAL CRU E-MAIL ARCHIVE

1. Recognising that the e-mails improperly released into the public domain represent only a tiny fraction (less than 0.3%) of the e-mails archived by the key individuals in the CRU, the Review team sought to set these in context. The backup server (CRUBACK3) had been taken as evidence by the police as part of their own investigation and was held by police contracted forensic investigators. A full context could only be established by some form of access to the information held on this server. In seeking to gain this access a number of legal issues arose, notably that:

the server and its contents were evidence in the continuing police investigation; and in the opinion of UEA‘s legal advisers, unconstrained access to the contents of e-mails on the server by th e Review would raise potential privacy and data protection issues.

2. The compromise eventually reached with both the police investigative team and the UEA Registrar was for:

the University to contract an independent computer forensic analyst; the police fo rensic consultants to extract from CRUBACK3 all the e-mails from the various archived mailboxes of key UEA staff and to provide these under strict security conditions to the independent forensic analyst; the independent analyst, respecting the high evidential security requirements set out by the police team, to work within secure premises authorized by the police; the independent analyst to seek to determine the search or selection parameters that had extracted the improperly released e-mails from the multiple archives and to determine whether this process highlighted any additional material of

relevance to the Review ; and any material identified by the the analyst to be redacted by the University, in terms of protecting the identity of non-UEA recipients or authors, prior to being made available to the Review.

3. This whole process took an extended period to negotiate and implement. It became clear that a full analysis would require considerable further time and extensive manual intervention. It would introduce significant delay to the publication of the Review ‘s report. A decision was reached not to pursue this further on grounds of both time and cost against likely results. The Review had always regarded the e-mails as pointers to areas for detailed investigation and this had been complemented by extensive public requests for submissions and any other information in the public domain. A summary report by the independent forensic analyst has been placed on the Review website.APPENDIX 6

147 4. The Review ‘s own analysis of t hose e-mails already in the public domain is highlighted in the following charts, including a  ̳heat map‘ of correspondence intensity by month.

5. The selected e-mails relate largely to controversial issues, although they do not appear to have been selected on the basis of a simple word search, as indicated in the table below which shows the number of e-mails in which interesting words (or a component thereof) occur. Some of the more infamous e-mails contain words such as trick or hide and these occur comparatively infrequently:-APPENDIX 6

148 6. The periods of greatest intensity generally coincide with repeated selection of embedded e-mails having the same core title

– with 4 or more core title repetitions being viewed as significant in the table below.

Date Cor e Title Repeats Subject

Oct

- 98 climate of the last millennia... 4 Meeting to discuss multi

- proxy data
prompted by Jones 1998 1

May

- 99 Straight to the Point 4 Perceived criticism of Briffa 1999 2

Sep

- 99 IPCC revisions 7 Making the case for tree ring proxies in IPCC TAR Chapter 2 Apr

- 02 Your letter to Science 5 Criticism of draft letter Mann 2002 3

Mar

- 03 Soon & Baliunas 9 Criticism of Soon 2003 4

Jun

- 03 Review

- confidential 4 Validity of proxy

- based temperature

reconstruction

Oct

- 03 draft 4 of proposed EOS response to Soon 2003 4

Feb

- 04 Stephen McIntyre 4 Requesting Russian tree ring data

Jan

- 05 [Wg1

- ar4

- ch06] IPCC last 2000 years

data 5 Drafting of IPCC chapter

Apr

- 05 WG1 LA2 meeting

- Overlap cluster

A 5 Coordination between IPCC Lead Authors

Jul

- 05 MWP figure 5 for IPCC report

Dec

- 05 HadCRUT2v 4 Missing Southern Pole data

Feb

- 06 some figures at last! 4 Proxy reconstruction for IPCC AR4

Mar

- 06 latest draft of 2000

- year section text 7 IPCC Chapter 6 in the SOD for the AR4 May

- 06 Wahl & Ammann paper 4 Use of Wahl 2006 5 in the SOD for the AR4

Jul

- 06 Special instructions/timing adjustment 4 Proxy reconstruction for IPCC Sep

- 06 No Subject 4 Significance of recent temperatures in IPCC

report

Jan

- 07 not so fast 4 J. Eddy figure in IPCC report Apr

- 07 FYI 4 Climate Audit attacks on Wang and Jones

Dec

- 07 sorry to take your time up, but really

do need a scrub of this singer/christy/etc effort 11 Response to Douglas 2007 6

Jan

- 08 Update on response to Douglass et al. 5 Publication options to rebut Douglas 2007 6

May

- 08 EA 21389

- Probabilistic information to inform EA decision making on climate change impacts

- PCC(08)01 4 Tender pack for Environment Agency bid

May

- 08 JOC

- 08

- 0098

- International Journal of Climatology 4 Review of Santer 2008 7

May

- 08 Our d3* test 4 Validation of climate models

Jan

- 09 Temperatures in 2009 5 Prognosis for 2009 and UHI effect

Jun

- 09 2009JD012960 (Editor

- Steve
Ghan):Decision Letter 4 Review of McLean 2009 8

Jul

- 09 ENSO blamed over warming

- paper
in JGR 14 Response to McLean 2009 8

Sep

- 09 attacks against Keith 5 Selection of tree

- ring data

Oct

- 09 BBC U

- turn on climate 10 Recent 'lack' of warmingAPPENDIX 6

149 

## References

1. P.D. Jones, K.R. Briffa, T.P. Barnett and S.F.B. Tett, ― High-resolution palaeoclimatic records for the last millennium: interpretation, integration and comparison with General Circulation Model control-run temperatures ‖

, The

Holocene 8, 4 (1998), pp. 455

– 471

2. KR. Briffa and T.J. Osborn, ― Seeing the wood for the trees ‖

, Science Vol. 284, 7 May 1999, pp. 926-9 27

3. M.E. Mann and M.K. Hughes, ― Tree-ring chronologies and climate variability ‖

, Science , Volume 296, Issue 5569, 3 May 2002, pp. 848-849,

4. W. Soon, S. Baliunas, ― Proxy climatic and environmental changes of the past 1000 years ‖

, Climate Research Volume 23: 89 -110, January 2003

5. E.R. Wahl, D.M. Ritson, C.M. Ammann, ― Reconstructing Past Climate from Noisy Data ‖

, Science 312, 529 (2006)

6. D.H. Douglass, J.R. Christy, B.D. Pearson and S.F. Singer, ― A comparison of tropical temperature trends with model predictions ‖

, International Journal of Climatology , (2007)

7. B.D. Santer, P.W. Thorne, L. Haimberger, K.E. Taylor, T.M.L. Wigley, J.R. Lanzante, S. Solomon, M. Free, P.J. Gleckler, P.D. Jones, T.R. Karl, S.A. Klein, C. Mears, D. Nychka, G.A. Schmidt, S.C. Sherwood, and F.J. Wentz,

― Consistency of modelled and observed temperature trends in the tropical troposphere ‖

, International Journal of Climatology, 28: 1703

– 1722 (2008)

8. J.D. McLean, C.R. de Freitas, and R.M. Carter, ― Influence of the Southern Oscillation on tropospheric temperature ‖

, Journal of Geophysical Research , Vol 14, D14104 (2009)APPENDIX 7

150 

## APPENDIX 7: LAND STATION TEMPERATURE DATA

1. Contents

This Appendix supports Chapter 6 of the report addressing land temperature data. It

contains further information on two issues:

the Trial Analysis, referred to in Sections 6.3 and 6.4 of Chapter 6, and the particular issue of matching station identifiers, referred to in Section 6.5.1 of Chapter 6.



## 2. The Trial Analysis

2.1 Introduction

The Review has conducted a trial analysis to demonstrate what an independent researcher is able to do, using publicly available land station temperature information, should they wish to replicate the CRUTEM analysis. We have only carried out each step as far as necessary to demonstrate proof of principle and to compare results at a level sufficient to inform our findings in respect of allegations made against CRU. It is important to note that we have not carried out all of the steps with the necessary degree of scientific rigour needed to form scientific conclusions, and we emphasise that no such conclusions should be drawn from this

work.

We have performed the following steps:

1. Obtained raw station temperature data from two repositories as ASCII files.

2. Written computer code in C++ to read this data into memory in a suitable

format.

3. Written computer code to process this data and create gridded global average temperature trends.

4. Compared the results to those of CRU and others in order to inform our findings in respect of the allegations made against CRU.

We have not carried out the full process of homogenising all data. There is no fundamental barrier to doing so. It is a time consuming step and requires the experience and expertise of a climate research scientist to process each of the raw station data entries before averaging. We comment upon the significance of homogenisation below.



## 2.2 Primary Station Data

Primary station data was obtained from: NCAR:http://www.ncar.ucar.edu/tools/datasets/ WWR: http://dss.ucar.edu/datasets/ds570.0/ GHCN: http://www.ncdc.noaa.gov/oa/climate/ghcn-monthly/index.php.APPENDIX 7

151 The GHCN data was freely available. GHCN provided several files including a list of stations and locations, a data set of unadjusted data, and a dataset of adjusted data. This contained 7280 stations in the unadjusted file and 4771 in the adjusted file. This data set contained in some cases several duplicates from the same station.

For NCAR it was necessary to register (free) first. The NCAR data was a single file containing all necessary information and contained 4646 stations.

The formats of each file were easy to understand. The formats were different from

each other.



## 2.3 Code to Process the Data

Code was written in C++ to:

Parse the sou rce files and assemble the information into computer memory. Pass over each station and calculate the monthly ―normals‖ defined as the average over a set period. In this case we required good data from at least 10 years in the period 1955-1995. This is larger than was used by CRUTEM in order to obtain more valid normals without having to resort to external

records. In the case of GHCN, choose the duplicate with the most valid monthly

normals. Calculate the anomaly for each monthly measurement for each station. The anomaly is defined as the monthly temperature minus the normal for that

month. Either choose to use all stations, or only those matched to the CRUTEM3 station list published in 2009. Pass over the data set and assemble average annual anomalies in each cell of a 5x5 degree grid of latitude and longitude. Calculate the overall annual average anomaly by averaging over grid cells. Do this for each of GHCN (unadjusted), GHCN (adjusted) and NCAR, and plot the annual average anomaly as a function of time. Separately read the published CRUTEM3 product and form the same average over grid cells and plot this on the same figure for comparison.

This code was straightforward to write from scratch. The minimal code needed to read and process the data sets amounts to only a few hundred executable lines and took about 2 days to develop.



## 2.4 Results of Global Temperature Trends

Figure A1 - A4 show the comparison of the time series obtained from the three data sets GHCN (unadjusted), GHCN (adjusted) and NCAR. A simple 5 year smoothing is applied. Also shown is the result obtained from the CRUTEM3 5x5 gridded product.APPENDIX 7

152 The set of input stations used for each line is not identical - the numbers are GHCN (unadjusted) [7280], GHCN (adjusted)[4771], NCAR [4646]. Of these, all stations

wh ich have a valid monthly normal and valid monthly data are included in Fig A1.

In Fig A2 we have approximately matched the stations in each dataset to those used in

CRUTEM3. This is described in more detail below. The numbers matched are GHCN (unadjusted) [3722], GHCN (adjusted) [2962], NCAR [1957]. CRUTEM itself used

4138 stations.

Figure A3 is the same as A2 but showing only the two lines which correspond to unadjusted data (GHCN and NCAR).

Figure A4 is the same as A2 but showing only the two lines which correspond to adjusted data (GHCN (adjusted) and CRUTEM).

Figure A5 reproduces a comparison of different studies taken from the IPCC 4 th
Report Chapter 3.

Figure A1: Temperature anomaly time series created by the Inquiry Team‘s own trial analysis using a 5x5 grid with 5 year smoothing. Shown are results obtained from GHCN (blue)

, GHCN-adjusted (yellow) and NCAR (green) . Also shown is the CRUTEM3 line (black). The Y-axis is degrees x 10. The X-axis is year.APPENDIX 7

153 Figure A2: Same as Fig A1 but with approximate match of stations to those used in CRUTEM3. Axes same as Fig A1.

Figure A3: Same as Fig A2 but showing only the unadjusted lines (GHCN and NCAR) Axes same as Fig A1.

Figure A4: Same as Fig A2 but showing only the adjusted lines (GHCN (adjusted) and CRUTEM). Axes same as Fig A1.APPENDIX 7

154 Figure A5: Taken from IPCC 4 th Report Chapter 3.

The exercise and comparison of all figures demonstrates that:

1. Any independent researcher may freely obtain the primary station data. It is impossible for any group to withhold data.

2. It is impossible for any group to tamper improperly with data unless they have done so to the GHCN and NCAR (and presumably the NMO) sources

themselves.

3. The steps needed to create a temperature trend are straightforward to implement.

4. The computer code necessary is straightforward to write from scratch and could easily be done by any competent programmer.

5. The shape obtained in all cases is very similar: in other words if one does the same thing with the same data one gets very similar results.

6. The result does not depend significantly on the exact list of stations.

7. Adjustments make little difference.

By performing this simple test one determines easily that the results of the CRUTEM analysis follow directly from the published description of the method, and that the resultant temperature trend is not significantly different from the other results regardless of stations used or adjustments made. The test is therefore sufficient to demonstrate that, with respect to the declared method, the CRUTEM analysis does not contain either error or adjustments which are responsible for the shape of the resultant temperature trend.APPENDIX 7

155 A researcher can evidently produce a study which would test the CRUTEM analysis quite precisely, without let or hindrance from CRU.

We do not make any statement regarding the correctness of any of these results in representing global temperate trends. We do not address any alleged deficiencies such as allowance for non climatic effects or the significant drop in station number post 1991. We do not address any possible deficiencies of the gridding method. These are entirely matters for proper scientific study and debate and lie outside the scope of

this Review.



## 2.5 Discussion of Adjustments (Homogenisation)

In our trial analysis the GHCN and NCAR lines are without homogenisation adjustment. The CRUTEM3 and GHCN (adjusted) lines are both after homogenisation adjustments. The adjustments to CRUTEM and GHCN are made in different ways by different people. Even so the results are extremely similar. It is evident from this comparison alone that neither CRU nor those assembling the GHCN (adjusted) dataset have made inappropriate adjustments in order to bias or falsify the global temperature trend simply because those adjustments have no significant effect on the global average. Nevertheless we consider additional evidence that the adjustments made have no significant effect on the global temperature trend.

In the CRUTEM3 product the homogenisation adjustments are (i) made to only 298 (10%) of the stations and (ii) are known to be approximately evenly distributed about zero. Thus they would not be expected to have a large effect in the global average.

CRU have submitted detailed evidence of this (p49 and p54 of their submission) reproduced here as figure A6 and A7. Figure A6 shows a comparison of CRUTEM3 with the results obtained by (i) not applying adjustments to the 298 stations and (ii) leaving out the 298 adjusted stations. There is no significant difference as expected. Figure A7 shows the CRUTEM3 analysis compared to unadjusted results from other groups. Again no significant difference is seen. This is fully in accord with the results of the trial analysis.

Comparison of all of these plots demonstrates that adjustments which have been made are largely immaterial to the overall shape of the temperature series and are not on their own the cause of the rising temperature trend in recent years.

We have not addressed the opposite question of whether the adjustments made are adequate to allow for non climatic effects. This is entirely a matter for proper scientific study and debate and outside the scope of this review.APPENDIX 7

156 FIGURE A6 CRUTEM3: Removing AdjustmentsAPPENDIX 7

157 FIGURE A7 CRUTEM3: Comparison with unadjusted results from other groups



## 3. Matching Station Identifiers

We have used the information supplied by CRU from its website http://www.cru.uea.ac.uk/cru/data/landstations/. This lists the 4138 stations used in CRUTEM3. We have attempted to match these stations to those appearing the GHCN

and NCAR data sets.

This process is not straightforward (at least was not straightforward to us as nonAPPENDIX 7

158 climate scientists). Problems arose through Lack of uniqueness of station identification numbers; Different identifiers appearing for the same station in some cases; Different station names in a few cases (spelling, order of words); Locations differing by a few degrees;
Different conventions for location; Ambiguity due to duplicates in GHCN

We found it impossible to match stations purely from identifiers. Instead we used a combination of conjunctions of matching latitude and longitude, the first letters of station names, and station identifiers. We were in the end able to match the numbers

shown in the table.

Number in CRUTEM3 Matched in GHCN Matched in NCAR

4138 37 22 (90%) 1957 (47%)

The number matched in GHCN (90%) is in accord with that claimed by CRU in their written responses. We have presumed that the remaining data is available from National Meteorological Offices or as part of more recent update messages.

The Review feels that it has pursued this enough to understand that (i) the matching of meta data (lists of stations) across sources is not straightforward and (ii) obtaining the final 10% of the primary data would require some additional effort (although anyone familiar with the field would presumably have a better knowledge of how to do this). This is a secondary issue to this Review, but we make the following observations

(i) It would benefit the global climate research community if a standardised way of defining station metadata and station data could be agreed, preferably through a standards body, or perhaps the WMO. As example an xml based format which would make the interpretation, use, comparison, and exchange of data much more straightforward.

(ii) Without such standardisation there will remain residual problems in issuing unambiguous lists, and assembling primary data from them. We feel it would be in the public interest if CRU and other such groups developed a process to capture and publish a snapshot of the data used for each important publication.APPENDIX 7

159APPENDIX 8

160 

## APPENDIX 8 : SOFTWARE AND DATA STANDARDS

Software Standards - The available guidance on software development methodologies is very extensive ranging from  ̳formal methods‘, where the desired properties of the system are expressed in a mathematically formal language, through to the  ̳Unified Process‘ developments of this decade. The UK Royal Academy of Engineering has published extensively in this area 1

.

Data Management and Archiving Guidance is also available on Records Management and archiving from: the Joint Information Systems Committee (JISC) of the Higher Education Funding Council 2 ; the UK Research Integrity Office (UKRIO) 3 ; and the US National Academy of Sciences report ―Ensuring the Integrity, Accessibility and Stewardship of Research Data in the Digital Age‖ 4 . We note in particular Recommendation 2: “Research institutions should ensure that every researcher receives appropriate training in the responsible conduct of research, including the proper management of research data in general and within the researcher‟s field of specialization. Some research sponsors provide support for this training and for the development of training programs”. We also note Recommendation 5: “All researchers should make research data, methods, and other information integral to their publicly reported results publicly accessible in a timely manner to allow verification of published findings and to enable other researchers to build on published results, except in unusual cases in which there are compelling reasons for not releasing data. In these cases, researchers should explain in a publicly accessible manner why the data are being withheld from release”.

Information Security - There is again very extensive guidance available of information security issues including: from a standards perspective ISO/IEC 27002; for thought le adership the ―Jericho Forum‖ 5 and the ―Information Security

Forum‖ 6 ; and from a professional perspective the ―Institute of Information Security

Professionals‖ 7

1 http://www.raeng.org.uk/news/publications/list/reports/Engineering_values_in_IT.pdf http://www raeng.org.uk/news/publications/list/reports/Complex_IT_Projects.pdf

2 http://www.jiscinfonet.ac.uk/partnerships/records-retention-he/managing-research-records

3 See UKRIO  ̳Code of Practice for Research‘ Section 3.12 ―Collection and Retention of Data‖ at http://www.ukrio.org

4 http://www.nap.edu/catalog/12615 html

5 http://www.opengroup.org/jericho/

6 https://www.securityforum.org/

7 https://www.instisp.org/SSLPage.aspx