# Recent Inquir i es and Investigations of the CRU Emails and

th e IPCC Fourth Assessment Report

1. The disclosure of climate data from the Climate Research Unit at the University of East Anglia – UK House of Commons, Science and Technology Committee...................2

2. Report of the International Panel set up by the University of East Anglia to examine the research of the Climate Research Unit – University of East Anglia, Oxburgh Investigative Panel.........................................................................................................63

3. RA-10 Final Investigation Report Involving Dr. Michael E. Mann – The Pennsylvania State University.............................................................................................................72

4. Assessing an IPCC assessment – An analysis of statements on projected regional impact in the 2007 report – Netherlands Environmental Assessment Agency ............................91

5. The Independent Climate Change E-mails Review – University of East Anglia, Russell Investigative Panel ..........................................................................................191

Page numbers refer to PDF file pages and not the pages of the individual reports.HC 387-I Published on 31 March 2010 by authority of the House of Commons London: The Stationery Office Limited
£0.00 

# House of Commons Science and Technology Committee



# The disclosure of climate data from the Climatic Research Unit at the University of East Anglia

Eighth Report of Session 2009–10

Report, together with formal minutes

Ordered by the House of Commons to be printed 24 March 2010

### The Science and Technology Committee

The Science and Technology Committee is appointed by the House of Commons to examine the expenditure, administration and policy of the Government Office for Science. Under arrangements agreed by the House on 25 J une 2009 the Science an d Technology Committee
was established on 1 October 2009 with the same membership and Chairman as the former Innovation, Universities, Science and Skills Commi ttee and its proceedings were deemed to have been in respect of the Scie nce and Technology Committee.

Current membership

Mr Phil Willis (Liberal Democrat, Harroga te and Knaresborough)(Chair)
Dr Roberta Blackman-Woods (Labour, City of Durham)
Mr Tim Boswell (Conservative, Daventry) Mr Ian Cawsey (Labour, Brigg & Goole)
Mrs Nadine Dorries (Conservative, Mi d Bedfordshire)
Dr Evan Harris (Liberal Democrat, Oxford West & Abingdon)
Dr Brian Iddon (Labour, Bolton South East)
Mr Gordon Marsden (Labour, Blackpool South) Dr Doug Naysmith (Labour, Bristol North West) Dr Bob Spink (Independent, Castle Point)
Ian Stewart (Labour, Eccles) Graham Stringer (Labour, Manchester, Blackley)
Dr Desmond Turner (Labour, Brighton Kemptown)
Mr Rob Wilson (Conservative, Reading East)

Powers

The Committee is one of the departmental Select Committees, the powers of which are set out in House of Commons Standing Orders, principally in SO No. 152. These are available on the Internet via www.parliament.uk.

Publications

The Reports and evidence of the Committee are published by The Stat ionery Office by Order of the House. All publications of the Committee (inc luding press notices) are on the Internet at http://www.parliament.uk/ science. A list of reports from the Committee in this Parliament is
included at the back of this volume.

Committee staff

The current staff of the Committee are: Glenn McK ee (Clerk); Richard Ward (Second Clerk); Dr Christopher Tyler (Committee Spec ialist); Xameerah Malik (Committee Specialist); Andy Boyd (Senior Committee Assistant); Camilla Brace ( Committee Assistant); D ilys Tonge (Committee Assistant); Melanie Lee (Commi ttee Assistant); Jim Hudson ( Committee Support Assistant); and Becky Jones (Media Officer).

Contacts

All correspondence should be addressed to the Clerk of th e Science and Technology Committee, Committee Office, 7 Millbank, London SW1P 3JA. The telephone number for general inquiries is: 020 7219 2793; the Committee’s e-mail a ddress is: scitechc om@parliament.uk.The disclosure of climate data from the Climatic Research Unit at the University of East Anglia 1



# Contents

Report Page

Summary 3

1 Introduction 5

The Climatic Research Unit at UEA 5
The disclosure of climate data 5
The aftermath 6 The independent inquiries set up by UEA 7 Our inquiry 8 Our Report 9

2 Datasets 10

Climate science 10

Context 11 Complaints and accusations 11 Transparency 12 Dishonesty 19 Perverting the peer review process 21

3 Freedom of information issues 24 Freedom of Information legislation 24 Alleged breaches of the Freedom of Information Act 2000 26
The e-mails 26 Correspondence with the Deputy Information Commissioner 28 Volume of requests 33

4 Independent inquiries 36 The Independent Climate Change Email Review 36
Terms of reference 36
The Review team 38 Transparency 40 Scientific Appraisal Panel 41
Public view of the climate science 42 Need for a single review 44

5 Conclusions 46

Conclusions and recommendations 47

Formal Minutes 52

Witnesses 55

List of written evidence 552 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia

List of unprinted evidence 57

List of Reports from the Committee during the current Parliament 58The disclosure of climate data from the Climatic Research Unit at the University of East Anglia 3



# Summary

The disclosure of climate data from the Climatic Research Unit (CRU) at the University of East Anglia (UEA) in November 2009 had the potential to damage th e reputation of the climate science and the scientists involved.

We believe that the focus on CR U and Professor Phil Jones, Dire ctor of CRU, in particular, has largely been mispla ced. Whilst we are concerned that the disclosed e-mails suggest a blunt refusal to share scientif ic data and methodologies with others, we can sympathise with Professor Jones, who must have found it frus trating to handle reques ts for data that he knew—or perceived—were motivated by a desire simply to undermine his work.

In the context of the sharing of data and methodologies, we consider that Professor Jones’s actions were in line with common practice in the climate science community. It is not standard practice in climate science to publish the raw data and the compu ter code in academic papers. However, climat e science is a matter of grea t importance an d the quality of the science should be irreproachable. We therefore consider t hat climate scientists should take steps to make available all the da ta that support their work (including raw data) and full methodological workings (includi ng the computer code s). Had both been available, many of th e problems at UEA coul d have been avoided.

We are content that th e phrases such as “tri ck” or “hiding the decli ne” were colloquial terms used in private e-mails and the balance of e vidence is that they were not part of a systematic attem pt to mislead. Lik ewise the evidence t hat we have seen does not suggest that Professor Jones was trying to subvert th e peer review process. Academics should not be criticised for making informal comments on academic papers.

In the context of Freedom of Information (FOIA) , much of the resp onsibility should lie with UEA. The disclosed e-mails appear to sh ow a culture of non-disclosure at CRU and instances where information may have been deleted, to av oid disclosure. We found prima facie evidence to suggest that the UEA found ways to support the culture at CRU of resisting disclosure of information to climat e change sceptics. The failure of UEA to grasp fully the potential dama ge to CRU and UEA by the non-di sclosure of FOIA requests was regrettable. UEA needs to review its policy towards FOIA and re-asse ss how it can support academics whose expertise in this area is limited.

The Deputy Information Commissioner has given a clear indication th at a breach of the Freedom of Informatio n Act 2000 may have occurred but that a prosecution was timebarred; however no investigation has been carried out. In our vi ew it is unsatisfactory to leave the matter unresolved. We conclude that the ma tter needs to be resolved conclusively—either by the Independent Climate Change Email Review or by the Information Commissioner.

We accept the in dependence of the Clima te Change E-mail Review and recommend that the Review be open and transparent, taking or al evidence and conducting interviews in public wherever possible.

On 22 March UEA announced th e Scientific Appraisal Pane l to be chaired by Lord4 Optional header



### Oxburgh. This Panel should de termine whether the work of CRU has been soundly built and it would be premature fo r us to pre-ju dge its work.The disclosure of climate data from the Climatic Research Unit at the University of East Anglia 5



# 1 Introduction

1. On Friday 20 November 2009 it was reported across the world that hackers had targeted a “leading climate research unit” 1 

### and that e-mails from the University of East Anglia’s (UEA) Climatic Research Unit (CRU), one of the world’s foremost centres of climate science, had been published in the internet. 2 

### The story of the substantial file of private emails, documents and data that had been leaked helped igni te the global warming debate in the run up to the Cope nhagen climate change conference in Decembe r 2009. As reported by the press, exchanges on the internet allege d that data had been manipulated or deleted, in order to support evide nce on global warming.



### The Climatic Resear ch Unit at UEA

2. UEA was founded in 1963 and in 1972 UEA established CRU. 3 

### CRU’s website describes the Unit as being “widely recognised as one of the world’s leading institutions concerned with the study of natural and anthropogenic [h uman caused] climate change”. 4 

### CRU has a staff of around thirty research scientists and students. 5 

### But as we heard in oral evidence, it is in fact “a very small Un it [with only] three full-time members of academic staff”. 6



### 3. CRU has developed a number of the datasets widely used in climate research, including the global temperatu re record used to moni tor the state of the clim ate system, as well as statistical software packages an d climate models. In its written submissi on to the inquiry UEA outlined CRU’s “pioneering role” in the science of understanding the world’s changing climate. CRU’s co ntributions included the co mpilation of a global land temperature record and the development of increasingly sophistica ted methods by which to represent the average temperat ure of the globe and changes in that average over time. 7 

### Professor Edward Acton , the Vice-Chancellor of UEA, indicated that he was “immensely proud of what they have done; [as] without them humanity wo uld be vastly less able to understand c limate change.” 8



### The disclosure of climate data

4. In mid November 2009 it appea red that a server used by CRU had been accessed with 160 MB of data containing more than 1,000 e-mails and 3,000 other documents being

1 “Hackers target leading climate research unit”, BBC News website , 20 November 2009 news.bbc.co.uk/1/hi/sci/tech/8370282.stm

2 For example: “Hacked E-Mail Is New Fodder for Climate Dispute”, New York Times website , 21 November 2009 www.nytimes.com/2009/11/2 1/science/earth/21clim ate.html?_r=4 and “Hackers leak em ails, stoking cl imate debate”, Sydney Morning Herald website , 23 November 2009, www.smh.com.au/t echnology/technology-news/hackers-leakemails-stoking-climate-debate-20091123-iu6u.html

3 Ev 17, paras 1.2 and 1.5

4 “About the Climatic Research Unit”, CRU website, www.cru. uea.ac.uk/cru/about/

5 As above

6 Q 92

7 Ev 17, paras 1.5-1.6

8 Q 1526 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia



### copied. 9 

### A UEA spokeswoman confirmed that th e information was not available on a server that could be easily accessed and could not have been inadvertently released. 10 

### It is not known exactly when the breach occurred; the RealCli mate website, “a commentary site on climate science by working climate scientists for the inte rested public and journalists”, 11 

### indicated that UEA had been notified of the possible se curity breach on 17 November. 12 

### The following was pos ted anonymously on the climate-sceptic blog, The Air Vent :
November 17, 2009 at 9:57 pm

We feel that climate science is, in the cu rrent situation, too important to be kept under wraps.

We hereby release a random selection of correspondence, co de, and documents. Hopefully it will give some insight into the scie nce and the peop le behind it. 13



### From here the debate was “blown wide open”. 14 

### The Guardian ran the story on 20 November with the headline: “Climate sceptics claim leaked e-mails are evidence of collusion among scientists”. 15



### 5. UEA issued a statement on 20 November: “This information has been obtained and published without our permission and we took immediate action to remove the server in question from operation. We are undertakin g a thorough internal investigation and we have involved the police in this inquiry.” 16 

### The e-mails containe d technical and routine aspects of climate research, in cluding data analysis and details of scie ntific conferences. The controversy has focused on a small number of e-mails, par ticularly those sent to, or written by, climatologist Professor Ph il Jones, the Director of CRU.

The aftermath

6. Condemnation of alleged malpractices found within the leaked CRU e-mails was quickly disseminated on the internet. Co ntributors to climate change debate websites and written submissions to us claimed that these e-mails showed a delibera te and systemat ic attempt by leading climate scientis ts to manipulate climate data, arbitrarily adjusting and “cherrypicking” data that su pported their global warming claims and deleting adverse data that questioned their theories. 17 

### It was alleged t hat UEA may not have complied with the requirements of the Freedom of Informatio n Act 2000, that inap propriate statistical methods and defective computer programmes may have been used to analyse data and that

9 RealClimate website archive, November 2009, www.realclimate.org/index.php/archives/2009/11/the-cru-hack

10 “Scotland Yard call in to probe clim ate data leak from UEA in Norwich”, Norwich Evening News , 1 December 2009

11 RealClimate website ‘about’ page, www.realclimate.org

12 RealClimate website archive, November 2009, www.realclimate.org/index.php/archives/2009/11/the-cru-hack; the data may have been downloaded on to the RealClimate—see paragraph 12.

13 The Air Vent website, November 2009 archive, noconsensus.wordpress.com/2009/11/page/3/

14 As above

15 “Climate sceptics claim leaked emails are evidence of collusion among scientists”, The Guardian , 20 November 2009

16 “Sceptics publish climate e-mails ‘s tolen from East Anglia University’”, The Times , 21 November 2009

17 For examples see Ev 85 [Roger Helmer MEP], Ev 92 [Godfrey Bloom MEP], and Ev 144 [Stephen McIntyre]The disclosure of climate data from the Climatic Research Unit at the University of East Anglia 7



### CRU may have attemp ted to abuse the proce ss of peer review to pr event the publication of research papers with conflictin g opinions about climate change. 18



### 7. In a statement released on 24 November , Professor Trevor Davies, UEA pro-ViceChancellor with responsibility for research, rejected calls for Pr ofessor Jones’s resignation: “We see no reason for Profe ssor Jones to resign and, inde ed, we would not accept his resignation. He is a valued and important scientist.” 19 

### He also contested several of the claims of malpractice: “It is well known within the scientif ic community and particularly those who are sceptical of climate change that over 95% of the raw st ation data has been accessible through the Global Historical Clim atology Network for several years. We are quite clearly not hiding inform ation which seems to be the sp eculation on some blogs and by some media commenta tors”. He added:

There is nothing in the stolen material which indicates that peer-reviewed publications by CRU, and others, on the nature of global warming and related climate change are not of the highest-qua lity of scientific investigation and interpretation. CRU’s peer-reviewed publi cations are consistent with, and have contributed to, the overwhelming scientif ic consensus that the climate is being strongly influenced by human activity. 20



### 8. On 1 December, Professor Jone s announced that he would st ep aside from the Director’s role during the course of the independent review. 21



### The independent inquiries set up by UEA

9. On 3 December UEA announced that an in dependent review—the Independent Climate Change Email Review—into the allegations made against CRU would be carried out by Sir Muir Russell. 22 

### Professor Acton explained in a letter to us why Sir Muir was chosen to head the review:

Sir Muir is extremely experi enced in public life, has an understandin g of the conduct of universities and re search, and is entirely independent of any association with this University and with the climate change debate. 23



### 10. Alongside the Independen t Climate Change E-Mails Review, UEA decided on a separate scientific assessment of CRU’s key scie ntific publications; an external reappraisal of the science itself. The Royal Society agreed to assist UEA in identifying assessors with the requisite experience, standing and independence. 24 

### UEA announced on 22 March that Lord Oxburgh FRS would “chair an independent Scientific Assessment Panel to examine

18 For examples see Ev 90 [Phillip Bratby]; Ev 115 [David Holland], para 2; Ev 144 [Stephen McIntyre]; Ev 194 [Peabody Energy Company], para 24.

19 “Climate scientist at centre of leaked email row dismisses conspiracy claims”, The Guardian , 24 November 2009

20 UEA, “CRU update 2”, 24 November 2009, www.uea.ac.uk/mac/comm/media/press/2009/nov/CRUupdate

21 UEA, “CRU update 3”, 1 December 2009, www.uea. ac.uk/mac/comm/media/press/2009/nov/CRUupdate

22 “Sir Muir Russell to head the Independent Review into the allegations against the Climatic Research Unit (CRU)”, UEA Press Release, 3 December 2009, www.uea.ac .uk/mac/comm/media/pre ss/2009/dec/ CRUreview

23 Ev 16

24 Ev 18, para 2.38 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia



### important elements of the published science of the Climatic Research Unit (CRU) at the University of East Anglia”. 25



### Our inquiry

11. We were concerned by the press reports and on 1 Decem ber 2009 the Chair of the Committee wrote to the Vice-Chanc ellor of UEA. The letter explained th at we took a close interest in academic integrity and the systems in place to ensure th e quality of evidence from research and evide nce-based policy making. The letter requested a note on the recent events setting out:

a) what had taken place;

b) the steps that had been take n to investigate the allegations and to test the integrity of the data held and used by CRU;

c) how CRU justified its commitment to academic transparency; and

d) how the Vice-Chancellor proposed to restor e confidence in CRU and its handling of data.

We also asked for an assurance that none of the data referred to in the e-mails that had been publicised had been destroyed. 26



### 12. UEA replied on 10 December 20 09. It explained that “a sign ificant amount of material including emails and document s appears to have been accessed illegally from a back-up server in CRU and do wnloaded in whole, or possibly in part, on to the RealClimate website.” 27 

### This incident was the subj ect of a police enquiry and the Norfolk Constabulary investigation was expected to ta ke some time. UEA was keen to stress that this “episode is being treated very seri ously” and announced that it had set up the independent inquiry, headed by Sir Muir Ru ssell, to investigate the allegations ag ainst CRU. UEA said that “none of the adjusted station data re ferred to in the emails that have been published has been destroyed.” 28



### 13. In the light of the gravity of the allegations against CRU, the growing weight of damaging press coverage, on-goi ng concerns about the deleti on of data and the serious implications for UK science we de cided to hold an inquiry into the disclosure of the data at CRU. On 22 January 2010 we therefore anno unced the inquiry inviting submissions on three key issues:

• 

### What were the implications of the disclosure s for the integrity of scientific research?

• 

### Were the terms of referenc e and scope of the Independ ent Review announced on 3 December 2009 by UEA adequate?

25 “CRU Scientific Assessment Panel announced”, UEA Press Release, 22 March 2010, www.uea.ac.uk/mac/comm/media/press/CRUstatements/SAPannounce

26 House of Commons Science and Technology Committee Press Notice 04, 7 December 2009, Session 2009–10

27 Ev 16

28 Ev 17The disclosure of climate data from the Climatic Research Unit at the University of East Anglia 9

• 

### How independent were the ot her two international data sets (see paragraph 23)?

14. If there had been more time available before the end of this Parliament we would have preferred to carry out a wider inquiry into the science of global warming itself. In response to enquiries we issued a statemen t on 1 February making it clear that the inquiry would focus on the terms of reference announced on 22 January and that this was not an inquiry into global warming. 29



### 15. We set a deadline of 10 Febr uary for the submission of memoranda and we have received 58 submissions, not including supplementary memoranda. We held one oral evidence session on 1 Ma rch, when we took evide nce from five panels:

a) Rt Hon Lord Lawson of Blaby, Chairman , and Dr Benny Peiser, Director, Global Warming Policy Foundation;

b) Richard Thomas CBE, former Information Commissioner;

c) Professor Edward Acton, Vice-C hancellor, UEA and Professor Phil Jones, Director of CRU;

d) Sir Muir Russell, He ad of the Independent Climat e Change E-Mail s Review; and

e) Professor John Bedding ton, Government Ch ief Scientific Advise r, Professor Julia Slingo OBE, Chief Scientist, Met Office, and Professor Bo b Watson, Chief Scientist, Department for Environment, Food and Rura l Affairs.

16. We would like to thank everyone who cont ributed to the inquiry through written submissions or oral evidence. We also received unsolicited co pies of a numb er of books challenging anthropogenic globa l warming and reviewi ng events at CRU and the disclosed e-mails. 30



### Our Report

17. In the time left before the end of this Pa rliament we will not be able to cover all the issues raised by the events at UEA, nor cover all the ground that would be covered by the Independent Climate Change Em ail Review and the Scientific Appraisal Panel. We have therefore concentrated on what we believe to be key issues. Of cent ral concern is the accuracy and availability of CRU’s data, da tasets and computer programming, which we address in Chapter 2 of this Report; and related to the data and methodology is the question of access, or the wi thholding of access, under th e Freedom of Information Act 2000 which we cover in Chapter 3. Finally, in Chapter 4 we comment on the independent reviews that UEA has announced.

29 House of Commons Science and Technology Committee Press Notice 11, 1 February 2010, Session 2009–10

30 The Committee received the following books: Christopher Booker, The Real Global Warming Disaster , Continuum, 2009 A.W. Montford, The Hockey Stick Illusion , Stacey International, 2010 Steven Mosher and Tom Fuller, Climategate , St Matthew Publishing, 2010 Ian Plimer, Heaven and Earth , Quartet Books Limited, 200910 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia



# 2 Datasets

Climate science

18. Climate is distinct from weather : it is the average of weather conditions over a number of years. Climatologists study climates in differ ent parts of the world and for the Earth as a whole. CRU, according to its website: “has develo ped a number of the da ta sets widely used in climate research, in cluding the global temp erature record used to monitor the state of the climate system, as well as statist ical software packages and climate models”. 31



### 19. The process of calculating the Earth’s averag e global temperatures (past, present and future) is complicated and lengthy. Data from thousands of weat her stations all around the world, on land and at sea, must be co llected, checked for qu ality, adjusted for inconsistencies and error margins , and then mapped onto a seri es of grids on the Earth’s surface. The methods, results and conclusions are then pres ented to the academic world, first by passing the peer revie w process prior to publication, an d second, after presentation, the scrutiny of the wider academic community.

20. Climate science, like any ot her science, uses the scie ntific method to make its assessments of past and present climate and pr edictions about the future climate. The key characteristics of the scientific method can be described as: characte risations, hypotheses, predictions, and experiments.

• 

### Characterisations: considerat ion of a problem, and examin ation of whether or not an explanation exists for it.

• 

### Hypotheses: if no such explanation exists, a new explanation is stated.

• 

### Predictions: what co nsequences follow from a new explanation?

• 

### Experiments: is the outc ome consistent with the predicted consequences?

Each of these is subject to peer review prio r to the formal sharing of knowledge through publication. Through peer review scientists al low their views and methods to be critically appraised expertly and externally.

• 

### Replication and verification

To have the results and conc lusions survive crit icism or scepticism and be part of the accepted canon of scientific knowledge, most experiments will hav e to be demonstrably replicable (by the same group) to pass peer review and will of ten need to be verified by other independent researchers taking similar approaches.

21. Therefore climatologists are, like other scient ists, required to test their theories—such as global warming and the caus es of warming—against observat ional data. They must also replicate and verify their experiments, by holding independent da tasets and conducting independent analyses of these datasets, and by publishing th eir full methods and results for

31 www.cru.uea.ac.uk/cru/aboutThe disclosure of climate data from the Climatic Research Unit at the University of East Anglia 11



### scrutiny. Ultimately, these ideas are put up to th e threat of falsification by other scientists working in the field.

22. In this Chapter we discuss some aspects of this process.



### Context

23. There are three main international climate da tasets, which have be en built up from direct temperature measurements on land and sea at weather stations all around the world:

a) the National Climatic Data Center (NCDC) of the National Oceanographic and Atmospheric Administration (NOAA) in Asheville, North Carolina, USA;

b) the Goddard Institute of Space Studies (GIS S), part of the National Aeronautic and Space Administration (NASA) in New York, USA; and

c) CRUTEM3, at CRU, UEA. 32



### 24. In addition, there are two oth ers, one in Russia and one in Japan, that use similar methods. 33 

### There are also two that use satellite ob servations, by the Un iversity of Alabama at Huntsville and by Remote Sensing Sy stems, California. 34



### 25. Professor Jones, commenting on the differe nt climate research groups around the world in the UK, US, Russia and Japan, 35 

### told us that:

we are all working independently so we may be using a lot of co mmon data but the way of going from the raw data to a derived product of gridded tem peratures and then the average for the hemisphere and th e globe is totally in dependent between the different groups. 36



### 26. What sets the CRU dataset apar t is its comprehensiveness:

The CRU dataset, which form s the land surface componen t of the HadCRUT global temperature record, was compiled with th e aim of comprehens iveness. The majority of the data in it are derived from the same freely-available raw data sets used by NOAA and NASA. However, it also includes data derive d from station data that were obtained directly from countries, institutions and scientists on the understanding that they would not be passed on. 37



### Complaints and accusations

27. The complaints and accusations made agains t CRU in relation to the scientific process come under two broad headings. Th e first is transparency: that CRU failed to abide by best

32 Ev 21, para 4.2

33 Q 78

34 Ev 104 [D.R. Keiller], para 2

35 Q 79

36 Q 80

37 Ev 64 [John Beddington and Julia Slingo]12 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia



### scientific practice by refusing to share its ra w data and detailed me thods. The second is honesty: that CRU ha s deliberately misreprese nted the data, in or der to produce results that fit its preconceived views about the anthropogenic warmin g of the climate. We take each of these complaints and accusatio ns in turn.

Transparency

Raw data

28. Warwick Hughes, a “freelance earth scientist from Australia”, 38 

### had asked Professor Jones for CRU’s raw data. He received the following reply:

I should warn you that some data we have we are not supposed [to] pass on to others. We can pass on the gridded da ta—which we do. Even if WMO [World Meteorological Organization] ag rees, I will still not pass on the data. We have 25 or so years invested in the wo rk. Why should I make the data available to you, when your aim is to try and find something wrong with it. 39



### 29. On the face of it, this looks like an unreason able response to a reasonable request. As Lord Lawson put it: “Ask any decent scientist an d they will say the keystone for integrity in scientific research is full and transparent disclo sure of data and methods”. 40 

### However, Professor Jones, while confessing that he has sent some “awful” e-mails, 41 

### defended his position.

30. First, in answer to the question of whethe r the raw data are acce ssible and verifiable, Professor Jones told us that:

The simple answer is yes, most of the sa me basic data are ava ilable in the United States in something called the Global Hi storical Climatology Network. They have been downloadable there for a number of years so people ha ve been able to take the data, do whatever method of assessment of the quality of the data and derive their own gridded product and compare that with other workers. 42



### 31. In addition, of course, there are the sources of the data, th e weather stations, to which any individual is free to go and collect the data in the same way that CRU did. This is feasible because the list of stations that CRU used was published in 2008. 43



### 32. Even if CRU had wanted to, it would have been unable to publish all of these data because, as Professor Acto n explained, some of the da ta are bound by commercial agreements with differ ent national meteorol ogical organisations:

38 www.warwickhughes.com

39 Ev 158, Appendix 1

40 Q 9

41 Q 103

42 Q 78

43 Q 98The disclosure of climate data from the Climatic Research Unit at the University of East Anglia 13



### Unfortunately, several of these countries impose con ditions and say you are not allowed to pass [on the data]. Seven coun tries have said “No, yo u cannot”, half the countries have not yet answered, Canada and Poland are amongst those who have said, “No you cannot publish it” and also Sw eden. Russia is very hesitant. We are under a commercial promise, as it were, not to; we are long ing to publish it because what science needs is the most openness. 44



### (The issue with Sweden has since been resolved. The Swedish Meteorological and Hydrological Institute gave pe rmission for CRU to publish its Swedish data on the UEA website on 8 March 2010. 45 

### )

33. Second, as UEA explained in its submission, it is:

sometimes necessary to adjust temperature data because ch anges in stat ion location, instrument or observation ti me, or in the methods used to calculate monthly average temperatures can introduce fa lse trends. These have to be removed or adjusted, or else the overall seri es of values will be incorrect. In th e early 1980s, CRU painstakingly examined the long-term ho mogeneity of each station temperature series which it acquired. As a result, data were adjusted for about 11% of the sites, that is approximately 314 sites out of a then-total of some 3,276. This was in complete accordance with st andard practice, and all ad justments were documented. 46



### 34. Professor Jones added, when he gave oral evidence:

It is all documented [...] what [adjustments we made to the data] in the 1980s and since then we have obviously added more station data as more has become available, as countries have digiti sed more data; we have added th at in and we have reported on that in our peer review publications in 2003 and 2006. 47



### 35. These kinds of adjustments to raw data take a lot of time. That is why, in the words of Professor Jones, “Most scientists do not want to deal with the raw station data, they would rather deal with a derived product”. 48



### 36. A third point was made by Professor Ac ton that CRU should not be under any obligation to pr ovide raw data:

May I also point out that it is not a national archive, it is not a library, it is a research unit. It has no special duty to conserve an d its data is the copy of data provided by over 150 countries, whose national meteor ological stations turn the data into the average for a month. 49

44 Q 94

45 Ev 39, para B

46 Ev 18, para 3.4

47 Q 81

48 Q 107

49 Q 9214 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia



### 37. CRU’s refusal to release the raw data gave some the impr ession that it was deliberately keeping its work private so that its studies could not “be replicated and critiqued”. 50 

### The Peabody Energy Company said of CRU that “they appeared to be particularly concerned that putting their infor mation in the public domain wo uld expose their work to criticism”. 51 

### Even an effort to conduct a simple qua lity check was said to be thwarted by CRU’s unwillingness to shar e the data it had used. 52 

### In contrast, NASA has been able to make all its raw data available as well as its programmes. 53



### 38. We recognise that some of the e-mails suggest a blunt re fusal to shar e data, even unrestricted data , with others. We acknowledge that Professor Jones must have found it frustrating to handle requests for data th at he knew—or perceived—were motivated by a desire simply to seek to undermine his work. But Professo r Jones’s failure to handle helpfully requests for data in a field as im portant and controversia l as climate science was bound to be viewed with suspicion. He was obviously frustrat ed by other workers in the field trying to “undermine” his work, but his action s were inevitably counterproductive. Professor Jones told us that the published e-mails represented only “one tenth of 1%” of his ou tput, which amounts to one mi llion e-mails, and that we were only seeing the end of a protracted se ries of e-mail exchanges. We consider that further suspicion could have b een allayed by releasing all the e-mails. In addition, we consider that had the available raw data been available online from an early stage, these kinds of unfortunate e-mail exchanges would not have oc curred. In our view, CRU should have been more open with its raw data and followed the more open approach of NASA to making data available.

39. We are not in a position to set out any further the extent, if any, to which CRU should have made the data av ailable in the interests of transparency, and we hope that the Independent Clim ate Change Email Review will reach specific conclusions on this point. However, transparency and accountability are of ar e increasing importance to the public, so we recommend th at the Government reviews the rules for the accessibility of data sets collected and an alysed with UK public money.

Methods

40. The Royal Society of Chemistry in its submission made it clear that:

It is essential that the pu blic and all non-specialists re main truly conf ident in the scientific method to provide a sound scientif ic evidence-base on which strong decisions can be made. 54



### There have been critici sms that Professor Jones and colleagues have not shared their methodologies. Andrew Montford, author of The Hockey Stick Illusion

, 55 

### pointed out in his memorandum that:

50 Ev 194 [Peabody Energy Company], para 20

51 As above

52 Ev 152 [Steven Mosher], para 8

53 Q 150 [Professor Jones]

54 Ev 170, summaryThe disclosure of climate data from the Climatic Research Unit at the University of East Anglia 15



### The scientific method demands that findings be subject to testing and verification by others. The refusal of CRU sc ientists to release information to those who they felt might question or threaten their findings have led many to conclude that the CRU’s work is not trustworthy. 56



### 41. Professor Jones contested thes e claims. According to him, “The methods are published in the scientific pape rs; they are relatively si mple and there is nothing that is rocket science in them”. 57 

### He also noted: “We have made all th e adjustments we have made to the data available in these reports 58 

### ; they are 25 years old now”. 59 

### He added that the programme that produced the global temperature average had been available from the Met Office since December 2009. 60



### 42. On this basis, he ar gued, it was unnecessary to provide the exact codes that he used to produce the CRUTEM3 chart. The Met Office had released its code and it produced exactly the same result. 61



### 43. In answer to the charge that the computer codes that were stolen from CRU’s computer network were defective, 62 

### Professor Jones pointed out that:

Those codes are from a much earlier time, they are from the period about 2000 to 2004. [They] do not relate to the prod uction of the global and hemispheric temperature series. They are no thing to do with that, they are to do with a different project [...] that was funded by the British Atmospheric Da ta Centre, which is run by NERC, and that was to produce more gr idded temperature data and precipitation data and other variables. A lo t of that has been released on a Dutch website and also the BADC website. 63



### 44. CRU’s alleged refusal to discl ose its assumptions and method ologies gave credence to the view that exposure to “i ndependent scrutiny would have undermined the AGW [anthropogenic global warming] hypothesis”. 64 

### However, the failure to publish the computer code for CRUTEM3 left CRU vulnerable when concerns emerged that other codes it used had faults. Jo hn Graham-Cumming, a professi onal computer programmer, told us that:

55 Andrew Montford, The Hockey Stick Illusion: Climateg ate and the corruption of science , Stacey International, 2010

56 Ev 159, para 4

57 Q 92

58 Raymond Bradley, Mick Kelly, Phil Jones and others, A Climatic Data Bank for Northern Hemisphere Land Areas,
1851-1980 , US DoE, Technical Report TRO17, 1985, p 335; Phil Jones, Sarah Raper, Ben Santer, and others, A Grid Point Surface Air Temperature Data Set for the Northern Hemisphere , DoE Technical Report No. TR022, US Department of Energy, 1985, p 251; Phil Jones, Sarah Raper, Claire Goodess, and others, A Grid Point Surface Air Temperature Data Set for the Southern Hemisphere, 1851-1984 , DoE Technical Report No. TR027, US Department of Energy, 1986, 73

59 Q 97

60 As above

61 Qq 139-42

62 Ev 32, Q 137; Ev 196 [John Graham-Cumming]

63 Qq 137-38

64 Ev 94 [Clive Menzies], para 1.516 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia



### the organization writ ing the [other] co de did not adhere to st andards one might find in professional software engineering. The co de had easily identifi ed bugs, no visible test mechanism, was not apparently under version control and was poorly documented. It would n ot be surprising to find that other code written at the same organization was of similar quality. And given that I subs equently found a bug in the actual CRUTEM3 code only reinforces my opinion. 65



### 45. The conspiracy claims were fu elled by CRU’s refusal to sh are the most de tailed aspects of its methodologies, for ex ample, the computer codes for producing global temperature

averages. We note that the research passed the peer review process of some highly reputable journals. However, we note that CRU could have been more open at that time in providing the detailed met hodological working on its we bsite. We recommend that all publicly funded re search groups consider whether they are being as open as they can be, and ought to be, with the details of their methodologies.

Repeatability and verification

46. These complaints and concerns surroundin g transparency cut to the heart of the scientific process. It has been argued that without access to the raw data and detailed methodology it is not possible to check the resu lts of CRU’s work. The Institute of Physics pointed out that:

Published reconstructions may represent only a part of the raw data available and may be sensitive to the choices made and th e statistical techniques used. Different choices, omissions or statistical processe s may lead to different conclusions. This possibility was evidently the reason behind some of the (rejected) reques ts for further information. 66



### 47. This has substance if one considers CRU’s wo rk in isolation. But s cience is more than individual researchers or research groups. One should put research in context and ask the question: what would one hope to find by double checking th e processing of the raw data? If this were the only dataset in existence, and Professor Jones’s team had been the only team in the world to analyse it , then it might make sense to double check independently the processing of the raw data and the methods. But there are other datasets and other analyses that have been carried out as Professor Jones explained:

There are two groups in America that we [CRU] comp are with and there are also two additional groups, one in Russia and one in Japan, that also produce similar records to ourselves and they all show pret ty much the same so rt of course of instrumental temperature ch ange since the nineteenth century compared to today. 67



### [...] we are all working inde pendently so we may be using a lot of common data but the way of going from the raw data to a derived product of gr idded tempera tures and

65 Ev 196

66 Ev 167, para 4

67 Q 78The disclosure of climate data from the Climatic Research Unit at the University of East Anglia 17



### then the average for the hemisphere and th e globe is totally in dependent between the different groups. 68



### 48. In its memorandum UEA explai ned the differences between the methodologies used by three basic datasets for land areas of the world, NOAA, NASA and CRU/UEA:

All these datasets rely on primary obse rvations recorded by NMSs [National Meteorological Services] across the globe. 69



### GISS [70] 

### and NCDC [71] 

### each use at least 7,200 stat ions. CRUTEM3 us es fewer. In CRUTEM3, each monthly temp erature value is expressed as a departur e from the average for the base period 1961–90. This “anomaly method” of expressing temperature records demands an adequate amount of data for the base period; this limitation reduces the number of stations used by C RUTEM3 to 4,348 (from the dataset total of 5,121). The latest NCDC analysis [...] has now moved to the “anomaly method” though with different refinements from those of CRU. 72



### NCDC and GISS use differen t approaches to the problem of “absolute temperature” from those of CRUTEM3. Th e homogeneity procedures undertaken by GISS and NCDC are completely different from th ose adopted for CRUT EM3. NCDC has an automated adjustment procedure [...], whil st GISS additionally makes allowances for urbanization effects at some stations. 73



### 49. In our call for evidenc e we asked for submissions on the question of how independent the other international data se ts are. We have established to the extent that a limited inquiry of this nature can, that the NCDC/NOAA and GISS/NASA data sets measuring temperature changes on land an d at sea have arrived at simi lar conclusions using similar data to that used by CRU, but using inde pendently devised methodologies. We have further identified that there are two other data sets (Univer sity of Alabama and Remote Sensing Systems), using satellite observations that use entirely different data than that used by CRU. These also confirm th e findings of the CRU work. We therefore conclude that there is independent verifica tion, through the use of ot her methodolog ies and other sources of data, of th e results and conclusions of the Climate Research Unit at the University of East Anglia.

50. The fact that all the datasets show broadly the same sort of course of instrumental temperature change since the nineteenth ce ntury compared to tod ay was why Professor John Beddington, the Government Chief Scientific Adviser, ha d the confidence to say that

68 Q 80

69 Ev 21, para 4.3

70 Dataset held by the Goddard Institute for Space Studies (GISS, USA) part of the National Aeronautic and Space Administration (NASA)

71 Global Historical Climatology Network (GHCN) dataset held by National Climatic Data Center (NCDC), the National Oceanographic and At mospheric Administra tion (NOAA, USA)

72 Ev 21, para 4.4

73 Ev 21, para 4.518 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia



### human induced global warm ing was, in terms of the eviden ce to support t hat hypothesis, “unchallengeable”: 74



### I think in terms of datasets, of the way in which data is analyse d, there will always be some degree of uncertainty but when you get a series of fundamentally different analyses on the basic data and they come up with similar conclusions, you get a [...] great deal of certaint y coming out of it. 75



### 51. Even if the data that CRU used were no t publicly available—which they mostly are—or the methods not publis hed—which they have been—i ts published results would still be credible: the results from CRU agree with those drawn from other international data sets; in other words, the analyses have been repeated and th e conclusions have been verified.

52. That is probably part of w hy it has not been practice in the climate scie nce community to publish all the data and computer codes with the academic papers. We got to the crux of the issue during an interesting exchange with Pr ofessor Jones:

Graham Stringer: You are saying that every pa per that you have produced, the computer programmes, the we ather stations, all the info rmation, the codes, have been available to scientists so that they could test out how good your work was. Is that the case on all the papers you have produced?

Professor Jones: That is not the case.

Graham Stringer: Why is it not?

Professor Jones: Because it has not been standard practice to do that.

Graham Stringer: That takes me back to the original point, that if it is not standard practice how can the science progress?

Professor Jones: Maybe it should be stan dard practice but it is not standard practice across the subject. 76



### 53. Another reason why data an d the codes were not publis hed may be that norms for publication evolved in a period when the journals were only published in hard copy. In such circumstances it is un derstandable why an editor would not want to publish raw climate data (extremely long li sts of numbers) and code for the computer pr ogrammes that analyse the data (which run to hundreds of thousands of lines of code). However, in the age of the internet, these kinds of products ca n be made available mo re easily, and we are minded to agree with Professor Jones observation on this point that: “Maybe it should be standard practice”. 77

74 Q 191

75 Qq 191–92

76 Qq 100–02

77 Q 102The disclosure of climate data from the Climatic Research Unit at the University of East Anglia 19



### 54. It is not standard practice in climate science and many other fields to publish the raw data and the computer code in academic papers. We think that this is problematic because climate science is a matter of glob al importance and of public interest, and therefore the quality and transparency of the science should be irreproachable. We therefore consider that climate scientists should take steps to make available all the data used to generate their published work, including raw data ; and it should also be made clear and referenced where data has been us ed but, because of comm ercial or national security reasons is not available. Scientists are also, under Freedom of Information laws and under the rules of normal scientific cond uct, entitled to with hold data which is due to be published under the peer-review process. 78 

### In addition, scientis ts should take steps
to make available in full their methodological workings , including the computer codes. Data and methodological workings should be provided via the in ternet. There should be enough information publis hed to allow verification.

Dishonesty

55. Of all the e-mails released, on e dated 16 Nove mber 1999 has caused particular concern:

I’ve just completed Mi ke’s Nature trick of adding in the real temps to each series for the last 20 years (ie from 1981 onwards) amd [ sic ] from 1961 for Ke ith’s to hide the decline. 79



### 56. The word “trick” and the ph rase “hide the declin e” have been taken by some to demonstrate intent on the part of Professor Jones to “falsi fy data” and to “exaggerate warming”. 80



### “Trick”

57. In his submission, Peter Taylor, author of Chill

, 81 

### states that:

The tree ring data did not match the model expectation (ie the ‘ho ckey stick’ pattern of a sudden rise at the end of the period). Rather than admit this, the team-workers discuss using Michael Mann’s ‘trick’ of replacing th e offending tree-ring data and using instrumental data in its place in a spliced graph. 82



### 58. UEA interpreted the use of th e word “trick” differently:

as for the (now notorious) wo rd ‘trick’, so deeply appealing to the media, this has been richly misinterpreted an d quoted out of context. It was used in an informal email, discussing the di fficulties of statistica l presentation. It does not mean a ‘ruse’ or method of deception. In co ntext it is obvious that it is used in the informal sense

78 See paragraph 78 and following; section 22 of the FOIA provides an exemption from disclosure where the requested information is intended for future (but imminent) publication.

79 E-mail from Phil Jones to Ray Bradley, 16 November 1999

80 Ev 93 [Godfrey Bloom MEP], para 4

81 Peter Taylor, Chill, A Reassessment of Global Warming Theory: Does Climate Change Mean the World is Cooling, and If So What Should We Do About It? , Clairview Books, 2009

82 Ev 188, para 2220 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia



### of ‘the best way of doing so mething’. In this case it was ‘the trick or knack’ of constructing a statisti cal illustration which would comb ine the most re liable proxy and instrumental evidence of temperature trends. 83



### 59. These interpretations of the colloquial meaning of “trick” have been accepted by even the staunchest of critics:

Lord Lawson of Blaby: The sinister thing is not the word ‘trick’. In their [UEA’s] own evidence they say that what they me an by ‘trick’ is th e best way of doing something.

Chairman: You accept that?

Lord Lawson of Blaby: I accept that. 84



### 60. Critics of CRU have suggested that Profess or Jones’s use of the word “trick” is evidence that he was part of a conspiracy to hide evidence that did not fit his view that recent global warming is pr edominately caused by huma n activity. Th e balance of evidence patently fails to support this view. It appears to be a colloquialism for a “neat” method of handling data.

“Hide the decline”

61. Lord Lawson did, however, de scribe CRU’s tr eatment of the data as “reprehensible”, 85 

### because, in his view, Professor Jones deliberately h id data that demonstrated a decline in temperatures. 86



### 62. The data that he believed to be “hidden” are a set of tree ri ng data that disagree with other data sources regarding te mperature trends. Lord Lawson said: “when the proxy series [...] departed from the measured temperature series, a normal pers on will say maybe that means the proxy series is not all that reliable”. 87 

### In that context he made two specific claims:

• 

### that the tree ring data were fl awed because “for a long period before 1421 they relied on one single pine tree”; 88 

### and

• 

### that the divergence problem was not just for data after the 1960s, “it is not a good fit in the latter half of the ni neteenth century either”. 89



### 63. It is outside the remit of th e terms of reference of this inquiry to make a detailed assessment of the science, but it is worth noting that Professor Jones had a very different perspective. On the first point, he commented:

83 Ev 19, para 3.5.6

84 Qq 25–26

85 Q 26

86 Qq 26–28

87 Q 26

88 As above

89 Q 28The disclosure of climate data from the Climatic Research Unit at the University of East Anglia 21



### That particular reconstructio n went back to 1400, or ju st after 1400, and that is because there are insufficient trees to go back before that , there are more than just one. We have criteria to determine how far you can go back in terms of the number of trees you have at a ce rtain number of sites. 90



### 64. On the second point, he told us:

One of the curves was based on tree ring data which showed a very good relationship between the tree rings and the temperature fr om the latter part of the nineteenth century through to 1960, an d after that there was a dive rgence where the trees did not go up as much as the real temperatures had. 91



### 65. Professor Jones has pu blished on this issu e on several occasion s, including a 1998

Nature paper 92 

### and subsequent papers. 93 

### He contested the view that he was trying to hide the decline in the sense that he was trying to pretend that these data did not exist and thereby exaggerate global warm ing: “We do not accept it was hidden because it was discussed in a paper [94] 

### the year before and we have discussed it in every paper we have written on tree rings and climate”. 95 

### Rather, what was meant by “hide the decline” was remove the effects of data kn own to be problematic in the sense that the data were known to be misleading. UEA made it clear in its written submission that:

CRU never sought to disguise th is specific type of tree-ri ng “decline or divergence”. On the contrary, CRU has publi shed a number of pioneeri ng articles th at illustrate, suggest reasons for, and di scuss the implicatio ns of this intere sting phen omenon. 96



### 66. Critics of CRU have suggested that Profe ssor Jones’s use of the words “hide the decline” is evidence that he was part of a conspiracy to hide evidence that did not fit his view that recent global warming is predom inantly caused by human activity. That he has published papers—including a paper in Nature —dealing with t his aspect of the science clearly refutes this allegation. In ou r view, it was shorthand for the practice of discarding data known to be erroneous. We expect that this is a ma tter the Scientific Appraisal Panel will address.

Perverting the peer review process

67. The main allegations on the suppression or distortion of others’ findings concern the role of CRU in the operat ion of the peer review pr ocess. It has been alle ged that scientists at CRU abused the peer review pr ocess to prevent those with dissenting views on climate change the opportunity in getti ng papers published. There are three key accusations. First,

90 Q 125

91 Q 122

92 Q 122; Keith Briffa and others, “Reduced sensitivity of recent tree-growth to temperature at high northern latitudes”, Nature , vol 391 (1998), pp 678-82

93 For example: Edward Cook, Paul Krusic and Phil Jones, “Dendroclimatic signals in long tree-ring chronologies from the Himalayas of Nepal”, International Journal of Climatology , Vol 23 (2003), pp 707-32

94 Keith Briffa and others, “Trees tell of past climates: but are they speaking less clearly today?”, Philosophical Transactions of the Royal Society of London Series B-Biological Sciences , vol 353 (1998), pp 65-73

95 Q 124

96 Ev 19, para 3.5.522 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia



### David Holland, an auth or of several FOIA re quests that were ment ioned in the leaked emails, claimed that climate scientists at CRU corr upted the IPCC process:

The emails show that a group of influential climate scientists co lluded to subvert the peer-review process of the IPCC and scienc e journals, and thereby delay or prevent the publication and assessment of research by scientis ts who disagreed with the group’s conclusions about global warming. They manu factured pre-determined conclusions through the corru ption of the IPCC process and deleted procedural and other information hoping to avoid its disclosure under freedom-of-information requests. 97



### 68. In one e-mail, Professor Jones appeared to su ggest that he and an other scientist would deliberately try to “keep out” two papers from the IPCC’s Fourth Assessment Report. 98



### 69. The second is that climate scientists tried to suppress a paper on research fraud. As Dr Benny Peiser, Director of the Global Warming Policy Fo undation, put it:

The CRU e-mails under invest igation suggest that climate scientists (not only at CRU but also elsewhere) have actively soug ht to prevent a paper on alleged research fraud from being published in violation of principles of academic integrity. 99



### 70. The third allegation is made by Dr Sonj a Boehmer-Christiansen, a former peer reviewer for the IPCC, editor of the journal, Energy & Environment , and Reader Emeritus

97 Ev 115, para 2

98 www.eastangliaemails.com

99 Ev 164, para 2

From: Phil Jone s <p.jones@xxxxxxxxx.xxx> To: "Michael E. Mann" <mann@xxxxxxxxx.xxx> Subject: HIGHLY CONFIDENTIAL Date: Thu Jul 8 16:30:16 2004

Mike,

Only have it in the pdf form. FYI ONLY - don’t pass on. Relevant paras are the last

2 in section 4 on p13. As I said it is worded carefully due to Adrian knowing Eugenia for years. He knows the’re wrong, but he succumbed to her almost pleading with him to tone it down as it might affect her proposals in the future !

I didn’t say any of this, so be careful how you use it - if at all. Keep quiet also that you have the pdf. The attachment is a very good paper - I’ve been pushing Adrian over the last weeks to get it submitted to JGR or J. Climate. The main re sults are great for CRU and also for ERA-40. The basic message is clear - you have to put enough surface and sonde obs into a model to produce Reanalyses. The jumps when the da ta input change stand out so clearly. NCEP does many odd things also around sea ice and over snow and ice. The other paper by MM is just garbage - as you knew. De Freitas again. Pielke is also losing all credibility as well by replying to the mad Finn as well - frequently as I see it. I can’t s ee either of these papers being in the next IPCCreport. Kevin and I will keep Them out somehow - even if we have to redefine what the peerreview literature is !The disclosure of climate data from the Climatic Research Unit at the University of East Anglia 23



### at Hull University, who stated in her memora ndum that her journal became the focus of attacks from CRU scientists:

As editor of a journal wh ich remained open to scie ntists who challenged the orthodoxy, I became the target of a numb er of CRU manoeuvres. The hacked emails revealed attempts to ma nipulate peer review to E&E’s disadvantage, and showed that libel threats were considered against its editorial team. Dr Jo nes even tried to put pressure on my university department. The emailers expressed anger over my publication of severa l papers that questioned the ‘hockey stick’ graph and the reliability of CRU te mperature data. The de sire to control the pe er review process in their favour is expressed se veral times. [... ] CRU clearly disli ked my journal and believed that “good” climate scientists do not read it. 100



### 71. When we asked Professor Jones about these accusations, he con tested each of them.

• 

### On the claim that he tried to keep two papers out of the IPCC report, he explained that the papers were alread y published and that “I was just commenting that I did not think those papers were very good”. 101

• 

### On the claim by he tried to su ppress papers that alleged research fraud, he told us:

Dr Benny Peiser [...] was editing a series of papers in Energy & Environment . He asked me to comment on a pa rticular paper and I sent hi m some views back that I did not think the paper was very good. It was not a formal review, he wa s just asking me for my views. 102

• 

### On the claims made by Dr Bo ehmer-Christiansen, he noted: “I was sending an email to the head of department ab out a complaint that she had made about me to the UK Climate Impacts Programme, so I was just responding there”. 103



### 72. In summary, Profe ssor Jones argued:

I do not think there is anything in those emai ls that really supports any view that I or CRU have been trying to pervert the peer re view process in any w ay. I have just been giving my views on specific papers. 104



### 73. The evidence that we have s een does not suggest that Pro fessor Jones was trying to subvert the peer review proce ss. Academics shou ld not be critic ised for making informal comments on acad emic papers. The Independen t Climate Ch ange Email Review should look in detail at al l of these claims.

100 Ev 125, paras 4.1–4.3

101 Q 154

102 Q 157

103 As above

104 Q 15924 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia



# 3 Freedom of information issues

74. We are not a tribunal reviewi ng whether breaches of the Freedom of Information Act 2000 (FOIA) have taken place but see as our role in this inqu iry as considering whether:

(a) the arrangements for examining whet her CRU breached FOIA are adequate;

(b) whether the six-month time limit on th e initiation of a prosecution where a public authority acts so as to prevent in tentionally the disclo sure of requested information needs to be revised; and

(c) whether UEA ensured that CRU was ab le to meet the re quirements of the legislation when it re ceived FOIA requests.



### Freedom of Inform ation legislation

75. The FOIA creating new rights of access to information came into operation on 1 January 2005. CRU, as part of UEA, is classed as a “public authority” for the purposes of the FOIA. In his submission Richard Thomas, who was In formation Commissioner from 2002 until June 2009, ex plained the applicati on of the FOIA to scient ific data held by UK
universities:

the public must be satisfied that publicly-funded universit ies, as with any other public authority in re ceipt of public fundin g, are properly acco untable, adopt systems of good governance and can inspire public trust and confidence in their work and operations [...] The fact that the FOIA reques ts relate to complex scientific data does not detract from this proposit ion or excuse non-compliance. 105



### 76. When he gave oral evidence, we asked Mr Thomas if the le gislation drew a distinction between, on the one hand, scientific da ta and modelling an d, on the other hand, administrative reco rds. He replied:

the broad answer [...] is no [...] First of all, the legi slation applies to information held by the public authority, and info rmation is not elaborated in that sense. [...] It is not ownership. The legislation uses the wo rd “held”, and in the Environmental Information Regulations [EIR] th at phrase “held” is slightly elaborat ed. If I can quote the regulation for you there, “It is held by a public authority if th e information: (a) is in the authority’s possession and has been produced or re ceived by the authority, or (b) is held by another person on behalf of the authority.” So that is an elaboration of the concept of “held”. It is not ownership. 106



### 77. Mr Thomas considered that the issues in this case which were most relevant to the information law appeared to be:

(a) the relevance and impact of the information laws on scientific and academic research conducted within universities;

105 Ev 8, para 3.2

106 Qq 59–60The disclosure of climate data from the Climatic Research Unit at the University of East Anglia 25



### (b) the adequacy of section 77 of FOIA to deal with su ggestions that CRU researchers deleted inform ation, not in the course of normal work, but to frustrate FOIA/EIR 107 

### requests;

(c) the handling of a large number of FOIA/EIR requests by UEA relating especially to climate change research which (within CRU) it “held”; and

(d) whether this case illus trates that there is scope to extend the “proactive” disclosure provisions of FOIA as they relate to universities. 108



### 78. Parliament has created a presumption in favour of disclosure but there are exclusions. 109 

### Mr Thomas explained:

There are over 20 exemptio ns to the fundamental duty to disclose requested information in FOIA.[...] Eig ht of the main exemptions are absolute and 16 are qualified. Qualified means th at there is a “public intere st override,” which means that, even where the exemption applies, th e public interest cons iderations must be considered. In formal term s, there must still be di sclosure—even though the qualified exemption applies—u nless the public interest in the exemption outweighs the public interest in disclosure.

Mr Thomas added that:

The exemptions are similar to those found in other Freedom of Information laws in force in the world. I am not aware wh ich exemptions were considered by the University as potentia lly applicable to some or all of the requests to CRU. I can speculate that some or al l of the following [...] m ight have been considered:

(a) Section 22—where the requested info rmation is intended for future (but imminent) publication;

(b) Section 40—where disclosure of person al data would breach any of the data protection principles;

(c) Section 41—where the information had been obtained from elsewhere in such circumstances that its disclosure would constitute an actionable breach of confidence unde r common law;

(d) Section 43 (qualified)—where disclo sure would, or would be likely to, prejudice the commercial interests of any person, including the public authority;

(e) Section 44—where disclosure is prohibited by another enactment or inconsistent with an EU obligation (which may in clude some intellectual property restrictions); and

107 EIR: Environmental Information Regulations 2004. Deriving from European Directive 2003/4/EC these give rights of public access to environmental inform ation held by public authorities.

108 Ev 8, para 2.2

109 Ev 9, para 3.626 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia



### (f) Section 14 (not an exemption, strict ly speaking)—where the request is
vexatious. 110



### 79. We were grateful to Mr Thomas for explaini ng the operation of the FOIA and EIR. He did, however, point out that he did not have detailed knowledge of even ts at UEA since leaving the Information Commissioner’s Office:

I have no idea at all what has happened inside my former o ffice. I cannot say because this is a serious matter. It depends a great deal on the ci rcumstances of the particular case, the evidence. I have had no direct contact with the office as to how this case is being handled. 111



### Alleged breaches of the Freed om of Information Act 2000

The e-mails

80. Some of the hacked e-mails ap pear to reveal scientists en couraging their colleagues to resist disclosure and to delete e-mails, apparently to prevent them from being revealed to people making FOIA requests. Below are examples, in chronological order, of e-mails sent by Professor Jones which address FO IA and requests for information.

E-mail: 

### 1107454306 [Extract] At 09:41 AM 2/2/2005, Phil Jones wrote: Mike,[...]Just sent loads of station data to Scott. Make sure he documents everything better this time! And don’t leave stuff lying around on ftp sites - you never know who is trawling them. The two MMs have been after the CRU station data for years. If they ever hear there is a Freedom of Information Act now in the UK, I think I’ll delete the file rather than send to anyone. Does your similar act in the US force you to respond to en quiries within 20 days? - our does ! The UK works on precedents, so the first request will test it. We al so have a data protection act, which I will hide behind. Tom Wigley has sent me a worried email when he heard about it - thought people could ask him for his model code. He has retired officially from UEA so he can hide behind that. IPR should be relevant here, but I can see me getting into an argument with so meone at UEA who’ll say we must adhere to it !. [...]

E-mail: 1219239172 [Extract] From: Phil Jones <p.jones@xxxxxxxxx.xxx> To: Gavin Schmidt <gschmidt@xxxxxxxxx.xxx> Subject: Re: Revised version the Wengen paper Date: Wed Aug 20 09:32:52 2008

[...] Keith/Tim still getting FOI re quests as well as MOHC and Reading. All our FOI officers have been in discussions and are now using the same exceptions not to respond - advice they got from the Information Commissioner. As an aside and just between us, it seems that Brian Hoskins has withdrawn himself from the WG1 Lead nominations. It seems he doesn’t want to have to deal with

110 Ev 9, para 3.7

111 Q 58The disclosure of climate data from the Climatic Research Unit at the University of East Anglia 27

this hassle.

The FOI line we’re all using is this. IPCC is exempt from any countries FOI – the Sceptics have been told this. Even though we (MOHC, CRU/UEA) possibly hold relevant info the IPCC is not part our remit (mission statement, aims etc) therefore we don’t have an obligation to pass it on. Cheers
Phil

E-mail: 1228330629 From: Phil Jones <p.jones@xxxxxxxxx.xxx> To: santer1@xxxxxxxxx.xxx, Tom Wigley <wigley@xxxxxxxxx.xxx> Subject: Re: Schles suggestion Date: Wed Dec 3 13:57:09 2008 Cc: mann <mann@xxxxxxxxx.xxx>, Gavin Schm idt <gschmidt@xxxxxxxxx.xxx>, Karl Taylor <taylor13@xxxxxxxxx.xxx>, peter gleckler gleckler1@xxxxxxxxx.xxx

Ben, When the FOI requests began here, the FOI person sa id we had to abide by the requests. It took a couple of half hour sessions - one at a screen, to convince them otherwise showing them what CA was all about. Once they became aware of the type s of people we were de aling with, everyone at UEA (in the registry and in the Environmental Sc iences school - the head of school and a few others) became very supportive. I’ve got to know the FOI person quite well and the Chief Librarian - who deals with appeals. The VC is also aware of wh at is going on - at least for one of the requests, but probably doesn’t know the number we’re dealing with. We are in double figures.

One issue is that these requests aren’t that wide ly known within the School. So I don’t know who else at UEA may be getting them. CRU is moving up the ladder of requests at UEA though - we’re way behind computing though. We ’re away of requests going to others in the UK - MOHC, Reading, DEFRA and Imperial Coll ege. So spelling out all the detail to the LLNL management should be the first thing you do. I hope that Da ve is being supportive at PCMDI. The inadvertent email I sent last month has led to a Data Protection Act request sent by a certain Canadian, saying that the email maligned his scientific credibility with his peers!

If he pays 10 pounds (which he hasn’t yet) I am supposed to go through my emails and he can get anything I’ve written about him. About 2 months ago I deleted loads of emails, so have very little - if anything at all. This legislation is different from the FOI - it is supposed to be used to find put why you might have a poor credit rating ! In response to FOI and EIR requests, we’ve put up some data - mainly paleo data. Each request generally leads to more - to explain what we’ve put up. Every time, so far, that hasn’t led to anything being added - in stead just statements saying read what is in the papers and what is on the web site! Tim Osborn sent one such response (via the FOI person) earlier this week. We’ve never sent programs, any codes and manuals.

In the UK, the Research Assessment Exercise results will be out in 2 weeks time. These are expensive to produce and take too much time, so from next year we’ll be moving onto a metric based system. The metrics will be # and amounts of grants, papers and citations etc. I did flippantly suggest that the # of FOI requests you get should be another.

When you look at CA, they only look papers from a handful of people. They will start on another coming out in The Holocene early next year. Gavin and Mike are on this with loads of others. I’ve told both exactly what will appear on CA once they get access to it! Cheers28 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia

Phil

E-mail: 1237496573 [Extract] From: Phil Jones <p.jones@xxxxxxxxx.xxx> To: santer1@xxxxxxxxx.xxx Subject: Re: See the link below Date: Thu Mar 19 17:02:53 2009

[...] CRU has had numerous FOI requests since the beginning of 2007. Th e Met Office, Reading, NCDC and GISS have had as well – many related to IPCC involvement. I know the world changes and the way we do things changes, but these requ ests and the sorts of simple mistakes, should not have an influence on the way things have been adequately dealt with for over a century. Cheers
Phil



### 81. In his submission Andrew Montford stated that:

Research materials should be made available to outsiders as a requirement of the scientific method. That scientis ts have failed to do so is reprehensible, but the fact that they have apparently also resorted to breaches of the Freedom of Information Act in order to do so requires urgent atte ntion from policymakers. 112



### 82. As we explained in the previous chapter, David Holland was the au thor of several FOIA requests that were mentioned in the leaked e- mails. In his submission he pointed out that on 9 May [2008] in e-mail 1210367056, Prof essor Jones sent “my formal information request to ‘team’ members Mann , Hughes and Ammann” writing:

You can delete this attachment if you want. Keep this quiet also, but this is the person who is putting in FOI requests for all emails Keith and Tim have written and received re Ch 6 of AR4. 113 

### We think we’ve foun d a way around this. 114



### 83. Mr Holland also drew attention to e-ma il 1212063122 dated 29 May 2008 in which Professor Jones aske d Professor Mann:

Can you delete any emails you may have had with Keith re AR4? Keith will do likewise. Can you also email [Eu]Gene [Wahl ] and get him to do the same? I don’t have his new email address. We will be getting Caspar [Ammann] to do likewise. 115



### Correspondence with the Deputy Information Commissioner

84. On 22 January 2010 , when the Deputy Information Commissio ner, Graham Smith, issued a statement which sugges ted that at l east some of the requested in formation should

112 Ev 159, para 6

113 Intergovernmental Panel on Climate Change: Fourth Assessment Report

114 Ev 117, para 23

115 Ev 118, para 32The disclosure of climate data from the Climatic Research Unit at the University of East Anglia 29



### have been disclosed in the absence of applicable exemptions, it ga ve support to the criticisms of CRU’s handling of FOIA requests. Mr Smith said:

The FOI Act makes it an of fence for public auth orities to act so as to prevent intentionally the disclosure of requested information. Mr Holla nd’s FOI requests were submitted in 2007/8, but it has only recently come to ligh t that they were not dealt with in accordance with the Act. Th e legislation requires action within six months of the offenc e taking place, so by the time the action c ame to light the opportunity to consider a prosecution was long gone. 116



### 85. Mr Thomas commented that this was “clearl y a reference to section 77 of the Act and/or the near-identical Regulation 19 of EIR”. 117 

### Section 77 of the FOIA provides:

1. Where:

(a) a request for information has be en made to a pu blic authority,

(b) under section 1 of this Act or sectio n 7 of the Data Prote ction Act 1998, the applicant would have been en titled (subject to pa yment of any fee) to communication of any information in accordance with that section,
any person to whom this subsec tion applies is guilty of an of fence if he alters, defaces, blocks, erases, destroys or conceals any re cord held by the public authority, with the intention of preventing the dis closure by that au thority of all, or any part, of the information to the communicati on of which the applicant would have been entitled.
2. Subsection (1) applies to the public author ity and to any person who is employed by, is an officer of, or is subject to the direction of, the public authority.
3. A person guilty of an offence under this section is liable on summary conviction to a fine not exceeding level 5 on the standard scale. 118



### 86. Mr Thomas added that the Deputy Commiss ioner also appeared “to have in mind” section 127(1) of the Magistrates Co urt Act 1980, which provides that

a magistrates’ court shall not try an info rmation or hear a co mplaint unless the information was laid, or the complaint made, with in 6 months from the time when the offence was commi tted, or the matter of complaint arose. 119



### Mr Thomas confirmed in oral evidence that

because of the interaction with the Magist rates Court Act, any prosecution must be brought within six months of the offence being committed. 120



### 87. In its memorandum to our inquir y, UEA defended its actions:

116 Ev 9, para 4.1

117 Ev 10

118 Ev 10, para 4.1

119 Ev 10, para 4.2

120 Q 5630 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia



### CRU has been accused of re fusing to release data reques ted under the FOIA. There are many obstacles outside CRU’s control surrounding th e release of data provided by NMSs [National Meteoro logical Services]. Many FO IA requests made to CRU related to primary data provided by the NM Ss. Some of these da ta are subject to formal non-publication agreem ents between the NMS and CRU. Ot her primary data had been provided to CRU on an individual-to-in dividual basis, wi th accompanying verbal agreements that they may be used within the gridded dataset, but should not be passed on to others. CRU responded to the FOIA requests for primary data by pointing out that approximately 90% of the stations in the CRU dataset are available from other sources, particularly GHCN. 121



### 88. On 29 January there was an exchange between UEA and Mr Smith, the Deputy Commissioner. Brian Summers, th e Registrar and Secr etary of UEA responded forcibly to Mr Smith’s 22 January press stat ement, which asserted that UEA had not dealt with FOIA requests “as they should have been under the legislation”. 122 

### He did not co nsider it was “acceptable that such a statem ent which has led to an extrem ely damaging co mmentary on the University [was] first communicated to the Univer sity by a journalist”. 123 

### His letter goes on to defend UEA’s actions in detail and to ask that, if the Information Commissioner’s Office (ICO) cannot retract the 22 January stat ement, it issue a clarification regarding the alleged breaches of the FOIA. A response from the ICO was issued the same day. It did not retract the original statemen t but offered clarification:

1. [No] decision notice has yet been issued and no alleged brea ches have yet been put to the University for comment. That matter has yet to be addressed, but it will be over coming months.

2. The fact that the elements of a section 77 offence may have been found here, but cannot be acted on because of the elapsed time, is a ve ry serious matter. The ICO is not resiling from its position on this.

3. The ICO’s position is as st ated in point 2 above. Th e statement may be read to indicate that. 124 

### Under section 77, an offence may be committed by an individual, not necessarily the pu blic authorit y itself.

4. Errors like this are frequen tly made in press report s and the ICO cannot be expected to correct them, particularly wh en the ICO has not itself referred to penalties or sanctions in its own statement. 125

121 Ev 20, para 3.7.2

122 “Scientists in stolen e-mail scandal hid climate data”, The Times , 28 January 2010

123 Registrar and Secretary to Deput y Information Commissioner - 29 January 2010, UEA website, Correspondence between University of East Anglia and the Information Commissioner’s Office, www.uea.ac.uk/mac/comm/media/press/ CRUstatements/ICOcorrespondence

124 UEA had asked the Deputy Commissioner to confirm that “your state ment cannot be taken to mean that there has been a demonstrable breach of Section 77, which is a breach of the FOI which can result in prosecution”; Registrar and Secretary to Deputy Information Commissioner, 29 January 2010, UEA website, Correspondence between University of East Anglia and th e Information Commissioner’s Office, www.uea.ac.uk/mac/comm/media/pres s/CRUstatements/ICOcorrespondence

125 Deputy Information Commissioner to Registrar and Secretary - 29 Januar y 2010, UEA website, Correspondence between University of East Anglia and the Information Commissioner’s Office, www.uea.ac.uk/mac/comm/media/pres s/CRUstatements/ICOcorrespondenceThe disclosure of climate data from the Climatic Research Unit at the University of East Anglia 31



### 89. UEA responded on 1 February thanking the IC O for the clarificatio n but setting out its concerns relating to the press covera ge of the ICO’s original statement:

Your clarification that the press cannot infer from your statement to the Sunday Times that it has been es tablished that the University (or in deed any individual associated with the University) has br eached the terms of the Freedom of Information Act is welcome. [UEA’s] reputa tion which has been subjected to these damaging and incorrect assertions claiming to be based on your statement and we must take some steps to put this right. We will be writing to the media which carried reports based on your statement, pointing out the inaccuracies and asking them to rectify the position. 126



### 90. In his oral evidence Professor Acton questi oned the ICO statement of 22 January:

our principle is that prima facie evidence is evidence whic h on the face of it and without investigation suggests that there is a case to answer. To my mind if there is prima facie evidence; why did I set up the Muir Russell independent review? Prima facie evidence is not the same as, you have been found to breach. [...] If it is sub judice, if, as we had in the lett er ten days ago from the IC O, the investigation has not even begun, I am puzzled how we could have been found to breach if there has been no investigation. 127



### 91. The ICO’s most recent letter, dated 3 March, in UEA’s view, “makes plain that there is no assumption by the ICO, prior to investigati on, that UEA has breached the Act; and that no investigation has yet been completed.” 128 

### The ICO’s letter confirmed that the “ICO is not pursuing any investig ation under section 77 of the Act. That matter is closed as far as the ICO is concerned, given the statutory time limits for action”. It added that:

The ICO acknowledges your concern about the statement made and the subsequent media and blog reports. G iven that the Deputy Commi ssioner has already been publicly associated with the matte r, any Decision Notice w ill be reviewed and signed off by another authorised signatory. 129



### We regret that the ICO made a statement to the press that went be yond that which it could substantiate and that it took over a month for the ICO properly to put the record straight. We recommend that the ICO develop procedures to ensure that its public comments are checked and that mechanisms exist to swiftly correct any mis-statements or misinterpretations of such statements.

92. The disclosed e-mails appear to show a culture of non-disclosure at CRU and instances where information (disclosable or otherwise) may have been deleted, to avoid disclosure. The Deputy Information Commissioner’s lett er of 29 January gives a clear indicati on that a

126 Registrar and Secretary to Deputy Information Commissioner - 1 February 2010, UEA website, Correspondence between University of East Anglia and the Information Commissioner’s Office, www.uea.ac.uk/mac/comm/media/pres s/CRUstatements/ICOcorrespondence

127 Q130

128 Ev 39, para A

129 Ev 39, annex32 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia



### breach of the FOIA may have occurred but that a prosecution was time-barred. 130 

### As, however, UEA pointed ou t, no investigation ha s been carried out.

93. It seems to us that both sides have a point. There is prima facie evidence that CRU has breached the Freedom of Inform ation Act 2000. It would, however, be premature, without a thorough investig ation affording each party the opportunity to make representations, to conclude that UEA was in breach of the Act. In our view, it isunsatisfactory to leave the matter unresolved simply because of the operation of the sixmonth time limit on the init iation of prosecutions. Much of the reputation of CRU hangs on the issue. We conclude that the matter needs to be resolved conclusively— either by the Independent Climate Change Email Review or by the Information
Commissioner.

94. On the question of the six-mo nth time limit on the initiation of prosecutions, Mr Thomas pressed for a revision of the law. He pointed out that apart from in the most blatant cases “it will usua lly be impossible for the ICO to detect an of fence within 6 months of its occurrence” and thus to be able to initiate a prosecution. 131 

### He drew attention to a recent debate in the House of Lords on a proposal to amend the time limit. In reply, in the debate the Parliamentary Under-Secretary of State at the Mi nistry of Justice said that:

The Freedom of Inform ation Act 2000 came in to force only in 2005 , and [...] we have no evidence at present that the current six-month time limit presents a systemic problem for the Information Commissioner or any other prosecutor in taking action under Section 77. [...] We w ill listen to the vi ews of the Informat ion Commissioner and other interested pa rties on this point, and if there is evide nce that the current legislation is causing systemic difficulties, we will look for ways to addr ess the matter, if necessary by means of an alternative legislative veh icle in the future. However, I cannot go further than that to day on behalf of the Government. 132



### No change was made to the legislation.

95. We consider that events at CRU throw lig ht on the operation of the Freedom of Information Act 2000 and, in pa rticular, whether there is a ne ed to amend the time limit on prosecutions from six months from th e time the alleged of fence was committed. If the
Minister was correct to assert in July 2009 that the Government had no evidence that the current six-month ti me limit presents a systemic prob lem, then it is now clear that such evidence exi sts. Irrespective of whether or not CRU breached the Freedom of Information Act 2000, we re commend that the Government review the operation of section 77 of the 2000 Act and the six month limit on the initiati on of prosecutions provided by section 127(1) of the Magistrates Court Act 1980.

130 UEA website, Correspondence between University of East Ang lia and the Informatio n Commissioner’s Office, www.uea.ac.uk/mac/comm/media/pres s/CRUstatements/ICOcorrespondence

131 Ev 10, para 4.3

132 HL Deb, 21 July 2009, col 1571The disclosure of climate data from the Climatic Research Unit at the University of East Anglia 33



### Volume of requests

96. In the face of allegations of poor handling of FOIA requ ests, one of the explanations offered by UEA was that in:

July 2009 UEA received an unpreced ented, and frankly administratively overwhelming, deluge of FOIA requests related to CRU. These amounted to 61 requests out of a 2009 total of 107 related to CRU, compare d to annual totals of 2 in 2008 and 4 in 2007 (University totals for those ye ars were 204, 72 and 44 respectively). 133



### 97. At the oral evidence se ssion Lord Lawson commented on the increase in the volume of FOIA requests:

what had happened was ther e had been a very, very small number of FOI Act requests to begin with and it was in response to those th at there was all the evasion, the lack of disclosure and all the other th ings which we have seen in the emails: discussions about possibly dest roying evidenc e and so on. All that came well before the 2009 flood of stuff. The 20 09 flood, if you look at th e sequence of events, was a response to the refusal to give disclosure of various thin gs before. That was what came first. 134



### 98. There are two issues here: the adequacy of CRU’s handling of the FOIA requests and whether the increase in the numb er of requests in July 2009 was a deluge. On the latter, Mr Thomas said that, whilst agreeing that UEA had faced a significant rise in FOIA requests in July 2009, he did not consider that a total of 61 was a “huge number”. 135



### 99. On handling, CRU claimed that it could not cope with the significant rise in FOIA requests because it only had three full-time academic staff. 136 

### We therefore wrote to UEA on 2 March 2010 to ask what ex tra resources were provided to assist CRU cope with these requests. UEA responded that:

additional support was provide d to the Univer sity’s Information Policy Compliance Manager (IPCM) who handles FOI requests. This included resc heduling workloads to allow him to concentrate on the CRU FOI requests and diverting secretarial support to provide additional resource. Given the high vo lume of requests received, the Director of Information Ser vices (DoIS) also took an ac tive role in the first stage of a number of requests, th us providing additional supp ort to the IPCM. (Should any cases where the DoIS was directl y involved in the fi rst stage be appealed then we have arranged for the PVC Academic to adjudicate to ensure impartiality). ISD also fasttracked the merging of the Security Policy and Complia nce team to ensure that a fully trained back-up to the IPCM was available. 137

133 Ev 20, para 3.7.4

134 Q 9

135 Q 68

136 Q 92 [Professor Acton], Ev 20, para 3.7.4; Ev 37, Q 1

137 Ev 37, para 134 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia



### 100. The Science Facul ty also provided additio nal administrative suppo rt, including that of the Director of Faculty Administration, th e most senior member of the Faculty’s administrative staff. UEA pointed out that many of the requests were of a very technical nature and:

required scientific knowledge and understanding of the subject area in order to provide the details. Despite the addition al administrative resources provided, the requirement to respond to the 61 requests received in July 2009 impacted considerably upon the work of CRU. 138



### 101. We also asked UEA to outline what legal advice and guidance on handling had been offered to CRU in handling these FOIA requests. UEA confirmed that the:

IPCM provided advice to CRU on the requirem ents of the Act both generally, and in relation to any app licable sections, exemptions or exceptions pertaining to the specific request. In this latte r role, the IPCM set out the requirements of any possible exemption or exception, incl usive of the public in terest test, and elicited from CRU staff whether the public interest test had been met. Additio nal advanced training was provided to the ‘FOI Contact’ for the Faculty of Science, the Dire ctor of Faculty Administration. In this role, the FOI co ntact acted as a support to CRU in the location and retrieval of information and provided as sistance to th e IPCM in exploring the application of the Act to the specific requests. 139



### 102. On the evidence we took we have concerns about the handling of FOIA requests by CRU. First, the disclosed e-mai ls betray an attitude to fr eedom of infor mation that was antipathetic to the spirit of disclosure in the legislation. Mr Thomas pointed out that:

the simplest approach, particularly where requ ests tend to generate either a defensive attitude or place a great burd en on the public authority, is proactive disclosure in the first place.[...] Public authorit ies ought to decide what really has to be kept away from the public. If it is partic ularly sensitive or there is a good reason for withholding it, fair enough, but where there is no good reason for withholding information, then why not proactively disclose it and avoid the hassle of large numbers of requests? 140



### 103. Whether or not CRU liked it, those making FOIA requests were entitled to have their requests dealt with in accordance with the legi slation and, if the info rmation sought did not fall within one of the exclus ions provided by the FOIA, it should have been disclosed. We have already recommended in paragraph 54 above that in future information, including data and methodology, should be published proactively on the internet wherever possible. Howeve r, a culture of withholding information—from those perceived by CRU to be host ile to global warming—appears to have pervaded CRU’s approach to FOIA requests from the outset. We consider this to be unacceptable.

104. In the face of such an un helpful approach we are not su rprised that FOIA requests multiplied. When the surge in FOIA requests hit CRU in July 2009 UEA provided extra

138 Ev 37, para 1

139 Ev 37, para 2

140 Q 70The disclosure of climate data from the Climatic Research Unit at the University of East Anglia 35



### resources but because of their technical nature the same small group of staff at CRU had a pivotal role in handling th e requests. We are not clea r that the cu lture changed. We cannot reach a firm conclusion on the basis of the evidence we took but we must put on record our concern about the manner in which UEA allowed CRU to handle FOIA requests. Further, we found prima facie evidence to suggest that th e UEA found ways to support the culture at CRU of resisting disclosure of information to climate change sceptics. The failure of UEA to grasp fully the pote ntial damage to CRU and UEA by the nondisclosure of FOIA requests was regrettabl e. UEA needs to review its policy towards FOIA and re-assess how it can support academics whose e xpertise in this area is limited.36 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia



# 4 Independent inquiries

105. There are two reviews underway: the Indepe ndent Climate Change Email Review led by Sir Muir Russell; and a scie ntific assessment panel revi ewing CRU’s ke y scientific publications. The Vice-Chan cellor explained to us in oral evidence on 1 March 2010 that the reviews would focus on different matters:

Muir Russell’s independent revi ew is not looking at the science, it is looking at allegations about malpractice. As for the science itself, I have not actually seen any evidence of any flaw in the science but I am hoping, later this we ek, to announce the chair of a panel to reassess the science and make sure there is nothing wrong. 141



### In the event the announcement was not made un til 22 March.



### The Independent Climate Change Email Review

106. The Independent Climate Chan ge Email Review is being conducted by a team, led by Sir Muir Russell. Accord ing to the Review’s website the te am has more t han 100 years’ collective expertise of scient ific research methodology and a wide range of scientific backgrounds. None have any links to the Climat ic Research Unit, or the United Nations’ Intergovernmental Panel on Climate Change (IPCC). 142



### Terms of reference

107. The Review’s terms of refe rence are as follows:

The Independent Review will in vestigate the key allegations that arose from a series of hacked e-mails from the University of Ea st Anglia’s Climatic Research Unit (CRU). The re view will:

1.1. Examine the hacked e-mail exchanges, other relevant e-mail exchanges and any other information held at CR U to determine whether ther e is any evidence of the manipulation or su ppression of data which is at od ds with acceptab le scientific practice and may therefore call into que stion any of the research outcomes.

1.2. Review CRU’s poli cies and practices for acquiring, assem bling, subjecting to peer review and di sseminating data and research findings, and their compliance or otherwise with best scientific practice.

1.3. Review CRU’s co mpliance or otherwise with the University’s policies and practices regarding requests under the Fr eedom of Information Act (‘the FOIA’) and the Environmental Information Regulations (‘the EIR’) for the release of data.

141 Q 129

142 www.cce-review.org/About.phpThe disclosure of climate data from the Climatic Research Unit at the University of East Anglia 37



### 1.4. Review and make recommendation s as to the appro priate management, governance and securi ty structures for CRU and the security, integrity and release of the data it holds. 143



### 108. Sir Muir has discretion to amend or add to the terms of reference if he feels necessary, devise his own methods of working, and call on appropriate expertise, in order to investigate the allegations fully . UEA has asked for the Review to be completed by Spring 2010 and this will be made pub lic along with UEA’s response. 144



### 109. Lord Lawson, in both his wr itten submission and his oral evidence, considered that the terms of refe rence “may be a bit too CRU-centric” 145 

### and “needed to be extended to include more fully the issue of the dissenting scientists”. 146 

### These points were echoed in written submissions to us. Andrew Montford suggested that:

The independence of the review is not assured. Sir Muir Russell was appointed to head the review by the vice-chancellor of th e University of East Anglia, [...] Edward Acton. However, the emails disclosed implicate [his] pr edecessor in an apparent breach of the Freedom of Information Act and there is therefore a prime-facie case that the review is not sufficiently independen t. [...] The review must take evidence from sceptics. At time of writing it appe ars that no prominent sceptic has been contacted by Sir Muir with a view to providing evidence. Without complainants being able to make thei r case to the review, it is unli kely that the fi ndings will be sound or accepted by the sceptic community. 147



### Mike Haseler, creator of the Nu mber 10 Petition regarding the CRU, was also critical of the Review saying that it “seems to serve no real purpose except the PR of the University to appear to be doing something.” 148



### 110. Others offered amendments to the terms of refe rence. Professor Ross McKitrick, a professor of environmental economics, recomm ended that the terms of reference “should consider whether CRU scientis ts whose respon sibilities include providi ng climate data to the IPCC should not serve as IPCC Lead Authors (or Coordina ting Lead Authors) on any Report or Chapter that assesse s evidence for or against it s quality for cli matic research purposes.” 149



### 111. The Royal Society of Chemistry conside red the terms of re ference “adequate” 150 

### and Professor John Bedding ton suggested that th ey “give sufficient scop e for the issue to be investigated in full”. 151 

### Professor Peter Cox, a former lead author on the last IPCC Working

143 Ev 39

144 “Sir Muir Russell to head the Independent Review into the allegations against the Climatic Research Unit (CRU)” UEA Press Release, 3 December 2009, www.uea.ac .uk/mac/comm/media/pre ss/2009/dec/ CRUreview

145 Q 5, Ev 1, annex containing lett er dated 26 January 2010 from th e Foundation to Sir Muir Russell ( not printed )

146 Q 3

147 Ev 161, paras 22 and 24

148 Ev 139, para 27

149 Ev 140, para 3.2

150 Ev 172, para 12

151 Ev 45, para 738 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia



### Group, suggested t hat the “Inquiry should hear evidence on the re viewing of scientific papers and the exclusion of paper s from the IPCC report. It w ill be critical to determine whether these decisions were carried out on the basis of scientific merit alone”. 152



### 112. In response to criticisms Sir Muir pointed out that the re view “is not ac tually about the big science of global wa rming and making forecasts fo r the next hundred years”. 153 

### He said that “it will not be window dressing”, and UEA had “not interfered at all”. 154



### 113. We accept the assurances that Sir Muir Russell has given about the independence of the Independent Climate Ch ange Email Review and we expec t him to be scrupulous in preserving its impartiality. We see no reason why the Review’s conclusions and UEA’s response have to be published togethe r. Indeed, it could give the impression that UEA was being given an advantage when it co mes to responding. We consider that the Review’s conclusions and recommendations should not be conveyed to UEA in advance of publication.

114. With regards to the terms of reference of the Review, we consider that as well as measuring CRU against current acceptable scientific practi ce, the Review should also make recommendations on best practice to be followed by CRU in the futu re. We invite Sir Muir Russell to respond formally to our Report to the extent that he sets out whether, on the basis of its contents, he finds the Terms of Refe rence of his inquiry need to be changed.

The Review team

115. The Review Team membership, as announced, consisted of:

Sir Muir Russell Professor Geo ffrey Boulton Dr Philip Campbell [ subsequently resigned ] Professor Peter Clarke Mr David Eyton Professor Jim Norton. 155



### 116. Sir Muir and the Review team held a press briefing at the Scie nce Media Centre in London on 11 February 2010 to announce its membership, pu blish its workplan and issue a call for submissions fro m interested parties. Al most immediately it wa s beset by claims of partiality. On the same day as the launch Sir Muir Russell accepted the resignation of Dr Philip Campbell, Editor of Chief of Nature , after a recording of an interview given by Dr Campbell to China Radio Internatio nal in December 2009 was a lleged to raise doubts over his impartiality. Dr Campbell said:

I made the remarks in good fa ith on the basis of media repo rts of the leaks. As I have made clear subsequently, I support the need for a full review of the facts behind the

152 Ev 132, para 2

153 Q 163

154 Q 166

155 Ev 40The disclosure of climate data from the Climatic Research Unit at the University of East Anglia 39



### leaked e-mails. There must be nothing that calls into question the a bility of the independent Review to complete this task, and therefore I have decided to withdraw from the team. 156



### 117. Sir Muir said “I have spok en to Philip Campbell, and I understand why he has withdrawn. I regret the loss of his ex pertise, but I respect his decision." 157 

### Further allegations arose on 12 February that Profe ssor Geoffrey Boulton’s background and views affected his ability to be a member of the Review. 158 

### These have been reje cted by Sir Muir Russell and by Professor Boulton. Professor Boulton said:

At the Review press conferen ce (on February 11), I pointed out that I had worked full-time in the School of Environmental S ciences at UEA from its inception in 1968 to 1980, and that I had a part-time appo intment between 1980 and 1986, whilst working primarily in the University of Amsterdam. Since then, I have had no professional contact with th e University of Ea st Anglia or the Climatic Research Unit. I was equally clea r that although my research is not in the field of modern or recent climate change, I am familiar with its scientif ic basis and uncertainties surrounding it. I decl ared my current view of the balanc e of evidence: that the earth is warming and that human ac tivity is implicate d. These remain the views of the vast majority of scientists who rese arch on climate ch ange in its different aspects. They are based on extensive work worldwide, not that of a single institution. As a sceptical scientist, I am prepared to change those view s if the evidence meri ts it. They certainly do not prevent me from being heavily bia sed against poor s cientific practice, wherever it arises. 159



### Sir Muir Russell said:

This Review must determ ine if there is evidence of poor scientific practice, as well as investigate allegations around the manipulati on and suppression of data. As others have pointed out, it would be impossible to find somebody with the qualifications and experience we need who has not formed an opinio n on climate change. I am completely confident that each member of the Review team has the integrity, the expertise, and the experience to complete our work impartially. 160



### 118. In his oral evidence Sir Muir outlined his approach in choosing the team:

156 “Dr Philip Campbell withdraws from the Review”, Independent Climate Change Email Review News release , 12 February 2010, www.cce-review.org/News.php

157 As above

158 There has been pressure on Pr ofessor Boulton to step down. The Scotsman reported: “Dr Benny Peizer, [ sic ] director of the Global Warming Policy Foundation, a think tank which claims the debate on climate change has become distorted, called for Prof Boulton to step down, too. He said: ‘Prof Boulton obviously is a very distinguished geologist. The problem is, he is a very outspoken cam paigner on this issue and he’s given talks calling for galvanising public opinion. He also worked at the very instit ution that he is now going to be investigating. That, we think, is a conflict of interest.’” ( “Sen ior Scots scientist in climate probe row”, The Scotsman , 13 February 2010) Sir Muir has rejected the call. (“Allegations of bias against Review member rejected”, Independent Climate Change Email Review News release , 15 February 2010)

159 “Allegations of bias against Review member rejected”, Independent Climate Change Email Review News release , 15 February 2010,www.cce-review.org/News.php

160 As above40 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia



### You can see as you look at th e composition of the team that I needed to be looking at climate science in general bu t not somebody who was asso ciated with this particular stream of work but would un derstand what was going on. There were going to be huge data handling issues, there was a lot of work on computin g and data security and so on and that the wo rk was going to have a resonance out there in the real world and around the wo rld. Really on that basis I came up with this set of names that you can see. In relation to Dr C ampbell, the others that I had got together thought that it would be extremely i mportant to have somebo dy who knew about peer review and that wa s really the qualification that brought him in. 161



### 119. It is unfortunate that th e Independent Review got off to a bad start with the necessary resignation of Dr Campbell. The qu estion of the operatio n of peer review is going to be a critical issue in the inquiry and the Review Team needs to take steps to ensure the insight and experience he would have brought are replaced.

Transparency

120. Contributors to our inquiry have suggested the importan ce that the Independent Review is open and transparent. Lord Lawson , in his oral evidence , said that he was:

concerned about the openne ss and transparen cy, [...] there sh ould be public hearings, like you are having here—I think th at is very, very important—and I regret the fact that it appears that they do not intend to do this. 162



### Andrew Montford commented:

The review must be held in public. Sir Muir Russell has stated that he wants to retain the confidence of global warming sceptics. Ho wever, in his letter to Mr Willis of 10 December 2009, [... ] the vice-chancellor of UEA, states that Si r Muir will present his findings to [him], who will in turn present a report to the council of th e university. We are asked to believe that Sir Muir will properly in vestigate [the ViceChancellor’s] role in the alleged FoI breaches, and that [he] will pass on the findings that Sir Muir makes on this subject to the university council. 163



### 121. When answering our question on transparency Sir Muir indicated that the Review team “plans to put on its website the evidence that we receive”. 164 

### When pressed on the question of holding pu blic evidence sess ions Sir Muir re sponded that:

all my predispositions and those of the fe llow team members are to do it that way [via written evidenc e] rather than to do it in a hearing of perhaps this kind or in a series of one-to-one interviews or whatev er. Where we have interviews with people in CRU or elsewhere, those w ill be written up and they wi ll be part of the record but at the moment I am not really sure that getting to th e stage of putting people in a

161 Q 160

162 Q 3

163 Ev 161, para 23

164 Q 172The disclosure of climate data from the Climatic Research Unit at the University of East Anglia 41



### hearing context is going to be a particula rly effective way of adding value to the objective evidence that we want to get our hands on. 165



### 122. We agree that the Review must be open and transparent. We conclude that, when the Independent Review holds oral he arings or interviews, they should be carried out in public wherever possible and that it should publish all the writte n evidence it receives on its website as soon as possible.



### Scientific Appraisal Panel

123. In its evidence to us the Independent Clim ate Change Email Revie w stated that its remit does not invite it to re-appraise the scie ntific work of CRU. That re-appraisal is being separately commissioned by UEA, with the assistan ce of the Royal Society. 166 

### In a statement released on 11 February UEA said that:

The Royal Society will as sist the Univ ersity in identifying asse ssors with th e requisite expertise, standing and independence. “Published pa pers from CRU have gone through the rigorous and intensive peer review process which is the keystone for maintaining the integrity of scientific research,” said Professor Trevor Davies, the University’s Pro-Vice- Chancellor for Research, Enter prise and Enga gement. “That process and the findings of our researchers have been the subject of significant debate in recent months . Colleagues in CRU have st renuously defended their conduct and the published work and we believe it is in the intere sts of all concerned that there should be an additional assessment cons idering the scie nce itself.”

The independent reassessmen t will complement Si r Muir Russell’s Review of the key allegations about the handling of data arisin g from the publication of a series of emails hacked from CRU. Sir Muir’s Review is expected to announce its finding in Spring 2010.

The reassessment of CRU’ s key publications will be completed at the earliest date the assessors can manage. The findings will be made public. 167



### 124. Details of the panel we re announced on 22 Ma rch. It will be headed by Lord Oxburgh. His appointment was made on the recommendation of the Ro yal Society, which was also consulted on the choice of th e six scientists on the panel: Professor Huw Davies, Professor of Physics at the Institute fo r Atmospheric and Climate Scie nce at ETH Zürich; Professor Kerry Emanuel, Professor of Meteorology at Mass achusetts Institut e of Technology; Professor Lisa Graumlich, Director of the School of Natural Resources and the Environment at The University of Arizona; Professor David Han d, Professor of Statistics in the Department of Mathematics at Imperial College; Professo r Herbert Huppert, Professor of Theoretical Geophysics at the University of Cambridge; and Professor Michael Kelly, Prince Philip Professor of Te chnology at the University of Cambridge. The panel will have

165 Q 176

166 Ev 40, para 4

167 UEA, 11 February 2010, www.uea.ac.uk/mac/comm/media/press/CRUstatements/New+sc ientific+assessment+of+climatic+research+publicatio ns+announced42 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia



### access to any publications or materials it requests , and all information considered will be listed in the Report. UEA, in consultation wi th the Royal Society, ha s suggested that the panel looks in particular at ke y publications, from the body of CRU’s research referred to in the UEA submission to our inquiry. According to the anno uncement on 22 March, the panel will meet in Norwich in April and will have the opportunity to see original data and speak to those who did the work and it compri ses of scientists who use techniques similar to those used in CRU but who la rgely apply them to ot her areas of research, as well as those with experience in climate or re lated research. 168



### 125. Announcing the Panel, Prof essor Trevor Davies, UEA’ s Pro-Vice-Chancellor for Research, said that:

Our concern has been to br ing together a distinguished group of independent scientists who understand the difference between asse rtion and evide nce, and are familiar with using the latt er to judge the validity of co nclusions arising from science research. The panel members have the right mix of skills to understand the complex nature of climate research and the discip line-based expertise to scrutinise CRU’s research. How they do this will be entirely down to the panel.

The choice of scientists is sure to be th e subject of discussion, and experience would suggest that it is impossible to find a group of eminent scientists to look at this issue who are acceptable to every interest group which has expressed a view in the last few months. Similarly it is un likely that a group of pe ople who have the necessary experience to assess the science, but hav e formed no view of their own on global warming, could be found. 169



### Public view of the climate science

126. There is no doubt that the e-mail discl osure from CRU in November 2009, and especially the extensive media coverage that has followed it ever since, has affected the general public view of climate science, both in the UK and further afield. Professor Bob Watson, Defra’s Chief Scientific Adviser, told us that “the media has certainly portrayed the UEA issue as a crisis, so I think to th e public it has been po rtrayed as a crisis”. 170 

### Professor Peter Cox, a cli mate scientist and a lead-author on the last IPCC 171 

### Working Group, in his written submission to us, said as much: “I am co ncerned that public confidence in the s cience of climate change has been undermined by the email leak”. 172 

### In its submission the Ro yal Society of Chemistry said that the:

true nature of science dictates that rese arch is transparent and robust enough to survive scrutiny. A lack of willingness to disseminate scientific information may infer that the scientific results or methods used are not robust enough to face scrutiny, even if this conjecture is not well-founded. This has far-reaching consequences for

168 “CRU Scientific A ssessment Panel announced ”, UEA Press Releas e, 22 March 2010, www.uea.ac.uk/mac/comm/media/press/CRUstatements/SAPannounce

169 As above

170 Q 198

171 Intergovernmental Panel on Climate Change

172 Ev 132, para 1The disclosure of climate data from the Climatic Research Unit at the University of East Anglia 43



### the reputation of science as a whole, with the ability to un dermine the public’s confidence in science. 173



### 127. The majority of submissions submitted to ou r inquiry has been from those who stated that the disclosed e-mails confirmed their wo rries that the climate change orthodoxy has serious flaws and the actions of CRU seriously impugned the integrity of climate change research. 174 

### A representative example was the me morandum from Dr Ph illip Bratby, “a semi-retired energy co nsultant”, who said that having examined the disclosures:

It is concluded that over at least a period of 20 years, climate s cience has been seriously compromised by the actions of a small group of scientists who have attempted to control the debate about cl imate change. The e ffects of this are potentially profound. For example a genera tion of work may have been corrupted and may be unreliable. A gene ration of students may have been corrup ted and their work may be unreliable. 175



### 128. Others offered a different perspective. Dr Timothy Osborn, a ful l-time member of staff at CRU, defended CRU:

It is impossible to draw firm conclusi ons from the hacked documents and emails. They do not represent the complete recor d, and they are not a random selection from the complete recor d. They are clearly selected with a purpose in mind and it is easy for people to fall into the trap s set by those who did the selection. 176



### 129. Beyond CRU, Professor Hans von Storch and Dr Myles Allen, profes sional statistical climatologists, agreed th at the publication of the hacked e-mails had initiated an intense debate about the credibility of climate science an d that “unfortunatel y, this debate sometimes goes so far as to questi on a key result of climate science”, 177 

### and the

language used in some of these e-mails ha s created concern, among both scientists and the public, about the openness and integr ity of the scientific process. But at the same time it is critical to point out that no gr ounds have arisen to doubt the validity of the thermometer-based temperature re cord since 1850, nor any results based upon it. 178



### 130. We put the concerns about th e threat to the reputation of science to the fifth panel who gave oral evidence: Professor John Be ddington, Government Ch ief Scientific Adviser, Professor Julia Slingo, Chief Scientist, Met Office, and Professor Bob Watson, Chief Scientist, Department for Envi ronment, Food and Rural Affairs. Prof essor Beddington did

173 Ev 171, para 4

174 For examples, see Ev 68 [R ichard S Courtney]; Ev 77 [Walter Radtke]; Ev 78 [Geoffrey Sherringt on]; and Ev 93 [Clive Menzies]

175 Ev 92, para 21

176 Ev 130, para 3

177 Ev 172, para 1

178 As above44 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia



### not consider that “UK sc ience has been damaged”. 179 

### The Met Office, in its written submission stated that

the UK enjoys a reputation for strong and robust science on the international stage. In the field of climat e research the Met Office is widely acknowle dged as world leading. 180



### Professor Slingo conf irmed in oral evidence that she has “absolute confidence in the science that we produc e at the Met Office”, 181 

### and Professor Watson, looking at the wider situation, attested that “there is absolutely no adverse effe ct on any of the conclusions of the IPCC.” 182



### 131. In our view, reputation has to be built on the solid fo undation of excellent, peerreviewed science. The review of the science to be carried out by the Scientific Appraisal Panel, which UEA announced on 22 March, should determ ine whether the work of CRU has been soundly built and it would be prem ature for us to pr e-judge that review.

132. Reputation does no t, however, rest solely on the quality of work as it should. It also depends on perception. It is self-evident that the disc losure of CRU e-mails has damaged the reputation of UK climate scienc e and, as views on global warming have become polarised, any deviation from the highest scient ific standards will be pounced on. As we explained in chapter 2, the practi ces and methods of climate science are a key issue. If the practices of CRU are found to be in line with the rest of climate science, the question would arise whether climate science methods of operation need to change. In this event we would recommend that the scientific community should consider changing those practi ces to ensure grea ter transparency.



### Need for a single review

133. The final issue is whether the best interests of science are served by having two reviews or inquiries. We found this difficult to evalua te as details of the S cientific Appraisal Panel were released in a late stage in our inquiry. When we asked Sir Muir whether it would be better to have a single in quiry, he responded:

It would have been poss ible, obviously, to have construc ted an inquiry that looked at both aspects of that, and t hat was not what I was asked to do. Whether I would have been the right person to be asked to do it I do not know but certainl y it obviously became clear to the Vice Chan cellor that there was this different issue about the confidence that one should have not in al l the methodological and handling issues but in the higher level s et of conclusions about wh at was actually happening. 183



### 134. The process of two revie ws or inquiries is underway. In our view there is the potential for overlap between the two inquiries—for example, the ques tion of the operation of peer

179 Q 194

180 Ev 46, para 1

181 Q 197

182 Q 198

183 Q 181The disclosure of climate data from the Climatic Research Unit at the University of East Anglia 45



### review needs to examine both methodology and quality of the science subject to review. The two reviews or inquiries need to map thei r activities to ensure that there are no unmanaged overlaps or gaps. If there are, th e whole process c ould be undermined.46 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia



# 5 Conclusions

135. Consideration of the complaints and accu sations made against CRU has led us to three broad conclusions.

136. Conclusion 1 The focus on Professor Jone s and CRU has been largely misplaced. On the accu sations relating to Professor Jones’s refusa l to share raw data and computer codes, we consider that his actions were in line with comm on practice in the climate science community. We have suggested that the community consider becoming more transparent by publishing raw data and detailed methodologies. On accusations relating to Freedom of Info rmation, we consider that much of the responsibility should li e with UEA, not CRU.

137. Conclusion 2 In addition, insofar as we have been able to consider accusations of dishonesty—for exa mple, Professor Jones’s alleged attempt to “hide the decline”—
we consider that there is no case to answer . Within our limited in quiry and the evidence we took, the scientific repu tation of Professor Jones and CRU remains intact. We have found no reason in this un fortunate episode to challenge the scientific consensus as expressed by Pr ofessor Beddington, that “global warming is happen ing [and] th at it is induced by human activity”. 184 

### It was not our purpose to examine, nor did we seek evidence on, the science produced by CRU. It will be for the Scient ific Appraisal Panel to look in detail into all the evidence to determine whet her or not the consensus view remains valid.

138. Conclusion 3 A great responsibility rests on the shoulders of climate science: to provide the planet’s d ecision makers wi th the knowledge they need to s ecure our future. The challenge that this poses is extensive and some of thes e decisions risk our standard of living. When the prices to pay are so la rge, the knowledge on which these kinds of decisions are taken had better be right. The science must be irreproachable.

184 Q 191The disclosure of climate data from the Climatic Research Unit at the University of East Anglia 47



# Conclusions and recommendations

Datasets

1. We recognise that some of the e-mails suggest a blunt re fusal to share data, even unrestricted data, with others. We ackn owledge that Professor Jones must have found it frustrating to handl e requests for data that he knew—or perceived—were motivated by a desire simply to seek to undermine his work. But Professor Jones’s failure to handle helpfully requ ests for data in a field as important and controversial as climate science was boun d to be viewed with su spicion. He was obviously frustrated by other workers in the fie ld trying to “undermine” his work, but his actions were inevitably coun terproductive. Professor Jones told us that the published e-mails represented only “one tenth of 1%” of his output, which amounts to one million e-mails, and th at we were only seeing the end of a protracted se ries of e-mail exchanges. We consider that further suspicion could have been allayed by releasing all the e-mails. In a ddition, we consider th at had the available ra w data been available online from an early stage, these kinds of unfortunate e-mail exchanges would not have occurred. In our view, CR U should have been more op en with its raw data and followed the more open appr oach of NASA to ma king data available. (Paragraph 38)

2. We are not in a position to set out any further the ex tent, if any, to which CRU should have made the data available in th e interests of transparency, and we hope that the Independent Climat e Change Email Review will reach specific conclusions on this point. However, tr ansparency and ac countability are of are increasing importance to the public, so we recommend that the Go vernment reviews the rules for the accessibility of data sets collected and analysed with UK public money. (Paragraph 39)

3. We note that the research passed the peer review process of so me highly reputable journals. However, we note that CRU could have been more open at that time in providing the deta iled methodologic al working on its websi te. We recommend that all publicly funded research groups consider whether they ar e being as op en as they can be, and ought to be, with the details of their methodologies. (Paragraph 45)

4. We therefore conclude that there is inde pendent verification, through the use of other methodologies and other sources of da ta, of the results and conclusions of the Climate Research Unit at the University of East Anglia. (Paragraph 49)

5. Even if the data that CRU used were not publicly availabl e—which they mostly are— or the methods not published —which they have been—i ts published results would still be credible: the resu lts from CRU agree with those drawn from other
intern ational data sets; in other words, the analyses have been repeated and the conclusions have been verified. (Paragraph 51)

6. It is not standard practice in climate science and many othe r fields to publish the raw data and the computer code in academic papers. We think that this is problematic because climate sc ience is a matter of global impo rtance and of public interest, and therefore the quality and transparency of the science should be irreproachable. We48 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia



### therefore consider that climat e scientists should take steps to make available all the data used to generate their published work, includ ing raw data; and it should also be made clear and referenced where data has b een used but, becaus e of commercial or national security reasons is not available . Scientists are also , under Freedom of Information laws and under th e rules of normal scientif ic conduct, entitled to withhold data which is due to be published under the peer-review process. In addition, scientists sh ould take steps to make available in full their methodological workings, including the computer codes. Data and methodological workings should be provided via the internet. There should be enough informatio n published to allow verification. (Paragraph 54)

7. Critics of CRU have suggested that Professor Jones’s us e of the word “trick” is evidence that he was part of a conspiracy to h ide evidence that did not fit hi s view that recent global warming is predominately caused by human activity. The balance of evidence patently fails to support this vi ew. It appears to be a colloquialism for a “neat” method of handli ng data. (Paragraph 60)

8. Critics of CRU have su ggested that Profe ssor Jones’s use of the words “hide the decline” is evidence that he was part of a conspiracy to hide ev idence that did not fit his view that recent global warming is predominantly caused by human activity. That he has published papers—including a paper in Nature —dealing with this aspect of the science clearly refutes this allegation . In our view, it wa s shorthand for the practice of discarding data known to be erroneous. We ex pect that this is a matter the Scientific Appraisa l Panel will address. (Paragraph 66)

9. The evidence that we have seen does not su ggest that Professor Jones was trying to subvert the peer review process. Academ ics should not be criticised for making informal comments on acad emic papers. The Independen t Climate Change Email Review should look in detail at all of thes e claims. (Paragraph 73)

Freedom of Information issues

10. We regret that the ICO made a statement to the press that went beyond th at which it could substantiate and that it took over a month for th e ICO properly to put the record straight. We recommend that the IC O develop procedures to ensure that its public comments are checked and that mechanisms exist to swiftly correct any misstatements or misinterpretations of such statements. (Paragraph 91)

11. There is prima fac ie evidence that CRU has breached the Freedom of Information Act 2000. It would, however, be prematur e, without a thorough investigation affording each party the opportunity to ma ke representations, to conclude that UEA was in breach of the Act. In our view, it is unsatisfactory to leave the matter unresolved simply because of the operatio n of the six-month time limit on the initiation of prosecutions. Much of the re putation of CRU hangs on the issue. We conclude that the matter needs to be resolved co nclusively—either by the Independent Climate Change Email Review or by the Information Commissioner. (Paragraph 93)The disclosure of climate data from the Climatic Research Unit at the University of East Anglia 49



### 12. If the Minister was correct to assert in July 2009 that the Government had no evidence that the current six-mo nth time limit presents a sy stemic problem, then it is now clear that such evidence exists. Irrespective of whether or not CRU breached the Freedom of Information Act 2000, we recommen d that the Government review the operation of section 77 of the 2000 Act and the six month limit on the initiation of prosecutions provided by section 127(1 ) of the Magistrates Court Act 1980. (Paragraph 95)

13. We have already recommended in paragraph 54 above that in future information, including data and methodol ogy, should be published pr oactively on the internet wherever possible. However, a culture of withholding information—from those perceived by CRU to be ho stile to global warming—appe ars to have pervaded CRU’s approach to FOIA requests from the outset. We consider this to be unacceptable. (Paragraph 103)

14. We cannot reach a firm conclusion on the basis of the evidence we took but we must put on record our concern about the manne r in which UEA all owed CRU to handle FOIA requests. Further, we found prima facie evidence to sugge st that the UEA found ways to support the cult ure at CRU of resisting disclosure of information to climate change sceptics. The failure of UE A to grasp fully the potential damage to CRU and UEA by the non-disclosure of FO IA requests was regrettable. UEA needs to review its policy to wards FOIA and re-assess how it can support academics whose expertise in this area is limited. (Paragraph 104)

The Independent Climate Change Email Review

15. We accept the assurances th at Sir Muir Russell has given about the independence of the Independent Climate Change Email Revi ew and we expect him to be scrupulous in preserving its impartialit y. We see no reason why the Review’s conclusions and UEA’s response have to be published together. Indeed, it could give the impression that UEA was being given an advantage when it comes to responding. We consider that the Review’s conclusions and recomme ndations should not be conveyed to UEA in advance of publica tion. (Paragraph 113)

16. With regards to the terms of reference of the Review, we consider that as well as measuring CRU against current acceptable scientific practice, th e Review should also make recommendations on best practice to be followed by CRU in the future. We invite Sir Muir Russell to respond formally to our Report to the extent that he sets out whether, on the basis of its contents, he finds the Terms of Reference of his inquiry need to be chan ged. (Paragraph 114)

17. It is unfortunate that the Independent Review got off to a bad star t with the necessary resignation of Dr Campbell. Th e question of the operation of peer review is going to be a critical issue in the inquiry and the Review Team needs to ta ke ste ps to ensure the insight and experience he would ha ve brought are replaced. (Paragraph 119)

18. We conclude that, when the Independent Review holds oral hearings or interviews, they should be carried out in public wherever poss ible and that it should publish all the written evidence it receive s on its website as soon as possible. (Paragraph 122)50 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia



### The Scientific Appraisal Panel

19. In our view, reputation has to be built on th e solid foundation ofexcellent, peerreviewed science. The review of the science to be carr ied out by the Scientific Appraisal Panel, which UEA announced on 22 March, should determine whether the work of CRU has been soundly built and it would be prem ature for us to pre-judge that review. (Paragraph 131)

20. Reputation does not, however, rest solely on the quality of work as it should. It also depends on perception. It is self-evident that the disclos ure of the CRU e-mails has damaged the reputation of UK climate scienc e and, as views on global warming have become polarised, any devia tion from the highest scie ntific standards will be pounced on. As we explained in chapte r 2, the practices an d methods of climate science are a key issue. If the practices of C RU are found to be in li ne with the rest of climate science, the question would ar ise whether climate sc ience methods of operation need to change. In this event we would recommend that the scientific community should consider ch anging those practices to ensure greater transparency. (Paragraph 132)

The two inquiries

21. The two reviews or inqu iries need to map their activities to ensure that there are no unmanaged overlaps or gaps. If there are, the whole process could be undermined. (Paragraph 134)

Conclusions

22. The focus on Professor Jones and CRU has been largely misplaced. On the accusations relating to Professor Jones’s refusal to share raw data and computer codes, we consider that his actions were in line with common pr actice in the climate science community. We hav e suggested that the comm unity consider becoming more transparent by publi shing raw data and detail ed methodologies. On accusations relating to Freedom of Infor mation, we consider that much of the responsibility should lie with UEA, not CR U. (Paragraph 136)

23. In addition, insofar as we ha ve been able to consider ac cusations of dishonesty—for example, Professor Jones’s al leged attempt to “hide the de cline”—we consider that there is no case to answer. Within our lim ited inquiry and the evidence we took, the scientific reputation of Professor Jones and CRU remains intact . We have found no reason in this unfortunate episode to challenge the scient ific consensus as expressed by Professor Beddington, that “global warming is happening [and] that it is induced by human activity”. It was not our purpos e to examine, nor did we seek evidence on, the science produced by CRU. It will be for the Scientific Appraisal Panel to look in detail i nto all the evidence to determine whet her or not the conse nsus view remains valid. (Paragraph 137)

24. A great responsibility rests on the shoulders of climate science: to provide the planet’s decision makers with the knowle dge they need to secure our future. The challenge that this poses is extensive and some of these de cisions risk our standard ofThe disclosure of climate data from the Climatic Research Unit at the University of East Anglia 51



### living. When the prices to pay ar e so large, the knowledge on which these kinds of decisions are taken had bett er be right. The science must be irreproachable. (Paragraph 138)52 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia



# Formal Minutes

Wednesday 24 March 2010

Members present:

Mr Phil Willis, in the Chair

Mr Tim Boswell
Dr Evan Harris Dr Brian Iddon Graham Stringe r

The Committee considered this matter.

Draft Report (The disclosure of climate data from the Climatic Research Unit at the University of East Anglia), proposed by the Chair, brought up and read.

Ordered , That the draft Report be read a second time, paragraph by paragraph.

Paragraphs 1 to 46 read and agreed to.

Paragraph 47 read.

Question put, That the paragraph stand part of the Report.

The Committee divided.

Ayes, 3 Mr Tim Boswell
Dr Evan Harris Dr Brian Iddon Noes, 1 Graham Stringer

Paragraphs 48 to 50 read and agreed to.

Paragraph 51 read.

Question put, That the paragraph stand part of the Report.

The Committee divided.

Ayes, 3 Mr Tim Boswell
Dr Evan Harris Dr Brian Iddon Noes, 1 Graham Stringer

Paragraphs 52 to 65 read and agreed to.

Paragraph 66 read.

Amendment proposed, to leave out from the beginning to “We” in line 6 and insert “We have not taken enough evidence on this matter to come to a final conclusion”.—( Graham Stringer. )

Question put, That the Amendment be made.The disclosure of climate data from the Climatic Research Unit at the University of East Anglia 53

The Committee divided.

Ayes, 1 Graham Stringer Noes, 3 Mr Tim Boswell
Dr Evan Harris Dr Brian Iddon

Paragraph 66 agreed to.

Paragraphs 67 to 131 read and agreed to.

Paragraph 132 read.

Amendment proposed, to leave out from “science” in line 6 to the end and add “it would be necessary for the whole of climate science to increase its transparency and improve its scientific methodology”.—( Graham Stringer .)

Question put, That the Amendment be made.

The Committee divided.

Ayes, 1 Graham Stringer Noes, 3 Mr Tim Boswell
Dr Evan Harris Dr Brian Iddon

Paragraph 132 agreed to.

Paragraph 133 read and agreed to.

Paragraph 134 read.

Amendment proposed, at the end of line 5 to insert “ Given the increasingly hostile attitudes of both sides on this issue, it is vital that these two inquiries have at least one member each who is a reputable scientist, and is sceptical of anthropogenic climate change ” .—( Graham Stringer. )

Question put, That the Amendment be made.

The Committee divided.

Ayes, 1 Graham Stringer Noes, 3 Mr Tim Boswell
Dr Evan Harris Dr Brian Iddon

Paragraphs 135 and 136 read and agreed to.

Paragraph 137 read.

Amendment proposed, after “answer” in line 3 add “ Within our limited inquiry and the evidence we took, the scientific reputation of Prof essor Jones and CRU remains intact. ”—( Dr Evan Harris. )

Question put, That the Amendment be made.54 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia

The Committee divided.

Ayes, 3 Mr Tim Boswell
Dr Evan Harris Dr Brian Iddon Noes, 1 Graham Stringer

Question put, That the paragraph, as amended, stand part of the Report.

The Committee divided.

Ayes, 3 Mr Tim Boswell
Dr Evan Harris Dr Brian Iddon Noes, 1 Graham Stringer

Paragraph 138 read and agreed to.

Summary brought up and read.

Question put, That the summary be added to the Report.

The Committee divided.

Ayes, 3 Mr Tim Boswell
Dr Evan Harris Dr Brian Iddon Noes, 1 Graham Stringer

Motion made, and Question put, That the Report be the Eighth Report of the Committee to the House.

The Committee divided.

Ayes, 3 Mr Tim Boswell
Dr Evan Harris Dr Brian Iddon Noes, 1 Graham Stringer

Resolved , That the Report be the Eighth Report of the Committee to the House.

Ordered , That the Chair make the Report to the House.

Ordered , That embargoed copies of the Report be made available, in accordance with the provisions of Standing Order No. 134.

Written evidence was ordered to be reported to the Hous e for printing with the Report, together with written evidence reported and ordered to be published on 24 February and 1 March 2010.

Written evidence was ordered to be reported to th e House for placing in the Library and Parliamentary Archives.

[The Committee adjournedThe disclosure of climate data from the Climatic Research Unit at the University of East Anglia 55



# Witnesses

Wednesday 1 March 2010

The Rt Hon Lord Lawson of Blaby, Chairman, and Dr Benny Peiser, Director, Global Warming Policy Foundation Ev 2

Richard Thomas CBE, former Information Commissioner Ev 11

Professor Edward Acton, Vice-Chancellor, University of East Anglia and Professor Phil Jones, Director of the Climatic Research Unit Ev 27

Sir Muir Russell, Head of the Independent Climate Change E-Mails Review Ev 41

Professor John Beddington, Government Chief Scientific Adviser, Professor Julia Slingo OBE, Chief Scientist, Met Office, and Professor Bob Watson, Chief Scientist, Defra Ev 58



# List of written evidence

1 Andrew Montford Ev 159

2 Anne Stallybrass Ev 169

3 Aporia Ev 98

4 Climate Change E-Mails Review Team Ev 39

5 Clive Menzies Ev 93

6 David Andrew Cockroft Ev 168

7 David Holland Ev 115

8 David Shaw Ev 99

9 Douglas J. Keenan Ev 181

10 Dr. Benny Peiser Ev 164

11 Dr. D. R. Keiller Ev 103

12 Dr. Michael Simons Ev 97

13 Dr. Sonja Boehmer-Christiansen Ev 124, Ev 127

14 Dr. Timothy J. Osborn Ev 129

15 Edward Dilley Ev 76

16 Eric Rasmusen Ev 89

17 G R Ryan Ev 78

18 Geoffrey Sherrington Ev 78

19 Global Warming Policy Foundation Ev 1

20 Godfrey Bloom MEP Ev 92

21 Ian Goddard Ev 82

22 Institute of Physics Ev 167

23 J Ronan Ev 19756 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia

24 John F Kelly Ev 191

25 John Graham-Cumming Ev 195

26 John Wadsworth Ev 81

27 Lalu Hanuman Ev 81

28 Martin Brumby Ev 82

29 Met Office Ev 46, Ev 64

30 Mike Haseler Ev 133

31 Nicholas Barnes and David Jones Ev 197

32 Peabody Energy Company Ev 191

33 Peter Sinclair Ev 82

34 Peter Taylor Ev 186

35 Phillip Bratby Ev 90

36 Professor Darrel Ince Ev 152

37 Professor Hans von Storch a nd Dr. Myles R. Allen Ev 172

38 Professor John Beddington, Government Chief Scientific Adviser Ev 45, Ev 64

39 Professor Peter Cox Ev 132

40 Professor Ross McKitrick Ev 140

41 Public Interest Research Centre Ev 176

42 Research Councils UK Ev 175

43 Richard S Courtney Ev 68

44 Richard Thomas CBE Ev 7

45 Richard Tyrwhitt-Drake Ev 162

46 Roger Helmer MEP Ev 85

47 Ronald K Bolton Ev 119, Ev 123

48 Royal Society of Chemistry Ev 170

49 Royal Statistical Society Ev 185

50 Stephen McIntyre Ev 82, Ev 144

51 Stephen Prower Ev 86

52 Steven Mosher Ev 151

53 Stuart Huggett Ev 77

54 Susan Ewens Ev 83

55 University of East Anglia Ev 16, Ev 17, Ev 25, Ev 34, Ev 37, Ev 38

56 Walter Radtke Ev 77

57 Warwick Hu ghes Ev 153The disclosure of climate data from the Climatic Research Unit at the University of East Anglia 57



# List of unprinted evidence

The following written evidence has been reported to the House, but has not been printed and copies have been placed in the House of Commons Library, where they may be inspected by Members. Other copies are in the Parliamentary Archives (www.parliament.uk/archives), a nd are available to the public for inspection. Requests for inspection should be addressed to The Par liamentary Archives, Houses of Parliament, London SW1A 0PW (tel. 020 7219 3074; e-mail archives@parliament.uk). Opening hours are from 9.30 am to 5.00 pm on Mondays to Fridays.

CRU 27 The Global Warming Policy Foundation annexes CRU 58/58a Dr Nigel Dudley memoranda58 The disclosure of climate data from the Climatic Research Unit at the University of East Anglia



# List of Reports from the Committee during the current Parliament

The reference number of the Government’s re sponse to each Report is printed in brackets after the HC printing number.

Session 2009–10
First Report The work of th e Committee in 2008–09 HC 103
Second Report Evidence Check 1: Early Literacy Interventions HC 44 (HC 385)
Third Report The Government’s review of th e principles applying to the treatment of independent scientific advice provided to government HC 158-I (HC 384)

Fourth Report Evidence Check 2: Homeopathy HC 45
Fifth Report The Regulation of Geoengineering HC 221
Sixth Report The impact of sp ending cuts on science and scie ntific research HC 335-I
Seventh Report Bioengineering HC 220
Eighth Report The disclosure of climate data fro m the Climatic Research Unit at the University of East Anglia HC 387-I

Session 2008–09
First Report Re-skilling for recovery: After Leitch, implementing skills and training policies HC 48–I (HC 365)

Second Report The Work of the Committee 2007–08 HC 49
Third Report DIUS’s Department al Report 2008 HC 51–I (HC 383)
Fourth Report Engineering: turning ideas into reality HC 50–I (HC 759)
Fifth Report Pre-appointment hearing with the Chair-elect of the Economic and Social Research Council, Dr Alan Gillespie CBE HC 505

Sixth Report Pre-appointment hearing with the Chair-elect of the Biotechnology and Biological Sciences Research Council, Professor Sir Tom Blundell HC 506

Seventh Report Spend, spend, spend? – The misma nagement of the Learning and Skills Council’s capital programme in further education colleges HC 530 (HC 989)

Eighth Report Putting Science and E ngineering at the Heart of Government Policy HC 168–I (HC 1036)
Ninth Report Pre-appointment hearing with the Chair-elect of the Science and Technology Facilities Council, Professor Michael Sterling HC 887

Tenth Report Sites of Special Sc ientific Interest HC 717 (HC 990)
Eleventh Report Students and Universities HC 170–I (HC 991)

Session 2007–08
First Report UK Centre for Medical Re search and Innovation HC 185 (HC 459)
Second Report The work and operation of the Copyright Tribunal HC 245 (HC 637)
Third Report Withdrawal of funding for equivalent or lower level qualifications (ELQs) HC 187–I (HC 638)

Fourth Report Science Budget Allocations HC 215 (HC 639)
Fifth Report Renewable elec tricity-generation technologies HC 216–I (HC 1063)
Sixth Report Biosecurity in UK res earch laboratories HC 360–I (HC 1111)The disclosure of climate data from the Climatic Research Unit at the University of East Anglia 59

Seventh Report Pre-legislative Scrutiny of the Draft Apprenticeships Bill HC 1062–I (HC (2008–09)262)
First Special Report The Funding of Science and Disc overy Centres: Government Response to the Eleventh Repo rt from the Science and Technology Committee, Session 2006–07 HC 214

Session 2007–08 (Continued)
Second Special Report The Last Report: Government Res ponse to the Thirteenth Report
from the Science and Technol ogy Committee, Session 2006–07 HC 244

Fourth Special Report Investigating the Oceans: Governme nt Response to the Science and Technology Committee’ s Tenth Report of Session 2006–07 HC 506 [incorporating HC 469–i]

### 1

Report of the International Panel set up by the University of East Anglia to examine the research of the Climatic Research Unit.

Introduction

1. The Panel was set up by the University in consultation with the Royal Society to assess the integrity of the research published by the Climatic Research Unit in the light of various external assertions. The Unit is a very small academic entity within the School of Environmental Sciences. It has three full time and one part time academic staff members and about a dozen research associates, PhD students and support staff. The essence of the criticism that the Panel was asked to address was that climatic data had been dishonestly selected, manipulated and/or presented to arrive at pre-determined conclusions that were not compatible with a fair interpretation of the original data. The members of the Panel are listed in Appendix A at the end of this report.

2. The Panel was not concerned with the question of whether the conclusions of the published research were correct. Rather it was asked to come to a view on the integrity of the Unit’s research and whether as far as could be determined the conclusions represented an honest and scientifically justified interpretation of the data. The Panel worked by examining representative publications by members of the Unit and subsequently by making two visits to the University and interviewing and questioning members of the Unit. Not all the panel were present on both occasions but two members were present on both occasions to maintain continuity. About fifteen person/days were spent at the University discussing the Unit’s work.

3. The eleven representative publications that the Panel considered in detail are listed in Appendix B. The papers cover a period of more than twenty years and were selected on the advice of the Royal Society. All had been published in international scientific journals and had been through a process of peer review. CRU agreed that they were a fair sample of the work of the Unit. The Panel was also free to ask for any other material that it wished and did so. Individuals on the panel asked for and reviewed other CRU research materials.

4. The Panel’s work began with a detailed reading of the published work. Every paper was read by a minimum of three Panel members at least one of whom was familiar with the general area to which the paper related. At least one of the other two was a generalist with no special climate science expertise but with experience of some of the general techniques and methods employed in the work. Most of the members of the Panel read all the publications. The publications provided a platform from which to gain a deeper understanding of the Unit’s research and enabled the Panel to probe particular questions in more detail.

### 2

5. Broadly the work of the Unit falls into two parts:

• Construction and interpretation of tree ring chronologies extending over some thousands of years with a view to gaining information about past climates:

• Studies of temperatures over the last few hundred years from direct observations.

Dendroclimatology

1. Tree growth is sensitive to very many factors including climate. By piecing together growth records from different trees, living or dead, it is possible to determine the temporal variation of growth patterns going back many hundreds of years. The dendroclimatological work at CRU seeks to go beyond this and to extract from the dated growth patterns the local and regional history of temperature variations. The Unit does virtually no primary data acquisition but has used data from published archives and has collaborated with people who have collected data.

2. The main effort of the dendroclimalogists at CRU is in developing ways to extract climate information from networks of tree ring data. The data sets are large and are influenced by many factors of which temperature is only one. This means that the effects of long term temperature variations are masked by other more dominant short term influences and have to be extracted by statistical techniques. The Unit approaches this task with an independent mindset and awareness of the interplay of biological and physical processes underlying the signals that they are trying to detect.

3. Although inappropriate statistical tools with the potential for producing misleading results have been used by some other groups, presumably by accident rather than design, in the CRU papers that we examined we did not come across any inappropriate usage although the methods they used may not have been the best for the purpose. It is not clear, however, that better methods would have produced significantly different results. The published work also contains many cautions about the limitations of the data and their interpretation.

4. Chronologies (transposed composites of raw tree data) are always work in progress. They are subject to change when additional trees are added; new ways of data cleaning may arise (e.g. homogeneity adjustments), new measurement methods are used (e.g. of measuring ring density), new statistical methods for treating the data may be developed (e.g. new ways of allowing for biological growth trends).

5. This is illustrated by the way CRU check chronologies against each other; this has led to corrections in chronologies produced by others. CRU is to be commended for continuously updating and reinterpreting their earlier chronologies.

### 3

6. With very noisy data sets a great deal of judgement has to be used. Decisions have to be made on whether to omit pieces of data that appear to be aberrant. These are all matters of experience and judgement. The potential for misleading results arising from selection bias is very great in this area. It is regrettable that so few professional statisticians have been involved in this work because it is fundamentally statistical. Under such circumstances there must be an obligation on researchers to document the judgemental decisions they have made so that the work can in principle be replicated by others.

7. CRU accepts with hindsight that they should have devoted more attention in the past to archiving data and algorithms and recording exactly what they did. At the time the work was done, they had no idea that these data would assume the importance they have today and that the Unit would have to answer detailed inquiries on earlier work. CRU and, we are told, the tree ring community generally, are now adopting a much more rigorous approach to the archiving of chronologies and computer code. The difficulty in releasing program code is that to be understood by anyone else it needs time-consuming work on documentation, and this has not been a top priority.

8. After reading publications and interviewing the senior staff of CRU in depth, we are satisfied that the CRU tree-ring work has been carried out with integrity, and that allegations of deliberate misrepresentation and unjustified selection of data are not valid. In the event CRU scientists were able to give convincing answers to our detailed questions about data choice, data handling and statistical methodology. The Unit freely admits that many data analyses they made in the past are superseded and they would not do things that way today.

9. We have not exhaustively reviewed the external criticism of the dendroclimatological work, but it seems that some of these criticisms show a rather selective and uncharitable approach to information made available by CRU. They seem also to reflect a lack of awareness of the ongoing and dynamic nature of chronologies, and of the difficult circumstances under which university research is sometimes conducted. Funding and labour pressures and the need to publish have meant that pressing ahead with new work has been at the expense of what was regarded as non-essential record keeping. From our perspective it seems that the CRU sins were of omission rather than commission. Although we deplore the tone of much of the criticism that has been directed at CRU, we believe that this questioning of the methods and data used in dendroclimatology will ultimately have a beneficial effect and improve working practices

Temperatures from Historical Instrumental Records

1. The second main strand of work at CRU has been the collection and collation of instrumental land temperature records from all over the world and the construction of regional, hemispherical and global scale temperature records. These records are irregularly distributed in space and time. Modern records come largely from land-based meteorological stations but their geographical distribution is uneven and strongly biased in favour of the northern hemisphere

### 4 where most of the Earth’s land masses are located. Oceans cover two thirds of the Earth’s surface and away from the main shipping routes coverage is thin. For earlier centuries the record is much sparser. Deriving estimates of past temperatures on a global, hemispheric and regional scale from incomplete data sets is one of the problems faced by the Unit and in consequence an important current interest is the discovery of useable old temperature records from a variety of sources.

2. In the latter part of the 20 th 

### century CRU pioneered the methods for taking into account a wide range of local influences that can make instrumental records from different locations hard to compare. These methods were very labour intensive and were somewhat subjective. Much of this work was supported by the US Department of Energy and was published with the details of station corrections several times a year. Since the 1980s the Unit has done no more of this work and have concentrated on the merging and interpretation of data series corrected by others. There have been various analyses of similar publicly available data sets by different international groups. Although there are some differences in fine detail that reflect the differences in the analytical methods used, the results are very similar.

3. The Unit has devoted a great deal of effort to understanding how instrumental observations are best combined to derive the surface temperature on a variety of time and space scales. It has become apparent from a number of studies that there is elevation of the surface temperature in and around large cities and work is continuing to understand this fully.

4. Like the work on tree rings this work is strongly dependent on statistical analysis and our comments are essentially the same. Although there are certainly different ways of handling the data, some of which might be superior, as far as we can judge the methods which CRU has employed are fair and satisfactory. Particular attention was given to records that seemed anomalous and to establishing whether the anomaly was an artefact or the result of some natural process. There was also the challenge of dealing with gaps in otherwise high quality data series. In detailed discussion with the researchers we found them to be objective and dispassionate in their view of the data and their results, and there was no hint of tailoring results to a particular agenda. Their sole aim was to establish as robust a record of temperatures in recent centuries as possible. All of the published work was accompanied by detailed descriptions of uncertainties and accompanied by appropriate caveats. The same was true in face to face discussions.

5. We believe that CRU did a public service of great value by carrying out much time-consuming meticulous work on temperature records at a time when it was unfashionable and attracted the interest of a rather small section of the scientific community. CRU has been among the leaders in international efforts to determining the overall uncertainty in the derived temperature records and where work is best focussed to improve them.

### 5

6. The Unit has demonstrated that at a global and hemispheric scale temperature results are surprisingly insensitive to adjustments made to the data and the number of series included.

7. Recent public discussion of climate change and summaries andpopularizations of the work of CRU and others often contain oversimplifications that omit serious discussion of uncertainties emphasized by the original authors. For example, CRU publications repeatedly emphasize the discrepancy between instrumental and tree-based proxy reconstructions of temperature during the late 20 th 

### century, but presentations of this work by the IPCC and others have sometimes neglected to highlight this issue. While we find this regrettable, we could find no such fault with the peer-reviewed papers we examined

Conclusions

1. We saw no evidence of any deliberate scientific malpractice in any of the work of the Climatic Research Unit and had it been there we believe that it is likely that we would have detected it. Rather we found a small group of dedicated if slightly disorganised researchers who were ill-prepared for being the focus of public attention. As with many small research groups their internal procedures were rather informal.

2. We cannot help remarking that it is very surprising that research in an area that depends so heavily on statistical methods has not been carried out in close collaboration with professional statisticians. Indeed there would be mutual benefit if there were closer collaboration and interaction between CRU and a much wider scientific group outside the relatively small international circle of temperature specialists.

3. It was not the immediate concern of the Panel, but we observed that there were important and unresolved questions that related to the availability of environmental data sets. It was pointed out that since UK government adopted a policy that resulted in charging for access to data sets collected by government agencies, other countries have followed suit impeding the flow of processed and raw data to and between researchers. This is unfortunate and seems inconsistent with policies of open access to data promoted elsewhere in government.

4. A host of important unresolved questions also arises from the application of Freedom of Information legislation in an academic context. We agree with the CRU view that the authority for releasing unpublished raw data to third parties should stay with those who collected it.

Submitted to the University 12 April 2010

### 6

Addendum to report, 19 April 2010

For the avoidance of misunderstanding in the light of various press stories, it is important to be clear that the neither the panel report nor the press briefing intended to imply that any research group in the field of climate change had been deliberately misleading in any of their analyses or intentionally exaggerated their findings. Rather, the aim was to draw attention to the complexity of statistics in this field, and the need to use the best possible methods.

### 7

APPENDIX A PANEL MEMBERSHIP

Chair: Prof Ron Oxburgh FRS (Lord Oxburgh of Liverpool)

Prof Huw Davies, ETH Zürich Prof Kerry Emanuel , Massachusetts Institute of Technology Prof Lisa Graumlich , University of Arizona. Prof David Hand FBA, Imperial College, London. Prof Herbert Huppert FRS , University of Cambridge Prof Michael Kelly FRS , University of Cambridge

### 8

APPENDIX B

Peer-reviewed publications for assessment

.

1. Brohan, P., Kennedy, J., Harris, I., Tett, S.F.B. and Jones, P.D., 2006: Uncertainty estimates in regional and global observed temperature changes: a new dataset from 1850. J. Geophys. Res. 111 , D12106.

2. Briffa, K. R., F. H. Schweingruber, P. D. Jones, T. J. Osborn, S. G. Shiyatov, and E. A. Vaganov. 1998a. Reduced sensitivity of recent tree-growth to temperature at high northern latitudes. Nature 391 :678-682.

3. Briffa, K. R., F. H. Schweingruber, P. D. Jones, T. J. Osborn, I. C. Harris, S. G. Shiyatov, E. A. Vaganov, and H. Grudd, 1998b. Trees tell of past climates: but are they speaking less clearly today? Philosophical Transactions of the Royal Society of London Series B – Biological Sciences 353, 65-73.

4. Briffa, K. R. 2000. Annual climate variability in the Holocene: interpreting the message of ancient trees. Quaternary Science Reviews 19 , 87-105.

5. Briffa, K.R., Osborn, T.J., Schweingruber, F.H., Harris, I.C., Jones, P.D., Shiyatov, S.G. and Vaganov, E.A., 2001: Low-frequency temperature variations from a northern tree-ring density network. J. Geophys. Res. 106 , 2929-2941.

6. Briffa, K. R., V. V. Shishov, T. M. Melvin, E. A. Vaganov, H. Grudd, R. M. Hantemirov, M. Eronen, and M. M. Naurzbaev. 2008. Trends in recent temperature and radial tree growth spanning 2000 years across northwest Eurasia. Philosophical Transactions of the Royal Society B-Biological Sciences 363 , 2271-2284.

7. Jones, P.D. and Moberg, A., 2003: Hemispheric and large-scale surface air temperature variations: An extensive revision and an update to 2001. J. Climate 16

, 206-223.

8. Jones, P.D., Raper, S.C.B., Bradley, R.S., Diaz, H.F., Kelly, P.M. and Wigley, T.M.L., 1986a: Northern Hemisphere surface air temperature variations: 1851-1984. Journal of Climate and Applied Meteorology 25 , 161-179.

9. Jones, P.D., Raper, S.C.B. and Wigley, T.M.L., 1986b: Southern Hemisphere surface air temperature variations: 1851-1984. Journal of Climate and Applied Meteorology 25 , 1213-1230.

10. Jones, P.D., Groisman, P.Ya., Coughlan, M., Plummer, N., Wang, W-C. and Karl, T.R., 1990: Assessment of urbanization effects in time series of surface air temperature over land. Nature 347, 169-172.



### 11. Jones, P.D., Lister, D.H. and Li, Q., 2008: Urbanization effects in large-scale temperature records, with an emphasis on China. Journal of Geophysical Research, 113, D16122.

Supporting documentation

Briffa and Melvin (2009) which is online at http://www.cru.uea.ac.uk/cru/people/briffa/yamal2009 /

### 9

TR017 – Bradley, R.S., Kelly, P.M., Jones, P.D., Goodess, C.M. and Diaz, H.F., 1985: A Climatic Data Bank for Northern Hemisphere Land Areas, 1851-1980, U.S. Dept. of Energy, Carbon Dioxide Research Division, Technical Report TRO17 , 335 pp.

TR022 – Jones, P.D., Raper, S.C.B., Santer, B.D., Cherry, B.S.G., Goodess, C.M., Kelly, P.M., Wigley, T.M.L., Bradley, R.S. and Diaz, H.F., 1985: A Grid Point Surface Air Temperature Data Set for the Northern Hemisphere, U.S. Dept. of Energy, Carbon Dioxide Research Division, Technical Report TRO22 , 251 pp.

TR027 – Jones, P.D., Raper, S.C.B., Cherry, B.S.G., Goodess, C.M. and Wigley, T.M.L., 1986: A Grid Point Surface Air Temperature Data Set for the Southern Hemisphere, 1851-1984, U.S. Dept. of Energy, Carbon Dioxide Research Division, Technical Report TR027 , 73 pp.RA-1O Final Investigation Report Involving Dr. Michael E, Mann
The Pennsylvania State University
June 4, 2010

Composition of the Investigatory Committee:

Sarah M. Assmann, Waller Professor Department of Biology

Welford Castleman, Evan Pugh Professor and Eberly Distinguished Chair in Science Department of Chemistry and Depmtment of Physics

Mary Jane Irwin, Evan Pugh Professor Department of Computer Science and Electrical Engineering

Nina G. Jablonski, Department Head and Professor Department of Anthropology

Fred W. Vondracek, Professor Department of Human Development and Family Studies

Research Integrity Officer:

Candice Yekel, Director of the Office for Research Protections

Background of the alleged misconduct as described in the RAlO Inquiry Report:

On and about November 22, 2009, The Pennsylvania State University began to receive

numerous communications (emails, phone calls and letters) accusing Dr. Michael E,
Mann of having engaged in acts, beginning in approximately 1998, that included manipulating data, destroying records and colluding to hamper the progress of scientific
discourse around the issue of anthropogenic global warming, These accusations were based on perceptions of the content of the emails stolen from a server at the Climatic
Research Unit of the University of East Anglia in Great Britain as widely reported,

Given the sheer volume of the communications to Penn State, the similarity of their

content and the variety of sources, which included University alumni, federal and state politicians, and others, many of whom had had no relationship with Pel1l1 State, Dr. Eva J. Pell, then Senior Vice President for Research and Dean of the Graduate School, was asked to examine the matter. The reason for having Dr. Pell examine the matter was that
the accusations, when placed in an academic context, could be construed as allegations of research misconduct, which would constitute a violation of Penn State policy,

Under The Pennsylvania State University's policy, Research Administration Policy No, 10, (hereafter referred to as RA-I 0), Research Misconduct is defined as:

Page 1 of 19(1) fabrication, falsification, plagiarism or other practices that seriously deviate from accepted practices within the academic community for proposing, conducting, or reporting research or other scholarly activities; (2) callous disregard for requirements that ensure the protection of researchers, human participants, or the public; or for ensuring the welfare oflaboratory animals; (3) failure to disclose significant financial and business interest as defined by Penn State Policy RA20, Individual Conflict of Interest; (4) failure to comply with other applicable legal requirements governing research or other scholarly activities.

RA-IO further provides that "research misconduct does not include disputes regarding
honest error or honest differences in interpretations or judgments of data, and is not intended to resolve bona fide scientific disagreement or debate."

On November 24, 2009, two days after receipt of the allegations, Dr. Pell initiated the

process articulated in RA-1O by scheduling a meeting with the Dean of the College of
Earth and Mineral Sciences (Dr. William Easterling), the Associate Dean for Graduate
Education and Research of the College of Em1h and Mineral Sciences (Dr. Alan Scaroni),
the Director of the Office for Research Protections (Ms. Candice Yekel), and the Head of
the Department of Meteorology (Dr. William Brune).

At this meeting, all were informed of the situation and of the decision to initiate an inquiry under RA-I O. Dr. Pell then discussed the responsibilities that each individual
would have according to the policy. Dean Easterling recused himself from the inquiry
due to a conflict of interest. As the next administrator in the line of management for the college, Dr. Scm'oni was asked to take on Dean Easterling's function in the ensuing inquiry.

The Inquiry Committee assigned to conduct the inquiry into the matter consisted of Dr.
Eva J. Pell, Senior Vice President for Research, Ms. Candice Yekel, Director of the
Office for Research Protections, and Dr. Alan Scaroni, Associate Dean for Graduate
Education and Research of the College of Earth and Mineral Sciences. Dr. William Brune, Head of the Department of Meteorology, was to serve in a consulting capacity for
the Inquiry Committee. Dr. Henry C. Foley, then Dean of the College ofInformation
Sciences and Technology, was added to the Inquiry Committee in an ex-officio role for
the duration of2009, since he had been named to succeed Dr. Pell as the next Vice
President for Research, beginning January 1,2010.

At the time of initiation of the inquiry, no formal allegations accusing Dr. Mann of
research misconduct had been submitted to any University official. Therefore, the emails
and other communications were reviewed by Dr. Pell, and from these she synthesized the following four formal allegations. To be clear, these were not allegations that Dr. Pell put forth but rather her best effo11 to reduce to reviewable allegations the many different
accusations that were received from pm1ies outside of the University. The four synthesized allegations were as follows:

Page 2 of 19I. Did you engage in, or participate in, directly or indirectly, any actions with the intent to
suppress or falsify data?

2. Did you engage in, or participate in, directly or indirectly, any actions with the intent to delete, conceal or otherwise destroy emails, information and/or data, related to AR4,as suggested by Phil Jones?

3. Did you engage in, or participate in, directly or indirectly, any misuse of privileged or confidential information available to you in your capacity as an academic scholar?

4. Did you engage in, or participate in, directly or indirectly, any actions that seriously deviated from accepted practices within the academic community for proposing, conducting, or reporting research, or other scholarly activities?

On November 29, 2009, Dr. Pell and Dr. Foley met with Dr. Mann to inform him personally that he had been accused of research misconduct and that an inquiry under
RA-1O would take place. On November 30, 2009, a letter was delivered by Dr. Pell to Dr.
Mann to notify him of these allegations and Dr. PeWs decision to conduct an inquiry
under RA-IO. The inquiry phase ofRA-IO was thereby formally initiated on November 30,2009.

From November 30 to December 14,2009, staff in the Office for Research Protections
culled through the 1073 files that contained emails or email strings that were purloined
from a server at the University of East Anglia. A subset of the files containing emails or email strings was reviewed. This subset of files included emails that were sent by Dr. Mann, were sent to Dr. Mann, were copied to Dr. Mann, or discussed Dr. Mann (but were neither addressed nor copied to him). In summary, the following were found:

206 files that contained emails or email strings that contained messageltext from
Dr. Mann somewhere in the chain;
91 files that contained emails or emails strings that were received by Dr. Mann, but in which he did not participate; and
79 files that contained emails or email strings that dealt with Dr. Mann, his work

or publications but that he neither authored nor was listed as copied.

From among these 376 files, the Inquiry Committee focused on 47 files that contained
emails or email strings that were deemed relevant. On December 17, 2009, the Inquiry
Committee (Pell, Scaroni, Yekel), Dr. Brune and Dr. Foley met to review the emails, the
RA-1O inquiry process, and their respective activities. It was agreed that these individuals would meet again in early January and that they would use the time until that meeting to review the relevant information, including the above mentioned e-mails, journal articles,
OP-ED columns, newspaper and magazine articles, the National Academy of Sciences report entitled "Surface Temperature Reconstructions for the Last 2,000 Years," ISBN: 0-
309-66144-7 and various blogs on the internet.

Page 3 of 19On January 4, 2010, Dr. Foley, in his capacity as the new Vice President for Research
and Dean of the Graduate School, became the convener of the Inquiry Committee as Dr. Pell had left the University to become the Under-Secretary of Science for the Smithsonian Institution. On January 8, 2010, Dr. Foley convened the Inquiry Committee

to discuss their thoughts on the evidence presented in the emails and other publically available materials. At this meeting, it was decided that each Inquiry Committee member would send to Dr. Foley specific questions to be used by the Inquiry Committee during
the interview of Dr. Mann. During the interview, Dr. Foley would moderate the
interview and ask each of the initial questions with follow-up questions coming from the other Inquiry Committee members.

On January 12,2010, the Inquiry Committee (Foley, Yekel, Scaroni) and Dr. Brune met with Dr. Mann. Dr. Mann was asked to address the four allegations leveled against him
and to provide answers to the fifteen additional questions that the Inquiry Committee had compiled. In an interview lasting nearly two hours, Dr. Mann addressed each of the questions and follow-up questions. A recording was made of the interview and was later
transcribed. The Inquiry Committee members asked occasional follow-up questions. Dr.
Mann answered each question carefully:

• He explained the content and meaning of the emails about which the Inquiry Committee inquired;

• He stated that he had never falsified any data, nor had he had ever manipulated data to serve a given predetermined outcome;

• He stated that he never used inappropriate influence in reviewing papers by other
scientists who disagreed with the conclusions of his science;

• He stated that he never deleted emails at the behest of any other scientist, specifically including Dr. Phil Jones, and that he never withheld data with the
intention of obstructing science; and

• He stated that he never engaged in activities or behaviors that were inconsistent with accepted academic practices.

On January 15,2010, Dr. Foley conveyed via email on behalf of the Inquiry Committee

an additional request to Dr. Mann. Dr. Mann was asked to produce all emails related to the fourth IPCC report ("AR4"), the same emails that Dr. Phil Jones had suggested that
he delete. On January 18,2010, Dr. Mann provided a zip-archive of these emails and an explanation of their content. In addition, Dr. Mann provided a ten page supplemental
written response to the matters discussed during his interview.

On January 22, 2010, the Inquiry Committee and Dr. Brune met again to review the evidence, including but not limited to Dr. Mann's answers to the Inquiry Committee's questions, both in the interview and in his subsequent submissions. Dr. Foley reviewed
the relevant points of his conversation with Dr. Gerald North, a professor at Texas A&M University and the first author of the NAS 2006 report that included Dr. Mann's research

on paleoclimatology. Dr. Foley also relayed the sentiment and view of Dr. Donald Kennedy of Stanford University and the former editor of Science Magazine about the controversy currently swirling around Dr. Mann and some of his colleagues. Both were

Page 4 of 19very supportive of Dr. Mann and of the credibility of his science. Dr. Brune gave his opinions and suggestions for next steps of the process, and then was dismissed from
fmther discussion pursuant to RA-1O policy role which was consult to the rest of the Inquiry Committee members.

On January 26, 2010, Dr. Foley convened the Inquiry Committee, along with University counsel, Mr. Wendell Courtney, Esq., in case issues of procedure arose.

After a careful review of all written material, and information obtained from the purloined emails, the interview of Dr. Mann, the supplemental materials provided by Dr.
Mann and all the information from other sources, the Inquiry Committee found as follows with respect to each allegation:

Allegation 1: "Did you engage in, or participate in, directly or indirectly, any actions with the intent to suppress or falsify data? "

Decision 1: The Inquiry Committee determined there was no substance to this allegation and further investigation of this allegation was not warranted.

Allegation 2: "Did you engage in, or participate in, directly or indirectly, any actions
with the intent to delete, conceal or otherwise destroy emails, information and/or data, related to AR4, as suggested by Phil Jones?"

Decision 2: The Inquiry Committee determined there was no substance to this allegation and further investigation of this allegation was not warranted.

Allegation 3: "Did you engage in, or participate in, directly or indirectly, any misuse of privileged or confidential information available to you in your capacity as an academic scholar?"

Decision 3: The Inquiry Committee determined there was no substance to this allegation and further investigation of this allegation was not warranted.

Allegation 4: "Did you engage in, or participate in, directly or indirectly, any actions
that seriously deviated from accepted practices within the academic community for proposing, conducting, or reporting research or other scholarly activities?"

Decision 4: The Inquiry Committee determined that "given that information emerged in the form of the emails purloined from CRU in November 2009, which have raised questions in the public's mind about Dr. Mann's conduct of his
research activity, given that this may be undermining confidence in his findings as

a scientist, and given that it may be undermining public trust in science in general
and climate science specifically, an Investigatory Committee of faculty peers from diverse fields should be constituted under RA-I 0 to further consider this allegation."

Page 5 of 19An Investigatory Committee of faculty members with impeccable credentials was appointed and asked to present its findings and recommendations to Dr. Henry C. Foley
within 120 days of being charged.

The charge to the RA-lO Investigatory Committee:

The Investigatory Committee was charged by Dr. Henry C. Foley, Vice President for Research, on March 4, 2010, as follows:

The Investigatory Committee's charge is to determine whether or not Dr. Michael Mann engaged in, or participated in, directly or indirectly, any actions that seriously deviated
from accepted practices within the academic community for proposing, conducting, or reporting research or other scholarly activities.

Sources of support for the related research or publications:

Dr. Mann's research has been sponsored by many different agencies including the
National Science Foundation, the Department of Energy and the National Oceanic and Atmosphere Administration.

Documents available to the Investigatory Committee:

•



### •

•

•

•



### •

•

•



### •

• 376 files containing emails stolen from the Climate Research Unit (CRU) of the University of East Anglia and originally reviewed by the Inquiry Committee
Documents collected by the Inquiry Committee
Documents provided by Dr. Mann at both the Inquiry and Investigation phases
Penn State University's RA-IO Inquiry Report
House of Commons Report HC387-I, March 31,2010 National Academy of Science letter titled, "Climate Change and the Integrity of
Science" that was published in Science magazine on May 7, 2010
Information on the peer review process for the National Science Foundation (NSF) Department of Energy's Guide to Financial Assistance
Information on National Oceanic and Atmospheric Administration's peer review

process Information regarding the percentage of NSF proposals funded
Dr. Michael Mann's curriculum vitae

Interview process:

The interviews were audio-taped and verbatim transcripts were prepared. All interviewed
individuals were provided an opportunity to review the transcripts of their interviews for

accuracy. The transcripts will be maintained in the Office for Research Protections as part of the official record. Statements or information relevant to the Investigatory
Committee's findings are noted in the paragraphs below. The Investigatory Committee
interviewed the following individuals:

Page 6 of 19April 12, 2010: Dr. William Easterling, Dean, College of Earth and Mineral Sciences,
The PelIDsylvania State University April 14, 2010: Dr. Michael Mann, Professor, Department of Meteorology, The Pennsylvania State University April 20, 2010: Dr. William Curry, Senior Scientist, Geology and Geophysics Department, Woods Hole Oceanographic Institution April 20, 2010: Dr. Jerry McManus, Professor, Department of Earth and Environmental Sciences, Columbia University May 5, 2010: Dr. Richard Lindzen, Alfred P. Sloan Professor, Department of Earth, Atmospheric and Planetary Sciences, Massachusetts Institute of Technology

Summary ofluvestigatory Committee's Interview with Dr. Michael E. Mann, Professor, Department of Meteorology, Penn State University

- April 14, 2010

On April 14, 2010, the RA-1O Investigatory Committee (Assmann, Castleman, Irwin, Jablonski, and Vondracek) and Candice Yekel interviewed Dr. Michael Mann. In advance of the interview, the Investigatory Committee prepared several questions focusing on whether Dr. Mann "engaged in, or participated in, directly or indirectly, any actions that seriously deviated from accepted practices within the academic community
for preparing, conducting, or reporting research or other scholarly activities." In addition

to the prepared questions, Investigatory Committee members asked a number of followup questions. Dr. Mann answered the questions in a detailed manner.

The first question was "Would you please tell us what you consider in your field to be accepted, standard practice with regard to sharing data?" A follow-up question asked how
Dr. Mann had dealt with requests for data that were addressed to him during the period
covered by the stolen emails. Dr. Mann offered a brief historical perspective on the issue
of sharing data in his field, concluding with the observation that data are made generally
available (e.g., in the NOAA public database) after those scientists who obtained the data
have had a chance to be the first to publish findings based on the data. He noted that
sometimes data are made available on a collegial basis to specific scientists before those
who collected the data have published their initial findings. Typically, this involves a request to not release the data to others until the data are made publically available by the
scientists who obtained the data. Dr. Mann concluded his answer by stating that he has always worked with data obtained by other scientists, and that when such data were not already in the public domain, he made them available as soon as he was permitted to do

so by those who initially obtained the data.

Dr. Mann drew a distinction between actual data and intermediate data that are produced

as part of the analytic procedures employed. He indicated that while such intermediate
data may occasionally be shared with colleagues, it is not standard practice to publish or make generally available this intermediate data (to which he and others refer to as "dirty laundry" in one of the purloined emails). Finally, he indicated that someone who wanted

to reproduce his work would be able to independently reproduce this intermediate data
and that, in fact, other researchers had done this.

Page 7 of 19The Investigatory Committee next inquired how he constructed his source codes and
what he considered to be accepted practice in his field for publishing source codes. Dr.
Mann indicated that in his field of study, in contrast with some other fields such as economics, publishing the source code was never standard practice until his work and
that of his colleagues came under public scrutiny, resulting in public pressure to do so. He indicated that he initially was reluctant to publish his source codes because the
National Science Foundation had determined that source codes were the intellectual property of the investigator. Also, he developed his source codes using a programming language (FORTRAN 77) that was not likely to produce identical results when run on a computer system different from the one on which it was developed (e.g., different

processor makes/models, different operating systems, different compilers, different compiler optimizations). Dr. Mann reported that since around 2000, he has been using a

more accessible programming style (MATLAB), and since then he has made all source codes available to the research community.

The next question was "Do you believe that the perceived hostility and perceived ulterior
motives of some critics of global climate science influenced your actions with regard to the peer review process, particularly in relation to the papers discussed in the stolen
emails?" Dr. Mann responded by affirming his belief in the importance of the peer review process as a means of ensuring that scientifically sound papers are published, and

not as a means of preventing the publication of papers that are contrary to one's views.
I-Ie elaborated by stating that some of the emails regarding this issue dealt with his

concern (shared by other scientists, the publisher, and some members of the editorial
board of the journal in question) that the legitimacy of the peer review process had been subverted.

Next, Dr. Mann was asked "Did you ever, without first getting express permission from
the original authors, forward to a third party an in-press or submitted manuscript on which you were not a co-author?" In response to this question, Dr. Mann first responded by saying that to the best of his knowledge he had not done so. He then clarified that he

may have forwarded such a manuscript to a specific, close colleague, in the belief that permission to do so had been implicit, based on his close collegial relationships with the paper's authors. An illustrative case of such a circumstance would have been the manuscript by Wahl and Ammann, which Dr. Mann forwarded to Dr. Briffa. In response
to a follow-up question, Dr. Mann asselied that such judgments about implied consent are quite typical in his field, but they are made only as long as it is understood that such sharing would take place only among trusted colleagues who would maintain the confidentiality of the manuscript.

The next question for Dr. Mann was posed as follows: "What is your reply to the email

statements of Dr. McIntyre (a) that he had been referred to an incorrect version of your data at your FTP site (b) that this incorrect version was posted prior to his request and

was not formulated expressly for him and (c) that to date, no source code or other
evidence has been provided to fully demonstrate that the incorrect version, now deleted,
did not infect some of Mann's and Rutherford's other work?" Dr. Mann responded by

Page 8 of 19stating that neither he, nor many of his colleagues, put much reliability in the various
accusations that Dr. McIntyre has made, and that, moreover, there is "no merit
whatsoever to Mr. McIntyre's claims here." Specifically, Dr. Mann repeated that all data,

as well as the source codes requested by Dr. McIntyre, were in fact made available to him. All data were listed on Dr. Mann's FTP site in 2000, and the source codes were made available to Dr. McIntyre about a year after his request was made, in spite of the
fact that the National Science Foundation had ruled that scientists were not required to do

so. The issue of an "incorrect version" of the data came about because Dr. McIntyre had requested the data (which were already available on the FTP site) in spreadsheet format,
and Dr. Rutherford, early on, had unintentionally sent an incorrectly formatted spreadsheet.

In response to a couple of follow-up questions, Dr. Mann stressed that the stolen emails represent part of a larger context of active communication among scientists, and that he
remains on friendly terms with scientists with whom he has had ongoing, and sometimes heated, disagreements about scientific matters. He also commented that he and other
scientists fear that the stolen emails will have a chilling effect on the way scientists
communicate with each other, partly because members of the public may not appreciate
the lingo or jargon (e.g., "dirty laundry" or "trick") that scientists often use when communicating with each other about their science.

At the conclusion of the interview, Dr. Mann indicated that he would be very happy to provide additional information if the Investigatory Committee felt that this would be helpful.

Summary of Investigatory Committee Interview with Dr. William Easterling, Dean, College 

### of Earth and Mineral Sciences, Penn State University

- 

### April 12, 2010

On April 12, 2010, the RA-lO Investigatory Committee (Assmann, Castleman, Irwin, Jablonski, and Vondracek) and Candice Yekel interviewed Dr. William Easterling, Dean
of the College of Earth and Mineral Sciences, Penn State University. The Investigatory
Committee had a number of prepared questions, starting with a request to learn how Dr. Easterling knew Dr. Mann. Dr. Easterling reported that he had known Dr. Mann for
about six or seven years prior to his appointment at Penn State in 2008. In response to a question about when and how he had become aware of the allegations against Dr. Mann,
Dr. Easterling reported that it was the week before Thanksgiving 2009, when he started receiving emails suggesting a connection between the stolen East Anglia emails and Dr.
Mann's work.

The next question for Dr. Easterling was posed as follows: "In your judgment, are accepted and ethical research practices in scientific fields related to global climate change significantly different from such practices in other fields of scientific inquiry?" Dr. Easterling's response to that question was "Absolutely not!" In a follow-up question, Dr. Easterling was asked whether he saw any difference between certain kinds of experimental scientific fields and observational ones like paleoclimatology. He responded by stating that much of what we know about climate change is the result of a combination

Page 9 of 19of observation and numerical modeling, making the classic idea of falsification of a hypothesis, which may be applicable to a laboratory science, of limited applicability in
the study of climate change. Thus, even though there are a number of highly sophisticated, physically sound models that are used to analyze and predict various
features of the earth's climate system, humanjudgments are invariably involved, and a certain amount of subjectivity is introduced.

Another follow-up question inquired about the likely number of different statistical
models that might be applicable to Dr. Mann's work. Dr. Easterling indicated that Dr.
Mann and his colleagues were primarily interested in looking at historical data (which
tend to be "noisy"), using a relatively small number of statistical models, such as principal components analysis, which has a long tradition in various sciences.

The next question addressed to Dr. Easterling was whether, in his judgment, Dr. Mann's
work was very aggressive, very conservative, or somewhere in the middle in how it pOltrayed global warming. Dr. Easterling responded by stating that Dr. Mann's early
work showed a more dramatic upturn in warming, but that his more recent work has led

to the conclusion that the change has been slightly less dramatic. Moreover, Dr. Easterling added that Dr. Mann's findings have been replicated by independent teams of researchers.

Dr. Easterling was asked whether he knew of any other investigations related to the
stolen emails other than the University of East Anglia and Penn State University, and he responded that he was unaware of any others.

The Investigatory Committee then questioned Dr. Easterling about various scientists in
the field of climate science who might be interviewed by the Investigatory Committee regarding their views of what constitutes accepted and ethical practice with regard to the
conduct of research in the field. The Investigatory Committee wanted a choice of
scientists who had disagreed with Dr. Mann's findings as well as others who had agreed
but who had not collaborated with Dr. Mann or his collaborators.

At the conclusion of the interview, Dr. Easterling offered to be available to the Investigatory Committee if the Investigatory Committee members thought that this would
be helpful.

Summary ofInvestigatory Committee Interview with Dr. William Curry, Senior Scientist, Geology and Geophysics Department, Woods Hole Oceanographic Institution

- April 20, 2010

On April 20, 2010, the RA-l 0 Investigatory Committee (Assmann, Castleman, Irwin, Jablonski, and Vondracek) and Candice Yekel interviewed Dr. William Curry, Senior Scientist, Geology and Geophysics Department, Woods Hole Oceanographic Institution.
The Investigatory Committee had four prepared questions, but Investigatory Committee
members were free to ask additional questions as well as follow-up questions as they saw fit.

Page 10 of 19The first question addressed to Dr. Curry was: "Would you please tell us what you consider in your field to be accepted standard practice with regard to sharing data and unpublished manuscripts?" With regard to sharing data, Dr. Curry indicated that standard practice is that once a publication occurs, the pertinent data are shared via some electronic repository. He stated that not all researchers actually comply with this practice,
and that there may be special arrangements with the funding agency, or the journal that publishes the research, that specify when data need to be made available to other
researchers. 

### In Dr. Curry's case, for example, the National Science Foundation allows a two-year window during which he has exclusive rights to his data. After that period he

must make it available to others.

On the issue of sharing unpublished manuscripts, Dr. Curry stated that if the manuscript

was accompanied by a request to keep it confidential, he would not share it with anyone; if it was not accompanied by an explicit request for confidentiality, he might talk about it with colleagues but would not usually forward it.

Next, Dr. Curry was asked: "Would you please briefly explain how codes are developed
in the process of evaluating data in your field, e.g., are these codes significantly different
from published software packages? Then please tell us what you consider in your field to be accepted standard practice with regard to sharing codes." Dr. Curry reported that in
his area, most codes are fairly basic and researchers use software packages to construct them. 

### I-Ie also reported that he was not aware of any public archive for such codes, but
that he was fairly certain that if he asked another researcher to share such codes, he
would most likely get them. He added that overall compliance with requests to share
codes would probably be equal to the rate of compliance with requests for sharing data.

Next, Dr. Curry was asked to respond to the following: "How do the processes of data acquisition, analysis and interpretation in paleoclimatology affect practices of data sharing in the field? Are any of these processes unique to paleoclimatology?" Dr. Curry
asked for clarification and was told that the question referred to whether the laborious and expensive way in which most data are collected in paleoclimatology had an effect on data sharing. He then responded that requests for raw data would be the exception rather than
the rule, because transforming the raw data into usable information is labor intensive and
difficult. Nevertheless, because of NSF requirements, he would release all data after two

years. He added that some scientists, however, do seek to maintain proprietary access to their data even after two years.

Finally, Dr. Curry was asked whether he wanted to share anything else with the Investigatory Committee. 

### In his concluding comments to the Investigatory Committee,
Dr. Curry noted that in the last ten years things have changed rather rapidly with regard

to sharing data and information. He reported that he has become more aware of how he
would be affected if people started asking him step-by-step details of his work, and that
while he has always been diligent about documenting his work, ten years ago he would

not have been able to document every single step in his analytical work. Thus, "accepted practices" are not fixed and are always evolving.

Page II of 19Summary of Investigatory Committee Interview with Professor Jerry McManus, Professor, Department of Earth and Environmental Sciences, Columbia University

- April 20, 2010

On April 20, 2010, the RA-1O Investigatory Committee (Assmann, Castleman, Irwin, Jablonski, and Vondracek) and Candice Yekel interviewed Dr. Jerry McManus, Professor, Department of Earth and Environmental Sciences, Columbia University. The Investigatory Committee had four prepared questions, but Investigatory Committee
members were free to ask additional questions as well as follow-up questions as they saw fit.

To start the interview, Dr. McManus was asked to respond to the following question:
"Would you please tell us what you consider in your field to be accepted standard practice with regard to sharing data [and]

... with regard to sharing unpublished manuscripts?" Dr. McManus responded by first drawing a distinction between published
and unpublished data, noting, however, that there is a range of standard practices with regard to both. Nevertheless, the mode of behavior regarding unpublished data is to share
"in a fairly limited fashion with individuals or groups who make specific requests and typically who are known to the researcher." Regarding published data, Dr. McManus
indicated that standard practice is to make such data available through any of a broad

range of means, including providing access to electronic repositories and institutional archives.

Regarding the sharing of unpublished manuscripts, Dr. McManus indicated that there is a broad range of typical and accepted behaviors, with such manuscripts commonly shared
with a limited number of colleagues. In a follow-up question, it was inquired whether it

may be considered standard practice to share an unpublished manuscript with others
without getting express permission to do so from the author. Dr. McManus responded by saying "no" to such sharing as standard practice, but allowing that there is not necessarily only one acceptable practice, as permission may be given implicitly or explicitly. Without specific encouragement for wider distribution, however, it is generally understood, according to Dr. McManus, that unpublished papers are not intended for third-party
distribution.

The next question was stated as follows: "Would you please briefly explain how codes

are developed in the process of evaluating data in your field (e.g., are these codes significantly different from published software packages)? Then please tell us what you consider in your field to be accepted, standard practice with regard to sharing codes." Dr.
McManus indicated that most, but not all, details of such methods are usually reported
when research is published, and that some of these details may be shared in a "somewhat
ad hoc basis." Generally, however, the tendency is to "try to provide the conditions by
which any research can be replicated

.... " Dr. McManus agreed that generally, codes are treated the same way as any other method.

Page 12 of 19Summary of Investigatory Committee Interview with Dr. Richard Lindzen, Alfred
P. Sloan Professor, Department of Earth, Atmospheric, and Planetary Sciences,
Massachusetts Institute of Technology

- May 5, 2010

On May 5 20 I 0, the RA-I 0 Investigatory Committee (Assmann, Irwin, Jablonski, Vondracek; Dr. Castleman was not available) and Candice Yekel interviewed Dr.
Richard Lindzen, Professor, Department of Earth, Atmospheric, and Planetary Sciences,
Massachusetts Institute of Technology. The Investigatory Committee had four prepared questions, but Investigatory Committee members were free to ask additional questions as well as follow-up questions as they saw fit.

Before the Investigatory Committee's questioning began, Dr. Lindzen was given some general background information regarding the process of inquiry and investigation into allegations concerning Dr. Mann, with a focus on the particular allegation that is the subject of the current review by the Investigatory Committee. Dr. Lindzen then requested, and was provided with, a brief summary of the three allegations previously
reviewed. When told that the first three allegations against Dr. Mann were dismissed at the inquiry stage of the RA-lO process, Dr. Lindzen's response was: "It's thoroughly amazing. I mean these are issues that he explicitly stated in the emails. I'm wondering what's going on?"

The Investigatory Committee members did not respond to Dr. Lindzen's statement. Instead, Dr. Lindzen's attention was directed to the fourth allegation, and it was explained to him that this is the allegation which the Investigatory Committee is charged

to address. Dr. Lindzen was then asked the first question formulated by the Investigatory
Committee: "Would you please tell us what you consider in your field to be accepted,
standard practice with regard to sharing data, and the second part of the question is would

you tell us what you consider in your field to be accepted, standard practice with regard

to sharing unpublished manuscripts?"

Dr. Lindzen responded by stating that "with respect to sharing data, the general practice
is to have it available." With respect to unpublished manuscripts, he indicated that "those

are generally not made available unless the author wishes to." 

### In response to a number of follow-up questions, Dr. Lindzen indicated that if an unpublished manuscript is sent to a scientist by the author, it would be common practice to ask for permission before sharing
it with others; if it was sent by someone else it would be common practice to ask if they
had permission to share the paper. According to Dr. Lindzen, a scientist might conclude
that there is implicit permission to disseminate an unpublished paper only when the
author made it clear that the results may be disseminated.

The next question inquired whether, in Dr. Lindzen's view, climatologists normally make
their codes (used in the analysis of data) available for other people to download. Dr.
Lindzen responded by stating that "it depends." He elaborated, saying that if the codes

are very standard, it is unnecessary to share them, but if it's an unusual analysis it would
be his practice to make the codes available to anyone who wishes to check them. 

### In a follow-up question, Dr. Lindzen was asked whether he would have issues with people

Page 13 of 19running into compatibility issues or compilation issues. He responded by saying that even if people "screw it up" or if you have reservations about sharing codes, "if somebody
asks you how did you get this, you really should let them know."

The next questions presented to Dr. Lindzen were as follows: "How do the processes of
data acquisition, analysis, and interpretation in paleoclimatology affect practices of data sharing in the field? Are any of these processes unique to paleoclimatology?" Dr.
Lindzen indicated that he did not think that these processes are unique to paleoclimatology, and that since most of the data are acquired using public funds, there is

no basis for investigators being proprietary with their data. In response to a follow-up question, Dr. Lindzen acknowledged that prior to publication, scientists may have a variety of reasons to keep things confidential, but after publication "there's an obligation

to explain exactly how you got them, especially if they're controversial."

Standard of proof used 

### by the Investigatory Committee:

Preponderance a/the evidence (happen more likely than not or 51 % certainty). All
committee votes are unanimous unless otherwise indicated.

Level of intent considered 

### by the Investigatory Committee:

The Investigatory Committee considered various levels of intent in order of increasing severity from careless, to reckless, to knowingly, to intentional. These terms are defined

as follows:

• careless

- a reasonable person would not have known better or honest errorthis is not considered research misconduct. reckless

- a reasonable person should have known better. knowingly

- a reasonable person knew better but did it anyway.

• intentional (purposejil1)

- a reasonable person knew better but did it anyway with the intent to deceive.

The level of intent regarding the specific allegation will be addressed below.

Summary of Investigation:

The Investigatory Committee investigated the following potential acts of misconduct:

"Did Dr. Michael Mann engage in, or participate in, directly or indirectly, any actions
that seriously deviated from accepted practices within the academic community for proposing, conducting, or reporting research or other scholarly activities?"

The Investigatory Committee was given access to 376 files that contained emails stolen
from the Climate Research Unit (CRU) of the University of East Anglia. These emails

were either sent by Dr. Mann, sent to Dr. Mann, copied to Dr. Mann, or discussed Dr.
Mann (but were neither addressed nor copied to him). The Investigatory Committee also

Page 14 of 19reviewed the documents collected by the Inquiry Committee, as well as the Inquiry
Committee's findings and report. In addition, the Investigatory Committee reviewed a number of documents provided by Dr. Mann in response to requests from both the Inquiry and Investigatory Committees. A number of public documents were also made
available to the Investigatory Committee, including a number of editorials, both pro and

con Dr. Mann, an open letter from 255 members of the National Academy of Sciences, published in Science magazine, May 7, 2010, and the full text of the British House of
Commons' Science and Technology Committee report on "The disclosure of climate data
from the Climatic Research Unit at the University of East Anglia," which was published

on March 31, 2010.

In the course of the investigation, the Investigatory Committee interviewed Dr. Michael Mann, as well as his immediate supervisor, Dr. William Easterling, Dean of the College
of Earth and Mineral Sciences at the Pennsylvania State University. Dean Easterling and
Dr. David Verardo, National Science Foundation Program Director for Paleo Perspectives on Climate Change, agreed to suggest names of eminent scientists who might agree to be interviewed by the Investigatory Committee in its efforts to establish
the range of "accepted practices within the academic community for proposing, conducting, or reporting research or other scholarly activities." As previously described,
the Investigatory Committee contacted, and subsequently interviewed, three eminent
scientists from the field of climate research: Dr. William Curry, Senior Scientist, Geology
and Geophysics Department, Woods Hole Oceanographic Institution; Dr. Richard Lindzen, Alfred P. Sloan Professor, Department of Earth, Atmospheric, and Planetary Sciences, Massachusetts Institute of Technology; and Dr. Jerry McManus, Professor, Department of Earth and Environmental Sciences, Columbia University.

Based on the documentary evidence and on information obtained from the various interviews, the Investigatory Committee first considered the question of whether Dr.
Mann had seriously deviated from accepted practice in proposing his research activities. First, the Investigatory Committee reviewed Dr. Mann's activities that involved proposals

to obtain funding for the conduct of his research. Since 1998, Dr. Mann received funding
for his research mainly from two sources: The National Science Foundation (NSF) and
the National Oceanic and Atmospheric Administration (NOAA). Both of these agencies
have an exceedingly rigorous and highly competitive merit review process that represents

an almost insurmountable barrier to anyone who proposes research that does not meet the highest prevailing standards, both in terms of scientific/technical quality and ethical
considerations.

NOAA and NSF research grant proposals are both evaluated through similarly rigorous
and transparent merit review (peer review) processes. To illustrate, we describe the NSF
review process, which has two stages. In Stage 1, proposals are sent out to several
external experts for merit review (mail review) based on the two NSF review criteria
established by the National Science Foundation Board

-- Intellectual Merit and Broader Impacts. In Stage 2, the proposal and its external expert reviews (mail reviews) are taken

to a 8-15 person external expert panel and evaluated over a several day period (panel review). Panel review members are not the same persons as the mail review members. In

Page 15 of 19Stage I, the external reviewers only see individual proposals and rate them on a 5-point
scale in descending order from Excellent, Very Good, Good, Fair, and Poor. In Stage 2,
the entire panel (except those members who have a conflict of interest with the proposal)

see all the proposals in the competition (usually about 140 proposals in the NSF program
to which Dr. Mann has typically submitted his proposals) and rate them based on the

same two NSF criteria on the same rating scale, but at this stage they evaluate the proposals in comparison with all the other proposals that were submitted. All reviews are then taken under advisement by the director of the particular NSF program to which the proposal was submitted, who then recommends whether a project should be funded. The

program director is guided by the expert reviews, but may also take programmatic
balance and other NSF criteria into account before making a final recommendation. The

rate of funding varies by program, but rarely exceeds 25 percent.

The results achieved by Dr. Mann in the period 1999-2010, despite these stringent requirements, speak for themselves: He served as principal investigator or co-principal investigator on five NOAA-funded and four NSF-funded research projects. During the

same period, Dr. Mann also served as co-investigator of five additional NSF-and
NOAA-funded research projects, as well as on projects funded by the Department of Energy (DOE), the United States Agency for International Development (USAID), and
the Office of Naval Research (ONR). This level of success in proposing research, and obtaining funding to conduct it, clearly places Dr. Mann among the most respected
scientists in his field. Such success would not have been possible had he not met or exceeded the highest standards of his profession for proposing research.

The second part of the Investigatory Committee's charge was to investigate whether Dr.
Mann had engaged in any actions that seriously deviated from accepted practices within
the academic community for conducting research or other scholarly activities. One focus
of the committee's investigation centered on whether Dr. Mann had deviated from accepted practice with regard to sharing data and source codes with other investigators. First, the Investigatory Committee established that Dr. Mann has generally used data
collected by others, a common practice in paleoclimatology research. Raw data used in
Dr. Mann's field of paleoclimatology are laboriously collected by researchers who obtain

core drillings from the ocean floor, from coral formations, from polar ice or from glaciers, or who collect tree rings that provide climate information from the past millennium and beyond. Other raw data are retrieved from thousands of weather stations
around the globe. Almost all of the raw data used in paleoclimatology are made publicly available, typically after the originators of the data have had an initial opportunity to evaluate the data and publish their findings. In some cases, small sub-sets of data may be protected by commercial agreements; in other cases some data may have been released to close colleagues before the originators had time to consummate their prerogative to have

a limited period (usually about two years) of exclusivity; in still other cases there may be legal constraints (imposed by some countries) that prohibit the public sharing of some climate data. The Investigatory Committee established that Dr. Mann, in all of his published studies, precisely identified the source(s) of his raw data and, whenever possible, made the data and or links to the data available to other researchers. These

Page 16 of 19actions were entirely in line with accepted practices for sharing data in his field of research.

With regard to sharing source codes used to analyze these raw climate data and the
intermediate calculations produced by these codes (referred to as "dirty laundry" by Dr.
Mann in one of the stolen emails) with other researchers, there appears to be a range of accepted practices. Moreover, there is evidence that these practices have evolved during
the last decade toward increased sharing of source codes and intermediate data via
authors' web sites or web links associated with published scientific journal articles. Thus, while it was not considered standard practice ten years ago to make such
information publicly available, most researchers in paleoclimatology are today prepared

to share such information, in part to avoid unwarranted suspicion of improprieties in their

treatment of the raw data. Dr. Mann's actual practices with regard to making source codes and intermediate data readily available reflect, in all respects, evolving practices
within his field. Dr. Mann acknowledged that early in his career he was reluctant to publish his source codes because the National Science Foundation had determined that

source codes were the intellectual propelty of the investigator. Moreover, because he developed his source codes using a specific programming language (FORTRAN 77),
these codes were not likely to compile and run on computer systems different from the

ones on which they were developed (e.g., different processor makes/models, different operating systems, different compilers, different compiler optimizations). Since then, however, he has used a more accessible method for developing his source codes (MA TLAB) and he has made all source codes, as well as intermediate data, available to the research community, thereby meeting and exceeding standard practices in his field. Moreover, most of his research methodology involves the use of Principal Components Analysis, a well-established mathematical procedure that is widely used in climate
research and in many other fields of science. Thus, the Investigatory Committee
concluded that the manner in which Dr. Mann used and shared source codes has been
well within the range of accepted practices in his field.

The issue of whether Dr. Mann had engaged in any actions that seriously deviated from accepted practices within the academic community for conducting research or other scholarly activities was examined by the Investigatory Committee via a number of
additional means. When a scientist's research findings are well outside the range of findings published by other scientists examining the same or similar phenomena, legitimate questions may be raised about whether the science is based on accepted practices or whether questionable methods might have been used. Most questions about
Dr. Mann's findings have been focused on his early published work that showed the "hockey stick" pattern of climate change. In fact, research published since then by Dr.
Mann and by independent researchers has shown patterns similar to those first described by Dr. Mann, although Dr. Mann's more recent work has shown slightly less dramatic changes than those reported originally. In some cases, other researchers (e.g., Wahl & Ammann, 2007) have been able to replicate Dr. Mann's findings, using the publicly
available data and algorithms. The convergence of findings by different teams of researchers, using different data sets, lends further credence to the fact that Dr. Mann's
conduct of his research has followed acceptable practice within his field. Further support

Page 17 of 19for this conclusion may be found in the observation that almost all of Dr. Mann's work

was accomplished jointly with other scientists. The checks and balances inherent in such

a scientific team approach further diminishes chances that anything unethical or inappropriate occurred in the conduct of the research.

A particularly telling indicator of a scientist's standing within the research community is
the recognition that is bestowed by other scientists. Judged by that indicator, Dr. Mann's work, from the beginning of his career, has been recognized as outstanding. For example,
he received the Phillip M. Orville Prize for outstanding dissertation in the earth sciences

at Yale University in 1997. In 2002, he received an award from the Institute for
Scientific Information for a scientific paper (published with co-authors) that appeared in
the prestigious journal Nature; also in 2002, he co-authored a paper that won the Outstanding Scientific Paper Award from the NOAA Office of Oceanic and Atmospheric Research, and Scientific American named him as one of 50 leading visionaries in science
and technology. In 2005, Dr. Mann co-authored a paper in the Journal of Climate that

won the John Russell Mather Paper award from the Association of American Geographers, and in the same year, the website "RealClimate.org" (co-founded by Dr. Mann) was chosen as one of the top 25 "Science and Technology" websites by Scientific
American. In 2006, Dr. Mann was recognized with the American Geophysical Union
Editors' Citation for Excellence in Refereeing (i.e., reviewing manuscripts for Geophysical Research Letters). All of these awards and recognitions, as well as others

not specifically cited here, serve as evidence that his scientific work, especially the
conduct of his research, has from the beginning of his career been judged to be outstanding by a broad spectrum of scientists. Had Dr. Mann's conduct of his research
been outside the range of accepted practices, it would have been impossible for him to receive so many awards and recognitions, which typically involve intense scrutiny from
scientists who mayor may not agree with his scientific conclusions.

The third area of investigation was to address whether Dr. Mann had engaged in any actions that seriously deviated from accepted practices within the academic community
for reporting research or other scholarly activities. Dr. Mann's record of publication in

peer reviewed scientific journals offers compelling evidence that his scientific work is highly regarded by his peers, thus offering de facto evidence of his adherence to established standards and practices regarding the reporting of research. To date, Dr.
Mann is the lead author of39 scientific publications and he is listed as co-author on an additional 55 pUblications. The majority of these publications appeared in the most highly respected scientific journals, i.e., journals that have the most rigorous editorial and peer reviews in the field. In practical terms, this means that literally dozens of the most highly qualified scientists in the world scrutinized and examined every detail of the scientific
work done by Dr. Mann and his colleagues and judged it to meet the high standards

necessary for publication. Moreover, Dr. Mann's work on the Third Assessment Report (2001) of the Intergovernmental Panel on Climate Change received recognition (along
with several hundred other scientists) by being awarded the 2007 Nobel Peace Prize. Clearly, Dr. Mann's reporting of his research has been successful and judged to be outstanding by his peers. This would have been impossible had his activities in reporting
his work been outside of accepted practices in his field.

Page 18 of 19One issue raised by some who read the stolen emails was whether Dr. Mann distributed privileged information to others to gain some advantage for his interpretation of climate change. The privileged information in question consisted of unpublished manuscripts that

were sent to him by colleagues in his field. The Investigatory Committee determined that

none of the manuscripts were accompanied by an explicit request to not share them with
others. Dr. Mann believed that, on the basis of his collegial relationship with the manuscripts' authors, he implicitly had permission to share them with close colleagues. Moreover, in each case, Dr. Mann explicitly urged the recipients of the unpublished manuscripts to first check with the authors if they intended to use the manuscripts in any
way. Although the Investigatory Committee determined that Dr. Mann had acted in good
faith with respect to sharing the unpublished manuscripts in question, the Investigatory
Committee also found that among the expelis interviewed by the Investigatory
Committee there was a range of opinion regarding the appropriateness of Dr. Mann's
actions. Opinions ranged from one expert who contended that it is never acceptable to share an unpublished manuscript without first obtaining explicit permission from the author(s) to do so, to another expert who felt that, when working with close colleagues, it
is sometimes acceptable to do so by assuming that implicit permission had been granted.
The Investigatory Committee considers Dr. Mann's actions in sharing unpublished manuscripts with third parties, without first having received express consent from the
authors of such manuscripts, to be careless and inappropriate. While sharing an unpublished manuscript on the basis of the author's implied consent may be an acceptable practice in the judgment of some individuals, the Investigatory Committee believes the
best practice in this regard is to obtain express consent from the author before sharing an unpublished manuscript with third parties.

The Investigatory Committee would like to note that Dr. Mann, after being questioned by
the Investigatory Committee about this issue, requested and received confirmation that
his assumption of implied consent was correct from the author of one of the papers in question. This "after the fact" communication was not considered by the Investigatory
Committee in reaching its decision.

Conclusion of the Investigatory Committee as to whether research misconduct
occurred:

The Investigatory Committee, after careful review of all available evidence, determined
that there is no substance to the allegation against Dr. Michael E. Mann, Professor, Department of Meteorology, The Pennsylvania State University.

More specifically, the Investigatory Committee determined that Dr. Michael E. Mann did

not engage in, nor did he participate in, directly or indirectly, any actions that seriously
deviated from accepted practices within the academic community for proposing, conducting, or reporting research, or other scholarly activities.

The decision of the Investigatory Committee was unanimous.

Page 19 of 19snijlijn snijlijn rug PBL-omslag zonder flappen
340 x 240 mm

**Hier komt bij voorkeur een flaptekst te staan, in plaats van deze algemene tekst** 

Het Planbureau voor de Leefomgeving is hét nationale instituut voor strategische beleidsanalyses op het gebied van milieu, natuur en ruimte.

Het PBL draagt bij aan de kwaliteit van het strategische overheidsbeleid door een brug te vormen tussen wetenschap en beleid en door gevraagd en ongevraagd,onafhankelijk en wetenschappelijk gefundeerd, verkenningen, analyses en evaluaties te verrichten waarbij een integrale benadering voorop staat.

Planbureau voor de Leefomgeving

Locatie Bilthoven Postbus 303 3720 AH Bilthoven
T: 030 274 274 5 F: 030 274 4479 E: info@pbl.nl www.pbl.nl

Planbureau voor de Leefomgeving, <maand> 200X **Assessing an IPCC assessment**The Netherlands Environmental Assessment Agency (PBL) has investigated the scientific foundations for the IPCC summary conclusions of the Fourth Assessment Report of 2007 on projected regional climate-change impacts, at the request of the Dutch Minister for the Environment. Overall the summary conclusions are considered well founded and none were found to contain any significant errors.

The Working Group II contribution to the Fourth Assessment Report shows ample observational evidence of regional climate-change impacts, which have been projected to pose substantial risks to most parts of the world, under increasing temperatures. However, in some instances the foundations for the summary statements should have been made more transparent. While acknowledging the essential role of expert judgment in scientific assessments, the PBL recommends to improve the transparency of these judgments in future IPCC reports.

In addition, the investigated summary conclusions tend to single out the most important negative impacts of climate change. Although this approach was agreed to by the IPCC governments for the Fourth Assessment Report, the PBL recommends that the full spectrum of regional impacts is summarised for the Fifth Assessment Report, including the uncertainties. The PBL believes that the IPCC shouldinvest more in quality control in order to prevent mistakes and shortcomings, to the extent possible.

Netherlands Environmental Assessment Agency, July 2010 

# Assessing an IPCC assessment An analysis of statements on projected regional impacts in the 2007 report

# Assessing an IPCC assessment An analysis of statements on projected regional impacts in the 2007 report**Assessing an IPCC assessment. An analysis of statements on projected regional impacts in the 2007 report**  ©Netherlands Environmental Assessment Agency (PBL) The Hague/Bilthoven, 2010

PBL publication number: 500216002 Cover photo: Hollandse Hoogte

Corresponding author: leo.meyer@pbl.nl

This publication can be downloaded from: www.pbl.nl/en. A hard copy may be ordered from: reports@pbl.nl, citing the PBL publication number or ISBN.

Parts of this publication may be reproduced, providing the source is stated, in the form: Netherlands Environmental Assessment Agency, the publication title.

The Netherlands Environmental Assessment Agency (PBL) is the national institute for strategic policy analysis in the field of environment, nature and spatial planning. We contribute to improving the quality of political and administrative decision-making by conducting outlook studies, analyses and evaluations in which an integrated approach is considered paramount. Policy relevance is the prime concern in all our studies. We conduct solicited and unsolicited research that is both independent and always scientifically sound.

Netherlands Environmental Assessment Agency Office The Hague Office Bilthoven
PO Box 30314 PO Box 303 2500 GH The Hague 3720 AH Bilthoven

The Netherlands The Netherlands

T elephone: +31 (0)70 328 8700 T elephone: +31 (0)30 274 2745 Fax: +31 (0)70 328 8799 Fax: +31 (0)30 274 4479

E-mail: info@pbl.nl

W ebsite: www.pbl.nl/enForeword 5 

# Foreword

In January 2010, the media reported two errors in specific parts of the Fourth Assessment Report of 2007 by the Intergovernmental Panel on Climate Change (IPCC). The media reports gave rise to questions about the credibility of the overall IPCC assessment: could politics and society still rely on the IPCC for an assessment of the scientific knowledge on climate change?

On 28 January 2010, the Dutch Parliament asked Jacqueline Cramer, the then Dutch Minister for the Environment, for an investigation into the implications of these errors in the IPCC report. Minister Cramer subsequently requested the Netherlands Environmental Assessment Agency (PBL) **1)** to make an assessment of the reliability of the regional chapters (9 to 16) of the IPCC Working Group II contribution to the Fourth Assessment Report (the part on Impacts, Adaptation and Vulnerability), and to assess the effects of any errors on the summary conclusions drawn by the IPCC.

From the outset it was clear that the PBL would not be able to repeat the original IPCC Working Group II assessment, given the sheer volume of the scientific work reported in the IPCC exercise. **2)** What we were able to do, however, was investigate the extent to which the IPCC in their summaries had presented existing scientific knowledge to the world of policymakers, in a way that was supported by the underlying texts and scientific references. What is more, we also investigated what could be learned from our findings so that informed choices can be made in future assessments. This seemed the more reasonable approach, as behind the request from the Dutch Parliament was a more general concern: could policymakers and the public at large still trust the IPCC's key messages?

A scientific assessment – an analysis and weighing of the actual scientific knowledge – of a complex problem such as climate change is a tremendously difficult task. The IPCC is a bridge between science and policy: a so-called ‘science– policy interface’. The IPCC is not conducting science, nor is it making policy, but should provide the guarantee that policymakers can come to their decisions on the basis of the best available knowledge regarding climate change. W e have

1) T he Netherlands Environmental Assessment Agency (PBL) is an independent governmental body that by statute providesthe Dutch Government and Parliament – and the European Commission, European Parliament and UN organisations – with scientific advice on problems regarding the environment, sustainability and spatial planning. PBL authors have contributed to several assessments of the IPCC. During the Third and Fourth Assessments, the PBL hosted the Co-Chair and Technical Support Unit of Working Group III (Mitigation on Climate Change). 2) E lsewhere we have reported on new insights into climate science since the Fourth Assessment Report (cf. News in Climate Science and Exploring Boundaries , 2009).Assessing an IPCC assessment 6 investigated how this task was taken on by the IPCC Working Group II in their assessment of the regional impacts of climate change. Based on this investigation we have formulated some recommendations to further improve the quality of
these assessments.

The investigation was conducted realising that climate science and policymaking are currently taking place in a new era, characterised by a high degree of politicisation, a much more dynamic interaction between science and public discourse, and vocal citizens who either want to know whether policy measures under discussion are all really necessary, or who are of the opinion that suggested measures do not go far enough. This new context comes with new requirements for the IPCC assessments. Our suggestions are thus not always meant as criticism of the ‘architecture’ of the climate report of Working Group II as this was being drawn up in the years leading up to its publication in 2007. Yet we believe that, by being critical and self-critical of work contained in the Fourth Assessment Report, we can contribute to further improving the reliability of future IPCC reports. It is an exercise in future-oriented learning.

Accomplishing our task would have been impossible without the cooperation of the IPCC authors of Working Group II, who were willing to devote a considerable amount of their time to answering our questions, and did so within the strict time tables we had set them. It is clear that the responsibility for the research, as well as for the presentation of the findings, the conclusions drawn, and the recommendations made, lies solely with the Netherlands Environmental Assessment Agency (PBL).

Our investigation is focused in character and size. However, a number of conclusions and recommendations concern the set up of the procedures and processes of the IPCC. Hopefully, those suggestions can still play a role in the review by the InterAcademy Council, that will become available by late August 2010, focusing on procedural and process issues of the IPCC. **3)**

Professor Maarten Hajer Director, Netherlands Environmental Assessment Agency (PBL)

3) InterAcademy Council 2010, Review of the IPCC; an evaluation of the procedures and process of the Intergovernmental Panel on Climate Change, http://reviewipcc.interacademycouncil.net/index.htmlForeword 7 

# Contents

 Executive summary 9

P art I – Main findings

 1 I ntroduction 1 9

1.1 S cientific assessment in aid of policy 1 9

1.2 S ome background on IPCC Working Group II 2 2

1.3 S cope, objectives and limitations of this report 2 4

1.4 St ructure of the report 2 6

 2 M ethodology 2 7

2.1 I nvestigative approach 2 7

2.2 T ypology of errors and qualifications 2 9

 3 Re sults and discussion 3 3

3.1 St atus of IPCC summary statements on regional climate-change impacts 3 3

3.2 E rrors 3 5

3.3 R isk-oriented approach 3 7

3.4 Co mments 4 0

3.5 Co nclusion 4 3

 4 Re commendations 4 5

4.1 M inimising the risk of errors 4 5

4.2 I nvesting in the improvement and transparency of foundations of summary conclusions 4 6

4.3 St rengthening the quality control by the chapter teams 4 7

4.4 St rengthening the review process 4 7

4.5 T iming of the assessments 4 8

4.6 B alancing the assessment of climate-change impacts 4 8

4.7 I nvesting in climate-change impact science 5 0

Part II – Detailed analysis of regional chapters and summaries

 5 A frica 5 3

5.1 A nalysis of statements in Table SPM.2 of the Synthesis Report 5 3

5.2 A dditional findings 5 5

5.3 F indings from the PBL registration website 5 6Assessing an IPCC assessment 8  6 A sia 5 9

6.1 A nalysis of statements in Table SPM.2 of the Synthesis Report 5 9

6.2 A dditional findings 6 0

 7 A ustralia and New Zealand 6 3

7.1 A nalysis of statements in Table SPM.2 of the Synthesis Report 6 3

7.2 A dditional findings 6 4

 8 E urope 6 7

8.1 A nalysis of statements in Table SPM.2 of the Synthesis Report 6 7

8.2 A dditional findings 6 8

8.3 F indings from the PBL registration website 7 0

 9 L atin America 7 3

9.1 A nalysis of statements in Table SPM.2 of the Synthesis report 7 3

9.2 A dditional findings 7 4

 10 N orth America 7 7

10.1 A nalysis of statements in Table SPM.2 of the Synthesis Report 7 7

10.2 A dditional findings 78

 11 P olar regions (Arctic and Antarctic) 7 9

11.1 A nalysis of statements in Table SPM.2 of the Synthesis Report 7 9

 12 S mall islands 8 1

12.1 A nalysis of statements in Table SPM.2 of the Synthesis Report 8 1

12.2 A dditional findings 8 1

 Annex A T able SPM.2 of the Synthesis Report: examples of some projected regional impacts 8 3

 Annex B T he error on the melting of the Himalayan glaciers 86

 Annex C T he error on the Dutch land area below sea level 89

 Annex D S ea level rise: consequences for the Netherlands 9 1

 Annex E A bbreviations 9 6

 Colophon 9 9Executive summary 9 

# Executive summary

The Netherlands Environmental Assessment Agency (PBL) has investigated the scientific foundations for the IPCC summary conclusions of the Fourth Assessment Report of 2007 on projected regional climate-change impacts, at the request of the Dutch Minister for the Environment. Overall the summary conclusions are considered well founded and none were found to contain any significant errors. The Working Group II contribution tothe Fourth Assessment Report shows ample observational evidence of regional climatechange impacts, which have been projected to pose substantial risks to most parts of the world, under increasing temperatures. However, in some instances the foundations for the summary statements should have been made more transparent. While acknowledging the essential role of expert judgment in scientific assessments, the PBL recommends to improve the transparency of these judgments in future IPCC reports. In addition, the investigated summary conclusions tend to single out the most important negative impacts of climate change. Although this approach was agreed to by the IPCC governments for the Fourth Assessment Report, the PBL recommends that the full spectrum of regional impacts is summarised for the Fifth Assessment Report, including the uncertainties. The PBL believes that the IPCC should invest more in quality control in order to prevent mistakes and shortcomings, to the extent possible.

N o significant errors found in summary conclusions The foundations for thirty-two IPCC Fourth Assessment summary conclusions on the regional impacts of climate change have been investigated. These conclusions show examples of projections of climate-change impacts on food, water, ecosystems, coastal regions and health, for all the earth’s continents. These conclusions have not been undermined by errors, although one of the conclusions contains a minor inaccuracy: in hindsight, not 75 to 250 million people, but 90 to 220 million people are projected to be exposed to increased water stress due to climate change in Africa, by 2020. Given the large uncertainties surrounding such projections, this difference is not significant.

Provenance of summary statements needs to become more transparent in future
reports Seven of the investigated 32 conclusions on the regional impacts of climate change contain information that we were unable to sufficiently trace to the underlying chapters in the IPCC Working Group II Report or to the references therein. For two of these conclusions, our critical comments pertain to insufficiently founded generalisations from existing scientific research, in both cases from local to regional scales, and for one of them also from one type of livestock to livestock in general. The PBL recommends to invest in the improvement and transparency of the foundations of summary conclusions in future IPCC reports.Assessing an IPCC assessment 10 The regional chapters: one significant error and some comments In the underlying regional chapters, in addition to the two already known errors about the melting of the Himalayan glaciers and about the Dutch land area below sea level, another significant error was found: a 50 to 60% decrease in productivity in anchovy fisheries on the African west coast was projected on the basis of an erroneous interpretation of the literature references. It appeared to be about a 50 to 60% decrease in extreme wind and seawater turbulence, with some effects on the anchovy population that were not quantified. We found certain inaccuracies, ranging from (very) small errors in numbers to imprecise literature references. In addition, the PBL has some critical comments to make. One of these relatesto the fact that the report does not specify how many of the additional heatrelated deaths projected for Australian cities are actually attributable to climate change – a sizeable fraction is due to demographic changes alone. However, these shortcomings do not affect the investigated 32 summary conclusions or other parts
of the IPCC summaries.

Examples of negative impacts dominate at summary level The IPCC Working Group II Report focuses on climate impacts, adaptation and vulnerability. It was found that, in the IPCC’s highest level summary, the conclusions that were derived from the regional chapters of the Working Group II contribution to the Fourth Assessment Report single out examples of projections of negative climate-change impacts. The IPCC authors considered these to be the most relevant to policymakers. The PBL has labelled this as a ‘risk-oriented’ approach, which had been implicitly endorsed by the governments that constitute the IPCC (including that of the Netherlands). The PBL subscribes to the importance of an approach that highlights what may go wrong under unmitigated climate change, but the Working Group II Report lacked a clear explanation of the choice of approach and its consequences. Alternatively, it could be argued that policymakers should be presented with a complete picture in the Summaries for Policymakers, not just with negative examples (without suggesting that potential positive effects cancel out potential negative effects). We recommend that, for the Fifth Assessment Report, the choice of the approach taken is made explicit. Moreover, we suggest that, at Summary for Policymakers level, two separate sections are included dealing with projected regional impacts on water, food, ecosystems, coastal regions and health:

 One section that describes the projected full range of climate-change impacts, including uncertainties, positive impacts, and the relative contributions from other important areas, such as industrialisation, population growth, and land

use;

 One section that describes the most important negative impacts, including a worst-case risk approach, based on a clearly explicated risk-assessment methodology.

N o consequences for overarching conclusions Our findings do not contradict the main conclusions of the IPCC on impacts, adaptation and vulnerability related to climate change. There is ample observational evidence of natural systems being influenced by climate change on regional levels. The negative impacts under unmitigated climate change in the future pose substantial risks to most parts of the world, with risks increasing at higher global average temperatures.Executive summary 11 Other recommendations Additional recommendations for future IPCC publications include:

 Create a public website for the submission of possible errors found in the published reports;

 Provide stronger underpinning of generalisations of case studies to entire regions or sectors, also making use of regional modelling studies;

 Ensure that statements that attribute impacts to climate change are well founded in scientific research, including systematic observations, modelling and statistics. The climate change component of impacts should be carefully
characterised.

 Be careful with phrasing of statements that could be perceived by readers as heightening the projected impacts of climate change;

 IPCC governments should provide financial support for hiring chapter assistants to assist with quality control;

 Assure that the reviews of all draft texts are fully covered by several expert reviewers;

 Strengthen the expert and government reviews of the foundation for and provenance of statements in the summaries;

 IPCC governments should increase their investments in climate-change observations and modelling in developing countries.

T he Dutch Parliament resolution and the subsequent assignment given to the PBL In January 2010, worldwide media attention was given to two errors that were discovered in a part of the Fourth Assessment Report of the IPCC of 2007: an erroneously high rate of melting of the Himalayan glaciers, and an erroneously high percentage of land area in the Netherlands lying below sea level. The commotion in the Dutch media and the subsequent discussion at the political level in the Netherlands led to a resolution by the Dutch Parliament on 28 January 2010, which declared in the preamble that the reliability of the IPCC should be undisputed but was now at issue. In the resolution, the parliament required that the government would instruct the Netherlands Environmental Assessment Agency (PBL) to provide a new update on climate science, including the implications of the said errors.

Based on this resolution and the debate in parliament, the Minister for the Environment decided to limit her question to PBL to an investigation in the implications of possible errors in the regional chapters of the IPCC report of 2007 on climate-change impacts, adaptation and vulnerability, paying specific attention to the Himalayan glaciers.

Specification of the assignment The PBL, subsequently, broadened the assignment somewhat by including an investigation of the conclusions on regional impacts, as described in the highest summary level of the combined IPCC 2007 reports (the ‘Synthesis Report’). In addition, the investigation report would also include annexes on the Himalayan glaciers, on the percentage of land area in the Netherlands lying below sea level, and on the protection of the Netherlands against sea level rise. The last annex was added, since the errors about the Himalayan glacier melting and about the percentage of Dutch land area below sea level had triggered a political discussion on whether Dutch policies dealing with sea level rise would need to be revised.Assessing an IPCC assessment 12 For the ‘update on climate science’, as requested by the Dutch Parliament, we refer to a PBL study ‘News in Climate Science and Exploring the Boundaries’ (December, 2009). That report, drawn up together with the Royal Netherlands Meteorological Institute (KNMI) and Wageningen University and Research Centre (WUR), revisited the IPCC conclusion that global warming since the middle of the 20th century is very likely to have been due to human influence on the global climate. This statement was found to be robust, even when taking into account peer-reviewed scientific literature expressing doubts on this relation. We took this and other findings of the Working Group I as a point of departure for our investigation into the IPCC statements on the projected regional impacts of climate change. **1)**

Given the constraints regarding time and capacity, it was not possible for the PBL to check a hundred per cent of all texts and references in the eight regional chapters of the Working Group II Report for errors, considering that it had taken hundreds of authors and reviewers to produce the report over the course of five years. Instead, we limited ourselves to the IPCC summary statements, and framed the central questions of this report as follows:

Are the summary conclusions on regional impacts well founded on the underlying chapters and literature references? Are there errors in statements that have travelled from the scientific literature references and/or the main texts through to the summary conclusions? If errors are found, do they affect the validity of these conclusions? What recommendations can we derive from our investigation in order to further improve the quality of the assessment process for the Fifth Assessment Report (due in 2014)?

The role of IPCC assessments and the role of the PBL Policymakers need to be well informed about the state of the art in climate science. Obtaining a reliable overview of the scientific literature, with its many different and sometimes contradicting messages, is something policymakers can obviously not achieve on their own. What is more, a knowledge base that is shared and accepted as credible and legitimate by all parties involved, can be of great value to the policy process. Therefore, in 1989, the UN member governments that constitute the IPCC arranged a process in which worldwide groups of carefully selected expert authors, periodically, would assess the ever growing amounts of scientific literature on climate. Working within the confines of the IPCC, authors arrive at collective judgments that are supposed to present the state of the art in climate science, in a comprehensive, unbiased, non-policy-prescriptive way. The strict procedures established by the IPCC member governments are to guarantee precisely this. The credibility of an IPCC report depends both on the knowledge and integrity of the author teams and on the thoroughness and expertise of expert and government
reviewers.

The PBL generally has an intermediate role in interpreting IPCC reports, so that these can be applied in policy-making by the Dutch Government and Parliament. In this case, the PBL has had to assess the reliability of some key messages derived from a particular IPCC report; the Working Group II Report. To this end, we needed

1) T here is a forthcoming publication by the Royal Netherlands Academy of Sciences (KNAW) that will examine this fundamental part of climate science in more detail.Executive summary 13 to rely on the expert judgments of IPCC authors, to some extent, as it would have been impossible to track down and assess all theory, measurements, observations, model calculations, and all arguments that the IPCC authors may have used during the five-year assessment period in order to arrive at certain conclusions. However, we could investigate whether the key messages in the summary were being backed up by material discussed in the main chapters of the IPCC assessments, whether relevant information in the main chapters, in our opinion, had been founded on sound scientific sources, and whether there would be any obvious errors or mistakes in the (presentation of) the material investigated.

IPCC assessments are always snapshots in time – representative of the knowledge of science as it is at that moment. As science proceeds, insights may change. However, in our investigation, we have not included new scientific information that has become available since the Fourth Assessment Report was produced – our task was not to redo the IPCC assessment, but to investigate its underpinning at the time of finalisation, in early 2007.

Focus of the investigation The 2007 Fourth Assessment Report of the IPCC contains almost 3,000 pages. The focus of our investigation was on only a small part of this report. The IPCC report contains four volumes: the Working Group I Report on the physical science basis of climate change; the Working Group II Report on climate impacts, adaptation and vulnerability; the Working Group III Report on mitigation of climate change; and a Synthesis Report, integrating the main outcomes of these three W orking Group reports.

The Working Group II Report contains 20 chapters, eight of which deal with the regions Africa, Asia, Australia and New Zealand, Europe, Latin America, North America, polar regions (Arctic and Antarctic), and small islands (that is, small-island states).

In the Synthesis Report a table is presented, in its Summary for Policymakers, describing ‘examples of some projected regional impacts’, derived from the eight regional chapters in the Working Group II Report, each containing four statements, which together form 32 statements (see Annex A for a copy of this table). W e have considered this table in the Synthesis Report as the highest summary level presenting the most important regional impacts, with the highest visibility to the policymakers. **2)**

The primary focus of this investigation, therefore, was on the foundation of these 32 statements, which are a subset of largely identical statements in the Summary for Policymakers of the Working Group II Report.

2) T he Synthesis Report was released a few weeks before the important world climate Summit of the UNFCCC (the UN climate convention) in Bali in December 2007 (also indicated as CoP13). At this conference, the UN parties to the convention took the first steps towards a long-term climate policy strategy following the first commitment period of the Kyoto protocol.Assessing an IPCC assessment 14 Analysis of the main texts and the summaries Each IPCC report consists of several layers. Each chapter in a W orking Group report contains a main text and its literature references, with an ‘Executive Summary’ up front. Each Working Group report has both an extensive ‘Technical Summary’ which synthesises and summarises the information of all chapters, and a brief ‘Summary for Policymakers’.

We analysed the Executive Summaries of each of the eight regional chapters, and checked their provenance from the relevant parts of the underlying texts in the main chapters and their key references. We did the same with regional information in the Technical Summary and the Summary for Policymakers of the Working Group II Report. In addition, we checked how these conclusions were entered into the main text of the Synthesis Report, and how they ultimately landed in its Summary for Policymakers (see Figure 2.1).

Registration website The PBL launched a public website that would be available for the course of one month, in order to give all experts in the Netherlands the opportunity to contribute to our investigation. We asked for submissions of possible errors found in regional chapters of the Working Group II Report. By the end of that month, the PBL had registered 40 submissions; however, most of them were about issues related to the Working Group I Report. Two submissions qualified to be addressed in our report. All submissions and PBL’s responses have been published on the PBL website.

Process We invited the IPCC authors of the regional chapters of the Working Group II Report to comment on our draft findings, and in our communication we went through several iterations. In some cases, we also contacted the authors of cited references. The draft of our report was also reviewed by selected internal and external, national and international experts. Moreover, this project has been conducted under the independent supervision of the Royal Netherlands Academy of Arts and Sciences (KNAW).

Methodology This investigation has resulted in a report on:

1. findings referring to the foundation for the 32 summary conclusions on regional impacts in the Synthesis Report (main objective);

2. additional findings at lower levels, such as those referring to the foundations for statements on regional impacts in the T echnical Summary and Executive Summaries of the regional chapters, and also some findings on parts of the main text that we came across while working on other findings. As our focus was on tracing possible errors that would affect the key messages to policymakers, we were unable to systematically check all statements in the main texts.

For our investigation, we used several criteria for assessing the quality of IPCC statements. First of all, we employed a distinction between obvious factual ‘errors’ – which we believe require an erratum on the IPCC Fourth Assessment web pages – and ‘comments’, the latter being critical remarks, made from our specific position of having to assess these statements on behalf of Dutch policymakers. TheseExecutive summary 15 criteria were developed ‘on the job’, while categorising our findings. In total, there are nine criteria, and associated with these are two types of ‘errors’ (E1-E2) and seven types of ‘comments’ (C1-C7):

 E1: Inaccuracy of a statement;

 E2: Inaccuracy of literature referencing;

 C1: Insufficiently substantiated attribution, when a certain impact had been attributed to climate change without convincing substantiation;

 C2: Insufficiently founded generalisation, when case studies had been extrapolated to whole regions/sectors without convincing foundation;

 C3: Insufficiently transparent expert judgment, when we could not trace the reasoning behind a statement from underlying texts or literature references given;

 C4: Inconsistency of messages, when found between the different layers in the IPCC report;

 C5: Untraceable reference, when a literature reference could not be found;

 C6: Unnecessary reliance on grey referencing, when reference had been made only to grey literature (literature other than from peer-reviewed journals), while adequate peer-reviewed scientific journal references would have been available;

 C7: Statement unavailable for review, when new information had been introduced after the last review, but was not clearly derived from a content issue
raised in this last review.

When a certain statement in the IPCC Fourth Assessment Report would meet all the criteria, the statement was considered to be well founded and reliable.

M elting of Himalayan glaciers In response to the request from the Dutch Minister for the Environment to provide an update on the melting of Himalayan glaciers, we performed an assessment of the new scientific information that had become available since the publication of the IPCC’s Fourth Assessment Report (see Annex B). The average, region-wide, glacier area retreat rate is probably between 0.1 and 0.5% per year. Although the glacier area will shrink substantially this century, especially in the most vulnerable eastern zone of the Himalayas, glaciers (such as Khumbu and Imja), will not disappear entirely, or even mostly, by 2035, as stated in the Working Group II Report.

The annual contribution from Himalayan glaciers to sea level rise is 0.06±0.04 mm or ~2% of the current annual sea level rise of 3.1±0.7 mm. The rate of Himalayan glacier melting appears to have increased over time, but so has sea level rise, and therefore the 2% contribution may still apply.

Dutch land area below sea level The error about the percentage of Dutch land area below sea level stems from a text submitted by the PBL (see Annex C). The Working Group II Report states that 55% of Dutch land area is below sea level. This should have read that 55% of the Netherlands is prone to flooding: 26% of the country is at risk because it lies below sea level and another 29% is susceptible to river flooding.Assessing an IPCC assessment 16 Sea level rise and its consequences for the Netherlands Errors in the IPCC report regarding the Himalayan glaciers and the Dutch land area below sea level have triggered a discussion on whether Dutch policies that deal with sea level rise should be revised (see Annex D). The answer is clearly ‘no’: the contribution from melting Himalayan glaciers to sea level rise is very limited (around 2%) and the policy concerning safety against flooding in the Netherlands, as formulated in the National Water Policy Plan 2009-2015 (Ministry of Transport, Public Works and Water Management, 2009) and elaborated in the Delta Programme, is well founded on the available knowledge on climate change, sea level rise, river discharges and uncertainties. The current discussions on the IPCC have no effect on the assumptions under the National Water Policy Plan and the Delta Programme.Part I – Main findings 17 

# Part I – Main findingsAssessing an IPCC assessment 18Introduction 19 

# Introduction

1.1 Scientific assessment in aid of policy

The characteristics of climate change qualify it as the most complex type of policy problem. It resists direct perception, hence, making policymakers and the public depend on experts to describe its potentially problematic features. It is a phenomenon, the problematic features of which may manifest themselves within a matter of decades, while preventative measures assume more direct action. It is also a global phenomenon which can only be handled effectively through joint participation of all countries (climate change is a classic ‘free rider’ problem). Addressing climate change is not a matter of controlling one clearly identified source, as was the case with the depletion of the ozone layer, but rather of controlling a multiplicity of sources. This characteristic of ‘wickedness’ (cf. Rittel and Webber, 1973) makes that climate science plays a very central role in the policy debate. To make things worse, a full understanding of the subject depends on the input from a large variety of scientific fields, with differing degrees of standing knowledge relevant to measuring and understanding processes of climate change, its effects and possible remedial strategies. Since both the stakes and the uncertainties are high, the problem of climate change needs to be addressed by using ‘post-normal science’; a problem-solving strategy that involves paying specific attention to qualitative judgments and value commitments permeating science (Funtowicz and Ravetz, 1993).

T he role of assessments Historically, the IPCC was established to present international policymakers with undisputed, policy-relevant knowledge, while acknowledging uncertainties (see e.g. Petersen, 2006). Since its inception in 1989, the IPCC has been effective at this. However, the science on climate-change impacts is still beset with large uncertainties, some of which are irreducible. Next to the nature of climate science, it is relevant to remember that perceptions of climate-change risk vary widely, both across the globe and within societies (cf. e.g. Hulme, 2009).

Hence, the issue of climate change is complex, both in causes and effects and in terms of policy implications. The uncertainties, different interests and different risk perceptions pertaining to all aspects of causes, consequences and response options are inextricably linked to the issue of climate change.

In such situations, scientific assessments are the institutional vehicle for presenting policymakers with information that would help them define and decide on policy strategies. Scientific assessments present the state of knowledge relevant to a 

# 1Assessing an IPCC assessment 20 particular policy problem. It is a subtle tool. For complex problems such as climate change, recent history has shown that a traditional, linear science-for-policy model – one that is based on a clearly formulated policy question followed by a sequence of scientific research from hypotheses to experiments and, finally, agreed on answers – will not work. The nature of the issue is too complex and uncertainties will always remain. However, the fact that uncertainties about climate change are large does not mean that scientific information should not or could not play a role in decision-making. On the contrary, well-formulated scientific information is considered a crucial component in addressing complex policy problems.

The tool of scientific assessment, such as that implemented through the Intergovernmental Panel on Climate Change (IPCC), has been developed as a way of organising the science–policymaker dialogue (see, e.g., Social Learning Group, 2001). Scientific assessment is the analysis and review of information derived from existing research, in the context of specific decision-making processes, in order to better understand the wide implications of a policy problem and to identify and evaluate possible response actions. Such assessment usually does not entail new research. The design of the process of scientific assessment needs to ensure that uncertainties are properly addressed and that different societal perspectives are included in the process, so that politicians may obtain a better sense of the possible futures facing the world, and of the choices that could be made. One of the functions of independent assessments is to assess and communicate uncertainty, and to make sure that the problem is analysed from different value perspectives. The proper course of action then is for policymakers to decide what – if anything –
should be done.

The IPCC assessment process This report investigates the quality of IPCC’s key messages on regional impacts of climate change, which were the outcome of a long production process. W e have not reiterated the elaborate IPCC production process with its procedures for review. This has been described extensively in the Principles Governing IPCC Work (http://www.ipcc.ch/pdf/ipcc-principles/ipcc-principles.pdf; for a summary, see http://www.ipcc.ch/pdf/ipcc-principles/IPCC%20Procedures.pdf). These procedures are meant to guarantee a responsible assessment process.

It is important to note, however, that an IPCC assessment is not the product of an extensive bureaucratic operation. Instead, its organisational structure is aptly characterised as being a network organisation that spans the whole globe. Scientific experts from varying continents collaborate and assemble overviews of the knowledge in their respective fields. The quality and integrity of the IPCC process depends, to a large extent, on the quality and integrity of its procedures, and, hence, on the commitment of all contributing authors to adhere to those rules and procedures. In response to experiences and criticism, these procedures have been regularly upgraded over the course of more than 20 years to guarantee increasing assurance of high levels of quality and integrity.

We have chosen not to elaborate on the institutional set up of the IPCC, but have considered it a given. For the present report, we had set ourselves the task of assessing the transparency with which scientific facts on climate-change impactsIntroduction 21 have ‘travelled’ from their source (e.g., scientific peer-reviewed literature) to their destinations: the Summaries for Policymakers. This final destination of scientific facts within the IPCC process has driven the investigation presented in the present report. The IPCC process should deliver useful and reliable knowledge (cf. Morgan, 2010) to policymakers; one route to check this – followed for this report – is by tracing summary conclusion back to their origins.

The role of expert judgment This report deals with the questions that have been raised on the reliability of the assessment process run by the IPCC. In particular, questions have been raised concerning the Working Group II contribution to the Fourth Assessment Report
with its focus on Impacts, Adaptation and Vulnerability and including the regional chapters that were home to the two errors that caused the upset in the media and in society, in the winter of 2009–2010. This report has been based on the assumption that the quality of the IPCC scientific assessment can be measured by tracing statements on the effects of climate change, for example, as presented at the highest level of summaries for policymakers (in the Synthesis Report and the Working Group II Report), back to their original sources. There is obviously a limit to the traceability of such statements. Given the mismatch between the state of standing knowledge on complex environmental phenomena, on the one hand, and the questions raised by policymakers, on the other, those conducting
an assessment often have to arrive at so-called expert judgments : a weighing of understandings and analyses to come up with the best possible assessment of observations, or of what one might expect to happen. Such expert judgments are a crucial component of any assessment. Given the fact that these judgments cannot be traced back to any one source in the (peer-reviewed) literature, the quality of these assessments grows as the expert judgments are made more transparent, that is, by explaining how particular conclusions have been arrived at. This procedure allows others to follow a certain line of reasoning, similar to how this generally is done in articles published in scientific journals. Another way of enhancing the quality of expert judgments is to present them in the form of a deliberation, stating the way in which key conclusions have been arrived at. This then allows other scientists to contradict inferences, or come up with alternative explanations.

Expert judgments are inextricably linked to assessments and are used in all phases of the IPCC assessment process. The lead authors of the regional chapters of the Working Group II Report were supposed to have read all the literature relevant to that particular region, and to be familiar with the environmental policy issues of that region. Based on their individual expertise, the lead authors collectively arrived at judgments on the impacts of climate change. The review process of the IPCC relies on two rounds of expert review and one round of government review, as well as on Review Editors, who umpire the review process. The objective of this process is to ensure a balance in the lead authors’ judgments as these are published in the report. The chapters themselves are very detailed. This explains the need for summary statements that convey the most important conclusions to policymakers. The Coordinating Lead Authors drafted the Summary for Policymakers of the Working Group II Report, and a smaller group of authors (consisting of two regional Coordinating Lead Authors, the two Co-Chairs and the head of the T echnical Support Unit of Working Group II) was involved in the drafting of the Summary forAssessing an IPCC assessment 22 Policymakers of the Synthesis Report. Moreover, in both of these Summaries for Policymakers, the role of expert judgments had been crucial for determining the main policy-relevant findings from the underlying chapters.

Since it is impossible for most readers of IPCC reports to understand and check every reference, all data, models, calculations and measurements, they must be able to rely on the quality control and quality assurances given by the assessment process. To some extent, those readers have no choice other than to assume an expert judgment to reflect the best of current knowledge. However, they should be able to trace the plausibility of a judgment in a Summary for Policymakers back to the main text and references. When dealing with issues as complex as climate change, expert judgment is inevitable. But precisely because this is a matter of sound judgment, there is a virtue in maximising the possibility for others to verify and validate the analyses made by the IPCC authors.

1.2 Some background on IPCC W orking Group II

Since the Third Assessment Report (finalised in 2001), the work of the IPCC has been organised into the following three distinct Working Groups:

 W orking Group I addresses the physical science of climate change;

 W orking Group II addresses the impacts of and vulnerability to climate change and the means for adaptation;

 W orking Group III addresses the mitigation of greenhouse gas emissions.

Working Group II addresses impacts, vulnerability and adaptation to climate change with respect to human and natural systems. This particular focus calls for the mobilisation of authors mostly from physical, geographical and ecological sciences, but also from the social sciences. The particular thematic focus of W orking Group II immediately complicates its assessment task. The interactions within biological systems and between biological systems and physical chemical and social systems, are very complex. What is more, the patterns of interaction vary, depending on
local situations and on human choices and actions.

Literature on climate impacts is not only very diverse, and the intensity of academic scholarship is divided very unevenly over the globe. This increases the challenge of making good assessments of what the impacts, effects and vulnerabilities of climate change are in those regions that have the least data to draw on. This point is well illustrated by a map from the IPCC report itself (see Figure 1.1).

Furthermore, impact analyses are not only spread unevenly around the world, it is also a field of study where often insights are found outside of the strict academic tradition. For instance, many impact analyses are presented in reports by governments and inter-governmental or private, non-academic institutions – given the fact that not all topics in these reports are equally suitable for, or indeed, targeted to scientific publication – which complicates the task of assessors. The IPCC guidelines specifically allow for this literature also to be assessed (in order to provide relevant input about local and sectoral situations and to minimise bias in information between developing and industrialised countries, for instance, becauseIntroduction 23 impacts in developing countries are reported even less in scientific peer-reviewed journals).

The scientific uncertainties that the IPCC, including Working Group II, are confronted with, are large and deep. Since its inception, the IPCC has given increasing attention to the management and reporting of uncertainties (Swart et al., 2009). The socio-ecological systems studied by Working Group II are not always suitable for numerical modelling; hence, there is a lack of information on quantitative uncertainty. Impact analyses or the study of vulnerabilities often depend on empirical observation using different scientific techniques, such as field research or case study analysis. Because of the uncertainties involved, it is difficult to establish which impacts occur, and why and to what extent observed effects could be attributed to climate change. Often authors need to make assessments of the risks that are associated with various levels of climate change, rather than indicating best-estimates and clearly defined uncertainty levels. In order to estimate Locations of significant observations of changes in physical an biological systems together with changes in surface air temperature, over the 1970–2004 period. Source: Climate Change 2007: Impacts, Adaptation and V ulnerability. Working Group II Contribution to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change, Figure 1.8. Cambridge University Press. **Figure 1.1**

Observed data series Physical systems (snow, ice and frozen ground; hydrology; coastal processes) Biological systems (terrestrial, marine, and freshwater)

* Circles in Europe represent 1 to 7,500 data series. 1 – 30

31 – 100
101 – 800 801 – 1,200

1,201 – 7,500 Temperature change o C 1970 – 2004

-1.0 3.52.01.00.2-0.2 **Observed changes in biological and physical systems**

Europe * **Figure 1.1**Assessing an IPCC assessment 24 the possible risks of climate change, also events with very low probability but with large consequences have to be dealt with.

In the context of this report, it is important to understand the mandate of Working Group II. Formally, Working Group II ‘assesses the scientific, technical, environmental, economic and social aspects of vulnerability (sensitivity and adaptability) to climate change and the negative and positive consequences for ecological systems, socio-economic sectors and human health, with an emphasis on regional, sectoral and cross-sectoral issues.’ (see http://www.ipcc.ch/working_ groups/working_groups.htm)

1.3 Scope, objectives and limitations of this report

In January 2010, worldwide media attention was drawn to two errors that were discovered in a part of the Fourth Assessment Report of the IPCC of 2007: an erroneously high rate of melting of the Himalayan glaciers, and an erroneously high percentage of land area in the Netherlands lying below sea level. The latter error was not the fault of the IPCC but of the PBL (see Annex C). The commotion in the Dutch media and subsequent discussions at the political level in the Netherlands led to a resolution by the Dutch Parliament on 28 January 2010, which declared in the preamble that the reliability of the IPCC, which should be undisputed, was now at
issue.

In the resolution, the parliament required that the government would charge the Netherlands Environmental Assessment Agency (PBL) with providing a new update of climate science, including the implications of the said errors. Based on this resolution and the debate in parliament, the Minister for the Environment decided for the PBL to limit its investigation to the implications of possible errors in the regional chapters of the IPCC report of 2007 on climate-change impacts, adaptation and vulnerability.

The assessment presented in this report was based on an investigation of the original sources of the IPCC’s Fourth Assessment summary conclusions (in particular, the Synthesis Report, Table SPM.2, see Annex A) that pertain to a part of one of the Fourth Assessment Reports (that is, Chapters 9 to 16 of the W orking Group II Report), and was aimed to derive general observations and identify possible improvements.

Two additional tasks were taken on. First of all, the Minister for the Environment asked the Netherlands Environmental Assessment Agency (PBL) to make an assessment of the scientific literature on Himalayan ice, published since 2007. This was triggered by the need to communicate what scientists do know about this topic; after all, the public turmoil concerned the IPCC error that stated that all Himalayan ice would be melted by 2035. The results from this assessment are presented in Annex B.

Second, the parliamentary debate also showed concerns over the possible repercussions of the state of knowledge on the issue of sea level rise for climate-Introduction 25 change adaptation measures in the Netherlands. Therefore, the state-of-the-art
scientific foundation on this issue is summarised in Annex D.

The PBL has taken the utmost care to prevent possible prejudice in their present assessment of the conclusions derived from the regional chapters of the W orking Group II Report. A crucial mechanism to avoid any prejudice was the installation of a supervisory committee under the responsibility of the Royal Netherlands Academy of Arts and Sciences (KNAW). The KNAW was asked to supervise the quality of this PBL study. The KNAW appointed a scientific committee, whose role it was to judge whether the PBL performed its research without prejudice and according to established procedures of quality control. **1)**

Realistically speaking, a thousand-page assessment by hundreds of authors involving thousands of reviewers conducted within a limited timeframe could hardly be expected to be free of errors. Therefore, it is to be expected that some inaccuracies, insufficiently justified statements or other irregularities, escape even the most thorough drafting and review procedures. Our evaluation methodology was to take the statements in the various IPCC summaries as starting points and consider their source of origin (where did they come from?). A full evaluation of the reliability of statements would entail redoing the original IPCC assessments. Our limited evaluation could never have repeated or improved on the IPCC drafting and review process in a comprehensive way, nor was it our aim to redo the IPCC assessment. Nevertheless, we think that such a limited evaluation by scientifically educated policy analysts, who are not experts in climate-impacts science, is a useful means for the IPCC to receive feedback from an extended peer community (cf. Funtowicz and Ravetz, 1993). Through this evaluation, we reflect on the qualitative judgments and value commitments that necessarily permeate the IPCC assessment

process.

Without recruiting experts with the range of expertise and following the careful, multi-step review that has characterised the IPCC reports, any review of the contents of IPCC chapters only has limited potential. The current PBL assessment, therefore, extends only to checking whether observations and interpretations had been captured conscientiously, how statements had been substantiated within IPCC reports, and to evaluate the transparency with which important expert judgments had been made. The analysis focused on investigating the ‘internal’ consistency of the report, and did not perform new, fresh reviews of the evidence available at the time the Fourth Assessment Report was produced. In sum, the analysis focused on assessing statements by looking for errors (inaccuracies in statements and referencing) and by applying a short list of quality criteria (see Chapter 2), which was developed specially for the purpose of this analysis.

In sum, the central questions addressed in this report are the following:

Are the summary conclusions on regional impacts well founded on the underlying chapters and literature references? Are there errors in statements that have travelled

1) T he terms of reference of the supervision by the Royal Netherlands Academy of Arts and Sciences can be found at http://www.knaw.nl/pdf/Instellingsbesluit_Toezicht_Cie_PBL.pdf (in Dutch).Assessing an IPCC assessment 26 from the scientific literature references and/or the main texts through to the summary conclusions? If errors are found, do they affect the validity of these conclusions? What recommendations can we derive from our investigation in order to further improve the quality of the assessment process for the Fifth Assessment Report (due in 2014)?

1.4 Structure of the report

Part I of the report contains the main findings. Chapter 2 describes the methodology of our assessment. Chapter 3 presents the results from our analysis of IPCC Fourth Assessment’s statements on regional climate-change impacts. Chapter 4 contains our recommendations for the IPCC assessment process, following from our analysis in Chapter 3. Part II provides our detailed analysis of the regional chapters and the summaries.

Re ferences
Funtowicz, S.O. and Ravetz, J.R., 1993. ‘Science for the post-normal age’, Futures 25: 739–755. Hulme, M., 2009. Why We Disagree about Climate Change: Understanding Controversy, Inaction and

Opportunity. Cambridge: Cambridge University Press. Morgan, M.S., 2010. ‘Traveling facts’, in P. Howlett and M.S. Morgan, eds., How Well Do Facts Travel?,

Cambridge: Cambridge University Press. (http://www2.lse.ac.uk/economicHistory/pdf/Morgan/ Travelling%20facts%20_2_.pdf) Petersen, A.C., 2006. Simulating Nature: A Philosophical Study of Computer-Simulation Uncertainties

and Their Role in Climate Science and Policy Advice. Het Spinhuis Publishers, Apeldoorn and Antwerp. Download from http://hdl.handle.net/1871/11385. Rittel, H.W.J. and M. Webber, 1973. ‘Dilemmas in a general theory of planning’, Policy Sciences, 4 (2):

155-169. Social Learning Group, 2001. Learning to Manage Global Environmental Risks, V olume 1: A

Comparative History of Social Responses to Climate Change, Ozone Depletion, and Acid Rain, Cambridge, MA: MIT Press. Swart, R., L. Bernstein, M. Ha-Duong, A. Petersen, 2009. ‘Agreeing to disagree: uncertainty

management in assessing climate change, impacts and responses by the IPCC’, Climatic Change

92: 1–29.Methodology 27 

# Methodology

2.1 Investigative approach

The two primary aims of the investigation were to assess, first of all, whether errors would affect the content of the IPCC’s key statements on regional climate-change impacts, and, second, whether these key statements would meet seven quality criteria formulated by the PBL (see Section 2.2). These key statements were taken from the Summary for Policymakers of the IPCC Working Group II Report (2007) and, in part, presented in Table SPM.2 of the IPCC Synthesis Report (2007) (see Annex A). Figure 2.1 offers a graphical explanation of the area of investigation, Figure 2.1A places the statements on regional impacts of climate change in the context of the whole IPCC Fourth Assessment Report, and Figure 2.1B represents the approach to the chapter analysis. Table SPM.2, containing 32 bulleted statements on the most important projected impacts (four per region), is located at the top of a ‘pyramid’.

A secondary aim of our investigation was to identify errors and assess the quality of statements on regional climate-change impacts at levels below the Summary for Policymakers of the Working Group II Report.

In Figure 2.1B, the ‘pyramid’ shows the analysis levels that we used in our investigation. The Working Group II 2007 Report itself contains four layers. Level 1 contains the main chapter texts, level 2 contains the Executive Summaries of the individual chapter texts, level 3 contains the Technical Summary (giving an extensive overview of the whole report), and level 4 is the Summary for Policymakers (a short document, approved line-by-line by the governments involved in the IPCC). Each chapter text refers to hundreds of literature sources on which the assessment was founded (‘level 0’). In some cases, we also investigated a few sources that were mentioned in the literature references (‘level −1’)

The Synthesis 2007 Report has only two layers, a main report, divided into six topics, and a Summary for Policymakers. The information in the Synthesis Report has a higher level of aggregation than that in the Working Group Reports (level 5). The information in the Summary for Policymakers of the Synthesis Report has the highest level of aggregation (level 6).

The heuristic approach that was followed by PBL policy analysts involved in this study, was to start by reading the Executive Summaries (level 2) of the regional chapters with a critical eye, and from there work upwards to level 6 (Summary for Policymakers of the Synthesis Report) as well as downwards, to level −1. W e made 

# 2Assessing an IPCC assessment 28 Figure 2.1A shows the structure of the IPCC’s Fourth Assessment Report, 2007. The Working Group II Report contains eight chapters (from a total of 20 chapters) describing the impacts in eight world regions. The main conclusions of the reports by the three W orking Groups were summarised per report, and also included in the Fourth Assessment Synthesis Report. The last contains examples of some projected regional impacts; see T able SPM.2 (Annex A). This table was the starting point in our investigation. Figure 2.1 B shows the approach to the regional chapter analysis. The statements from Table SPM.2, at the top of the ‘pyramid’, were analysed by investigating their foundation in the five consecutive levels, and in the relevant underlying literature. **Figure 2.1 Stucture of the Fourth Assessment Report and regional chapter analysis** 

Literature references

Underlying literature Regional chapter texts Working Group II Regional chapter including Executive summaries • Ch. 9 Africa • Ch. 10 Asia • Ch. 11 Australia – New Zealand • Ch. 12 Europe Technical Summary Working Group II 2007 report • Section 4.2, Regional Impacts, Adaptation and Vulnerability p. 48 – 58 • Box TS.6, The main projected impacts for regions, p. 59 – 63 • Tabel TS.4, Example of regional impacts, p.67 Summary for policy makers Working Group II 2007 report (SPM WG II) p. 13 – 15. Synthesis report 2007 Topic 3 (Climate Change and its impacts), section 3.2, para 3.3.2, Impacts on Regions, p.50 and 52 Summary for policy makers (SPM) Synthesis report (SYR) Table SPM.2, Examples of some projected regional impacts – see Annex A

Executive summaries chapter texts Working Group II Technical summary Working Group II Summary for policy makers Working Group II Synthesis report main text Level of aggregation

Level of detail Summary for policy makers (SPM) Synthesis report (SYR)

• Ch. 13 Latin America • Ch. 14 North America • Ch. 15 Polar regions (Arctic and Antarctic)
• Ch. 16 Small Islands B. Regional chapter analysis pyramid

-1 0 1 2 3 4 5 6 A. Structure of the Fourth Assessment Reports

Table SPM.2 Example of some projected regional impacts (see Annex A) Working Group I report The Physical
Science Basis Working Group II report Impacts, Adaptation and Vulnerability

The AR4 Synthesis Report Working Group III report Mitigation of Climate Change **Figure 2.1**Methodology 29 sure that we covered all the bulleted statements in Table SPM.2, looking for any possible weaknesses, thus, effectively working from the top layer of the Synthesis Report down to the more detailed regional chapter texts of the W orking Group II Report, and in many cases to the underlying key literature references. In our search for the underpinning of statements, we also looked for evidence in the introductory and sectoral chapters of the Working Group II Report and, when this was deemed necessary, in the Working Group I Report.

Re gistration website The PBL launched a public website that would be available for the course of one month, in order to give all experts in the Netherlands the opportunity to contribute to this investigation. We asked for submissions of possible errors found in regional chapters of the Working Group II Report. By the end of that month, the PBL had registered 40 submissions; however, most of them were about issues related to the Working Group I Report. Five submissions were related to the regional chapters of the Working Group II Report; two of those five were included in our report, three submissions did not qualify for inclusion. All submissions and PBL’s responses have been published on the PBL website.

P rocess We invited the IPCC authors of the regional chapters of the Working Group II Report to comment on our draft findings, and in our communication we went through several iterations. In some cases, we also contacted the authors of cited references. The draft of our report was also reviewed by selected internal and external, national and international experts. Moreover, this project has been conducted under the independent supervision of the Royal Netherlands Academy of Arts and Sciences (KNAW).

2.2 T ypology of errors and qualifications

The typology of errors and qualifications (‘comments’) presented here was arrived at through an inductive process – the results from which are outlined in this section. A distinction was made between ‘major’ and ‘minor’ errors or comments. Issues were labelled to be ‘major’ when we judged them to have a significant quantitative impact on statements pertaining to food, water, ecosystems, coastal areas, and
health.

E rrors We used a sharp definition: ‘errors’ were considered inaccuracies in the IPCC Fourth Assessment Report that we believed to require repair through an erratum or a redoing of the assessment of the specific issue at hand. Experts – in particular, PBL policy analysts and IPCC authors – reached a high level of agreement on what they considered to be an ‘error’ in this precise sense. Both the Himalayan glacier melting by 2035 and the 55% of the Netherlands that was said to be below sea level, were
considered to be such ‘errors’.Assessing an IPCC assessment 30 We distinguished two types of errors:

E1. Inaccurate statement Statements should be an accurate and correct reflection of the material found in literature, augmented by the expert judgment of a writing team. If a state

ment appeared to be factually incorrect, then we considered whether this was a minor imprecision or inaccuracy or if a major mistake had been made that could have led to misinterpretations by policymakers. We applied the following subcategories to this type of error:

E1a) Inaccuracies that can be corrected by issuing an erratum – This would be mistakes that could be simply corrected without new scientific assessment considerations, such as: typographical errors, incorrect phrasing of part of a sentence, wrong dimensions, and wrong reference years. The error on the percentage of Dutch land area below sea level belongs to this subcategory (see Annex C).

E1b) Errors that would in fact require a redoing the assessment of the specific issue at hand , such as: establishing a new range of numbers by revised calculations from the reference sources available during the assessment period, and/ or rephrasing of the expert judgment including its uncertainty labelling. **1)** The Himalayan glacier error belongs to this subcategory (see Annex B).

E2. Inaccurate referencing
References used in statements should be correct. Whenever we found a refe

rence to a wrong source, or when the source was not correctly cited, we quali

fied this to be ‘inaccurate referencing’ . In all cases, an erratum was proposed.

Qualifications In addition to the errors in the Fourth Assessment Report that could be corrected. either in an erratum or through reassessment, we distinguished between seven types of qualifications or comments that would signify the quality of a statement.

C1. Insufficiently substantiated attribution In some cases, observed or projected impacts on systems are described as being ‘due to climate change’, ‘temperature increase’ or other climate parame

ters. Since, often, there are other drivers that could be involved in causing such an impact (e.g. population growth, industrialisation, migration, and changes in land use and land cover), care must be taken to provide the proper context in IPCC statements about the effects of climate change, by referring to other causes and/or stresses that may influence the impacts discussed. Statements that, in our judgment, did not sufficiently signal the presence of multiple stresses, or suggested a one-sided attribution of impacts to climate change while other factors also would have been expected to play a critical role, were qualified as ‘insufficiently substantiated attribution’

.

1) T his, of course, excludes new findings in literature published after completion of the assessment
reports.Methodology 31 C2. Insufficiently founded generalisation In some cases, the impacts, for instance, on certain types of livestock or countries, were generalised or extrapolated to include other livestock or entire regions. In such cases, we believe authors ought to make clear why they think that the evidence in the references justifies such generalisation. Statements, for which such argumentation was lacking, and for which such evidence could not be found in the report or in the references, were qualified as ‘insufficiently founded generalisation’

.
C3. Insufficiently transparent expert judgment Expert judgment on results that can be found in the scientific literature, is an essential part of the IPCC assessment process; this includes giving an indication of uncertainties. An assessment is thus not a simple summary of the literature. The expertise of an international group of authors adds value by judging the importance and relative weight of the information from the literature. For the sake of transparency, we believe that the reasoning behind these judg

ments, including the reasoning behind their levels of likelihood and/or confi

dence **2)** , should be accessible to a non-expert reviewer who attempts to trace the travelling of facts from the referenced literature to the Summaries for Policymakers.

If, in our opinion, this was not the case, and this fact was found to be proble

matic, we qualified the particular judgment as an ‘insufficiently transparent expert judgment’ . This does not imply that this IPCC expert judgment was wrong, since the authors as a group may have had their reasons, and may have conside

red additional information or knowledge that was not explicitly referred to. However, in such cases a convincing substantiation of the statement was lacking in the report.

C4. Inconsistency of messages When moving up from statements in the main chapter texts to statements in the Summary for Policymakers of a Working Group Report or of the Synthesis Report, the sentences were inevitably shortened and the wording may have needed some adjustments. However, Summary for Policymaker texts need to be consistent with the main text, as required by the IPCC procedures. This holds both for the meaning of the message as well as for the level of certainty. SPM approval session decisions include a list of issues that need to be checkedfor consistency with the Technical Summary and the main chapters in preparation of the final publication. Whenever we found that message content and/ or confidence level had changed when it moved from main text to Summary

2) The IPCC authors were in fact requested to produce a ‘traceable account’ of their assessment of uncertainty in expert judgments in the Fourth Assessment process (see Guidance Notes for Lead Authors of the IPCC Fourth Assessment Report on Addressing Uncertainties, July 2005, http://www. ipcc-wg1.unibe.ch/publications/supportingmaterial/uncertainty-guidance-note.pdf). For the ThirdAssessment, a similar guidance was given (Uncertainties in the IPCC TAR, 2000, http://stephenschneider.stanford.edu/Publications/PDF_Papers/UncertaintiesGuidanceFinal2.pdf). It is our general impression that this part of the guidance has never been fully implemented in the assessment process. Also, the option to have separate traceable accounts underlying the reports has not been pursued.Assessing an IPCC assessment 32 for Policymakers and also if there was any inconsistency between chapters, we qualified this as an ‘ inconsistency of messages’.

C5. Untraceable reference As stated before, references used in statements should be accurate. If we could not find the reference at all, we qualified this as an ‘untraceable reference’

.
C6. Unnecessary reliance on grey referencing Grey literature was an indispensable part of the Working Group II assessment (this also applies to Working Group III), since not all relevant literature had been published in peer-reviewed scientific journals. The grey literature needed to meet quality standards before it could be used. There are huge differences in the types of literature, going from thoroughly reviewed scientific books to much less reviewed – or even non-reviewed – publications. When a substan

tial scientific statement only referred to a grey publication, although strong peer-reviewed journal references could have been referred to at the time, we qualified this to be ‘unnecessary reliance on grey referencing’.

C7. Statement unavailable for review In the preparation of a Final Draft Report, the results from government and expert reviews were included. This could mean that new literature sources would be added in order to properly address the Second Order Draft (SOD)
review comments. Since reviews of a SOD often asked for the addition of sources, such additions had to be considered as acceptable revisions. However, if a new statement contained new literature findings that were not clearly derived from a content issue raised in the Second Order Draft review, then this new material would have been kept out of the review process. This is not supposed to happen; expert reviewers or governments must have access to all findings that will be part of the published report. In the cases in which these new literature findings were added as described above, we qualified them as a ‘statement unavailable for review’.

It is important to note that the assignments to these seven qualifications were not necessarily agreed on by all the experts involved in our investigation, including the IPCC authors. These assignments were based on the judgment of PBL policy analysts who had to establish the reliability of IPCC reports as part of their work at the science–policy interface. The aim of this exercise has been to identify potential areas for improvement in future IPCC assessments. The underlying assumption being that times have changed and that, consequently, quality and transparency requirements for the IPCC have been raised.Results and discussion 33 

# Results and
Discussion

This Chapter consists of four parts. First, it presents the appraisal of the statements on regional climate-change impacts that appear in the Summary for Policymakers of the Synthesis Report. Second, an overview is given of errors found in the W orking Group II Fourth Assessment Report. Third, it addresses the approach followed by Working Group to arrive at its summary conclusions on the regional impacts of climate change. And fourth, a thematic overview is given of our comments on the IPCC report, aimed at further improving the IPCC process.

Our detailed analysis of the regional chapters and summaries can be found in Part II of the present report.

3.1 Status of IPCC summary statements on regional climate-change impacts

In this investigation we have not encountered errors that undermine the summary conclusions of the Fourth Assessment Report of the IPCC on the regional impacts of climate change. The Working Group II Report presents ample evidence of observations of climate-change responses by physical and biological systems. Furthermore, the consequences of climate change generally impose a serious risk given the projections on future climate change of Working Group I. However,
there is considerable room for scientific debate on statements that refer to climate-change impacts on a regional scale. We have found that, the more specific statements become in time and space, the larger the uncertainties. In addition, the body of scientific publications is smaller, in particular where it concerns local and regional changes occurring in developing countries.

Of all the summary statements on the regional impacts in the Working Group II Report – a subset of which is included in the 32 statements in T able SPM.2 of the Synthesis Report – only one was found to be affected by a minor error. This minor error, in our view, requires a response and reassessment by the IPCC.

Besides checking the summary statements for possible errors, we also tested the quality of these statements with respect to a range of issues: attribution, generalisation, transparency of expert judgment, consistency of messages, traceability of references, credibility of references, and availability of statements for review (see Section 2.2). We have one or more comments to make on seven 

# 3Assessing an IPCC assessment 34 **Results from PBL analysis for 32 statements contained in Table SPM.2 of the Synthesis Report** 

**Region Topic (and system/sector)**  **Error in statement? Comments?** 

Africa Water stress (freshwater resources) Yes (minor) No

Yields from rain-fed agriculture (food, fibre and forest products) No Yes (major)

Cost of adaptation to sea level rise (coastal systems and low-lying areas) No Yes (minor)

Arid and semi-arid land (ecosystems) No No

Asia Freshwater availability (freshwater resources) No Yes (major)

Flooding from sea and rivers (coastal systems and low-lying areas) No No

Natural resources and the environment (ecosystems) No No

Diarrhoeal disease (health) No No

Australia and New Zealand Biodiversity loss (ecosystems) No Yes (minor)

Water security (freshwater resources) No No

Production from agriculture and forestry (food, fibre and forest products) No No

Risks from sea level rise and increases in storms and coastal flooding (coastal systems and low-lying areas) No No

Europe Inland flash floods, coastal flooding and erosion(industry, settlement and society; coastal systems and low-lying areas) No No

Mountainous areas (ecosystems) No Yes (minor)

Southern Europe (most systems/sectors) No No

Heat waves and forest fires (health) No No

Latin America Disappearance of tropical forest (ecosystems) No No

Biodiversity loss (ecosystems) No No

Productivity in crops and livestock (food, fibre and forest products) No Yes (major)

Water availability (freshwater resources) No No

North America Western mountains and water resources (freshwater resources) No No

Rain-fed agriculture (food, fibre and forest products) No Yes (minor)

Heat waves (health) No No

Coastal communities and habitats (coastal systems and low-lying areas) No No

Polar regions Biophysical effects (ecosystems) No No

Impacts for human communities (industry, settlement and society) No No

Infrastructure and indigenous lifestyle (industry, settlement and society) No No **Table 3.1**Results and discussion 35 of the 32 statements contained in Table SPM.2 of the Synthesis Report. If the criteria, as used in this investigation for assessing the quality of statements, would be systematically applied in future IPCC assessments, it would become easier for the users of these assessments, such as the PBL, to establish the reliability of IPCC conclusions. Table 3.1 summarises the results for the statements contained in Table SPM.2 of the Synthesis Report.

3.2 Errors

Inaccurate statements (code: E1) Table 3.2 presents all newly found errors in the regional chapters of the Fourth Assessment Working Group II Report, as well as the error regarding the Himalayan glacier melting and the error about the percentage of Dutch land area below sea level. Of these errors, only one travelled up to the Summaries for Policymakers. The range for the number of people projected to be exposed to increased water stress due to climate change in Africa, by 2020, turned out to have been erroneously calculated by adding up the ranges for African sub-regions that were given in the underlying reference. The author of the original study – who had recently discovered this error – performed a recalculation, which led to a slightly smaller uncertainty range (90 to 220 million people, instead of 75 to 250 million). W e considered this a minor mistake, since the difference does not significantly change the message and the original paper did not provide the details that would have allowed the correct computation of this range. In our view, the IPCC should quickly respond to and assess this slight inaccuracy, even though it is not significant given the uncertainties surrounding such projections.

Only one major new error was found, in Chapter 9 on Africa. A projected decrease by 50 to 60% in extreme wind and turbulence over fishing grounds was mistakenly represented as a 50 to 60% decrease in productivity as a result of changes in wind and turbulence. However, this error and the remaining five other errors that were newly found did not travel to the level of the summaries, and therefore do not affect the summary conclusions. This is also the case for the two errors that were discovered before we began our investigation (concerning the melting rate of the Himalayan ice and the percentage of Dutch land area below sea level). **Region Topic (and system/sector)**  **Error in statement? Comments?**  Vulnerability of ecosystems and habitats (ecosystems) No No

Small islands Sea level rise (coastal systems and low-lying areas) No No

Coastal conditions (coastal systems and low-lying areas) No No

Water resources (freshwater resources) No No

Invasion by non-native species (ecosystems) No NoAssessing an IPCC assessment 36 Inaccurate referencing (code: E2) Table 3.3 provides an overview of all newly found errors in referencing in the regional chapters of the Fourth Assessment Working Group II Report. We found four instances of inaccurate referencing, which could and should be repaired by issuing errata. These inaccuracies have no effect on the summary conclusions. **Inaccurate statements identified by the PBL in the IPCC Working Group II Report and the Synthesis Report of 2007**  **a**

**Region Pages Error Major/**  **minor**

**error Simple**  **erratum/**
**response needed**  **Effect on Summary for Policymakers**  **WG II and Synthesis Report** 

Africa 11 (SYR), 13, 435, 444 Increased water stress by 2020 ‘between 75 and 250 million people are project to be exposed’ should be ‘between 90 and 220 million people are projected to be exposed’ (see Section 5.1) Minor Response needed Yes

435, 444 Increased water stress by 2050s ‘350- 600 million people’ should be ‘350- 490 million people’ (see Section 5.2) Minor Response needed No

448 Productivity decrease in fisheries under doubling of CO **2** ‘extreme wind and turbulence could decrease productivity by 50-60%, while turbulence will probably bring about a 10% decline in productivity in the spawning grounds, and an increase of 3% in the main feeding grounds’ should be ‘ the frequency of extreme wind and turbulence events could decrease by 50-60%, while mean turbulence will probably decline by 10% in the spawning grounds and increase by 3% in the main feeding ground of anchovy (IPCC proposed an alternative erratum; see Section 5.2) Major Erratum No

Asia 493 Melting of Himalayan glaciers The second paragraph of Section 10.6.2 of the Working Group II contribution and a repeat of part of the paragraph in Box TS.6 refer to poorly substantiated estimates on rate of recession and date for the disappearance of Himalayan glaciers (2035). (See Annex B) Major Response needed No

Australia and New Zealand 50 Observed warming since 1950

‘ 0.3 to 0.7 °C’ should be ‘ 0.4 to 0.7 °C’ (see Section 7.2) Minor Erratum No

Europe 547 ***** Percentage of Dutch land area below sea level

‘ below sea level ’ should be ‘ at risk of flooding ’ (see Section 8.2) Major Erratum No

555 Decreases in crop yields by 2050 ‘legumes −30 to +5% ’ should be ‘legumes −14 to +1% ’; ‘sunflower −12to +3% ’ should be ‘sunflower −12to +1% ’; and ‘tuber crops −14 to +7% ’ should be ‘tuber
crops −9 to +8% ’ (see Section 8.2) Minor Erratum No

Latin America 583 Frequency of hurricanes ‘frequency and intensity of hurricanes’ should be ‘frequency and intensity of intense hurricanes’ (see Section 9.2) Minor Erratum No

598 People living in water-stressed watersheds

‘ Net increases in the number of people living in water-stressed watersheds’ should be ‘ Number of people living in water-stressed watersheds and net increases due to climate change ’ (see Section 9.2) Minor Erratum No

a These statements pertain to regional impacts of climate change (addressed in Working Group II Chapters 9–16), together with the two known inaccuracies. Unless stated otherwise, the page numbers refer to the Working Group II Report. * Erratum already published by IPCC (last checked June 25th). **Table 3.2**Results and discussion 37 The errors traced in the investigation presented above do not undermine the conclusions of the IPCC on the regional effects of climate change.

3.3 Risk-oriented approach

The authors of the Summary for Policymakers of the Working Group II Fourth Assessment Report – and especially those of the Summary for Policymakers of the Synthesis Report – during the Fourth Assessment were asked by the Working Group II Co-Chairs to identify the conclusions in the Working Group II assessment as ‘key findings’. These key findings would, in turn, be described in two ways: as textual descriptions of ‘main projected impacts’, and as a tabulated range of possible impacts scaled to climate change. With regard to the main projected impacts, examples of those that travelled to the Synthesis Report, Co-Chair Dr. Parry, explained the approach to us as follows:

‘In outline: i) we asked author teams to identify a) a tabulated range of impacts for which uncertainty ranges were known and b) a description of the ‘main projected impacts’ (as well as vulnerabilities and adaptation issues); ii) we made clear that by ‘main’ we meant the most significant challenges and opportunities that need to be faced; iii) we asked that each author team’s list should cover the main relevant impact fields (e.g. water, food, health, coasts, ecosystems, etc.) but not exhaustively if some were not relevant; iii) reviewers and review editors commented on the selection of main projected impacts, and the lists were frequently revised; v) the order of listing is not a rank order; vi) there should be a line of sight connection between the listings of conclusions, i.e. from executive summary of the chapter, through the Technical Summary, the Summary for Policymakers of the Working Group II Report and the Synthesis Report, but vii) the wording of the conclusions might need to vary according to its context in these different places. . . .

[Thus] our conclusions list the main projected impacts. Most of these impacts involve potential losses in human welfare and therefore are negative. This stems from two things: i) because the balance of impacts is, indeed, negative in welfare terms and also ii) because positive impacts can often be exploited by mechanisms other than policy, i.e. autonomous adaptation.’ (Parry, personal communication, 20 June 2010) **Inaccurate references newly identified by the PBL in the IPCC Working Group II Report of 2007, Chapters 9–16** 

**Region Page Proposed erratum** 

Asia 59, 471 Coral reefs The reference to Wilkinson (2004) should be replaced by Wilkinson (2000) (see Section 6.2)

Australia and New Zealand 518 Great Barrier Reef The references to Berkelmans et al. (2004), Crimp et al. (2004) and Jones et al. (2004b) should be replaced by a reference to Sheehan et al. (2006) (see Section 7.1)

Latin America 598 Water supply The references to Vásquez (2004) and UNMSM (2004) should be interchanged (see Section 9.2)

Small islands 697 Freshwater lens The reference to World Bank (2000) after ‘would reduce the freshwater lens by 65%’ should be replace by Falkland (1999) (see Section 12.2) **Table 3.3**Assessing an IPCC assessment 38 The approach of Working Group II towards the selection of specific conclusions from the regional chapters has thus focused on which impacts would be the most important for policymakers to respond to.

The Synthesis Report, on page 47 (Section 3.3 on ‘Impacts of future climate change’), gives a brief explanation of the selection of the particular key findings:

‘The following is a selection of key findings14 regarding the impacts of climate change on systems, sectors and regions, as well as some findings on vulnerability, for the range of climate changes projected over the 21st century.’

Footnote 14 specifies the selection criteria:

‘Criteria of choice: magnitude and timing of impact, confidence in the assessment, representative coverage of the system, sector and region.’

And in the Summary for Policymakers of the Synthesis Report, the note to T able SPM.2 (‘Examples of some projected impacts’) states:

‘The magnitude and timing of impacts that will ultimately be realised will vary with the amount and rate of climate change, emissions scenarios, development pathways and adaptation.’

As a result of the approach that was followed, mostly negative impacts were singled out (see Table SPM.2, reproduced in Annex A of the present report). In fact, the reason why hardly any example of a positive impact of climate change can be found in Table SPM.2 of the Synthesis Report is that, according to the IPCC authors, most of the positive impacts were not of sufficient policy relevance. The W orking Group II Co-Chair further explained the following:

‘i) most main impacts are negative, because mankind’s activities are generally tuned to current climate, not to something different from that; most change therefore implies a cost; ii) in net welfare terms, as Working Group II found, aggregate impacts are indeed estimated to be negative (see much of Chapter 20 of the Working Group II Report, summarised on page 17 of the Summary for Policymakers); iii) autonomous adaptation is better at making more of opportunities than protecting against the adverse, and iv) policy is experienced in providing protection against the adverse.’ (Parry, personal communication, 20 June 2010)

We labelled this approach to summarising as a ‘risk-oriented’ approach – mainly, because it implicitly guides assessors to focus more on risk than on opportunities. We believe that this risk-oriented approach has been implicit in the process to arrive at conclusions based on the chapters and that by accepting its outcomes it has in effect been approved by all the governments. The consequence of this approach for which facts did or did not travel to the highest levels of summary, can be illustrated by several examples. For instance, the focus in T able SPM.2 of the Synthesis Report is on crops for which yields are likely to be reduced in Africa, but the table does not mention the crops for which yields are likely to increase due to climate change. For Australia and New Zealand, Table SPM.2 shows many risks but only one benefit of climate change (initial benefits for agricultural productionResults and discussion 39 in New Zealand), while Section 11.4 on key future impacts and vulnerabilities concludes: ‘climate change is likely to give rise to six key risks in specific sectors. [...]. There are also four key benefits for particular sectors.’ Furthermore, in Africa,
also decreases in the numbers of people suffering from water-stress due to climate change are projected for some areas. Of course, these numbers cannot be simply subtracted from increases in water stress elsewhere (other people are involved), but they constitute potential positive effects of climate change. Finally, an example of a selection against mentioning the positive effects of climate change was found with respect to the likely positive effect of climate change on forest production in North Asia (Russia), which some policymakers may find to be of relevance to their decision making. This potentially positive effect did not reach the Summaries for Policymakers, partly because the authors attached a ‘medium confidence’ to their

statement.

A second aspect of the risk-oriented approach that was followed by the W orking Group II Fourth Assessment Report, is that in summary statements on the impacts of climate change, information rightfully was provided on the climate-change risk. However, this information was often not contextualised in the summary statements by also mentioning other impact factors. This was even the case when these other factors were much larger than the impact that was attributable to climate change. To give an example, Arnell (2004) showed that the number of people living in water-stressed watersheds (defined as having less than 1,000 m **3** of water per year, per person), even without climate change, would rise strongly over time (a rise of 1.5 to 2 billion people by 2025, globally, compared to 1995, see his Table 5), mainly due to population growth in already water-stressed areas. This increase is much larger than the additional increase related to climate change. Again, some policymakers may wish to see both numbers – that is, changes with and without climate change – within the same context in a summary.

A third, and final, characteristic of the risk-oriented approach is that, often, the upper ends of uncertainty ranges (the worst outcomes that are projected) were highlighted. One example comes from the chapter on Australia and New Zealand, in which 60% of the Great Barrier Reef was projected to regularly bleach, by 2020. This percentage is at the upper end of the uncertainty range. Another example, in the Synthesis Report, in Table SPM.2, is the illustration of the risk of species loss in the European mountains, by the statement ‘in some areas up to 60% under high emissions scenarios by 2080.’ This statement highlights the upper end of the uncertainty range (the most sensitive situations and the scenario with the most impact), and does not specify the whole uncertainty range; although the words ‘up to’ signal that there is a range, only the upper end was specified, causing it to be highlighted.

We believe it is appropriate – and even necessary – to apply a risk-oriented approach to climate-change impact assessment. It is a common approach in the fields of, for example, public health, public safety, and anti-terrorism. Nevertheless, we note that the nature of such an approach, especially for the Summary for Policymakers of the Synthesis Report, could easily go unrecognised by readers. The legitimacy of the findings could be increased if the reader would be explicitly informed about the fact that a particular approach has been employed. However,Assessing an IPCC assessment 40 we feel that a risk-oriented approach is not enough. After all, some policymakers may wish to also receive information on the positive regional consequences and the full ranges of uncertainty at the highest level of summary. Therefore, a more explicit discussion on this issue in the IPCC Plenary may be considered.

3.4 C omments

In this investigation, besides looking for errors, we have evaluated the quality of statements on regional climate-change impacts in the IPCC Fourth Assessment Report by using seven criteria (see Section 2.2). A full account of our findings is given in Part II. In the section below, an overview is given of our main comments, grouped along our seven criteria. **1)**

Attribution (code: C1) The regional chapters in the Working Group II Report are typically judicious in separating out the impacts that they attribute to anthropogenic climate change. In the text examined, only one instance was found of an ‘insufficiently substantiated attribution’ (major comment), in which a future attribution is questionable and more care should have been taken to specify what part of a projected change depends on climate change as its driver.

In the Technical Summary, in Table TS.4, the number of additional heat-related deaths per year is given for Australia, with the suggestion that all 3,000 to 5,000 additional heat-related deaths projected for 2050 would be dependent on an increase in temperature. However, a large part of this increase is solely due to changes in population size and age distribution (see Section 7.2). In future assessments, even more care must be taken to prevent insufficiently substantiated
attributions.

Generalisation (code: C2)In most cases, the generalisations contained in the regional chapters are wellunderpinned by the chapter material. In the text examined only two instances were found where the generalisation is questionable (both major comments). T wo statements in Table SPM.2 of the Synthesis Report involve ‘insufficiently founded generalisations’. First, it is questionable whether freshwater availability will decrease in all parts of Asia as indicated (Central, South, East and South-East Asia) (see Section 6.1). Second, it is questionable, on the basis of evidence presented in the chapter, whether productivity will decrease also for livestock farming other than cattle, and, more generally in Latin America, because the evidence provided relates only to Bolivia and Argentina (see Section 9.1). In future assessments, more care must be taken to prevent insufficiently founded generalisations.

1) P art II also presents five of our comments that do not fit our qualification categories. Amongthese, there is one major comment on the ambiguity of the meaning of phrases such as ‘in the Sahelian region’ (see Section 5.3).Results and discussion 41 Transparency of expert judgment (code: C3) One of the main findings of our investigation is that, for six out of the 32 key statements on climate-change impacts in different regions (Table SPM.2), the regional chapters insufficiently convey the IPCC authors’ reasoning behind their weighing of the evidence that was available from the literature (‘insufficiently transparent expert judgment’). For other important statements in the regional chapters, also, information on the reasoning behind them was lacking. Expert judgment will always have to play a crucial role in IPCC assessments, or in fact in any assessment. But in those cases where readers may wish to ascertain the reliability of a statement, having the authors’ reasoning explained in the text would be of great help in improving the credibility of the claims made.

Below, by way of example, is a list of six questions pertaining to statements in T able SPM.2, which we were unable to find the answer to, on the basis of the regional chapters and the references contained therein:

1. Why is a projected decline in cereal yields in dry years in Morocco, by 2020, representative of an overall decline in yields in ‘rain-fed agriculture’ in ‘some countries’ in Africa? (see Section 5.1) (major comment)

2. With respect to the cost of adaptation to the projected sea level rise in Africa towards the end of the 21st century, we asked: how was the range of 0 to 156% of GDP for different countries in the referenced study translated to a 5 to 10% range for the whole continent? (see Section 5.1) (minor comment)

3. In which way is the information about changes in ‘freshwater availability’ in Asia (which in the chapter is given in both absolute flows and per-capita availability) combined to arrive at the assessment that freshwater availability is projected to decrease in many parts of Asia by the 2050s? (see Section 6.1) (major comment)

4. The report provides information on the range of temperature change for which a regular bleaching of 60% of the Great Barrier Reef is expected; how was this temperature range converted into the statement that this level of bleaching would be reached by the year 2020? (see Section 7.1) (minor comment)

5. What is the underpinning for the projected loss of up to 60% in plant and animal species, in some European mountainous areas, under high emissions scenarios by 2080 – given that the references named in the chapter only state this percentage in relation to plant species? (see Section 8.1) (minor comment)

6. How was the large range (from negative to positive) of projected changes in yields of rain-fed crops (varying over crops and locations) for North America translated to a 5 to 20% increase in ‘aggregate yields’? (see Section 10.1) (minor comment)

Relying on expert judgment is a necessary practice in situations where wellresearched findings are scarce and potential risks are great. In such situations, the joint deliberation of experts may simply be the best available way to make an

assessment. Y et, the transparency of the reasoning in such a deliberation would enhance the verifiability and hence the potential credibility of the assessments that experts arrived at. This investigation ex post reconstructed some of the reasoning. Through communication with the IPCC authors, a large part of the answers was received (these answers have been included in Part II), which led to an increase in our confidence about the statements. Although, for a significant part of the cases, the IPCC authors, when asked, were able to reconstruct their reasoningsAssessing an IPCC assessment 42 and convey them to us, we are convinced that, especially for statements that are included in the Summaries for Policymakers, it is necessary in future assessments to have such reasoning available in the underlying material (chapter texts).

Consistency of messages (code: C4) Our investigation revealed only two instances of inconsistencies (minor comments) within the report. This concerned inconsistencies in the likelihood level of increases in fires in Australia and New Zealand (‘likely’ according to the main text of Chapter 11 and ‘virtually certain’ according to the Executive Summary of the Chapter) and inconsistencies in the confidence level about the relevance of past experience with extreme events to policy-making on adaptation in Europe (‘high confidence’ in the Executive Summary of Chapter 12 and ‘very high confidence’ in the T echnical Summary and Summary for Policymakers). Such minor flaws could have been prevented by even more careful cross-checking and editing than was done for the Working Group II Fourth Assessment Report.

Another issue related to consistency of messages is the general lack of specification in the summaries about baseline years or periods related to the statements. Part II does not address this issue, since it deals with individual statements; here it is flagged as a more general issue. Variable baselines (20th century; 1961-1990; 1990; 1995; ‘today’) are used in the regional chapters of the Working Group II Report (as well as in the underlying references), but many statements in the summaries are vague about which particular baseline is being assumed for the projected ‘changes’. For qualitative accounts this is less important, but baselines need to be crystal clear when numbers are involved, since these are sensitive to the choice of baseline. Ideally, baselines should be the same throughout a report, but this is not always possible.

Traceability of references (code: C5) Of the references that we checked, only very few turned out to be untraceable. For instance, the projected ‘up to 50%’ reduction in yields in rain-fed agriculture for some countries in Africa, was ultimately based on an untraceable reference (MATUHE, 2001). Given the high importance attached to the statement, this is a major comment. On several other examples we had only minor comment to make. A hyperlink turned out to be outdated; it probably has been functioning in 2006/2007, but it was not in 2010 (FAO, 2002, see Section 9.2). Yet another example concerns a paper on a projected increase in coffee-leaf miners in Brazil, which was listed in the references of Chapter 13 as ‘submitted’ to Climatic Change , but we found that the paper had never been accepted by this particular journal, which makes this reference untraceable. Although we found a paper with the same title that was published in 2008, in Pesquisa Agropecuária Brasileira, Brasília , the paper had not been accepted by any journal at the time the IPCC report was published. It would be our advice that papers may only be used as literature reference in future IPCC reports if they have been accepted for publication prior to the publication date of the IPCC assessment report.

Credibility of references (code: C6) Only two instances were found of statements relying on references to grey literature, when literature published in peer-reviewed journals was also available toResults and discussion 43 which referencing would have been more appropriate. For example, the statement that up to 40% of the Amazon Rainforest could react drastically to even a slight reduction in precipitation was underpinned by a reference to a peer-reviewed WWF/IUCN report of 2000 on forest fires, while – in this case – also more relevant high profile peer-reviewed journal articles had been published by Cox et al. (2000; 2004). In our opinion, this issue deserves more attention from IPCC authors in the
future. We consider it a minor comment.

Availability of statements for review (code: C7) In one instance, we found that a new, quantitative conclusion was added after the Second Order Draft had been completed, without there being sufficient grounds from review comments for such an addition. This statement, on the ‘up to 50%’ reduction in yields from rain-fed agriculture for some countries in Africa, thus, had been unavailable for review, while it was allowed to travel from an untraceable source up to the highest level of summary (Table SPM.2 of the Synthesis Report). This is a major comment. It is our conviction that such high-impact information should have been made available for review, and that this is applied in future IPCC reports.

3.5 C onclusion

No significant errors found in summary conclusions The foundations for thirty-two IPCC Fourth Assessment summary conclusions on the regional impacts of climate change have been investigated. These conclusions show examples of projections of climate-change impacts on food, water, ecosystems, coastal regions and health, for all the earth’s continents. These conclusions have not been undermined by errors, although one of the conclusions contains a minor inaccuracy: in hindsight, not 75 to 250 million people, but 90 to 220 million people are projected to be exposed to increased water stress due to climate change in Africa, by 2020.

The provenance of summary statements needs to become more transparent in future reports Seven of the investigated 32 conclusions on the regional impacts of climate change contain information that we were unable to sufficiently trace to the underlying chapters in the IPCC Working Group II Report or to the references therein. For two of these conclusions, our critical comments pertain to insufficiently founded generalisations from existing scientific research, in both cases from local to regional scales, and for one of them also from one type of livestock to livestock in general. While acknowledging the essential role of expert judgment in scientific assessments, the PBL recommends to improve the transparency of these judgments in future IPCC reports.

The regional chapters: one error and some comments In the underlying regional chapters, in addition to the two already known errors about the melting of the Himalayan glaciers and about the Dutch land area below sea level, another significant error was found: a 50 to 60% decrease in productivity in anchovy fisheries on the African west coast was projected on theAssessing an IPCC assessment 44 basis of an erroneous interpretation of the literature references. W e found certain inaccuracies, ranging from (very) small errors in numbers to imprecise literature references. In addition, we have some critical comments to make. One of themrelates to the fact that the report does not specify how many of the additional heatrelated deaths projected for Australian cities are actually attributable to climate change – a sizeable fraction is due to demographic changes alone. However, these shortcomings do not affect the investigated 32 summary conclusions or other parts
of the IPCC summaries.

Examples of negative impacts dominate at summary level The IPCC Working Group II Report focuses on climate impacts, adaptation and vulnerability. It was found that, in the IPCC’s highest level summary, the conclusions that were derived from the regional chapters of the Working Group II contribution to the Fourth Assessment Report single out projections of negative climate-change impacts. Our investigation revealed that IPCC authors considered these to be the most relevant to policymakers. The PBL has labelled this as a ‘risk-oriented’ approach, which had been implicitly endorsed by the governments that constitute the IPCC (including that of the Netherlands). The PBL subscribes to the importance of an approach that highlights what may go wrong under unmitigated climate change, but the Fourth Assessment Report lacked a clear explanation of the choice of approach and its consequences. Alternatively, it could be argued that policymakers should be presented with a complete picture in the Summaries for Policymakers, not just with negative examples (without suggesting that potential positive effects cancel out potential negatives effects).

No consequences for overarching conclusions Our findings do not contradict the main conclusions of the IPCC on impacts, adaptation and vulnerability related to climate change. There is ample observational evidence of natural systems being influenced by climate change on regional levels. And the negative impacts under unmitigated climate change in the future pose substantial risks to most parts of the world, with risks increasing at higher global average temperatures.Recommendations 45 

# Recommendations

Based on our findings and conclusions in the previous chapters, we present recommendations for the Fifth Assessment Report process. Again, there will be four assessment reports (by Working Groups I, II, and III and a Synthesis Report), due in 2013 and 2014. The report by Working Group II will consist of two separate volumes; a sectoral part A and a regional part B (IPCC, 2009). Our recommendations primarily apply to part B of the Fifth Assessment Report by Working Group II, although most of our recommendations may also apply to the entire Fifth Assessment Report.

4.1 Minimising the risk of errors

In comprehensive scientific assessments of thousands of pages that refer to a large corpus of scientific work, errors are inevitable, especially when dealing with quantitative information. The current IPCC review process is meant to be instrumental in removing errors that creep into the texts during the writing of the several drafts. If errors do occur in the main texts, they rarely propagate to the level of the Summary for Policymakers, since this relatively short text of approximately 15 pages is reviewed and approved line by line, word by word by the IPCC Plenary. In our investigation, we found only one minor error in the investigated 32 statements.

However, the full range of IPCC reports is supposed to be authoritative, also in the underlying texts. Therefore, as much as possible, errors need to be detected and removed. With regard to finding and correcting errors, we believe that an open, pro-active and transparent policy, involving wide circles of the assessment report users, would increase the reliability of IPCC, given its recent increased visibility and public exposure. It is a complex and highly dynamic field. Small mistakes belong to this process and should not be seen to jeopardise the overall credibility of science. However, concerns about statements that are made in such elaborate assessments are inevitable. By enabling constant updating of documents and by publishing ‘errata’ as an integral part of the process, the IPCC may strengthen the societal legitimacy of its work.

In light of the above, we recommend:

 C reate a public registration website for the submission of possible errors found in the published report . Errors are inevitable, therefore, the IPCC should make provisions to deal with these, immediately following publication of an IPCC report. Since a few months, erratum pages are available on the IPCC website for the Fourth Assessment Reports of 2007. For the Fifth Assessment Reports to be published 

# 4Assessing an IPCC assessment 46 in 2013 and 2014, such erratum pages should be made available directly after publication. In addition, we suggest the launch of a public registration website for the reporting of possible errors. Subsequently, these errors (e.g. regarding numbers or typographic errors), when found to be genuine, could be quickly cor

rected by publishing them on the erratum website. For more substantial errors, additional comments and explanations from the IPCC should be added and also posted on the website. From our own recent experience, we believe that such a registration website, if designed properly, would be manageable. Experience with this approach may be gained by setting up experimental registration web

sites for the two upcoming Special Reports in 2011.

 Check and double-check references . More checks should be done with regard to accurate referencing to the scientific literature, which should always betraceable and available for reviewers. With regard to the traceability, a userfriendly option would be to create hyperlinks within the text leading to the original publications.

 Investigate possibilities of pre-publication ‘crowd sourcing’ . Finding errors, notably in numbers, is a time-consuming effort for authors, reviewers, Co-Chairs and Technical Support Units, who already are limited in terms of time and resources. The possibility of using ‘crowd sourcing’ technologies as part of the review during the last stages of completion, therefore, should be investigated. Crowd sourcing has the advantage of providing a large reservoir of voluntary resources to be tapped. A disadvantage could be the amount of resources needed to manage and moderate tasks that would be handed out to the public.

4.2 Investing in the improvement and transparency of foundations of summary conclusions

The findings in Section 3.4, and the more detailed analysis in Part II of this report, lead us to the following recommendations:

 Make sure that statements that attribute impacts to climate change are well founded in scientific research, including systematic observations, modelling and statistics. The climate-change component of impacts should be carefully
characterised.

 Make sure that expert judgments that are incorporated in summaries are made transparent and plausible by explaining the line of reasoning behind them in the main text. This applies especially to:

**–** case-study generalisations to whole sectors, areas or regions; such generalisa

tions should be well supported by scientific evidence;

**–** choosing certain intervals of specific numbers from a wide range of figures. This would be a great help to any institute that is asked to assess the plausibility
and reliability of statements, and could enhance the credibility of the assessment process. Authors are faced with the dilemma that strict limitations on page lengths are imposed by the Working Group management, limiting the amount of
information that can be included.

 Check and double-check consistency of messages and confidence intervals/ likelihood levels between chapters and different summaries.

 Grey literature should only be used when peer-reviewed scientific journal literature on a certain subject is not available.Recommendations 47 4.3 Strengthening the quality control by the chapter teams

IPCC authors work on a voluntary basis, or their work is a part of their regular jobs as scientists and, often, it is an addition to their regular workload. The budget for the IPCC organisation is relatively small, and assessment time lines are kept very strict. The high cost-effectiveness is both a strength and a weakness in the IPCC

process.

The amount of scientific literature on climate change is skyrocketing, policy implications become more visible and, recently, the credibility of the IPCC has come under scrutiny by the media. This implies a need for higher quality standards, next to a future increase in the amount of assessment work. Moreover, Coordinating Lead Authors have to manage their writing teams, and deal with large flows of information within a limited amount of time. Therefore, we recommend that a procedure is set up – within the shortest possible time frame, and which will be implemented at the very beginning of the writing process – for appointing and employing chapter assistants who will help the Coordinating Lead Authors with quality-control issues, such as the checking of references (including basic checks on quantitative information), the cross-checking between findings on regional and sectoral levels, the cross-checking between statements concerning regional climate projections in the Working Group II Report and those in the report by Working Group I. In order to make all this possible, a serious increase in the contributions from the governments that are involved in the IPCC would be necessary.

4.4 Strengthening the review process

We recommend that certain improvements are made in the review process, in order to further minimise the risk of errors and to improve the foundation of the summary conclusions:

 Ensure that all texts are fully covered by expert reviewers . The current review process is generally rather passive — expert reviewers are free to comment on those parts of the text that they are interested in, but there is no guarantee that qualified expert reviewers will scrutinise all parts of all texts including the references. A pro-active targeted expert review is therefore recommended. The IPCC could establish a web-based nomination procedure, in which experts subscribe to reviewing certain parts of a report. The Co-Chairs and T echnical Support Units could then select the experts and see to it that all parts of the texts are covered by at least a few qualified reviewers.

 Do not allow new material to be introduced after the last review if this m aterial is not related to the review . After the Second Order Draft review, new material should only be added if it is clearly in response to, or related to substantive issues raised in review comments. Otherwise, newly introduced material would be excluded from expert review. The IPCC procedures should stipulate more clearly how new
material should be dealt with.

 Do not allow referencing to scientific articles that have been submitted bu t not (yet) accepted for publication by the time the IPCC assessment report is published. This has already been good practice in many of the IPCC reports, but IPCC procedures should provide some ruling on this issue.Assessing an IPCC assessment 48  Review by governments of the foundation and provenance of statements in Summaries for Policymakers . Governments have an important responsibility in the review process and, therefore, should seriously invest in the government/expert review of Second Order Drafts. Apart from the reviewing of issues of specific national interest, governments, assisted by independent experts, could review provenance and foundations for statements in the draft version of the Summary for Policymakers. This report provides an example of how this could be done methodologically. Since this would be a time-consuming effort that draws from many resources, the IPCC could consider extending the expert/government review period for the Fifth Assessment Report. Governments could work together in this process and share resources. In addition, this would provide them with a solid frame of reference for the line-by-line approval session of the final Summary for Policymakers.

4.5 Timing of the assessments

The quality of regional chapters could be further improved by making better use of information that is available in the sectoral chapters of Working Group II reports. The best way to achieve this would be to finalise regional chapters not until sectoral chapters have become final. This would help to integrate information at regional levels and contribute to the transparency of the expert judgments. In particular, this can help to provide stronger underpinning of generalisations of case studies to entire regions or sectors, by also making use of regional modelling studies.

4.6 Balancing the assessment of climate-change impacts

The section on ‘risk-oriented approach’ (3.3) reveals that, at summary level, the most important negative impact projections have been highlighted. This approach is understandable and justifiable, but it has not been made explicit. W e believe that such a risk-oriented approach, although essential, is also one-sided. It could be argued that the Summary for Policymakers should provide policymakers with a full picture; it should not only present the most important negative examples (without suggesting that potential positive effects cancel out potential negative effects). W e recommend that the Fifth Assessment Report explains which approach has been
taken.

In addition, we suggest that a Summary for Policymakers includes two separate sections dealing with projected regional impacts on water, food, ecosystems, coastal regions and health, divided as follows:

 One section that describes robust outcomes, with the full range of projections of climate-change impacts, including not only the uncertainties, but also any positive impacts, containing:

**–** A description of uncertainties, as has been employed by Working Group III in

their report in the Fourth Assessment. Per statement, an indication could be given of the amount of evidence (number and quality of independent sources) and level of agreement (on a particular finding). At summary level, the first choice would be to highlight statements that are based on a large number ofRecommendations 49 publications all converging to the same conclusions. ‘Likelihood’ statements could only be made if sufficient scientific evidence would be available, and ‘confidence’ statements would be expert judgments, which are subjective by nature and, normally, are untraceable for the reader;

**–** An indication of the relative importance of other drivers that affect food,

water, ecosystems, coastal areas, and health, compared to drivers of climate change, such as population growth, industrialisation and land use. The impacts that can be attributed to climate change should be separated, as clearly as possible, from those that are caused by other factors.

 One section that explicitly describes the most important negative impacts, as was done for the Fourth Assessment Report, but this time including a worstcase risk approach, based on clearly explicated risk assessment meth odology. This section could highlight future impacts that may be unlikely to occur, but could have major consequences, and therefore would pose significant risks. Such a section could also include more speculative projections of high climate sensitivity, accelerated warming, and tipping points in ecosystems. An exploration of such an approach has been presented in the PBL study ‘ News in Climate Science and Exploring the Boundaries’ (December, 2009).

Reasons to consider this line of action include:

 The outsiders of the scientific community that study impacts, adaptation and vulnerability, will not automatically understand the ‘risk-oriented approach’,
as we have labelled the W orking Group II approach that leads to a sample of the most important negative impacts. Working Group I followed a different approach, establishing the full range of possible projections including its uncertainties. Without proper explanation, the results at the summary level of Working Group II could easily be interpreted as being an alarmist view (as was our interpretation at the outset of our analysis, before we received further explanation from the Working Group II Co-Chair and authors).

 W e believe that policymakers (and their analysts) need to see the complete picture at summary level. Positive impacts are important, because they may also become smaller when mitigation measures are introduced. There may be offsets possible between positive and negative impacts of climate change, for instance, in relation to agricultural yields. Although there are justifiable objections against making a cost-benefit analysis of positive and negative impacts, to not mention them would be ‘policy-pre-emptive’.

 In order to obtain a complete picture, we also believe that policymakers – inboth public and private sectors – would need to be informed about the worstcase risks, when they are developing risk management, disaster prevention and insurance policies.

 Finally, some of the comments made and the errors found – including those received through our registration website – were related to the issue of accentuating or even heightening the severity of climate-change impacts. W e recommend that the authors of the Fifth Assessment Report (AR5) be made specifically aware of this issue. Care should be taken with phrasing of statements that could be perceived by readers as heightening the projected impacts of climate change.Assessing an IPCC assessment 50 4.7 Investing in climate-change impact science

Climate-change impact science is relatively young, although the amount of literature is expanding rapidly. Figure 1.8 in the Working Group II Report and Figure SPM.1 in the Synthesis Report (see Figure 1.1 in this report) show that there are many observational data available on Europe and the United States that indicate physical and biological changes, over time, but there is only a small amount of observational data available on such changes in developing countries, while developing countries are more vulnerable to climate change. In addition, model projections become more uncertain at geographically more detailed levels, while policymakers ask for concrete information on their specific region or location.

This calls for intensifying investments in regional climate impact observation systems, regional modelling and capacity building, particularly in developing
countries

Re ferences
IPCC, 2009. Chapter outline of the Working Group II contribution to the IPCC Fifth Assessment report

(AR5), http://www.ipcc.ch/meetings/session31/doc20-rev1.pdf, PBL, 2009. News in Climate Science and Exploring the Boundaries. http://www.rivm.nl/bibliotheek/

rapporten/500114013A.pdfPart II – Detailed analysis of regional chapters and summaries 51 

# Part II – Detailed analysis of regional chapters and
summaries

This Part presents:

 our findings referring to the foundation for the 32 summary conclusions on regi

onal impacts in the Synthesis Report (see Annex A) – below, these conclusions have been typeset as headers, followed by our findings;

 additional findings at lower levels, such as those referring to the foundations for statements on regional impacts in the Technical Summary and Executive Summaries of the regional chapters, and also some findings on parts of the main text that we came across while working on other findings – below, statements have been typeset as headers, followed by our findings.

For our investigation, we used several criteria for assessing the quality of IPCC statements. First of all, we employed a distinction between obvious factual ‘errors’ – which we believe require an erratum on the IPCC Fourth Assessment web pages – and ‘comments’, the latter being critical remarks, made from our specific position of having to assess these statements on behalf of Dutch policymakers. These crite

ria were developed ‘on the job’, while categorising our findings. In total, there are nine criteria, and associated with these are two types of ‘errors’ (E1-E2) and seven types of ‘comments’ (C1-C7):

 E1: Inaccuracy of a statement ;

 E2: Inaccuracy of literature referencing ;

 C1: Insufficiently substantiated attribution , when a certain impact had been attributed to climate change without convincing foundation;

 C2: Insufficiently founded generalisation , when case studies had been extrapolated to whole regions/sectors without convincing foundation;

 C3: Insufficiently transparent expert judgment , when we could not trace the reasoning behind a statement from underlying texts or literature references given;

 C4: Inconsistency of messages , when found between the different layers in the IPCC report;

 C5: Untraceable reference , when a literature reference could not be found;

 C6: Unnecessary reliance on grey referencing , when reference had been made to grey literature (literature other than from peer-reviewed journals), while adequate peer-reviewed scientific journal references would have been available;Assessing an IPCC assessment 52  C7: Statement unavailable for review , when new information had been introduced after the last review but was not clearly derived from a content issue raised in
this last review.

When a certain statement in the IPCC Fourth Assessment Report would meet all the criteria, the statement was considered to be well founded and reliable.Africa 53 

# Africa

5.1 Analysis of statements in T able SPM.2 of the Synthesis Report

By 2020, between 75 and 250 million of people are projected to be exposed to increased water stress due to climate change. (E1; minor) This statement was based on Table 11 in a study by Arnell (2004), which contains results on increased water stress by 2025 (instead of 2020) as computed by several climate models for five sub-regions in Africa (and other continents). In our communication with Nigel Arnell, concerning the addition of numbers in the table (which gives a range of between 75 and 240 million people, consistent with the rounded numbers in the statement), he informed us that he had recently noticed that, in this case, the addition of ranges was an unreliable method for determining the total range. For the HADCM3 model the table contains a range of model outcomes from three model runs of the SRES A2 scenario and a range from two model runs of the B2 scenario, but the low and high values did not originate from the same model run. Simply adding up these extreme values from the model outcomes for the sub-regions may thus overstate the range. It must be added here that, although this type of error is well known to occur in scientific practice, the IPCC authors, including Nigel Arnell who was a Lead Author of a sectoral chapter (Chapter 3) of the Working Group II Report, considered the straightforward addition of those extreme values appropriate at the time. With the benefit of hindsight, a relatively simple reanalysis of the data would have brought this error to light. According to Arnell (2010), the correct range that results from adding the right numbers for Africa would be: 88 to 217 million people. This implies that ‘between 75 and 250 million’ should have read ‘between 90 and 220 million’ when rounded to the nearest 10 (E1b). However, since the difference is only small and uncertainties are considerable, and because the original paper did not provide the details that would have allowed correct computing of this range, we consider this to be a minor inaccuracy.

By 2020, in some countries, yields from rain-fed agriculture could be reduced by up to 50%. (C3,C5,C7; major) This statement is not directly a statement on climate change, but on climate variability: in individual years, droughts can cause up to 50% in yield reductions. The implicit message here is that when droughts would become more frequent due to climate change, more years with up to 50% in yield reductions would occur. The statement could easily mislead readers into thinking that average annual yields could be reduced by up to 50% due to climate change. In the Summary for Policymakers of the Working Group II Report, the paragraph that contains this 

# 5Assessing an IPCC assessment 54 statement starts with a sentence introducing the notion of climate variability, which puts the statement more into context. We have several major comments to make about its provenance. The statement was based on a report by the International Institute for Sustainable Development (IISD) (Agoumi, 2003) that referred to studies performed within the framework of a UNEP–GEF project that we were unable to trace (C5) and to the Initial National Communications (INCs) of Morocco, Algeria and Tunisia. However, only the INC of Morocco (Kingdom of Morocco, 2001) reported a decline in yields for 2020: ‘a 50% reduction in cereal yields in Morocco might occur in dry years and 10% in normal years’. The IPCC authors explained to us that present-day climates and projected future climate change in the three countries are very similar – with a reference to Figure 11.2 in the W orking Group I Report of AR4 – and also that only cereals are grown without irrigation. Furthermore, using information from the European statistical information service (EUROSTAT) the authors made plausible that, due to current climate variability, the yields in Algeria, Morocco and Tunisia have been varying annually, including yield reductions of nearly 70% in individual years, in the period between 2000 and 2006. In hindsight, these additional explanations could have provided further foundations for the statement, had they been included in Chapter 9 (C3).

The Initial National Communication of Morocco indicated that this statement was based on ‘The Study of Morocco’s Vulnerability to CC Effects’ (MATUHE, 2001). However, we were unable to trace this study, nor could it be provided by the IPCC authors (C5). Although the IPCC does not formally require that authors check references of references of references, it would have been very helpful in this case;the IPCC requires that the chapter team reviews the quality and validity of nonpeer-reviewed sources (C5). As it stands, the statement cannot be traced back to any original scientific research.

Finally, the statement seemed to have been added to the main text of Chapter 9 after the Second Order Draft had been reviewed, but we were unable to find the particular substantive comment(s) made in this review that could have led to the addition of this statement (C7).

Agricultural production, including access to food, in many African countries is projected to be severely compromised. This would further adversely affect food security and exacerbate malnutrition. This statement – which together with the previous statement comprises the second bulleted statement on Africa in Table SPM.2 (see Annex A) – is fully supported by the underlying material.

Towards the end of the 21st century, projected sea level rise will affect low-lying coastal areas with large populations. The cost of adaptation could amount to at least 5 to 10% of Gross Domestic Product (GDP). (C3; minor) The first sentence of this statement is fully supported by the underlying material. We have a comment to make about the second sentence of the statement, which was based on a table in a study by Niang-Diop (2005). The text does not explicate how the experts derived the 5 to 10% in adaptation costs from this study, since it covered a range from 0.5 to 156.2% (C3). The IPCC authors explained to us that they used 5 of 12 studies summarised in the table in Niang-Diop (2005). Those thatAfrica 55 scored below the range of 5 to 10% were either countries that did an assessment not covering the entire country (Cameroon: 0.5%; Gambia: 1%), or those with a very short coastline (Democratic Republic of the Congo: 4.1%), or relatively rich countries (Nigeria: 2.2%). Those that scored well above this range were either very small island states (Comoros: 49.7%) or countries for which the costs had been overestimated (Mauritania: 156.2%). This answer explains the 5 to 10% range. W e believe it would have been helpful if this explanation had been provided in Chapter 9.

By 2080, an increase of 5 to 8% of arid and semi-arid land in Africa is projected under a range of climate scenarios. This statement is fully supported by the underlying material.

5.2 Additional findings

The population at risk of increased water stress in Africa is projected to be between 75–250 million and 350–600 million people by the 2020s and 2050s, respectively (E1; minor). This statement is part of the Executive Summary and is also included in T able TS.4 on page 67 of the Technical Summary. The range of 75 to 250 million people is discussed above in Section 5.1 (first statement). From Nigel Arnell, the author of the underlying 2004 study, we learned that also the second range (350 to 600 million people in the 2050s) was incorrectly computed from Table 11 in his study; the extreme values for the sub-regions cannot be added together. This remained unnoticed during the writing of the Working Group II Report. According to Arnell, in hindsight, the correct range should have been 350 to 490 million (E1b). Given the large uncertainties, we consider this a minor error. This inaccuracy has no impact on the IPCC conclusions in the various Summaries for Policymakers.

In South Africa, fisheries could be affected [...]. Recent simulations based on the NCAR GCM under a doubling of carbon dioxide indicate that extreme wind and turbulence could decrease productivity by 50-60%, while turbulence will probably bring about a 10% decline in productivity in the spawning grounds and an increase of 3% in the main feeding grounds. (E1; major) This statement originates from Section 9.4.4 of the main text, and was based on a study by Clark et al. (2003), which indicated that the changes in the frequency of extreme wind and turbulence events is expected to decrease quite substantially, by 50 to 60%; it did not refer to a similar decrease in fishery productivity (E1a). Also, in the latter part of the sentence it should have been the changes in mean turbulence that decline by 10% and increase by 3%, and not the productivity. The IPCC authors initially indicated that the reference to ‘productivity’ should have been deleted and ‘for anchovy’ should have been added at the end. In that case, the statement would
read as follows:

Recent simulations based on the NCAR GCM under a doubling of carbon dioxide indicate that the frequency of extreme wind and turbulence events could decrease by 50 to 60%, while mean turbulence will probably decline by 10% in the spawning grounds and increase by 3% in the main feeding grounds for anchovy.Assessing an IPCC assessment 56 From the IPCC authors we received the following proposal for an erratum:

Recent simulations based on the NCAR GCM under a doubling of carbon dioxide indicate that changes in extreme wind and turbulence could decrease potential spawning and increase potential larval feeding of anchovy.

This major error has no impact on the IPCC conclusions in the various Summaries for Policymakers.

5.3 Findings from the PBL registration website

Agricultural intensification and/or expansion into marginal lands can trigger additional conflicts, cause crop failure, exacerbate environmental degradation (e.g., Olsson et al., 2005). (minor) This statement can be found on page 441, Section 9.2.2.4, of the main text. One of the reactions we received on the registration website indicated that the reference to Olsson et al. (2005) in relation to this particular statement is not the most relevant one, because the article does not focus primarily on the intensification and/or expansion of agriculture and its consequences. Olsson et al. (2005) is about changes in precipitation patterns that have resulted in the greening of the Sahel over the past 10 years (5 years at the time of writing). We believe that other, more appropriate references would have been available for this statement, but we have not assigned this less appropriate referencing to one of our qualification categories. It has no impact on the IPCC conclusions in the various Summaries for Policymakers.

In the Sahelian region of Africa, warmer and drier conditions have led to a reduced length of growing season with detrimental effects on crops. (major) This statement was taken from page 9 of the Summary for Policymakers of the Working Group II Report. One of the reactions we received indicated that this summary statement did not adequately convey the nuances contained in Chapter

9. **1)** In response to that reaction, we wish to make a major comment on this statement. It was based on two sentences in Section 1.3.6.1 of the main text: ‘In Sahelian countries, increasing temperature in combination with rainfall reduction has led to a reduced length of vegetative period, no longer allowing present varieties to complete their cycle (Ben Mohamed et al., 2002).’ and: ‘In the case of the Sahel region of Africa, warmer and drier conditions have served as a catalyst for a number of other factors that have accelerated a decline in groundnut production (Van Duivenbooden et al., 2002).’ Even though the statement does not directly refer to the regional chapters, it is a statement in the Summary for Policymakers on a regional impact.

Although, the Summary for Policymakers’ text is consistent with the formulations in the main text, we believe that the subject of both case studies, which deal with millet, groundnut and cowpeas in Niger, cannot be generalised to describe ‘the Sahelian region’ and to ‘effects on crops’ . In reaction to this, the IPCC authors replied
that ‘ the Sahelian region’ is not the same as ‘ the entire Sahelian region’ and ‘ effects on

1) T his reaction and the previous one were received as one combined reaction; in our report we treat them separately, since they pertain to different statements.Africa 57 crops’ is not the same as ‘ effects on all crops ’. In addition, the authors have pointed to the heading of this section (p. 9, Working Group II Summary for Policymakers) that says these are ‘examples’ and ‘have not yet become established trends’. The authors further commented that these are obviously not universal statements. In their opinion, it would be unnecessary for every statement on certain parts of a place or region to say ‘parts of’, and similarly about statements on systems to say ‘some of’, because in their view this non-universality is implicit.

Indeed, the disclaimer in the header does explain that these are examples and not established trends, but to us it is not obvious that the statement therefore refers

to some crops in a part of, for instance, the Sahelian region, and we believe it would have been better if these specifications had been added. This major comment has no consequences for the IPCC conclusions in the various Summaries for Policymakers.

References
Agoumi, A., 2003. Vulnerability of North African countries to climatic changes: adaptation and

implementation strategies for climatic change. Developing Perspectives on Climate Change: Issues and Analysis from Developing Countries and Countries with Economies in T ransition. IISD/ Climate Change Knowledge Network, 14 pp. http://www.cckn.net//pdf/north_africa.pdf Arnell, N.W., 2004. Climate change and global water resources: SRES emissions and socio-economic

scenarios. Global Environ. Change, 14, 31-52.
Arnell, N.W., 2010. Personal communication. Ben Mohamed A., Duivenbooden, N. van, Abdoussalam, S., 2002, Impact of climate change on

agricultural production in the Sahel- part I. methodological approach and case study for millet in Niger, Climatic Change, 54, pg 327-348. Clark, B.M., S. Young and A. Richardson, 2003. Likely effects of global climate change on the purse

seine fishery for Cape anchovy Engraulis capensis off the west coast of Southern Africa (SE Atlantic). http://swfsc.noaa.gov/uploadedFiles/Education/lasker_events.doc. Van Drunen, M.A., R. Lasage and C. Dorland, Eds., 2005. Climate Change in Developing Countries: An

Overview of Study Results from the Netherlands Climate Change Studies Assistance Programme. CABI Publishing,Amsterdam, 320 pp. Duivenbooden, N. van, Abdoussalam, S., Ben Mohamed A., 2002. Impact of climate change on

agricultural production in the Sahel- part II> Cae study for groundnut and cowpea in Niger, Climatic Change, 54, pg 349-368. Kingdom of Morocco, 2001. Initial National Communication on the United Nations Framework

Convention on Climate Change, Ministry of Land-use Planning, Housing, and the Environment, October. http://unfccc.int/resource/docs/natc/mornc1e.pdf MATUHE, 2001. Etude de Vulnérabilité du Royaume du Maroc face aux impacts des changements

climatiques. Ministère chargé de l’Aménagement du Territoire, de l’Urbanisme, del’Habitat et de l’Environnement, septembre 2001. Niang-Diop, I., 2005. Impacts of climate change on the coastal zones of Africa. Coastal Zones in

Sub-Saharan Africa: A Scientific Review of the Priority Issues Influencing Sustainability and Vulnerability in Coastal Communities, IOC, Ed., IOCWorkshop Report No. 186. ICAM Dossier No. 4,

27-33. Olsson, L., Eklundh, L., Ardö, J., 2005. A recent greening of the Sahel—trends, patterns and potential

causes. Journal of Arid Environments, 63, pg 556–566. Otter, L., Olago, D.O. and Niang, I. (eds), 2007. Global Change Processes and Impacts in Africa: A

Synthesis. START/East African Educational Publishers, Nairobi. 346 pages. September 2007.Assessing an IPCC assessment 58Asia 59 

# Asia

6.1 Analysis of statements in T able SPM.2 of the Synthesis Report

By the 2050s, freshwater availability in Central, South, East and South-East Asia, particularly in large river basins, is projected to decrease. (C2,C3; major) This statement originates from the Executive Summary of Chapter 10 of the main text. Section 10.4.2 of the main text does not clearly reveal projections indicating an absolute decrease in freshwater availability in all indicated areas by the 2050s. There is no indication of how freshwater availability should be interpreted in this statement, that is, whether it concerns absolute flows or per-capita availability, which makes it hard to establish the line of reasoning that was followed, since a combination of evidence using different metrics has been cited in the chapter (C3). In the main chapter text, there are various referenced projections of ‘water stress’ (availability relative to demand) for South and Southeast Asia by 2050, projections of reduced run-off in mid-China by the end of the 21st century, and reductions in
the per-capita availability of fresh water in India by 2050. There is a reference to a region in Central Asia (Hindu-Kush) and its dependence on meltwater from glaciers, but no long-term projections are given. However, according to model calculations, the runoff projections for 2100 in the sectoral Chapter 3 (‘Freshwater resources and their management’) show an increase in annual runoff in South, East and Southeast Asia (Figure 3.8, on page 201). This, of course, does not contradict possible severe water stress during dry seasons, but neither does it support the generalisation that led to the statement in the Summary for Policymakers (C2)

.
Coastal areas, especially heavily populated megadelta regions in South, East and South-East Asia, will be at greatest risk due to increased flooding from the sea and, in some megadeltas, flooding from the rivers. This statement is fully supported by the underlying material.

Climate change is projected to compound the pressures on natural resources and the environment associated with rapid urbanisation, industrialisation and economic development. This statement is fully supported by the underlying material.

Endemic morbidity and mortality due to diarrhoeal disease primarily associated with floods and droughts are expected to rise in East, South and South-East Asia due to projected changes in the hydrological cycle. (C3; minor) This statement is fully supported by the underlying material. 

# 6Assessing an IPCC assessment 60 6.2 Additional findings

The already discovered error on the melting rate of glaciers (Section 10.6.2 of the main text) is treated in Annex B.

T he frequency and extent of forest fires in northern Asia is expected to increase in the future due to climate change and extreme weather events that could likely limit forest expansion. (C3,C6; major) See the Executive Summary, page 471, the Technical Summary, pages 49, and Box TS.6 on page 60. We have a comment to make on this statement. In the underlying text, nuances and uncertainties are clearly described: ‘The observations in the past 20 years show that the increasing intensity and spread of forest fires in North and South-East Asia were largely related to rises in temperature and declines in precipitation in combination with increasing intensity of land uses (see Section 10.2.4.4).Whether this trend will persist in the future or not is difficult to ascertain in view of the limited literature on how the frequency and severity of forest and brush fires will likely respond to expected increase in temperature and precipitation in North and South-East Asia (see Section 10.3.1).’ In the text, the expected future increase was solely underpinned by a reference to a single conference paper, which was based on a speech by the Vice-Minister of the Ministry of the Russian Federation for Civil Defence and did not include any references (Vorobyov, 2004). Given the response by the authors that the statement is an adequate reflection of other literature not listed in the chapter, we concluded that the statement is an insufficiently transparent expert judgment (C3) and that the referencing was unnecessarily relying on grey literature (C6). The IPCC authors explained to us the policy relevance of including this statement in the Executive Summary of the chapter and in the Technical Summary. Even though the research in this area was in an early stage and the modelling of future impacts was limited in 2006, the authors considered this subject to be sufficiently important because of its potential scale of effect. The statement was awarded only medium confidence (having an even chance of being correct), but given the use of the word ‘likely’, we think it could have been worded more carefully. These major comments have no consequences for the IPCC conclusions in the various Summaries for Policymakers.

Recent risk analysis of coral reef suggests that between 24% and 30% of the reefs in Asia are likely to be lost during the next 10 years and 30 years, respectively. (E2; minor) For this statement, see the Executive Summary, page 471, and the T echnical Summary, Box TS.6 on page 59. We identified an inaccurate reference affecting this statement. The statement cannot be deduced from the figures presented in the included table (Table 10.6). Also, the underlying reference (Wilkinson, 2004) does not provide information to support the statement. The IPCC authors indicated that (Wilkinson, 2004) was cited incorrectly; this should have been (Wilkinson, 2000) (E2). This minor inaccuracy has no impact on the IPCC conclusions in the various Summaries for Policymakers.Asia 61 The Third Assessment Report (TAR) predicted that the area-averaged annual mean warming would be about 3°C in the decade of the 2050s and about 5°C in the decade of the 2080s over the land regions of Asia as a result of future increases in atmospheric concentration of greenhouse gases (Lal et al., 2001a). (minor). For this statement, see Section 10.1.1 of the main text, page 471. W e have one minor comment to make on this statement (which could not be placed in one of the categories). In the First Order Draft, the statement used the word ‘reported’ instead of ‘predicted, and one of the comments in the review of the First Order Draft was that ‘The TAR reported that...’ should be changed to ‘The TAR predicted that...’ or ‘The TAR forecasted that...’. This comment was responded to, saying that ‘IPCC Assessment Reports do not predict or forecast – they only report’, however, the sentence was changed nevertheless to ‘The TAR predicted that...’. It may be useful to note that in the IPCC reports, normally, the verb ‘to project’ is used instead of ‘to predict’ to denote the fact that a statement is made relative to a particular scenario. A comment on the Second Order Draft, made by the Republic of Korea, was to ‘specify the SRES scenario used for this assessment statement’. This comment was accepted according to the authors’ response (‘Appropriate revisions and editing made’), but no changes were implemented in the final texts. This minor comment has no consequences for the IPCC conclusions in the various Summaries for Policymakers.

References
Vorobyov,Y., 2004. Climate change and disasters in Russia. Proc. World Climate Change Conference,

Moscow, Y. Izrael, G. Gruza, S. Semenov and I. Nazarov, Eds., Institute of Global Climate and Ecology, Moscow, 293-298. Wilkinson, C., Ed., 2000. Status of Coral Reefs of the World: 2000.Australian Institute of Marine

Science, Townsville, 363 pp. Wilkinson, C., Ed., 2004. Status of Coral Reefs of the World: 2004, Volume 1. Australian Institute of

Marine Science, Townsville, 302 pp.Assessing an IPCC assessment 62Australia and New Zealand 63 

# Australia and New Zealand

7.1 Analysis of statements in T able SPM.2 of the Synthesis Report

By 2020, significant loss of biodiversity is projected to occur in some ecologically rich sites, including the Great Barrier Reef and Queensland Wet Tropics. (E2,C3; minor) In the text of Chapter 11 of the Working Group II Report, which underpins the statement, we found an inaccurate reference. Biodiversity loss on the Great Barrier Reef is covered by Table 11.6, in which is stated, for 2020: ‘Bleaching and damage to the Great Barrier Reef equivalent to that in 1998 and 2002 in up to 50% of years. (Berkelmans et al., 2004; Crimp et al., 2004) and 60% of the Great Barrier Reef regularly bleached (Jones, 2004b).’ The references for the first and second statement are not correct (E2). The IPCC authors indicated that the correct reference for both statements is Sheehan et al. (2006).

Furthermore, we were unable to derive the statements in Table 11.6 from the provided, correct reference (Sheehan et al, 2006). The reference relates impacts to temperature changes, not to specific years, which in this case would have been 2020. Implicitly, the IPCC authors converted warming into years by using T able 11.4, that is, a near-coast warming in the range of 0.1 to 1.0 °C by 2020. As also confirmed by the IPCC authors, this conversion should have been explicitly stated in the caption of Table 11.6 (C3). We consider this to be a minor comment.

By 2030, water security problems are projected to intensify in southern and eastern Australia and, in New Zealand, in Northland and some eastern regions. This statement is fully supported by the underlying material.

By 2030, production from agriculture and forestry is projected to decline over much of southern and eastern Australia, and over parts of eastern New Zealand, due to increased drought and fire. However, in New Zealand, initial benefits are projected in some other regions. This statement is fully supported by the underlying material.

By 2050, ongoing coastal development and population growth in some areas of Australia and New Zealand are projected to exacerbate risks from sea level rise and increases in the severity and frequency of storms and coastal flooding. This statement is fully supported by the underlying material. 

# 7Assessing an IPCC assessment 64 7.2 Additional findings

3,000 to 5,000 more heat related deaths per year. (C1; major) This example of regional impact was taken from Table TS.4 of the Technical Summary. We have a major comment to make on this statement.

From Table TS.4 readers may mistakenly conclude that 3,000 to 5,000 more heatrelated deaths could be directly attributed to temperature change (as is the case with the ‘additional people suffering from increased water stress’ also shown in the table). However, the largest part of the 3,000–5,000 range is dependent on expected changes in population sizes and age distributions within cities (C1). With the other drivers remaining constant, temperature change would only be responsible for 300 to 900 more heat-related deaths (this can be calculated from Table 2 in McMichael et al., 2003). This is only a small part of the 3,000 to 5,000 more heat-related deaths mentioned in the table. The underpinning for the statement in Table TS.4 is given in Section 11.4.11 on health. The relevant sentence
is: ‘Assuming no planned adaptation, the number of deaths is likely to rise from 1,115/ yr at present in Adelaide, Melbourne, Perth, Sydney and Brisbane to 2,300 to 2,500/yr by 2020, and 4,300 to 6,300/yr by 2050, for all SRES scenarios, including demographic change (McMichael et al., 2003)’. The stated numbers include both demographic and temperature effects. In the chapter is not mentioned that the effect of temperature change alone is relatively small, compared to the combined effect of temperature change and changes in population sizes and age distributions.

We believe that the total number of heat-related deaths by 2050 (3,000–5,000) is a number that IPCC should publish, as policymakers should be made aware of all heat-related deaths, not just those due to climate change. However, it is also our opinion that an IPCC report on climate-change impacts should show the different components including the climate-change related component, if available or traceable. It is no easy matter to establish a climate-change related component, but one way of doing this would be to make a comparison between changes in the demographic situation of 2050 with climate change and without. A simple calculation performed by the PBL resulted in 1,000 to 3,000 additional heat-related death per year, as a consequence of climate change, which is significantly less than the 3,000 to 5,000 range given in Table TS.4. This major comment has no consequences for the IPCC conclusions in the various Summaries for Policymakers.

Heat waves and fires are virtually certain to increase in intensity and frequency. (C4; minor) This statement originates from the Executive Summary. We have a minor comment to make on this statement. In our opinion, there is an inconsistency between the level of likelihood in the Executive Summary and in the chapter text. The relevant chapter text in Section 11.3.1 repeatedly uses ‘likely’ for increases related to fires, and it not clear why ‘likely’ was subsequently translated to ‘virtually certain’ in the Executive Summary (C4). The IPCC authors have concluded, after consultation with authors of relevant references, that the likelihoods in the chapter may have been understated. This minor comment has no consequences for the IPCC conclusions in the various Summaries for Policymakers.Australia and New Zealand 65 Since 1950 there has been a 0.3 to 0.7 °C warming in the region. (E1; minor) This statement, taken from the Technical Summary on page 50, is different from the Executive Summary, which states: ‘ Since 1950, there has been 0.4 to 0.7°C warming,... ’. This is a minor inaccuracy (E1a). The authors confirmed this to be a typographical error in the Technical Summary; 0.4 °C is the correct number. This minor inaccuracy has no impact on the IPCC conclusions in the various Summaries for Policymakers.

References
Berkelmans, R., G. De’ath, S. Kininmonth and W.J. Skirving, 2004. A comparison of the 1998 and 2002

coral bleaching events of the Great Barrier Reef: spatial correlation, patterns and predictions. Coral Reefs, 23, 74-83. Crimp, S., J. Balston, A. Ash, L. Anderson-Berry, T. Done, R. Greiner, D. Jones, R.N., 2004b. Managing

climate change risks. The Benefits of Climate Policies: Analytical and Framework Issues, J. Corfee Morlot and S. Agrawala, Eds., OECD, 251-297.

h ttp://www.oecd.org/document/35/0,2340,en_ 2649_34361_ 34086819_1_1_1_1,00.html. McMichael, A., R. Woodruff, P. Whetton, K. Hennessy, N. Nicholls, S. Hales, A. Woodward and

T. Kjellstrom, 2003. Human Health and Climate Change in Oceania: A Risk Assessment 2002. Commonwealth Department of Health and Ageing, 128 pp. http://www.health.gov.au/internet/ main/publishing.nsf/Content/2D4037B384BC05F6CA256F1900042840/$File/env_climate.pdf Sheehan, P., R. Jones, A. Jolley, B.L. Preston, M. Clark, P.J. Durack, S. Islam, F.Sun, and P.H.

Whetton, 2006. Climate Change and the Global Knowledge Economy: An Immediate Challenge. CSES Climate Change Working Paper No. 11. Victoria University Centre for Strategic Economic Studies, Melbourne, 35 pp.

h ttp://www.cfses.com/documents/climate/11_Sheehan_Jones_et_al_Climate_Change_.pdfAssessing an IPCC assessment 66Europe 67 

# Europe

8.1 Analysis of statements in T able SPM.2 of the Synthesis Report

Climate change is expected to magnify regional differences in Europe’s natural resources and assets. Negative impacts will include increased risk of inland flash floods and more frequent coastal flooding and increased erosion (due to storminess and sea level rise). This statement is fully supported by the underlying material.

Mountainous areas will face glacier retreat, reduced snow cover and winter tourism, and extensive species losses (in some areas up to 60% under high emissions scenarios by 2080). (C3; minor) We have one minor comment to make on this statement (which is also included, verbatim, in the Summary for Policymakers of the Working Group II Report). The Executive Summary only addresses extensive species loss in ‘mountain plant communities, which face a loss in species of up to 60% under high emission scenarios’, but does not mention mountain fauna , nor does Thuiller et al. (2005), the publication on which the example was based. Thus, the example given in the statement in the Summary for Policymakers (‘in some areas up to 60%...’) was not fully derived from the text in the Executive Summary or from the main text of Chapter 12. In addition, it is not transparent how the authors have underpinned the example (C3). After an additional literature search on our part, it appeared that there was a reference that the authors could have used for underpinning the example. A report by the Potsdam Institute for Climate Impact Research (PIK) (Schröter et al., 2004) that was not included in the references of the chapter, explicitly presented quantitative results which refer to species loss in mountain flora and fauna of up to 60% (Figure 25 in Schröter et al., 2004). W e judged this omission of a reference and of a more traceable line of reasoning from the chapter to the Summary for Policymakers to be of minor effect.

In southern Europe, climate change is projected to worsen conditions (high temperatures and drought) in a region already vulnerable to climate variability, and to reduce water availability, hydropower potential, summer tourism and, in general, crop productivity. This statement is fully supported by the underlying material.

Climate change is also projected to increase the health risks due to heat waves and the frequency of wildfires. This statement is fully supported by the underlying material. 

# 8Assessing an IPCC assessment 68 8.2 Additional findings

The already discovered error on the Dutch land area below sea level (in the main text of Chapter 12, on page 547) is discussed in Annex C.

A daptation to climate change is likely to benefit from experiences gained inreactions to extreme climate events, by specifically implementing proactive climatechange risk management adaptation plans (very high confidence). (C4; minor) This statement appears in the Technical Summary (page 53) and in the Summary for Policymakers of the Working Group II Report (page 14). However, the exact same statement received a lower confidence level (‘high confidence’) in the Executive Summary (page 543) (C4). The IPCC authors replied that the correct level of confidence should have been ‘very high confidence’ This was an editorial mistake; IPCC has indicated that it will issue an erratum. This minor inconsistency has no impact on the IPCC conclusions in the Summary for Policymakers of the Synthesis Report.

In southern Europe, general decreases in yield (e.g., legumes -30 to +5%; sunflower -12 to +3% and tuber crops -14 to +7% by 2050) and increases in water demand (e.g., for maize +2 to +4% and potato +6 to +10% by 2050) are expected for spring sown crops (Giannokopoulos et al., 2005; Audsley et al., 2006). (E1; minor) The statement, which was taken from the main text of Chapter 12 (page 555), is slightly inaccurate (E1a). In Giannokopoulos et al. (2005), four Mediterranean regions have been distinguished. Only two are located in Europe (those labelled ‘N-W’ and ‘N-E’). The numbers (ranges) presented in the IPCC Working Group II Report refer to all four regions, including North Africa, Jordan, Libya and Egypt, and should not have been reported in the chapter on Europe. The correct ranges for the European regions should read: Legumes (beans/soybean)

- 14 % to +1 %; C3 (sunflower)

- 12% to + 1%; and Tubers (potato)

- 9% to +8 %). The IPCC authors have told us that this finding will lead to an IPCC erratum. These minor inaccuracies have no impact on the IPCC conclusions in the various Summaries for Policymakers.

Annual average runoff is projected to increase in northern Europe (north of 47°N) by approximately 5 to 15% up to the 2020s and 9 to 22% up to the 2070s , for the SRES A2 and B2 scenarios and climate scenarios from two different climate models (Alcamo et al., 2007) (Figure 12.1). Meanwhile, in southern Europe (south of 47°N), runoff decreases by 0 to 23% up to the 2020s and by 6 to 36% up to the 2070s (for the same set of assumptions). (C3; minor) We have a minor comment to make on this statement. The quantitative information in this statement (Chapter 12, page 549), as well as in Figure 12.1, could not be found in the specific paper cited (Alcamo et al., 2007) (C3). In a reaction, the IPCC authors responded by saying that – for the accomplishment of the most up-to-date review of scientific knowledge – they sometimes also used underlying data that had formed the basis of the results in the peer-reviewed publications; they obtained this background data by contacting the authors of these publications and asking for specific figures. Our comment does not imply that such information should not be used, but only that the provenance of this information could have been made more transparent. This minor comment has no consequences for the IPCC conclusions in the various Summaries for Policymakers.Europe 69 Climate-related increases in crop yields are expected mainly in northern Europe, e.g., wheat: +2 to +9% by 2020, +8 to +25% by 2050, +10 to +30% by 2080 (Alexandrov et al., 2002; Ewert et al., 2005; Audsley et al., 2006; Olesen et al., 2007), and sugar beet +14 to +20% until the 2050s in England and Wales (Richter and Semenov, 2005),... The impacts on autumn sown crops are more geographically variable; yield is expected to strongly decrease in most southern areas, and increase in northern or cooler areas (e.g., wheat: +3 to +4% by 2020, -8 to +22% by 2050, -15 to +32% by 2080 ) (Santos et al., 2002; Giannakopoulos et al., 2005; Audsley et al., 2006; Olesen et al., 2007). (C3; minor) We have a minor comment to make on this statement. It is not transparent how the quantitative information on wheat yield and sugar beet as presented in this statement (Chapter 12, page 555) was derived from the numbers given in the cited references (C3). The IPCC authors explained to us in more detail how these numbers had been established on the basis of the cited references. Although we have no reason to doubt the correct underpinning of these results, we do think that the provenance of presented quantitative information in IPCC reports – if it cannot be derived from the cited references – should be clearly explained for reasons of transparency and traceability. This minor comment has no consequences for the IPCC conclusions in the various Summaries for Policymakers.

However, on balance health risks are very likely to increase (C3; major) We have a major comment to make on this statement, which was taken from the Executive Summary (page 543) and summarises the IPCC authors’ expert judgment on the balance between positive and negative effects of climate change on health. Supportive material is presented in the main text (see e.g. Section 12.4.11 on pages 557–558; Figure 12.3 on page 558; Table 12.1 on page 546, and Table 12.4 on page 565), which also provides a more regionally differentiated picture. However, a quantitative underpinning for the statement is lacking in the main text (C3).

The IPCC authors indicated to us that they weighed the positive and negative effects and this was their final assessment. Pages 557 to 558 of Chapter 12 give a detailed account of all impacts, and we found the negative impacts to be the norm, not the positive ones. However, although the list of negative effects is longer, no indication of their relative weights is given, which makes the expert judgment about the ‘balance’ in our view insufficiently transparent.

Although the statement was not included in the various Summaries for Policymakers, many readers may consider it to be highly policy relevant.

This major comment has no consequences for the IPCC conclusions in the various Summaries for Policymakers.Assessing an IPCC assessment 70 8.3 Findings from the PBL registration website

For the first time, wide-ranging impacts of changes in current climate have been documented: retreating glaciers, longer growing seasons, shift of species ranges, and health impacts due to a heat wave of unprecedented magnitude. (minor) We received a reaction to this statement which criticised the presenting of the 2003 heat wave as a consequence of climate change. In response to that reaction, we wish to make a minor comment on this statement (although it does not fit one of our qualification categories). The Working Group II Summary for Policymakers states, for Europe, on page 14: ‘For the first time, wide-ranging impacts of changes in current climate have been documented: retreating glaciers, longer growing seasons, shift of species ranges, and health impacts due to a heat wave of unprecedented magnitude. The observed changes described above are consistent with those projected for future climate change.’ This text, as well as its counterparts in Table TS4.2 of the Technical Summary (page 51) and in the Executive Summary (page 543), present the health impacts from the 2003 heat wave as an example of ‘wide-ranging impact of changes in current climate’. Thus, the text implicitly suggests that the 2003 heat wave is the result of recent climate change. However, one can never attribute a specific extreme weather event of the past – such as that particular heat wave –to changes in current climate. In fact, Schär and Jendritzky (2004) concluded: ‘The European heatwave of 2003: was it merely a rare meteorological event or a first glimpse of climate change to come? Probably both.’ The IPCC authors have indicated that the statement in the Summary for Policymakers does not contain a formal attribution – otherwise, the second sentence ‘the observed changes described above are consistent with those projected for future climate change’ would not have been necessary; the authors’ reaction is factually correct, but the wording remains suggestive in our view. Chapter 1 of the Working Group II Report contains statements that are more careful, assessing the evidence of observed increases in heat waves in Europe. This minor comment has no consequences for the Summary for Policymakers of the Synthesis Report.

References
Alcamo, J., M. Floerke and M. Maerker, 2007. Future long-term changes in global water resources

driven by socio-economic and climatic changes. Hydrological Sciences, 52, 247-275. Alexandrov, V., J. Eitzinger, V. Cajic and M. Oberforster, 2002. Potential impact of climate change on

selected agricultural crops in north-eastern Austria. Glob. Change Biol., 8, 372-389. Audsley, E., K.R. Pearn, C. Simota, G. Cojocaru, E. Koutsidou, M.D.A. Rounsevell, M. T rnka and V.

Alexandrov, 2006. What can scenario modelling tell us about future European scale agricultural land use, and what not? Environ. Sci. Pol., 9, 148-162. Ewert, F., M.D.A. Rounsevell, I. Reginster, M.J. Metzger and R. Leemans, 2005. Future scenarios of

European agricultural land use I. Estimating changes in crop productivity. Agr. Ecosyst. Environ.,
107, 101-116. Giannakopoulos, C., M. Bindi, M. Moriondo, P. LeSager and T. Tin, 2005. Climate Change Impacts

in the Mediterranean Resulting from a 2oC Global Temperature Rise. WWF report, Gland Switzerland. Accessed 01.10.2006 at http://assets.panda.org/downloads/medreportfinal8july05. pdf.Europe 71 Olesen, J.E., T.R. Carter, C.H. Díaz-Ambrona, S. Fronzek, T. Heidmann, T. Hickler, T. Holt, M.I.

Mínguez, P. Morales, J. Palutikhof, M. Quemada, M. Ruiz-Ramos, G. Rubaek, F. Sau, B. Smith and M. Sykes, 2007: Uncertainties in projected impacts of climate change on European agriculture and terrestrial ecosystems based on scenarios from regional climate models. Climatic Change, 81,
S123-S143. Richter, G. and M. Semenov, 2005: Re-assessing drought risks for UK corps using UKCIP02 climate

change scenarios. Final report of Defra project CC0368. Schär, C. and G. Jendritzky, 2004: Climate change: hot news from summer 2003. Nature, 432, 559-

560. Schröter, D., et al., 2004. Advanced Terrestrial Ecosystem Analysis and Modelling (ATEAM): Final

Report 2004. Section 5 and 6 and Annex 1 to 6. Accessed 22-4-2010 at http://www.pik-potsdam.de/ ateam/ateam_final_report_sections_5_to_6.pdf. Santos, F.D., K. Forbes, and R. Moita, Eds., 2002: Climate change in Portugal: scenarios, impacts and

adaptation measures. SIAM project report, Gradiva, Lisbon, 456 pp. Thuiller, W., S. Lavorel, M.B. Araújo, M.T. Sykes and I.C. Prentice, 2005: Climate change threats plant

diversity in Europe. Proc. Natl. Acad. Sci. U.S.A., 102, 8245-8250.Assessing an IPCC assessment 72Latin America 73 

# Latin America

9.1 Analysis of statements in T able SPM.2 of the Synthesis report

By mid century, increases in temperature and associated decreases in soil water are projected to lead to gradual replacement of tropical forest by savanna in eastern Amazonia. Semi-arid vegetation will tend to be replaced by arid-land vegetation. This statement is fully supported by the underlying material.

There is a risk of significant biodiversity loss through species extinction in many areas of tropical Latin America. This statement is fully supported by the underlying material.

Productivity of some important crops is projected to decrease (a) and livestock productivity to decline (b), with adverse consequences for food security. In temperate zones, soybean yields are projected to increase (c). Overall, the number of people at risk of hunger is projected to increase (d). (C2; major) We subdivided this statement by inserting (a) to (d) between issues. We have one comment to make on part (b).

Part (a) ( Productivity of some important crops is projected to decrease ) This is fully supported by the underlying material.

Part (b) ( Livestock productivity is projected to decline ). The ‘livestock’ appears to be limited to ‘cattle’ and this part of the statement is underpinned by references to studies on Bolivia and central Argentina. The generalisation to other livestock, which would include pigs, poultry, small ruminants and other animals (together also comprising a substantial fraction of livestock production in Latin America) and to the whole of Latin America, is not underpinned in Chapter 13. We therefore believe that this is an ‘insufficiently founded generalisation’ (C2).

Statement (c) ( In temperate zones, soybean yields are projected to increase ). This is fully supported by the underlying material.

Statement (d) ( The number of people at risk of hunger is projected to increase ). This is fully supported by the underlying material.

Changes in precipitation patterns and the disappearance of glaciers are projected to significantly affect water availability for human consumption, agriculture and energy generation. This statement is fully supported by the underlying material. 

# 9Assessing an IPCC assessment 74 9.2 Additional findings

Table 13.6 Net increases in the number of people living in water-stressed watersheds. (E1; minor) Table 13.6 (on page 598 of Chapter 13) combines the absolute number of people that live under water-stressed conditions (for the non-climate cases, columns 2, 3 and 5) with the number of people that experience an increase in water stress due to climate change (Arnell, 2004). The table’s caption, however, suggests all numbers are increases, but this is not correct (E1a). This issue also had been recognised by the IPCC authors, who have indicated to us that they intend to have an erratum issued. Note that we consider this to be a minor inaccuracy and that we have no comment to make about the table’s content. This minor inaccuracy has no impact on the IPCC conclusions in the various Summaries for Policymakers.

...the frequency of occurrence of weather and climate extremes is likely to increase in the future; as is the frequency and intensity of hurricanes in the Caribbean Basin. (E1; minor) This statement was taken from the Executive Summary of Chapter 13, page 583, right column, second paragraph. The statement contains a minor inaccuracy. The Technical Summary of the Working Group I 2007 Report (p. 74) states: ‘A synthesis of the model results to date indicates, for a warmer future climate, increased peak wind intensities and increased mean and peak precipitation intensities in future tropical cyclones, with the possibility of a decrease in the number of relatively weak hurricanes, and increased numbers of intense hurricanes. However, the total number of tropical cyclones globally is projected to decrease’. We conclude that ‘frequency and intensity of hurricanes’ should have read ‘frequency and intensity of strong hurricanes’ (E1a). This minor inaccuracy has no impact on the IPCC conclusions in the various Summaries for Policymakers.

Up to 40% of the Amazonian forests could react drastically to even a slight reduction in precipitation; this means that the tropical vegetation, hydrology and climate system in South America could change very rapidly to another steady state. (C6; minor) We have a minor comment to make on this statement, which originates from Section 13.4.1 of Chapter 13 (page 596). The statement was based on Rowell and Moore (2000), which is a peer-reviewed report by the World Wide Fund for Nature and the International Union for Conservation of Nature (WWF/IUCN) on a global review of forest fires, and not a study on changes in vegetation due to climate change. That report, in turn, was mainly based on Nepstad et al. (1999) (in Nature ). In our opinion, both documents were not the most obvious choice of reference in this case, as their focus is on forest fires (and logging). More adequate peer-reviewed, scientific journal literature would have been available to support this statement, such as Cox et al. (2000; 2004) (C6). This minor comment has no consequences for the IPCC conclusions in the various Summaries for Policymakers.

...some plants will become locally extinct because the elevation range would not permit natural adaptation to temperature increase (FAO, 2002). (C5; minor) This statement originates from Section 13.4.1 of Chapter 13 (page 596, right column). We were unable to trace FAO (2002) and the URL in the references doesLatin America 75 not provide a link to the FAO website (C5). This issue also has been recognised by the IPCC authors, who have indicated that they intend to have an erratum issued. This minor comment has no consequences for the IPCC conclusions in the various Summaries for Policymakers.

...the incidence of the coffee leafminer (Perileucoptera coffeella) and the nematode Meloidogyne incognita are likely to increase in future in Brazil’s production area. The number of coffee leaf miner cycles could increase by 4%, 32% and 61% in 2020, 2050 and 2080, respectively, under the SRES A2 scenarios (Ghini, 2007). (C5; minor) This statement originates from Section 13.4.2 of Chapter 13 (page 597, right column). We have a minor comment to make on this statement. The reference to the paper by Ghini et al. (2007) was listed as having been submitted to the journal Climatic Change, but was never published in Climatic Change (C5). It was published

in Pesquisa Agropecuária Brasileira, Brasília 43 (2): 187-194, in 2008 (Ghini et al., 2008). We are of the opinion that referring to submitted papers that have not beenaccepted for publication is incorrect, since they have not yet completed the peerreview process. Moreover, the reference may end up to be untraceable, as was the case here. In our opinion, papers should at least be accepted for publication, or be in press, so that traceability is guaranteed. This minor comment has no consequences for the IPCC conclusions in the various Summaries for Policymakers.

A highly stressed condition is projected between 2015 and 2025 in the water availability in Colombia, affecting water supply and ecosystem functioning in the páramos (IDEAM, 2004), and very probably impacting on the availability of water supply for 60% of the population of Peru (Vásquez, 2004). The projected glacier retreat would also affect hydroelectricity generation in some countries, such as Colombia (IDEAM, 2004) and Peru; one of the more affected rivers would be the Mantaro, where an hydroelectric plant generates 40% of Peru’s electricity and provides the energy supply for 70% of the country’s industries, concentrated in Lima (UNMSM, 2004). (E2; minor) In this statement, taken from Section 13.4.3 of Chapter 13 (page 598, right column) we found that the references to UNMSM and Vasquez had been mixed up. The statement on the water supply in Peru is not supported by the Vásquez (2004) paper, but by UNMSM (2004). and the statement on hydro-electricity in the Mantaro river is supported by Vásquez (2004) (E2). This minor inaccuracy has no impact on the IPCC conclusions in the various Summaries for Policymakers.

References
Arnell, N.W., 2004. Climate change and global water resources: SRES emissions and socio-economic

scenarios. Global Environ. Change, 14, 31-52. Cox, P.M., R.A. Betts, C.D. Jones, S. Spall and I.J. Totterdell, 2000. Acceleration of global warming

due to carbon-cycle feedbacks in a coupled climate model. Nature, 408, 184-187. Cox, P.M., R.A. Betts, M. Collins, P. Harris, C. Huntingford, C.D. Jones, 2004. Amazon dieback under

climate-carbon cycle projections for the 21st century. Theor. Appl. Climatol. 78, 137–156. (doi:10. 1007/s00704-004-0049-4) FAO, 2002. El cambio climático y los bosques. Boletín electrónico Julio 2002, FAO, Rome. http://

www.ecosur.net/cambio_climatico_y_los_bosques.html. Ghini, R., E. Hamada, M.J. Pedro Júnior and J.A. Marengo, 2007. Climate change and coffee pests in

Brazil. Climatic Change, submitted.Assessing an IPCC assessment 76 Ghini, R., E. Hamada, M.J. Pedro Júnior and J.A. Marengo, 2008. Risk analysis of climate change on coffee nematodes and leaf miner in Brazil. Pesquisa Agropecuária Brasileira, Brasília 43 (2): 187-

194. Nepstad, D.C., A. Veríssimo, A. Alencar, C. Nobre, E. Lima, P. Lefebvre, P. Schlesinger, C. Potter,

P. Mountinho, E. Mendoza, M. Cochrane, V. Brooks, 1999. Larg e scale impoverishment of Amazonian forests by logging and fire, Nature, Vol. 398, 8 April. Rowell, A. and P.F. Moore, 2000. Global Review of Forest Fires. WWF/IUCN, Gland, Switzerland, 66

pp. http://data.iucn.org/dbtw-wpd/edocs/2000-047.pdf. UNMSM, 2004. Calor intenso y largas sequías. Especials, Perú. http://www. unmsm.edu.pe/Destacados/contenido.php?mver=11. Vásquez, O.C., 2004. El Fenómeno El Niño en Perú y Bolivia: Experiencias en Participación Local.

Memoria del Encuentro Binacional Experiencias de prevención de desastres y manejo de emergencias ante el Fenómeno El Niño, Chiclayo, Peru. ITDG, 209 pp.North America 77 

# North America

10.1 Analysis of statements in T able SPM.2 of the Synthesis Report

Warming in western mountains is projected to cause decreased snowpack, morewinter flooding and reduced summer flows, exacerbating competition for overallocated water resources. This statement is fully supported by the underlying material.

In the early decades of the century, moderate climate change is projected to increase aggregate yields of rain-fed agriculture by 5 to 20%, but with important variability among regions. Major challenges are projected for crops that are near the warm end of their suitable range or which depend on highly utilised water resources. (C3; minor) From Chapter 14 and the references therein we could not derive the reasoning that led to the 5 to 20% range of yield increases under moderate climate change. Of the six references named in the chapter, Adams et al. (2003), appears to be the main reference. Adams et al. report (in their Table I) changes in rain-fed yield, from –8% to +41% for 2060 (with adaptation) (C3). Adams et al. and other references (Thomson et al. 2005, Tsvetsinkaya et al., 2003) show there are regions with significant yield
decreases for specific rain-fed crops (5 to 25%), even if adaptation measures are adopted. It could be argued that this observation is covered by the phrase ‘but with important variability among regions’, but this could not be derived from the chapter text. The IPCC authors provided us with the reasoning that had been followed: for Adams et al. (2003), the projected yield changes for the four food crops considered (corn, soy, wheat, sorghum) are +7, +17, +13, and +5% respectively, according to the Regional Climate Model with adaptation. The whole idea of that paper was that the Regional Climate Model gave more credible projections than the Global Climate Model, so those are the numbers that the IPCC authors used. Furthermore, they confirmed that they highlighted variability due to, for example, different varieties of wheat with the statement that there is ‘important variability among regions’. They found it hard to see how they could have been more precise without reporting individual values from tables. We believe that the above explanation of the reasoning behind the 5 to 20% range could have been made more transparent with just a few extra lines of text. We wish to add here that the IPCC authors’ expert judgment had been based on a consideration of a vast body of literature and we have no reason to doubt the quality and correctness of the expert judgment. 

# 10Assessing an IPCC assessment 78 Cities that currently experience heat waves are expected to be further challenged by an increased number, intensity and duration of heat waves during the course of the century, with potential for adverse health impacts). This statement is fully supported by the underlying material.

Coastal communities and habitats will be increasingly stressed by climate change impacts interacting with development and pollution. This statement is fully supported by the underlying material.

10.2 Additional findings

Figures TS.15 and 14.1 on the economic damages from hurricanes. (minor) Figures TS.15 (Technical Summary, page 55) and 14.1 (Chapter 14, page 621) in the Working Group II Report illustrate the statement in the Technical Summary
that reads ‘Over the past several decades, economic damage from hurricanes in North America has increased over fourfold, due largely to an increase in the value of infrastructure at risk’ . We have a minor comment to make on the figures. Because of a shorter averaging period at the end of the time series (6 years instead of 10), the most recent increase was higher, compared with the increase that would have resulted from applying a uniform averaging period at the end of the series. However, a different choice of statistical method would not have affected the statement. Furthermore, we have found this statement to be fully supported by the underlying material. This minor comment has no consequences for the IPCC conclusions in the various Summaries for Policymakers.

References
Adams, R.M., B.A. McCarl and L.O. Mearns, 2003. The effects of spatial scale of climate scenarios on

economic assessments: An example from U.S. agriculture. Clim. Change, 60, 131-148. Brown, T. J., B. L. Hall and A. L.Westerling, 2004. The impact of twenty-first century climate change

on wildland fire danger in the western United States:An applications perspective. Clim. Change,
62, 365-388. Flannigan, M.D., K.A. Logan, B. D.Amiro,W. R. Skinner and B. J. Stocks, 2004. Future area burned in

Canada. Clim. Change, 72, 1-16. Thomson, A.M., R.A. Brown, N.J. Rosenberg, R.C. Izaurralde and V. Benson, 2005. Climate change

impacts for the conterminous USA:An integrated assessment. Part 3:Dryland production of grain and forage crops. Clim. Change, 69, 43-65. Tsvetsinskaya, E.A., L.O. Mearns, T. Mavromatis, W. Gao, L. McDaniel and M.W. Downton, 2003. The

effect of spatial scale of climatic change scenarios on simulated maize, winter wheat, and rice production in the southeastern United States. Clim. Change, 60, 37-72.Polar regions (Arctic and Antarctic) 79 

# Polar regions (Arctic and Antarctic)

11.1 Analysis of statements in T able SPM.2 of the Synthesis Report

The main projected biophysical effects are reductions in thickness and extent of glaciers, ice sheets and sea ice, and changes in natural ecosystems with detrimental effects on many organisms including migratory birds, mammals and higher predators. This statement is fully supported by the underlying material.

For human communities in the Arctic, impacts, particularly those resulting from changing snow and ice conditions are projected to be mixed. This statement is fully supported by the underlying material.

Detrimental impacts would include those on infrastructure and traditional indigenous ways of life. This statement is fully supported by the underlying material.

In both polar regions, specific ecosystems and habitats are projected to be vulnerable, as climatic barriers to species invasions are lowered. This statement is fully supported by the underlying material. 

# 11Assessing an IPCC assessment 80Small islands 81 

# Small islands

12.1 Analysis of statements in T able SPM.2 of the Synthesis Report

Sea level rise is expected to exacerbate inundation, storm surge, erosion and other coastal hazards, thus threatening vital infrastructure, settlements and facilities that support the livelihood of island communities. This statement is fully supported by the underlying material.

Deterioration in coastal conditions, for example through erosion of beaches and coral bleaching, is expected to affect local resources. This statement is fully supported by the underlying material.

By mid-century, climate change is expected to reduce water resources in many small islands, e.g. in the Caribbean and Pacific, to the point where they become insufficient to meet demand during low-rainfall periods. This statement is fully supported by the underlying material.

With higher temperatures, increased invasion by non-native species is expected to occur, particularly on mid- and high-latitude islands. This statement is fully supported by the underlying material.

12.2 Additional findings

Lower rainfall coupled with accelerated sea-level rise compounds the threat on water resources; a 10% reduction in average rainfall by 2050 is likely to correspond to a 20% reduction in the size of the freshwater lens on Tarawa Atoll, Kiribati. (C3; minor) We have a minor comment to make on this statement. The statement appears in the Technical Summary (Table TS2, page 58; Box TS.6, page 63), in the Executive Summary of Chapter 16 (page 689) and in the main chapter text (pages 696 and 697), and is indicated to have been based on a World Bank report (World Bank, 2000). However, this reference indicates a 14% reduction in the freshwater lens, as a consequence of a 10% reduction in precipitation. Figures change when other factors combine with changes in precipitation; −12% change in the freshwater lens is projected for 0.4 m sea level rise, and −38% for 0.4 m sea level rise in combination with reduced island size due to erosion. The World Bank report also notes that a 10% increase in evapotranspiration would lead to a 6% reduction in the freshwater lens. The chapter text is not clear on why the effect of a projected 10% increase in evapotranspiration should be added to the effect of a projected 10% reduction in 

# 12Assessing an IPCC assessment 82 precipitation. The Coordinating Lead Authors explained to us that these changes are correlated and that, hence, the 6% effect from evapotranspiration change should indeed be added to the 14% effect from changes in precipitation. The statement is correct, but we believe that this last point should have been explained in the text. This minor comment has no consequences for the IPCC conclusions in the various Summaries for Policymakers.

Less rainfall coupled with accelerated sea-level rise would compound this threat. Studies conducted on Bonriki Island in Tarawa, Kiribati, showed that a 50 cm rise in sea level accompanied by a reduction in rainfall of 25% would reduce the freshwater lens by 65% (World Bank, 2000). (E2; minor) Another instance of inaccurate referencing was found in the main text of Chapter 16 (page 697). However, the World Bank study does not report a 50 cm sea level rise. The reference should have been to a study by Falkland (1999) (E2). This minor inaccuracy has no impact on the IPCC conclusions in the various Summaries for Policymakers.

References
World Bank, 2000: Cities, Seas and Storms: Managing Change in Pacific Island. Econ omies. Vol. IV:

Adapting to Climate Change. World Bank, Washington, District of Columbia, 135 pp Falkland, A., 1999: Impacts of climate change on water resources of Pacific islands . PACCLIM 19

Workshop, Modelling the Effects of Climate Change and Sea Level Rise in Pacific Island 20 Countries. Auckland, New Zealand, 33pp.Annex A 83 

# Annex A T able SPM.2 of the Synthesis Report: examples of some projected regional impacts

This table including its footnote is a direct copy of the original table SPM.2 on page
11 of Climate Change 2007 – Synthesis Report, A report of the IPCC

.Assessing an IPCC assessment 84 **Table SPM.2. Examples of some projected regional impacts {3.3.2}**

Africa • By 2020, between 75 and 250 million of people are projected to be exposed to increased water stress due to climate change.

• By 2020, in some countries, yields from rain-fed agriculture could be reduced by up to 50%. Agricultural

production, including access to food, in many African countries is projected to be severely compromised. This would further adversely affect food security and exacerbate malnutrition.

• T owards the end of the 21st century, projected sea level rise will affect low-lying coastal areas with large populations. The cost of adaptation could amount to at least 5 to 10% of Gross Domestic Product (GDP).

• By 2080, an increase of 5 to 8% of arid and semi-arid land in Africa is

projected under a range of climate scenarios (TS).

Asia • By the 2050s, freshwater availability in Central, South, East and South-East Asia, particularly in large river basins, is projected to decrease.

• Coastal areas, especially heavily populated megadelta regions in South, East and South-East Asia, will be at

greatest risk due to increased flooding from the sea and, in some megadeltas, flooding from the rivers.

• Climate change is projected to compound the pressures on natural resources and the environment

associated with rapid urbanisation, industrialisation and economic development.

• Endemic morbidity and mortality due to diarrhoeal disease primarily associated with floods and droughts

are expected to rise in East, South and South-East Asia due to projected changes in the hydrological cycle.

Australia and New Zealand • By 2020, significant loss of biodiversity is projected to occur in some ecologically rich sites, including the Great Barrier Reef and Queensland Wet Tropics.

• By 2030, water security problems are projected to intensify in southern and eastern

Australia and, in New Zealand, in Northland and some eastern regions.

• By 2030, production from agriculture and forestry is projected to decline over much of southern

and eastern Australia, and over parts of eastern New Zealand, due to increased drought and fire. However, in New Zealand, initial benefits are projected in some other regions.

• By 2050, ongoing coastal development and population growth in some areas of

Australia and New Zealand are projected to exacerbate risks from sea level rise and increases in the severity and frequency of storms and coastal flooding.

Europe • Climate change is expected to magnify regional differences in Europe’s natural resources and assets. Negative impacts will include increased risk of inland flash floods and more frequent coastal flooding and increased erosion (due to storminess and sea level rise).

• Mountainous areas will face glacier retreat, reduced snow cover and winter tourism, and

extensive species losses (in some areas up to 60% under high emissions scenarios by 2080).

• In southern Europe, climate change is projected to worsen conditions (high temperatures

and drought) in a region already vulnerable to climate variability, and to reduce water availability, hydropower potential, summer tourism and, in general, crop productivity.

• Climate change is also projected to increase the health risks due to heat waves and the frequency of wildfires.

Latin America • By mid-century, increases in temperature and associated decreases in soil water are projected to lead to gradual replacement of tropical forest by savanna in eastern Amazonia. Semi-arid vegetation will tend to be replaced by arid-land vegetation.

• There is a risk of significant biodiversity loss through species extinction in many areas of tropical Latin America.

• Productivity of some important crops is projected to decrease and livestock productivity to decline, with

adverse consequences for food security. In temperate zones, soybean yields are projected to increase. Overall, the number of people at risk of hunger is projected to increase (TS; medium confidence).

• Changes in precipitation patterns and the disappearance of glaciers are projected to significantly

affect water availability for human consumption, agriculture and energy generation. **Table A.1**Annex A 85 North America • W arming in western mountains is projected to cause decreased snowpack, more winter flooding and reduced summer flows, exacerbating competition for over-allocated water resources.

• In the early decades of the century, moderate climate change is projected to increase aggregate yields of rainfed agriculture by 5 to 20%, but with important variability among regions. Major challenges are projected for crops that are near the warm end of their suitable range or which depend on highly utilized water resources.

• Cities that currently experience heat waves are expected to be further challenged

by an increased number, intensity and duration of heat waves during the course of the century, with potential for adverse health impacts.

• Coastal communities and habitats will be increasingly stressed by climate

change impacts interacting with development and pollution.

Polar Regions • The main projected biophysical effects are reductions in thickness and extent of glaciers, ice sheets and sea ice, and changes in natural ecosystems with detrimental effects on many organisms including migratory birds, mammals and higher predators.

• For human communities in the Arctic, impacts, particularly those resulting from

changing snow and ice conditions, are projected to be mixed.

• Detrimental impacts would include those on infrastructure and traditional indigenous ways of life.

• In both Polar Regions, specific ecosystems and habitats are projected to be

vulnerable, as climatic barriers to species invasions are lowered.

Small Islands • Sea level rise is expected to exacerbate inundation, storm surge, erosion and other coastal hazards, thus threatening vital infrastructure, settlements and facilities that support the livelihood of island communities.

• Deterioration in coastal conditions, for example through erosion of beaches

and coral bleaching, is expected to affect local resources.

• By mid-century, climate change is expected to reduce water resources in many small islands, e.g. in the

Caribbean and Pacific, to the point where they become insufficient to meet demand during low-rainfall periods.

• With higher temperatures, increased invasion by non-native species is

expected to occur, particularly on mid- and high-latitude islands.

Note:Unless stated explicitly, all entries are taken from the text of the Working Group II Summary for Policymakers. They are either ‘very high confidence’ or ‘high confidence’ statements, related to different sectors (agriculture, ecosystems, water, coasts, health, industry and settlements). The Working Group II Summary for Policymakers refers to the source of the statements, timelines and temperatures. The magnitude and timing of impacts that may ultimately occur will vary according to the amount and rate of climate change, emissions scenarios, development pathways and adaptation.Assessing an IPCC assessment 86 

# Annex B T he error on the melting of the Himalayan glaciers

In the media, much attention was paid to this erroneous statement in the chapter on Asia, page 493 of Section 10.6.2, which suggested the disappearance of Himalayan glaciers by 2035:

G laciers in the Himalaya are receding faster than in any other part of the world [...] the likelihood of them disappearing by the year 2035 and perhaps sooner is very high if the Earth keeps warming at the current rate. Its total area will likely shrink from the present 500,000 to 100,000 km **2** by the year 2035 (E1; major) Here, we provide an analysis of this error and some background information, which also provides an update that includes literature that became available after production and publication of the Fourth Assessment Report.

In the Synthesis Report (IPCC, 2007) and in the Summary for Policymakers (SPM), and the Technical Summary (TS) of the Working Group I contribution to the IPCC Fourth Assessment Report (Solomon et al., 2007), no explicit statements were made on Himalayan glacier retreat. In the Summary for Policymakers of the Working Group II Report (Parry et al., 2007), there was only one reference to the Himalayas:

Glacier melt in the Himalayas is projected to increase flooding, and rock avalanches from destabilised slopes, and to affect water resources within the next two to three decades. This will be followed by decreased river flows as the glaciers recede

.

This statement is rather general and correct, although the impact on water resources due to glacier retreat is very likely to be rather limited in the next two to three decades. However, in the Technical Summary, a major error occurred in Box TS6, on page 59:

If current warming rates are maintained, Himalayan glaciers could decay at very rapid rates, shrinking from the present 500,000 km2 to 100,000 km **2** by the 2030s.

This text was based on page 493 of Section 10.6.2 of Working Group II Report,
which states:Annex B 87 Glaciers in the Himalaya are receding faster than in any other part of the world [...] the likelihood of them disappearing by the year 2035 and perhaps sooner is very high if the Earth keeps warming at the current rate. Its total area will likely shrink from the present 500,000 to 100,000 km **2** by the year 2035 (WWF 2005).

The first mention of ‘2035’ was taken from a publication by the World Wide Fund for Nature (WWF, 2005), which cited a news story (Pearce, 1999) about an unpublished study (Hasnain, 1999) that does not estimate a date for disappearance of Himalayan glaciers.

The second mention of ‘2035’, again a major error, was not taken from the (WWF, 2005), but could be traced to a rough estimate (Kotlyakov, 1996) on the shrinkage
of all non-polar glaciers (excluding those in basins of internal drainage) between the present day and the year 2350. Furthermore, the Himalayan glaciers do not cover 500,000 km **2** , but about 30,000 km **2** , as was correctly indicated in the first sentence of Section 10.6.2, which stated that: Himalayan glaciers cover about three million hectares

.

In conflict with knowledge of glacier–climate relationships, disappearance by 2035 would require a 25-fold acceleration of the loss rate during the 1999-2035 period, compared to the estimated past loss rate between 1960 and 1999 (Dyurgerov and Meier, 2005), a situation which is extremely unlikely to occur.

Here, we provide some additional background information, mainly based on a study by Oerlemans et al. (2007) and on a recent American Geophysical Union lecture by leading experts in this field (Kargel et al., 2010). The lecture was held on 14 December 2009 in San Francisco, in response to media attention concerning the claim in the IPCC Working Group II Report that the Himalayan glaciers might be gone by 2035 (Parry et al., 2007).

Glacier retreat and water supply The region-wide average glacier area retreat rate is probably between 0.1 and 0.5% per year. Although the size of the glacier area will change substantially this century, especially in the most vulnerable eastern zone of the Himalayas, glaciers (such as Khumbu and Imja), will not disappear entirely, or even mostly, by 2035, as stated in the Working Group II contribution to the Fourth Assessment Report of the IPCC.

All glaciers combined store roughly 3,400 km **3** of fresh water. This represents almost three years in combined water discharges from the Indus, the Ganges and the Brahmaputra. The prevalent negative balances of Himalayan glaciers (meaning the annual net withdrawals of ice) total between 0.4 and 1.2% of current river flows, which is not very significant, basin-wide. So, although over half a billion people draw water from these melting glaciers, this only represents between one and a few per cent of their water supplies.

Increased melting may further increase water discharge by 1 to 2%, in the next few decades. Within 50 to 100 years, the discharge could decrease by several per cent, as the current negative balance contribution decreases. However, on the scale of the subcontinent, hydrological problems are likely to stem more from populationAssessing an IPCC assessment 88 growth, inefficiency in water resource distribution and application, and from changes in the Monsoon and the Westerlies, due to global warming and changes in the Elevated Heat Pump (EHP). Changes in the Monsoon and the Westerlies already appear to be substantial and are likely to be increasingly impactful.

On a local scale, the seasonal influence of glacier melt can be strong, and even rivers as large as the Indus fluctuate widely throughout the year because of the seasonality of water storage (mostly in the form of ice), and melting of both glaciers and high altitude snow fields. The greatest vulnerability of large numbers of people to glacier state and dynamics is to those in arid and semi-arid lands, such as north-eastern Afghanistan, Uzbekistan, parts of Pakistan, and western China. This concerns tens of millions of people.

Sea level rise Since 1900, the majority of the observed glaciers around the world are retreating, including those in central Asia. Based on Oerlemans et al. (2007), global glacial contribution to sea level rise was 4.5±0.7 cm during the 1900-2000 period, compared to a total sea level rise of ~19.5 cm in the 1870-2004 period. This estimate is higher than the IPCC estimate of 3.0±1.0 cm for the 20th century, and better explains the observed sea level rise as caused by thermal expansion and changes in the cryosphere. Based on Oerlemans et al. (2007) and Kargel et al. (2010), it is roughly estimated that the annual contribution from Himalayan glaciers to sea level rise is 0.06±0.04 mm, or ~2% of the current annual sea level rise of 3.1±0.7 mm (Parry et al., 2007). The rate of Himalayan glacier melting appears to have increased over time, but so has sea level rise, and therefore the 2% contribution may still apply.

References Dyurgerov, M.B. and M.F. Meier, 2005. Glaciers and the changing earth system, a 2004 snapshot,

Occasional Paper No. 58. Boulder, Institute of Arctic and Alpine Research, University of Colorado:

117 Hasnain, S.I., 1999. Report on Himalayan Glaciology. New Dehli, ICSI working group on Himalayan

Glaciology: 22. IPCC, 2007. Climate Change 2007. Synthesis report. Contribution of W orking Groups I, II and III to the

Fourth Assessment Report of the Intergovernmental Panel on Climate Change. Core writing team: Pachauri, R.K. and Reisinger, A. (Eds.). Geneva, Switzerland. Kargel, J.S., R. Armstrong et al., 2010. Satellite-era glacier changes in High Asia. Background support

presentation for NASA “Black Carbon and Aerosols” press conference associated with Fall AGU, Dec. 14, 2009. Updated and expanded Feb. 17, 2010. Kotlyakov, V.M., 1996. Variations of Snow and Ice in the past and at present on a Global and Regional

Scale. I. H. PROGRAMME. Paris, UNESCO. Oerlemans, J., M. Dyurgerov et al., 2007. "Reconstructing the glacier contribution to sea-level rise

back to 1850." The Cryosphere 1: 59-65. Parry, M.L., O.F. Canziani et al., 2007. Climate Change 2007: Impacts, Adaptation and V ulnerability.

Cambridge, Cambridge University Press. Pearce, F., 1999. Flooded Out. New Scientist. 162. Solomon, S.D., M.M. Qin et al., 2007. Climate Change 2007: The Physical Science Basis. Contribution

of Working Group I to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change., Cambridge University Press, Cambridge, United Kingdom and New Y ork, NY, USA. WWF, 2005. An overview of Glaciers, glacier retreat and its subsequent impacts in Nepal, India and

China: 67.Annex C 89 

# Annex C T he error on the percentage of Dutch land area below sea level

In February 2010, an error was detected in a text supplied by the Netherlands Environmental Assessment Agency (PBL) for Chapter 12 (on Europe), page 547, Subsection 12.2.3, of the Working Group II contribution to the Fourth Assessment Report, regarding the risks of flooding in the Netherlands:

T he Netherlands is an example of a country highly susceptible to both sea-level rise and river flooding because 55% of its territory is below sea level where 60% of its population lives and 65% of its Gross National Product (GNP) is produced. (E1; major) The error was discovered by a Dutch journalist of a weekly newspaper (V rij Nederland) who checked information about the Netherlands, triggered by the publicity around the error on the Himalayan glaciers. On page 547 of the chapter text, it says that 55% of the Netherlands is located below sea level . This should have read that 55% of the Netherlands is at risk of flooding (E1); 26% of the country is at risk because it is located below sea level, and another 29% is susceptible to river flooding. **1)** Examples of the latter are the near floodings in the mid-1990s, of areas along the rivers Meuse and Waal – areas that are well above sea level. The Netherlands is sensitive to climate change. Sea level rise as well as peak river discharges have been analysed frequently, and according to government policy this requires precautionary measures.

The original text that included this error was submitted by the Netherlands Environmental Assessment Agency (PBL) in mid 2005. A similar error also occurred in publications by others. A commentary that was published in Nature in 2005 (Kabat et al., 2005) refers to 60% of the Dutch land area as being located below sea level. In 2007, an English version of a note from the Dutch Ministry of T ransport also stated (Dutch Ministery of Transport, 2007) that about 60% of the country was

1) H ere, the area of the Netherlands that is prone to flooding has been divided in a binary manner, starting with determining the part of the country that lies below NAP (a particular reference sea level) – which is 26% – the rest of the country at risk of flooding is then categorised as being at risk of flooding by rivers; of course, more finegrained subdivisions could also have been made, since obviously the part of the country that lies below sea level is also at risk of flooding by rivers, and parts of the country that lie above NAP are also at risk of flooding by the sea.Assessing an IPCC assessment 90 situated below sea level. It would have been correct to state that about 60% of the Netherlands would be prone to flooding by sea if the sea level would rise by 1 m above the 2000 level and a severe storm would hit the coast during spring tide – but this is not the same thing. In the review process of the IPCC W orking Group II Report, there were several comments and suggested changes relating to this chapter, but this error remained unnoticed.

Immediately after detection, the PBL reported the error to the international community. It was also reported to the IPCC, and on 27 April 2010 is was included in a list of errata on IPCC reports. The incorrect wording in the IPCC report does not affect the message of the conclusion. The error has no significance for the main conclusions in the chapter, or for those on Europe, nor for the conclusions in the various Summaries for Policymakers.

We acknowledge that this error was not the fault of the IPCC (Coordinating) Lead Authors or Co-Chairs. The error was made by a Contributing Author from the PBL, and the (Coordinating) Lead Authors are not to blame for relying on Dutch information provided by a Dutch agency. The lesson to be learned for an assessment agency such as ours, is that quality control is needed at the primary level of a literature assessment. This should be done by checking basic data, to the extent feasible, even when authoritative references are available. Of course, there are limitations to checking data, given the vast amounts of specific regional
information.

References
Kabat, P., Vierssen, W. van, Veraart, J.A., Vellinga, P., Aerts, J., 2005. Climate proofing the

Netherlands. Nature, vol. 438, p. 283-284 Ministry of Transport, Public Works and Water Management, 2007. Flood maps in the Netherlands,

http://www.safecoast.org/editor/databank/File/folder%20engels%20def%201%20febr07.pdfAnnex D 91 

# Annex D S ea level rise: consequences for the Netherlands

Summary Errors in the IPCC report regarding the Himalayan glaciers and the Dutch land area below sea level have triggered a discussion on whether the Dutch policies that deal with sea level rise should be revised. The answer is clearly ‘no’: the contribution from melting Himalayan glaciers to sea level rise is very limited (around 2%). Moreo

ver, the policy concerning safety against flooding in the Netherlands, as formulated in the National Water Policy Plan 2009-2015 (Ministry of Transport, Public Works and Water Management, 2009) and elaborated in the Delta Programme, is well founded on the available knowledge on climate change, sea level rise, river discharges and
uncertainties. The current discussions related to the IPCC have no effect on the assumptions under the National Water Policy Plan and the Delta Programme.

There is scientific agreement that the main causes of mean global sea level rise are thermal expansion of sea water and melting of land ice due to increasing temperatures. However, future projections of sea level rise are uncertain because of the large range of temperature projections and the limited knowledge about the melting behaviour of especially the arctic ice shields (PBL2007, Katsman et al., 2008).

Sea level rise: limited contribution from the Himalayan glaciers Over the last century, the sea levels have risen, world wide, by 14 to 20 cm, cor

responding with an average of 1.7±0.3 mm a year (Church et al., 2006). Along the Dutch coast, the sea level rise was of the same order of magnitude: 19 cm with an average of 1.8±0.2 mm a year (PBL, 2010). Satellite data indicate an accelerated trend in global sea level rise of 3.1±0.7 mm a year, over the past 15 years (Church and White, 2006; Rahmstorf et al., 2007). It is unclear whether this acceleration will continue. The effect of glacier melting in the Himalayas on global sea level rise is limited: based on Oerlemans et al. (2007) and Kargel et al. (2010), it is esti

mated that the annual contribution from the Himalayan glaciers to sea level rise is 0.06±0.04 mm, that is, around 2% of the current annual sea level rise of 3.1±0.7 mm (Parry et al., 2007). Therefore can be concluded that the contribution from the melting of the Himalayan glaciers to sea level rise is very limited.Assessing an IPCC assessment 92 Estimates on future sea level rise for the Netherlands: large uncertainties There is scientific agreement that the main causes of mean global sea level rise are thermal expansion of sea water and the melting of land ice due to increasing temperatures. However, future projections of sea level rise are uncertain because of the large range of temperature projections and the limited knowledge about the melting behaviour of especially the arctic ice shields (PBL, 2007; Katsman et al., 2008). The range for temperature rise is due to i) natural variability, such as the recent slowing down of the increase in average global temperature, ii) the climate sensitivity of the atmosphere, iii) uncertainties in atmospheric processes, and iv) uncertainties in future emission of greenhouse gases.

When considering the possible sea level rise for the Netherlands, the most recent estimates on sea level rise for the Netherlands cover a range of 35 to 85 cm for 2100 (KNMI, 2006), or in case of high-end/worst-case estimates, the rise is between 130 cm (Deltacommissie, 2008) and 150 cm (PBL, 2007).

In 2007, in collaboration with the Royal Netherlands Meteorological Institute (KNMI), the PBL explored plausible future extreme sea level rise scenarios for the Netherlands and – based on paleoclimatological data – estimated a worst-case sea level rise of 1.5 m/century (PBL, 2007). Given the technical adaptive capacity of the Netherlands and the considered safety margins, the PBL concluded that, with the available techniques, the delta region of the Netherlands could be kept safe even in case of such an extreme sea level rise, but that in the long term, spatial measures could be required (PBL, 2007). In 2008, the Dutch Delta committee (Deltacommis

sie) presented a high-end estimate on sea level rise of between 65 and 130 cm for 2100, based on a temperature-rise scenario of 6 degrees, and also concluded that technically the delta region of the Netherlands could protect itself against flood

ing (Deltacommissie 2008). The maximum ranges for sea level rise given by the IPCC (2007) and the KNMI (2006), were based on scenarios with a maximum of 4 degrees temperature rise. A 6-degree temperature rise thus exceeds the 4-degree scenario of the IPCC, but still lies within the total range of uncertainty with respect to temperature rise (e.g. PBL, 2009, News in Climate Science).

Consequences for policy on safety against flooding Following the advice of the Delta committee in 2008, the policy on safety against flooding and freshwater management will be revised, given the increases in popula

tion and economical values in the area that has been sensitive to flooding since the 1960s, and the challenges imposed by climate change. The recent National Water Policy Plan 2009-2015 (Dutch Ministry of Transport, Public Works and Water Management, 2009) has announced that i) the national standards for the safety against flooding will be reconsidered, and ii) a long-term adaptive strategy on climate proofing of the Netherlands will be constructed, including an analysis of possible consequences for spatial development. To effectuate these points, the Delta Programme has been launched, together with a Delta Act, Delta Fund and a Delta Commissioner responsible for the coordination of the programme. In this Delta Programme, the four available KNMI scenarios (KNMI, 2006) are taken as the plausible range for future uncertainties for the Netherlands, with respect to climate change and sea level rise.Annex D 93 Current investments in the protection against flooding in the Netherlands arejustified by the fact that a substantial number of dikes (24%) did not meet presentday safety standards because of insufficient maintenance and new hydrological conditions. The investments that are required for the improvement of the dikes are not fully known yet, because for 32% of the dikes, the quality status is currently unknown (Dutch Ministry of Transport, Public Works and Water Management, 2006).

Given that sea level rise is a slow process and that the uncertainties in future projec

tions are considerable, future additional investments in the safety of the Nether

lands will depend on progressing scientific insights. Sea level rise is addressed in the current policy on safety against flooding. For safety measures in coastal areas, decisions with a short lead time (5 years) are based on the present sea level rise of 20 cm/century and – based on earlier KNMI scenarios – decisions with lead times of 50 to 100 years are based on 60 cm sea level rise. Spatial reservations for the long term (100 to 200 years) are based on a high sea level rise of 85 cm. Possible impacts of climate change from floods due to extreme river discharges have also been incorporated in the management strategy ‘Room for the River’ through spatial reservations in certain areas for which additional measures could be required in the future. Finally, the process of regular evaluations of the quality of the dikes, every six years, will include the advancing knowledge about climate change, sea level rise and its consequences (Dutch Ministry of Transport, Public Works and Water Management, 2009).

An additional, important factor that will determine river discharges in the Nether

lands is the river and safety management upstream in the Rhine and Meuse river basins. Since 2009, the EU High Water Directive has been operational. It provides a legal basis for international appointments in flood risk management on the scale of transboundary river basins.

Considering the slow process of sea level rise, the adaptive capacity of the Nether

lands with respect to technical measures (e.g., strengthening of dikes, storm bar

riers, dunes) is thought to be sufficient – even in worst-case scenarios – provided that the required budgets become available (PBL, 2007; Deltacommissie, 2008). However, the study by the PBL (2007) and Ligtvoet et al. (2009) point out that, in the long run, management of peak river discharges will be the main challenge for the Netherlands, and that adaptive capacity concerning spatial developments is limited, because of their relative slowness and irreversibility. Therefore, choices that will be made in the coming decades, in urban development, nature develop

ment and configuration of the main water system in the Netherlands, will also determine the future climate resilience of the Netherlands (PBL, 2009). It has been shown that a combination of spatial and technical measures regarding river areas, IJsselmeer area and the south-western delta, may contribute most to reducing the vulnerability of the Netherlands, in both the short term and the long term (Ligtvoet et al., 2009).

Conclusion The current policy on safety against flooding in the Netherlands, as formulated in the National Water Policy Plan and elaborated in the Delta Programme, is wellAssessing an IPCC assessment 94 founded on the available knowledge on climate change, sea level rise, river dis

charges and uncertainties. The current discussions about the IPCC have no effect on the assumptions underlying the Dutch National Water Plan.

The required investments in the period up to 2020 (although not fully known yet) are to ensure that the quality of the dikes will meet present-day standards (W ater Act). The required future maintenance and investment levels will depend on the new safety standards and the measures that will be adopted within the frame

work of a long-term adaptive strategy. The recently launched Delta Programme within the framework of the National Water Policy Plan 2009-2015 (Dutch Ministry of Transport, Public Works and Water Management, 2009) contains an extensive research agenda to focus future research on safety against flooding and on fresh

water supply in the Netherlands. In addition, the agenda lists the drafting of a longterm adaptive strategy, addressing the still large uncertainties concerning climate change. For 2015, further policy decisions are expected on extra measures and investments for safety against flooding and for freshwater supply. On a political level, it has been decided that within the context of the Delta Programme, by 2020,
financial reservations will be made for additional investments and maintenance.

References
Church, J.A., and N.J. White, 2006. A 20th century acceleration in global sea-level rise, Geophys.

Res. Lett., 33, L01602, doi:10.1029/2005GL024826. Update data to 2007 on PSMSL website (link to data) Deltacommissie, 2008. Samen werken aan water. Bevindingen van de Deltacommissie.

Deltacommissie, Den Haag, the Netherlands. Kargel, J.S., R. Armstrong et al., 2010. Satellite-era glacier changes in High Asia.Background support

presentation for NASA “Black Carbon and Aerosols” press conference associated with Fall AGU, Dec. 14, 2009. Updated and expanded Feb. 17, 2010. Katsman, C.A., W.Hazeleger, S.S. Drijfhout, G.J. van Oldenborgh & G. Burgers, 2008. Climate

scenarios of sea level rise for the northeast Atlantic Ocean: a study including the effects of ocean dynamics and gravity changes induced by ice melt, Climatic Change 91:351–374 KNMI, 2006. Climate change scenarios for the Netherlands. Scientific report WR 2006-01. Koninklijk

Nederlands Meteorologisch Institute, Bilthoven, the Netherlands. Ligtvoet, W.J. Knoop, B. Strengers & A. Bouwman, 2009. Flood protection in the Netherlands:

framing long term challenges and options for a climate resilient delta. PBL publication number 500078004. Netherlands Environmental Assessment Agency – PBL, Bilthoven/Den Haag, the

Netherlands. Ministry of Transport, Public Works, and Water management, 2006. Primaire Waterkeringen

getoetst, Landelijke rapportage Toetsing 2006. Ministry of Public Works, Transport and Watermanagement, Den Haag, the Netherlands Ministry of Transport, Public Works, and Water management, 2009. Nationaal Waterplan 2009-2015.

Ministry of Public Works, Transport and Water management, Den Haag, the Netherlands. Oerlemans, J., M. Dyurgerov et al., 2007. "Reconstructing the glacier contribution to sea-level rise

back to 1850." The Cryosphere 1: 59-65. Parry, M. L., O.F. Canziani et al., 2007. Climate Change 2007: Impacts, Adaptation and V ulnerability.

Cambridge, Cambridge University Press. PBL, 2007. The Netherlands in the future. Second Sustainability Outlook - part Physical Environment

of the Netherlands. Netherlands Environmental Assessment Agency – PBL, Bilthoven/Den Haag,

The NetherlandsAnnex D 95 PBL, 2009. Roadmap to a climate-proof Netherlands. PBL publication number 500078003.

Netherlands Environmental Assessment Agency – PBL, Bilthoven/Den Haag, The Netherlands PBL, 2009. News in Climate Science. Netherlands Environmental Assessment Agency – PBL,

Bilthoven/Den Haag, The Netherlands. PBL, 2010. Zeespiegelstand langs de Nederlandse kust en mondiaal, 1891-2008. V ersie 7.

Compendium voor de Leefomgeving. http://www.compendiumvoordeleefomgeving.nl/ indicatoren/nl022907-Zeespiegelstand-Nederland.html?i=9-54. Netherlands Environmental Assessment Agency – PBL, Bilthoven/Den Haag. Statistics Netherlands, Den Haag/Heerlen, Wageningen UR, Wageningen, The Netherlands . Rahmstorf, S., 2007. A semi-empirical approach to projecting future sea-level rise. Science315: 367-

370Assessing an IPCC assessment 96 

# Annex E A bbreviations

ASTER Advanced Spaceborne Thermal Emission and Reflection Radiometer (a remote sensing instrument)

CLA Coordinating Lead Author

CO **2** Carbon dioxide

CRU Climate Research Unit (at the University of East Anglia, United Kingdom)

EHP Elevated Heat Pump

FAO Food and Agriculture Organization

FOD
First Order Draft

GCM Global Circulation Model (a type of climate model)

GDP
Gross Domestic Product

GNP
Gross National Product

H-K Himalaya-Karakoram

IPCC Intergovernmental Panel on Climate Change

IUCN
International Union for Conservation of Nature

KNAW Royal Netherlands Academy of Arts and Sciences

KNMI Royal Netherlands Meteorological Institute

NAP
Amsterdam Ordnance Datum NCAR National Center for Atmospheric Research (Boulder, USA)

NGO Non Governmental Organisation

SOD
Second Order Draft

SPM Summary for Policymakers

SRES IPCC Special Report on Emission Scenarios

SYR IPCC Synthesis Report (of Working Groups I, II and III reports)

TAR Third Assessment Report of the IPCC

TS Technical Summary

UN
United Nations

UNFCCC
United Nations Framework Convention on Climate Change

UNMSM Universidad Nacional Mayor de San Marcos (Lima, Peru)

URL Uniform Resource Locator (e.g. an internet address)

WG Working Group

WG I IPCC Working Group I (on the Science of Climate Change)

WG II IPCC Working Group II (on Impacts, Adaptation and Vulnerability)Annex E 97 WG III IPCC Working Group III (on Mitigation of Climate Change)

WWF
World Wide Fund for NatureAssessing an IPCC assessment 98Colophon 99 

# Colophon

Responsibility Netherlands Environmental Assessment Agency

Editors
L.A. Meyer, A.C. Petersen

Steering group M.A. Hajer, J.A. Oude Lohuis, L.A. Meyer, A.C. Petersen, G. de Lange, J. Timmerhuis,

E. Buitelaar

Research
J.R.M. Alkemade, M. Bakkenes, G.J. van den Born, A.F. Bouwman, J.J.M. van Grinsven, H.B.M. Hilderink, P.H.M. Janssen, W. Ligtvoet, A.J.G. Manders, J.G. van Minnen, B.J. Strengers, H. Visser, M. Vonk, M. de Vos (PBL), R.J. Swart (WUR)

Translation and editing A. Righart (PBL), C. Ettema (Shades of Green - Natural Translations)

Other contributors F.S. de Blois, C. Campman, J.H.J. Dolmans, S.A. van Esch, M. Hartman, A.M. Oosterhuis, J.F. de Ruiter, C. Spelbos

Design and layout Uitgeverij RIVM

Corresponding Author L. Meyer, leo.meyer@pbl.nlsnijlijn snijlijn rug PBL-omslag zonder flappen
340 x 240 mm

**Hier komt bij voorkeur een flaptekst te staan, in plaats van deze algemene tekst** 

Het Planbureau voor de Leefomgeving is hét nationale instituut voor strategische beleidsanalyses op het gebied van milieu, natuur en ruimte.

Het PBL draagt bij aan de kwaliteit van het strategische overheidsbeleid door een brug te vormen tussen wetenschap en beleid en door gevraagd en ongevraagd,onafhankelijk en wetenschappelijk gefundeerd, verkenningen, analyses en evaluaties te verrichten waarbij een integrale benadering voorop staat.

Planbureau voor de Leefomgeving

Locatie Bilthoven Postbus 303 3720 AH Bilthoven
T: 030 274 274 5 F: 030 274 4479 E: info@pbl.nl www.pbl.nl

Planbureau voor de Leefomgeving, <maand> 200X **Assessing an IPCC assessment**The Netherlands Environmental Assessment Agency (PBL) has investigated the scientific foundations for the IPCC summary conclusions of the Fourth Assessment Report of 2007 on projected regional climate-change impacts, at the request of the Dutch Minister for the Environment. Overall the summary conclusions are considered well founded and none were found to contain any significant errors.

The Working Group II contribution to the Fourth Assessment Report shows ample observational evidence of regional climate-change impacts, which have been projected to pose substantial risks to most parts of the world, under increasing temperatures. However, in some instances the foundations for the summary statements should have been made more transparent. While acknowledging the essential role of expert judgment in scientific assessments, the PBL recommends to improve the transparency of these judgments in future IPCC reports.

In addition, the investigated summary conclusions tend to single out the most important negative impacts of climate change. Although this approach was agreed to by the IPCC governments for the Fourth Assessment Report, the PBL recommends that the full spectrum of regional impacts is summarised for the Fifth Assessment Report, including the uncertainties. The PBL believes that the IPCC shouldinvest more in quality control in order to prevent mistakes and shortcomings, to the extent possible.

Netherlands Environmental Assessment Agency, July 2010 

# Assessing an IPCC assessment An analysis of statements on projected regional impacts in the 2007 report

# The Independent Climate Change
E

- mail s Review

July 2010

Chair: Sir Muir Russell

Review team: Professor Geoffrey Boulton
Professor Peter Clarke David Eyton Professor James Norton

### 2



## INDEX OF CONTENTS

GLOSSARY 8

Chapter 1 EXECUTIVE SUMMARY 10

1.1 Introduction 10 1.2 The Review Process 11 1.3 Findings 11 1. 3 .1 Land Station Temperatures 12 1.3.2 Temperature Reconstructions from Tree Ring Analysis 12 1.3

. 3 Peer Review and Editorial P olicy 13 1.3

. 4 Misuse of IPCC P rocess 13 1.3

. 5 Compliance with the Freedom of Information Act (FoIA) and the Environmental Information Regulations (EIR) 14

1.3 .6 Other Findings on Governance 14 1.4 Recommendations 14 1.5 Broader Issues 14

Chapter 2 INTRODUCTION 18

2.1 Background 18 2.2 The Review 19

Chapter 3 TER MS OF REFERENCE AND METHOD OF E NQUIRY 22

3.1 Terms of Reference 22 3.2 Method of E nquiry 22

Chapter 4 C ONTEXT OF THE E

- MAILS 26

4.1 Characterising the E

- mails 26 4.2 The T imeline 27 4.3 What the E

- mails Tell U s 32

Chapter 5 THE CHANGING CONTEXT OF MODERN SCIENCE 36

5.1 The Scientific P rocess 36 5.2 The Nature of Climate S cience 36 5.3 Data A ccessibility in the D igit al A ge 37 5.4 Handling U ncertainty 38 5.5 Scientific Journals and the Peer Review P rocess 39 5.6 The Responsibilities of Scientists in Communicating in the Public D omain 40 5.7 Communicating to P olicymakers 41 5.8 The C ha nging Forum for D ebate and the B logosphere 41

Chapter 6 LAND STATION INSTRUMENTAL TEMPERATURE DATA 44

6.1 Background 44 6.2 The Allegations 44 6.3 The Approach Adopted by the Review Team 45 6.4 The Results of the Analysis 46 6.5 Checking Specific Details in the CRUTEM Analysis 49 6.5.1 Identification of Data Sources 49 6.5.2 The Availability of Computer Codes 49 6.6 Use of Local temperature Data from China 6.7 Conclusions and Recommendations 51 53

### 3

### 4 Chapter 7 TEMPERATURE RECONSTRUCTIONS FROM TREE RING ANALYSIS 54

7.1 Background 54 7.2 The Allegations 54 7.3 Findings 56 7.3.1 IPPC Reports 56 7.3.2 Divergence 59 7.3.3 Withholding Data 60 7.3.4 Mishandling Data 61 7.4 Conclusions and R ecommendations 62

Chapter 8 PEER REVIEW AND INFLUENCING EDITORIAL POLICY OF SCIENTIFIC

JOURNALS 64

8.1 Background : Peer Review, Testing and V erification 64 8.2 The Allegations 64 8.3 The Soon and Baliunas A ffair & Climate Research 64 8.4 The C onflict with Dr Boehmer

- Christiansen 66 8.5 Peer Review and Professor Briffa‘s E ditorship of Holocene 67 8.6 Conclusion s 68

Chapter 9 COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC 70

9.1 Background 70 9.2 The Allegations 70 9.3 The CRUTEM Temperature S eries 71 9.3.1 The Scientific C hallenge 71 9.3.2 The A llegations 71 9.3.3 Evidence in S u pport of the A llegations 71 9.3.4 Jones‘ R esponse 73 9.3.5 Eviden ce from IPCC Review Editor for C hapter 3 (Professor Sir Brian Hoskins) 74

9.3.6 Findings 75 9.4 The Tree Ring Proxy Temperature S eries 77 9.4.1 The Scientific C hallenge 77 9.4.2 The A llegations 77 9.4.3 Evidence in Support of the A llegations 78 9.4.4 Responses from Briffa 80 9.4.5 Eviden ce from IPCC Review Editor for C hapter 6 (Professor John Mitchell) 82

9.4.6 Findings 83 9.5 Conclusions 84

Chapter 10 COMPLIANCE WITH FoIA / ENVIRONMENTAL INFORMATION

REGULATIONS 86

10.1 Introduction and Method of Enquiry 86 10.2 The Allegations 86 10.3 General Context 86 10.4 Investigation 89 10.5 Findings 91 10.6 Recommendations 94

### 5

### 6 Chapter 11 GOVERNANCE 98

11.1 Introduction and Method of E nquiry 98 11.2 Research Management Systems 98 11.2.1 Background 98 11.2.2 Funding Management 99 11.2.3 Funders‘ R equirements 99 11.2.4 Good Research Practice 99 11.2.5 Financial Controls 100 11.2.6 Risk Management 100 11.2.7 Findings on Research Management Systems 100 11.3 Software, Data Management and Data Security 101 11.3.1 General Context 101 11.3.2 Issues and the Investigation 102 11.3.3 Findings on Software, Data Management and Data Security 103 11.4 Recommendations 103

ACKNOWLEDGEMENTS 106

APPENDIX 1: REVIEW TEAM MEMBERS 108 APPENDIX 2: APPROACH AND WORK PLAN 110 APPENDIX 3: ISSUES FOR EXAMINATION 112 APPENDIX 4: INDEX OF MEETINGS, INTERVIEWS, SUBMISSIONS, 120

FOLLOW UP ENQUIRIES AND RESPONSES APPENDIX 5: PEER REVIEW 126 APPENDIX 6: DATA MINING

– ACCESS TO THE ORIGINAL CRU E

- MAIL 146

ARCHIVE APPENDIX 7: LAND STATION TEMPERATURE DATA 150 APPENDIX 8: SOFTWARE AND DATA STANDARDS 160

### 7

### 8



## GLOSSARY

AR4 Fourth Assessment Report of the Intergovernmental Panel on

Climate Change in 2007
CLA Coordinating Lead Author

COP Conference of the Parties

COPE Committee on Publication Ethics

CRU Climatic Research Unit

CRUT EMX Land air temperature anomalies on a 5° by 5° grid

- box basis

,

version X
DEFRA UK Government Department for Environment, Food and Rural

Affairs
DPA Data Protection Act 1998

E&E Energy and Environment

EC European Community

EIR Environmental Information Regulations

ENSO El Niño

- Southern Oscillation

ENV U niversity of E ast A nglia School of Environmental Sciences

FOI Freedom of Information

FoIA Freedom of Information Act 2000

GHCN Global Historical Climatology Network

GISS Goddard Institute for Space Studies

GISTEMP Goddard Institute for Space Studies Surface Temperature Analysis

GWPF Global Warming Policy Foundation

HadCRUTX Combined land and marine temperature anomalies on a 5° by 5°

grid

- box basis , version X
IAC Inter Academy Council

ICCER Independent Climate Change E

- Mails Review

ICO Information Commissioner‘s Office

ICT Information Communications Technology

IDL Interactive Data Language

IEC International Electrotechnical Commission

IPCC Intergovernmental Panel on Climate Change

IPCM Information Policy and Compliance Manager

IS Information Systems

ISSC Information Systems Strategy Committee

IT Information Technology

JANET The United Kingdom‘s Education and Research Network

JISC Joint Information Systems Committee

LA Lead Authors

LIA Little Ice Age

MBH Mann, Bradley and Hughes

MM McKitrick and Michaels

### 9

M&M McIntyre and McKitrick

MWP Medieval Warm Period

NAO North Atlantic Oscillation

NASA National Aeronautics and Space Administration

NCAR National Center for Atmospheric Research

NCDC National Climatic Data Center

NMO National Meteorological Office

NOAA National Oceanic and Atmospheric Administration

PERL Practical Extraction and Reporting Language

PI Principal Investigator

REE Research, Enterprise and Engagement

S&B Soon and Baliunas

SAP Scientific Assessment Panel

SPM Summary for Policy Makers

TAR Third Assessment Report of the IPCC in 2001

UEA University of East Anglia

UHI Urban Heat Island

UKRIO United Kingdom Research Integrity Office

UN United Nations

UNEP United Nations Environment Programme

WMO World Meteorological Organization
WWR World Weather RecordsCHAPTER 1: EXECUTIVE SUMMARY



### 10 

## CHAPTER 1: EXECUTIVE SUMMARY

1. The main findings of the Independent Climate Change E-mails Review (―the Review‖) are set out in Section 1.3 below, and the main recommendations in Section 1.4. We comment in Section 1.5 on some of the more general issues raised by the Review that we think are important about the context in which scientists operate and in which science contributes to public policy.



## 1.1 Introduction

2. In November 2009, approximately 1000 e-mails from the Climatic Research Unit (CRU) of the University of East Anglia (UEA) were made public without authorisation.

3. CRU is a small research unit which over the last 30 years has played an important role in the development of climate science, in particular in their work on developing global temperature trends.

4. The e-mails fuelled challenges to the work of CRU, to the reliability of climate science generally, and to the conclusions of the Intergovernmental Panel on Climate Change (IPCC). All this happened shortly before the Copenhagen Summit, and was extensively referred to there.

5. In response, the UEA commissioned two inquiries. The first led by Lord Oxburgh, into the science being undertaken at CRU, has already reported. This document is the report of the second inquiry

– The Independent Climate Change E-mails Review

–

which examines the conduct of the scientists involved and makes recommendations to the University of East Anglia. Our inquiry addresses a number of important allegations that were made following the e-mail release.

6. The allegations relate to aspects of the behaviour of the CRU scientists, such as their handling and release of data, their approach to peer review, and their role in the public presentation of results.

7. The allegations also include the assertion that actions were taken to promote a particular view of climate change by improperly influencing the process of advising policy makers. Therefore we have sought to understand the significance of the roles played by those involved from CRU and of the influence they had on the relevant outcomes.

8. The Review examines the honesty, rigour and openness with which the CRU scientists have acted. It is important to note that we offer no opinion on the validity of their scientific work. Such an outcome could only come through the normal processes of scientific debate and not from the examination of e-mails or from a series of interviews about conduct.CHAPTER 1: EXECUTIVE SUMMARY



### 11 

## 1.2 The Review Process

9. The approach taken by the Review was to identify and investigate the allegations to which the e-mails gave rise. This reflected our reading of the emails and the comments made on them. An online consultation was undertaken to ensure that the Team‘s initia l analysis of the allegations and concerns was sound. The method of investigation is explained in the relevant Chapters and Appendices to the report. The Review‘s evidence base is published on the website, which it intends to archive.

10. In addressing the allegations about CRU‘s impact on climate science , we sought evidence to place these into perspective:

On handling global temperature data, we went to global primary sources and tested how data was handled. On tree-ring temperature reconstructions, we looked at the overall picture painted in Chapter 6 of the Fourth Assessment Report of the IPCC in 2007 (AR4) and examined the influence of CRU. On peer review, we sought independent input (from the Editor of The Lancet ) on how the system works, to provide a context for our judgement. On influencing the IPCC process, we sought advice from the Review editors on the role individual contributors can play.

11. This work provided a context in which we considered the evidence about the specific allegations made in the submissions and identified in our interviews with CRU and others.

12. Reflecting this approach, the report and conclusions are set out as follows. The heart of the report lies in Chapters 6 through 10 where the important allegations arising from the e-mail release are examined. Chapters 2 and 3 contain introductory material, 4 deals with the body of e-mails and 5 presents important contextual material. The report concludes with Chapter 11 on other governance issues.



## 1.3 Findings

13. Climate science is a matter of such global importance, that the highest standards of honesty, rigour and openness are needed in its conduct. On the specific allegations made against the behaviour of CRU scientists, we find that their rigour and honesty as scientists are not in doubt.

14. In addition, we do not find that their behaviour has prejudiced the balance of advice given to policy makers. In particular, we did not find any evidence of behaviour that might undermine the conclusions of the IPCC assessments.

15. But we do find that there has been a consistent pattern of failing to display the proper degree of openness , both on the part of the CRU scientists and on the part of the UEA, who failed to recognise not only the significance of statutoryCHAPTER 1: EXECUTIVE SUMMARY



### 12 requirements but also the risk to the reputation of the University and, indeed, to the credibility of UK climate science.


## 1.3.1 Land Station Temperatures

16. On the allegation of withholding temperature data, we find that CRU was not in a position to withhold access to such data or tamper with it. We demonstrated that any independent researcher can download station data directly from primary sources and undertake their own temperature trend analysis.

17. On the allegation of biased station selection and analysis, we find no evidence of bias. Our work indicates that analysis of global land temperature trends is robust to a range of station selections and to the use of adjusted or unadjusted data. The level of agreement between independent analyses is such that it is highly unlikely that CRU could have acted improperly to reach a predetermined outcome. Such action would have required collusion with multiple scientists in various independent organisations which we consider highly improbable.

18. On the allegation of withholding station identifiers we find that CRU should have made available an unambiguous list of the stations used in each of the versions of the Climatic Research Unit Land Temperature Record (CRUTEM) at the time of publication. We find that CRU‟s responses to reasonable requests for information were unhelpful and defensive.

19. The overall implication of the allegations was to cast doubt on the extent to which CRU‟s work in this area could be trusted and should be relied upon
and we find no evidence to support that implication.


## 1.3.2 Temperature Reconstructions from Tree Ring Analysis

20. The central implication of the allegations here is that in carrying out their work, both in the choices they made of data and the way in which it was handled, CRU scientists intended to bias the scientific conclusions towards a specific result and to set aside inconvenient evidence. More specifically, it was implied in the allegations that this should reduce the confidence ascribed to the conclusions i n Chapter 6 of the IPCC 4 th 

### Report, Working Group 1 (WG1).

21. We do not find that the way that data derived from tree rings is described and presented in IPCC AR4 and shown in its Figure 6.10 is misleading. I n particular, on the question of the composition of temperature reconstructions, we found no evidence of exclusion of other published temperature reconstructions that would show a very different picture. The general discussion of sources of uncertainty in the text is extensive, including reference to divergence. In this respect it represented a significant advance on the IPCC Third Assessment Report (TAR).

22. On the allegation that the phenomenon of “divergence” may not have been properly taken into account when expressing the uncertainty associated with reconstructions, we are satisfied that it is not hidden and that theCHAPTER 1: EXECUTIVE SUMMARY



### 13 subject is openly and extensively discussed in the literature, including CRU

papers.

23. On the allegation that the references in a specific e- mail to a „trick‟ and to „hide the decline‟ in respect of a 1999 WMO report figure show evidence of intent to paint a misleading picture, we find that, given its subsequent iconic sign ificance (not least the use of a similar figure in the IPCC Third Assessment Report), the figure supplied for the WMO Report was misleading. We do not find that it is misleading to curtail reconstructions at some point per se , or to splice data, but we believe that both of these procedures should have been made plain

– ideally in the figure but certainly clearly described in either the caption or the text.

24. On the allegations in relation to withholding data, in particular concerning the small sample size of the tree ring data from the Yamal peninsula, CRU did not withhold the underlying raw data (having correctly directed th e single request to the owners). But it is evidently true that access to the raw data was not simple until it was archived in 2009 and that this delay can rightly be criticized on general principles. In the interests of transparency, we believe that CRU should have ensured that the data they did not own, but on which their publications relied, was archived in a more timely way.


## 1.3.3 Peer Review and Editorial Policy

25. On the allegations that there was subversion of the peer review or editorial process we find no evidence to substantiate this in the three instances examined in detail. On the basis of the independent work we commissioned (see Appendix 5) on the nature of peer review, we conclude that it is not uncommon for strongly opposed and robustly expressed positions to be taken up in heavily contested areas of science. We take the view that such behaviour does not in general threaten the integrity of peer review or publication.



## 1.3.4 Misuse of IPCC Process

26. On the allegations that in two specific cases there had been a misuse by CRU scientists of the IPCC process, in presenting AR4 to the public and policy makers, we find that the allegations cannot be upheld. In addition to taking evidence from them and checking the relevant records of the IPCC process, we have consulted the relevant IPCC review Editors. Both the CRU scientists were part of large groups of scientists taking joint responsibility for the relevant IPCC Working Group texts, and were not in a position to determine individually the final wording and content.CHAPTER 1: EXECUTIVE SUMMARY



### 14 

## 1.3.5 Compliance with the Freedom of Information Act (FoIA) and the Environmental Information Regulations (EIR)

27. On the allegation that CRU does not appear to have acted in a way consistent with the spirit and intent of the FoIA or EIR, we find that there was unhelpfulness in responding to requests and evidence that e-mails might have been deleted in order to make them unavailable should a subsequent request be made for them. University senior management should have accepted more responsibility for implementing the required processes for FoIA and EIR compliance.



## 1.3.6 Other Findings on Governance

28. Given the significance of the work of CRU, UEA management failed to recognise in their risk management the potential for damage to the University‟s reputation fuelled by the controversy over data access.



## 1.4 Recommendations

29. Our main recommendations for UEA are as follows:

Risk management processes should be directed to ensuring top management engagement in areas which have the potential to impact the reputation of the university. Compliance with FoIA/EIR is the responsibility of UEA faculty leadership and ultimately the Vice-Chancellor. Where there is an organisation and documented system in place to handle information requests, this needs to be owned, supported and reinforced by University leadership. CRU should make available sufficient information, concurrent with any publi cations, to enable others to replicate their results.



## 1. 5 Broader Issues

30. Our work in conducting the Review has led us to identify a number of issues relevant not only to the climate science debate but also possibly more widely, on which we wish to comment briefly.

31. The nature of scientific challenge. We note that much of the challenge to CRU‘s work has not always followed the conventional scientific method of checking and seeking to falsify conclusions or offering alternative hypotheses for peer review and publication. We believe this is necessary if science is to move on, and we hope that all those involved on all sides of the climate science debate will adopt this approach.

32. Handling Uncertainty

– where policy meets science. Climate science is an area that exemplifies the importance of ensuring that policy makers

–CHAPTER 1: EXECUTIVE SUMMARY



### 15 particularly Governments and their advisers, N on -Governmental Organisation s and other lobbyists

– understand the limits on what scientists can say and with what degree of confidence. Statistical and other techniques for explaining uncertainty have developed greatly in recent years, and it is essential that they are properly deployed. But equally important is the need for alternative viewpoints to be recognized in policy presentations, with a robust assessment of their validity, and for the challenges to be rooted in science rather than rhetoric.

33. Peer review - what it can/cannot deliver. We believe that peer review is anessential part of the process of judging scientific work, but it should not be overrated as a guarantee of the validity of individual pieces of research, and the significance of challenge to individual publication decisions should be not exaggerated.

34. Openness and FoIA. We support the spirit of openness enshrined in the FoIA and the EIR. It is unfortunate that this was not embraced by UEA, and we make recommendations about that. A well thought through publication scheme would remove much potential for disruption by the submission of multiple requests for information. But at the level of public policy there is need for further thinking about the competing arguments for the timing of full disclosure of research data and associated computer codes etc, as against considerations of confidentiality d uring the conduct of research. There is much scope for unintended

consequen ces that could hamper research: US experience is instructive. We recommend that the ICO should initiate a debate on these wider issues.

35. Handling the blogosphere and non traditional scientific dialogue. One of the most obvious features of the climate change debate is the influence of the blogosphere. This provides an opportunity for unmoderated comment to stand alongside peer reviewed publications; for presentations or lectures at learned conferences to be challenged without inhibition; and for highly personalized critiques of individuals and their work to be promulgated without hindrance. This is a fact of life, and it would be foolish to challenge its existence. The Review team would simply urge all scientists to learn to communicate their work in ways that the public can access and understand. That said, a key issue is how scientists should be supported to explain their position, and how a public space can be created where these debates can be conducted on appropriate terms, where what is and is not uncertain can be recognised.

36. Openness and Reputation. An important feature of the blogosphere is the

e xtent to which it demands openness and access to data. A failure to recognise this and to act appropriately, can lead to immense reputational damage by feeding allegations of cover up. Being part of a like minded group may provide no defence. Like it or not, this indicates a transformation in the way science has to be conducted in this century.

37. Role of Research Sponsors . One of the issues facing the Review was the release of data. At various points in the report we have commented on the formal requirements for this. We consider that it would make for clarity for researchers if funders were to be completely clear upfront in their requirements for the release of data (as well as its archiving, curation etc).CHAPTER 1: EXECUTIVE SUMMARY



### 16 38. The IPCC. We welcome the IPCC‘s decision to review its processes, and can only stress the importance of capturing the range of viewpoints and reflecting appropriately the statistical uncertainties surrounding the data it assesses. Our conclusions do not make a judgement on the work of IPCC, though we acknowledge the importance of its advice to policy makers.CHAPTER 1: EXECUTIVE SUMMARY



### 17CHAPTER 2: INTRODUCTION



### 18 

## CHAPTER 2: INTRODUCTION

2.1 Background

1. Prior to the 1960‘s there had been little investigation of past climatic changes and variability, except by geologists and botanists and on geological timescales. The Climatic Research Unit (CRU) in the UEA, established in 1972, was one of the first institutes in the world to address the science of climate change on more recent timescales. Its objective was ―to establish the past record of climate over as much of the world as possible, as far back in time as was feasible, and in enough detail to recognis e the basic processes, interactions, and evolutions in the earth‘s fluid envelopes and those involving the earth‘s crust and its vegetation cover‖. A useful history of its work is available on its website: http://www.cru.uea.ac.uk/cru/about/history.

2. CRU is a small organisation: at present there are around 16 staff, with 3.5 established posts and the rest postgraduate students and post-doctoral researchers. Two of the main areas of its work are the focus of this review: the development of methodologies for calculating the extent to which the average temperature of the earth ‘ s land masses is changing, using instrumental temperature measurements made over the past 160 years (the period for which reliable measurements have been available); and the estimation of the earth ‘ s temperature over the last millennium, using tree ring data as a proxy for temperature.

3. One of CRU‘s most important contributions to climate science is the production of a land based, gridded temperature data set showing how the temperature has varied year by year since 1850 relative to the 1961 to 1990 average. This work was started in 1978 and continues today. CRU is now one of a number of organisations working in this area. Others carrying out similar development of temperature records include the Goddard Institute for Space Studies (GISS) and the National Climatic Data Center (NCDC).

4. Over the period that CRU has been in existence and especially over the last 20 years, there has been a transformation in the importance attached to climate science. One clear indication of this was the creation of the Intergovernmental Panel on Climate Change (IPCC) in 1988, a body established by the United Nations Environment Programme (UNEP) and the World Meteorological Organization (WMO) ―to provide the world with a clear scientific view on thecurrent state of climate change and its potential environmental and socioeconomic consequences‖. It assesses the most recent scientific, technical and socio-economic information produced worldwide that is relevant to the understanding of climate change. Members of CRU have played several significant roles on the IPCC. In its successive assessment reports the IPCC has sought to achieve a scientific consensus, but many continue to challenge the basis of its work and its conclusions.

5. The pioneering work conducted by CRU therefore began to assume a great deal of significance in the international debate that surrounded climate science. Not surprisingly, given the enormous level of public interest in the subject, the debateCHAPTER 2: INTRODUCTION



### 19 soon moved out of the confines of climate scientists and became highly polarized in websites, journals and conferences across the world. As a result, the work conducted by CRU became the focus of intense scrutiny and challenge with multiple demands from both fellow scientists and laymen for background information and data.

6. In November 2009, the nature of the debate and challenge took on a whole new significance when approximately 1000 of CRU‘s e-mails were made public without authorization. Th is material fuelled more challenges to their work, to the reliability of climate science generally, and to the conclusions of the IPCC. All this happened shortly before the Copenhagen Summit, and was extensively referred to there.



## 2.2 The Review

7. The material in the e-mails led to a set of allegations against the leading members of CRU, focusing on whether CRU had operated in accordance with best scientific practice at the relevant time, and in particular: whether data had been manipulated or suppressed; whether peer review and the dissemination of findings had bee n properly handled; and whether CRU had complied with the Freedom of Information Act.

8. In response to this, the Vice-Chancellor of the UEA established the Independent Climate Change E-mails Review, to be led by Sir Muir Russell. Sir Muir was

g iven complete freedom to develop the terms of reference as necessary and to assemble an appropriate team and appropriate support; and was asked if possible to report in spring 2010. The terms of reference are in Chapter 3 and details of the team members are in Appendix 1.

9. Two other formal inquiries have addressed the matter. First, the House of Commons Science and Technology Committee held a hearing on 1 March 2010, reporting on 31 March 2010 1 

### . Secondly, the Vice-Chancellor commissioned Lord Oxburgh to review CRU ‘ s scientific output. The Oxburgh Scientific Assessment Panel and this Review proceeded entirely independently, though the latter took steps to ensure, following the House of Commons Science and Technology Committee ‘ s Report, that Lord Oxburgh was aware of the approach it was taking to issues that might bear on his work. Lord Oxburgh ‘ s report was published on 14 April 2010 2



### .

10. In addition investigations have been initiated by the police, looking at the circumstances of the unauthoris ed disclosure of the e-mails; and by the ICO

, looking at compliance with the Data Protection Act, the FoIA and the EIR.

11. Responding to a joint request by the Chairman of the IPCC and the Secretary-

1 Science and Technology Committee - Eighth Report: The disclosure of climate data from the Climatic Research Unit at the University of East Anglia

2 Report by Lord Oxburgh‘s Science Assessment Panel http://www.uea.ac.uk/mac/comm/media/press/CRUstatements/S APCHAPTER 2: INTRODUCTION



### 20 General of the United Nations, an independent review of the IPCC‘s processes and procedures is now under way by the InterAcademy Council (IAC), chaired by economist Harold T. Shapiro, former president of Princeton University.

12. Although the focus of this Review is quite clearly on the behaviour of a number of scientists, it is also apparent that there are many wider issues raised by the events described above. These include issues such as the workings of the peer review system, the reporting of uncertainty in the translation of scientific findings into policy, the handling and release of data, and the role of research funders. As a contribution to the wider scientific debate about the conduct of science in the 21 st 

### century, this Review also considers these questions.CHAPTER 3: TE RMS OF REFERENCE AND METHOD OF ENQUIRY



### 21CHAPTER 3: TE RMS OF REFERENCE AND METHOD OF ENQUIRY



### 22 

## CHAPTER 3: TERMS OF REFERENCE AND METHOD OF ENQUIRY

3.1 Terms of Reference

1. The Review‘s terms of reference are as follows:

― The Independent Review will investigate the key allegations that arose from a series of hacked* e-mails from CRU. The Review will:

Examine the hacked e-mail exchanges, other relevant e-mail exchanges and any other information held at CRU to determine whether there is any evidence of the manipulation or suppression of data which is at odds with acceptable scientific practice and may therefore call into question any of the research

outcomes. Review CRU ‘ s policies and practices for acquiring, assembling, subjecting to peer review and disseminating data and research findings, and their compliance or otherwise with best scientific practice.
Review CRU ‘ s compliance or otherwise with the University ‘ s policies and practices regarding requests under the Freedom of Information Act (  ̳ the
Fo IA ‘ ) and the Environmental Information Regulations (  ̳ the EIR ‘ ) for the release of data. Review and make recommendations as to the appropriate management, governance and security structures for CRU and the security, integrity and release of the data it holds. ‖

*Note: The word  ̳ hacked ‘ as contained in the Review ‘ s terms of reference has been challenged in submissions to the Review, on the basis that the means by which the unauthorized disclosure of the e-mails was made has not been established. This matter is subject to police enquiries and the Review has made no judgment on the question.



## 3 .2 Method of Enquiry

2. The Review sought to operate in an open and transparent way. This is described in the  ̳ Approach and Work Plan ‟ paper (Appendix 2) made public at the launch at the Science Media Centre in February 2010. The Review called for submissions,which it undertook to publish on its website: http://www.ccereview.org/Evidence.php

3. To provide a focus for submissions the Review produced and placed on its website an  ̳ Issues for Examination ‟ paper (Appendix 3) which addresses the main allegations made against the members of CRU. This paper set out our initial understanding based on a preliminary reading of the e-mails. Approximately 100 submissions were then received, including one from CRU itself. Some provided fur ther background considered relevant by the authors, but there was no suggestion in the submissions that the original statement of the issues was fundamentally misdirected. As its work progressed, the team kept the issues paper

c onstantly under review and gradually developed a sharper focus on the keyCHAPTER 3: TE RMS OF REFERENCE AND METHOD OF ENQUIRY



### 23 allegations.

4. Thus this report is structured to address the principal allegations. It is important to

re cognise that this is not a detailed inquiry into the precise meaning of every email.

5. The team proceeded to investigate the allegations by interviewing members of CRU and others from the University. We considered that the nature of our inquiry was such that holding public hearings to gather evidence, as some had urged, would be unlikely to add significant value over and above the written record. Nor have we produced transcripts of the interviews. This is because our conclusions are founded on information given in submissions and at interviews relating to facts that can be checked and referenced, rather than on interview testimony as such. The team found that this process helped it follow up key points, leading to supplementary submissions and references.

6. The team did not carry out interviews other than with CRU and other UEA staff (apart from preliminary discussions with ICO and the police and interviews with two relevant IPCC Review Editors). We recognise that natural justice requires that those in respect of whom findings will be made should have an opportunity to be heard: this does not apply to the authors of submissions and other parties, in respect of whom the Review has made no findings.

7. Under our p ublication policy notes of the t eam‘s meetings and interviews have been placed on the website. Interview notes have been checked for factual accuracy with those involved. Only those that are potentially sensitive, such as notes of discussions with the police, have been withheld. The Team has sought to

pu blish all the submissions received unless they were explicitly described as confidential, subject only to the redaction, on legal advice, of content the publication of which could have led to legal action. In addition, one submission ha s been withheld on legal advice. The Review has invited the author to make it available directly to enquirers. It is a matter for the author whether to proceed in this way. This submission has been given full consideration.

8. A full list of submissions and details of meetings can be found in Appendix 4.

9. It is important to note that the allegations relate to aspects of the behaviour of the CRU scientists and not to the content of their work and hence this Review addressed scientific questions only to the extent necessary to place this behaviour in context.

10. On the subject of peer review, in addition to investigating the specific allegations the team commissioned a contextual paper from a distinguished Journal editor

– Dr Richard Horton of The Lancet

– on which Elizabeth Wager, Chair of the Committee on Publication Ethics (COPE), was invited to comment in a personal capacity. This material forms Appendix 5 and is referred to Chapters 5 and 8.

11. The Review maintained contact with the police and the ICO. It was important to ensure that the work of the team did not compromise their statutory responsibilities, and also to avoid duplication of effort. The t eam ‘ s findings andCHAPTER 3: TE RMS OF REFERENCE AND METHOD OF ENQUIRY



### 24 recommendations respect this division of responsibilities and the fact that investigations are continuing.

12. The Review intends that the website and the submissions and evidence published on the website should be archived by the British Library Web Archive.CHAPTER 3: TE RMS OF REFERENCE AND METHOD OF ENQUIRY



### 25CHAPTER 4: CONTEXT OF THE E-MAILS



### 26 

## CHAPTER 4: CONTEXT OF THE E-MAILS

4.1 Characterising the E-mails

1. This Chapter gives an overview of the e-mails and seeks to relate their content to key events happening in the climate change world, thus giving context to the criticism of CRU.

2. The information released comprises a very small (less than 0.3%) subset of files which were held on the back-up server at CRU, which include e-mails and other documents

–

such as text files, Word documents, Excel spreadsheets, PDF documents, and computer code.

3. The focus of this Chapter is e-mails which spanned the period 7 th 

### March 1996 to
12 th 

### November 2009. The  ̳ primary ‘ e-mails number 1073 in total with 166 authors. There are more e-mails and authors if the associated e-mail chains are included. When printed on A4 paper the e-mails run to 3,375 pages and contain many embedded duplicates. Self-evidently each of the primary e-mails was either

se nt by or received by CRU members, but this is not the case for many of theassociated e-mail chains. Those who authored the largest numbers of primary emails are as follows.

Autho r Number Role
Philip Jones 174 Director, CRU, UEA and Coordinating Lead Author IPCC 4 th 

### Assessment Report

Michael
Mann 140 Director, Earth System Science Centre, Pennsylvania State University (from 2005), and Lead Author IPCC 3 rd 

### Assessment Report
Keith Briffa 117 Professor, CRU, UEA and Lead Author IPCC 4 th 

### Assessment Report

Jonathan Overpeck 90 Institute Director, University of Arizona and Coordinating Lead Author IPCC 4 th 

### Assessment Report
Tim Osborn 59 Academic Fellow, CRU, UEA and Contributing Author IPCC 4 th 

### Assessment Report

Ben Santer 51 Researcher, Lawr ence Livermore National Lab, US and Contributing Author IPCC 4 th 

### Assessment Report

Tom Wigley 35 Scientist, University Corporation for Atmospheric Research, Contributing Author IPCC 4 th 

### Assessment Report and a former Director of CRU

4. The e-mails relate to a number of the major developments in recent climate science from the Kyoto Summit in December 1997, through the 4 th 

### Assessment Report of the IPCC in 2007, to the run-up to the 15 th 

### C onference of the Parties (COP) in Copenhagen. Since CRU played a significant role in providing scientific input to these events, the release of the emails resulted in reduced globalCHAPTER 4: CONTEXT OF THE E-MAILS



### 27 confidence in climate science and more specifically in the findings of the IPCC.



## 4.2 The Timeline

5. The  ̳ story ‘ which underlies the selected e-mails, and hence is reflected in them, is summarised in the table and text which follow.

Year Event

1997 Kyoto Summit

1998 MBH98 1 

### including what has become known as the ― Hockey Stick ‖

1999 MBH99 2 

### reconstruction extended to 1000 AD WMO Statement on the Status of the Global Climate in 1999 3



### 2001 IPCC 3 rd 

### Assessment Report 4



### 2002 Esper et al 2002 5



### 2003 Soon & Baliunas 2003 6



### , McIntyre &McKitrick 2003 7 

### criticism of MBH Mann et al 2003 8

, 9



### 2004 von Storch 2004 10 

### questioning the statistical methods used in MBH Launch of RealClimate 11 

### website

1 M.E. Mann, R.S. Bradley and M.K. Hughes, ― Global-scale temperature patterns and climate forcing over the past six centuries ‖

, Nature 392, 779-787 23 April (1998)

2 M.E. Mann, R.S. Bradley and M.K. Hughes, ― Northern hemisphere temperatures during the past millennium: Inferences, uncertainties, and limitations ‖

, Geophysical Research Letters , Volume 26. No. 6, pp. 759-762 (1999)

3 http://www.wmo.ch/pages/prog/wcp/wcdmp/statemnt/wmo913.pdf

4 http://www.grida.no/publications/other/ipcc_tar/

5 J. Esper, E.R. Cook, F.H. Schweingruber, ― Low-Frequency Signals in Long Tree-Ring Chronologies for Reconstructing Past Temperature Variability ‖

, Science , Volume 295, 22 March (2002) pp. 2250- 2253

6 W. Soon, S. Baliunas, ― Proxy climatic and environmental changes of the past 1000 years ‖

, Climate Research Volume 23: 89-110, January 2003

7 S. McIntyre, R. McKitrick, ― Corrections to the Mann et. al. (1998) proxy data base and Northern hemispheric average temperature series ‖

, Energy & Environment

, Volume 14, No. 6 (2003) pp. 751

- 771

8 M.E. Mann and P.D. Jones, ― Global surface temperatures over the past two millennia ‖

, Geophysical Research Letters Volume 30, No. 15, 1820, August 2003

9 M.E Mann, C.M Ammann, R.S. Bradley, K.R. Briffa, T.J. Crowley, M.K. Hughes, P.D. Jones, M. Op penheimer, T.J. Osborn, J.T. Overpeck, S. Rutherford, K.E. Trenberth, T.M.L. Wigley, ― On past temperatures and anomalous late 20 th century warmth ‖

, Eos , 84, 256-258, 2003.

10 H. von Storch, E. Zorita, J.M. Jones, Y. Dimitriev, F. González-Rouco, S.F.B. Tett

, ― Reconstructing Past Climate from Noisy Data ‖

, Science , 22 October 2004, Volume 306. no. 5696, pp. 679

– 682

11 http://www.realclimate.org/CHAPTER 4: CONTEXT OF THE E-MAILS



### 28 Year Event (continued)
2005 UK Freedom of Information Act 12



### Launch of ClimateAudit 13 

### website McIntyre &McKitrick 2005 14 

### criticism of MBH Rutherford et al 2005 15 

### defence of MBH
2006 US National Research Council review 16 

### US Committee on Energy & Commerce (Wegman) statistical review 17



### 2007 IPCC 4 th 

### Assessment Report 18



### Wahl & Ammann 2007 19 

### defence of MBH
2009 15 th 

### COP in Copenhagen

CRU‘s work assumed progressively greater significance as it became apparent that both the global temperature record and tree ring data it was collecting and analysing were being cited prominently in the work of the IPCC.

Building in part on the work of CRU, Mann, Bradley & Hughes (MBH) published a paper in Nature in 1998 20 

### which sought to reconstruct historic temperatures back to 1400 AD, well before significant numbers of instrumental climate records began in around 1850. They usedmeasurements including those from tree rings, ice cores and corals, socalled proxy data, which can reflect local temperature changes. This paper was updated in 1999 21 

### with the reconstruction extended back to 1000 AD in the Northern Hemisphere. The resulting temperature profile had a  ̳ hockey stick ‘ shape. In particular it reduced the significance of what is termed the Medieval Warm Period (MWP), that is believed to have reached its peak between about 950 and 1100 AD, and the Little Ice Ag e (LIA) (from about 1500 to 1850 AD, and centered around 1700 AD), relative to previous representations. Temperatures over the last 50 years appeared unprecedented in the past thousand years. These data were used to create an iconic representation of anthropogenic global warming, as supplied by Jones, in the 1999 World Meteorological Organisation for its

12 http://www.ico.gov.uk/what_we_cover/freedom_of_information.aspx

13 http://climateaudit.org/

14 S. McIntyre, R. McKitrick, ― Hockey sticks, principal components, and spurious significance ‖

, Geophysical Research Letters

, Volume 32, No. 3 (2005)

15 S. Rutherford, M.E. Mann, T.J. Osborn, R.S. Bradley, K.R. Briffa, M.K. Hughes, P.D. Jones,

― Proxy-based Northern Hemisphere surface temperature reconstructions: sensitivity to methodology, predictor network, target season and target domain ‖

, Journal of Climate (2005); 18: 2308- 2329

16 Surface Temperature Reconstructions for the Last 2,000 Years http://www.nap.edu/catalog.php?record_id=11676

17 Ad Hoc Committee Report on the  ̳Hockey Stick‘ Global Climate Reconstruction (2006) for the US Committee on Energy and Commerce

18 http://www.ipcc.ch/

19 E.R. Wahl, C.M. Ammann

, ― Robustness of the Mann, Bradley, Hughes reconstruction of Northern Hemisphere surface temperatures: examination of criticisms based on the nature and processing of proxy climate evidence ‖ Climatic Change , Volume 85, Numbers 1-2, pp.33-69, November 2007

20 Ibid. 1

21 Ibid. 2CHAPTER 4: CONTEXT OF THE E-MAILS



### 29 Statement on the Status of the Global Climate in 1999 22 

### . They were also referenced in the Third Assessment Report (TAR) of the IPCC in 2001 23



### .

The IPCC process occurs over a number of years, with drafts being produced periodically, leading up to the final report and summary for policy makers (SPM). The process is clearly documented 24 

### with extensive voluntary involvement of many eminent scientists in authoring and reviewing chapters of the report. Key players in CRU were involved inthe development of both the TAR and AR4 in 2007 and many of the emails relate to their involvement in the process.

Questions were increasingly asked about MBH. In 2002 Esper published

a paper 25 

### which suggested that MBH had underestimated the strength of the MWP, but it was Soon & Baliunas ‘ s (S&B) papers published in Climate Research 26 

### and with minor amendments in Energy &
Environmen t (E&E) 27 

### which first challenged MBH ‘ s results directly. An e-mail by Osborn circulating S&B ‘ s 2003 paper 28 

### is one of the most duplicated in the released e-mails triggering nine of the subsequently selected e-mail chains. S&B ‘ s papers were followed in 2003 by another paper by McIntyre and McKitrick (M & M) 29 

### also critical of MBH. At issue in the whole exchange of e-mails were the following key scientific issues:-

(i) the selection of tree ring data (particularly those from Yamal

and the Polar Urals); (ii) the statistical methods for extracting information; (iii) the record of local and hence global temperatures (especially the impact of the urban heat island (UHI) effect); (iv) relating proxy records to instrumental temperature records; and (v) uncertainties in comparing recent warmth with that over the last 2000 years.

Prior to the publication of S&B ‘ s papers in 2003 those critical of MBH had not had a paper published in a mainstream journal. The publications in Climate Research and Energy & Environment were significant not only because they challenged MBH but also because they had been peer reviewed. Not only were rebuttals published in 2003 by Mann, Jones, Briffa, Osborn et al 30

, 31



### , but also the process of peer review at Climate Research was questioned. The editor, de Freitas, sought initially to

22 Ibid. 3

23 Ibid. 4

24 http://www.ipcc.ch/organization/organization_procedures htm

25 Ibid. 5

26 Ibid. 6

27 W. Soon, S. Baliunas, C. Idso, S. Idso. D.R. Legates, ― Reconstructing climatic and environmental changes of the past 1000 years: a reappraisal ‖

, Energy & Environment , Volume 14: Nos. 2 & 3, 2003, pp. 233- 296

28 Ibid. 6

29 Ibid. 7

30 Ibid. 8

31 Ibid. 9CHAPTER 4: CONTEXT OF THE E-MAILS



### 30 defend himself as the e-mail extract below shows but he ultimately resigned as did members of the editorial board of Climate Research. The matter is discussed in detail in Section 8.3 of Chapter 8.

I n an e-mail dated 18/6/03 (1057944829.txt) de Freitas wrote:

“I have spent a considerable amount of my time on this matter and had my integrity attacked in the process. I want to emphasize that the people leading this attack are hardly impartial observers. Mike himself refers to "politics" and political incitement involved. Both Hulme and Goodess are from the Climate Research Unit of UEA that is not particularly well known for impartial views on the climate change debate. The CRU has a large stake in climate change research funding as I understand it pays the salaries of most of its staff.”

In 2004 von Storch 32 

### questioned the statistical methods used in MBH and, at around the same time, the RealClimate 33 

### and Climate Audit 34 

### websites were launched, the former defending and the latter critical of majority climate science. This marked a new phase of more public climate science debate and criticism, and was seen as an unwelcome development by those in CRU, who wanted criticism to follow the normal channels of private peer review and publication. This view remained relatively unchanged for over a decade, as shown by the following:

In an e-mail dated 6/5/99 (926031061.txt) Jones wrote:

“I must admit to having little regard for the Web. Living over here makes that easier than in the US - but I would ignore the so-called skeptics until they get to the peer-review arena. I know this is harder for you in the US and it might become harder still at your new location. I guess it shows though that what we are doing in [sic] important. The skeptics are fighting a losing battle.”

In an e-mail dated 30/9/09 (1254323180.txt) Jones wrote:

“Another issue is science by blog sites - and the then immediate response mode. Science ought to work through the peer-review system..... sure you've said all these things before. We're getting a handful of nasty emails coming and requests for comments on other blog sites.”

Critics of CRU claimed that the y were unable to reproduce CRU‘s work du e to a lack of access to data. The UK EIR and FoIA 35 

### should have encouraged the release of information, but this does not seem to have been the initial result. The volume of requests grew over time, more formally under the FoIA starting in 2007 and spectacularly in 2009, particularly related to details of the CRUTEM land temperature dataset s

32 Ibid. 10

33 Ibid. 11

34 Ibid. 13

35 Ibid. 12CHAPTER 4: CONTEXT OF THE E-MAILS



### 31 and the role CRU played in the development of IPCC AR4.

In an email dated 7/5/04 (1083962601.txt) Jones wrote:

“Many of us in the paleo field get requests from skeptics (mainly a guy called Steve McIntyre in Canada) asking us for series. Mike and I are not sending anything, partly because we don't have some of the series he wants, also partly as we've got the data through contacts like you, but mostly because he'll distort and misuse them. Despite this, Mike and I would like to make as many of the series we've used in the RoG [Reviews of Geophysics] plots available from the CRU web page.”

We do not suggest that the allegations made against McIntyre are correct.

In the run up to IPCC AR4 further papers were published criticising (McIntrye & McKitrick 2005 36 

### ) and defending MBH (Rutherford 2005 37



### , Wahl and Ammann 2007 38 

### ). In parallel, political interest in AR4 grew

a nd in the United States of America two reviews were commissioned, one by the National Research Council 39 

### into surface temperature reconstructions for the last 2000 years and the other by the Committee on Energy and Commerce (Wegman) 40 

### into the quality of the statistical analysis used in MBH. These reviews were critical of MBH; nevertheless the AR4 SPM 41 

### contained the following statement.

“Average Northern Hemisphere temperatures during the second half of the 20 th 

### century were very likely higher than during any other 50-year period in the last 500 years and likely the highest in at least the past 1300 years.”

In the aftermath of AR4, attempts were made to demonstrate that the IPCC process had been abused by CRU members. Jones was accused of attempting to exclude reference to a 2004 paper by McKitrick and Michaels 42 

### , and Briffa of including reference to Wahl 2007 43 

### before it was in publication, contrary to IPCC rules.

These accusations were accompanied by Freedom of Information FOI requests. In an email dated 10/12/08 (1228922050.txt) Jones wrote:

“Anyway requests have been of three types - observational data, paleo data and who made IPCC changes and why. Keith has got all the latter

–

36 Ibid. 14

37 Ibid. 15

38 Ibid. 19

39 Ibid. 16

40 Ibid. 17

41 AR4 WG1 Summary for policymakers: http://www.ipcc.ch/publications_and_data/ar4/wg1/en/spmsspm-a-palaeoclimatic.html

42 R. McKitrick and P.J Michaels, ― A test of corrections for extraneous signals in gridded surface temperature data ‖

, Climate Research 26, 159-173, 2004

43 Ibid. 19CHAPTER 4: CONTEXT OF THE E-MAILS



### 32 and there have been at least 4. We made Susan aware of these - all came from David Holland. According to the FOI Commissioner's Office, IPCC is an international organization, so is above any national FOI. Even if UEA holds anything about IPCC, we are not obliged to pass it on, unless it has anything to do with our core business - and it doesn't!”



## 4 .3 What the E-mails Tell Us

6. Since the communication was assumed to be private, it was generally informal, using slang, jargon and acronyms. Now that the e-mails have become public, some are doubtless regretted by their authors. Indeed, some submissions have charac terised them as  ̳unprofessional‘, or as evidence of CRU‘s contribution to a  ̳poisoned atmosphere‘ in climate science. The question is whether this was unusual, or simply characteristic of the normal mode of expression in e-mail communication.

7. This is a well known issue in social psychology and communication. Early authoritative works 44

, 45

, 46 

### , reinforced by later authors, explore normative language in electronic communication in comparison with other written and oral forms. They demonstrate that e-mail communication is less inhibited than other written and spoken forms, and suggest reasons why. Extreme forms of language are frequently applied to quite normal situations by people who would never use it in other communication channels.

8. It is also clear from the submissions that it is possible to place different interpretations on the same phrase. In such circumstances, only the original author can really know what their intentions were. For example, on 16/11/99 (942777075.txt) Jones wrote his now infamous e-mail including the following:

“ I' ve just completed Mike's Nature trick of adding in the real temps to each series for the last 20 years (ie. from 1981 onwards) amd [ sic] from 1961 for Keith's to hide the decline. ”

McIntyre in his submission 47 

### to the Review Team, states

“The IPCC “trick” was not a “clever” mathematical method – it was merely the d eletion of inconvenient data after 1960.”

This compares with Jones‘s own commentary in the UEA submission 48 

### to the Review Team.

“The email was written in haste and for a limited and “informed” audience (the people that had provided data). The word “trick” was not intended to imply any

44 Sproull, L. and Kiesler, Sara. 1986. Reducing Social Context Cues: Electronic Mail in Organisational Communications. Management Science , 32, p.1492- 1512

45 Sproull, L. and Kiesler, S. 1991. Connections. MIT Press, Cambridge MA.

46 Shapiro, N and Anderson, R. 1985. Towards and Ethics and Etiquette for Electronic Mail. Rand Corporation.

47 McIntyre Submission (No.23), 1 March 2010, paragraph 22

48 Climatic Research Unit Submission (No.5), 1 March 2010, 3.1 on p.36CHAPTER 4: CONTEXT OF THE E-MAILS



### 33 deception, simply the “best way of doing or dealing with something”. The reconstruction from the tree-ring density network was not shown after 1960, and thus in this sense it is “hidden” – but justifiably so: excluding the anomalous

tree -ring density data is justified if the purpose is to illustrate the most likely course of temperature based on a combination of proxy and measured temperatures. Again, no deception was intended or practised.”

9. E-mails are rarely definitive evidence of what actually occurred. They are open to interpretation, but they are also indicative. Having identified specific allegations against CRU, based on the e-mails, the Review then sought to obtain evidence to substantiate or refute these allegations, as described in the subsequent Chapters of this report.

10. The presumption is that e-mails were selected to support a particular viewpoint. Recognising that they were a tiny fraction of those archived, the Review Team sought to learn more about the full contents of the back-up server. This attempt, summarised in Appendix 6, was largely unsuccessful due to the sheer scale of the task and ongoing police investigation.

11. In some instances the leaked e-mails contain statements which might be viewed as supporting the behavior of CRU and countering the allegations in paragraph 9 above. The following extracts from e-mails by Briffa are examples.

E-mail dated 22/9/99 (938018124.txt):

“ the early (pre-instrumental) data are much less reliable as indicators of global temperature than is apparent in modern calibrations that include them and when we don't know the precise role of particular proxies in the earlier portions of

recon struction it remains problematic to assign genuine confidence limits at multidecadal and longer timescales. ”

E-mail dated 3/2/06 (1138995069.txt):

“ we are having trouble to express the real message of the reconstructions - being scientifically sound in representing uncertainty , while still getting the crux of the information across clearly...We have settled on this version (attached) of the Figure which we hoe [sic] you will agree gets the message over but with the rigor required for such an important document. ”

12. On the other hand there are some e-mails which show that CRU members may have gone out of their way to make life difficult for their critics, just as they perceived the critics to be frustrating their work. Some examples follow:

Jones e-mail dated 27/4/05 (1114607213.txt):

“ I got this email from McIntyre a few days ago. As far as I'm concerned he has the data - sent ages ago. I'll tell him this, but that's all - no code. If I can find it, it is likely to be hundreds of lines of uncommented fortran ! I recall the program did a lot more that just average the series. I know why he can't replicate the results early on - it is because there was a variance correction for fewer series.”CHAPTER 4: CONTEXT OF THE E-MAILS



### 34 Jones e-mail dated 3/12/08 (1228330629.txt ):

“ The inadvertent email I sent last month has led to a Data Protection Act request sent by a certain Canadian, saying that the email maligned his scientific credibility with his peers! If he pays 10 pounds (which he hasn't yet) I am supposed to go through my emails and he can get anything I've written about him. About 2 months ago I deleted loads of emails, so have very little - if anything at all.”

Osborn e-mail dated 23/6/08 (1214228874.txt ):

“ I've just had a quick look at CA. They seem to think that somehow it is an advantage to send material outside the formal review process. But *anybody* could have emailed us directly. It is in fact a disadvantage! If it is outside the formal process then we could simply ignore it, whereas formal comments had to be formally considered. Strange that they don't realise this and instead argue for some secret conspiracy that they are excluded from! I'm not even sure if you sent me or Keith anything, despite McIntyre's conviction! But I'd ignore this guy's request anyway. If we aren't consistent in keeping our discussions out of the public domain, then it might be argued that none of them can be kept private. Apparently, consistency of our actions is important.”

13. During the work of the Review an additional concern surfaced, namely that of financial controls, for example see extract below. This is addressed in Chapter 11 (paragraph 18). In an e-mail dated 7/3/96 (826209667.txt) Shiyatov (A Russian tree ring researcher) wrote:

“It is important for us if you can transfer the ADVANCE money on the personal accounts which we gave you earlier and the sum for one occasion transfer (for example, during one day) will not be more than 10,000 USD. Only in this case we can avoid big taxes and use money for our work as much as possible. ”

14. Finding: The extreme modes of expression used in many e-mails are characteristic of the medium. Crucially, the e-mails cannot always be relied upon as evidence of what actually occurred, nor indicative of actual behaviour that is extreme, exceptional or unprofessional.

15. The Chapters which follow address specific behavioural allegations based in particular on a comparatively small number of e-mails. Despite the fact that all the e-mails released are unlikely to be representative of the larger set on the CRU back -up server, there nevertheless would appear to be a pattern of behaviour. The re is little doubt about the polarisation of views in the world of climate science, which has overstepped the line dividing heated scientific debate from outright hostility. One camp comprises the main authors of the e-mails in paragraph 3, who are acknowledged leaders in majority climate science as indicated by the roles they all played in the IPCC. The other camp is their critics, for whom pejorative terms such as ―prat, dishonest, appalling, rubbish and crap‖CHAPTER 4: CONTEXT OF THE E-MAILS



### 35 were used by some CRU members to refer to them or their work. More generally the majority climate scientists appear to have been united in their defence against criticism. Whilst perhaps understandable, given the nature and methods of criticism, some of which impugns their personal integrity as well as challenging their work, this may have blinded some CRU members to the possibility of merit therein. Such denial then fuelled yet further antagonism. There needs to be better communication, as well as greater openness enabling more scientific debate. We comment on this in Chapters 5 and 10.CHAPTER 5: THE CHANGING CONTEXT OF MODERN SCIENCE



### 36 

## CHAPTER 5: THE CHANGING CONTEXT OF MODERN SCIENCE

1. In this Chapter we review the changing context of modern science as a framework against which CRU‘s practices can be judged.

2. When CRU was created in the early 1970s, climate change was a relatively obscure area of science. But it has developed into an area of great political and public concern. CRU found itself ―in the eye of the storm‖ in generating much of the data necessary to address one of the most important issues of the century.

3. This has come at a time of transformation in the need for openness in the culture of publicly-funded science. It is being driven by a range of pressures including new recommendations from national academies addressing the timely release of data, increased demands for scientific input to public policy, important changes in the law and challenge from the blogosphere.



## 5.1 The Scientific Process

4. Scientific hypotheses and theories are presumed to be provisional: they can be refuted by testing but they cannot be verified as correct or true in an absolute sense. Verification is however possible of the results of the experiments upon which theories are built or that attempt to test a theory. If they can be repeated, and produce the same results, they are said to be validated. It is important to recognise that science progresses by substantive challenges based on rigorously logical, published arguments that present a different view of reality from that which they challenge. Criticising and attacking process and behaviour is not the same as an attack on a scientific hypothesis. Failure in one is not necessarily failure in the other.

5. Given the nature of our remit, our concern is not with science, whether data has been validated or whether the hypotheses have survived testing, but with behaviour; whether attempts have been made to misrepresent, or ―cherry pick‖ data with the intention of supporting a particular hypothesis, or to withhold data so that it cannot be independently validated, or to suppress other hypotheses to prevent them being put to the test.



## 5.2 The Nature of Climate Science

6. Modern climate science is largely a creation of the last 30-40 years. It involves a wide range of disciplines, including geography, physics, chemistry, a wide variety of earth sciences, palaeo-biology and computer modelling. The synthesis of such diverse sources of evidence generated through wide variety of scientific approaches and traditions has been a major task, in which the IPCC has played an import ant role.

7. One of the main contributions of CRU has been in the laborious assembly of datasets of instrumental measures of climate change and of data relating to tree ring s that are proxies for climatic variables.CHAPTER 5: THE CHANGING CONTEXT OF MODERN SCIENCE



### 37 

## 5.3 Data Accessibility in the Digital A ge

8. Modern digital technologies permit the acquisition and manipulation of very much larger datasets than formerly. To enable proper validation of the conclusions, such datasets must be made freely available, along with details of the associated computational manipulation. It is often difficult and time-consuming to describe this  ̳meta

- data‘ in ways that permit others to verify experimental results and test the inferences drawn from them. The preservation of such data also presents a novel problem of archiving and stewardship compared with the archives of former times.

9. An important shift in attitude is represented by a recent report by the US National Academies 1 

### , which highlights these issues. It recommends a new approach to the integrity, accessibility, and stewardship of data in publicly-funded science, arguing that researchers should make all research data, methods, and other information underlying the results publicly accessible in a timely manner. These recommendations would require a substantial shift of behaviour amongst many scientists. The e-mails suggest that this would be true for CRU

– for example: Jones: “We have 25 or so years invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it ” 2



### .

10. Digital datasets in specific areas of science are increasingly being placed in international archives. The World Data System, run by the International Council for Scientific Unions 3 

### , for example, covers a very wide range of data types including the International Tree-Ring Data Bank. The majority of the source data used to create the CRUTEM gridded product is available through the Global Historical Climatology Network (GHCN) managed by the NCDC 4 

### and the US National Center for Atmospheric Research (NCAR) 5



### .

11. Some funders of research require that applications for funding include plans for data archiving. The Natural Environment Research Council (NERC), the UK research council most relevant to the work of CRU, has a policy that requires recipients of grants to offer to deposit with NERC a copy of datasets resulting from the supported research 6



### .

12. Over the period under review, data accessibility was also being driven by major changes in the law that ha d important implications for research scientists working in the UK. Whilst the precursors of the Environmental Information Regulations date back to the 1990s, the current EIR came into force on 1 st 

### January 2005, on the same date as the general right of access under the Freedom of Information Act 2000 (FoIA). These are described in more detail in Chapter 10. The EIR applies exclusively within a rather broad definition of environmental and related

1 National Academy of Sciences. 2009. Ensuring the Integrity, Accessibility, and Stewardship of Research Data in the Digital Age . 188pp.

2 Referred to in Sherrington submission (No. 38), 17 February 2010, page 10)

3 http://www.ukssdc.ac.uk/wdcmain/guide/wdcguide html

4 http://www.ncdc noaa.gov/oa/ncdc.html

5 http://www.ncar.ucar.edu/

6 NERC Data Policy (Version February 2010)CHAPTER 5: THE CHANGING CONTEXT OF MODERN SCIENCE



### 38 information; this area is then excluded from the FoIA. The extent to which both regimes apply to access to information from publicly funded (or part-funded) research is still an evolving area. Whilst it is broadly recognised that these do apply to data and supporting metadata, algorithms, etc. at the point when a

re search paper is published, how to interpret the legislation and regulations in terms of data held for a long period prior to publication and also pre-publication drafts and other correspondence is work in progress. The  ̳public interest test‘, applicable to both EIR and FoIA, again described in Chapter 10, remains key to such interpretation.



## 5.4 Handling Uncertainty

13. There is a widespread misconception that science produces unequivocal and absolutely precise answers. It does not, and cannot. All scientific results contain uncertainties, and it is important that these are made clear to and are understood by those who use them. There are two fundamental sources of uncertainty in science: uncertainty in measuring a phenomenon and uncertainty in determining causes and causal relationships.

14. Over the period in question, the increasing amount of quantitative data available on climate change has made it easier to assess and represent uncertainty. This can clearly be seen in the representation of what is termed the Medieval Warm Period. Prior to the 1980s this was shown as a single line, without error bars 7 

### . Subsequent quantitative palaeo-temperature work on tree rings, in which CRU has been a leader, has permitted errors to be assessed as a consequence of the statistical correlations involved (see Chapter 7). Although the recent Oxburgh Scientific Assessment Panel 8 

### has been cr itical of some of the statistical work of CRU in relation to the tree ring series, that issue lies beyond our remit. Our concern is whether tree ring series have been improperly selected and whether the uncertainties have been properly presented.

15. The processes of hypothesis, experiment, testing and refutation generally reduce uncertainties about causes and relationships but do not entirely remove them. Some of the conflicts involving CRU have been scientific disagreements about causes and relationships that in principle should be resolved by new data which exposes existing hypotheses to rigorous testing. One of the allegations against members of CRU has been that they have attempted improperly to prevent ideas with which they disagreed from being effectively discussed, for example in relation to the publication of papers (Chapter 8) and in parts of the IPCC process (Chapter 9).

7 e.g. Houghton, J.T., Jenkins, G.J. and Ephraums J.J. 1990. Climate Change: the IPCC Scientific
Assessment . Cambridge University Press.

8 Report of the International Panel set up by the University of East Anglia to examine the research of the Climatic Research Unit . 2010. Available at: http://www.uea.ac.uk/mac/comm/media/press/CRUstatements/oxburghCHAPTER 5: THE CHANGING CONTEXT OF MODERN SCIENCE



### 39 

## 5.5 Scientific Journals and the Peer Review Process

16. Allegations against CRU include improper attempts to influence the editorial policy of scientific journals and the peer review process. To help it understand the context, the Review commissioned a contextual paper from a distinguished Journal editor

–

Dr Richard Horton of The Lancet

– on which Elizabeth Wager, Chair of CO PE , was invited to comment in a personal capacity. (Appendix 5).

17. Peer review is important because of the way that scientific journals provide the frame of reference for most formal scientific debate. Scientific ideas that are not expressed in a scientific journal are rarely addressed. Edward Wilson, for example, commented that  ̳ a discovery does not exist until it is safely reviewed and in print ‟

. 9



### 18. Access to publication in scientific journals is therefore a crucial issue. Editors have used peer review by acknowledged experts of papers submitted for publication to advise them whether they contain errors, are important, trivial or simply repeat what has previously been published.

19. Richard Horton‘s essay explores typical patterns of behaviour relating to journals and peer review in contentious areas of science, and what behaviour should be regarded as improper. His conclusions are summarised as follows:

What an editor seeks from a reviewer is a powerful critique of the manuscript

– testing each assumption, probing every method, questioning all results, sceptically challenging interpretations and conclusions and ensuring that uncertainties are fully acknowledged, measured, and reported.

Armed with such a critique, the editors decide, and take full responsibility for deciding, whether to publish.

There is always the risk of group-think among experts which resists alternative perspectives. Editors try to reduce the risk of group-think by sending papers to different and widely dispersed reviewers, deliberately seeking or even provoking critical reviews.

Editors send manuscripts to reviewers based on a principle of confidentiality. The author expects the editor to maintain a covenant of trust between the two parties. Disclosure to a third party without the prior permission of the editor would be a serious violation of the peer review process and a breach of confidentiality.

Many who are far from the reality of the peer review process would like to believe that peer review is a firewall between truth on the one hand and error or dishonesty on the other. It is not. It is a means of sieving out evident error, currently unacceptable practices, repetition of previously published work without acknowledgement, and trivial contributions that add little to

9 Wilson, E.O. 1998. Consilience: The Unity of Knowledge . New York: Knopf.CHAPTER 5: THE CHANGING CONTEXT OF MODERN SCIENCE



### 40 knowledge.

It does not and cannot guarantee veracity. Many published papers have proved deeply flawed: many good ones have been rejected. Nor has it been efficient in identifying fraud, which has usually come to light by different routes.

However, journals, as the gatekeepers of scientific publication, have come to exert an increasing influence on the reputations of scientists, research units and universities. Many measures of academic success, promotion, tenure, grants, fame, and personal wealth depend upon journal publication. It is not surprising therefore that journals, and peer review, are the subject of constant tension and occasionally explosive controversy.

20. In conclusion, it is common for editors to have multiple, intense, and sometimes sharp and passionate interactions with authors and reviewers. The tone of their exchanges and communications with editors can be attacking, accusatory, aggressive, and even personal. If a research paper is especially controversial and word of it is circulating in a particular scientific community, third-party scientists or critics with an interest in the work may get to hear of it and decide to contact the journal. They might wish to warn or encourage editors. This kind of intervention is entirely normal. It is the task of editors to weigh up the passionate opinions of authors and reviewers, and to reflect on the comments (and motivations) of third parties. Occasionally, a line might be crossed into highly improper behaviour leading, for example, to censorship of ideas that might normally pass peer review. Defining that line is the crucial task when judging the role of CRU scientists, and determining whether, as has been alleged, they acted to subvert peer review by slowing or blocking the publication of research which disagreed with their own views. Was their activity part of the normal, robust hurly-burly surrounding publication in important highly contended fields, or was an important line crossed? We address this in Chapter 8.



## 5.6 The Responsibilities of Scientists in Communicating in the Public Domain

21. The scientific literature is relatively opaque to non-specialists. Scientific understanding that is transmitted into the public domain must be comprehensible to non-specialists, make appropriate and not excessive claims, and include careful statements of the uncertainties surrounding that understanding. These principles apply universally to scientific advice, and are embedded in the codes of practice for scientific advice that have been adopted by many governments 10 

### . One of the allegations against CRU is that they have not been sufficiently frank in communicating uncertainties about their reconstructions into the public domain. This is particularly relevant to graphical presentation such as the  ̳hockey stick‘ which has taken on iconic significance. Images have a great power to persuade, and this is particularly true when complex issues are faced by lay audiences, who may often infer a level of certainty that does not in fact exist. The danger is

10 For example, the ― Code of Scientific Ethics ‖ published in 2007 by the UK Chief Scientific Advisor, and accepted by UK Government, includes the responsibility to: ―present and review scientific evidence, theory or interpretation honestly and accurately.‖CHAPTER 5: THE CHANGING CONTEXT OF MODERN SCIENCE



### 41 obviously heightened where an image is being used to support arguments for policy change. Therefore, if images are likely to be used in this way, it is essential that qualifications such as uncertainties are given a closely coupled prominence and explanation.



## 5.7 Communicating to Policymakers

22. The interface between science and public policy is a crucial one in matters of great publi c importance such as climate change. The IPCC was set up to provide just such an interface. Its job was to draw on and synthesise the diverse strands that contribute to modern climate science (see paragraph 6) and to make this accessible to the public, policymakers and other stakeholders in a way that is comprehensible and that does justice to underlying uncertainties. The importance of this process is underlined by the potential magnitude of the economic and social consequences of governmental decisions in the domain of global climate change. It has been alleged that CRU scientists subverted IPCC processes by minimising uncertainties and blocking ideas that disagreed with their established views.

23. The IPCC produces assessments of the current state of understanding of climate change, its causes and implications. Its approach is to produce the most probable account of these issues; together with their uncertainties, and to identify where there is insufficient evidence to discriminate between different interpretations of a phenomenon. Its purpose is to produce a  ̳best estimate‘ of what is currently understood, through the work of a group of scientists chosen for their expertise and experience to make reasoned assessments on the balance of evidence. It is not to produce a review of the scientific literature.

24. The IPCC assessment reports were published in 1990, 1996, 2001 and 2007. AR4, published in 2007, consisted of three components: The Physical Science Base (Working Group 1

–

WG1); Impacts, Adaptation and Vulnerability (WG2); and Mitigation of Climate Change (WG3). The issues that concern CRU are a consequence of their involvement in WG1. The core activities of these WGs are undertaken by writing teams consisting of Coordinating Lead Authors (CLAs) and Lead Authors (LAs). There are two drafting cycles prior to the production of the Final Draft. The first draft is submitted to Expert Reviewers for comment. The second draft is submitted for both expert and governmental review. Review Editors are appointed for each of the review cycles to ensure that critical comments are properly dealt with. The final draft is the source of a Summary for Policymakers, which is also reviewed by governments prior to publication.



## 5.8 The C ha nging Forum for Debate and the Blogosphere

25. The development in recent years of the internet as a vehicle for easy, instantaneous transmission of news and opinion has changed the nature of the debate about scientific issues. Prior to these developments, scientific debate largely took place in journals and conferences that effectively excluded the public from active engagement. Experts tended to introduce their conclusions to the public in ways that were difficult to challenge.CHAPTER 5: THE CHANGING CONTEXT OF MODERN SCIENCE



### 42 26. The mode has now changed and the field of climate change exemplifies this. There continues to be a scientific debate about the reality, causes and uncertaintiesof climate change that is conducted through the conventional mechanisms of peerreviewed publication of results, but this has been paralleled by a more vociferous, more polarised debate in the blogosphere and in popular books. In this the protagonists tend to be divided between those who believe that climate is changing and that human activities are contributing strongly to it, and those that are sceptical of this view. This strand of debate has been more passionate, more rhetorical, highly political and one in which each side frequently doubts the motives and impugns the honesty of the other, a conflict that has fuelled many of the views expressed in the released CRU emails, and one that has also been dramatically fuelled by them. It is difficult at the moment to predict whether and how the necessary cooler, rigorous scientific debate and the vital public policy interface will develop, or the effect that it will have on scientific publication or peer review.

27. Arguably the most significant change produced by the blogosphere is a transformation in the degree of openness now required of scientists whose work directly affects policy making. Without such openness, the credibility of their work will suffer because it will always be at risk of allegations of concealment and hence mal-practice. The extent to which this change was fully recognised by both CRU and UEA administration is an important issue for the Review.

28. Therefore, the Review would urge all scientists to learn to communicate their work in ways that the public can access and understand; and to be open in providing the information that will enable the debate, wherever it occurs, to be conducted objectively. That said, a key issue is how scientists should be supported to explain their position, and how a public space can be created where these debates can be conducted on appropriate terms, where what is and is not uncertain can be recognised. The learned societies may have an expanded role to play here in encouraging debate. We would also commend the work of bodies such as the Science Media Centre at the Royal Institution for encouraging and helping scientists to take their work to lay audiences through the media, and advising them on how best to do this.CHAPTER 5: THE CHANGING CONTEXT OF MODERN SCIENCE



### 43CHAPTER 6: LAND STATION INSTRUMENTAL TEMPERATURE DATA



### 44 

## CHAPTER 6: LAND STATION INSTRUMENTAL TEMPERATURE DATA

6.1 Background

1. Terrestrial temperature data is recorded at more than seven thousand land stations across the world. This is referred to as primary data.

2. The steps involved in producing CRU‘s analysis of global temperature trends involve assembling and quality checking the primary data from approximately four thousand stations, determining the monthly ―anomalies‖ 1 

### for each station, and then assembling averages based on a uniform 5x5 degree grid covering the globe.

3. This work, requiring considerable experience and man-years of effort, was started by CRU in 1978 and it continues today. The results were published in a series of CRUTEM datasets over a period of 20 years between 1986 and 2006.

4. In assembling a gridded temperature dataset, some adjustment of primary data

ma y be needed in order to allow for what are referred to as  ̳non

- climatic effects‘. A simple example might involve adjusting temperatures at a particular station to allow for obvious recording errors, or the elimination of an obvious discontinuity in results if a station had been moved (referred to as homogenization). Such adjustments are both necessary and ubiquitous in scientific research.

5. This Chapter deals with the availability of these data to anyone wishing to carry out an equivalent study, the effect of adjustments, the comparison with independent studies and the availability of the information required to explicitly check the CRUTEM analysis. Finally, it addresses specific allegations about Chinese data used in a paper about the effect of urbanization on temperature.



## 6 .2 The Allegations

6. The broad allegations which are prevalent in the public domain are:

That CRU prevented access to raw data. That CRU adjusted the data without scientific justification or adequate explanation. Some allegations imply that this was done to fabricate evidence for recent warming. That CRU inappropriately withheld data and computer code, thus inhibiting others from checking the conclusions of the CRUTEM analysis.

The overall implication of these allegations is to cast doubt on the extent to which CRU‘s work in this area can be trusted and should be relied upon.

1 A station  ̳normal‘ is the average recorded te mperature over a defined time period. In the case of CRUTEM this is 1961- 1990. The temperature ―anomaly‖ is defined as the difference between the recorded temperature and the normal.CHAPTER 6: LAND STATION INSTRUMENTAL TEMPERATURE DATA



### 45 7. While very few of the submissions to this Inquiry make the allegations above explicitly, they are nevertheless implied. The full text of the submissions is contained on the team website. It is not the intent here to review each one, but only to highlight the most relevant and significant.

8. The most comprehensive and substantive submission critical of CRU is from McKitrick 2 

### . He addresses the question of data adjustments and also clarifies that the issue is not generally one of data availability, but more specifically the availability of a list of additional stations used in the CRUTEM analysis since 1986. He draws attention to the series of requests to CRU for station identifiers over the period 2002-2007, eventually culminating in a series of FOI requests, also referred to by Matthews 3 

### and Cockroft 4 

### . Two submissions 5 

### allege that CRU has withheld access to primary data and by implication stopped others from repeating analyses, but present no evidence. A common theme in the submissions from the Global Warming Policy Foundation (GWPF), McKitrick and McIntyre 6 

### is the lack of cooperation exhibited by CRU in their dealings with selected third parties.

9. In support of CRU, several submissions 7

, 8

, 9 

### point out that the data is in the public domain and can be accessed by anyone wishing to repeat a temperature trend analysis. Santer also defends the work done by CRU on regional adjustments, maintaining that it made very little difference to the global results. He also highlights that studies conducted independently from CRU, based on different choices and adjustment protocols, produce very similar results

– a point also mad e by Joos 10 

### and Tett.

10. In its own submission under issue 4, CRU 11 

### defends its position with regard to adjustments and openness. The website also contains the minutes of the meeting which took place on the subject of data sets on March 4 th 

### , 2010 between members of CRU and the Review Team.


## 6.3 The Approach Adopted by the Review Team

11. In order to test the principal allegations of withholding data and making inappropriate adjustments, the Review undertook its own trial analysis of land station temperature data. The goal was to determine whether it is possible for an independent researcher to (a) obtain primary data and (b) to analyse it in order to produce independent temperature trend results. This study was intended only to test the feasibility of conducting such a process, and not to generate scientific conclusions. The process followed is described in Appendix 7

.

2 McKitrick submission (no.15), 26 February

3 Matthews submission (no.16), 1 March

4 Cockroft submission (no. 6), 28 February

5 Global Warming Policy Foundation submission (no. 22), 28 February, and Cockroft (Ibid.)

6 McIntyre submission (no. 23), 2 March

7 Tett submission (no. 21), 1 March

8 Santer submission (no. 44), 28 February

9 Barnes submission (no. 1), 6 March

10 Joos submission (no. 12), 1 March

11 CRU submission (no. 5), 1 MarchCHAPTER 6: LAND STATION INSTRUMENTAL TEMPERATURE DATA



### 46 12. The Team then went on to address a further question, namely the extent to which an independent researcher could check the CRUTEM analysis. In particular, this considered:

whether the data sources were properly explained the availability of software required in the analysis



## 6.4 The Results of the Analysis

13. To carry out the analysis we obtained raw primary instrumental temperature station data. This can be obtained either directly from the appropriate National Meteorological Office (NMO) or by consulting the World Weather Records (WWR) available through the National Centre for Atmospheric Research (NCAR), or the Global Historical Climatology Network (GHCN):

NCAR: http://www.ncar.ucar.edu/tools/datasets/ WWR: http://dss.ucar.edu/datasets/ds570.0/ GHCN: http://www.ncdc.noaa.gov/oa/climate/ghcn-monthly/index.php.

14. Anyone working in this area would have knowledge of the availability of data from these sources. There are also other sources but we have not investigated these.

15. The Review extracted a comprehensive set of global temperature records from both GHCN and NCAR websites to test ease of availability from either. The stations included in these sources have much overlap as would be expected. In addition, as a test case, primary data was also obtained directly from the Japanese NMO.

16. In a true analysis the researcher would at this point be able to make whatever quality control, station selection and adjustments they considered appropriate, which should be justified in the published record of research.

17. Figure 6.1 below gives the results of the Team‘s trial analysis to produce three different global temperature anomaly average series. These use a 5x5 degree grid and a 5-year smoothing (for details see Appendix 7). In each case we started with the full dataset, requiring only that a station had enough data to construct the relevant normals when constructing the gridded averages. Apart from this we made no selection or other adjustments of our own. We show the result obtained using the GHCN unadjusted data set (blue), the GHCN adjusted data set (yellow) and the NCAR dataset which is unadjusted (green). On the same figure we show as a black line the global average series obtained from the land air temperature anomalies on a 5x5 degree grid-box basis (CRUTEM3) gridded analysis (which uses adjusted data). The green and blue lines are both from unadjusted datasets. The yellow and black lines are from adjusted datasets.

18. All the lines, whether from adjusted or unadjusted datasets, agree very well.CHAPTER 6: LAND STATION INSTRUMENTAL TEMPERATURE DATA



### 47 Figure 6.1: Temperature anomaly time series created by the Review Team‘s own trial analysis using a 5x5 degree grid with 5 year smoothing. Shown are results obtained from GHCN (blue)

, GHCNadjusted (yellow) and NCAR (green) . Also shown is the CRUTEM3 line (black). The Y-axis is 10 x the anomaly in degrees. The X-axis is year.



### 19. In Figure 6.2 we show the results of independent scientific studies, where individual researchers have brought to bear their own judgment on station selection and adjustments to the data. Such studies have been carried out by NASA-GISS and NOAA-NCDC using the GHCN dataset. The figure from the IPCC 4 th 

### Repo rt Chapter 3 is reproduced below to show these in comparison to CRUTEM3 12 

### . Once again there is good agreement.

12 The reader is referred to the comprehensive submission from the UK Met Office (CRU 54) to the House of Commons Committee on Science and Technology Inquiry which shows a similar unsmoothed figure.CHAPTER 6: LAND STATION INSTRUMENTAL TEMPERATURE DATA



### 48 Fig 6.2 Reproduced from the IPCC 4 th Report Chapter 3.



### 20. Finding: This simple analysis and the comparisons in figures 6.1 and 6.2 give rise to the following findings:

Any independent researcher may freely obtain the primary station data. It is impossible for a third party to withhold access to the data.

It is impossible for a third party to tamper improperly with the data unless they have also been able to corrupt the GHCN and NCAR sources. We do not consider this to be a credible possibility, and in any case this would be easily detectable by comparison to the original NMO records or other sources such as the Hadley Centre.

T 

### he steps needed to create a global temperature series from the data ar e straightforward to implement.

The required computer code is straightforward and easily written by a competent researcher.

The shape of the temperature trends obtained in all cases is very similar: in other words following the same process with the same data obtained from different sources generates very similar results.CHAPTER 6: LAND STATION INSTRUMENTAL TEMPERATURE DATA



### 49 21. By performing this simple test one determines easily that the results of the CRUTEM analysis follow directly from the published description of the method, and that the resultant temperature trend is not significantly different from the other results regardless of stations used or adjustments made. The test is therefore sufficient to demonstrate that, with respect to the declared method, the CRUTEM analysis does not contain either error or adjustments which are responsible for the shape of the resultant temperature trend.

22. A researcher can evidently produce a study which would test the CRUTEM analysis quite precisely, without requiring any information from CRU to do so.

23. Finding: The high level trial analysis shows that adjustments make little difference to the overall conclusions on temperature trends. However, this has been a matter about which much comment has been made and it is therefore discussed in more detail in Appendix 7. Adjustments were made by CRU to the data from 10% of stations. The Review Team is satisfied that this had no significant effect upon results. Very strong evidence for this is furnished by the level of agreement between CRUTEM and the other lines shown in Figure 6.1, as well as other studi es detailed in the Appendix, showing the effect of de-adjusting and of removing the adjusted stations.

24. It should be noted that in making these findings, the Review Team is making no statement regarding the correctness of any of these analyses in representing global temperature trends. We do not address any alleged deficiencies such as allowance for non climatic effects or the significant drop in station number post 1991. We do not address any possible deficiencies of the method. These are entirely matters for proper scientific study and debate and lie outside the scope of this Review.



## 6.5 Checking Specific Details in the CRUTEM Analysis

25. The goal here is to consider the requirements for carrying out an exact replication of the CRUTEM work to produce an identical gridded analysis. The work above demonstrates that this procedure is not necessary if the goal is simply to produce an alternative temperature trend analysis. As discussed in Appendix 7, and as follows from the close agreement between the lines in Figures 6.1 and 6.2, it is also not necessary to test the robustness of the CRUTEM3 analysis. However, the Review upholds the important principle that any scientific result should be open to such scrutiny for whatever reason. It therefore considered whether all the necessary information was made available.



## 6.5.1 Identification of Data Sources

26. In order to reproduce exactly a CRUTEM study, the independent researcher requires the exact list of stations used in order to source the primary data themselves.

27. The original version of the CRUTEM gridded averages was published in 1986 (in this report we call this CRUTEM1986) using 3276 stations (314 of which were adjusted). Subsequent revisions were produced in 1994, 2003 and 2006, referredCHAPTER 6: LAND STATION INSTRUMENTAL TEMPERATURE DATA



### 50 to as CRUTEM1, 2 and 3. CRUTEM3 used 4138 stations (298 were adjusted). Whereas the 3276 stations used in CRUTEM1986 are listed in the respective references 13 

### , this is not the case for the additional approximately 1000 stations included in CRUTEM3 14 

### . The additional stations are described only in general terms in the respective publication via references to the sources.

28. For some years prior to the coming into force of the general right of access to information under FoIA, CRU had received requests for data, including station identifiers. An example of the attitude to these requests is given in the following e-mail extract:

Jones to Mann on 2 nd 

### February 2005 (1107454306.txt):

“And don't leave stuff lying around on ftp sites - you never know who is trawling them. The two MMs have been after the CRU station data for years. If they ever hear there is a Freedom of Information Act now in the UK, I think I'll delete the file rather than send to anyone. Does your similar act in the US force you to respond to enquiries within 20 days? - our does ! The UK works on precedents, so the first request will test it. We also have a data protection act, which I will hide behind. Tom Wigley has sent me a worried email when he heard about it - thought people could ask him for his model code. He has retired officially from UEA so he can hide behind that. IPR should be relevant here, but I can see me getting into an argument with someone at UEA who'll say we must adhere to it !”

29. A request was made to CRU for both station identifiers and access to the raw data for these stations. This was formally logged by the UEA Information Policy & Compliance Manager (IPCM) in January 2007 and was dealt with under the FoIA

. The request was initially refused entirely on the grounds that raw station data was publicly available (from GHCN, NCAR and the NMOs), but without reference to providing station identifiers. Following further correspondence, an internal UEA appeal process and further prompting by the applicant, the list of the 4138 stations used in CRUTEM3 was finally released on 1st October. The issues that this raises

13 Bradley, R.S., Kelly, P.M., Jones, P.D., Goodess, C.M. and Diaz, H.F., 1985: A Climatic Data Bank for Northern Hemisphere Land Areas, 1851-1980, U.S. Dept. of Energy, Carbon Dioxide Research Division, Technical Report TRO17 , 335 pp. Jones, P.D., Raper, S.C.B., Santer, B.D., Cherry, B.S.G., Goodess, C.M., Kelly, P.M., Wigley, T.M.L., Bradley, R.S. and Diaz, H.F., 1985: A Grid Point Surface Air Temperature Data Set for the Northern Hemisphere, U.S. Dept. of Energy, Carbon Dioxide Research Division, Technical Report TRO22 , 251 Jones, P.D., Raper, S.C.B., Cherry, B.S.G., Goodess, C.M. and Wigley, T.M.L., 1986: A Grid Point Surface Air Temperature Data Set for the Southern Hemisphere, 1851-1984, U.S. Dept. of Energy, Carbon Dioxide Research Division, Technical Report TR027 , 73 pp. Jones, P.D., Raper, S.C.B., Bradley, R.S., Diaz, H.F., Kelly, P.M. and Wigley, T.M.L., 1986: Northern Hemisphere surface air temperature variations: 1851-1984. Journal of Climate and Applied Meteorology 25 , 161-179. Jones, P.D., Raper, S.C.B. and Wigley, T.M.L., 1986: Southern Hemisphere surface air temperature variations: 1851-1984. Journal of Climate and Applied Meteorology 25 , 1213 1230

14 Brohan, P., Kennedy, J., Harris, I., Tett, S.F.B. and Jones, P.D., 2006: Uncertainty estimates in regional and global observed temperature changes: a new dataset from 1850. J. Geophys. Res. 111

, D12106, doi:10.1029/2005JD006548. Jones, P.D. and Moberg, A., 2003: Hemispheric and large-scale surface air temperature variations: An extensive revision and an update to 2001. J. Climate 16 , 206-223.CHAPTER 6: LAND STATION INSTRUMENTAL TEMPERATURE DATA



### 51 on compliance with the FoIA are dealt with in Chapter 10.

30. The Review Team verified that matching of stations to a simple list of identifiers is not a straightforward process. The identification numbers are not unique across data sets, station names differ, and in the case of GHCN several replicas of the same data exist. This is primarily a problem caused by the lack of standardisation of metadata within the global climate science community. We make a recommendation on this below.

31. The Review Team was able to match 90% of stations given in the CRU list to GHCN (see Appendix 7). CRU has stated in a written submission 15 

### that that the remaining 10% can be obtained from other sources including the NMOs. Thus substantial work is required to take the CRU published list and assemble 100% of the primary station data from global repositories and NMOs. We make a recommendation for the future below.

32. Finding: The Review finds that as a matter of good scientific practice, (and having established the precedent with CRUTEM1986) CRU should have made available an unambiguous list of the stations used in each of the versions of CRUTEM at the time of publication. In the absence of this, CRU was unhelpful and defensive and should have responded throughout to requests for this information in a more timely way.



## 6.5.2 The Availability of Computer Codes

33. Finding: The computer code required to read and analyse the instrumental temperature data is straightforward to write based upon the published literature. It amounts a few hundred lines of executable code (i.e. ignoring spaces and comments). Such code could be written by any research unit which is competent to reproduce or test the CRUTEM analysis. For the trial analysis of the Review
Te am, the code was written in less than two days and produced results similar to other independent analyses. No information was required from CRU to do this.

34. The Met Office Hadley centre has published Practical Extraction and Reporting Language (PERL) codes which are a few hundred lines long. The submission by
Barnes is also very helpful and illustrative in this context.



## 6. 6 Use of Local Temperature Data from China

35. The above account of our work has focused on global data, its handling and accessibility. We also considered the implication in a submission to us by Dr Benny Peizer of the Global Warming Policy Foundation 16 

### that Jones was complicit in malpractice in failing to respond appropriately to allegations of fraud made against a climate scientist at the State University of New York, Albany, Professor Wei-Chyung Wang in relation to more local data. An important paper

15 Follow up request (20 April) to Professor Jones and response in relation to raw instrument station availability for each CRUTEM set

16 Peizer/Global Warming Policy Foundation submission (no. 22), 28 FebruaryCHAPTER 6: LAND STATION INSTRUMENTAL TEMPERATURE DATA



### 52 on the effect of urbanization on temperature by Jones et al (1990) 17 

### included data from China (Wang et al, 1990) 18 

### that contained the statement that only instrument stations were selected where there had been few, if any, changes in instrumentation, location or observation times. It has been claimed by Keenan (2007) 19 

### that this latter statement was knowingly untrue (“fabricated”), that stations had been moved, thus that their record of temperature change would be unreliable, and that hard copies of details of station histories had been lost or withheld. A subsequent paper by Jones et al. (2008) 20 

### verified the original conclusions for the Chinese data for the period 1954

– 1983, showing that the precise location of weather stations was unimportant to the outcome

36. Peizer, as Editor of Energy and Environment , to which the Keenan paper had been submitted, had contacted Jones asking him to comment on the Keenan paper, which Jones did, by denying its criticisms and advising rejection of the

paper 21 

### . Jones was encouraged by colleagues to respond in writing to the journal, but chose not to do so "until the SUNY process has run its course." 22 

### (The State University of New York, Albany, investigated the accusation of fraud against Wang, which it rejected 23 

### ). Peizer has included some detail of email traffic on this issue in his submission to us. 24



### 37. The allegation against Jones that is relevant to the Review 25 

### is that since significant doubts about the reliability of Chinese climate data were raised, Jones has taken no public steps to clear up the discrepancies regarding Wang's claims and data, and that “it is unacceptable that the scientist who disseminates a dat a product on which international treaties are based, as well as IPCC reports and countless government policies, should actively seek to suppress information that calls the quality of the data into question”.

38. We note that, in at least a Nature interview referring to the loss of records 26 

### , Jones is reported as acknowledging “It‟s not acceptable...not best practice ...the stations „probably did move‟...I thought it was the right way to get the data. I was specifically trying to get more rural station data that wasn‟ t routinely available in real time from [meteorological] services.” Although Jones chose not to respond in

17 Jones P.D., Groisman P.Y., Coughlan M., Plummer N., Wang W.- C., Karl T.R. (1990), ―Assessment of urbanization effects in time series of surface air temperature over land‖, Nature , 347: 169

– 172.

18 Wang W.- C., Zeng Z., Karl T.R. (1990), ―Urban heat islands in China‖, Geophysical Research
Letters , 17: 2377

– 2380.

19 Keenan, D.J. Fraud allegation against some climatic research of Wei-Chyung Wang. Energy and Environment, 18 (7- 8) , 985-995.

20 Jones, P.D., Lister, D.H. and Li, Q. 2008. Urbanization effects in large-scale temperature records, with an emphasis on China. Journal of Geophysical Research, 113 (D16122) , 12pp.

21 Ibid. 16

22 Ibid. 16

23 http://www.informath.org/apprise/a5620.htm. This site included a letter to Keenan from the State University of New York, Albany, that states that ―The Investigation Committee finds no evidence of the alleged fabrication of results and nothing that rises to the level of research misconduct having been committed by Dr. Wang.‖ We are not aware of a published version of this report.

24 Ibid. 16

25 Ibid. 16

26 Nature News, published online 15 February 2010 http://www.nature.com/news/2010/100215/full/news.2010.71.htmlCHAPTER 6: LAND STATION INSTRUMENTAL TEMPERATURE DATA



### 53 writing when Keenan‟s claim was made, he did respond through an analysis in a paper in a front rank peer-reviewed journal only one year later that verified the conclusions of the earlier work 27 

### . By that time the SUNY investigation had concluded in Wang‟s favour

. 

## 6.7 Conclusions and Recommendations

39. In summary, with regard to the allegations concerning the temperature dat a, the conclusions of the Review Team are as follows:

Regarding data availability, there is no basis for the allegations that CRU prevented access to raw data. It was impossible for them to have done so.

Regarding data adjustments, there is no basis for the allegation that CRU made ad justments to the data which had any significant effect upon global averages and through this fabricated evidence for recent warming.

We find that CRU was unhelpful in dealing with requests for information to enable detailed replication of the CRUTEM analysis.

Crucially, we find nothing in the behaviour on the part of CRU scientists that is the subject of the allegations dealt with in this Chapter to undermine the validity of their work.

40. Reflecting the analysis in Appendix 7, the Revi ew has the following recommendations:

It would benefit the global climate research community if a standardised way of defining station metadata and station data could be agreed, preferably through a standards body, or perhaps the WMO. We understand that this is not straightforward and could be a lengthy process, but the process should start now. As example an xml based format would make the interpretation use, comparison, and exchange of data much more straightforward.

Without such standardisation there will always be problems in issuing unambiguous lists, and assembling primary data from them. It would be in the public interest if CRU and other such groups developed a standard process to capture and publish a snapshot of the data used for each important publication.

27 Ibid. 20CHAPTER 7: TEMPERATURE RECONSTRUCTIONS FROM TREE RING ANALYSIS



### 54 

## CHAPTER 7: TEMPERATURE RECONSTRUCTIONS FROM TREE RING ANALYSIS

7.1 Background

1. This Chapter considers the criticisms that have been made of CRU scientists in relation to their work on obtaining and analysing tree ring data to reconstruct land temperature records back over thousands of years, and in relation to the use made of that work in the IPCC.

2. Data is recorded from tree core samples from many sites around the globe. These data comprise measurements of width and density taken from the cores and do not in themselves represent temperature directly. The processes of assembling and standardising the tree data, into what are termed tree ring chronologies, are separate from the later stage of producing temperature reconstructions.

3. Temperature reconstructions are obtained through regression models which are calibrated against instrument temperature data during periods of overlap. A temperature reconstruction from any specific region does not in itself indicate global temperature patterns. Tree and other proxy reconstructions are combined to produce reconstructions pertaining to a wider area, and such reconstructions from many authors may be aggregated in order to provide an indication of global patterns. Even so the coverage is not as extensive as that of instruments.

4. The uncertainty associated with temperature reconstructions in the distant past is much larger than that of present day instrument records. There is a wide and healthy debate in the published literature on all aspects of this.

5. CRU scientists have developed temperature reconstructions based on a range of tree ring series. The current leader in CRU is Professor Briffa. Their work i s extensively described in the relevant scientific literature. It also contributes to the development of a global picture such as that produced by IPCC in its 4 th 

### Assessment Report (Working Group 1, Chapter 6, Palaeoclimate). In particular this contained conclusions on the likelihood that the second half of the 20 th 

### Century was the warmest 50 year period in the last 500 or 1300 years in the Northern Hemisphere (different statements of probability are made for each

– see paragraph 20 ).



## 7.2 The Allegations

6. The release of the e-mails triggered the expression of a range of criticisms (many of long standing) of the work of CRU scientists. There was a clear implication that in carrying out their work, both the choices they made of data and the way it was handled, were intended to bias the scientific conclusions towards a specific result and to set aside inconvenient evidence. The overall implication is that this work is misleading and should not be trusted.CHAPTER 7: TEMPERATURE RECONSTRUCTIONS FROM TREE RING ANALYSIS



### 55 7. We address the following specific allegations which were explicit or can be inferred from the comments made on the e-mails and from the submissions we received

.

8. In relation to IPCC:
That the CRU work was ―flawed‖ and yet had a distinct influence on the balance of judgements made by IPCC Working Group 1 in Chapter 6 of their
4 th 

### Assessment Report, and therefore that less confidence should be ascribed to the conclusions reached by the authors, mentioned in paragraph 5 above. The criticism here is often captured by the proposition that today‘s temperatures are not unusual compared to the MWP.

That Yamal and other chronologies constructed by CRU are unrepresentative of temperature trends (in recent years), and had an undue influence on all of the lines appearing in Chapter 6 of the 4 th 

### IPCC Report.

That a majority of the reconstructions would look significantly different if certain component series were replaced with others, and that if this were done then the conclusions reached in respect of the likelihood associated with ranking of recent warmth with respect to the past would be significantly different.

9. In relation to divergence: That the phenomenon of ―divergence‖ (see discussion below) may not have been properly taken into account when expressing the uncertainty associated with reconstructions; and th at this was not addressed with sufficient openness in IPCC reports.

That the reference in a specific e- mail to a ―trick‖ and to ―hide the decline‖ in respect of a 1999 WMO report figure show evidence of intent to paint a misleading picture.

10. In relation to withholding data: That access to tree ring data from the Yamal peninsula in Siberia (used by Professor Briffa) was withheld, and that as a particular consequence of this the small sample size of the Yamal series in the most recent years was not acknowledged.

11. In relation to mishandling data: That improper manipulations have been performed

– specifically with respect to the Yamal and Tornetrask tree series.CHAPTER 7: TEMPERATURE RECONSTRUCTIONS FROM TREE RING ANALYSIS



### 56 

## 7.3 Findings

7.3.1 IPCC Reports

12. The IPCC reports are at the centre of allegations of presenting misleading messages produced by CRU. We concentrate upon AR4 as it is the most recent and most comprehensive.

13. On the replacement of series in reconstructions, much of the critical material submitted presents several examples of differing results obtained from a tree series (A) compared to another tree series (B) in the same region. This is to be expected, and taken in isolation has no implication for the validity of either. To know whether replacing one with another would have a significant effect upon statements regarding Northern Hemisphere temperature series, would require the fulfilment of several conditions:

That it has been proven that series B is more representative over the whole period than series A. That the difference between series A and B is important in a temperature reconstruction in which it is included. That if A were replaced with B in each of the reconstructions in which A is include d then the conclusions drawn in respect of the likely ranking of the past to the present would change significantly when considering all reconstructions together (including those in which it does not appear).

To support an allegation of improper conduct would require that the scientists involved, knowing the above, still use and promote non-representative series A in order to reach improperly a false conclusion.

14. Finding: We are unaware of any analysis to demonstrate that any of the above conditions are fulfilled for Yamal or any of the series cited in relation to CRU work (i.e. Tornetrask, Taymir). The Review is naturally aware that partial studies and comments referring to CRU‘s published work appear elsewhere . However these criticisms of CRU‘s work are not in peer reviewed journals, and we have not found that these are anywhere assembled into a coherent, comprehensive and scrutinised case which demonstrates the proposition in respect of any of the series cited.

15. Finding: To make the case that replacing series in reconstructions calls in question the validity of the picture painted in Chapter 6 of the IPCC report (or to sustain a charge of impropriety on the part of its many authors) would require that all the conditions in paragraph 13 were met. No evidence of this has been presented to the Review.

16. Finding: The influence of the Yamal series of Briffa 2000 1 

### is often called into question. The conditions in paragraph 13 above have not been demonstrated

1 Briffa, K. R. 2000. Annual climate variability in the Holocene: interpreting the message of ancient

trees. Quaternary Science Reviews 19:87- 105CHAPTER 7: TEMPERATURE RECONSTRUCTIONS FROM TREE RING ANALYSIS



### 57 anywhere which has been brought to our attention. This series is used in none of the IPCC TAR and only 4 2 

### out of 12 reconstructions in AR4.

17. We now turn to the question of uncertainty. There are multiple sources of uncertainty in respect of proxy temperature reconstructions and they are the subject of an ongoing and open scientific debate.

18. Elements of that debate are touched on in the e-mails. For example, the following e-mails from Briffa show some concern to ensure that uncertainty was properly recognised.

E-mail dated 29/4/03 (1051638938.txt):

“Can I just say that I am not in the MBH camp - if that be characterized by an unshakable "belief" one way or the other , regarding the absolute magnitude of the global MWP. I certainly believe the " medieval" period was warmer than the 18th century - the equivalence of the warmth in the post 1900 period, and the post 1980s, compared to the circa Medieval times is very much still an area for much better resolution. I think that the geographic / seasonal biases and dating/response time issues still cloud the picture of when and how warm the Medieval period was . On present evidence , even with such uncertainties I would still come out favouring the "likely unprecedented recent warmth" opinion - but our motivation is to further explore the degree of certainty in this belief - based on the realistic interpretation of available data.”

E-mail dated 3/2/06 (1138995069.txt):

“we are having trouble to express the real message of the reconstructions
- being scientifically sound in representing uncertainty , while still getting the crux of the information across clearly...We have settled on this version (attached) of the Figure which we hoe [sic] you will agree gets the message over but with the rigor required for such an important document.”

19. What is clear is that the uncertainty associated with any estimate of past temperatures from reconstructions is much larger than that of recent instrument temperature data. This is demonstrated in the figure below taken from IPCC AR4 Figure 6.10. The middle plot shows the variability both within and between different reconstructions each using an aggregation of proxy data (i. e. several tree and non-tree proxies). The lower plot gives an indication of uncertainty bands from those reconstructions. Simply looking at any individual reconstruction line alone makes only a partial statement about Northern Hemisphere temperatures with a large uncertainty. It is obviously even less meaningful to look at an individual tree series in isolation.

2 A fifth reconstruction is disputed.CHAPTER 7: TEMPERATURE RECONSTRUCTIONS FROM TREE RING ANALYSIS



### 58 Figure 7.1: From IPCC AR4 (Figure 6.10)



### 20. Understanding requires proper statistical interpretation, i.e. to determine the confidence level associated with a statement such as “ the present is likely warmer than the past”

. To do this as objectively as possible would require a complex (and difficult) study to perform hypothesis testing in a mathematically rigorous way, taking proper account all of the uncertainties and their correlations. We are not aware that this has been done in the production of IPCC reports to date, but instead qualitative statements have been made based on definitions of ―likely‖, ―very likely‖ etc according to criteria lai d down by the IPCC ( ̳Likely‘ mea ns a probability greater than 66 %, and  ̳Very Likely‘ means a probability greater than 90%).CHAPTER 7: TEMPERATURE RECONSTRUCTIONS FROM TREE RING ANALYSIS



### 59 21. Finding: We do not find that the data described in IPCC AR4 and shown in Figure 6.10 is misleading, and we do not find that the question marks placed over the CRU scientists‘ input casts doubt on th e conclusions. In particular:

The variation within and between lines, as well as the depiction of uncertainty is quite apparent to any reader.

It presents all relevant published reconstructions we are aware of, i.e. there has been no exclusion of other published temperature reconstructions which would show a very different picture.

The general discussion of sources of uncertainty in the text is extensive, including reference to divergence and it therefore cannot be said that anything has been suppressed. Presenting uncertainty in this way is a significant advance on the TAR.



## 7.3.2 Divergence

22. Divergence : The phenomenon of divergence is an ongoing research area which is discussed widely in the literature and the submissions. Tree core data yields both width and density measurements and both of these are used to reconstruct temperature records. In some cases the temperature reconstruction from one or other of these will track instrumental temperature measurements on a short timescale, but diverges on a longer timescale. The phenomenon of divergence is not observed in all tree series. There is as yet no complete understanding of the source of divergence although studies are suggesting possible contributory effects.

23. Finding: The question of whether divergence has been adequately taken into account when estimating uncertainties is a complex and ongoing question of science and is therefore outside the scope of this Review. We have however investigated this matter enough to be satisfied that it is not hidden and that the subject is openly and extensively discussed in the literature, including CRU

papers.

24. The WMO 1999 front cover figure : The allegation is that the figure appearing on the front cover of the WMO report of 1999 ( Statement on the Status of Global Climate , 1999 3 

### ) did not disclose that it combined proxy and temperature records and failed to show a declining proxy line due to divergence; and that this was done to deceive.

Jones‘ e-mail on 16/11/99 (942777075.txt)

“I've just completed Mike's Nature trick of adding in the real temps to each series for the last 20 years (ie from 1981 onwards) and from 1961 for Keith's to hide the decline.”

is perhaps the prime example of the fact that opposing interpretations can be obtained from the same statement. The word ―trick‖ has been widely taken to

3 http://www.wmo.ch/pages/prog/wcp/wcdmp/statement/wmo913.pdfCHAPTER 7: TEMPERATURE RECONSTRUCTIONS FROM TREE RING ANALYSIS



### 60 confirm the intention to deceive, but can equally well, when used by scientists, mean for example a mathematical approach brought to bear to solve a problem. It is the latter explanation that Jones has given the Review, quoted in paragraph 6 of Chapter 4.

25. The WMO report is a short document produced annually. It does not have the status or importance of the IPCC reports. The figure in question was a frontispiece and there is no major discussion or emphasis on it in the text. The caption of the figure states: ― Northern Hemisphere temperatures were reconstructed for the past 1000 years (up to 1999) using palaeoclimatic records (tree rings, corals, ice cores, lake sediments, etc.), along with historical and long instrumental records ”.

26. Finding: In relation to ―hide the decline‖ we find that, given its subsequent iconic significance (not least the use of a similar figure in the TAR), the figure supplied for the WMO Report was misleading in not describing that one of the series was truncated post 1960 for the figure, and in not being clear on the fact that proxy and instrumental data were spliced together. We do not find that it is misleading to curtail reconstructions at some point per se , or to splice data, but we believe that both of these procedures should have been made plain

– ideally in the figure but certainly clearly described in either the caption or the text.



## 7.3.3 Withholding Data

27. Yamal - withholding data and information on sample size : This issue concerns the Briffa 2000 4 

### and Briffa 2008 5 

### publications. The underlying raw data were not archived at the time of publication. Since it was not owned by CRU, they had no right to archive it . The data were owned by others including Hantemirov and Shiyatov. CRU has stated that it directed the request it received for data to the owners. Whether it was a result of this, or otherwise, the requester was given a

copy 6 

### of the data in 2004 by Hantemirov. Later, following publication of Briffa 2008 7 

### , and again following requests for the data, CRU has stated that it asked the owners to archive the data, which they did.

28. Because the raw data was not archived, the information on the number of core counts used in each year was not easily available. As a result the reader could not know that the core count in the most recent years had dropped substantially. This fact grew in importance as the need to understand uncertainty grew. The information was actually published and in the public domain in 2002 8 

### but it is arguable that a user of the CRU Yamal chronology would not have known to look

4 Ibid. 1

5 Briffa, K. R., Shishov, V.V., Melvin, T. M., Vaganov, E. A., Grudd, H., Hantemirov, R. M., Eronen, M., and Naurzbaev, M. M. 2008. Trends in recent temperature and recent tree growth spanning 2000 years across Northwest Eurasia. Philosophical Transactions of the Royal Society B- Bi ological Sciences 363:2271-2284

6 It has also been pointed out to us by CRU that McIntyre acknowledged this in a comment (number 61) made underneath a Climate Audit post which states that McIntyre did not realize that this was the series used by Briffa when he received it. http://climateaudit.org/2009/10/05/yamal-and-ipcc-ar4- review-comments/.

7 Ibid. 5

8 Hantemirov, R. M., and Shiyatov, S. G. 2002. A continuous multimillenial ring-width chronology in Yamal, northwestern Siberia. Holocene 12:717-726CHAPTER 7: TEMPERATURE RECONSTRUCTIONS FROM TREE RING ANALYSIS



### 61 there unless explicitly directed. Hantemirov and Shiyatov 2002 was referenced in Briffa 2008 9



### .

29. Finding: It is evidently true that access to the raw data was not simple until it was archived in 2009 and this can rightly be criticized on general principles of transparency, although it may have been common practice at the time of the original publication. We find that CRU has not withheld the data (having correctly directed the single request to the owners). However, in the interests of transparency, we believe that CRU should have ensured that the data it did not own, but had relied upon in publications, was archived in a more timely way. This i s another example where a more open approach would have been helpful.


## 7.3.4 Mishandling Data

30. We have not focussed upon disagreements over comparisons of results using individual tree series for two reasons. First, because this is a science question and should be addressed through the normal channels of peer reviewed publication. Secondly, because as we have indicated above, it requires much more work to show that one is unequivocally more representative than the other, and even then requires the conjunction of several additional conditions to lead to an allegation of wrongdoing. We have addressed these conditions above, and we have found no evidence to substantiate them. Nevertheless we comment briefly upon Yamal as it has received so much attention and the Tornetrask series as it is subject of much misunderstanding.

31. Finding on Yamal: The Briffa 200 0 10 

### paper presents a collection of many chronologies (Yamal is simply one of them) assembled from many authors. It is quit e clear that Briffa simply reprocessed data from other authors (Hantemirov and Shiyatov in this case) and no selection was performed. In their submission to the Review CRU included a copy of work they had made available on the web 11



### . This was posted in response to criticisms published at Climate Audit 12 

### which described examples of variability between specific tree series in the region. This CRU response demonstrates the consistency of the original Yamal publication with that obtained using an updated analysis incorporating a comprehensive set of contemporary data. The Review is not aware of any equivalent comprehensive analysis which demonstrates that the conclusions of Briffa 2000 are inconsistent with the best analysis available today.

32. Finding on ―Bodging‖ in respect of Tornetrask

. Th e term ―bodging‖ has been used, including by Briffa himself, to refer to a procedure he adopted in 1992 13



### . The  ̳bodge‘ refers to the upward adjustment of the low -frequency behaviour of the density signal after 1750, to make it agree with the width signal. This ad hoc process was based on the conjecture that the width signal was correct. There is nothing whatsoever underhand or unusual with this type of procedure, and it was

9 Ibid. 5

10 Ibid. 1

11 http://www.cru.uea.ac.uk/cru/people/briffa/yamal2009/

12 http://climateaudit.org/2009/09/27/yamal-a-divergence-problem/

13 Briffa, K. R., Jones, P. D., Bartholin, T. S., Eckstein, D., Schweingruber, F. H., Karlen, W., Zetterberg, P., and Eronen, M. 1992. Fennoscandian Summers from AD 500: temperature changes on long and short timescales. Climate Dynamics 7:111-119.CHAPTER 7: TEMPERATURE RECONSTRUCTIONS FROM TREE RING ANALYSIS



### 62 fully described in the paper. The interpretation of the results is simply subject to this caveat. The conjecture was later validated 14 

### when it was shown to be an effect due to the standardisation technique adopted in 1992. Briffa referred to it as a

― bodge‖ in a private e -mail in the way that many researchers might have done when corresponding with colleagues. We find it unreasonable that this issue, pertaining to a publication in 1992, should continue to be misrepresented widely to imply some sort of wrongdoing or sloppy science.



## 7.4 Conclusions and recommendations

33. We do not f

ind that the data described in AR4 and shown in Figure 6.10 is misleading, and in particular we do not find that the question marks placed over the CRU scientists’ input cast doubt on the conclusions.

34. The variation within and between lines, as well as the depiction of uncertainty is quite apparent to any reader. All relevant published reconstructions of which we are aware are presented, and we find no evidence of exclusion of other published temperature reconstructions which would show a very different picture. The general discussion of sources of uncertainty in the text is extensive, including reference to divergence and it therefore cannot be said that that anything has been suppressed. Presenting uncertainty in this way is a significant advance on the
TAR.

35. We have seen no evidence to sustain a charge of impropriety on the part of CRU staff (or the many other authors) in respect of selecting the reconstructions in AR4 Chapter 6. This would require that all the conditions in paragraph 13 were met in respect of tree chronologies either used by, or created by, CRU. No evidence of this has either been presented to the Review, nor has it been assembled as a scientific study published elsewhere and subjected to scrutiny. For the same reasons we found no evidence that there is anything wrong with the CRU publications using the Yamal or other tree series.

36. We find that divergence is well acknowledged in the literature, including CRU

papers.

37. In relation to ―hide the decline‖ we find that, given it s subsequent iconic significance (not least the use of a similar figure in the IPCC TAR), the figure supplied for the WMO Report was misleading in two regards. It did not make clear that in one case the data post 1960 was excluded, and it was not explicit on the fact that proxy and instrumental data were spliced together. We do not find that it is misleading to curtail reconstructions at some point per se, but that the reason for doing so should have been described.

38. We find that CRU has not withheld the underlying raw Yamal data (having correctly directed the single request to the owners). But it is evidently true that access to the raw data was not simple until it was archived in 2009 and this can rightly be criticised on general principles. In the interests of transparency, we believe CRU should have ensured that the data they did not own, but had relied upon in publications, was archived in a more timely way.

14 Briffa KR and Melvin TM (2010 in press)CHAPTER 7: TEMPERATURE RECONSTRUCTIONS FROM TREE RING ANALYSIS



### 63 39. It is a matter for the IPCC Review to determine whether the conclusions were in line with IPCC processes and guidelines for levels of likelihood. In respect of that Review we offer the suggestion that putting the combination of different reconstructions upon a more rigorous statistical footing would help in the future to make confidence levels more objective.CHAPTER 8: PEER REVIEW AND INFLUENCING EDITORIAL POLICY OF SCIENTIFIC JOURNALS



### 64 

## CHAPTER 8: PEER REVIEW AND INFLUENCING EDITORIAL POLICY OF SCIENTIFIC JOURNALS

8.1 Background: Peer Review, Testing and Verification

1. The rigorous development of scientific ideas leans heavily on peer-reviewed publication in scientific journals that are subject to testing and challenge by other, published, scientifically derived evidence.

2. The processes whereby new scientific concepts are peer-reviewed and published

c an be complex, contentious and partial (see Chapter 5 and Appendix 5). Peer review is not a ―gold standard‖ that ensures validity, as some claim. The processes of testing, validation and replication are the means whereby error is discarded, uncertainty reduced and robust scientific progress made. Peer review attempts initial sieving-out of the demonstrably erroneous or trivial, thereby setting a high standard that encourages rigour. But it is not infallible. Many well-founded concepts are rejected and many erroneous ideas accepted.

3. If peer review is subverted to exclude apparently well-founded alternative views, or if journal editors are intimidated from considering their publication, progress on an important issue such as climate change can be seriously slowed or skewed.



## 8 .2 The Allegations

4. The broad allegation is that CRU made improper attempts to influence the peer review system, pressuring journals to reject submitted articles that did not support a particular view of climate change.

5. This can only be properly assessed by analysis of the individual cases cited. Specific examples used to support this allegation are the events surrounding the publication of a paper by Soon and Baliunas (2003) 1 

### , the issues relating to Dr Boehmer-Christiansen, editor of the journal Energy and Environment , and certain actions of Professor Briffa when he was editor of the Journal Holocene

.


## 8.3 The Soon and Baliunas Affair & Climate Research

6. The paper by Soon and Baliunas (2003) entitled Proxy climatic and environmental changes of the past 1000 years , published in the journal Climate Research

, reviewed 240 previously published papers on temperature trends during the last millennium. It challenged the conclusion of Mann et al (1998, 1999) 2 

### that the late 20 th 

### century was the warmest period of the last millennium on a hemispheric scale, and claimed that recent temperatures were by no means unprecedented over

1 Soon, W. and Baliunas, S. 2003. Proxy climatic and environmental changes of the past 1000 years. Climate Research 23 , 89- 110.

2 Mann ME, Bradley RS, Hughes MK (1998) Global-scale temperature patterns and climate forcing

ov er the past six centuries. Nature 392 :779

– 787. Mann ME, Bradley RS, Hughes MK (1999) Northern Hemisphere temperatures during the past millennium: inferences, uncertainties, and limitations. Geophys Res Lett , 26:759

– 762.CHAPTER 8: PEER REVIEW AND INFLUENCING EDITORIAL POLICY OF SCIENTIFIC JOURNALS



### 65 this period. It was greeted with enthusiasm by those sceptical of the hypothesis of anthropogenic global warming 3 

### . However it received a negative reception from many other climate scientists 4 

### on scientific grounds, viz. that it conflated qualitative data on temperature and precipitation from many sources that could not be combined into a consistent proxy record. That hostility is reflected in the released CRU e-mails (e.g. CRU 1051156418.txt, 1051202354.txt), and the following e-mail from Jones to Mann on 11.3.03 (1047388489.txt):

“I think the skeptics will use this paper to their own ends and it will set paleo back a number of years if it goes unchallenged. I will be emailing the journal to tell them I'm having nothing more to do with it until they rid themselves of this troublesome editor (de Freitas), a well-known skeptic in NZ. A CRU person is on the board but papers get dealt with by the editor assigned by Hans von Storch."

7. The S&B paper had been seen by four reviewers, none of whom had recommended rejection 5 

### , and had been accepted by de Freitas (one of 10 Climate Research review editors; papers could be submitted to any one of them). A number of review editors resigned as a reaction against the publication of what they regarded as a seriously flawed paper. The journal‘s publisher admitted that the journal should have requested appropriate revisions of the manuscript prior to publication 6 

### . The Editor in Chief resigned on being refused permission by the publisher to write an editorial about what he regarded as a failure of the peer review system. De Freitas was said to have described these events as ―a mix of a witch- hunt and the Spanish Inquisition‖ 7



### .

8. These events, and the e-mail quote in paragraph 6, have led to the allegation that normal procedures of publications were being improperly undermined by a group that included Jones 8



### .

9. Jones has responded in evidence 9 

### to us that the reaction to the S&B paper was not improper or disproportionate given what he saw as the self evident errors of the

paper. The arguments presented in the Eos article, if correct, are strongly put, and suggest that the reaction was based on a belief, for which evidence was adduced, that the science was poor. In light of the reaction of the Journa l‘s publisher , we do not believe that any criticism of Jones can be justified in this regard.

10. Finding: This was clearly a bruising experience for all concerned. But Richard Horton‘s paper (Appendix 5 ) and the comments on it in Chapter 5 suggest to the Team that this scale of reaction is not unusual in contested areas, and the peer review process does not provide insulation from it. The Review makes no judgement about the correctness or otherwise of the Soon and Baliunas paper, but we conclude that the strong reaction to it was understandable, and did not amount to undue pressure on Climate Research.

3 New York Times , 5 August 2003.

4 e.g. Eos , Vol. 84, No. 27, 8 July 2003.

5 http://www.sgr.org.uk/climate/StormyTimes_NL28 htm

6 A statement from Inter-Research, 2003. Climate Research, 24 , 197- 198

7 http://www.sgr.org.uk/climate/StormyTimes_NL28 htm

8 Montford, A.W. 2010. The Hockey Stick Illusion . Stacey International, London.

9 See the note of the CCER meetings held on 9 April 2010CHAPTER 8: PEER REVIEW AND INFLUENCING EDITORIAL POLICY OF SCIENTIFIC JOURNALS



### 66 

## 8.4 The Conflict with Dr Boehmer-Christiansen

11. Dr Boehmer-Christiansen is Reader Emeritus in Geography at the University of Hull and the editor of the scientific journal Energy and Environment . As part of her evidence to us, she submitted a copy of her evidence to the House of Commons Science and Technology Committee Select Committee for its inquiry into ― The disclosure of climate data from CRU at the UEA ‖. In it she wrote 10 

### :

“ As editor of a journal which remained open to scientists who challenged the orthodoxy, I became the target of a number of CRU manoeuvres. The hacked emails revealed attempts to manipulate peer review to E&E‟s disadvantage, and showed that libel threats were considered against its editorial team. Dr Jones even tried to put pressure on my university department. The emailers expressed anger over my publication of several papers that questioned the „hockey stick‟ graph and the reliability of CRU temperature data. The desire to control the peer review process in their favour is expressed several times”.

12. Apart from the allegation of  ̳ att empts to manipulate peer review‘ for which Boehmer-Christiansen presents no evidence in relation to the journal that she edits, her statement 11 

### to us implies that pressure on her University was designed to undermine her role as editor of Energy and Environment. She comments that ― a message was send to my head of department late last year by Phil Jones in relation to some other matter, which suggested that the University of Hull should dissociate itself from me as editor of Energy & Environment (Multi-Science) as I was causing difficulties for CRU .‖

13. This episode appears to have been precipitated by an email sent on 2 October 2009 (1254746802.txt) from Boehmer-Christiansen to Stephanie Ferguson at the UK Climate Impacts Programme, and copied to a number of others. Its title was : “Please take note of potetially [sic] serious scientific fraud by CRU and Met Office.” The email was brought to the attention of Jones by the ―help desk‖ of UKCP09 in the UK Department for Environment, Food and Rural Affairs (DEFRA). He then wrote an e-mail on 27 October 2009 (1256765544.txt) to Professor Graham Haughton, Boehmer-Christianse n‘s Head of Department, complaining about the e- mail as ― very malicious”. Haughton responded sympathetically, but commented that “I‟d want to protect another academic‟s freedom to be contrary and critical”, to which Jones replied: “I don‟t think there is anything more you can do. I have vented my frustration and have had a considered reply from you”.

14. Finding: We see nothing in these exchanges or in Boehmer- Christiansen‘s evidence that supports any allegation that CRU has directly and improperly attempted to influence the journal that she edits. Jones‘ response to her accusation of scientific fraud was appropriate, measured and restrained.

10 Memorandum submitted by Dr Sonja Boehmer-Christiansen (CRU 26), at 4.1

11 Dr Sonja Boehmer-Christiansen submission (no.43), 23 FebruaryCHAPTER 8: PEER REVIEW AND INFLUENCING EDITORIAL POLICY OF SCIENTIFIC JOURNALS



### 67 

## 8.5 Peer Review and Professor Briffa‟s E ditorship of Holocene

15. An e-mail (1054748574.txt) was sent by Briffa, as Editor of the journal Holocene

, on 4.06.03 to Ed Cook, a reviewer, as follows:

“I am really sorry but I have to nag about that review – Confidentially I now need

a hard and if required an extensive case for rejecting

– to support Dave Stahle‟s and really as soon as you can.”

This has been assumed to be an editor who wishes to have the paper rejected, for reasons that are not explicit, seeking help from a reviewer to do so. In general, reviewers are not, or should not be invited to reject, merely to review. This e-mail has been widely interpreted as Briffa perverting the purpose of peer review by asking a colleague for help in rejecting a paper that contained research findings contradictory to his own views 12



### .

16. However, put in context, the e- mail‘s significance changes. The reality is r eflected in a series of emails provided by Briffa 13 

### and which have been redacted to remove names, apart from that of Stahle, as follows :

17.06.02

–

Briffa requests references from Dr Dave Stahle and referee B for a paper submitted to Holocene on the use of a specific tree ring record.
07.08.02

–

Stahle submits a referee report to Briffa. It is strongly critical of some of the underlying analysis, though it comments that the paper is well written. Advises rejection but suggests that if substantial additional work were done, it could be suitable for publication.
09.08.02

–

Briffa to Stahle, thanking him for the review, and committing himself to reading the paper very carefully in anticipation of receipt of a second review.
28.05.03

– Briffa to referee B, pressing to send a second review.
0 4.06.0 3

– Briffa reiterates his request to referee B, sending the email copied in paragraph 15.
04.06.03

–

Referee B sends review to Briffa. In the event, the review is not as negative as that of Stahle: the paper is ―marginal at best, could justifiably be rejec ted‖. This is an immediate reply to the email copied in paragraph 15.
23.07.03

– One of the authors e-mails to Briffa enquiring about a decision on the paper.
24.07.03

–

Briffa to author. Apologis es for the delay (―the manuscript could have been dealt wit h much better‖) and encloses referees‘ comments. Offers possible fast-track publication if the referees comments could be dealt with.

We can find no evidence that the article was subsequently published but the evidence above demonstrates that the possibility of publication was not rejected.

12 e.g. Channel 4 News 3.2.10

13 Copies of Communications relating to Keith Briffa‟s editorial treatment of a submitted manuscript

. Available at www.cce-review.org/Evidence.phpCHAPTER 8: PEER REVIEW AND INFLUENCING EDITORIAL POLICY OF SCIENTIFIC JOURNALS



### 68 17. Finding: Although much has been made of the e-mail in paragraph 15 as evidence of an unprincipled approach to the role of editor, we see nothing in these exchanges that supports the interpretations of subverting the peer review process that have been placed upon it. It appears to reflect an Editor with a strongly negative review in hand, and who presumably has read the paper, asking for confirmation that the paper should be rejected, possibly to reduce one of the many complications that assail an editor; and in view of the delay in communicating to authors, hoping for a strong decision from the referee. On receiving a second, more equivocal review, he offers the authors the opportunity to re-submit. These exchanges illustrate some of the complications of an E ditor‘s life as referred to in Chapter 5 and in Appendix 5. They do not provide evidence of subversion of process in rejecting contradictory ideas as has been alleged.



## 8.6 Conclusions

18. In our judgement none of the above instances represents subversion of the peer

re view process nor unreasonable attempts to influence the editorial policy of journals. It might be thought that this reflects a pattern of behaviour that is partial and aggressive, but we think it more plausible that it reflects the rough and tumble of interaction in an area of science that has become heavily contested and where strongly opposed and aggressively expressed positions have been taken up on both sides. The evidence from an editor of a journal in an often strongly contested area such as medicine (Appendix 5) suggests that such instances are common and that they do not in general threaten the integrity of peer review or publication.CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC



### 69CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC



### 70 

## CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC

9

. 1 Background

1. Where scientific results and concepts are relevant to issues of public or policy interest, their complexities are often such that they need to be communicated in simpler, comprehensible language that includes explicit statements about uncertainties and errors.

2. The IPCC process and its assessment reports have been the prime routes through which the complexities of climate change are transmitted into the international governmental domain. Actions that undermined the rigour, honesty and

e xpressions of scientific uncertainty in such communications would be serious infringements of good scientific practice.


## 9.2 The Allegations

3. The above principles are the frame for allegations levelled at CRU that relate to the Review‘s remit . In broad terms the claim is that CRU attempted to prevent proper consideration of views which conflicted with their own through their roles in the IPCC.

4. If this is the case it would represent a failure to represent current scientific understanding impartially at the vital interface between science and policy. If this has been done, it has the potential to understate uncertainty, and to undermine the rigour of risk-based evaluations by government, with the potential for severe social and economic consequences.

5. There are two specific, analogous allegations. They primarily relate to papers that challenge important elements of IPCC work; the first about the interpretation of the CRUTEM instrumental temperature series and the second about the reconstruction of tree ring proxy temperature series, discussed earlier in different contexts in Chapters 6 and 7 respectively. It is alleged that these papers were not properly considered by the relevant IPCC writing groups, and that members of CRU played the major role in ensuring that this was not done.

6. The following approach is adopted for each of these two sets of allegations:

summary of the scientific basis of the challenge;

allegations about the improper treatment of the papers by CRU members in their roles in IPCC writing groups;

evidence in support of the allegations;

the response of the CRU members alleged to have been primarily involved;

evidence from the IPCC Review Editors (see Chapter 5, section 5.7) about the procedures of the relevant IPCC writing groups;

findings of the Review.CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC



### 71 

## 9. 3 The CRUTEM Temperature Series

9. 3.1 The Scientific Challenge

7. A paper by McKitrick & Michaels (2004, hereafter MM2004) 1 

### argued that a large proportion of the measured late 20 th 

### century warming was a consequence of increased economic activity and that many meteorological measurement sites had become increasingly influenced by the warming effect of urbanisation around them, the so- called ―heat island effect‖, and that the CRUTEM d ata set was not adequately adjusted for such non-climatic effects. They argued that removing the effects of this socio-economic transition would reduce the trend of land surface temperatures from 0.27 deg C/decade to 0.11 deg C/decade and possibly to 0.06 deg C/decade. If this claim were correct it would, for many, fundamentally unde rmine the argument that the increasing concentration of atmospheric carbon dioxide as a consequence of human activity was a major driver of climate change.



## 9. 3.2 The Allegations

8. The allegation that CRU withheld their data from scrutiny and adjusted it without justification is dealt with in Chapter 6. Here we are concerned with the specific allegation that CRU improperly dismissed contrary views about the interpretation of the CRUTEM series and prevented its serious consideration by the IPCC.



## 9. 3.3 Evidence in Support of the Allegations

9. The reaction of Jones, who was then one of two CLAs for Chapter 3 ( Obse rvations: Surface and Atmospheric Climate Change ) of the Working Group 1 report of IPCC AR4, to the MM2004 paper is revealed in an e-mail (1089318616.txt) sent on 8 th 

### July 2004 to Mann, that included: “The other paper by MM (McKitrick & Michaels, 2004) is just garbage. ... I can‟t see either of these papers being in the next IPCC report. Kevin (Trenberth, the other coordinating lead author for ch. 3) and I will keep them out somehow

– even if we have to redefine what the peer- review literature is!” It indicates clearly his awareness of the paper prior to any drafts of IPCC AR4, and his determination to exclude it from consideration by IPCC.

10. It has been suggested 2 

### that this determination to exclude is reflected in the fact that MM2004 was not referred to in either the first order draft of Chapter 3 in August 2005 or in the second order draft of March 2006. Nor was a paper drawing similar conclusions by de Laat and Maurellis (2006) 3 

### referred to. It has been suggested to us by McIntyre 4 

### that it was Jones who kept reference to these papers out of the first and second order drafts, with the implication that this was done improperly to prevent incorporation of conclusions contrary to those held by the CRU group.

1 McKitrick, R. and Michaels, P.J. 2004. A test of corrections for extraneous signals in gridded surface temperature data. Climate Research, 26 , 159- 173.

2 McKitrick submission (no.15), 26 February

3 de Laat, A.T.J. and Maurellis, A.N. 2006. Evidence for influence of anthropogenic surface processes on lower tropospheric and surface temperature trends. International Journal of Climatology, 26, 897

- 913.

4 McIntyre submission (no. 23), 2 MarchCHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC



### 72 11. The omission of reference to MM2004 was criticised by expert reviewer Vincent Gray for Chapter 3, but this was rejected by the IPCC writing team for this Chapter (see Chapter 5 paragraph 24) with the argument that large scale patterns of atmospheric circulation over the continents and stronger warming of th e continents would produce patterns of warming such as those recorded by temperature stations 5



### .

12. The final draft of Chapter 3 did however include reference both to McKitrick & Michaels (2004) and de Laat and Maurellis (2006), in the following paragraph:

“McKitrick and Michaels (2004) and De Laat and Maurellis (2006) attempted to demonstrate that geographical patterns of warming trends over land are strongly correlated with geographical patterns of industrial and socioeconomic development, implying that urbanisation and related land surface changes have caused much of the observed warming. However, the locations of greatest socioeconomic development are also those that have been most warmed byatmospheric circulation changes (Sections 3.2.2.7 and 3.6.4), which exhibit largescale coherence. Hence, the correlation of warming with industrial and socioeconomic development ceases to be statistically significant. In addition, observed warming has been, and transient greenhouse-induced warming is expected to be, greater over land than over the oceans (Chapter 10), owing to the smaller thermal capacity of the land.”

13. It has been surmised in a submission to the Review by McKitrick 6 

### that Jones wrote the above paragraph and bears responsibility for its inclusion. The same submission alleges that it represents a ―fabricated conclusion‖ or ―invented evidence‖ and that only the derivation of a p -value 7 

### from a statistical test that is compatible with the claim of statistical insignificance would rebut this allegation of fabrication. It alleges that this is evidence of bias, and that after attempts to exclude evidence that conflicted with the preferred CRU interpretation of the CRUTEM data series from drafts 1 and 2, reasons were contrived in the published draft for the specific purpose of rejecting this evidence. If this were correct, such actions would appear to violate the principles in Chapter 5, of the duty of scientists to ensure that uncertainties are clearly transmitted to those that have the responsibility for deciding on any contingent actions.

5 Gray: “The „corrections‟ to the surface temperature record have always been based on very poor evidence. The many references to studies on individual or regional stations which find the need for much higher corrections than are currently applied, are ignored. Now you have ignored the persuasive evidence of McKitrick and Michaels 2004 Climatic Research 26 156-173 who have shown a significant influence on your „corrected‟ figures of a series of socioeconomic factors. You cannot just ignore this paper.” Response of the IPCC writing team: “ References are plentiful. Those of value are cited. Rejected. The locations of socioeconomic development happen to have coincided with maximum warming, not for the reasons given by McKitrick and Michaels (2004) but because of the strengthening of the Arctic Oscillation and the greater sensitivity of land than ocean to greenhouse forcing owing to the smaller thermal cap acity of land. Parker (2005) demonstrates lack of urban influences.” http://pds.lib harvard.edu/pds/view/7795947?n=7&s=4&imagesize=1200&jp2Res=.25&rotation=0 (comment 3-34).

6 McKitrick submission (no.15), 26 February

7 The p -value in statistics is the probability of obtaining a statistic at least as extreme as the one actually observed, assuming that there is no relationship between the measured phenomena. The lower the p-value, the less likely that there is no relationship. McKitrick and Michaels (2004) obtained a pvalue of 0.002, suggesting a significant relationship between climatic and socio-economic trends.CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC



### 73 14. It has been further alleged 8 

### that the changing response to the McKitrick and Michaels (2004) paper between the response to Gray and the statement in paragraph 13 of AR4, both assumed to have been by Jones, is evidence of a preparedness to accept any argument, irrespective of how well founded, that would refute conclusions to which he was opposed.



## 9. 3.4 Jones‟ R esponse

15. These allegations were put to Jones. We summarise his oral and written responses as follows 9

, 10 

### :

Jones comments that the e-mail of July 8, 2004 was sent on the spur of the moment and quickly forgotten. No pattern of behaviour with respect to his IPCC work could be construed from this one email.

The reason for the strong response of the email and the justification for not including reference MM2004 in the early drafts is that it can be readily shown to be scientifically flawed.

The basis for this statement is that if the CRUTEM3 trend is reduced by the factor claimed by MM2004, the land-based record then becomes incompatible with the ocean and the satellite record. MM2004 make no mention of this in their paper. In writing Chapter 3 of AR4 the author team were mindful of this. MM2004‘s analysis of the land surface temperature record is completely at odds with the rest of the surface and lower tropospheric temperature records. MM2004 also fails to take into account the effects of changes in the atmospheric circulation.

In summary, the atmospheric circulation has been shown, in numerous studies, to account for large scale patterns of temperature change. Before undertaking the kind of analysis in MM2004, it is essential to account for known signals (i.e . the N orth Atlantic Oscillation (NAO) 11 

### and El Nino Southern Oscillation (ENSO) 12 

### and possibly others) and then examine the residuals. It does not make sense to calculate the p -value without allowing first for the atmospheric effects and their spatial autocorrelation in that calculation.

In view of these arguments, it was reasonable to exclude MM2004, as the IPCC reports are assessments, not reviews of the literature. The author teams were chosen for their experience and expertise, and reliance was placed on this in determining which work should be included and discussed in the

8 McKitrick submission (no.15), 26 February

9 Response from Professor Jones to questions from Professor Geoffrey Boulton (20 April)

10 Clarification from Professor Jones to additional questions from Professor Geoffrey Boulton (3 May)

11 The North Atlantic Oscillation (NAO) and the associated Arctic Oscillation are parts of the global wave structure in the northern hemisphere atmosphere. The Icelandic low and the Azores high are parts of this wave structure, and their east-west oscillations control the strength and direction of westerly winds and storm tracks across the North Atlantic.

12 ENSO (El Niño-Southern Oscillation) is a 5-year (on average) oscillation between warm and upwelling cold waters in the tropical eastern Pacific, which has a major impact on atmospheric pressure in the region and knock-on effects on weather in many other parts of the world.CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC



### 74 assessment.

Jones commented that the decision about which papers to include and which to exclude was a collective one of the author team of Chapter 3 of AR4. Jones stated that he did not write the relevant text in the final report as has been assumed: the lead on the relevant section (3.2) was another member of the writing team. The suggestion to make a response did not come from Jones, as he was not the responsible person for the section. He did, however, agree with its inclusion in the final draft as a part of the overall writing team and a CLA fo r Chapter 3.

Jones explained that the decision to include MM2004 (and de Laat and
Maur ellis, 2006) was made at the final plenary team meeting in Bergen, and as stated the text was seen by the whole writing team. It had not been possible to include de Laat and Maurellis (2006) until then, as it was not published until after the third Lead Author‘s meeting. Discussion of MM 20 04 can be seen in comments numbered 3-283 to 3-289 of the Second Order Draft of Chapter 3 13



### . In two of these comments (3-284 and 3-285) it was stated that reference would be made to MM 20 04 in Section 3.2.2.2 with some text, which would point out that the papers by MM 20 0 4 and de Laat and Maurellis were biased. The fact that the Chapter author team had now read de Laat and Maurellis is referred to in response to comment 3-289. The comments were signed off by the two Review Editors for the Chapter.

Jones also noted to us that there were three criticisms made of the draft of the Summary for Policy Makers (SPM) based on those of MM2004, but that these were rejected by the SPM writing team (which did not include any members of the Chapter 3 team). They regarded them as inconsistent with a large body of the climatological literature addressed in Chapter 3 (comments numbers 482, 864 and 1005) 14



### .



## 9. 3.5 Evidence from IPCC Review Editor for Chapter 3 (Professor Sir Brian Hoskins)

16. The role of Review Editors was to ensure that all comments from expert and government reviewers were given appropriate consideration, to advise lead authors how to handle contentious and controversial issues and to ensure that genuine controversies were reflected adequately in the text. Professor Hoskins was one of three Review Editors for Chapter 3. Telephone evidence was sought from Hoskins about the ways of working of the author team for Chapter 3. The full summary of the interview with Hoskins is available on the Review website 15



### .

17. Hoskins confirmed that LAs, working individually and as small groups, were responsible for the collation and primary assessment of material relevant to the

13 Comments are available at http://hcl.harvard.edu/collections/ipcc/ by scrolling down to appropriate

comments.
14 Ibid. 13

15 Evidence from Review Editors for Chapters 3 and 6 of the IPCC Fourth Assessment Report on “The Physical Science Basis” (June 2010)CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC



### 75 topics for which they were responsible. The CLAs led the plenary meetings of the writing team prior to production of each of the drafts, led the process of overall collation of the Chapter material and the production of the initial drafts of the First and Second Draft Reports and the Final Draft Report of the Chapter. These drafts were discussed and agreed during plenary meetings of the whole writing team.

18. There were a very large number of comments from reviewers, of which a majority were from a relatively small group. The Review Editors made sure that they were all given proper consideration, and that they were either responded to by a change in the text or by an adequate reason for omission that was recorded in the author responses to expert and governmental review comments 16 

### . Hoskins, as a Review Editor, took part in the Chapter 3 plenary discussions and ensured that conflicting views were addressed.

19. Led by the two CLAs, Jones and Trenberth, the writing team for Chapter 3 was assiduous in dealing with comments. Hoskins was very impressed by Jones‘ attention to detail, and the rigour of the Chapter 3 process.

20. The levels of confidence and uncertainty reflected in the drafts were based on the consensus of a group of CLAs and LAs who were chosen for their expertise and experience in relevant fields. Irrespective of whether a paper is published in a peer reviewed journal, it is the responsibility of the whole team to assess whether a paper‘s conclusions are robust and to justify whether its arguments should carry weight in the assessment. These decisions for each draft were taken in plenary sessions of the whole team. Hoskins said that it is inconceivable that a paper making significant claims relevant to the work of IPCC and the Chapter 3 team would not be considered by the team as a whole. The basis for rejecting one of the papers that is a focus of the allegation is included in IPCC records 17 

### . Decisions about the inclusion of the MM2004 paper would have been taken by the whole team. Jones‘ voice would have been one amongst many.



## 9. 3.6 Findings

21. The essential issues in determining whether there is substance to the allegation that Jones misused his position to exclude well-founded conclusions that conflicted with his prior views and ―invented‖ an explanation in order to reject them are as follows:

i) Was the decision to exclude the MM2004 paper from the first and second drafts an unreasonable one?

ii) Was the reason given for the rebuttal of MM2004 in the final draft ―invented‖?

iii) If the answer to either or both of these questions is ―yes‖, w as Jones the major

16 Available at: http://hcl.harvard.edu/collections/ipcc

17 The issue is discussed further in discussion of reviewers comments on the 2 nd order draft of Chapter 3 (footnote to comments 3-275 to 3-389), in part of which McKitrick objected to the exclusion of the MM2004 and de Laat and Michaelis 2004, to which the writing team responded by the addition of text to section 3.2.2.2 of the draft. This was included in the final draft of Chapter 3. The comments and responses can be found at: http://pds.lib harvard.edu/pds/view/7786376?n=40&s=4&imagesize=1200&jp2Res=.25&rotation=0CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC



### 76 influence in excluding early consideration and/or in providing an ―invented‖ explanation?

22. The answer to i) depends upon the scientific credibility of the article. The contemporary emails indicate a highly critical response to it, and Jones‘ email (paragraph 9) is very strongly negative. Was the absence of a discussion of MM2004 in the first and second drafts reasonable, or could it reflect suppression of a view merely because it conflicted with that of the writing team or of Jones? Was the analysis in MM2004, and later in de Laat and Maurellis (2006), so evidently flawed in an issue as important as the significance of the instrumental record of climate change (see Chapter 6), that it could be readily rejected in the AR4? Jones believes that it was (paragraph 15, bullets 2-4), although the persistence of a debate on the issue in peer reviewed journals, including on one side MM2004, de Laat and Maurellis (2006), McKitrick and Michaels (2007) 18 

### and McKitrick (2010) 19 

### , and on the other side Benestad (2004) 20 

### , Parker (2010) 21



### , Jones et al (2008) 22 

### and Schmidt (2009) 23 

### , suggests at least a continuing margin of doubt. Those within the writing team took one view, and a group outside it took another. It is not in our remit to comment on the rights and wrongs of this debate, but those within the team had been entrusted with the responsibility of forming a view, and that is what they did. They initially rejected inclusion of reference to MM2004, and in the final draft included a commentary on it explaining why they disagreed with its conclusions. It may be that the conclusions of MM2004 conflicted so strongly with a generally held view among climate scientists tha t rejection was made too easily; but in the absence of better evidence, this is mere speculation. The mechanisms of the IPCC did however ensure that a reference to the article (and to de Laat and Maurellis, 2006) was contained in the final draft.

23. The answer to ii) depends upon the implication that the response to MM2004 in the published Chapter 3 was not scientifically credible. Having read most of the relevant papers however, we observe a consistence of view amongst those who disagree with MM2004 that has been sustained over the last 6 years, that the large scale organisation of atmospheric circulation produces a spatially integrated response to forcing. Although we do not comment on the relative merits of the two views, we see no justification of the view that that this response was ―invented‖, or even that its various expressions in the response to reviewer Gray 24 

### or the final text are fundamentally different.

24. Irrespective of the above comments on issues i) and ii), the evidence of the Review Editor underlines the team responsibility for the text, and the unlikelihood

18 McKitrick R and Michaels P. 2007. Quantifying the influence of anthropogenic surface processes and inhomogeneities on gridded global climate data. Journal of Geophysical Research 112 : D24S09.

19 McKitrick, R., 2010: Atmospheric circulations do not explain the temperature-industrtrialization correlation. Statistics, Politics and Policy (in press

– summer 2010).

20 Benestad, R. 2004. Are temperature trends affected by economic activity? Comment on McKitrick and Michaels. Climate Research, 27 , 171- 173.

21 Parker, D. E., 2010: Urban heat island effects on estimates of observed climate change, Wiley Interdisciplinary Reviews: Climate Change

, 1 (1), 123-133, doi:10.1002/wcc.21.

22 Jones, P.D., Lister, D.H. and Li, Q., 2008: Urbanization effects in large-scale temperature records, with an emphasis on China. J. Geophys. Res

. 113 , D16122, doi:10.1029/2008/JD009916.

23 Schmidt, G.A., 2009: Spurious correlations between recent warming and indices of local economic activity. Int. J. Climatol

. 29 , 2041-2048.

24 Ibid. 5CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC



### 77 that a single voice could dominate on an important issue. Jones ‘ evidence is that McKitrick‘s ― surmise that Professor Jones ... wrote the paragraph (in Chapter 3) alone or in consultation with Trenberth, and bears responsibility for its inclusion” 25 

### is false, and that the lead responsibility for the relevant section was another specified member of the writing team.

25. We conclude that there is evidence that the text was a team responsibility. It is clear that Jones (though not alone) had a strongly negative view of the paper but we do not find that he was biased, that there was any improper exclusion of material or that the comments on the MM2004 paper in the final draft were ―invented‖ given the (continuing) nature of the scientific debate on this issue.



## 9. 4 The Tree Ring Proxy Temperature Series

9. 4.1 The Scientific Challenge

26. A paper by McIntyre and McKitrick (2003; hereafter referred to as M&M2003) 26 

### argued that the so called ―hockey stick‖ plot (Mann, Bradley and Hughes, 1998; hereafter referred to as MBH98) 27 

### contained both simple errors and serious statistical errors. It suggested that the ―hockey stick‖ shape of the MBH98 reconstruction was largely an artefact of these errors and of the selection of specific tree ring series.

27. This work posed a significant challenge to interpretation of the assessment by the IPCC of the history of climate of the last millennium prior to the instrumental record of the last 150 years. It cast doubt on the validity of the claim of MBH98 and of Mann, Bradley and Hughes (1999; hereafter MBH99) 28 

### that the northern hemisphere was warmer in the late 20 th 

### century than at any time during the last millennium, which has been assumed by many to be evidence in support of the argument for strong, human induced greenhouse warming over recent decades.



## 9. 4.2 The Allegations

28. It has been alleged that Briffa, in his role as lead author for Chapter 6 in Working Group 1 for AR4, and as the member of the writing team with the most relevant expertise, attempted to bias the scientific conclusions towards those of the MBH98/99 and to set aside the inconvenient evidence of M&M2003. It is alleged this behaviour was calculated to favour one particular view of climate change and its causes, and to discredit or ignore opposing views, without, at the time, an adequate scientific reason for doing so. It would therefore represent a failure to discharge a scientist‘s responsibility impartially to represent current scientific understanding at the vital interface between science and policy.

25 McKitrick submission (no. 15), 26 February

26 McIntyre, S. and McKitrick, R. 2003. Corrections to the Mann et al (1998) proxy database and northern hemisphere average temperature series. Energy and Environment, 14, 751- 771.

27 Mann, M.E., Bradley, R.S. and Hughes, M.K. 1998. Global-scale temperature patterns and climate forcing over the past six centuries. Nature, 392 , 779-787.

28 Mann, M.E., Bradley, R.S. and Hughes. M.K. 1999. Northern hemisphere temperatures during the last millenium: inferences, uncertainties and limitations. Geophysical Research Letters, 26(6), 759

- 762.CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC



### 78 

## 9. 4.3 Evidence in Support of the Allegations

29. The Chapter 6 writing team relied heavily on a paper that was in preparation by Wahl and Ammann (eventually published as Wahl and Ammann, 2007, hereafter referred to as WA2007) that purported to refute the arguments of M&M2003. I t has been alleged that Briffa played a central role in improperly using WA2007 to refute M&M2003, to discredit a paper that conflicted with his core hypothesis, and in doing so was willing to break IPCC rules:

The claim of refutation made in the second order and the published drafts of Chapter 6 was knowingly misleading, because the method used to support the conclusions of WA2007 was dubious, and in any case, relied upon material that was rejected for publication in Geophysical Research Letters , and was not made available until August 2008 in an online supplement to Ammann and Wahl 2007, hereafter referred to as AW2007, long after acceptance of the final draft of Chapter 6. Thus even if the paper did represent an effective refutation of the M&M2003 argument, it was not available prior to the acceptance of the final draft of Chapter 6 of AR4.

IPCC rules require papers that are to be referenced should be at least  ̳accepted‘ by journals by specific deadline dates. W A2007 missed these deadlines and should not have been quoted as evidence.

Briffa broke confidence by asking Wahl, who was not involved in the IPCC process, to comment on Chapter 6 text.

30. The second order draft text of IPCC 2007 WGI Chapter 6, assumed in the allegation to have been written by Briffa, and sent late in March 2006 to the Government and Expert Reviewers, included on page 29 the following text that relied on WA2007 to rebut M&M2003:

“McIntyre and McKitrick (2003) reported that they were unable to replicate the results of Mann et al. (1998). Wahl and Ammann (accepted) demonstrated that this was due to the omission by McIntyre and McKitrick of several proxy series used by Mann et al. (1998). Wahl and Ammann (accepted) were able to reproduce the original reconstruction closely when all records were included”.

31. This text was criticised by the Reviewer for the Government of the United States of America, who wrote in comment 6-750 29 

### :
“The use of Wahl and Ammann (accepted) does not comply with WG1‟s deadlines and all text based on this reference should be deleted. WG1‟s rules require that all references be “published or in print” by De cember 16, 2005. Wahl and
Ammann was “provisionally accepted” on that date, and not fully accepted until February 28, 2006, at which time no final preprint was available. Substantial changes were made in the paper between December 16, 2005 and February 28,

29 Comments are available at http://hcl.harvard.edu/collections/ipcc/CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC



### 79 2006, including insertion of tables showing that the MBH98 reconstruction failed verification with r-squared statistics, as had been reported by McIntyre and McKitrick in 2003. These tables were not available in the draft considered by WG1 when developing the second- order draft.” The response to this was: “Rejected – the citation i s allowed under current rules” (Comment 6-1158) 30



### .

32. It was alleged that the material derived from WA2007 that was the rationale for the text in the final version of Chapter 6 was based on material that was not published or openly available until after the last deadline for the final draft. Their evidence should therefore not have been included.

33. In an email dated 18 July 2006 (1153470204), Briffa wrote to Wahl, who was not an official Expert Reviewer, as follows:

“Gene I am taking the liberty (confidentially) to send you a copy of the reviewers comments (please keep these to yourself) of the last IPCC draft chapter. I am concerned that I am not as objective as perhaps I should be and would appreciate your take on the comments from number 6-737 onwards, that relate to your reassessment of the Mann et al work. I have to consider whether the current text is fair or whether I should change things in the light of the sceptic comments. In practise this brief version has evolved and there is little scope for additional text, but I must put on record responses to these comments - any confidential help , opinions are appreciated . I have only days now to complete this revision and response. note that the sub heading 6.6 the last 2000 years is page 27 on the original (commented) draft. Cheers Keith”

34. It is alleged that this e- mail is an appeal to a strong proponent of the ―hockey stick‖ plot for assistance in coping with the comments of reviewers sceptical about it, that it hands confidential material to Wahl for him to help rebut the comments from Expert Reviewers critical of the Wahl and Ammann paper, and that it breaks rules of confidentiality at a stage when even official reviewers were denied ready access to review comments. It is implied that Briffa was prepared to go to exceptional and improper lengths to bolster a case that he supported and to defend it against alternative views.

35. Finally, it is alleged that the relevant paragraph on p.466 in Chapter 6 of the AR4 Final Report leaves the last word to Wahl and Ammann, and the reader is left with the clear impression that th e M&M2003 criticisms have been rebutted, although the work claimed to be the rigorous basis of this rebuttal had missed or was long after IPCC deadlines.

30 Ibid. 29CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC



### 80 

## 9. 4.4 Responses from Briffa

36. These allegations were put to Briffa. We summarise his oral and written responses as follows 31

, 32

, 33

, 34 

### :

a) Evaluation of the M&M2003 work by the Chapter 6 writing team

Briffa responded that the M&M2003 work was taken very seriously by the Chapter 6 writing group. There was genuine concern about whether its claim that the MBH98 reconstruction could not be replicated was correct. It was important to assess the merits of this, as a failure in replication would, if substantiated, be crucial in assessing confidence in the MBH99 reconstruction and the trend of past climate. Did M&M2003 provided sufficient grounds for dismissing the view of temperature change in MBH98?

The evaluation of this issue was very similar in both the second order and final, published versions of Chapter 6. The final version is:

“McIntyre and McKitrick (2003) reported that they were unable to replicate the results of Mann et al. (1998). Wahl and Ammann (2007) showed that this was a consequence of differences in the way McIntyre and McKitrick (2003) had implemented the method of Mann et al. (1998) and th at the original reconstruction could be closely duplicated using the original proxy data. McIntyre and McKitrick (2005a,b) raised further concerns about the details of the Mann et al. (1998) method, principallyrelating to the independent verification of the reconstruction against 19thcentury instrumental temperature data and to the extraction of the dominant modes of variability present in a network of western North American tree ring chronologies, using Principal Components Analysis. The latter may have some theoretical foundation, but Wahl and Ammann (2006NOTE) also show that the impact on the amplitude of the final reconstruction is very small (~0.05°C; for further discussion of these issues see also Huybers, 2005; McIntyre and McKitrick, 2005c,d; von
S torch and Zorita, 2005).”

The first sentence above reflects the concern that the results of MBH98 could not be replicated.

Briffa noted however that M&M2003 had not followed the MBH98 method and that Rutherford et al. (2005) 35 

### had pointed out that the M&M2003 approach to calculation resulted in the elimination of 77 out of the 95 pre-1500 tree ring

31 Response to specific questions raised by Professor Geoffrey Boulton, in his letter of 6 May 2010, in his role as a member of the Muir-Russell Review team (19 May)

32 Issues for discussion with Briffa and Jones on 9 April 2010 (April 10 )

33 Response to Additional Question regarding Keith Briffa‟s request to Eugene Wahl and his response (June 2010)

34 Copies of Communication s relating to Professor Briffa‟s editorial treatm ent of a submitted manuscript (June 20 10)

35 Rutherford, S., M.E. Mann, T.J. Osborn, R.S. Bradley, K.R. Briffa, M.K. Hughes and P.D. Jones, 2005: Proxy-based Northern Hemisphere surface temperature reconstructions: Sensitivity to methodology, predictor network, target season and target domain. Journal of Climate 18 , 2308-232.CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC



### 81 proxy series used by MBH98.

WA2007 had then shown that the results of MBH98 could be replicated very closely using their implementation of the MBH98 methods and using the same data. This is pointed out in the second sentence of the above paragraph from Chapter 6.

Briffa pointed out that the AR4 text did not state that WA2007 had disproved the concerns of M&M2003. Instead it considered their possible impact on the final reconstruction, citing papers that had assessed this impact including, but not exclusively, WA2007. These results indicated that the impact on the final reconstruction might be relatively small, leading to the view, contained in the AR4 text, that the criticisms raised against the MBH98 reconstruction were not sufficient to discount the evidence that it represented.

Briffa commented that he believed the above quoted paragraph from the published report represented a proper and balanced response to the issues raised by M&M2003, that the issues were important to the assessment and that the reference to WA2007 was a significant part of the debate.

Briffa and his colleague Osborn 36 

### commented that in any case the MBH98 was only one of 12 such reconstructions in figure 6.10 in Chapter 6, and does not therefore dominate the picture. The M&M2003 series is not presented in that figure as an alternative reconstruction, as McKitrick commented on the first order draft that, the draft ― trots out the straw man that we are selling an alternative climate history, despite our repeated and persistent statements that we are not trying to offer „our‟ climate history curve. ‖ 37



### Briffa also rejected the implication that this text was his responsibility, asserting that it was the responsibility of the whole writing group, not of any one person.

b) Deadlines

Briffa rejected the allegation that IPCC rules on deadlines were broken because of a determination to include reference to WA2007 (which was claimed to have been in press at the time). Details of these allegations and the responses to them from Briffa and Osborn are on the Review website 38



### .

Briffa pointed out that the scientific content in AW2007, referred to by WA2007, is equivalent to the content in the manuscript rejected by Geophysical Research Letters (this was not rejected for its scientific content, but for other editorial reasons).

c) Breaching confidentiality

36 Response to specific questions raised by Professor Geoffrey Boulton, in his letter of 6 May 2010, in his role as a member of the Muir-Russell Review team (May 2010)

37 Minute 6-1319 at http://hcl.harvard.edu/collections/ipcc/

38 Ibid. 33CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC



### 82 Briffa responded to the allegation of having broken confidentiality in sending draft text to Wahl to comment on, that there is no proscription in the IPCC rules to prevent the author team seeking expert advice when and where needed. The Technical Support Unit (TSU) and the CLAs of Chapter 6 agree that the author team was allowed to seek such advice. Copies of communications from both CLAs (Jansen and Overpeck) and the IPCC WG1 TSU are provided by Briffa (and published on the website) to provide support to Briffa‘s claim that his actions did not contravene IPCC procedures.

Briffa asserts that Wahl was asked for comment on text as a knowledgeable and objective arbiter and as such was a wholly reasonable judge of whether the responses were appropriate. Given his particular expertise on the details of the Mann et al. methodology and most importantly the implications for the character of the Mann et al. reconstruction, Briffa felt justified in seeking his advice and in using specific wording in a very few responses that were based on the text of a paper co-authored by Wahl (AW2007). Wahl did not write any of the main text, though he did make some suggestions for very minor edits. Briffa‘s evidence includes copies of the relevant email exchanges with Wahl 39



### , which also confirm that both Jansen and Overpeck, as CLAs, were aware of th e

a pproach to Wahl.



## 9. 4.5 Evidence from IPCC Review Editor for Chapter 6 (Professor John Mitchell)

37. Mitchell was one of two Review Editors for Chapter 6 of the AR4 Working Group 1 report. A telephone interview with him was conducted to establish how

unpu blished and ―in press‖ material was handled and how decisions about the text of successive drafts were made. He commented as follows 40 

### :

“I was not aware of the debate about whether the Wahl and Ammann paper had or had not met the deadline for the 2 nd 

### order draft for chapter 6, until after the event. The concentration on specific deadlines however misses the larger point. It must be recognised that if only published sources were used, the report would be two years old by the time of publication. In a fast-moving area such as climate change research, assessments could be significantly behind the times if important, but as yet unpublished, new results could not be used. The assessments for policymakers could also therefore be behind the times”.

“In earl ier assessments, there had been a relatively liberal regime in using unpublished material provided that there was a sound basis for regarding it as rigorous or reliable, although priority was always given to finding published sources. In AR4 however, the regime was tightened significantly, so that such material was only to be used under exceptional circumstances, but the use of unpublished material was not prohibited. „Hockey

- stick‟ issues were regarded at

39 Response to Additional Question regarding Keith Briffa‟s request to Eugene Wahl and his response (June 2010), available at: www.cce-review.org/Evidence.php

40 Evidence from Review Edit ors for Chapters 3 and 6 of the IPCC Fourth Assessment Report on “The Physical Science Basis”CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC



### 83 the time as sufficiently important to justify using new data. The dilemma between using only published material and being out of date, or using more recent unpublished material was increased in AR4 as the „latest publication date‟ was about 12 months earlier than in the process than in the previous assessm ent”.

“The email is problematic (para 33). On the one hand it appears to reflect an honest request to an expert for a comment about the extent to which the author is being balanced and fair. On the other hand, it stresses the need for confidentiality in three places, implying that the author realizes that the approach may be improper. There was also a leak of an early draft of the WG1 report to the press which led to IPCC emphasizing the need to maintain confidentiality in general which may have been at the back of the author‟s mind”.

“ In principle however there is nothing in IPCC rules that prevents an author from seeking external help, comment or judgement on text through consulting his peers. It is questionable whether expert reviewers‟ comments, in some * ways analogous to the comments of a peer reviewer for a journal, should be shared with an third party without their consent. (*Note that unlike most peer-reviewed journals, IPCC reviewers names and comments are made available at the end of the process).”



## 9. 4.6 Findings

38. The essential issues to be resolved in relation to the allegations are:

Was the McIntyre and McKitrick (2003) paper dealt with in a reasonable fashion in the draft and final versions of Chapter 6?

The evidence and narrative provided by Briffa is persuasive that these issues were dealt with in a careful and reasonable fashion that took into account the importance of the issues addressed in M&M2003. McIntyre or McKitrick may have wished to see them addressed in a different way, but they were addressed seriously and cogently.

Is there any evidence of any personal desire from Briffa to protect a fundamentalist line in defending the conclusions of MBH98/99?

The evidence of the Review Editor suggests that no one person in the writing team

cou ld have overridden the team responsibility for the text. Indeed, the evidence of a contemporary e-mail (1140039406; Feb 2006; to Overpeck) suggests that Briffa was unlikely to be an uncritical d efender of the MBH view of the  ̳ hockey stick ‘

, and wished to respect the view of the writing team as a whole:

― Peck, you have to consider that since the TAR, there has been a lot of argument re„hockey stick‟ and the real independence of the inputs to most subsequent analyses is minimal. True, there have been many different techniques used to aggregate and scale data - but the efficacy of these is still far from established. We should be careful not to push the conclusions beyond what we can securely justify - and this is not much other than a confirmation of the general conclusions of the TAR.....Just need to show the "most likely"course of temperatures over the last 1300 years - which we do well I think. Strong confirmation of TAR is a good result, given that we discuss uncertainty and base it on more data. Let us not try to over egg the pudding. For what it worth, the above comments are my (honestlyCHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC



### 84 long considered) views - and I would not be happy to go further. Of course this discussion now needs to go to the wider Chapter authorship, but do not let Susan [Solomon

– co -chair of IPCC WG1 for AR4] (or Mike) [Mann] push you (us) beyond where we know is right.”

Is the inclusion of references to WA2007 allegedly against the rules of IPCC evidence of a determination to rebut M&M2003 at all costs?

Taking into account the evidence of the Review Editor, IPCC papers, the statement from the CLA, and the importance of the issues raised by M&M2003, we consider it to be reasonable that work that might throw further light on these issues and to ensure that assessments were as up to date as possible should be included. We do not consider therefore that these were exceptional unwarranted efforts to defend a particular position, but reasonable attempt so use up to date information that might resolve an issue. They appear to be consistent with IPCC principles and to reflect a concern for objectivity.

Was there breach of confidentiality in having Wahl comment on draft text of the report and does this reflect determination to sustain a predetermined line ?

Altho ugh Briffa‘s e -mail 41 

### stressing confidentiality does imply an awareness of questionable conduct, the e-mail correspondence with Wahl 42 

### stresses in several places Briffa‘s concern to be fair to sceptical views. We see no evidence in the correspondence of anything other than a detailed determination to resolve a scientific issue. Nor do the IPCC Review Editor‘s comments to us suggest that what would normally be regarded in the research community as conventional requests for advice and help, were ruled out.

39. We conclude, in line with the comments made by Professor Mitchell, it was not unreasonable to include the WA2007 paper alongside the M&M2003 paper in presenting an up- to -date picture of the relevant scientific arguments. We do not find that Briffa acted improperly in the part he played in this, and note that in any case he did not have sole responsibility for the outcome.



## 9.5 Conclusions

40. In summary, we have not found any direct evidence to support the allegation that members of CRU misused their position on IPPC to seek to prevent th e publication of opposing ideas.

41. In addition to taking evidence from them and checking the relevant minutes of the IPCC process, we have consulted the relevant IPCC Review Editors. Both Jones and Briffa were part of large groups of scientists taking joint responsibility for the

41 Briffa to Wahl email: 1140039406.txt; Feb 2006 "I am taking the liberty (confidentially) to send you a copy of the reviewers comments (please keep these to yourself) of the last IPCC draft chapter. I am concerned that I am not as objective as perhaps I should be and would appreciate your take on the comments from number 6-737 onwards, that relate to your reassessment of the Mann et al work. I have to consider whether the current text is fair or whether I should change things in the light of the sceptic
comments. "

42 Response to Additional Question regarding Keith Briffa‟s request to Eugene Wahl and his response (June 2010)CHAPTER 9: COMMUNICATING INTO THE PUBLIC DOMAIN THROUGH THE IPCC



### 85 relevant IPCC Working Group texts and were not in a position to determine individually the final wording and content. We find that neither Jones nor Briffa behaved improperly by preventing or seeking to prevent proper consideration of views which conflicted with their own through their roles in the IPCC.CHAPTER 10 : COMPLIANCE WITH FoIA/ENVIRONMENTAL INFORMATION REGULATIONS



### 86 

## CHAPTER 10: COMPLIANCE WITH FoIA/ENVIRONMENTAL INFORMATION REGULATIONS

10.1 Introduction and Method of Enquiry

1. This Chapter addresses the third part of our remit: “ Review the Climatic Research Unit‘s compliance or otherwise with the University of East Anglia‘s policies and practices regarding requests under the Freedom of Information Act 2000 1 

### ( ̳the
FoIA ‘) and the Environmental Information Regulations 2 

### ( ̳the EIR‘) for the release of data‖. It also reviews subject access requests under the Data Protection Act 1998

– as amended 3



### . Whilst compliance with FoIA, EIR and DPA fall within the

more gen eral topic of  ̳Governance‘ (covered in the following Chapter), they are extracted and covered in detail here, given their very specific relevance to the Review.

2. Interviews were carried out with the: Information Policy & Compliance Manager

– IPCM; Science Faculty FoIA/EIR Contact; Director of Information Services; and Key staff within CRU.

3. Discussions were also held with representatives of the ICO, both to co-ordinate the work of the Independent Review with that of the concurrent ICO investigation and to seek advice.

4. Notes of all these interviews are available on the Review website.



## 10.2 The Allegations

5. It is alleged by a number of correspondents (for example the submission by Matthews 4 

### ) and commentators that requests under the FoIA and the EIR were incorrectly denied. Other correspondents (for example the submission by Mann 5 

### ) have suggested that a number of these FoIA requests were inappropriate or frivolous. Similarly it is alleged that subject access requests under the DPA for access to e-mails specifically referencing the applicant were not fully complied with.



## 10.3 General Context

6. The Freedom of Information Act 2000 created new statutory rights of access, subject to a number of defined exemptions, to information held by a wide range of public bodies. It replaced the earlier  ̳Code of Practice on Access to Government Information‘ tha t was a non-statutory scheme. The general right of access under

1 See: http://www.opsi.gov.uk/Acts/acts2000/ukpga_20000036_en_1

2 See: http://opsi.gov.uk/si/si2004/uksi_20043391_en.pdf

3 See: http://www.ico.gov.uk/what_we_cover/data_protection/legislation_in_full.aspx

4 Matthews submission (no. 16), 1 March

5 Mann submission (no. 42), 28 FebruaryCHAPTER 10 : COMPLIANCE WITH FoIA/ENVIRONMENTAL INFORMATION REGULATIONS



### 87 this Act came into force on 1 st 

### January 2005. In cases where access to information is refused, the Act requires the public authority to give reasons for that refusal, including detailing any exemptions the public authority believes may apply. There is also a duty under the Act to offer reasonable advice and assistance to applicants seeking information.

7. The current Environmental Information Regulations (EIR) also came into force on
1 st 

### January 2005. These contain a similar general right of access to that defined under the FoIA. Their genesis however dates back to initial regulations based on the European Community (EC) Council Directive 90/313 published in 1992, to which UEA would have been subject. It is not clear whether this was widely understood either by the University or by those seeking information.

8. Both the FoIA and EIR:

encourage the use of formal publication policies (―Schemes‖) and pro -active dissemination of information; have initial 20 working day time limits for the public authority to respond to requests; require public authorities to provide advice and assistance to applicants and to provide the information in the form or format requested wherever reasonably possible. use the  ̳Public Interest‘ test. Information can only be withheld if: ― in all the circumstances of the case, the public interest in maintaining the exception outweighs the public interest in disclosing the information ‖; have defined appeal processes (albeit with some subtle differences on time limits); and
have common recourse to the Information Commissioner and to the Information Tribunal.

9. Key differences 6 

### between the FoIA and EIR regimes are that, under the EIR: requests can be made verbally and do not have to be submitted in writing (or by e-mail) with a name and address; there is a clearer definition of  ̳information held‘ than under the FoIA : ― If the information is in the authority's possession and has been produced or received by the authority, or if it is held by another person on behalf of the authority ‖; a request cannot be rejected purely on grounds of cost; the allowed initial response deadline can be extended to 40 working days for particularly complex requests; withholding of information under exemptio ns remains subject to the  ̳public interest‘ test; there is an emphasis on improved decision making and participation; and there is no direct equivalent to the FoIA  ̳vexatious requests‘ provisions.

There are no guarantees of absolute confidentiality under the EIR.

10. The EIRs only cover environmental information, albeit this is subject to a very broad definition i 

### (see endnote after paragraph 35). The FoIA covers all other

6 See: http://www.defra.gov.uk/corporate/policy/opengov/eir/slides-leaflets htmCHAPTER 10 : COMPLIANCE WITH FoIA/ENVIRONMENTAL INFORMATION REGULATIONS



### 88 information held by public authorities, but with a specific exemption for environmental information.

11. The DPA 1998 (as amended) gives individuals the right to know what information is held about them. It provides a framework to ensure that personal information is handled properly. The Act states that anyone who processes personal information must comply with eight principles, which make sure that personal information is: Fairly and lawfully processed Processed for limited purposes Adequate, relevant and not excessive Accurate and up to date Not kept for longer than is necessary Processed in line with your rights
Secure Not transferred to other countries without adequate protection

12. More specifically in the context of this Chapter, the Act provides individuals with important rights, including the right to find out what personal information is held on computer and most paper records.

13. UEA‘s (and thus CRU‘s) formal processes are described in 3 documents: ―The Code of Practice for Responding to Requests for Information under the Freedom of Information Act 2000‖ 7 

### , which is based on the Lord Chancellor‘s Code and sets out the manner in which UEA processes requests. ―Guidance for Staff‖ 8 

### , which directs how UEA staff should respond to requests for information. ―Requests for Information‖ 9 

### , which explains how to make a request for information, and how the UEA expects to respond.

14. UEA‘s FoIA/EIR processes are based on concentric circles. The IPCM is considered the central point. He: should always be the co-ordinator for FoIA/EIR requests; maintains the formal FoIA/EIR logs on requests received; establishes a case file for each request; makes the initial determination as to whether to treat a request under the FoIA

or EIR regime; determines in cooperation with the Faculty contact and relevant staff whether the information sought is indeed ―held‖ i n statutory terms; and identifies and approaches the key provider via the appropriate faculty or central unit contact.

15. The IPCM is the key person within UEA with a detailed understanding of the FoIA and the EIR. He maintains liaison with the ICO. He is also the key training provider to others within UEA.

7 See: http://www.uea.ac.uk/is/strategies/infregs/FOIA+Code+of+Practice+for+Responding+to+Requests

8 See: http://www.uea.ac.uk/is/foi/guidance

9 See: http://www.uea.ac.uk/is/foi/foi_requestsCHAPTER 10 : COMPLIANCE WITH FoIA/ENVIRONMENTAL INFORMATION REGULATIONS



### 89 16. The next circle comprises FoIA/EIR contacts at the Faculty or Central Unit level. There are FoIA/EIR contacts in place for each of the four University Faculties and for each of the central university units (such as Information Systems and the Registrar‘s Department ). These receive initial training and then yearly updates. It is understood that participation in these training sessions has been good. Beyond the initial training, it is understood that much learning is ―on the job‖.

17. The final circle comprises individual members of staff/researchers. An initial brochure was sent to all units in the University when the FoIA/EIR regimes came into force in 2005. Awareness training is made available on a voluntary basis.


## 10.4 Investigation

18. As stated in this Review‘s  ̳ Issues for Examination ‟ document

(Appendix 3), it is alleged that requests under the FoIA, EIR and DPA were incorrectly denied by UEA on advice from CRU. This is the subject of a separate inquiry by the ICO, but does fall within the terms of reference of the Review. The Review has remained in regular contact with the ICO to ensure that both investigations could continue in parallel (see paragraph 19). The Review is particularly concerned to address: What formal processes were in place both centrally and within CRU to ensure fair and impartial assessment of FoIA requests? Were there any processes in place centrally to review recommendations from CRU that information should not be released?

19. The improperly released e-mails contain a number of references that raise specific concerns with respect to FoIA, EIR and the DPA. In particular, since CRU uses data obtained from other bodies, guidance was sought from the ICO on the extent to which a public authority that was not a primary repository of data might be expected to act as a secondary source of that data for the purposes of FoIA/EIR.

The advice received was that : Neither FoIA nor EIR make a distinction as to whether a public body is a primary or secondary source of information or data. The point is simply whether they hold the information for the purposes of the legislation, i.e.

se ction 3(2) of FoIA and Regulation 3(2) EIR. If the public authority holds the information for any reason or in any form they must provide it or rely on a provision in the legislation to refuse the request. If however a request is received and the information requested is publicly available, and reasonably accessible, the public body can rely on either section 21 FoIA or Regulation 6 EIR and point the requester to these sources. “Reasonably accessible” will normally mean the information is published or available on demand. There is no obligation for the public body to apply section 21 or Regulation 6: the public body is free to choose to supply the data.CHAPTER 10 : COMPLIANCE WITH FoIA/ENVIRONMENTAL INFORMATION REGULATIONS



### 90 20. The Review recognises three time periods into which information requests should be grouped: before 2005, where the FoIA was not in operation but a precursor regime to the current EIR was operating; from 2005 until the start of 2007, when responses to requests were being handled informally by CRU; and
after 25 th 

### January 2007 when the first request was formally logged by the IPCM.

21. The Review had access to the complete formal log of FoIA/EIR requests, compiled by the IPCM, with respect to CRU since both current regimes came into force at the start of 2005. CRU only started to treat such requests formally via the IPCM in 2007. It should be noted however that there are extensive indications, from e-mails now in the public domain, of requests for information and some release of information prior to 2007. An example is the e-mail from Jones on 21 st 

### February 2005 (1109021312.txt), ― PS I'm getting hassled by a couple of people to release the CRU station temperature data. Don't any of you three tell anybody that the UK has a Freedom of Information Act! ‖. Indeed, thre e e-mails from 2004 also made various information requests.

22. Formal FoIA/EIR requests were initially quite limited. In 2007 four requests were received, of which two were given full release of the requested information but two, despite appeals, were rejected. In 2008 two requests were received, one was granted full release, but the other was rejected, both initially and upon appeal. In the first half of 2009 only one request was received and this was responded to in full.

23. But in the third quarter of 2009 a wave of requests was received. In the five days starting on 24 th 

### July, some 60 requests were logged by the IPCM. A further 10 requests were logged between the 31 st 

### July and 14 th 

### August. Some related to the raw station data underpinning the CRUTEM data sets and the vast majority sought details of any confidentiality agreements related to this data. The wordings bear the hallmarks of an organised campaign. One applicant (UEA Log 09/97) appears to have forgotten to customise the request before dispatch. The text reads: ― I hereby make a EIR/FOI request in respect to any confidentiality agreements) restricting transmission of CRUTEM data to non-academics involving the following countries: [insert 5 or so countries that are different from ones already requested] the date of any applicable confidentiality agreements; the parties to such confidentiality agreement, including the full name of any organization; a copy of the section of the confidentiality agreement that "prevents further transmission to non-academics". a copy of the entire confidentiality agreement.”

24. In the final quarter of 2009 a further wave of 41 requests was received, starting on 20 th 

### November. These were mostly related to the unauthorised release of e-mails and seeking information in the context of that release. However a number raised other issues such as: the extent of UEA staff training in FoIA/EIR issues; sourcesCHAPTER 10 : COMPLIANCE WITH FoIA/ENVIRONMENTAL INFORMATION REGULATIONS



### 91 of funding; and statistics on FoIA/EIR requests received. Still others harked back to previous themes of access to the CRUTEM data set and related confidentiality agreements.


## 10.5 Findings

25. Education and culture. Whilst we did identify evidence that UEA had widely distributed initial guidance at the introduction of the FoIA/EIR regimes in 2005, we also found a lack of engagement by core CRU team in understanding EIR/FoIA legal requirements and how these might legitimately impact them. There was evident confusion within the CRU as to how these requirements might be applied for example to data, code, and personal correspondence. We found a tendency to assume that no action was required until precedents had been set. As an example, on 21 st 

### January 2005 Jones wrote (1106338806.txt) : ― On the FOI Act there is a little leaflet we have all been sent. It doesn't really clarify what we might have to do re programs or data. Like all things in Britain we will only find out when the first person or organization asks. I wouldn't tell anybody about the FOI Act in Britain. I don't think UEA really knows what's involved. ‖ There was insufficient priority given from the UEA centre to motivating staff and to prompting continuing education. Various requests received by the CRU between 2005 and the start of 2007 had not been formally logged with the IPCM.

26. Foresight lacking . We found a lack of recognition by both CRU, and the University‘s senior management of the extent to which earlier action to release information or give full guidance to locate primary sources and to provide station identifiers might have minimized the problems. There are many references in the e-mails now in the public domain to ―hiding information‖, ―finding ways around releasing‖, or finding excuses not to release information. There was a fairly swift shift towards a lack of sympathy with the requesters, as seen in an e-mail from
Jones sent on 7 th 

### May 2005 (1083962092.txt) : ― Mike and I are not sending anything, partly because we don't have some of the series he wants, also partly as we've got the data through contacts like you, but mostly because he'll distort and misuse them

. ‖

We do not suggest that the allegations made against McIntyre are correct.

27. Unhelpful responses . We found a tendency to answer the wrong question or to give a partial answer. For example the very first formal FoIA request (UEA Log 07 -04) asked for the list of stations used in the preparation of HadCRUT3 and the raw data for these stations. The initial response refers simply to the availability of the raw data from other sources and not to the station list. The requester then enters into an extended correspondence trying to extract the station identifiers. An extract from 14th April 2007 is given below:

“While it is good to know that the data is available at those two web sites, that information is useless without a list of stations used by Jones et al. to prepare the HadCRUT3 dataset. As I said in my request, I am asking for:

1) A list of the actual sites used by Dr. Jones in the preparation of the HadCRUT3 dataset, andCHAPTER 10 : COMPLIANCE WITH FoIA/ENVIRONMENTAL INFORMATION REGULATIONS



### 92 2) A clear indication of where the data for each site is available. This is quite important, as there are significant differences between the versions of each site's data at e.g. GHCN and NCAR."

Without knowing the name and WMO number of each site and the location of the source data (NCAR, GHCN, or National Met Service), it is not possible to access the information. Thus, Exemption 21 does not apply - I still cannot

ac cess the data.

I don't understand why this is so hard. All I am asking for is a simple list of the sites and where each site's data is located. Pointing at two huge piles of data and say ing, in effect, "The data is in there somewhere" does not help at all.

To clarify what I am requesting, I am only asking for a list of the stations used in HadCRUT3, a list that would look like this:

WMO# Name Source 58457 HangZhou NCAR 58659 WenZhou NCAR 59316 ShanTou GHCN 57516 ChongQing NMS

e tc. for all of the stations used to prepare the HadCRUT3 temperature data.

That is the information requested, and it is not available "on non-UEA websites", or anywhere else that I have been able to find.

I appreciate all of your assistance in this matter, and I trust we can get it resolved satisfactorily.”

These station identifiers are finally promised on appeal, but not provided until some six months later following further prompting.

28. Deliberate actions to avoid release . There seems clear incitement to delete emails, although we have seen no evidence of any attempt to delete information in respect of a request already made. Two e-mails from Jones to Mann on 2 nd



### F ebruary 2005 (1107454306.txt) and 29 th 

### May 2008 (in 1212063122.txt) relate to deletion :

2 nd 

### February 2005: ― The two MMs have been after the CRU station data for years. If they ever hear there is a Freedom of Information Act now in the UK, I think I'll delete the file rather than send to anyone”

.

29 th 

### May 2008: ― Can you delete any emails you may have had with Keith re AR4? Keith will do likewise. He's not in at the moment - minor family crisis. Can you also email Gene and get him to do the same? I don't have his new email address. We will be getting Caspar to do likewise ”

.CHAPTER 10 : COMPLIANCE WITH FoIA/ENVIRONMENTAL INFORMATION REGULATIONS



### 93 There is a clear statement that e-mails had been deleted

– for example, an e-mail from Jones to Santer sent on 3 rd 

### December 2008 (1228330629.txt ): ― About 2 months ago I deleted loads of emails, so have very little - if anything at all .‖

It seems likely that many of these  ̳deleted‘ e -mails subsequently became public following the unauthorised release from the backup server. There is evidence that the IPCM did try to warn Prof. Jones about deliberate deletion of information; for example, an email from Jones to Santer (1228922050.txt) 10 th 

### December 2008

states : ― I did get an email from the FOI person here early yesterday to tell me I shouldn't be deleting emails - unless this was 'normal' deleting to keep emails

manage able!”

.

29. Imbalance of authority . The current structure for handling FoIA/EIR requests within UEA focuses very much on the IPCM. At interview he indicated that he felt: “ Very much the bull’s eye at the centre of the target ”. We believe that the UEA senior staff need to take more explicit responsibility for these processes, thus enhancing the authority and standing of the IPCM. We found that the IPCM may have lacked such standing within the University

s tructure and the authority to challenge the assertions of senior professors. In this context, the IPCM and the Faculty FoIA contact may not have been empowered to be sufficiently rigorous: Jones to Santer sent on 3 rd 

### December 2008 (1228330629.txt): When the FOI requests began here, the FOI person said we had to abide by the requests. It took a couple of half hour sessions - one at a screen, to convince them otherwise showing them what CA was all about. Once they became aware of the types of people we were dealing with, everyone at UEA (in the registry and in the Environmental Sciences school - the head of school and a few others) became very supportive. ” However at interview the IPCM explicitly denied that he behaved in the way suggested in this e-mail.

30. Lack of constructive challenge . Whilst we found that efficient basic control, logging and progress chasing processes for FoIA/EIR requests were in place, we found a lack of independent working-level challenge in these systems. We found that the appeals mechanisms in place lacked the resources for effective challenge to basic assumptions. Similarly, the escalation processes failed to react sufficiently quickly to the dramatic change in the volume and character of requests and to provide timely high-level review and resources.

31. Limited internal communication . We found a lack of understanding within University central functions of the presence of extensive, and long duration, backups of e-mail and other materials despite these being on a server housed within the central Information Technology (IT) facilities. Awareness of these might have led to much greater challenge of assertions regarding non-availability of material by CRU, notably in the case of a subject access request made under the DPA for material naming the requesting individual.

32. The Review found an ethos of minimal compliance (and at times non-compliance) by the CRU with both the letter and the spirit of the FoIA and EIR. We believe that this must change and that leadership is required from the Un iversity‘s most senior staff in driving through a positive transformation of attitudes. Public trust in science depends on an inherent culture of honesty, rigour and transparency.CHAPTER 10 : COMPLIANCE WITH FoIA/ENVIRONMENTAL INFORMATION REGULATIONS



### 94 The requirements of FoIA and EIR must not be seen as impositions. They are a necessary part of the implicit contract between the scientist and broader society. Such an open culture will also lead to the best science.



## 10.6 Recommendations

33. The Review offers the following recommendations for action within the UEA: Change fundamentally the perception that responsibility for FoIA/EIR compliance lies with administrative staff . University senior staff need to make clear their commitment to a culture of honesty, rigour and transparency, plus the supporting processes and resources. Review the resourcing and standing of the FoIA/EIR/DPA compliance and request handling processes . Our findings have highlighted significant problems in the areas of: imbalance of authority; lack of effective challenge at appeal; over dependence on single individuals; inadequate escalation processes and limited strategic oversight. A concerted and sustained campaign to win hearts and minds . This should include: promotion of the University‘s formal publication policy; incorporating more information on Fo IA/EIR/DPA responsibilities in the induction processes for new staff members; developing a rolling awareness campaign to focus the attention of established staff, particularly in the context of the changing landscape e.g. Queens University judgment (see paragraph 3 4 ); and issuing annual reminders of the importance of transparency and of key FoIA/EIR/DPA responsibilities; Once the improved awareness measures and processes are in place, to run a programme of independent, external, tests with requests for information to verify the continuing effectiveness of these operations . This is a special case of the more general recommendation on  ̳Audit processes‘ given in the Governance Chapter.

34. The Review offers the following more general recommendations:

Definition of research data . There is extensive confusion and unease within the academic community as to exactly how FoIA/EIR should be applied in terms of the materials developed during a research process. The Review believes that all data, metadata and codes necessary to allow independent replication of results should be provident concurrent with peer-reviewed publication. However the situation regarding supporting materials such as early drafts, correspondence with research colleagues and working documents is widely regarded as unclear. The American experience is instructive here. The so called ―Shelby Amendment‖ in 1998 directed the US ―Office of Management & Budget (OMB)‖ to produce new standards requiring all data produced under Federally funded research to be made available under the US Freedom of Information Act. This resulted in great concern within the US Scientific community, expressed through Congressional testimony, that a very broad interpretation of this requirement could seriously impair scientific research and collaboration. In the final OMB guidelines 10 

### , recognising these concerns,

10 Federal Register: March 16, 2000 Volume 65, Number 52 Page 14406. http://frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=2000_register&docid=00-5674-filedCHAPTER 10 : COMPLIANCE WITH FoIA/ENVIRONMENTAL INFORMATION REGULATIONS



### 95 ―research data‖ is defined as: ― the recorded factual material commonly accepted in the scientific community as necessary to validate research findings, but not any of the following: preliminary analyses, drafts of scientific papers, plans for future research, peer reviews, or communications with colleagues ‖. The Review recommends that the ICO should hold consultations on a similar distinction for the UK FoIA/EIR.

Orchestrated campaigns . As detailed in paragraph 23 , CRU was the subject of an orchestrated campaign of FoIA/EIR requests in late July and early August 2009. The Review believes that CRU helped create the conditions for this

c ampaign by being unhelpful in its earlier responses to individual requests for station identifiers and the locations from which specific, detailed station raw data could be downloaded. Similarly a clearer publication policy, reflecting the wishes of both the University and the research funders might have avoided these challenges. The Review team can however conceive of situations where such orchestrated campaigns might recur, with literally overwhelming impacts on small research units. We urge the ICO to give guidance on how best to respond to such organised campaigns, consistent with the underlying principles of openness.

Greater clarity in an evolving landscape

. Particularly in the light of the recent Queens University Belfast determination by the ICO in respect of the release of Irish Tree Ring data 11 

### , it would be helpful if the ICO could re-engage more generally with the Higher Education sector about their understanding of FoIA

a nd EIR obligations and also consider what further guidance could be provided for that sector. It would be particularly useful if guidance were available as to how long it is reasonable to retain data without release, pending full publication as part of a peer reviewed paper. It is however recognised that often such determinations have to be made on a case- by -case basis against a ―public interest‖ test.

35. As a final comment we find that a fundamental lack of engagement by the CRU team with their obligations under FoIA/EIR, both prior to 2005 and subsequently, led to an overly defensive approach that set the stage for the subsequent mass of FoIA/EIR requests in July and August 2009. We recognise that there was deep suspicion within CRU, as to the motives of those making detailed requests. Nonetheless, the requirements of the legislation for release of information are clear and early action would likely have prevented much subsequent grief.

11 ICO Case Ref: FS50163282 Date: 29/03/2010 Publ ic Authority: Queen‘s University Belfast Summary: The complainant requested electronic data relating to tree ring research (dendrochronology). The public authority confirmed that it held the requested information but refused to provide it citing section 12 of the Act. The Commissioner indicated to the public authority that the withheld information fell within the definition of environmental information under the EIR. The public authority subsequently cited the exceptions at regulations 12(4)(d), 12(4)(b), 12(5)(c) and 12(5)(e) to refuse the

13 cont information. The Commissioner finds that none of the exceptions is engaged and the withheld information should therefore be disclosed. The Commissioner also recorded a number of procedural breaches in the public authority‘s handling of the request.CHAPTER 10 : COMPLIANCE WITH FoIA/ENVIRONMENTAL INFORMATION REGULATIONS



### 96 i 

### EIR definition of environmental information (see paragraph 10) :

Any information in written, visual, aural, electronic or any other material form on:

(a) the state of the elements of the environment, such as air and atmosphere,

wa ter, soil, land, landscape and natural sites including wetlands, coastal and marine areas, biological diversity and its components, including genetically modified organisms, and the interaction among these elements; (b) factors, such as substances, energy, noise, radiation or waste, including radioactive waste, emissions, discharges and other releases into the environment, affecting or likely to affect the elements of the environment referred to in (a); (c) measures (including administrative measures), such as policies, legislation, plans, programmes, environmental agreements, and activities affecting or likely to affect the elements and factors referred to in (a) and (b) as well as measures or activities designed to protect those elements;(d) reports on the implementation of environmental legislation; (d) reports on the implementation of environmental legislation; (e) cost-benefit and other economic analyses and assumptions used within the framework of the measures and activities referred to in (c); (f) the state of human health and safety, including the contamination of the food chain, where relevant, conditions of human life, cultural sites and built structures inasmuch as they are or may be affected by the state of the elements of the environment referred to in (a) or, through those elements, by any of the matters referred to in (b) and (c).CHAPTER 10 : COMPLIANCE WITH FoIA/ENVIRONMENTAL INFORMATION REGULATIONS



### 97CHAPTER 11 : GOVERNANCE



### 98 

## CHAPTER 11: GOVERNANCE

11.1 In troduction and Method of Enquiry

1. This C hapter addresses the fourth part of the Review‘s remit, ―Review and make recommendations as to the appropriate management, governance and security measures for CRU and the security, integrity and release of data it holds‖, to the extent that these issues are not addressed in previous Chapters. The previous Chapter deals extensively with the major governance issue of FoIA and EIR compliance. This Chapter considers other relevant aspects of the governance framework, first as it relates to Research Management Systems in Section 11.2 and then secondly to the specific issue of the Management of Software, Data Management and Security in Section 11.3. The Chapter concludes with a set of recommendations in Section 11.4 on governance matters relating to both issues.

2. The Review Team decided to focus on the efficacy of current control systems in UEA relevant to CRU‘s research, while developing a broad understanding of the evolution of UEA‘s policies and practices.

3. Interviews were carried out with the:

Vice-Chancellor PVC, Research Enterprise and Engagement Registrar and Secretary Associate Dean for Research, Faculty of Science Director of Information Services; Information Communications Technology (ICT) Systems Director; and CRU (part-time) IT Manager Director of Research, Enterprise and Engagement and Manager of Research Services Research Finance Management Accountant and Faculty of Science Finance Manager
Director of Human Resources

4. Notes of the relevant meetings and supporting documentation are accessible on the website.



## 11.2 Research Management Systems

11.2.1 Background

5. CRU is located within the School of Environmental Sciences (ENV) of UEA. Historically CRU had a high degree of autonomy, with separate reporting lines and distinct budgets. Apart from the Director, its staff were dependent on ―soft‖ money, that is on grants and other sources of funding mainly from sources such as UK Research Councils, EU programmes and the US Department of Energy. The position changed with the University‘s decision to invest strongly in environmental science. More recently up to 5 of the CRU staff have heldCHAPTER 11 : GOVERNANCE



### 99 established positions and the current full-time equivalent is 3.5.

6. The research policies and strategy of the University are set by a hierarchy of committees, with CRU being represented on the ENV Committee. The Director of CRU reports to the Head of ENV.



## 11.2.2 Funding Management

7. Responsibility for the University‘s funding management process rests with Research, Enterprise and Engagement (REE), through which major bids for funding are channelled. More routine funding bids are handled by Faculty Research Officers. The documentation supporting bids requires sign-off by the relevant principal investigator (PI) and Head of School, confirming their collective and individual responsibilities in delivering the research project in accorda nce with the funder‘s terms and conditions and the University‘s and School‘s policies and procedures.



## 11.2.3 Funders‟ R equirements

8. Funders‘ requirements vary considerably and have developed over time. All seek publication of results from the work they support. The traditional expectation has been that this will be in peer reviewed journals. There is now some pressure for open access publication, which also involves peer review. The emphasis on protecting intellectual property varies. Regular progress reports are generally required, often as a condition of drawing down funds; and reports with specific formats and deadlines may be required on completion. There are rarely detailed requirements for the release of data or code.



## 11.2.4 Good Research Practice

9. The university has a well developed set of policies on good research practice, research ethics, misconduct in research (including whistle blowing and grievance procedures), which apply to all those working in research, as well as regular performance reviews. The system is continuously developing; the good practice guidelines were first introduced in 2003 (we saw the 2009 version). Details of these structures, policies and procedures are referenced with the records of our interviews with the relevant senior managers, on our website.

10. Prospective students and staff are sent weblinks before arrival. There is training for undergraduates carrying out research as part of their coursework, and for postgraduate students and for new members of staff there is probationary training. It is the responsibility of Principal Investigators to ensure that their people are fully trained

11. The University‘s statement of terms and conditions of appointment of academic, teaching and research staff cross-refers to the requirements of a wide range of university policies. There is an induction process that includes coverage of what the requirements are and includes the allocation of a ―mentor‖, to act as an advisory colleague to a new member of staff. There is a standard annual training programme for both supervisors and new recruits. Compliance with theCHAPTER 11 : GOVERNANCE



### 100 requirements under the terms and conditions is the responsibility of Heads of School, with support from Human Resources as necessary.

12. Outputs such as publication and citations are monitored on a yearly basis, research plans are prepared at institutional level and have to be signed off each year, and individuals receive appraisal with feedback every 2 years. UEA attaches importance to peer review as a check on people‘s work and publications, and encourages academics to seek outside involvement.

13. Whistle blowing has never been used for any research related issue (it has been used a couple of times in all, in relation to other matters). The research misconduct procedure has been used on only a few occasions, mainly in relation to individuals‘ doctorate or master‘s theses.



## 11 .2.5 Financial Controls

14. Lead universities are normally responsible for administration of grants; payments are controlled by funders against a profile of expenditure and pass through central accounts only.

15. Where cash advances are required, there are specific, senior approval procedures.
I n exceptional circumstances advances could be made to an employee‘s bank

account.

16. It was reported by UEA Finance that audit procedures are robust, both in general and in relation to certain EU projects where these are mandatory. UK Research Councils are increasingly asking for specific audit checks.


## 11.2.6 Risk Management

17. The University maintains a risk register, regularly reviewed by Senior Management and annually by the Council and the Audit Committee. In relation to Research, the main concern has been the risk to reputation from slippage in league tables, with a range of mechanisms to drive performance in the desired direction. The register did not address the pressures on CRU as a result of the significance of its work for the climate change debate and the severity of the challenge to that work, and therefore did not consider the support CRU might need in handling its data and in making it available transparently. The upsurge of FoIA requests in 2009 was not brought to the attention of senior management soon enough; and, when it was, there was no prepared framework for addressing the implications. Action is now being taken to raise awareness of the implications of FoIA and EIR, with leadership from senior management.



## 11.2.7 Findings on Research Management Systems

18. The Review t eam‘s reading of the improperly rel eased e-mails together with Sherrington 1 

### suggested that cash advances to researchers in Russia might not have been adequately controlled, see also Chapter 4, paragraph 13. Whereas nearly all

re cords from the mid- 1990s have now been destroyed, in keeping with UEA‘s policy, we have confirmed that CRU did not have its own bank account and any

1 Sherrington submission (no. 107), 4 MarchCHAPTER 11 : GOVERNANCE



### 101 such payment would have required authorisation in writing by the Director of Finance at that time. While we have been unable to deal exhaustively with the issue, we are advised that there is a payment of $5000 in November 1998 properly recorded in the UEA ledger which is consistent with e-mails 911405082.txt and 914256033.txt at that time.

19. Apart from those issues addressed in Chapters 6 to 10 , UEA‘s Research Management Systems appear adequate. Requirements have developed significantly over the period relevant to the Review, from fairly general expectations that terms and conditions set by funders would be complied with, to the highly developed system now in operation.

20. To summarise, the focus of risk management in research was on quality and standards. The critical pressure on CRU, with its attendant risk to the University‘s reputation, was not on the radar screen. Senior management were not sufficiently aware of the issue, were not alerted in time to its emergence, and had no preparation in place to respond adequately.


## 11.3 Software, Data Management and Data Security

11.3.1 General Context

21. In submissions, concerns were expressed to the Review over the quality of software development, testing and operational processes within CRU, though it

wa s also asserted that much of such software is transient, and, though poorly structured and documented, may still be fit for the limited purposes envisaged. Submissions from Bratby 2 

### and Barnes 3 

### are relevant. We noted that a number of requests received by UEA under FoIA/EIR also sought access to code. Concerns have also been expressed about potential loss of data by CRU and the level of information security for CRU systems, which were vulnerable to unauthorised disclosure of the personal e-mails and other material subsequently published widely on the Internet.

22. IT Organisation . In common with other areas of the Science Faculty, CRU operates largely independently of the central IS functions of the UEA. Central IS has, in recent years, made significant efforts to better support the Science Faculty and some use of central facilities (such as the Storage Area Network) has been achieved. The University IS team does not provide desktop, remote access, hosting, database or software support to CRU, nor any quality control or assessment. CRU has its own local architecture based on a mix of individual PC based and server based processing. In common with many other research groups across the university, this is distinct from the UEA preferred model of client

– server operation. Internet communications for CRU is however routed over the university network and through the university firewall. CRU has its own IT Manager for whom CRU is 40% of his workload. CRU originally had no central backup arrangements for the individual researchers‘ PCs however its IT Manager introduced automated backup (using open source software) to a simple server held securely within the Central IS machine room.

2 Bratby submission (no. 3), 22 February

3 Barnes submission (no. 1), 6 MarchCHAPTER 11 : GOVERNANCE



### 102 23. Policy

. A high level  ̳Information Systems Policy‘ and a related  ̳Information Security Policy‘ 4 

### were agreed and put in place in 2005 under the aegis of UEA‘s Information Systems Strategy Committee (ISSC), which includes representatives of all four Faculties. Low level, detailed, security policies had been developed and put in place by 2007 5



### .

24. Standards

. The Review is aware of a number of industry standards relevant to this Chapter dealing with: software development, data stewardship and security. For further details see Appendix 8.



## 1 1.3.2 Issues and the Investigation

25. The Director of Information Services indicated that, whilst the central IT function were aware of the existence of the CRU Backup Server, they had no knowledge of the nature of the information held on the server as it was managed from CRU.

26. The CRU IT Manager indicated that researchers within CRU worked individually or in small groups. There was no master index of resources, be these data, algorithms or software. No systematic approach to the creation of metadata existed. There was no central database of underlying climate data; rather individual researchers assembled sets of data drawn from different primary sources outside CRU (for example the Met. Office Hadley Centre for Climate Change). These might arrive by network ( The United Kingdom‘s Educational and Research Network, or JANET), or on portable hard disks.

27. With the exception of a small amount of tree ring data, CRU does not generate new raw data. It relies instead on accessing existing primary data sources. Nonetheless the importance of maintaining clear records of what data has been accessed for what purpose

– for example in terms of station records processed

– is clear. The benefits of such a central data catalogue (or ―data dictionary‖) had long been recognised within CRU and past attempts had been made to create such a resource. These attempts had foundered on the lack of resources

– research grants made no provision for this and central UEA funding had not been available. Whilst there was no policy for the systematic archiving of data, many of CRU‘s processed results datasets are available on the Web 6



### .

28. Individual researchers were responsible for acquiring or developing their own software applications (usually written in Fortran or Interactive Data Language, IDL). There was no formal quality control policy or review policy. However individual projects would comply with whatever quality control processes were specified as part of their funding arrangements.

29. At interview, the ICT Systems Director indicated that ―lessons had been learnt‖ and he expected (subject to the results of a security audit report) to bring forward proposals within the University for:

4 These can be downloaded from: http://www.uea.ac.uk/is/itregs/ictpolicies

5 A draft ―Security Manual‖ (not available for public download) was received by the Review on 8th February.

6 See: http://www.cru.uea.ac.uk/cru/data/hrg/CHAPTER 11 : GOVERNANCE



### 103 Greater compliance with centrally defined IS policies and architecture; An audit of research data held in digital storage across the University; and Clear data retention (and destruction) policies.


## 11.3.3 Findings on Software, Data Management and Data Security

30. The software development process . We found that, in common with many other small units across a range of universities and disciplines, CRU saw software development as a necessary part of a researcher‘s role, but not resourced in any professional sense. Small pieces of software were written as required, with whatever level of skill the specific researcher happened to possess. No formal standards were in place for:

Software specification and implementation; Code reviews; and Software testing

31. Data management and archiving

. We found that there were no formal processes in place within CRU with respect to the systematic retention and archiving of data sets, or more particularly of metadata (data that allows the data set to be correctly interpreted). Individual researchers took whatever actions they deemed appropriate within the context of specific agreements with research funding bodies.

32. Information security. We found that the basic security processes had been appropriately specified and documented by the UEA‘s Information Systems Strategy Committee. We are constrained in our detailed findings by the fact that a police investigation into the unauthorised release of information is ongoing.



## 11.4 Recommendations

33. Risk management

. UEA should be alert to the implications for their reputation of the sort of challenges we have seen in this case to the work of CRU and any other key groups. The risk register should reflect the range of external attitudes towards its key units and growing criticism or the attentions of pressure groups should be noted. Mitigation measures should be put in place, including increased security and a bias for openness and properly resourced policy on data management and availability. Reporting arrangements should ensure that key senior management are in touch with the issues and are informed quickly of problems; and response plans should be in place and rehearsed. These points are no doubt relevant to many other universities.

34. Training for researchers. We believe that Universities should develop formal approaches to the training of researchers in basic software development methodologies and best practice, as well as best practice in the handling and sharing of research data.

35. Provision of a formal metadata repository. Whilst we recognize and accept that CRU relies on other bodies both nationally and internationally to provide and toCHAPTER 11 : GOVERNANCE



### 104 archive basic weather station data, we believe that a formal approach to the storage and archiving of metadata is required. Such a repository would, for example, have made it far easier to respond quickly to requests for the list of station identifiers associated with particular CRUTEM datasets. Where a University is hosting a unit of such international significance, we believe that it should ensure funding is available for such a repository either through the research grant process or from central resources.

36. Role of research sponsors. We note the recent statement by the US National Science Foundation (NSF) 7 

### that, from October 2010, NSF plan to make inclusion of a ―Data Management Plan‖ a requirement for all research proposals. It will be important for such plans to recognize that in some areas of science huge volumes of data are created and a degree of processing and compression is inevitable before data suitable for storage is created. We agree that the way in which important research data (and the associated meta data to make that data useful) should be preserved, should be specified by those funding such research. Explicit budgetary and resource provision must be made. Sponsorship arrangements should include a clear statement of requirements on the extent to which such data should be placed in the public domain and any constraints on the timing of such release. The guidance from the UK Research Integrity Office (UKRIO) is helpful in this respect. 8



### 37. Making source code publicly available. We believe that, at the point of publication, enough information should be available to reconstruct the process of analysis. This may be a full description of algorithms and/or software programs where appropriate. We note the action of NASA‘s Goddard Institute for Space Science in making the source code used to generate the GISTEMP gridded dataset publically available. We also note the recommendation of the US National Academy of Sciences in its report ―Ensuring the Integrity, Accessibility and Stewardship of Research Data in the Digital Age‖ that: “...the default assumption should be that research data, methods (including the techniques, procedures and tools that have been used to collect, generate or analyze data, such as models, computer code and input data) and other information integral to a publically reported result will be publically accessible when results are reported ...‖. We commend this approach to CRU.

38. Audit processes. It is entirely acceptable that the central functions of a University should set, document and disseminate the standards expected across all governance areas, but without necessarily mandating the precise means by which these will be achieved. These standards will reflect the University‘s interpretation of applicable law (Data Protection, Computer Misuse, Health & Safety, Environmental Information Regulations) and best practice. In areas such as Information Systems, it may well be appropriate to allow a degree of local autonomy. However it is then essential that robust audit procedures are in place to ensure that, where local solutions are implemented, these do meet fully the standards specified.

7 NSF Press Release 10- 077 http://www.nsf.gov/news/news_summ.jsp?cntn_id=116928&org=NSF&from=news

8 See UKRIO  ̳Code of Practice for Research‘ Section 3.12 ―Collection and Retention of Data‖ at http://www.ukrio.orgCHAPTER 11 : GOVERNANCE



### 105

### 106



## ACKNOWLEDGEMENTS

The Review Team would like to acknowledge the help and support they received during the conduct of the Review and in the preparation of this report. Our thanks go

to:

William Hardie who provided Secretariat support.

David Walker who acted as our Project Manager.

Kate Moffat, Adrian Dias

, Helen Davison and Mike Granatt who provided communications support and advice, set up and handled the Review website, and managed the production of the report.

### 107APPENDIX 1



### 108 

## APPENDIX 1: REVIEW TEAM MEMBERS

TEAM CV‘s


## Sir Muir Russell, KCB, FRSE (Chair) Sir Muir Russell was Principal and Vice-Chancellor of the University of Glasgow from 2003 to 2009. During that period he was Convener of Universities Scotland, a member of the Universities UK Main Board, a Trustee of the Universities Superannuation Scheme, and a member of the UCAS Board. He graduated from the University of Glasgow in 1970 with a First in Natural Philosophy and took up a career in the civil service. He was appointed Permanent Secretary to the Scottish Office in 1998, and was the first Permanent Secretary to the Scottish Executive following the establishment of the Scottish Parliament in 1999. He has honorary degrees from the Universities of Strathclyde, Glasgow and Edinburgh. He currently chairs the Judicial Appointments Board for Scotland. He is also a Trustee of the Glasgow School of Art, a Member of the Board of the Moredun Research Institute, and the Chairman of the Dunedin Concert Trust.


## Professor Geoffrey Boulton, OBE, FRS, FRSE Professor Geoffrey Boulton is Regius Professor Emeritus of Geology and former Vice Principal of the University of Edinburgh. His research is in the fields of glaciology, glacial geology and Quaternary science, and has been awarded several international awards and honorary degrees for his scientific work. He currently has research projects in Antarctica and Iceland. He has been the UK representative to the International Union of Geosciences and to the International Union of Quaternary Sciences. He is a member of the UK Prime Minister‘s Council for Science and Technology, chairs the Advisory Board of the University of Heidelberg and is General Secretary of the Royal Society of Edinburgh, Scotland‘s national academy. He has been a member of the Councils of the Natural Environment Research Council and the Royal Society, a member of the Royal Commission on Environmental Pollution, the Scottish Science Advisory Committee and the Scottish Higher Education Funding Council. He was formerly Head of the Department of Geology and Geophysics and Provost of Science and Engineering in the University of Edinburgh.



## Professor Peter Clarke, FInstP, CPhys, FIET, CEng Peter Clarke is Professor of Physics at the University of Edinburgh. He has a B.Sc in Electronics Engineering (Southampton University, 1980) and a D.Phil in Particle Physics (Oxford 1985). He was a CERN Fellow before being appointed as a lecturer first at Brunel University in 1987 and then University College London in 1993. He was promoted to Reader and then Professor in 2001 and was Head of the Particle Physics Research Group between 2001-04. He moved to the University of Edinburgh in 2004 to take up the Chair of eScience and later become Director of the National eScience Centre 2006-09.


## David Eyton MA, MIoM 3 

## , CEng. David Eyton is Group Head of Research & Technology at BP, and was appointed in April 2008. He is accountable for technology strategy and its implementation across BP and conducting research and development (R&D) in areas of corporate renewal.APPENDIX 1



### 109 In this role, David oversees the R&D capability of the company and also sits on the UK Energy Technologies Institute and Science¦Business Innovation Boards. During his career he has held a number of Petroleum Engineering, Commercial and Business Management positions in the UK, Australia, Trinidad and USA.



## Professor James Norton, CDir, CEng, CITP, FIoD, FIET, FBCS, FRSA Aged fifty-seven, Jim Norton is an independent director and policy adviser. He is an external member of the Board of the UK Parliament's Office of Science & Technology (POST) and council member of the Parliamentary IT Committee (PITCOM). Jim is a Non-Executive Director of F&C Capital & Income Investment Trust plc, where he chairs the Audit & Management Engagement Committee. He is a Board Member and Trustee of the Foundation for Information Policy Research (FIPR), as well as a member of the 'Electronic Communications Expert Advisory Panel' for the Irish Commission for Communications Regulation (ComReg). Jim is a Vice-President and Trustee of the BCS

– Chartered Institute for IT, an External Examiner for the Institute of Directors, and Chair of the IT Policy Panel for the Institution of Engineering & Technology. He also chairs the Steering Group for the Secure Software Development Partnership (SSDP) of the Technology Strategy Board.APPENDIX 2



### 110 

## APPENDIX 2 : APPROACH AND WORK PLAN

1. This document sets out how the Review Team has approached its terms of reference, and the way in which it will fulfil its remit.

2. The terms of reference are:

―The Independent Review will investigate the key allegations that arose from a series of hacked e- mails from the University of East Anglia‘s Climatic Research Unit (CRU). The review will:

2.1. Examine the hacked e-mail exchanges, other relevant e-mail exchanges and any other information held at CRU to determine whether there is any evidence of the manipulation or suppression of data which is at odds with acceptable scientific practice and may therefore call into question any of the research outcomes.

2.2. Review CRU‘s policies and practices for acquiring, assembling, subjecting to peer review and disseminating data and research findings, and their compliance or otherwise with best scientific practice.

2.3. Review CRU‘s compliance or otherwise with the University‘s policies and practices regarding requests under the Freedom of Information Act ( ̳the
FoIA ‘) and the Environmental Information Regulations ( ̳the EIR‘) for the
release of data.

2.4. Review and make recommendations as to the appropriate management, governance and security structures for CRU and the security, integrity and release of the data it holds.‖

3. The remit requires the Review to address the specific allegations about the way in which CRU has handled its data, reflecting comments in the e-mail exchanges that have been made public. In a separate paper

– Issues for examination - the Team has set out its initial view of the questions that need to be addressed. It will seek written submissions from CRU and other appropriate parts of UEA. It will also invite interested parties to comment on what the Issues paper covers, and to propose any further matters that clearly fall within the Remit and should also be examined.

4. The Review‘s remit does not invite it to re-appraise the scientific work of CRU. That re-appraisal is being separately commissioned by UEA, with the assistance of the Royal Society. The Review‘s conclusions will complemen t that re-appraisal by pointing to any steps that need to be taken in relation to data, its availability and its handling.

5. The Team wishes to focus on the honesty, rigour and openness with which CRU handled its data. It wishes to gain a proper understanding of: The range of data involved, and how it has been indexed and archived.APPENDIX 2



### 111 The procedures, processes and relevant protocols used to handle the data, recognizing that these may have changed over time as data-handling capacity has developed.
The associated metadata, algorithms and codes used for analysis. The extent to which other independent analysis produces similar conclusions. The peer review process, examining how much was in common between the work of the reviewers and the reviewed.

6. In making its analysis and conclusions, the Team will test the relevant work against pertinent standards at the time it was done, recognizing that such standards will have changed. It will also test them against current best practice, particularly statements of the ethics and norms such as those produced by the UK Government Office for Science and by the US National Academy of Sciences. These identify principles relating to rigour, respect and responsibility in scientific ethics and to integrity, accessibility and stewardship in relation to research data. This overall approach will allow the Team to establish a conceptual framework within which it can make judgements and comment about key issues such as the level of uncertainty inherent in all science, and the particular confidence limits associated with the CRU work.

7. The police and the Information Commissioner are also considering issues in connection with the leaked e-mails, and the Team has established appropriate, continuing liaison with them.

8. The Team‘s an alysis and conclusions will include not only a view of what has happened in the past, but also comments and recommendations on best practice for the future. This will be done both at the level of CRU and the University as a whole, and may have broader implications for institutions undertaking scientific work.

9. The Team will operate as openly and transparently as possible. It is establishing a

we bsite which will eventually display all of the submissions received, correspondence, analyses and conclusions. The aim will be to publish all received subm issions quickly, unless there are wholly exceptional reasons to delay, for example legal issues.APPENDIX 3



### 112 

## APPENDIX 3

The questions, criticisms and allegations raised in the „Issues for Examination‟ paper below are not intended to be read in isolation. Readers are referred to the Review‟s findings on these points as set out in the body of the report.



## ISSUES FOR EXAMINATION

1. The Review Team’s remit is set out in its terms of reference as follows:

―The Independent Review will investigate the key allegations that arose from a series of hacked e- mails from the University of East Anglia‘s Climatic Research Unit (CRU). The review will:

Examine the hacked e-mail exchanges, other relevant e-mail exchanges and any other information held at CRU to determine whether there is any evidence of the manipulation or suppression of data which is at odds with acceptable scientific practice and may therefore call into question any of the research outcomes.

Review CRU‘s policies and practices for acquiring, assembling, subjecting to peer review and disseminating data and research findings, and their compliance or otherwise with best scientific practice.

Review CRU‘s compliance or otherwise with the University‘s policies and practices regarding requests under the Freedom of Information Act ( ̳the FoIA‘) and the Environmental Information Regulations ( ̳the EIR‘) for the release of data.

Review and make recommendations as to the appropriate management, governance and security structures for CRU and the security, integrity and release of the data it holds.‖

1. The remit reflects the reaction to the e-mails that became public, much of it questioning and critical. In following up those questions and criticisms the Team stresses that it has formed no view on whether they are fair or justified. In formulating in its own words an expression of the issues for examination the Team is not adopting those issues as its own criticisms.

2. The Team‘s approach is to distill the questions and criticisms into the broad questions set out below. Using its own enquiries and experience, it has added questions about the handling and dissemination of data, including the response to FOI requests. The issues addressed in the first three paragraphs of the terms of reference will inform recommendations for paragraph four, as to the appropriate management, governance and security structures for CRU and the release of data.

3. The Team w ill invite CRU and other parts of UEA to respond in writing to these questions, and will follow up those responses as required. The Team expect the CRU to provide original documentary evidence to support its responses.

4. The Team invites those with an interest in the matter to comment on this Issues paper. The Team stresses that its remit does not involve re-evaluation of theAPPENDIX 3



### 113 scientific conclusions of the CRU work, still less a reappraisal of the scientific debate about the existence and suggested causes of global warming. Please confine any comments on this paper to matters within the remit at paragraph 1 of this paper.

5. Written responses from CRU and others are requested by 1 March 2010 to the address below:

Email: correspondence@cce-review.org

Or post to,

Climate Change E-Mails Review Box 18 196 Rose Street Edinburgh
EH2 4AT



## ISSUES ARISING ON Para 1.1 OF THE TERMS OF REFERENCE

6. The allegation of ignoring potential problems in deducing palaeotemperatures from tree ring data that might undermine the validity of the so- called “hockey

- stick” curve.

7. In the late 20 th 

### century, the correlation between the tree ring record and instrumental record of temperature change diverges from that for the earlier period. The cause of this divergence does not appear to be understood. If the method used to deduce temperatures from tree ring proxy metrics for the earlier tree ring record is applied to the late 20 th 

### century tree ring series, then declining temperatures would be deduced for the late 20 th 

### century. It is alleged that if the cause of divergence between the tree ring and instrumental temperature record is unknown, it may have existed in earlier periods. Therefore if tree rings had similarly failed to reflect the warming of the early Middle Ages, they may significantly under-estimate the warming during the Medieval Warm Period, thus falsely enhancing the contrast between the recent warming and that earlier period. (It is this contrast that has led to statements that the late 20 th 

### century warming is unprecedented during at least the last 1000 years.)

QUESTIONS TO ADDRESS:

What method do you use to deduce palaeotemperatures from tree ring data? Does not the problem of divergence for the late 20 th 

### century record invalidate the deduction of tree ring palaeotemperatures for the period prior to the instrumental record? How open have you been about this issue? What attempts have you made to resolve it?APPENDIX 3



### 114 What is the evidence that the amplitude of warming during the Medieval Warm Period (MWP) is not underestimated by tree ring evidence? How does the tree ring evidence of the MWP compare with other proxy data? Have you showed how data from different sources compare or have you conflated them? If the latter, what is the justification? If tree ring proxies are removed from reconstructions, what evidence remains of the MWP? Have you been selective in utilizing tree ring evidence from Yamal in Siberia; and if so, what is the justification for selectivity and does the selection influence the deduced pattern of hemispheric climate change during the last millennium?

8. The allegation that CRU has colluded in attempting to diminish the significance of data that might appear to conflict with the 20 th 

### century global warming hypothesis

9. The CRU group, in consultation with Professor Michael Mann, is alleged to have systematically attempted to diminish the significance of the Medieval Warm Period, evidenced by an email from Mann 4 th 

### June 2003: ―I think that trying to adopt a timeframe of 2K, rather than the usual 1K, addresses a good earlier point that Peck made w/ regard to the memo, that it would be nice to try to "contain" the putative "MWP", even if we don't yet have a hemispheric mean reconstruction available that far back [Phil and I have one in review

-- not sure it is kosher to show that yet though

-- I've put in an inquiry to Judy Jacobs at AGU abou t this].‖ The use of the words ―contain‖ and ―putative‖ are alleged to imply an improper intention to diminish the magnitude and significance of the MWP so as to emphasise the late 20 th 

### century warming.

QUESTIONS TO ADDRESS

What does the word “contain” m ean in this context? What involvement have you had in “containing” the MWP? How important is the assertion of “unprecedented late 20 th 

### century warming” in the argument for anthropogenic forcing of climate?

10. It is alleged that proxy temperature deductions and instrumental temperature data have been improperly combined to conceal mismatch between the two data series

11. An attempt to hide the difficulty of combining these two data series and to mislead is alleged to be revealed in the following sentence in a November 1999 email from Professor Phillip Jones which is alleged to imply a conscious attempt to mislead: "I've just completed Mike's Nature trick of adding in the real temps to each series for the last 20 years (i.e. from 1981 onwards) and from 1961 for Keith's to hide the decline‖.

QUESTIONS TO ADDRESS

What is the meaning of the quotation from the 1999 email?APPENDIX 3



### 115 How do you justify combining proxy and instrumental data in a single plotted line? What method do you use?

12. It is alleged that there has been an improper bias in selecting and adjusting data so as to favour the anthropogenic global warming hypothesis and details of sites and the data adjustments have not been made adequately available

13. It is alleged that instrumental data has been selected preferentially to include data from warmer, urban in contrast to rural sites; that the rationale for the choice of high/low latitude sites is poor; and that the processes by which data has been corrected, accepted and rejected are complex and unclear.

QUE STIONS TO ADDRESS

What is the rationale for the choice of data stations worldwide? How has this choice been tested as appropriate in generating a global or hemispheric mean temperature (both instrumental and proxy data)? Describe as clearly as possible the protocols you have followed in selecting, correcting and rejecting data and stations. Has this been an orderly and objective process applied to all datasets? To what extent have different procedures for data of different vintages and different sources been unified? What means do you use to test the coherence of the datasets?



## ISSUES ARISING ON Para 1.2 OF THE TERMS OF REFERENCE

14. It is alleged that there have been improper attempts to influence the peer review system and a violation of IPCC procedures in attempting to prevent the publication of opposing ideas.

15. It is alleged that there has been an attempt to subvert the peer review process and exclude publication of scientific articles that do not support the Jones-Mann position on global climate change. A paper by Soon & Balunias was published in the Journal Climate Research arguing that the 20 th 

### century was abnormally warm. An email from Professor Michael Mann on 11 th 

### March 2003 contained the following: "I think we have to stop considering Climate Research as a legitimate peer-reviewed journal. Perhaps we should encourage our colleagues in the climate research community to no longer submit to, or cite papers in, this journal." The allegation is that journals might be pressured to reject submitted articles that do not support a particular view of climate change.

16. In an email to a fellow researcher in June 2003, Briffa wrote: ―Confidentially I now need a hard and if required extensive case for rejecting (an unnamed paper) to support Dave Stahle‘s and really as soon as you can.‖

17. In an email to Mann on 8 th 

### July 2004, Jones wrote: "The other paper by MM is just garbage. [...] I can't see either of these papers being in the next IPCC report.APPENDIX 3



### 116 Kevin and I will keep them out somehow

— even if we have to redefine what the peer-review literature is!" The allegation is of an attempt to prevent ideas being published and the author being prepared to subvert the peer review process for a journal and to undermine the IPCC principle of accounting properly for

con tradictory views.



## QUESTIONS TO ADDRESS

Give full accounts of the issue in relation to the journal Climate Research, the June 2003 email, and the March 2004 email to Mann (“recently rejected two papers (one for Journal of Geophysical Research & one for Geophysical Research Letters) from people saying CRU has it wrong over Siberia. Went to town over both reviews, hopefully successfully. If either appears I will be very surprised”. Are the first two instances evidence of attempts to subvert the peer review process? In relation to the third, where do you draw the line between rejecting a paper on grounds of bad science etc, and attempting to suppress contrary
views? To what extent is your attitude to reviewing conditioned by the extent that a paper will set back the case for anthropogenic global warming and the political action that may be needed to mitigate it? What is the justification for an apparent attempt to exclude contrary views from the IPCC process?

18. The scrutiny and re-analysis of data by other scientists is a vital process if hypotheses are to rigorously tested and improved. It is alleged that there has been a failure to make important data available or the procedures used to adjust and analyse that data, thereby subverting a crucial scientific process.

19. It is alleged that there has been a systematic policy of denying access to data that has been used in publications, referring to an email from Jones to Mann on 2 nd 

### February 2005 which contains the following: "And don't leave stuff lying around on ftp sites - you never know who is trawling them. The two MMs have been after the CRU station data for years. If they ever hear there is a Freedom of Information Act now in the UK, I think I'll delete the fi le rather than send to anyone. Does your similar act in the US force you to respond to enquiries within 20 days?

— our does! The UK works on precedents, so the first request will test it. We also have a data protection act, which I will hide behind‖.



## QUESTIONS TO ADDRESS

Do you agree that releasing data for others to use and to test hypotheses is an important principle? If so, do you agree that this principle has been abused? If so, should not data be released for use by those with the intention to undermine your case, or is there a distinction you would wish to make between legitimate and illegitimate use?APPENDIX 3



### 117 If not, do others have reasonable access to the data at all levels and to the description of processing steps, in order to be able to carry out such a re

analysis? Can you describe clearly the data-sets and relevant meta-data that have been released; what has not been released and to what extent is it in useable form? Where has it been released? Where access is limited, or not possible, or not meaningful, for legitimate reasons please explain why?

20. The keeping of accurate records of datasets, algorithms and software used in the analysis of climate data.

21. A key concern expressed by a number of correspondents and commentators has been as to whether datasets, and analyses based thereon, were deleted.



## QUESTIONS TO ADDRESS

Were formal „data dictionaries‟ kept of the data sets acquired by the CRU at various times from other bodies such as the UK Meteorological Office Hadley Centre and its equivalents around the World? Were comprehensive records kept of the way these various data sets were used, the statistical and other algorithms used in processing them, and the various software programmes and modules used to carry out that processing? Does a formal library of algorithms and software used by the CRU exist? What quality control measures were used to test the various algorithms and software modules developed by the CRU? What techniques did members of the CRU employ to ensure the integrity of the various applications used to process climate data? What policies are in place to ensure the formal archiving of data sets and resultant analyses for future use and review.



## ISSUES ARISING ON Para 1.3 OF THE TERMS OF REFERENCE

22. Response to Freedom of Information requests.

23. A number correspondents and commentators assert that requests under the Freedom of Information Act (FoIA) and the Environmental Information Regulations (EIR) were incorrectly denied by the University of East Anglia on advice from the CRU. This is the subject of a separate inquiry by the Information C ommissioner, but does fall within the terms of reference of the Review Team.



## QUESTIONS TO ADDRESS

What formal processes were in place both centrally and within the CRU to ensure fair and impartial assessment of FoIA requests?APPENDIX 3



### 118 Were there any processes in place centrally to review recommendations from the CRU that information should not be released? Over the five years to November 2009:

o how many requests were received?

o how many were rejected, and on what grounds?

o how many received full release of information?

o how many received partial release of information?

Independent Climate Change E-Mails Review, February 2010APPENDIX 3



### 119APPENDIX 4



### 120 

## APPENDIX 4: INDEX OF MEETINGS, INTERVIEWS, SUBMISSIONS, FOLLOW UP ENQUIRIES AND RESPONSES

Team Meetings were held on the following dates in 2010. Confirmed notes of Team Meetings can be viewed at: http://www.cce-review.org/Meetings.php

12 January 4 February 25 February 20 March 1 April 13 April 22 April 28 April 11 May 26 May
7 June 11 June 15 June

Other Meetings and Interviews conducted. Notes of the meetings and supporting documentation can be accessed from: http://www.cce-review.org/Evidence.php

18 December 2009, meetings conducted by Sir Muir Russell at the UEA. Notes were taken by Lisa Williams (Senior Assistant Registrar), UEA: Brian Summers (Registrar and Secretary), Professor Trevor Davies (PVC Research, Enterprise and Engagement), Professor David Richardson (Dean, Faculty of Science), UEA Stuart Holmes, (Chair, UEA Council), UEA Jonathan Colam-French (Director of Information Services), Iain Reeman (ICT Systems Director), Steve Mosley (ICT Policy Manager), UEA Brian Summers (Registrar and Secretary), David Palmer (Information Policy & Compliance Manager) and Jonathan Colam-French (Director of Information Services), UEA Alan Preece (Director of Marketing and Communications) and Annie Ogden (Head of Communications), UEA Professor Trevor Davies (PVC Research, Enterprise and Engagement) and

P rofessor Philip Jones (Climatic Research Unit), UEA Professor Trevor Davies (PVC Research, Enterprise and Engagement) and Professor Keith Briffa (Climatic Research Unit), UEA Superintendent Julian Gregory, Norfolk Constabulary (meeting note withheld to avoid pr ejudicing ongoing police investigation).APPENDIX 4



### 121 27 January 2010, meetings conducted by Sir Muir Russell and Professor Jim Norton at the UEA:

Mike Gorrill (Head of Enforcement) and David Clancy (Investigations Manager) of the Information Commissioner‘s Officer Jonathan Colam-French (Director of Information Services) and Mike Salmon (IT Manager to the CRU - 40% time), UEA Professor Philip Jones and Professor Keith Briffa, Climatic Research Unit, UEA Superintendent Julian Gregory and Andy Guy from Norfolk Constabulary (meeting note withheld to avoid prejudicing ongoing police investigation)

4 March 2010, interviews conducted by Professor Peter Clarke and Professor Jim Norton at the UEA:

Professor Philip Jones, Dr Tim Osborn and Ian Harris, Climatic Research Unit, UEA

24 March 2010, meeting conducted by Professor Jim Norton at the Information Commissioner‘s Office:

Mike Gorrill (Head of Enforcement) and Steve Wood (Assistant Commissioner FOI)

26 March 2010, meetings conducted by Sir Muir Russell and David Eyton at the UEA: Ian McCormick (Director of Research, Enterprise and Engagement) and Alan Walker (Manager of Research Services), UEA Rob Bell (Research Finance Management Accountant) and Laura McGonagle (Faculty of Science Finance Manager), UEA Professor Trevor Davies (PVC Research, Enterprise and Engagement), UEA Professor Trevor Davies (PVC Research, Enterprise and Engagement) and Professor David Russell (Associate Dean for Research, Faculty of Science),

UEA Cecile Piper (Director of Human Resources), UEA Brian Summers (Registrar and Secretary), UEA Professor Edward Acton (Vice-Chancellor), UEA

30 March 2010, meetings conducted by Sir Muir Russell and Professor Jim Norton at th e UEA: Michael McGarvie (Science Faculty FoIA contact), UEA David Palmer (Information Policy & Compliance Manager) and Jonathan Colam-French (Director of Information Services), UEA

9 April 2010, meetings conducted by Professor Geoffrey Boulton and Professor Peter Clarke at the UEA: Professor Keith Briffa, Professor Philip Jones, Dr Tim Osborn and Dr Tom Melvin (Climatic Research Unit), David Palmer (Information Policy &APPENDIX 4



### 122 Compliance Manager) and Jonathan Colam-French (Director of Information Services), UEA

15 June 2010, telephone interview conducted by Sir Muir Russell and David Eyton with Professor Trevor Davies, PVC Research, Enterprise and Engagement

Telephone interviews conducted by Professor Geoffrey Boulton: 1 May, Professor Sir Brian Hoskins, IPCC AR4 Review Editor Ch 3 (Surface and Atmospheric Climate Changes) 1 June, Professor John Mitchell, IPCC AR4 Review Editor Ch 6 (Palaeoclimate).

Follow- up enquiries, responses and related correspondence. This material can be accessed from: http://www.cce-review.org/Evidence.php

Presentations from 20 March Review Team Meeting Independent forensic analyst report and commentary on e-mail examination Redacted set of FOI and EIR requests received relating to the Climatic Research Unit since 2005 Correspondence with Lord Oxburgh in respect of the separate reviews Follow-up request to Professor Jones and response on issues relating to the IPCC Clarification request to Professor Jones and response on issues relating to the IPCC Follow-up request to Professor Jones and response in relation to raw instrument station availability for each CRUTEM data set Follow-up request to Professor Briffa and response in relation to best scientific practice in dissemination scientific evidence into the public domain Follow-up request and response on the scope of material that was backed up on the server that was compromised
Follow- up request and response in relation to the UEA‘s main university e

mail server domain Follow-up request and response in relation to governance and risk management issues Follow-up request and response in relation to financial controls Follow-up request to ICO and response in relation to the holding of information or data Response to the Science and Technology Committee‘s Eight Report on The

Disclos ure of Climate Data from the Climatic Research Unit ( Note: at the time of writing, the Science and Technology Commons Select Committee has not give n permission to publish this response)

Submissions The Review was launched on 11 February and submissions were invited from this date and accepted up to 16 April. The Review accepted a small number of submissions that it deemed to have significant relevance to its remit after this close date.APPENDIX 4



### 123Submissions to the Review can be viewed at: http://www.ccere view.org/Evidence.php

Submissions Published

Submission Name Date received Numbe r 1 Nicholas Barnes 06

- Mar
2 David Archer 25

- Feb
3 Dr Phillip Bratby 22

- Feb
4 Philip Brohan 24

- Feb
5 Climatic Research Unit 01

- Mar
6 David Cockroft 28

- Feb
7 Gill Chant 13

- Feb
8 Tor Berge S. Gjersvik 12

- Feb
9 Professor Gabriele Hegerl 02

- Mar
10 Michael Hughes 17

- Feb
11 Professor Mike Hulme 26

- Feb
12 Professor Dr Fortunat Joos 01

- Mar
13 Dr D.R. Keiller 14

- Mar
14 Andrew MacIntyre 12

- Feb
15 Dr Ross McKitrick 26

- Feb
16 Dr P.C. Matthews 01

- Mar
17 Forrest M.Mims III 18

- Feb
18 Andrew Montford 11

- Feb
19 Professor Raymond Bradley 01

- Mar
20 Jim Stathos 26

- Feb
21 Professor Simon Tett 01

- Mar
22 The Global Warming Policy Foundation 28

- Feb
23 Stephen McIntyre 02

- Mar
24 Professor Rob Wilby 26

- Feb
25 Dr Austin Woods 11

- Feb
26 Professor Malcolm Hughes 01

- Mar
27 W.F. Lenihan 15

- Feb
28 Richard Calhoun 14

- Feb
29 Dr Henry Barnard 27

- Feb
30 Ron Cram 15

- Feb
31 Charlie Kilgore 13

- Feb
32 Vincent Moran 14

- Feb
33 Dr Richard North 11

- Feb
34 Stephen Richards 15

- Feb
35 Cameron Rose 16

- Feb
36 Dr C.W. Schoneveld 15

- Feb
37 David Shepherd 13

- Feb
38 Geoffrey Sherrington 17

- Feb
39 Martin Vermeer 27

- Feb
40 Robert Wright 15

- FebAPPENDIX 4



### 124 41 Brent Hargreaves 16

- Feb
42 Professor Michael Mann 28

- Feb
43 Dr Sonja Boehmer

- Christiansen 23

- Feb
44 Dr Benjamin Santer 28

- Feb
45 Trevor Jones 28

- Feb
46 D R G Andrews 08

- Apr
47 Professor Joshua Halpern 08

- Apr
48 Mike Haseler 22

- Feb
49 John R. Smith 17

- Feb
50 Dr D.R. Keiller 17

- Feb
51 Dr D.R. Keiller 17

- Feb
52 Dr David Lehmiller 15

- Feb
53 Dr D.R. Keiller 24

- Mar
54 Bob Smith 19

- Feb
55 Robert Wright 19

- Feb
56 Ian MacDonald 28

- Feb
57 Dr David Lehmiller 15

- Feb
58 Stephen Richards 11

- Feb
59 Ray Soper 17

- Feb
60 David Sandeman 18

- Feb
61 Dr Josep Verges 15

- Feb
62 Dominic O'Kane 15

- Feb
63 Vid Stimac 17

- Feb
64 Chris Allen 12

- Feb
65 Ian MacDonald 20

- Feb
66 Tony Brown 13

- Feb
67 Gerry Morrow 12

- Feb
68 Stephen Graves 12

- Feb
69 Will Hawkes 12

- Feb
70 C. Barling 12

- Feb
71 Susan Ewens 13

- Feb
72 Malcolm McClure 13

- Feb
73 Robert Owen 13

- Feb
74 Bruce Garrick 13

- Feb
75 Stephen Johnson 13

- Feb
76 Jim Stathos 13

- Feb
77 Dr Barry Napier 14

- Feb
78 Joe Olson 16

- Feb
79 Roderick Campbell 16

- Feb
80 John R. Smith 16

- Feb
81 Ray Soper 18

- Feb
82 Julie Grace 17

- Feb
83 Doug Vickers 15

- Feb
84 David Shepherd 15

- Feb
87 Mike Haseler 12

- Feb
88 Dr D.R. Keiller 06

- MarAPPENDIX 4



### 125 89 Patrick Keane 17

- Feb
90 Patrick Keane 18

- Feb
91 Dr D.R. Keiller 20

- Feb
92 Richard Calhoun 11

- Feb
93 Robert Denton 18

- Feb
94 Robert Denton 27

- Feb
95 John Graham

- Cumming 17

- Feb
96 D.G. McBeath 22

- Feb
97 George Mihailides 28

- Feb
98 David Shepherd 11

- Feb
99 Patrick Keane 15

- Feb
100 Dr D.R. Keiller 23

- Feb
101 Dr D.R. Keiller 26

- Feb
102 Dr D.R. Keiller 28

- Feb
103 Climate Scientists Letter 28

- May
104 McIntyre Submission 9

- Jun
105 Dr McKitrick supplementary sub 1 13

- Apr
106 Dr McKitrick supplementary sub 2 13

- Apr
107 Geoffrey Sherrington 4

- Mar
108 Professor Tom Wigley 28

- Feb
109 Patrick Keane 12

- Feb

Submissions Not Published
David Holland Conor McMenemie (submission to be treated as confidential)

The Review reserved the right to withhold publication of submissions if they were abusive, potentially defamatory, anonymous, or there were other legal difficultie s which prevented publication.APPENDIX 5



### 126 

## APPENDIX 5 : PEER REVIEW

UNDERSTANDING UNCERTAINTY: A BRIEF HISTORY OF PEER REVIEW By Richard Horton, Editor of The Lancet

Amid the public and scientific furore over alleged events at the University of East Anglia‘s Climatic Research Unit (CRU), peer review has emerged as a central issue in the dispute. In the Times Higher Education , for example, Andrew Montford, author of The Hockey Stick Illusion: Climategate and the Corruption of Science (1), argued that events at the CRU had far-reaching implications for the world of scientific publishing (2). His charge sheet was extensive

– undermining the peer-review process, threatening editors who published work contrary to orthodox scientific opinion, organising mass resignations from editorial boards, and persuading colleagues to stop submitting papers to allegedly offending journals. Montford suggests that ―as many as

four different journals may have had their normal procedures interfered with‖. He continues,

“What is an ethical way to deal with a journal?... At what point does valid protest elide into something more sinister?...If, as the [CRU] emails suggest, some

scientists are in fact putting illegitimate pressure on journ als, either to influence the peer-review process or to prevent the release of data, it is easy to see how editors may find it difficult to respond.”

Implicit in Montford‘s argument is that peer review is critical to the process of – and
ther eby public trust in

– science. Writing in The Guardian , George Monbiot put it this way: ―science happens to be [a] closed world with one of the most effective form s of
self- regulation: the peer review process.‖(3).

The importance of peer review has been invoked by climate sceptics in other domains of the climate debate. Christopher Booker has challenged Dr Rajendra Pachauri, for example, for claiming that his Intergovernmental Panel on Climate Change (IPCC) included only peer-reviewed research (4). By contrast, Booker reports that a third of IPCC sources were newspaper articles, student dissertations, even press releases. Again, the suggestion is that the peer-reviewed literature is something special and

sa cred to science. One can make strong and reliable assertions if those statements are underpinned by peer-reviewed science. If evidence has not been peer-reviewed, it is
next to worthless.

In the context of the CRU, there have been claims that peer reviewers censoredAPPENDIX 5



### 127 evidence deliberately to exclude those findings from scientific journals. Questions have been raised about the line between rejecting a paper on grounds of bad science and rejecting it as part of an attempt to suppress contrary opinions. Peer reviewers have been accused of letting their politics get in the way of science. Did reviewers‘ concerns about the potential adverse policy and political implications of supporting perfectly good science at peer review override their scientific judgment? The suggestion is that they did.

The House of Commons Science and Technology Committee report on the CRU (5) has emphasised that ―the question of the operation of peer review is going to be a critical issue‖ in any evaluation of the CRU‘s work. In their report, in a section headed ―Perverting the peer review process‖, MPs set out several specific allegations against CRU scientists. First, that they colluded to subvert peer review, slowing or blocking the publication of research which disagreed with their own views. Second, that climate scientists planned to ―redefine what the peer

- review literature is‖ in their recommendations to the IPCC (6). Third, that they tried to suppress a paper on research fraud. And finally, that CRU staff exerted improper pressure on the editor of one journal, an editor who had been open to scientists challenging climate change orthodoxy. The House of Commons Committee concluded that there was no evidence to suggest that the CRU had tried to subvert the peer review process. They wrote that, ―Academics should not be criticised for making informal comments on academic papers‖.

Much of the concern

–

and, indeed, confusion

– about what took place at the CRU in relation to peer review may stem from misunderstandings about what peer review is and what it can be expected to do.



## Peer review: firewall or the weakest link?
For scientific journals, peer review is the confidential evaluation of a submitted manuscript by one or more individuals who are experts in an aspect of the work under scrutiny.

Who invented peer review? It‘s hard to be sure, but possibly the prize goes to Ishaq bin Ali Al Rahwi (AD 854-931) (7). In his book, Ethics of the Physician , Al Rahwi apparently encouraged doctors to keep contemporaneous notes on their patients, later to be reviewed by a jury of fellow physicians. But the serious business of journal peer review had to wait another 800 years. Henry Oldenburg, editor of Philosophical Transactions of the Royal Society , was the first modern editor to adopt peer review in the seventeenth century. He used it to famous effect, provoking often fractious, but illuminating, debates between scientists across Europe.

Technology

–

first the typewriter, then the photocopier, and now the Internet

– has greatly facilitated peer review. Any scientific journal that lays claim to respectabilityAPPENDIX 5



### 128 must have a robust peer review process. At The Lancet

, the process goes like this. A research paper is submitted electronically to a secure database and allocated by an editor to a colleague. The first or second editor can reject the manuscript at that early stage if the paper is judged to be scientifically poor, unsuitable for the journal‘s readership, unoriginal, or insufficiently topical. Journals differ here. For The Lancet around three-quarters of manuscripts are rejected at this point.

If a paper survives preliminary editorial review, it is discussed at a pre-review meeting to assess its suitability for external peer review. If judged a potential candidate for publication, the manuscript is sent to three expert advisors, commonly international and representing different methodological dimensions of the research, as well as a statistician. There is always the risk of group-think among experts. That is, there is an orthodox belief about a particular subject, strongly held, which resists alternative perspectives. Editors try to reduce the risk of group-think by sending papers to different and widely dispersed reviewers, deliberately seeking or even provoking critical reviews (just like Henry Oldenburg). Reviewers are not referees in the sense that they can blow a whistle and call time on the paper. We ask reviewers to provide written comments for the authors, confidential comments to the editors, and a detailed rating for each section of the paper. Those comments are collected, presented,
and discussed at a once- weekly manuscript meeting attended by all the journal‘s

editors.

At this stage, a paper can be rejected or we can open negotiations with authors. If we proceed, reviewers‘ questions and concerns are put to the authors, with appropriate guidance from editors. The authors will reply by answering each question from reviewers, submitting a revised manuscript that attempts to respond to all points raised by editors and reviewers alike. The authors may also disagree with or challenge reviewers with varying degrees of force. The revised paper is discussed again at a manuscript meeting. The options at this stage are to reject, accept, go back to the authors with further requests for clarification, or return to reviewers (old or new) for additional opinions. We proceed with further revisions of the paper until a final reject/accept decision is made. We know that with such a high rejection rate we may get it wrong. To limit errors of omission, we have a formal appeals process where editors promise to look again at a paper, weigh up the authors‘ arguments, and

reconsider our decision.

Once the paper is provisionally accepted, the peer review process is not over. The paper is then passed to a scientifically qualified assistant editor who edits the paper‘s technical content. Mistakes may still be found at this stage, leading to further editorial or expert review, even (though rarely) rejection. A lesson learned from sometimes bitter experience is that a paper is not fully accepted until it is published.

Here are some of the commonest questions asked about the peer review process (8).APPENDIX 5



### 129 Do reviewers make mistakes in their judgments ? Of course, and so do editors. Sadly, the scientific literature is littered with retractions of papers that once passed the test of peer review. Reviewers and editors are disappointingly human.

Are reviewers objective in their judgments ? Pure objectivity is impossible. For some subjects, an editor can predict the judgment of the reviewer based on past experience with that reviewer. But this misses the point of what an editor is seeking. It is not simply the judgment of reject/accept that an editor wants from a reviewer. That decision is the responsibility of the editor and the editor alone. What an editor really seeks is a powerful critique of the manuscript

– testing each assumption, probing every method, questioning all results, and sceptically challenging interpretations and

co nclusions. Armed with that critique, the editors decide

– and take full responsibility for deciding.

Are reviewers willing to accept new ideas ? Certainly, they are, although they might question those ideas to destruction. The vast majority of reviewers take their responsibility as advisors very seriously indeed. They themselves are often on the receiving end of peer review. Most try to be as open as possible to new findings, although we encourage them to ask difficult and awkward questions. There are occasional exceptions. For example, the world of complementary and alternative medicine (CAM) divides the medical community. Orthodox medicine mostly rejects papers about reflexology, iridology, and acupuncture treatment that invokes invisible pathways (meridians) of qi. CAM is served by a separate class of journals that have little overlap with the more mainstream medical literature. In this instance, ideas are
incommensurable.

Despite peer review, are authors able to get away with dishonest or dubious
research ? Yes, they are. Peer review does not replicate and so validate research. Peer review does not prove that a piece of research is true. The best it can do is say that, on the basis of a written account of what was done and some interrogation of the authors, the research seems on the face of it to be acceptable for publication. This claim for peer review is much softer than often portrayed to the general public. Experience shows, for example, that peer review is an extremely unreliable way to detect research
misconduct.

Are peer reviewers accountable for what they do ? Yes, to the editor. But in a broader sense, to the scientific community and to the public as well. To a large extent, the trust society places in science depends on the scientific process, including peer review and publication, getting it right most of the time.

Does peer review improve the quality of published research? In our everyday practice, we see that it does. And research suggests that it does too (9). Peer review improves discussion of the limitations of research. It emphasises uncertainty. It invites just ification of generalisability. As one study of peer review concluded, ―peer reviewAPPENDIX 5



### 130 is a negotiation between authors and journal about the scope of the knowledge claims
that will ultima tely appear in print‖ (9).

Is there still a need for peer review, given the extraordinary ability of the Internet to enable continuous open criticism of research once published (that is, surely a thousand readers as reviewers after publication are better than 4 reviewers selected by editors before publication)? There is no right answer to this question. Different views have been expressed. Certainly, post-publication peer review adds greatly to the understanding of a piece of research. But watching pre-publication peer review in

a ction - both at the macro level of external expert review and the micro level of technical editing - and seeing the extent to which research papers change (mostly for the better) after peer review, I think that pre-publication review still has an important part to play in science. At its best, pre-publication peer review clarifies, introduces uncertainty, insists on placing new work in the context of the totality of available evidence, demands a careful explanation of limitations, and prevents flights of fanciful over-interpretation.

Peer review has changed considerably during the past two decades (since I became an editor). First, the stakes are higher. Individual and institutional success depends on getting papers published in high-impact journals. Citation data are now a standard
metric for measuring research performance. This trend has increased competition and rivalry for places in the best journals. Second, the globalisation of science has expanded the geographic range of papers submitted to journals. Research originating from China, for example, is now far more common than even five years ago. The internationalisation of science has further intensified competition for publication.

Third, research papers are increasingly multi-disciplinary, requiring a much broader range of expertise during peer review. Fourth, science is a stronger part of our public culture now than it once was. What scientists used to write only for other scientists is today available to

–

and sometimes read by

– non-scientists, policy makers, and the
media . Fifth, the importance of statistics has grown substantially. Whereas twenty years ago The Lancet had no separate statistical peer review process, every paper we now publish has been carefully scrutinised by an independent statistical advisor. Editors are now far more aware of analytic errors in research. Sixth, to address the often conflicting results of individual research studies that are trying to answer the same (or a similar) question, a new type of research method has been devised

– the systematic, as opposed to the narrative, review. Systematic reviews aim to search for particular types of study (eg, the randomised trial), then select only the best according to pre-specified criteria, and, if possible, to combine those findings in a statistically meaningful way (which is called meta-analysis). Examples include the risk of cervical cancer among women taking hormonal contraceptives (10) and the effects of a class of medicines on heart disease (11). In biomedicine, the Cochrane Collaboration is the most mature example of an effort to create a database of systematic reviews on

treatments.APPENDIX 5



### 131 Finally, editors have had to face an upsurge in the discovery of episodes of research misconduct (fabrication, falsification, and plagiarism). The increasing awareness of research fraud had led not only to greater vigilance (hopefully not suspicion) among
editors but also to the birth of institutional mechanisms to set standards and advise on research practice (eg, the Committee on Publication Ethics).

Because of the faith journal editors have in peer review, together with the empirical evidence they believe exists to support peer review, they take it very seriously indeed (12). That said, editors are well aware that peer review is anything but uncontroversial. Scientific discoveries that later turn out to be flagrant episodes of dishonesty

–

from Woo- Suk Hwang‘s fabricated claims in Science about cloning embryonic stem cells, to Andrew Wakefield‘s falsifications in Th e Lancet

– are not uncommon. They raise troubling questions about the robustness of peer review. Editors are only too well aware of the limitations of the peer-review system. Authors, for example, can be deeply resistant to responding to questions from anonymous critics (this fact at least partly drives the argument for fully transparent peer review, where reviewers have to disclose their names to authors). The reluctance of some
authors

– and some very famous authors, at that

– to take the comments of their peers seriously stems from the fact that they believe they have no peers. As one historian of peer review put it, somewhat poetically, ―anyone who possessed the MD degree had no reason to defer to any colleague as an expert greater than he or she‖ (13).

So what is peer review in today‘s scientific culture? Various views have been more or less vividly expressed. Peer review is a ―sacred academic cow‖, according to one editor (14). She put it rather well:

“the „sacred‟ cow of peer review wanders the me adows of scientific publishing because together, scholars and editors, believe that it is the best mechanism we have to improve the quality of published papers...The one component that we cannot control is that of competition

in the academic world, an issue that continues to circle in all disciplines. We all have stories to tell about the viciousness of academic politics...although we might not be able to eradicate the politics, we can at least understand them for what they are ...”

Everyone

– scientists, the public, policymakers, politicians

– would like to believe that peer review is a firewall between truth and error (or dishonesty) (15). But as the editor of one leading specialist medical journal has rightly pointed out, ―Ther e is no question that, when it comes to peer review, the reviewers themselves are the weakest (or strongest) links‖ (16). This frustration among editors and scientists that peer review cannot always live up to the claims sometimes made for it produces frequentAPPENDIX 5



### 132 expressions of dismay. Is peer review a castle built on sand or the bedrock of scientific publishing (17)? Is peer review a landmark, landmine, or landfill (18)? Or, put bluntly, is peer review simply in crisis? (19). Is it ―a flawed process at the he art of science and journals‖ (20)?

Unfortunately, there is evidence of a lack of evidence for peer review‘s efficacy. In 2002, Tom Jefferson and colleagues published a startling systematic review of all the evidence about editorial peer review in biomedical journals. Their exhaustive search yielded only a handful of studies. The conclusion? ―Editorial peer review, although widely used, is largely untested and its effects are uncertain‖ (21). They went on, ―Given the widespread use of peer review and its im portance, it is surprising that so little is known of its effects.‖ Jefferson and his colleagues have confirmed their observations more recently (22). Their findings have been replicated by others (23). To be fair, there is some evidence that micro peer review

– technical editing

–

can improve papers in biomedical journals (24). But, once again, this evidence is not as robust as one would either like or have expected.

Jefferson extended his investigation of peer review by arguing that the objectives of the review process were also unclear (25). Without clear objectives, proving the value of peer review (or not) would be impossible. After almost 350 years of journal peer review, our zeal for and confidence in the peer review process seem inversely proportional to our knowledge about what it actually does for science. Those who make big claims for peer review need to face up to this disturbing absence of
evidence.

Worse still, what evidence is slowly accumulating should perhaps make scientists, policymakers, and the public pause. Many who place great weight on the reliability of the peer-reviewed scientific literature believe that it reflects the judgment of the scientific community about the quality of research. But evidence suggests that acceptance of research for publication may well depend on factors other than scientific quality alone (26). Furthermore, peer reviewers will disagree greatly in their recommendations to editors about a particular research paper. Yet editors seem to be significantly influenced by reviewers who, when the quality of their advice is measured independently, turn out to be extremely unreliable in their overall judgments (27). Editors, some critics could reasonably argue, need to pay less, not more, attention to the recommendations of their peer reviewers.

Scepticism about peer review is healthy. But every editor knows that peer review can be an indispensible aid to his or her work. Peer review can rescue science from embarrassment and error. An extreme example goes some way to showing why. Peter Duesberg is a well-known molecular virologist who believes that HIV is not the cause of AIDS. In 2009, the journal Medical Hypotheses published a paper by Duesberg arguing that the deaths attributed to AIDS in South Africa were false. The editor of Medical Hypotheses operated an editorial policy of no external peer review. TheAPPENDIX 5



### 133 justification was that peer review might suppress creative thinking. In the case of the Duesberg paper, the idea that HIV does not cause AIDS was not new. More imp ortantly, South Africa is only now reversing its disastrous denialist policies on

HIV- AIDS. To consider Duesberg‘s old (and discredited) idea at a critical moment for the country he was writing about would, most reasonable editors would conclude, require some kind of external peer review to assist decision-making. The editor did not seek expert reviews. He accepted the paper within a few days of its submission. Many scientists in the AIDS community were appalled. They wrote to the publishers (Elsevier, also the publishers of The Lancet ) to complain. Elsevier removed the paper from its online database pending the results of an independent investigation. The

Lancet was asked to review the paper. We did so and the reviews were uniformly and deeply critical. No journal could have conceivably published the Duesberg paper based on these reviews. The Duesberg paper remains retracted, excised from the scientific literature. Here is an example of what can happen when peer review is excluded from a journal‘s processes, and why peer review can bring important information to bear on judgments about the suitability of research for publication. Thanks to these events, the journal will now implement peer review. The publishers are seeking a new editor (28).



## Climate science and peer review
The events surrounding the peer review of certain climate science papers have raised important questions about peer review. But some of these questions may be based on a misinformed view of the peer review process. Here, I touch on general themes in peer review that have emerged during the debate about the role of climate scientists in research publication.

It is common for editors to have multiple, intense, and sometimes sharp interactions with authors and reviewers. Publication matters. Authors and reviewers are frequently passionate in their intellectual combat over a piece of research. The tone of their exchanges and communications with editors can be attacking, accusatory, aggressive, and even personal. If a research paper is especially controversial and word of it is circulating in a particular scientific community, third-party scientists or critics with an interest in the work may get to hear of it and decide to contact the journal. They might wish to warn or encourage editors. This kind of intervention is entirely normal. It is the task of editors to weigh up the passionate opinions of authors and reviewers, and to reflect on the comments (and motivations) of third parties. To an onlooker, these debates may appear as if improper pressure is being exerted on an editor. In fact, this is the ordinary to and fro of scientific debate going on behind the public screen of science. Occasionally, a line might be crossed. We experienced such a border crossing recently, where several reviewers and third parties encouraged us to delay publication of a paper for non-scientific reasons (29). Defining that line is the crucial task when judging the role of CRU scientists.APPENDIX 5



### 134 One issue that is important to solve for the peer review process to work effectively is
the full disclosure of all financial and relevant non-financial conflicts of interest. If a research paper about drug A for disease Y is sent to a reviewer who has shares in a company that makes drug B, also for disease Y, there is a potential for the introduction of bias into that reviewer‘s advice to the journal – favouring drug B over
dr ug A. The editor may still want and value that reviewer‘s advice, but s/he needs to know about the reviewer‘s financial conflict to judge the weight s/he gives to the review. Non-financial conflicts may be even more important. If a scientist has devoted a life‘s work to theory A about disease Y, then clearly s/he might be biased if s/he is sent a manuscript that criticises theory A and proposes an alternative and compelling theory B for that same disease. Again, the editor would expect the reviewer to declare any non-financial academic or intellectual conflicts that might have the potential to influence that reviewer‘s critique. In the field of climate research, con clusions about the meaning of the science may have been strongly held. The fact that these conclusions were strongly held is entirely to be expected and should simply have been fully disclosed to editors during the peer review process.

It would be wrong for editors not to listen to advice about publication even after acceptance of a paper. A paper is only fully accepted when it is published. New information that informs the decision to publish a provisionally accepted paper before publication can be very valuable. The Lancet has rejected papers in this twilight zone of peer review. After publication, criticism is common and welcome, even lethal criticism. This is the much vaunted self-regulation of science

– except that sometimes editors and authors are reluctant to act when things go wrong after publication. In climate science, disputatious research is unsurprising. It would be extremely surprising if that research did not provoke heated exchanges both before and after publication.

Much has been made of whether scientists should or should not take public positions on the meaning of their data, especially if those data relate directly to policy or practice. The reality is that they do, all the time. Science does not exist in a political vacuum. The idea that scientists, including climate scientists, are neutral observers, bereft of opinions, is naïve. In biomedical and public health research, scientists are often quick to make statements applying their data to the real world. They will often do so passionately and be well known for those passionate views. Indeed, the current climate of science is such that scientists are encouraged at every stage of their research to consider the impact

– economic or human

–

of what they do, and to trumpet that impact. Research assessments in the future are likely to include a measure of impact when judging the quality of a scientist‘s work. In relation to peer review, the scientific, policy, or political positions an author, reviewer, or editor may hold could intervene to bias a review in one particular direction. There have been many examples of such conflicts in other scientific disciplines

– eg, psychology (30) and genetic epidemiology (31). These episodes are troubling, but an almost inevitable consequence of the way peer review is ordinarily done.APPENDIX 5



### 135 The intersection of politics and science in climate change has arisen at least once
before. The Skeptical Environmentalist , written by Bjørn Lomborg and published by Cambridge University Press, led to huge pressure on the publishers to withdraw the book (32). Although the manuscript was reviewed by four experts who all recommended publication, the scientific backlash was acute. Letters of protest were written to newspapers. One scientist refused to work with Cambridge University Press ever again. Lomborg was attacked physically.

Chris Harrison, in his thoughtful reflections as the editor at Cambridge University Press who dealt with Lomborg‘s book, points out that peer review ―offers no guarantees of always ensuring the  ̳truth‘‖ (32). But in the case of The Skeptical
Environmentali st , the concerns were as much political as scientific. The publication of this book by a respected scholarly press might play ―to a particular political agenda and can be used and abused by vested corporate and political interests.‖ Harrison rejected the idea that he should have applied these kinds of value judgment in the editorial process. But he notes, ruefully,

“While it may not be the responsibility of an editor to second-guess how a publication will be received in political circles, it is clear that politicians and advocacy groups have always maintained a keen interest in what is published and that they have sought to influence editorial decisions.”

Harrison defended the scholarly publishing industry‘s commitment to pluralism. He

wrote,

“Given the scale of interest in the environment it is

perhaps peculiarly incumbent on the academy and general interest intermediaries [Cambridge University Press] to host as full and as open a debate as possible...the public and the academy can surely only be better served by an opportunity to review and debate a wide range of perspectives.”

This commitment to pluralism would be the likely view of many scientific editors, even when controversy follows. There are interesting parallels between the response
of t he scientific community to Lomborg‘s sceptical book and the reaction of some scientists to journal articles on climate change which expressed an opinion contrary to their own. One might conclude that these kinds of extreme debate, although difficult, are part of the normal fabric of scientific discourse. The question to be answered is: where is the line to be drawn between vigorous scientific exchange and improper attempts to close down debate (these two positions can be remarkably close to one another)? But one should also be conscious of what some observers have described asAPPENDIX 5



### 136 the ―chilling‖ effect of political controversy on science. A survey of US National Institutes of Health scientists revealed that many engaged in self-censorship after they found themselves the subject of political criticism for their work (33). Political disagreement over science can shape not only the behaviour of scientists but also the future of science itself. As Joanna Kempner has noted,

“There is a role for democratic public e ngagement in science. The policy challenge will be to encourage this public voice in scientific decision-making, while enabling scientists to submit and conduct innovative studies, even when they may provoke controversy.”

Increasingly, commercial, as well as political, interests are also intervening to threaten the integrity of peer review (34).

Peer review and publication can provoke important questions about access to data. During the review process, reviewers may seek more information. Except in allegations of fraud, it would be highly unusual to provide or request raw data (even then, journals expect institutions to take responsibility for investigating the authenticity and reliability of original data). But access to data may be sought after publication. This is a highly contentious and unresolved issue. For example, Andrew Montford argues that,

“the more important story in terms of the conduct of science in this country concerns the repeated refusals of CRU staff to release the data and code underlying their global- temperature index...sceptics are universally of the opinion that the scientific method requires all research material to be released to friend and foe alike...” (2)

In the field of medicine, these issues are currently the subject of much disagreement. While many parties might like to see greater sharing of data, this practice remains unusual. The Wellcome Trust is taking an especially strong interest in data access. It proposes a code of conduct calling for ―maximum public access t o data of public health importance.‖ The very fact that this proposal is being made illustrates the point that routine access to data is not a settled issue or a universal norm in science, as some
claim.

The issue of retention of records and exclusion of data in climate research is also a matter relevant to the peer review process and the ordinary working of journals. Journals do expect records to be kept for limited periods (say, 5 years, although journal practices vary). And they are comfortable with the exclusion of data provided
that those exclusions

–

and the reasons for exclusion

– are fully described, with appropriate sensitivity analyses being completed where necessary.APPENDIX 5



### 137 Two additional dimensions of peer review have been raised in the debate over C RU‘s work. One relates to confidentiality, the other to uncertainty. Editors send manuscripts to reviewers based on a principle of confidentiality. The author expects the editor to maintain a covenant of trust between the two parties. The editor will not misuse the author‘s work by circulating it outside of the confidential peer review process. The editor expects that covenant of trust to be honoured by the peer reviewer. No manuscript should be passed to a third party by a reviewer without the permission o f the editor, usually on the grounds of improving the quality of the critique of the manuscript by involving a colleague in the review process. A disclosure to a third party without the prior permission of the editor would be a serious violation of the peer review process

–

a breach of confidentiality. It is also of paramount importance to report fully in all published scientific papers both quantitative and qualitative measures of uncertainty. One of the main benefits of peer review is to focus on areas

o f potential uncertainty and to ensure that those uncertainties are fully acknowledged, measured, and reported.



## The future of peer review

Peer review is a human process and so will always contain flaws, produce errors, and occasionally mislead. Given that journals are the gatekeepers of scientific publication, they have enormous

– probably too much

– influence over the reputations of scientists, research units, and universities. Many measures of academic success depend upon journal publication

– promotion, tenure, grants, fame, and personal wealth. It is not surprising that journals, and the main decision aid used by journals (peer review), are the subject of constant tension and occasionally explosive controversy. At such moments, it is not only essential to be clear (and modest) about what peer review can do, but also to look for opportunities to do better.

Journal articles are highly stylised reports of research. The linear and logical style of the research they report rarely presents a true or accurate picture of how a piece of research was done. As the Nobel laureate, Peter Medawar, put it (35) in his essay ―Is the scientific paper a fraud?‖ (to which he answered that it was),

“[the scientific paper] misrepresents the processes of thought that accompanied or give rise to the work

that is described in the paper...The scientific paper in its orthodox form does embody a totally mistaken conception, even a travesty, of the nature of scientific thought.”

Medawar‘s point was that, ―There is no such thing as unprejudiced observations.‖ To add insult to injury, research papers may not even fully represent the views of theAPPENDIX 5



### 138 authors who completed the work (36), and when faults are found after publication those faults may be completely ignored in the subsequent use of that research (37).

There are actions that the scientific community could take to improve this far from happy state of affairs surrounding one its foundational processes. First, there are new opportunities and techniques available to search out, identify, and eliminate (or at least reduce) unwanted bias in the peer review process (38, 39). Second, all young scientists should receive formal training

– which they currently do not

– in the standards and ethics expected in the peer review process (40). It is scandalous that peer review is simply not taken as seriously as it should be in the training of scientists. The result is that peer review is often idiosyncratic and sometimes unreliable, fuelling

sc ientific controversies, such as that over climate science, rather than defusing those controversies. Strengthening the training, standards, and expectations around peer review would do much to make the quality of peer reviewing part of the formal appraisal of a scientist‘s contribution to his or her subject. There is a demand for training in peer review (41). And the ethical dimensions of the review process are now sufficiently concerning to scientists that they merit training as much as the more formal methodological aspects of reviewing (42). Disappointingly, existing training packages in peer review deliver little benefit to the quality of the peer review process (43-45).

Third, the peer review process is enormously inefficient. Individual journals will undertake peer review and reject manuscripts that will then cycle around other journals until either the paper is accepted or the authors are sufficiently exhausted that they abandon attempts at publication. In the face of such gross inefficiencies, some scientific communities have tried to bring journals together to cooperate and make the review process not only more efficient, but also less costly on the time and energy of reviewers, authors, and editors (46). Alternatively, there may be intra-journal procedures that can be introduced to deliver more efficient peer review (47).

Fourth, journal editors should adopt more effective methods to resolve disputes between authors, reviewers, and readers. Within the journal, an ombudsperson operating independently of the editors can be one useful way to resolve intractable disagreements about journal processes (48). If a dispute remains impossible to resolve, journal editors can take their concerns to the Committee on Publication Ethics, a charity that aims to set standards for journal practices, including peer review. Journal editors should consider using this facility more often than they currently do

– in some ways, it represents the collective wisdom of a wide range of journal editors, a collective wisdom that any scientific editor can draw upon in times of crisis.

L astly, peer review should be a subject for research in its own right. Although there is a small group of scientists who study peer review (a biomedical peer review congress is held every 4 years), that community is extraordinarily fragile when measured against the size and importance of the contribution peer review makes to science (49).APPENDIX 5



### 139 Historically, science funding bodies have been reluctant to invest in research on peer review. This reluctance is partly responsible for the present vacuum in our knowledge about the way scientific knowledge is constructed, reported, and discussed. One positive result of the debate over the role of CRU scientists in peer review might be to encourage funding bodies

– such as the Medical Research Council and the National
Institute for Health Research

– to take the science of peer review far more seriously.

Journals have inevitable limitations. When a paper with important policy implications is considered, editors can ask authors to balance their conclusions by putting the work in the context of existing evidence. Or we can commission an editorial that does the same. But a journal cannot adjudicate a public debate, and neither can conventional peer review. For those occasions when science meets (or clashes with) policy, there may be a case for referring the area of controversy to an independent body for a public inquiry. In the US, the model used is the Institute of Medicine, which tackles controversial aspects of health through its thorough and wide-ranging investigations and reports. There may be a case for such a body in the UK.

The best one might hope for the future of peer review is to be able to foster an environment of continuous critique of research papers before and after publication. Many writers on peer review have made such a proposal, yet no journal has been able to create the motivation or incentives among scientists to engage in permanent peer review (50-52). Some observers might worry that extending opportunities for criticism will only sustain maverick points-of-view. However, experience suggests that the best science would survive such intensified peer review, while the worst would find its deserved place at the margins of knowledge.

This process of weeding out weak research from the scientific literature can be accelerated through more formal mechanisms, such as the systematic review. A systematic approach to selecting evidence focuses on the quality of scientific methods rather than the reputations of scientists and their institutions. This more rigorous approach to gathering, appraising, and summing up the totality of available evidence has been profoundly valuable to clinical medicine. There may be useful lessons here for the IPCC. Climate sceptics and climate scientists, along with their colleagues in other scientific disciplines, would likely welcome this greater rigour and scrutiny. It would certainly promote quality and strengthen accountability to a more critical public (and media) with higher expectations of science. More importantly, intensified post as well as pre publication review would put uncertainty

– its extent and
boundaries

–

at the centre of the peer review and publication process. This new emphasis on uncertainty would limit the rhetorical power of the scientific paper (53), and offer an opportunity to make continuous but constructive public criticism of
research a new norm of science.

Finally, there are more general institutional lessons to be learned from the events that took place at the CRU. Peer review cannot eliminate controversy. Leaders of researchAPPENDIX 5



### 140 organisations

–

from University Vice-Chancellors to Unit Directors

– might consider a risk assessment of the research being done under their stewardship. Mapping research in this way

–

who is doing what, where, and how?

– would provide the institution with signals of potential controversy if the work addressed an issue of substantive public, policy, or political interest. This kind of awareness might enable
scientists

–

and, indeed, the institution itself

– to prepare better for any likely public disputes. Vigilance over the research being undertaken at a university could usefully anticipate difficulties, promote best practices, protect reputations, and build wider public trust and confidence in the research process. These outcomes, together with more realistic expectations about peer review and the processes of science, would be valuable consequences of the experiences at the University of East Anglia.

1. Montford A. The Hockey Stick Illusion : Climategate and the corruption of
science (Stacey International, 2010)

2 . Montford A. Heated discussions. Times Higher Education March 25, 2010: 43-44.

3. Monbiot G. Our narrow, antiquated school system is at the root of the climate email fiasco. The Guardian April 6, 2010: 25.

4. Booker C. Climategate: a scandal that won‘t go away. The Sunday Telegraph April 18, 2010: 29.

5. House of Commons Science and Technology Committee. The disclosure of climate data from the Climate Research Unit at the University of East Anglia (Stationery Office, 2010).

6. This quotation is taken from an email from Phil Jones to Michael Mann.

7. Spier R. The history of the peer review process. Trends in Biotechnology 2002; 20: 357-58.

8. Hernon P, Schwartz C. Peer review revisited. Library and Information Science Research 2006; 28: 1-3.

9. Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of Internal Medicine

. Ann Intern Med 1994; 121: 11 -21.

10. Smith JS, Green J, de Gonzalez AB, et al. Cervical cancer and use of hormonal contraceptives: a systematic review. Lancet 2003; 361 : 1159-67.

11. Jun M, Foote C, Lv J, et al. Effects of fibrates on cardiovascular outcomes: a systematic review and meta-analysis. Lancet 2010

12. Green SM, Callaham ML. Current status of peer review at Annals of Emergency Medicine. Ann Emerg Med 2006; 48: 304 -08.APPENDIX 5



### 141 13. Burnham JC. The evolution of editorial peer review. JAMA 1990; 263: 1323

- 9.

14. Fitzpatrick JJ. Peer review: a 2008 report on the sacred academic cow. Applied Nursing Research 2008; 21: 53.

15. Harms M. Peer review: the firewall of science. Physiotherapy 2006; 92: 193- 94.

16. DeMaria AN. Peer review: the weakest link. JACC 2010; 55: 1161-62.

17. Berger E. Peer review: a castle built on sand or the bedrock of scientific publishing? Ann Emerg Med 2006; 47: 157 -59.

18. Balistreri WF. Landmark, landmine, or landfill? The role of peer review in assessing manuscripts. J Pediatr 2007; 151: 107 -08.

19. Mulligan A. Is peer review in crisis? Oral Oncology 2005; 41: 135-41.

20. Smith R. Peer review: a flawed process at the heart of science and journals. J
R Soc Med 2006; 99 : 178-82.

21. Jefferson T, Alderson P, Wager E, Davidoff F. The effects of editorial peer review. JAMA 2002; 287: 2784 -86.

22. Jefferson T, Rudin M, Brodney Folse S, Davidoff F. Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database Syst Rev 2007; 2 : MR16.

23. Richards D. Little evidence to support the use of editorial peer review to ensure quality of published research. Evid Based Dent 2007; 8 : 88-89.

24. Wager E, Middleton P. Technical editing of research reports in biomedical journals. Cochrane Database Syst Rev 2008; 4: MR2.

25. Jefferson T, Wager E, Davidoff F. Measuring the quality of editorial peer review. JAMA 2002; 287: 2786 -90.

26. Aarssen LW, Lortie CJ, Budden AE, et al. Does publication in top-tier journals affect reviewer behaviour? PLoS ONE 2009; 4 : e6283.

27. Kravitz RL, Franks P, Feldman MD, et al. Editorial peer reviewers‘ recommendations at a general medical journal: are they reliable and do editors care? PLoS ONE 2010; 5 : e10072.

28. Enserink M. Elsevier to Editor: change controversial journal or resign. Science 2010; 327 : 1316.

29. Horton R. Maternal mortality: surprise, hope, and urgent action. Lancet 2010;APPENDIX 5



### 142 30. McCarty R. Science, po litics, and peer review: an editor‘s dilemma. American Psychologist 2002; 57: 198 -201.

31. Calnan M, Smith GD, Sterne JA. The publication process itself was the major cause of publication bias in genetic epidemiology. J Clin Epidemiol 2006; 59: 1312 -18.

32. Harrison C. Peer review, politics, and pluralism. Environmental Science and Policy 2004; 7: 357 -68.

33. Kempner J. The chilling effect: how do researchers react to controversy. PLoS Medicine 2008; 5 : e222.

34. Curfman GD, Morrissey S, Annas GJ, Drazen JM. Peer review in the balance.
N Engl J Med 2008; 358 : 2276-77.

35. Medawar PB. The Strange Case of the Spotted Mice (Oxford, 1996).

36. Horton R. The hidden research paper. JAMA 2002; 287 : 2775-78.

37. Horton R. Postpublication criticism and the shaping of knowledge. JAMA 2002; 287 : 2843-47.

38. Bornmann L, Mutz R, Daniel H-D. How to detect indications of potential sources of bias in peer review. J Informatics 2008; 2 : 280-87.

39. Ross JS, Gross CP, Desai MM, et al. Effect of blinded peer review on abstract acceptance. JAMA 2006; 295 : 1675-80.

40. Walbot V. Are we training pit bulls to review our manuscripts? J Biol 2009; 8 : 24.

41. Snell L, Spencer J. Reviewers‘ perceptions of the peer review proces s for a medical education journal. Med Educ 2005; 39 : 90-97.

42. Resnik DB, Gutierrez-Ford C, Peddada S. Perceptions of ethical problems with scientific journal peer review. Sci Eng Ethics 2008; 14 : 305-10.

43. Schroter S, Black N, Evans S, et al. Effects of training on quality of peer review. BMJ 2004 doi: 10.11136/bmj.38023.700775.AE.

44. Callaham ML, Tercier J. The relationship of previous training and experience of journal peer reviewers to subsequent review quality. PLoS Medicine 2007; 4 : e40.

45. Schroter S, Black N, Evans S, et al. What errors do peer reviewers detect, and does training improve their ability to detect them? J R Soc Med 2008: 101 : 507 -14.APPENDIX 5



### 143 46. Saper CB, Maunsell JHR. The Neuroscience Peer Review Consortium. Brain
Res 2009; 1272 : 1-2.

47. Johnston SC, Lowenstein DH, Ferriero DM, et al. Early editorial manuscript screening versus obligate peer review. Ann Neurol 2007; 61: A10-12.

48. Horton R. The journal ombudsperson: a step toward scientific press oversight.
JAMA 1998; 280 : 298-99.

49. Linkov F, Lovalekar M, LaPorte R. Scientific journals are ―faith based‖: is there science behind peer review? J R Soc Med 2006; 99 : 596-98.

50. von Segesser LK. Peer review versus public review

–new possibilities of online publishing! Interactive Cardiovascular and Thoracic Surgery 2002; 1: 61 -62.

51. Mandviwalla M, Patnayakuni R, Schuff D. Improving the peer review process with information technology. Decision Support Systems 2008; 46: 29-40.

52. Liesegang TJ. Peer review should continue after publication. Am J Ophthalmol 2010; March; 149 : 359-60.

53. Horton R. The rhetoric of research. BMJ 1995; 310 : 985-88.



## Commentary on „ Understanding uncertainty:a brief history of peer review’

Elizabeth Wager, Publications Consultant Chai r of the Committee on Publication Ethics (COPE)

Richard Horton suggests that  ̳ If evidence has not been peer-reviewed, it is next to worthless‘. Earlier commentators have gone even further. The evolutionary biologist Edward Wilson commented that  ̳ a discovery does not exist until it is safely reviewed and in print‘. 1



### Peer review has been used to select articles for publication since the 17 th 

### century. It is likely that when Oldenburg was using peer review at London‘s Royal Society, his contemporaries at the Royal Society in Edinburgh and across the channel at the Académie Royale de Médicine were using similar systems. 2



### Interestingly, the inability of peer review to guarantee scientific integrity was noted as early as 1785 when the Committee of Papers of the Literary and Philosophical Society of Manchester noted that  ̳Responsibility concerning the truth of facts, the soundness of reasoning ... [and] the accuracy of calculations is wholly disclaimed; and must rest alone, on the knowledge, judgement, or ability of th e authors‘. 3



### Horton describes the peer review system used at The Lancet . This is typical of major medical journals that employ several full-time editors. Such journals reject aAPPENDIX 5



### 144 considerable proportion of submissions on the basis of in-house review alone. For example, at JAMA (the journal of the American Medical Association) only 36% of submissions are sent for external review and the remaining 64% are rejected after review by the journal editors alone. 4 

### At journals such as The Lancet and JAMA

, everything accepted for publication will have been reviewed by experts who are independent of the journal, but many articles will be rejected without review byexternal experts. In contrast, journals that do not have large editorial offices or fulltime editors, but whose editors are academics (generally fulfilling this role in addition to their regular job), generally send virtually all submissions for external review.

The Lancet generally sends papers to three reviewers. Other journals may use fewer, or occasionally more, reviewers. A survey of 200 journals from a range of disciplines found that 73% used two reviewers, 18% used three, 6% used one and 3% used more than three. 5



### Horton notes that  ̳ the scientific literature is littered with retractions of papers that

on ce passed the test of peer review‘. The biomedical database Medline (which includes over 19 million citations) currently contains nearly 1500 retractions. There have also been well-documented cases of journals failing to recognise important work. There is even a website devoted to accounts of journals that have rejected work that later led to their authors winning the Nobel prize. [ttp://www2.uah.es/jmc/nobel/nobel.html#reje].

Churchill‘s famous observation about democracy has often been applied to peer

re view, namely that it is the worst system for selecting papers except for all the other forms that have been tried. Horton mentions the possibility of alternative systems such as post-publication or public review. A few journals have experimented with these systems but found them unworkable. When Nature ran a trial of open peer review in 2006 (by posting submitted papers on its website), almost half the papers received no comments. 6



### While formal evidence for the effectiveness of peer review is lacking, there is extensive anecdotal evidence from both editors and authors of its benefits. Editors and publishers should strive to optimise their peer review systems and to reduce potential harm or bias. Guidance is available from several organizations including the Committee on Publication Ethics (www.publicationethics.org).

While the internet has not produced dramatic new ways of assessing articles submitted to journals, information technology does offer possibilities which could increase the efficiency of disseminating and verifying scientific data. Gene sequence data is now routinely entered onto public databases, and such data posting is a requirement for publication in traditional journals in some cases. However, before such posting could be practical, even for such homogeneous data as gene sequences, agreements had to be developed (e.g. MIAME). 7 

### Such standardization may be possible for other types of data yet, as Horton notes, in many disciplines, data sharing remains either controversial, or limited by technological, practical or ethical issues. 8



### Even with increased research and investment it seems unlikely that, despite its shortcomings, conventional peer review will be replaced or transformed in the next few years. In the meantime, researchers will probably continue to criticise the peerAPPENDIX 5



### 145 review system, sometimes with justification (because the system is flawed) and sometimes without (because researchers are human, feel strongly about their work and do not accept that some rejections are justified). The research community should listen to the criticisms of peer review and work towards fixing the remediable faults and correcting the inevitable errors. Editors and publishers should continue to work hard to prevent and detect misconduct by authors, reviewers and editors and should have systems in place for responding appropriately when misconduct occurs. However, judgement must be exercised to ensure that resources are wisely spent and care should be taken to avoid overzealous responses to individual cases or disproportionate reactions to rare problems.



## References

1. Wilson EO. Consilience: The Unity of Knowledge. Knopf, New York, 1998

2. Wager E. Peer review in science journals: past, present and future. In Holliman R et

al (eds) Practising Science Communication in the Information Age . Oxford University

Press, Oxford, 2009

3. Kronick DA. Peer review in 18 th 

### -century scientific journalism. Journal of the

American Medical Association 1990;263:1321-2

4. Fontanarosa P, DeAngelis C. To JAMA peer reviewers and authors

– thank you. Journal of the American Medical Association 2010;303:780

5. Current Practice in Peer Review . ALPSP, EASE, ALSSS. www.alpsp.org

6. Anon Overview: Nature‘s peer revew trial, 2006 www.nature.com/nature/peerreview/debate/nature05535.html

7. Brazma A. Minimum information about a microarray experiment (MIAME)

-- toward standards for microarray data. Nature Genetics 2001;29:365-71

8. Wager E. Publishing clinical trial results: the future beckons. PLoS Clinical Trials 2006;1:e31



## Competing interests statement

Elizabeth Wager is a freelance publications consultant who provides consulting, training, and editorial services mainly to pharmaceutical companies, publishers and academic institutions. She chairs the Committee on Publication Ethics (but has written this commentary in her individual capacity). She was part of the team that produced systematic reviews on the effects of peer review and technical editing cited by Richard Horton. She received a fee for producing this commentary.APPENDIX 6



### 146 

## APPENDIX 6 : DATA MINING - ACCESS TO THE ORIGINAL CRU E-MAIL ARCHIVE

1. Recognising that the e-mails improperly released into the public domain represent only a tiny fraction (less than 0.3%) of the e-mails archived by the key individuals in the CRU, the Review team sought to set these in context. The backup server (CRUBACK3) had been taken as evidence by the police as part of their own investigation and was held by police contracted forensic investigators. A full context could only be established by some form of access to the information held on this server. In seeking to gain this access a number of legal issues arose, notably that:

the server and its contents were evidence in the continuing police investigation; and in the opinion of UEA‘s legal advisers, unconstrained access to the contents of e-mails on the server by th e Review would raise potential privacy and data protection issues.

2. The compromise eventually reached with both the police investigative team and the UEA Registrar was for:

the University to contract an independent computer forensic analyst; the police fo rensic consultants to extract from CRUBACK3 all the e-mails from the various archived mailboxes of key UEA staff and to provide these under strict security conditions to the independent forensic analyst; the independent analyst, respecting the high evidential security requirements set out by the police team, to work within secure premises authorized by the police; the independent analyst to seek to determine the search or selection parameters that had extracted the improperly released e-mails from the multiple archives and to determine whether this process highlighted any additional material of relevance to the Review; and any material identified by the the analyst to be redacted by the University, in terms of protecting the identity of non-UEA recipients or authors, prior to being made available to the Review.

3. This whole process took an extended period to negotiate and implement. It became clear that a full analysis would require considerable further time and extensive manual intervention. It would introduce significant delay to the publication of the Review ‘s report. A decision was reached not to pursue this further on grounds of both time and cost against likely results. The Review had always regarded the e-mails as pointers to areas for detailed investigation and this had been complemented by extensive public requests for submissions and any other information in the public domain. A summary report by the independent forensic analyst has been placed on the Review website.APPENDIX 6



### 147 4. The Review ‘s own analysis of t hose e-mails already in the public domain is highlighted in the following charts, including a  ̳heat map‘ of correspondence intensity by month.

5. The selected e-mails relate largely to controversial issues, although they do not appear to have been selected on the basis of a simple word search, as indicated in the table below which shows the number of e-mails in which interesting words (or a component thereof) occur. Some of the more infamous e-mails contain words such as trick or hide and these occur comparatively infrequently:-APPENDIX 6



### 148 6. The periods of greatest intensity generally coincide with repeated selection of embedded e-mails having the same core title

– with 4 or more core title repetitions being viewed as significant in the table below.

Date Cor e Title Repeats Subject
Oct

- 98 climate of the last millennia... 4 Meeting to discuss multi

- proxy data prompted by Jones 1998 1
May

- 99 Straight to the Point 4 Perceived criticism of Briffa 1999 2
Sep

- 99 IPCC revisions 7 Making the case for tree ring proxies in IPCC TAR Chapter 2 Apr

- 02 Your letter to Science 5 Criticism of draft letter Mann 2002 3

Mar

- 03 Soon & Baliunas 9 Criticism of Soon 2003 4

Jun

- 03 Review

- confidential 4 Validity of proxy

- based temperature
reconstruction
Oct

- 03 draft 4 of proposed EOS response to Soon 2003 4

Feb

- 04 Stephen McIntyre 4 Requesting Russian tree ring data
Jan

- 05 [Wg1

- ar4

- ch06] IPCC last 2000 years
data 5 Drafting of IPCC chapter

Apr

- 05 WG1 LA2 meeting

- Overlap cluster
A 5 Coordination between IPCC Lead Authors
Jul

- 05 MWP figure 5 for IPCC report
Dec

- 05 HadCRUT2v 4 Missing Southern Pole data
Feb

- 06 some figures at last! 4 Proxy reconstruction for IPCC AR4

Mar

- 06 latest draft of 2000

- year section text 7 IPCC Chapter 6 in the SOD for the AR4 May

- 06 Wahl & Ammann paper 4 Use of Wahl 2006 5 in the SOD for the AR4
Jul

- 06 Special instructions/timing adjustment 4 Proxy reconstruction for IPCC Sep

- 06 No Subject 4 Significance of recent temperatures in IPCC report
Jan

- 07 not so fast 4 J. Eddy figure in IPCC report Apr

- 07 FYI 4 Climate Audit attacks on Wang and Jones
Dec

- 07 sorry to take your time up, but really
do need a scrub of this singer/christy/etc effort 11 Response to Douglas 2007 6

Jan

- 08 Update on response to Douglass et al. 5 Publication options to rebut Douglas 2007 6 May

- 08 EA 21389

- Probabilistic information to inform EA decision making on climate change impacts

- PCC(08)01 4 Tender pack for Environment Agency bid

May

- 08 JOC

- 08

- 0098

- International Journal of Climatology 4 Review of Santer 2008 7

May

- 08 Our d3* test 4 Validation of climate models

Jan

- 09 Temperatures in 2009 5 Prognosis for 2009 and UHI effect
Jun

- 09 2009JD012960 (Editor

- Steve Ghan):Decision Letter 4 Review of McLean 2009 8

Jul

- 09 ENSO blamed over warming

- paper in JGR 14 Response to McLean 2009 8

Sep

- 09 attacks against Keith 5 Selection of tree

- ring data
Oct

- 09 BBC U

- turn on climate 10 Recent 'lack' of warmingAPPENDIX 6



### 149 

## References

1. P.D. Jones, K.R. Briffa, T.P. Barnett and S.F.B. Tett, ― High-resolution palaeoclimatic records for the last millennium: interpretation, integration and comparison with General Circulation Model control-run temperatures ‖

, The Holocene 8, 4 (1998), pp. 455

– 471 2. KR. Briffa and T.J. Osborn, ― Seeing the wood for the trees ‖

, Science Vol. 284, 7 May 1999, pp. 926-927 3. M.E. Mann and M.K. Hughes, ― Tree-ring chronologies and climate variability ‖

, Science , Volume 296, Issue 5569, 3 May 2002, pp. 848-849, 4. W. Soon, S. Baliunas, ― Proxy climatic and environmental changes of the past 1000 years ‖

, Climate Research Volume 23: 89-110, January 2003 5. E.R. Wahl, D.M. Ritson, C.M. Ammann, ― R econstructing Past Climate from Noisy Data ‖

, Science 312, 529 (2006)
6. D.H. Douglass, J.R. Christy, B.D. Pearson and S.F. Singer, ― A comparison of tropical temperature trends with model predictions ‖

, International Journal of Climatology , (2007)
7. B.D. Santer, P.W. Thorne, L. Haimberger, K.E. Taylor, T.M.L. Wigley, J.R.
L anzante, S. Solomon, M. Free, P.J. Gleckler, P.D. Jones, T.R. Karl, S.A. Klein, C. Mears, D. Nychka, G.A. Schmidt, S.C. Sherwood, and F.J. Wentz,

― Consistency of modelled and observed temperature trends in the tropical troposphere ‖

, International Journal of Climatology, 28: 1703

– 1722 (2008) 8. J.D. McLean, C.R. de Freitas, and R.M. Carter, ― Influence of the Southern Oscillation on tropospheric temperature ‖

, Journal of Geophysical Research , Vol 14, D14104 (2009)APPENDIX 7



### 150 

## APPENDIX 7: LAND STATION TEMPERATURE DATA

1. Contents

This Appendix supports Chapter 6 of the report addressing land temperature data. It contains further information on two issues:

the Trial Analysis, referred to in Sections 6.3 and 6.4 of Chapter 6, and the particular issue of matching station identifiers, referred to in Section 6.5.1 of Chapter 6.


## 2. The Trial Analysis

2.1 Introduction

The Review has conducted a trial analysis to demonstrate what an independent researcher is able to do, using publicly available land station temperature information, should they wish to replicate the CRUTEM analysis. We have only carried out each step as far as necessary to demonstrate proof of principle and to compare results at a level sufficient to inform our findings in respect of allegations made against CRU. It is important to note that we have not carried out all of the steps with the necessary degree of scientific rigour needed to form scientific conclusions, and we emphasise that no such conclusions should be drawn from this work.

We have performed the following steps:

1. Obtained raw station temperature data from two repositories as ASCII files. 2. Written computer code in C++ to read this data into memory in a suitable format. 3. Written computer code to process this data and create gridded global average temperature trends. 4. Compared the results to those of CRU and others in order to inform our findings in respect of the allegations made against CRU.

We have not carried out the full process of homogenising all data. There is no fundamental barrier to doing so. It is a time consuming step and requires the experience and expertise of a climate research scientist to process each of the raw station data entries before averaging. We comment upon the significance of homogenisation below.



## 2.2 Primary Station Data

Primary station data was obtained from: NCAR:http://www.ncar.ucar.edu/tools/datasets/ WWR: http://dss.ucar.edu/datasets/ds570.0/ GHCN: http://www.ncdc.noaa.gov/oa/climate/ghcn-monthly/index.php.APPENDIX 7



### 151 The GHCN data was freely available. GHCN provided several files including a list of stations and locations, a data set of unadjusted data, and a dataset of adjusted data. This contained 7280 stations in the unadjusted file and 4771 in the adjusted file. This data set contained in some cases several duplicates from the same station.

For NCAR it was necessary to register (free) first. The NCAR data was a single file containing all necessary information and contained 4646 stations.

The formats of each file were easy to understand. The formats were different from each other.



## 2.3 Code to Process the Data

Code was written in C++ to:
Parse the sou rce files and assemble the information into computer memory. Pass over each station and calculate the monthly ―normals‖ defined as the average over a set period. In this case we required good data from at least 10 years in the period 1955-1995. This is larger than was used by CRUTEM in order to obtain more valid normals without having to resort to external records. In the case of GHCN, choose the duplicate with the most valid monthly normals. Calculate the anomaly for each monthly measurement for each station. The anomaly is defined as the monthly temperature minus the normal for that month. Either choose to use all stations, or only those matched to the CRUTEM3 station list published in 2009. Pass over the data set and assemble average annual anomalies in each cell of a 5x5 degree grid of latitude and longitude. Calculate the overall annual average anomaly by averaging over grid cells. Do this for each of GHCN (unadjusted), GHCN (adjusted) and NCAR, and plot the annual average anomaly as a function of time. Separately read the published CRUTEM3 product and form the same average over grid cells and plot this on the same figure for comparison.

This code was straightforward to write from scratch. The minimal code needed to read and process the data sets amounts to only a few hundred executable lines and took about 2 days to develop.



## 2.4 Results of Global Temperature Trends

Figure A1 - A4 show the comparison of the time series obtained from the three data

se ts GHCN (unadjusted), GHCN (adjusted) and NCAR. A simple 5 year smoothing is applied. Also shown is the result obtained from the CRUTEM3 5x5 gridded product.APPENDIX 7



### 152 The set of input stations used for each line is not identical - the numbers are GHCN (una djusted) [7280], GHCN (adjusted)[4771], NCAR [4646]. Of these, all stations wh ich ha ve a valid monthly normal and valid monthly data are included in Fig A1.

I n Fig A2 we have approximately matched the stations in each dataset to those used in CRUTEM3. This is described in more detail below. The numbers matched are GHCN (unadjusted) [3722], GHCN (adjusted) [2962], NCAR [1957]. CRUTEM itself used 4138 stations.

Figure A3 is the same as A2 but showing only the two lines which correspond to unadjusted data (GHCN and NCAR).

Figure A4 is the same as A2 but showing only the two lines which correspond to adjusted data (GHCN (adjusted) and CRUTEM).

Figure A5 reproduces a comparison of different studies taken from the IPCC 4 th 

### Report Chapter 3.

Figure A1: Temperature anomaly time series created by the Inquiry Team‘s own trial analysis using a 5x5 grid with 5 year smoothing. Shown are results obtained from GHCN (blue)

, GHCN-adjusted (yellow) and NCAR (green) . Also shown is the CRUTEM3 line (black). The Y-axis is degrees x 10. The X-axis is year.APPENDIX 7



### 153 Figure A2: Same as Fig A1 but with approximate match of stations to those used in CRUTEM3. Axes same as Fig A1.

Figure A3: Same as Fig A2 but showing only the unadjusted lines (GHCN and NCAR) Axes same as Fig A1.

Figure A4: Same as Fig A2 but showing only the adjusted lines (GHCN (adjusted) and CRUTEM). Axes same as Fig A1.APPENDIX 7



### 154 Figure A5: Taken from IPCC 4 th Report Chapter 3.



### The exercise and comparison of all figures demonstrates that: 1. Any independent researcher may freely obtain the primary station data. It is impossible for any group to withhold data. 2. It is impossible for any group to tamper improperly with data unless they have done so to the GHCN and NCAR (and presumably the NMO) sources themselves. 3. The steps needed to create a temperature trend are straightforward to implement. 4. The computer code necessary is straightforward to write from scratch and could easily be done by any competent programmer. 5. The shape obtained in all cases is very similar: in other words if one does the same thing with the same data one gets very similar results.
6. The result does not depend significantly on the exact list of stations.
7. Adjustments make little difference.

By performing this simple test one determines easily that the results of the CRUTEM analysis follow directly from the published description of the method, and that the resultant temperature trend is not significantly different from the other results regardless of stations used or adjustments made. The test is therefore sufficient to demonstrate that, with respect to the declared method, the CRUTEM analysis does not contain either error or adjustments which are responsible for the shape of the resultant temperature trend.APPENDIX 7



### 155 A researcher can evidently produce a study which would test the CRUTEM analysis quite precisely, without let or hindrance from CRU.

W e do not make any statement regarding the correctness of any of these results in representing global temperate trends. We do not address any alleged deficiencies such as allowance for non climatic effects or the significant drop in station number post 1991. We do not address any possible deficiencies of the gridding method. These are entirely matters for proper scientific study and debate and lie outside the scope of this Review.



## 2.5 Discussion of Adjustments (Homogenisation)

In our trial analysis the GHCN and NCAR lines are without homogenisation adjustment. The CRUTEM3 and GHCN (adjusted) lines are both after homogenisation adjustments. The adjustments to CRUTEM and GHCN are made in different ways by different people. Even so the results are extremely similar. It is evident from this comparison alone that neither CRU nor those assembling the GHCN (adjusted) dataset have made inappropriate adjustments in order to bias or falsify the global temperature trend simply because those adjustments have no significant effect on the global average. Nevertheless we consider additional evidence that the adjustments made have no significant effect on the global temperature trend.

In the CRUTEM3 product the homogenisation adjustments are (i) made to only 298 (10%) of the stations and (ii) are known to be approximately evenly distributed about zero. Thus they would not be expected to have a large effect in the global average.

CRU have submitted detailed evidence of this (p49 and p54 of their submission) reproduced here as figure A6 and A7. Figure A6 shows a comparison of CRUTEM3 with the results obtained by (i) not applying adjustments to the 298 stations and (ii) leaving out the 298 adjusted stations. There is no significant difference as expected. Figure A7 shows the CRUTEM3 analysis compared to unadjusted results from other groups. Again no significant difference is seen. This is fully in accord with the results of the trial analysis.

Comparison of all of these plots demonstrates that adjustments which have been made are largely immaterial to the overall shape of the temperature series and are not on their own the cause of the rising temperature trend in recent years.

We have not addressed the opposite question of whether the adjustments made are adequate to allow for non climatic effects. This is entirely a matter for proper scientific study and debate and outside the scope of this review.APPENDIX 7



### 156 FIGURE A6 CRUTEM3: Removing AdjustmentsAPPENDIX 7



### 157 FIGURE A7 CRUTEM3: Comparison with unadjusted results from other groups


## 3. Matching Station Identifiers

We have used the information supplied by CRU from its website http://www.cru.uea.ac.uk/cru/data/landstations/. This lists the 4138 stations used in CRUTEM3. We have attempted to match these stations to those appearing the GHCN and NCAR data sets.

This process is not straightforward (at least was not straightforward to us as nonAPPENDIX 7



### 158 climate scientists). Problems arose through Lack of uniqueness of station identification numbers; Different identifiers appearing for the same station in some cases; Different station names in a few cases (spelling, order of words); Locations differing by a few degrees; Different conventions for location; Ambiguity due to duplicates in GHCN

We found it impossible to match stations purely from identifiers. Instead we used a combination of conjunctions of matching latitude and longitude, the first letters of station names, and station identifiers. We were in the end able to match the numbers shown in the table.

Number in CRUTEM3 Matched in GHCN Matched in NCAR
4138 37 22 (90%) 1957 (47%)

The number matched in GHCN (90%) is in accord with that claimed by CRU in their written responses. We have presumed that the remaining data is available from National Meteorological Offices or as part of more recent update messages.

The Review feels that it has pursued this enough to understand that (i) the matching of meta data (lists of stations) across sources is not straightforward and (ii) obtaining the final 10% of the primary data would require some additional effort (although anyone familiar with the field would presumably have a better knowledge of how to do this). This is a secondary issue to this Review, but we make the following observations

(i) It would benefit the global climate research community if a standardised way of defining station metadata and station data could be agreed, preferably through a standards body, or perhaps the WMO. As example an xml based format which would make the interpretation, use, comparison,

a nd exchange of data much more straightforward.

(ii) Without such standardisation there will remain residual problems in issuing unambiguous lists, and assembling primary data from them. We feel it would be in the public interest if CRU and other such groups developed a process to capture and publish a snapshot of the data used for

e ach important publication.APPENDIX 7



### 159APPENDIX 8



### 160 

## APPENDIX 8 : SOFTWARE AND DATA STANDARDS

Software Standards - The available guidance on software development methodologies is very extensive ranging from  ̳formal methods‘, where the desired properties of the system are expressed in a mathematically formal language, through to the  ̳Unified Process‘ developments of this decade. The UK Royal Academy of Engineering has published extensively in this area 1



### .
Data Management and Archiving Guidance is also available on Records Management and archiving from: the Joint Information Systems Committee (JISC) of the Higher Education Funding Council 2 

### ; the UK Research Integrity Office (UKRIO) 3 

### ; and the US National Academy of Sciences report ―Ensuring the Integrity, Accessibility and Stewardship of Research Data in the Digital Age‖ 4 

### . We note in particular Recommendation 2: “Research institutions should ensure that every researcher receives appropriate training in the responsible conduct of research, including the proper management of research data in general and within the researcher‟s field of specialization. Some research sponsors provide support for this training and for the development of training programs”. We also note Recommendation 5: “All researchers should make research data, methods, and other information integral to their publicly reported results publicly accessible in a timely manner to allow verification of published findings and to enable other researchers to build on published results, except in unusual cases in which there are compelling reasons for not releasing data. In these cases, researchers should explain in a publicly accessible manner why the data are being withheld from release”.

Information Security - There is again very extensive guidance available of information security issues including: from a standards perspective ISO/IEC 27002; for thought le adership the ―Jericho Forum‖ 5 

### and the ―Information Security Forum‖ 6 

### ; and from a professional perspective the ―Institute of Information Security Professionals‖ 7

1 http://www.raeng.org.uk/news/publications/list/reports/Engineering_values_in_IT.pdf http://www raeng.org.uk/news/publications/list/reports/Complex_IT_Projects.pdf

2 http://www.jiscinfonet.ac.uk/partnerships/records-retention-he/managing-research-records

3 See UKRIO  ̳Code of Practice for Research‘ Section 3.12 ―Collection and Retention of Data‖ at http://www.ukrio.org

4 http://www.nap.edu/catalog/12615 html

5 http://www.opengroup.org/jericho/

6 https://www.securityforum.org/

7 https://www.instisp.org/SSLPage.aspx