U.S. GOVERNMENT PUBLISHING OFFICE

WASHINGTON : 

For sale by the Superintendent of Documents, U.S. Government Publishing Office

Internet: bookstore.gpo.gov Phone: toll free (866) 512–1800; DC area (202) 512–1800

Fax: (202) 512–2104 Mail: Stop IDCC, Washington, DC 20402–0001

24–628PDF 2017

MAKING EPA GREAT AGAIN 

HEARING 

BEFORE THE

COMMITTEE ON SCIENCE, SPACE, AND 

TECHNOLOGY 

HOUSE OF REPRESENTATIVES 

ONE HUNDRED FIFTEENTH CONGRESS 

FIRST SESSION 

February 7, 2017 

Serial No. 115–01 

Printed for the use of the Committee on Science, Space, and Technology 

( 

Available via the World Wide Web: http://science.house.gov 
(II) 

COMMITTEE ON SCIENCE, SPACE, AND TECHNOLOGY 

HON. LAMAR S. SMITH, Texas, Chair 

FRANK D. LUCAS, Oklahoma 

DANA ROHRABACHER, California 

MO BROOKS, Alabama 

RANDY HULTGREN, Illinois 

BILL POSEY, Florida 

THOMAS MASSIE, Kentucky 

JIM BRIDENSTINE, Oklahoma 

RANDY K. WEBER, Texas 

STEPHEN KNIGHT, California 

BRIAN BABIN, Texas 

BARBARA COMSTOCK, Virginia 

GARY PALMER, Alabama 

BARRY LOUDERMILK, Georgia 

RALPH LEE ABRAHAM, Louisiana 

DRAIN LAHOOD, Illinois 

DANIEL WEBSTER, Florida 

JIM BANKS, Indiana 

ANDY BIGGS, Arizona 

ROGER W. MARSHALL, Kansas 

NEAL P. DUNN, Florida 

CLAY HIGGINS, Louisiana 

EDDIE BERNICE JOHNSON, Texas 

ZOE LOFGREN, California 

DANIEL LIPINSKI, Illinois 

SUZANNE BONAMICI, Oregon 

ALAN GRAYSON, Florida 

AMI BERA, California 

ELIZABETH H. ESTY, Connecticut 

MARC A. VEASEY, Texas 

DONALD S. BEYER, JR., Virginia 

JACKY ROSEN, Nevada 

JERRY MCNERNEY, California 

ED PERLMUTTER, Colorado 

PAUL TONKO, New York 

BILL FOSTER, Illinois 

MARK TAKANO, California 

COLLEEN HANABUSA, Hawaii 

CHARLIE CRIST, Florida 
(III) 

C O N T E N T S 

February 7, 2017 

Page 

Witness List ............................................................................................................. 2 

Hearing Charter ...................................................................................................... 3 

Opening Statements 

Statement by Representative Lamar S. Smith, Chairman, Committee on 

Science, Space, and Technology, U.S. House of Representatives ..................... 4 

Written Statement ............................................................................................ 6 

Statement by Representative Eddie Bernice Johnson, Ranking Member, Committee on Science, Space, and Technology, U.S. House of Representatives .... 9 

Written Statement ............................................................................................ 10 

Witnesses: 

The Honorable Jeffrey Holmstead, Partner, Bracewell LLP 

Oral Statement ................................................................................................. 11 

Written Statement ............................................................................................ 14 

Dr. Kimberly White, Senior Director, Chemical Products and Technology, 

American Chemistry Council 

Oral Statement ................................................................................................. 19 

Written Statement ............................................................................................ 21 

The Honorable Rush Holt, CEO, American Association for the Advancement 

of Science 

Oral Statement ................................................................................................. 30 

Written Statement ............................................................................................ 32 

Dr. Richard Belzer, Independent Consultant 

Oral Statement ................................................................................................. 37 

Written Statement ............................................................................................ 39 

Discussion ................................................................................................................. 52 

Appendix I: Answers to Post-Hearing Questions 

The Honorable Jeffrey Holmstead, Partner, Bracewell LLP ............................... 82 

Dr. Kimberly White, Senior Director, Chemical Products and Technology, 

American Chemistry Council .............................................................................. 88 

The Honorable Rush Holt, CEO, American Association for the Advancement 

of Science .............................................................................................................. 91 

Dr. Richard Belzer, Independent Consultant ........................................................ 95 

Appendix II: Additional Material for the Record 

Documents submitted by Representative Lamar S. Smith, Chairman, Committee on Science, Space, and Technology, U.S. House of Representatives .... 102 

Documents submitted by Representative Eddie Bernice Johnson, Ranking 

Member, Committee on Science, Space, and Technology, U.S. House of 

Representatives .................................................................................................... 113 

Documents submitted by Representative Randy K. Weber, Committee on 

Science, Space, and Technology, U.S. House of Representatives ..................... 187 
Page

IV 

Documents submitted by Representative Gary Palmer, Committee on Science, 

Space, and Technology, U.S. House of Representatives .................................... 190 
(1) 

MAKING EPA GREAT AGAIN 

TUESDAY, FEBRUARY 7, 2017 

HOUSE OF REPRESENTATIVES, 

COMMITTEE ON SCIENCE, SPACE, AND TECHNOLOGY, 

Washington, D.C. 

The Committee met, pursuant to other business, at 11:04 a.m., 

in Room 2318, Rayburn House Office Building, Hon. Lamar Smith 

[Chairman of the Committee] presiding. 
2 

LAMAR s. SMITH, Tsxiiis 

CHAIRMAN 

Q::ongrcss of the 'Bnitcd ~tatcs 

'!ilOUllC Of 'iRcprcscntatiUCll COMMITIEE ON SCIENCE, SPACE, AND TECHNOLOGY 

2321 RAYBURN HOUSE OFFICE 8U!LD!NG 

WASHINGTON, DC 20515~6301 

(202) 225-6371 www.scien;:;e,hollse.gov 

Full Committee 

Making EPA Great Again 

Tuesday, February 7, 2017 

11:00 a.m. 1:00 p.m. 

2318 Ray bum House Office Building 

The Honorable Jeffrey R. Holmstead, Partner, Bracewell & Giuliani 

EDDIE BERNICE JOHNSON, Texas 

RANKING M£MBER 

Dr. Kimberly White, Senior Director, Chemical Products and Technology, American 

Chemistry Council 

The Honorable Rush Holt, CEO, Amedcan Association for tbc Advancement of 

Science 

Dr. Richard Belzer, Independent Consultant 
3 

U.S. HOUSE OF REPRESENTATIVES 

COMMITTEE ON SCIENCE, SPACE, AND TECHNOLOGY 

HEARING CHARTER 

Tuesday, February 7, 2017 

TO: Members, Committee on Science, Space and Technology 

FROM: Majority Staff, Committee on Science, Space, and Technology 

SUBJECT: Full Committee Hearing: "Making EPA Great Again" 

The Committee on Science, Space and Technology will hold a hearing titled Making EPA 

Great Again on Tuesday, February 7, 2017, at II :00 a.m. in Room 2318 of the Rayburn House 

Office Building. 

Hearing Purpose: 

The purpose of this hearing is to examine the Environmental Protection Agency's process 

for evaluating and using science during its regulatory decision making activities. Witnesses will 

discuss how EPA can pursue environmental protection and protect public health by relying on 

sound science. 

Witness List 

• The Honorable Jeffrey Holmstead, Partner, Bracewell LLP 

• Dr. Kimberly White, Senior Director, Chemical Products and Technology, American 

Chemistry Council 

• The Honorable Rush Holt, CEO, American Association for the Advancement of 

Science 

• Dr. Richard Belzer, Independent Consultant 

Staff Contact 

For questions related to the hearing, please contact Majority Staff at 202-225-6371. 
4 

Chairman SMITH. The Committee on Science, Space, and Technology will come to order. Without objection, the Chair is authorized to declare recesses of the Committee at any time. 

Welcome to today’s hearing entitled ‘‘Making the EPA Great 

Again.’’ I’ll recognize myself for an opening statement and then the 

Ranking Member. 

Today, we will examine how the Environmental Protection Agency evaluates and uses science in its regulatory decision-making 

process. 

Sound science should be at the core of the EPA’s mission. Legitimate science should underlie all actions at the Agency, from research to regulations, and be an integral part of justifying their actions. Unfortunately, over the last eight years, the EPA has pursued a political agenda, not a scientific one. 

Time and again, we saw the EPA under the Obama Administration propose regulations that would have no significant impact on 

the environment. For example, the so-called Clean Power Plan, proposed by the EPA last June, set impossible targets for carbon emissions. Yet even EPA data shows that this regulation would only 

eliminate a miniscule amount of global carbon emissions and would 

reduce sea level rise by only 1/100 of an inch. In fact, the EPA has 

proposed some of the most expensive and expansive and ineffective 

regulations in history. 

The rules proposed and finalized by the EPA placed heavy burdens on American families. Often, huge costs were shouldered by 

the taxpayer with little to show for it. And the EPA routinely relied 

on questionable science based on nonpublic information that could 

not be reproduced, a basic requirement of the scientific method. 

Americans deserve to see the science for themselves. If the EPA 

had nothing to hide, why didn’t it make the scientific data it used 

for its regulations publicly available? What was the EPA hiding? 

The Committee conducted oversight of EPA’s use of suspect 

science to justify its claims. Our hearings culminated in legislation 

that required the EPA to make its data publicly available. 

With the transition to a new Administration, there is now an opportunity to right the ship at the EPA and steer the Agency in the 

right direction. The EPA should be open and accountable to the 

American people and use legitimate science. Though ignored by the 

previous Administration, the EPA does have internal processes to 

ensure this accountability. The internal review process at the EPA 

should be restored and strengthened. 

The Science Advisory Board provides critical feedback to the EPA 

on its proposals, but in recent years, SAB experts have become 

nothing more than rubberstamps who approve all of the EPA’s regulations. The EPA routinely stacks this board with friendly scientists who receive millions of dollars in grants from the federal 

government. The conflict of interest here is clear. 

Fortunately, the EPA can once again become an agency that is 

credible and respected. Simple changes, such as eliminating conflicts of interests, adding more balanced perspectives, and being 

more transparent can go a long way to restoring the Agency’s credibility. 

In recent years, the EPA has sought to regulate every facet of 

Americans’ way of life. Instead, we should invest in research and 
5 

development and let technology lead the way. Far too often, the 

EPA has deliberately used its regulatory power to undercut American industries and advance a misguided political agenda that has 

minimal environmental benefit. The new Administration has the 

opportunity to let technology and innovation protect our environment without government mandates that impose costly and unnecessary regulations on the American people. The EPA should focus 

on environmental policies that can be justified and are based on 

good science. 

Lastly, recent news stories report that another agency, NOAA, 

tried to deceive the American people by falsifying data to justify a 

partisan agenda. A senior scientist at NOAA has questioned the 

scientific integrity of a study written by Tom Karl while at NOAA 

that claimed that there was no stop in global warming from 1998 

to 2013. This official has provided evidence that Karl ‘‘had his 

thumb on the scale’’ throughout the entire process. The Karl study 

was published in Science, the journal overseen by the American Association for the Advancement of Science. 

In light of this new information, AAAS and Science should retract the Karl study. The Committee will continue our investigation 

of NOAA’s refusal to provide the Committee with responsive documents on this subject. 

It is clear that the Committee’s investigation is justified. Although NOAA, AAAS, and others attempted to block the Committee’s efforts, our goal remains to ensure that the scientific process 

funded by the American taxpayer is in fact open and honest. 

[The prepared statement of Chairman Smith follows:] 
6 

COMMITTEE ON 

SCIENCE, 

Lamar Smith. Chairman 

For Immediate Release 

February 07, 20 l 7 

& TECHNOLOGY 

Media Contact: Kristina Bourn 

(202) 225-6371 

Statement of Chairman Lamar Smith (R-Texas) 

Making EPA Great Again 

Chairman Smith: Today we will examine how the Environmental Protection Agency 

evaluates and uses science in its regulatory decision making process. 

Sound science should be at the core of the EPA's mission. Legitimate science should 

underlie all actions at the agency. from research to regulations, and be an integral 

part of justifying their actions. 

Unfortunately, over the last eight years, the EPA has pursued a political agenda, not a 

scientific one. 

Time and again, we saw the EPA under the Obama administration propose regulations 

that would have no significant impact on the environment. 

For example, the so-called Clean Power Plan, proposed by the EPA last June, set 

impossible targets for carbon emissions. 

Yet even EPA data shows that this regulation would only eliminate a miniscule amount 

of global carbon emissions and would reduce sea level rise by only 1/lOOth of an inch. 

In fact, the EPA has proposed some of the most expensive and expansive and 

ineffective regulations in history. 

The rules proposed and finalized by the EPA placed heavy burdens on American 

families. Often, huge costs were shouldered by the taxpayer with little to show for it. 

The previous EPA's regulations were all pain and no gain. 

And the EPA routinely relied on questionable science based on nonpublic information 

that could not be reproduced, a basic requirement of the scientific method. 

Americans deserve to see the science for themselves. If the EPA has nothing to hide, 

why not make the scientific data it uses for its regulations public ally available? What 

was the EPA hiding? 

This Committee conducted oversight of EPA's use of suspect science to justify its 

claims. Our hearings culminated in legislation that required the EPA to make its data 

publicly available. 
7 

This year we will pursue similar legislative remedies and hold the EPA accountable to 

the American people. 

With the transition to a new administration, there is now an opportunity to right the ship 

at the EPA and steer the agency in the right direction. 

The EPA should be open and accountable to the American people and use legit 

science. 

Though ignored by the previous administration, the EPA does have internal processes 

to ensure this accountability. The internal review process at the EPA should be restored 

and strengthened. 

The Science Advisory Board {SAB) provides critical feedback to the EPA on its 

proposals. But in recent years SAB experts have become nothing more than 

rubberstamps who approve all of the EPA's regulations. 

The EPA routinely stacks this board with friendly scientists who receive millions of dollars 

in grants from the federal government. The conflict of interest here is clear. 

Fortunately, the EPA can once again become an agency that is credible and 

respected. Simple changes, such as eliminating conflicts of interests, adding more 

balanced perspectives and being more transparent can go a long way to restoring 

the agency's credibility. 

In recent years, the EPA has sought to regulate every facet of Americans' way of life. 

Instead, we should invest in research and development and let technology lead the 

way. 

Far too often the EPA has deliberately used its regulatory power to undercut American 

industries and advance a misguided political agenda that has minimal environmental 

benefits. 

The new administration has the opportunity to let technology and innovation protect 

our environment without government mandates that impose costly and unnecessary 

regulations on the American people. 

The EPA should not pick winners and losers by regulating entire sectors of our 

economy. Instead, the EPA should focus on environmental policies that can be 

justified and are based on good science. Americans deserve nothing less. 

Lastly, recent news stories report that another agency, NOAA, has deceived the 

American people by falsifying data to justify a partisan agenda. 
8 

A senior scientist at NOAA has questioned the scientific integrity of a study written by 

Tom Kart while at NOAA that claimed that there was no stop in global warming from 

1998-2013. 

This official has provided evidence that Karl "had his thumb on the scale" throughout 

the entire process. The Karl study was published in Science, the journal overseen by 

the American Association for the Advancement of Science (AAAS), an organization 

run by Rush Holt. 

In light of this new information, it seems to me that AAAS and Science should retract 

the Karl study. The Committee will continue our investigation of NOAA's refusal to 

provide the Committee with responsive documents on this subject. 

It is clear that the Committee's investigation into this matter was justified. While NOAA 

AAAS, and others attempted to block the Committee's efforts- our goal remains to 

ensure that the scientific process funded by the American taxpayer is open and 

honest. 

### 
9 

Chairman SMITH. That concludes my opening statement, and the 

Ranking Member, Ms. Johnson, is recognized for hers. 

Ms. JOHNSON. Thank you very much, Mr. Chairman, and let me 

thank our witnesses for being here today. In particular, I’d like to 

thank our former colleague in the House, Dr. Rush Holt, a scientist, for being here to share his unique perspective. 

I would also like to welcome to the Committee our new colleagues on both sides of the aisle. I stated at our organizational 

meeting this morning it is my hope that we will be able to find 

common ground together on important issues this Congress. 

With that said, I’m disappointed but not really surprised our 

very first hearing of this Congress will be focused on attacking the 

Environmental Protection Agency, as was so often the theme of our 

hearings the last Congress. 

I would also note that, of the witnesses invited by the majority 

to testify today, we have a lobbyist for the industry, a representative from an industry trade group, and a consultant for the industry. That is not a panel likely to produce an objective examination 

of EPA’s activities. 

The efforts by some to undermine how the EPA and other federal 

agencies use science threatens our economy, threatens public 

health, threatens the environment, threatens public confidence in 

our government. This is especially true when such efforts rely on 

biased, incomplete, and misleading information, alternative facts, if 

you will, in an attempt to advance a probably false narrative 

against EPA. Regulatory activity to protect public health and the 

environment should be supported by robust analyses of the best 

available scientific evidence. That is what EPA does. Policies 

geared toward preemptively limiting scientific input into the process undermine EPA’s ability to take justifiable actions to protect 

the American public. Questioning the credibility of the scientific 

process, casting doubt on the scientific research used by EPA, or 

selectively limiting what resources—what sources of scientific information EPA may consider jeopardizes the effectiveness of the only 

government agency specifically tasked to protect human health and 

the environment. 

Simply put, limiting the science EPA uses only serves to limit 

the actions EPA may take to protect public health and the environment. I hope that my colleagues will listen today with a critical ear 

and ask themselves whether they want to support policies that will 

harm future generations instead of empowering them, remove public health safeguards instead of strengthening them, and reverse 

the progress made over the last 40 years instead of working to find 

a constructive path forward. 

Thank you, Mr. Chairman. I yield back. 

[The prepared statement of Ms. Johnson follows:] 
10 

OPENING STATEMENT 

Ranking Member Eddie Bernice Johnson (D-TX) 

House Committee on Science, Space, & Technology 

"Making EPA Great Again" 

February 7, 2017 

Thank you Mr. Chainnan, and thank you to the witnesses for being here today. In particular, I 

want to thank my former House colleague, Dr. Rush Holt, for being here to share his unique 

perspective. I would also like to welcome to the Committee our new Colleagues from both sides 

of the aisle. As I stated at our organizational meeting this morning, it is my hope that we will be 

able to find common ground work together on important issues this Congress. 

With that said, I am disappointed, but not really surprised, that our very first hearing this 

Congress will be focused on attacking the Environmental Protection Agency, as was so often the 

theme of our hearings last Congress. I would also note that, of the witnesses invited by the 

Majority to testifY today, we have a lobbyist for industry, a representative from an industry trade 

group, and a consultant for industry. That is not a panel likely to produce an objective 

examination of EPA's activities. 

The efforts by some to undermine how the EPA, and other federal agencies, use science 

threatens our economy, threatens public health, threatens the environment, and threatens public 

confidence in our government. This is especially true when such efforts rely on biased, 

incomplete, and misleading information-"alternative facts" if you will-in an attempt to 

advance a provably false narrative against the EPA. 

Regulatory activity to protect public health and the environment should be supported by a robust 

analysis of the best available scientific evidence, and that is what EPA does. Policies geared 

towards preemptively limiting scientific input into this process undermine EPA's ability to take 

justifiable actions to protect the American public. Questioning the credibility of the scientific 

process, casting doubt on the scientific research used by EPA, or selectively limiting what 

sources of scientific information EPA may consider jeopardizes the effectiveness ofthe only 

government agency specifically tasked to protect human health and the environment. 

Simply put, limiting the science EPA uses only serves to limit the actions EPA may take to 

protect public health and the environment. 

I hope that my colleagues will listen today with a critical ear, and ask themselves whether they 

want to support policies that will harm future generations instead of empowering them, remove 

public health safeguards instead of strengthening them, and reverse the progress made over the 

last 40 years, instead of working to find a constructive path forward. 

Thank you, I yield back. 
11 

Chairman SMITH. Okay. Thank you, Ms. Johnson. 

And I’ll proceed to introduce our witnesses today. Our first witness is the Hon. Jeffrey Holmstead, a partner at Bracewell. Mr. 

Holmstead is one of the country’s leading air quality lawyers and 

heads the Environment Strategies Group at Bracewell. He previously served as the Assistant Administrator at the EPA for the 

Office of Air and Radiation. He also served on the White House 

staff as Associate Counsel to former President George H.W. Bush. 

Mr. Holmstead received his bachelor’s degrees in economics and 

English from Brigham Young University and his law degree from 

Yale. 

Our second witness today is Dr. Kimberly White, Senior Director 

in the Chemical Products and Technology Division of the American 

Chemistry Council. For the past five years, Dr. White has served 

as a scientific advisor to industry for the development and execution of research to assess chemical hazards. She also has worked 

to identify emerging issues and trends in science policy and risk 

evaluation. Dr. White received her bachelor’s and master’s degrees 

in biology and a Ph.D. in environment toxicology from Texas Southern University. 

Our third witness is the Hon. Rush Holt, CEO of the American 

Association for the Advancement of Science. Dr. Holt has held positions as a teacher, scientist, administrator, and policymaker. From 

1987 to 1998 he was Assistant Director of the Princeton Plasma 

Physics Laboratory, a Department of Energy national lab. Dr. Holt 

served for 16 years as a Member of the U.S. House of Representatives representing New Jersey’s 12th Congressional District. In 

Congress, he was a member of the Natural Resources Committee 

and the Education and the Workforce Committee. Dr. Holt received 

his master’s degree and Ph.D. in physics from New York University. 

Our final witness today is Dr. Richard Belzer, independent consultant and former economist at the Office of Information and Regulatory Affairs. There, he was responsible for reviewing regulatory 

analyses prepared by the EPA, the Food and Drug Administration, 

and various components of the Departments of Agriculture, Commerce, Energy, and the Interior. Dr. Belzer has been an independent consultant in regulatory policy, economics, and risk analysis. In addition, he is the President of Regulatory Checkbook and 

the managing editor of NeutralSource.org. Dr. Belzer received his 

bachelor’s and master’s degrees in agricultural economics from the 

University of California at Davis and his master’s and Ph.D. in 

public policy from Harvard University. 

We welcome you all and appreciate your attendance today. And, 

Mr. Holmstead, we’ll begin with your testimony. 

TESTIMONY OF THE HONORABLE JEFFREY HOLMSTEAD, 

PARTNER, BRACEWELL LLP 

Mr. HOLMSTEAD. Thank you very much. Good morning. As the 

Chairman mentioned, I am a partner at the law firm of Bracewell, 

LLP, but I do want to make it clear I am not appearing on behalf 

of any clients this morning. I am here to share my own views as 

a former EPA official and as someone in private practice, who has 

spent more than 25 years working with EPA on a range of issues. 
12 

I want to start by saying I do believe that EPA plays a vital role 

in protecting human and the environment, but I also know that 

there are opportunities to improve the way the Agency operates 

and makes decisions. 

This morning, I would like to talk briefly about three reforms 

that would improve the way science is evaluated and used by the 

Agency. First, I would like to endorse this Committee’s efforts to 

ensure that the scientific and technical information that is used to 

support regulatory decisions is publicly available in a manner that 

allows for independent analysis. This is where the science reform— 

the Secret Science Reform Act can make important and meaningful 

reforms. 

The only legitimate concern that I have heard about this legislation is that in some cases data that is used to support rulemaking 

might include medical or other personal information about specific 

individuals and should not be released because of privacy concerns. 

I think this is a red herring. Certainly, no one believes that such 

information should be made public, but I cannot imagine the case 

in which personal information about any particular individual 

would be needed to support the types of regulatory actions taken 

by EPA. I do understand that documentation used in some studies 

does contain personal information, but names, addresses, and any 

other identifying information could and must be redacted before 

any such information is made public. 

EPA would certainly incur cost to review data and ensure that 

personal information is redacted before it is made public, but when 

regulations impose billions of dollars on consumers and businesses, 

it is surely appropriate for the government to spend a tiny fraction 

of this amount to ensure that the scientific information used to 

support those regulations can be publicly available. 

Second, I believe it is important to reform the Science Advisory 

Board, which is generally known as the SAB, and other advisory 

groups that provide scientific and technical advice to EPA. Such 

groups are referred to as independent advisory committees, but the 

EPA Administrator appoints the members of these groups based on 

recommendations from EPA staff. Not surprisingly, EPA staffers 

tend to recommend people who share their views about the issues 

under consideration. There’s no question that the members of the 

SAB and other advisory panels are well-qualified, but there are 

other scientists and researchers who are equally well-qualified but 

who do not get appointed because they may be more skeptical 

about EPA’s views on certain important issues. 

The SAB Reform Act would help to ensure that EPA decisions 

are influenced by experts from a variety of fields and backgrounds 

that are relevant to the issues under consideration. By focusing on 

disclosure rather than disqualification, the SAB Reform Act would 

allow for a wider range of viewpoints while ensuring that any possible conflicts, financial or otherwise, are publicly disclosed. 

Third, I would ask this Committee to continue to look at the 

need to reform EPA’s Integrated Risk Information System, which 

is generally known by the acronym of IRIS. The IRIS program evaluates specific chemicals to which the public may be exposed and 

then sets reference values that are used in a variety of EPA and 

state regulatory programs. Unfortunately, outside experts believe 
13 

that the IRIS program often overstates the actual risk posed by 

these specific chemicals. 

EPA is inclined to be conservative in making both regulatory decisions and scientific conclusion. For example, if there are three 

well-conducted—I’m sorry. If there are five well-conducted studies 

finding that a substance poses very little risk and one that finds 

a higher risk, EPA will typically place much more weight on the 

one rather than the five. Some argue that this is appropriate and 

that EPA should always err on the side of being overly protective. 

But this is an issue for policymakers, not for scientists. When decisions are made based on overly conservative science, it can mislead 

the public and impose unnecessary regulatory burdens on society. 

As this Committee considers how to improve the way that EPA 

deals with scientific and technical issues, I hope it will consider potential reforms to the IRIS program as well. 

I thank you for giving me the chance to testify this morning, and 

I look forward to answering any questions that you may have. 

[The prepared statement of Mr. Holmstead follows:] 
14 

"Making the Environmental Protection Agency Great Again" 

Testimony Before the House Science, Space and Technology Committee 

Tuesday, February 7, 20 17 - I 1 :00 am 

2318 Rayburn House Office Building 

I. Introduction 

Thank you Chairman, Ranking Member, and distinguished members of the Committee for 

inviting me to participate in today's hearing. 

My name is JeffHolmstead. I am a partner in the law firm Bracewell LLP and have been the 

head of the firm's Environmental Strategies Group (ESG) since 2006. For almost 25 years, my 

professional career has been focused on policy, regulatory, and legal issues arising under the 

Clean Air Act. From 1989 to 1993, I served in the White House Counsel's Office as Associate 

Counsel to President George H.W. Bush. In that capacity I was involved in many of the 

discussions and debates that led to the passage of the 1990 Amendments to the Clean Air Act 

and was then deeply involved in the initial efforts to implement the 1990 Amendments. From 

2001 to 2005, I was the Assistant Administrator of EPA for Air and Radiation and headed the 

EPA Office in charge of implementing the Clean Air Act. 

When not in the federal government, 1 have been an attorney in private practice, representing a 

wide variety of clients on Clean Air Act and other environmental issues. Since I joined 

Bracewell LLP in 2006, I have worked primarily with companies and trade groups in the energy 

industry. 

This hearing could not be more timely as the change in administration creates an opportune 

moment for refining the mission of EPA as it seeks to strike the right balance between the costs 

and benefits of environmental regulations. I have spent the last 25 years of my professional life 

working on EPA issues, and I can say with confidence that, if we focus on sound science and 

good regulatory design, we could have the environmental protection we all want at a much lower 

cost than we have today. 

That is why I want to thank the Committee for hosting today' s hearing, which I think will shed 

light on changes that can strengthen the work being done at EPA. To this end, I commend this 

Committee and its staff for considering the "Secret Science Reform Act" and the "EPA Science 

Advisory Board Refonn Act." I hope that this hearing will push both bills a few steps closer to 

enactment. 

II. The Secret Science Reform Act 

No matter one's political views, it is hard to disagree that transparency is an important principle 

when it comes to the development of public policy and regulations. Transparency not only stays 

true to our collective democratic ideals but also helps to ensure that well-informed debate occurs 

before new policies are made or new regulations are promulgated. This is where the Secret 

Science Reform Act can make important and meaningful reforms. 

#5396287.2 
15 

How can EPA be sure that it is relying upon the best available science when the scientific and 

technical information used to support its actions cannot be identified and made available to the 

public? Only when such information is made public can other interested and qualified parties 

conduct independent analysis and seek to reproduce research results. Transparency not only 

breeds accountability but also a healthy respect for dialogue and honest debate. 

I don't think anyone can object to the basic premise that scientific infonnation used to support 

regulatory actions should be made public. Former President Obama's memorandum on scientific 

integrity stated that "there should be transparency in the preparation, identification, and use of 

scientific and technological information in policymaking." 1 Furthermore, a White House Office 

of Science and Technology Policy memo, also from the Obama Administration, explains that 

agencies should expand and promote access to scientific information by making it available 

online in open formats. 2 The Secret Science Reform Act would overcome bureaucratic hurdles 

that stand in the way of these principles. 

As far as I know, the only legitimate concern that has been raised about this legislation is that, in 

some cases, data that has been used to support rulemaking might include medical or other 

personal information about specific individuals and should not be released because of privacy 

concerns. Certainly, no one believes that it is appropriate for anyone much less a federal 

agency- to publicly release such information. But I cannot imagine a case in which personal 

information about any particular individual or individuals would be needed to support the types 

of regulatory actions and policies decisions made by EPA. I do understand that documentation 

used in some studies docs contain personal information about some individuals. But names or 

other identifying information could certainly be redacted before any such information is made 

public. 

Admittedly, EPA would incur costs to review certain data and ensure than personal information 

is redacted before it is made public. But when regulations impose billions of dollars on 

consumers and businesses, it is surely appropriate for the government to spend a tiny fraction of 

this amount to ensure that the scientific information used to support those regulations can be 

made public. 

It is worth noting that EPA itself has recognized that "scientific research and analysis comprise 

the foundation of all major EPA policy decisions" and that "the Agency should maintain 

vigilance toward ensuring that scientific research and results are presented openly and with 

integrity, accuracy, timeliness, and the full public scrutiny demanded when developing sound, 

high-quality environmental science." 3 These are laudable goals, and the Secret Science Reform 

Act will ensure that EPA actually lives up to them when it relies on such information to support 

regulatory decisions. 

1 See: https://www.gpo.gov/fdsys/pkg/FR-2009-03-II/pdf/E9-5443.pdf 

2 See: http://www .sciencemag.org/news/20 I 0/ 12/white-house-releases-long -awaited-guidance-scientific-integrity 3 See: http://ethics.iit.edu/ecodes/node/5537 

-2- #5396287.2 
16 

III. Science Advisory Board Reform Act of 2015 

The Science Advisory Board (SAB) Reform Act would also make sensible reforms and increase 

the likelihood that EPA's regulatory decisions will not only be based on the best data but will be 

informed by the best possible analysis and interpretation of that data. EPA's Science Advisory 

Board (SAB) and subsidiary groups like the Clean Air Science Advisory Committee (CASAC) 

often advise EPA on scientific issues that are important for the Agency's regulatory actions. 

Such groups are generally known as independent advisory committees, but the EPA 

Administrator appoints the members of these groups based on recommendations from EPA 

staff. Not surprisingly, EPA tends to choose people who share EPA's views about the 

importance of environmental issues. The members of the SAB and other subsidiary groups are 

well-qualified and have good credentials, but there are other scientists and researchers who are 

equally well qualified but do not get appointed because they are more skeptical about EPA's 

views on certain important issues. 

The SAB Reform Act would help to ensure that EPA decisions are informed by experts from a 

variety of fields and backgrounds that are relevant to the issues under consideration. Throughout 

my professional career, I have seen how serious dialogue among thoughtful people with different 

perspectives can be used to inform both policy and science. By focusing on disclosure rather 

than disqualification, the SAB Reform Act would allow for a wider range of viewpoints while 

ensuring that any possible conflicts-- financial or otherwise are publicly disclosed. 

By requiring EPA to make public a list of nominees to the Board and accept public comments on 

the nominees, the Act comports with the maxims of transparency outlined above. And by 

instituting a requirement to balance scientific and technical points of view, the Act helps to 

ensure that the SAB provides the best advice and insights to EPA as it crafts regulations. 

IV. EPA's Integrated Risk Information System (IRIS) 

Another EPA program that I believe should receive scrutiny from the Committee is EPA's 

Integrated Risk Information System ("IRIS"). The IRIS program, which is located within EPA's 

National Center for Environmental Assessment ("NCEA"), endeavors to "develop impartial 

toxicity information independent of its use by EPA's program and regional offices to set national 

standards and clean up hazardous sites."4 While the IRIS program as a whole is laudable, outside 

experts believe that it often overstates the actual risk posed by specific chemicals. 

EPA is inclined to be "conservative" in making both regulatory decisions and scientific 

conclusions. For example, if there are 5 studies finding that a substance poses very little risk and 

one that finds a higher risk, EPA will typically place more weight on the one rather than the five. 

Some argue that this is appropriate, and the EPA should always err on the side of being overly 

protective, but this is a decision for policymakers, not for scientists. And when regulatory 

decisions are made based on overly conservative science, it can have serious effects on the 

regulated community, sometimes even threatening the viability of industrial facilities that 

provide important benefits to local workers and communities. 

4 See: https://www.epa.gov/iris/basic-information-about-integrated~risk-information-system 

-3- #5396287.2 
17 

The IRIS Program sometimes relies on very small and statistically limited subgroups to reach 

conclusions regarding risk of specific chemicals. In some cases, it has even relied on outdatedand poor quality- Russian and Chinese data instead of domestic data relied upon by experts 

throughout various domestic industries, simply because the foreign studies found risks that the 

widely used U.S. studies did not. And then, in establishing reference values (or "safe" levels of 

exposure), the IRIS program relies on additional conservative assumptions. 

As a result of these things, NCEA sometimes misleads the public and stokes unnecessary fears, 

causing serious real-world consequences for facilities that already are struggling to keep their 

doors open-and their jobs in the United States. The President and many other officials from 

both political policies have stressed the importance of keeping manufacturing capacity in the 

United States, but the IRIS program can, in some cases, imperil this important goal. Accordingly, 

as the Committee contemplates how best to craft transparent, fair, and predictable regulatory 

processes based upon sound scientific information, the IRIS program, in my view, should be an 

important part ofthat broader conversation. 

V. Conclusion 

Again, I very much appreciate the opportunity to appear before the Committee and hope my 

testimony will be helpful to you as you seek to shape the strategic direction at EPA. I commend 

the Committee for its work so far and I respectfully offer my input, as necessary, to you going 

forward. Thank you and I look forward to answering any questions that the Committee may 

have. 

### 

-4- #5396287.2 
18 

JeffHolmstead is a partner and Head of the Environmental Strategies Group at 

Bracewell LLP and has repeatedly been recognized by Chambers USA as one of 

the leading environmental lawyers in the country. He has worked on environmental 

and regulatory issues for more than 25 years in both the federal government and 

the private sector. From 1989 to 1993, he worked on the White House staff of 

President George H.W. Bush, where he was deeply involved in the passage and 

then the early implementation of the 1990 Amendments to the Clean Air Act. In 

early 2001, he was nominated by President George W. Bush and confirmed by the 

Senate to be the Assistant Administrator of EPA for Air and Radiation a position 

he held until late 2005. 

When not in the government, he has been a lawyer in private practice advising and 

representing clients on a range of environmental issues. Since he joined Bracewell 

in 2006, most of his work has been with energy companies and trade associations, 

including utilities, upstream oil and gas companies, and petroleum refineries. He 

represents a number of clients dealing with the development, implementation, and 

enforcement of EPA regulations. Much of work over the last few years has been 

focused on EPA's efforts to regulate greenhouse gas emissions under the Clean Air 

Act. 

Mr. Holmstead graduated from Yale Law School in 1987 and then served as a law 

clerk to Judge Douglas H. Ginsburg on the United States Court of Appeals for the 

DC Circuit. He graduated summa cum laude from Brigham Young University in 

1984 with degrees in Economics and English. He and his wife are both from 

Colorado and are the proud parents of four children. 
19 

Chairman SMITH. Thank you, Mr. Holmstead. 

Dr. White. 

TESTIMONY OF DR. KIMBERLY WHITE, 

SENIOR DIRECTOR, 

CHEMICAL PRODUCTS AND TECHNOLOGY, 

AMERICAN CHEMISTRY COUNCIL 

Dr. WHITE. Good morning, Chairman Smith, and Members of the 

Committee. My name is Dr. Kimberly White, a scientist with the 

American Chemistry Council. And I appreciate this opportunity to 

testify regarding EPA’s use of science in its regulatory decisionmaking process. 

The business of chemistry is a critical component for manufacturing safe, high-quality products. ACC member companies and the 

public rely on science to spur innovation, advance product stewardship, and improve the assessments of chemical risk. Similarly, they 

expect high-quality science and objective assessment processes to 

underpin regulatory decisions by the federal government. 

ACC has long maintained that EPA chemical assessments can 

and should reflect the most up-to-date and relevant science regarding potential impacts to human health and the environment from 

chemical exposures. Although EPA has made efforts to improve its 

scientific approach, the actual implementation has been slow and 

often lacking. This has been fundamentally due to the lack of a 

consistent science-based framework for conducting chemical evaluations. In 2016, Congress passed the Lautenberg Chemical Safety 

Act or the LCSA, which provides EPA a mandate for use of the 

best-available science and a weight-of-evidence approach in conducting risk evaluations for both new and existing chemicals. Implementing these new provisions under the LCSA will require significant changes to EPA’s scientific evaluation procedures. 

Unfortunately, as indicated in EPA’s proposed framework rule 

for risk evaluation, EPA believes existing practices meet the standards of the LCSA. ACC does not support this belief and will continue fostering approaches to advance the technical quality and objectivity of scientific evaluations, particularly by promoting more 

transparency in both what science is being considered and how 

that information is interpreted. 

My oral testimony today focuses on four areas to improve science 

evaluations at EPA, and my written testimony provides additional 

detail and some specific examples. 

First, EPA should ensure that its chemical assessments address 

information needs of decision-makers and are fit for purpose. This 

includes clearly defining the scope of the evaluation, the methods 

to be used, and the utility of the evaluation for regulatory decisions. EPA is currently interpreting the LCSA as requiring the 

Agency to evaluate all uses of a chemical. However, the LCSA 

clearly indicates that EPA has discretion to select those conditions 

of use for its scope of their risk evaluation. A clear scoping step allows the Agency to determine if a screening-level assessment identifies risk sufficiently or if a more refined risk evaluation is needed. 

This approach also enables EPA to meet the stringent deadlines of 

the LCSA and focus resources on those conditions of use where unreasonable risk cannot be ruled out. 
20 

Secondly, scientific evaluations must utilize transparent and consistent criteria for selecting the most relevant, high-quality science 

and evaluating the evidence to draw conclusions. It is critical that 

EPA rely on studies of the highest quality, not simply those studies 

that produce the lowest points of departure or the highest exposure 

estimates. 

Thirdly, EPA should employ a transparent weight-of-evidence 

framework that integrates evidence from human studies, animal 

research, and mechanistic data. The LCSA requires risk evaluations to integrate and assess available information on hazard and 

exposures for the conditions of use. The LCSA also requires EPA 

to make decisions using a weight-of-evidence approach. 

Lastly, EPA must implement an effective peer-review process. 

Peer review should be independent and objective, allowing for robust engagement with stakeholders to provide a thorough review. 

It should also include a quality-assurance process that explicitly 

evaluates whether peer-review recommendations and public comments were completely and adequately addressed. 

In conclusion, ensuring that up-to-date, high-quality science 

underlies EPA decision-making is critical to protecting human 

health and the environment. This can be achieved by consistent applications of processes throughout EPA to conduct risk evaluations 

using a weight-of-evidence process, as required by the LCSA. 

Thank you for this opportunity to provide testimony, and I look 

forward to working with members of the Committee to ensure that 

high-quality science is the foundation for the Agency’s decisionmaking. 

[The prepared statement of Dr. White follows:] 
21 

merican· 

Chemistry 

ouncil 

Testimony of 

Kimberly W. White, Ph.D. 

Senior Director, Chemical Products and Technology Division 

American Chemistry Council 

Making EPA Great Again 

House Committee on Science, Space, and Technology 

February 7, 2017 

americanchemistry.com® 700 Second St., NE I Washington, DC 20002 I (202) 249,7000 
22 

Summary 

The American Chemistry Council (ACC) 1 appreciates this opportunity to provide testimony on 

the U.S. Environmental Protection Agency's (EPA) process for evaluating and using science to 

support regulatory decision making. 

The business of chemistry is a critical component for manufacturing safe, high quality products 

and ACC member companies rely on science to conduct the research necessary to discover new 

chemistries and identify new applications of existing chemistries. They also rely on science to 

develop new tools for assessing the potential hazards, exposures and risks of chemical 

substances. Similarly, they expect high quality, up to date science and relevant reliable 

assessment processes to underpin regulatory decisions by the Federal government. Reliance on 

the highest quality, best available science is critical to ensuring public trust. Without it, 

consumers are at a severe disservice and lose confidence in regulatory decision making, leading 

to product de-selection that is not supported by science, unwarranted public alarm and 

unnecessary costs. ACC supports actions to enhance the integration of the best available 

scientific knowledge and methods as the foundation for regulatory decision making across EPA. 

We also support advancing the technical quality and objectivity of EPA evaluations, particularly 

through enhancing transparency in both what science is being considered and how it is being 

interpreted and integrated. 

Over the last 30 years, advances in scientific techniques and knowledge have improved our 

understanding of how chemicals interact with the human body and the environment. Research 

programs within industry, academia and government have expanded to investigate the underlying 

biological processes for chemical interactions and to improve the scientific basis of chemical 

policies and product stewardship efforts. It is simply not enough to have the science available for 

use. There must also be a transparent process and a willingness to enable integration of the 

science into EPA policies and practices. Current processes for evaluating scientific infonnation 

and conducting chemical assessments at EPA are not always based on transparent, objective or 

consistent use of the best available science. In recent years, there has been a focus on EPA's 

Integrated Risk Infonnation System (IRIS) program and addressing deficiencies identified by the 

National Academy of Sciences (NASl These deficiencies arc also evident in other EPA 

chemical assessment programs. 

1 The American Chemistry Council (ACC) represents the leading companies engaged in the business of chemistry. 

ACC members apply the science of chemistry to make innovative products and services that make people's lives 

better, healthier and safer. ACC is committed to improved environmental, health and safety perfommnce through 

Responsible Care®, common sense advocacy designed to address major public policy issues, and health and 

environmental research and product testing. The business of chemistry is a $797 billion enterprise and a key element 

of the nation's economy. It is one of the nation's largest exporters, accounting for fourteen percent of all U.S. 

exports. Chemistry companies are among the largest investors in research and development. 2 National Academy of Sciences (NAS). NRC (National Research Council). Review of EPA's Integrated Risk 

Information System (IRIS) Process. Board on Environmental Studies and Toxicology. Division on Earth and Life 

Studies. Washington, DC: The National Academies Press, 2014. Available at 

http://www.nap.edu/catalog.php?record id=\8764. 

2 
23 

ACC has consistently called upon EPA to improve the design and conduct of its chemical 

assessments. Recommended improvements have included adoption of consistent and transparent 

study evaluation methods to determine the quality and reliability of critical studies. We have also 

encouraged EPA to utilize an improved framework for integrating study results based on a 

weight of the scientific evidence approach that incorporates modem knowledge of mode of 

action to establish cause and effect. Furthermore, we have recommended that EPA improve its 

peer review and accountability practices for addressing both public comments and peer review 

recommendations. Although EPA has made efforts to identify practices for systematically 

reviewing the available science and to strengthen its peer review processes, the actual 

implementation of these practices has been slow and often lacking. This has been fundamentally 

due to the lack of a consistent science-based framework for conducting chemical evaluations 

within EPA. 

Over the past several years Congress has worked to update and reform the Toxic Substances 

Control Act (TSCA) and in 2016 passed the Frank R. Lautenberg Chemical Safety for the 21st 

Century Act (LCSA). The LCSA establishes new requirements for the review of new and 

existing chemicals manufactured and used to make U.S. products; including requiring use of the 

best available science and a weight of evidence process to evaluate scientific information. EPA 

now has a mandate to apply high quality, reliable scientific information while evaluating new 

chemicals and prioritizing and evaluating the risks of existing chemicals. Implementing these 

new provisions will require significant changes to EPA's scientific evaluation procedures, 

particularly for existing chemicals. However, as we have recently seen in EPA's proposed 

framework rule for risk evaluation, EPA believes that existing practices meet the standards of the 

LCSA. ACC does not support this belief and we plan to continue to be a constructive partner to 

both Congress and EPA in identifying approaches that enhance the chemical assessment process. 

ACC's testimony today outlines four areas that can improve the evaluation of scientific 

infonnation at EPA: 

Clear framework for conducting the chemical assessment; 

Application of consistent criteria for selecting and evaluating scientific data; 

Transparent and objective integration of scientific evidence; and 

• Independent and robust peer review. 

I. Clear Framework for Conducting Chemical Assessment 

EPA and other federal government agencies conduct chemical assessments to inform risk 

management decisions. As such, EPA should ensure that the assessments it conducts will 

address the infonnation needs of decision makers. EPA is tasked with evaluating new 

chemicals to be manufactured and used to make U.S. products and existing chemicals in the 

marketplace. As such, any assessment that EPA undertakes should be fit for purpose in order 

to effectively and efficiently utilize its limited resources. This can help ensure that chemical 

assessments are based on the best available information and are appropriately scaled and 

oriented to address relevant questions regarding risk. EPA should also make use of all 

available science evaluations, including primary scientific literature, grey literatures and 

reviews, conducted to inform the chemical assessment process and determine information 

needs. Jn this initial phase of chemical assessment, EPA can determine if a screening level 

3 
24 

assessment will identify and assess risk sufficiently or if a more refined risk evaluation is 

needed. 

EPA is currently interpreting the LCSA as requiring the Agency to evaluate all conditions of 

use of a chemical, regardless of how small, in the risk evaluation. This interpretation is 

unreasonable and inconsistent with other provisions in the LCSA which, clearly indicate that 

EPA has discretion to select the conditions of use that it will consider in the scope of a risk 

evaluation. There are significant questions about EPA's ability to meet the stringent 

evaluation deadlines of the LCSA if the Agency takes the position that all uses of a substance 

must be evaluated. A tiered approach, where EPA uses the scoping step to conduct a 

quantitative screening level analysis, will allow EPA to focus its limited resources on more 

robust refined risk evaluations for only those conditions of use where unreasonable risks 

cannot be ruled out. 

In order to adequately and effectively evaluate the available science to make timely and 

science based decisions regarding potential risk from exposures, methods for conducting a 

chemical assessment must be clearly defined up front. The protocol, developed before the 

chemical evaluation begins, defines the methodologies that will be used in the assessment. It 

is made publicly available before the assessment begins and becomes a living document that 

can be commented upon and modified as needed. The protocol includes: a clear testable 

question/hypothesis, the planned search strategy (including criteria for inclusion and 

exclusion of studies), the criteria that will be used for study quality and risk of bias 

evaluations (including, for example, consideration of study design and confounders), the plan 

for integrating/synthesizing scientific evidence using a weight of evidence approach, the plan 

for quantifying and presenting findings, and the plan for peer review of the assessment. 

Section 6(b)(4)(B) of the LCSA requires EPA to establish, by rule, "a process to conduct risk 

evaluations." Incorporation of a protocol which includes these important risk evaluation 

elements is missing from EPA's proposed rule for risk evaluation. Without these elements it 

is not clear how EPA can meet the LCSA requirements that, for the first time in federal law, 

provide a statutory requirement mandating best available science and weight of the scientific 

evidence requirements to inform agency decision making. 

In EPA's IRIS program there are similar concerns regarding scientific evaluation procedures. 

These concerns have been well articulated by the NAS. Assessment approaches also appear 

to be hampered by a lack of coordination among programs regarding the chemical 

assessments being undertaken and how those assessments can be utilized by other EPA 

programs. For instance, past assessments by EPA's IRIS program (e.g., n-butanol, 

trimethylbenzenes) did not seem to consider data developed by other EPA program offices in 

previous chemical assessments. It also has not been clear why the TSCA Work Plan 

chemicals program, within the Office of Pollution Prevention and Toxics (OPPT), at times 

evaluated the same chemicals that the IRIS program evaluated. 

ACC recommends that EPA identify a consistent framework for conducting chemical 

assessments, including the methods to be utilized in the assessment and the utility of the 

assessment for regulatory decision making, prior to initiating the assessment These practices 

should be consistent with requirements outlined in section 26 of the LCSA which, requires 

4 
25 

EPA to improve the quality, transparency and relevance of the scientific information, 

approaches, methods, protocols, and models that are used to evaluate chemical risks. EPA 

must additionally ensure that the information used is reasonable for and consistent with the 

intended use of the information.3 When assessments are being conducted to inform 

significant rulemakings, EPA must make certain that these important standards are being 

met. 

II. Application of Consistent Criteria for Selecting and Evaluating Scientific Data 

Scientific evaluations must utilize transparent and consistent criteria for selecting the most 

relevant scientific information and evaluating the evidence to draw scientifically defensible 

conclusions to support decision making. In particular, a systematic approach can ensure that 

EPA is using clear procedures and protocols to develop reproducible and scientifically sound 

assessments. It is critical that EPA rely on the studies of the highest quality not simply those 

studies that produce the lowest points of departure or the highest exposure estimates. A lack 

of sufficient review of study information may lead to establishing unrealistic risk 

characterizations and exposure concentrations that are not relevant to actual human 

exposures. For example, in the Work Plan chemical draft risk assessment of 1-

bromopropane, EPA did not provide information regarding the quality of the individual 

studies. Appendix M of the assessment identifies some quality considerations, but EPA did 

not provide any information regarding its own findings from its quality review of the 

individual studies. Additionally, no information was provided to describe how considerations 

were applied and what constitutes a study of "high quality" or "good quality."4 Simply 

choosing the lowest value is not consistent with the best available science approach 

envisioned under the LCSA. As noted before, this new language will require that EPA make 

significant changes to its risk evaluation practices. 

Given the lack of consistency in evaluating scientific information, EPA should develop, 

through an open and transparent process, (I) protocols that define the process for the 

acquisition of the scientific literature including study inclusion/exclusion criteria; and (2) a 

framework for evaluating studies for quality, reliability and relevance. Notably, the LCSA 

calls on the EPA risk evaluation process to comply with the best available science provision 

in Section 26(h), the weight of the scientific evidence provision in 26(i) and the transparency 

provision in 260). 

III. Transparent and Objective Integration of Scientific Evidence 

Considerable progress has been made over the years to improve understanding of the 

potential for risk from chemical exposures. In order for the Agency to reach scientifically 

robust conclusions, it must employ a transparent weight of evidence framework that 

3 Section 26(h)(l) states that the Administrator shall consider "the extent to which the scientific information, 

technical procedures, measures, methods, protocols, methodologies, or models employed to generate the information 

are reasonable for and consistent with the intended use of the information." 4 See Comments of the American Chemistry Council on the TSCA Work Plan Chemical Draft Risk Assessment of 

1-Bromopropane, Docket No. EPA-HQ-OPPT-2015-0084, May 9, 2016 

5 
26 

integrates evidence from human epidemiological studies, laboratory animal research and 

mechanistic research. This includes evaluating the strengths and limitations of the human and 

animal data, understanding the biological significance of responses in animal models and of 

mechanistic research, and applying current scientific knowledge to extrapolate those findings 

to humans. 

EPA's 2005 Guidelines for Carcinogen Risk Assessment5 emphasize the importance of 

weighing all of the evidence--including both studies that provide evidence of an effect as 

well as those that provide no evidence of an effect-in reaching conclusions about the 

potential for a chemical to be carcinogenic to humans. The weighing of the evidence includes 

addressing not only the likelihood of human carcinogenicity, but also the conditions under 

which such effects might occur. Weighing the scientific evidence entails clearly explaining 

the kinds of evidence available (e.g., epidemiology, toxicology, mechanistic) and how that 

evidence fits together in drawing conclusions regarding human relevance and dose-response. 

Section 6(b)(4)(F)(i) of the LCSA requires risk evaluations to integrate and assess available 

infonnation on hazards and exposures for the conditions of use of the chemical substance. 

Additionally, Section 26(i) of the LCSA requires EPA to make decisions using a weight of 

scientific evidence approach. The Congressional Record clearly describes how a weight of 

the scientific evidence approach requires the consideration of the strengths, limitations and 

relevance of each study. 6 

Unfortunately, it has been unclear how the EPA programs apply weight of evidence 

approaches or how the programs incorporate mode of action information when evaluating the 

science to reach decisions. There also appears a lack of acknowledgement in some EPA 

programs regarding science that supports a threshold for safe exposures to a chemical. In 

2011, the NAS7 reviewed the draft formaldehyde IRIS assessment and concluded that EPA 

had not sufficiently documented methods to identify or evaluate relevant scientific studies; 

and had not adequately integrated the lines of evidence from the available animal, human, 

and mechanistic data. The NAS report also called the draft formaldehyde assessment 

subjective and potentially problematic given the inconsistencies in the available scientific 

data. The NAS also noted areas where EPA's approaches may not be scientifically justified. 

For example, the NAS review noted that with regard to the biologically based dose-response 

(BBDR) model manipulations made by the EPA " ... some of the manipulations are extreme, 

may not be scientifically justified, and should not have been used as the basis of rejection of 

the use of the BBDR model in its assessment. Model manipulations that yield results that are 

implausible or inconsistent with available data should be rejected as a basis for judging the 

uti! ity of the model." 

5 EPA 2005. Guidelines for Carcinogen Risk Assessment. Available at 

http://www.epa.gov/sites/productionlfiles/20 13-09/ documents/cancer guidelines final 3-25-05 .pdf. 6 See Senate Congressional Record, June 7, 2016 at page S3518, available at: 

https://www.congress.gov/crec/2016/06/07/CREC-2016-06-07-ptl-PgS35ll.pdf. 

7 National Academy of Sciences (NAS). National Research Council (NRC). Review of the Environmental 

Protection Agency's Draft IRIS Assessment of Formaldehyde. Washington, DC: The National Academies Press, 

2011. 

6 
27 

In addition to identifying scientific concerns with the formaldehyde IRIS assessment the 

NAS also identified recurrent problems with EPA's process for evaluating chemicals more 

broadly. While the EPA is working to address these NAS recommendations, after more than 

5 years, the IRIS program is still falling short and has not yet released a final assessment that 

is fully consistent with these important NAS recommendations. In addition to the IRIS 

program, and more recently, in a 2016 Work Plan chemical review of 1-bromopropane, EPA 

had multiple studies for identified hazards, such as reproductive and developmental toxicity, 

and carcinogenicity. EPA also had multiple exposure studies to consider. However, the 

Agency failed to apply a weight of evidence approach. 

When there are multiple studies available, the only scientifically defensible approach is to 

weigh the studies by considering study characteristics and determining which studies are of 

higher quality and should be given greater weight in the assessment. Failure to employ a 

weight of evidence approach is a critical deficiency that seriously limits any conclusions that 

can be drawn. To ensure clarity and consistency in the scientific evaluation process, EPA 

should (I) develop a clear weight of evidence framework to identify and evaluate each 

stream of evidence, including strengths, limitations, and relevance of each study; and (2) 

integrate evidence based upon strengths, limitations, and relevance. This approach should be 

implemented in all programs and codified in EPA's risk evaluation framework regulations 

under the LCSA. 

IV. Independent and Robust Peer Review 

Peer review is essential in the evaluation of scientific information to ensure the development 

of scientifically defensible assessments. It also allows for the review of the underlying 

assumptions, methodology, criteria, and conclusions reached in the evaluation. EPA 

currently has several mechanisms to conduct peer review of scientific information including 

the Science Advisory Board, Science Advisory Panel, NAS contracted review and ad hoc 

peer review. As outlined in EPA's 2015 Peer Review Handbook,8 "the success and 

usefulness of any peer review depends on the quality of the draft work product submitted for 

peer review, the care given to the statement of the issues or "charge," the match between the 

peer review draft product and the form of peer review, the match between the peer review 

draft product and the scientific/technical expertise of the reviewers, and Agency use of peer 

review comments in the final product." 

Unfortunately, peer review processes and approaches are inconsistently applied throughout 

the Agency, including the selection of peer review panel members and the consideration 

given to public and peer review comments. During some EPA peer review meetings, the peer 

reviewers have appeared to be overly deferential to EPA and reluctant to be seen as 

criticizing EPA staff. We have also seen situations where peer reviewers have suggested 

discounting a study solely based on the funding source, without any considerations being 

given to the quality of the study. Also, EPA staff often comment throughout peer review 

meetings, essentially participating as peers, while industry experts are typically excluded 

8 EPA Peer Review Handbook 4th Edition, 20 I 5. Available at https://www.epa.gov/sites/production/files/20 16-

03/documents/epa peer review handbook 4th edition.pdf 

7 
28 

from the dialogue. This practice undermines the integrity of the reviewers' role as 

independent and external to the assessment itself. 

A critical element of peer review is also the consideration of public comments. The public 

plays an important role in the review process by helping identify key scientific information 

and potential concerns with the assessment being evaluated. Currently, there is no robust 

consideration of public comments in the peer review process. Reviewers on the EPA Science 

Advisory Board (SAB) arc not given clear advice regarding what it means to "consider" 

public comments. In fact we have seen SAB chairs ignore public input because they are not 

required to address it. When this has occurred, SAB staff have not clarified to the peer 

reviewers that they can and should respond to public input. 

In 2013, EPA's IRIS program announced a revised process that included an explicit response 

to comments step. However, 2016 IRIS assessments oftrimethylbenzenes and ammonia and 

the 2017 ethylene oxide assessment contained no response to public comments in the final 

documents and only addressed peer review comments. This is a clear departure from EPA's 

commitment in step 5 of its IRIS process which states that the Agency "Develops a 

disposition of peer review and public comments and provides these as an appendix to the 

IRIS assessment."9 Compounding concerns, the SAB committee reviewing the 

trimethylbezene assessment also did not respond to public comments, essentially creating a 

black hole where public comments are provided to the Agency but no clear responses arc 

provided. Peer review should be independent and objective allowing for robust engagement 

with stakeholders to provide a thorough review. It should also include a quality assurance 

process that explicitly evaluates whether the peer review recommendations and public 

comments were completely and adequately addressed. 

Conclusion 

The incorporation of up to date scientific information, approaches and methods to ensure that 

EPA decision making is firmly based on high quality science is critical to protecting human 

health and the environment. This can be achieved by transparent, objective and consistent 

application of evaluation processes throughout EPA to evaluate and integrate scientific 

information utilizing a weight of scientific evidence process as required under the LCSA. 

Further, a robust and independent peer review process must be employed. ACC looks forward to 

working with members of the Committee to enhance the approaches to ensure that high quality 

science is the foundation to regulatory decision making regarding potential chemical hazards and 

risks. 

9 EPA IRIS Process Flow Chart. Available at https://www.epa.gov/sites/production/files/2014-

03/documents/iris process flow chart.pdf. 

8 
29 

Kimberly W. White is a Senior Director in the American Chemistry Council's Chemical Products and 

Technology Division where she works in support of scientific research and chemical assessments that are 

firmly based on up-to-date scientific knowledge, meet the highest standards of scientific inquiry and are 

evaluated in accordance with the most relevant scientific approaches. For the past 5 years, Dr. White has 

served as a scientific advisor to industry partners for the development and execution of scientific research 

to inform chemical hazard assessments. She has also worked to identify emerging issues and trends in 

science policy and risk evaluation. Dr. White has presented at scientific symposia; collaborated to 

organize multi-stakeholder workshops to improve the conduct of chemical assessments; and managed 

scientific research programs. Additionally, Dr. White has coauthored publications on weight of evidence 

frameworks, problem formulation in chemical assessment and understanding potency information 

associated with human exposures. She has also been the lead representative in discussions with 

regulatory and chemical assessment agencies. In her most recent past position, Dr. White served as a 

Scientific Advisor with the American Petroleum Institute where she managed toxicology research, 

regulatory response, and product stewardship efforts for the oil and natural gas industry. She has also held 

positions as an Environmental Manager for Doar's Head Provisions Co., Inc. and as an Environmental 

Scientist for Resource Management Concepts, where she managed environmental compliance and 

sustainability efforts. Dr. White possesses D.S. and M.S. degrees in biology and a Ph. D in Environmental 

Toxicology. 
30 

Chairman SMITH. Thank you, Dr. White. 

And, Dr. Holt. 

TESTIMONY OF THE HONORABLE RUSH HOLT, CEO, 

AMERICAN ASSOCIATION FOR 

THE ADVANCEMENT OF SCIENCE 

Dr. HOLT. Good morning, Chairman Smith and Ranking Member 

Johnson and esteemed Members of the Committee. Thanks for the 

opportunity to testify today on behalf of the American Association 

for the Advancement of Science, the AAAS. AAAS is the largest 

general science membership society, publisher of the Science family 

of journals, and our mission is simple: to advance science, engineering, and innovation throughout the world for the benefit of all people. 

The hearing today, as I understand it, is general and not about 

specific bills, so I will talk in general about the use of good science 

as a basis for policy and regulation. 

I’m pleased to note from the title of today’s hearing that the 

Committee acknowledges that the EPA has been great. The success 

of EPA is really because the environment regulations of past decades have been based—such as the Clean Air Act—have been 

based on science. And that sets this EPA apart from a number of 

other regulatory agencies. It has worked. 

I want to state from the outset that I don’t want my presence 

here to be construed as advocating for a specific environment legislation. I want to talk about science and the process by which 

science is conducted. It must be recognized as the most reliable 

pathway to knowledge and the best basis for making public policy 

and regulations. Science is not a political construct or a belief system. It provides testable, fundamental knowledge of the world and 

how things work. It’s a set of principles dedicated to discovery and 

the use of evidence to continually test those discoveries. 

And although science gets a great deal of credit for advancing 

our understanding of the world, I think it is less understood for its 

foundational quality, humility in the face of evidence, and over 

time, when one’s cherished beliefs and partisan ideologies and 

wishful thinking have turned out to be wanting, the scientific evidence is most likely to remain. We need more reverence for evidence in our policymaking. Without reverence for evidence and by 

extension evidence-based policymaking, our country’s future is compromised. 

Science is not static. That’s why the process of science can converge on reliable knowledge. Science does not deal in cut-and-dried 

facts, ever-immutable. Sometimes, we’ll see the science push aside 

an understanding for a better, more verifiable understanding. 

That’s the job of scientists through the scientific process, not the 

job of politicians second-guessing the scientific process. 

However attractive any of us may find our own belief at any 

time, one’s odds of success are better if one goes with the scientifically established thinking. Scientific progress depends on openness, transparency, the free flow of ideas and people. These are 

principles that have helped the United States attract and benefit 
31 

from science talent, from the Apollo program and exploring the far 

reaches of the universe, to advancing biomedical research for curing disease, to harnessing science to build a thriving high-tech sector. The United States has been a leader in science, in education, 

and in innovation. And the principles are the same principles that 

have allowed EPA to base their regulations on science. 

Furthermore, scientists, whether industry, academia, or government, must have confidence that they can conduct their work in an 

atmosphere free of intimidation and undue influence. Policymakers 

should never dictate the conclusions of a scientific study, and they 

should base policy on a review of relevant research and provisions 

of relevant studies. In other words, the integrity of the process 

must be upheld. 

During the Bush and Obama Administrations, federal agencies 

worked to develop and implement scientific integrity and access to 

date policies. This bipartisan recognition of strengthening the scientific integrity in federal agencies lays a good foundation that 

should not be weakened. In other words, I’m here to say don’t try 

to reform the scientific process. It has served us well and will serve 

us well. 

I thank you again for the invitation to testify, and I look forward 

to working with you in this Congress. 

[The prepared statement of Dr. Holt follows:] 
32 

Written Testimony 

Before the 

Committee on Science, Space and Technology 

by 

Rush Holt, Ph.D. 

Chief Executive Officer 

American Association for the Advancement of Science 

Executive Publisher, Science 

February 7, 20 J 7 

Good Morning Chairman Smith, Ranking Member Johnson, and fellow colleagues of this 

esteemed committee. Thank you for the opportunity to testify before you today on behalf of the 

American Association for the Advancement of Science, or AAAS. AAAS is the largest general 

scientific society and publisher of the Science family of journals. Our mission is simple: to 

advance science, engineering, and innovation throughout the world for the benefit of all people. 

I want to state from the outset that I do not want my presence here to be construed as 

advocating for a specific environmental regulation. Science, and the process by which science is 

conducted, must be recognized as the most reliable pathway to knowledge and the best basis for 

making public policy and regulations. To quote a recent editorial in our journal Science, "science 

is not a political construct or a belief system. It provides testable, fundamental knowledge of the 

world and how things work." It is a set of principles dedicated to discovery and use of evidence 

to continually test these discoveries. Though science gets a great deal of credit for advancing our 

understanding of the world, it is less understood for its foundational quality: humility in the face 

of evidence. Overtime, when one's cherished beliefs, partisan ideologies, and wishful thinking 

have turned out to be wanting, the scientific evidence is likely to remain. 
33 

We need more reverence for evidence in our policy making. Without respect for 

evidence, and by extension evidence-based policymaking, our country's future, and indeed all of 

humanity's future, becomes dangerously compromised. 

Good regulations depend on scientific progress. Science is not static, that is why the 

process of science converges on reliable knowledge. Science does not deal with cut-and-dried 

facts, ever immutable. Sometimes we will see the science push aside an understanding for a 

better, more verifiable understanding. That is the job of scientists --through the scientific 

process. However attractive you may find your own belief at any time, your odds of success are 

better if you can go with scientifically established thinking. 

Scientific progress depends on openness, transparency, and the free flow of ideas and 

people. These are the principles that have helped the United States attract and richly benefit from 

scientific talent. From the Apollo Program and exploring the far reaches of the universe, 

advancing biomedical research for curing diseases, to harnessing science to build a thriving hightech sector, the United States has been a leader in science, education, and innovation. In order to 

remain the world leader, the U.S. must continue to foster this free exchange of ideas and talent. 

Furthermore, scientists-- whether in industry, academia, or the government-- must have 

confidence that they can conduct their work in an atmosphere free of intimidation or undue 

influence. Policymakers should never dictate the conclusions of a scientific study, and they 

should base policy on a review of relevant research and the provisions of relevant statutes. In 

other words, the integrity ofthe process must be upheld. During the Bush and Obama 

Administration federal agencies worked to develop and implement scientific integrity and access 
34 

to data policies. This bipartisan recognition of strengthening scientific integrity in federal 

agencies lays a crucial foundation that should not be weakened. 

Moreover, regulations and agency actions should be informed by the best available 

science and a rigorous scientific process. Undermining the integrity of the scientific process, or 

the ability of federal agencies to utilize rigorous science in establishing policies, could have 

long-term consequences ranging from a depletion of intellectual capital, to negative health 

outcomes for Americans and the world. It is with this in mind that we urge caution in setting 

laws that would make science a combat zone. Legislation removing concepts like reproducibility 

and independent analysis from the hands of scientists and into a legislative chamber or a court 

room would truly have a chilling effect on the scientific process and reduce the benefits that 

science could bring to society. Seeking to inlluence the scientific process has no place in how a 

government or other entity should conduct science. 

In recent decades, opinion and ideological assertions have crowded out scientifically 

validated evidence on some issues. If policymakers and citizens do not recognize the value that 

science plays in modern society, and the enormous opportunity for scientific evidence to help 

make better public decisions, research and innovation will not thrive. 

Thank you again for the invitation to testify today. I look forward to working with you in 

the weeks and months ahead. 
35 

Biography of Rush D. Holt, Chief Executive Officer 

Rush D. Holt, Ph.D., became the 18th chief executive officer of the American Association for the 

Advancement of Science (AAAS) and executive publisher of the Science family of journals in February 

2015. In this role, Holt leads the world's largest multi-disciplinary scientific and engineering society. 

Over his career, Dr. Holt has held positions as a teacher, scientist, administrator, and policymaker. From 

1987 to 1998, Holt was assistant director of the Princeton Plasma Physics Laboratory (PPPL), a 

Department of Energy national lab, which is the largest research facility of Princeton University and one 

of the largest alternative energy research facilities in the country. At PPPL, Holt helped establish the 

lab's nationally renowned science education program. From 1980 to 1988, Holt served on the faculty of 

Swarthmore College, where he taught courses in physics and public policy. In 1982, he took leave from 

Swarthmore to serve as an AAAS/American Physical Society Science and Technology Policy Fellow on 

Capitol Hill. The Fellowships program, dating to 1973, places outstanding scientists and engineers in 

executive, legislative, and Congressional branch assignments for one or two years; by early 2015, the 

program had served nearly 3,000 alumni working worldwide in the policy, academic, industry, and 

nonprofit realms. Holt has said that his AAAS S&T Policy Fellowship was "life changing," and served as a 

springboard to his role in Congress. He also served as an arms control expert at the U.S. State 

Department, where he monitored the nuclear programs of countries such as Iraq, Iran, North Korea, and 

the former Soviet Union. In 1981, Holt was issued a patent for an improved solar-pond technology for 

harnessing energy from sunlight. 

Before coming to AAAS, Holt served for 16 years as a member of the U.S. House of Representatives, 

representing New Jersey's 12th Congressional District. In Congress, Holt served as a senior member of 

the Committee on Natural Resources and the Committee on Education and the Workforce. On Capitol 

Hill, Holt established a long track record of advocacy for federal investment in research and 

development, science education, and innovation. He served on the National Commission on the 

Teaching of Mathematics and Science (known as the Glenn Commission), founded the Congressional 

Research and Development Caucus, and served as a co-chair of the Biomedical Research Caucus. Holt 

served eight years on the Permanent Select Committee on Intelligence and, from 2007 to 2010, chaired 

the Select Intelligence Oversight Panel, which worked to strengthen legislative oversight oft he 

intelligence community. His legislative work earned him numerous accolades, including being named 

one of Scientific American magazine's "50 National Visionaries Contributing to a Brighter Technological 

Future" and a "Champion of Science" by the Science Coalition. He has also received awards from the 

American Chemical Society, the American Association of University Professors, the National Association 

of Graduate-Professional Students, the American Institute for Medical and Biological Engineering, the 

Council of Scientific Society Presidents, the American Geophysical Union, and the Biotechnology Industry 

Organization. Holt is also a past recipient of two of AAAS' highest honors: the William D. Carey 

Lectureship Award {2005) and the Philip Hauge Abelson Award {2010). 

From December 2014 to February 2015, Holt was appointed a Director's Visiting Scholar at the Institute 

for Advanced Study in Princeton, New Jersey. 
36 

Holt is a Phi Beta Kappa graduate of Carleton College in Northfield, Minnesota, and he holds M.A. and 

Ph.D. degrees in physics from New York University. He is an elected fellow of AAAS, the American 

Physical Society, and Sigma Xi, and he holds honorary degrees from Monmouth University, Rider 

University, University of Toledo, and Thomas Edison State College. He is married to Margaret Lancefield, 

a physician, and they have three children and seven grandchildren. 
37 

Chairman SMITH. Thank you, Dr. Holt. 

And, Dr. Belzer. 

TESTIMONY OF DR. RICHARD BELZER, 

INDEPENDENT CONSULTANT 

Dr. BELZER. Thank you, Chairman Smith, Ranking Member 

Johnson, Members of the Committee. Thank you again for the opportunity to testify today. My testimony is informed by 30-plus 

years of experience with environment science and economics, and 

I’m not testifying on behalf of any past or present client. Most of 

my recent consulting work has involved intellectual property, and 

I don’t think that’s related at all to EPA. 

In 1983, then-Administrator Bill Ruckelshaus wrote an article 

published in Science that ‘‘Risk assessment at EPA must be based 

only on scientific evidence and scientific consensus. Nothing will 

erode public confidence faster than suspicion that policy considerations have been allowed to influence the assessment of risk.’’ 

But EPA risk assessments are chock full of policy considerations, 

so it should be no surprise that public confidence in the EPA has 

eroded, as Ruckelshaus predicted. You need not and should not 

take my word for it. In 2004, the EPA’s science advisor published 

a report on the Agency’s risk-assessment practices. The science advisor defended these practices as follows, ‘‘EPA seeks to adequately 

protect public health and environmental health by ensuring that 

risk is not likely to be underestimated.’’ 

Now, under the Ruckelshaus principle, when asked to measure 

a mouse, EPA is not supposed to give dimensions closer to that of 

an elephant because elephants are riskier, which they are. The 

EPA’s science advisor justifies mistaking mice for elephants because EPA is ‘‘a health and environmental protective agency.’’ 

That’s a non sequitur. 

The Central Intelligence Agency is a national security agency. 

Would that justify exaggerating the risk that Iraq possessed weapons of mass destruction? No. But it is just as improper for EPA to 

exaggerate the human health risks of contaminants in drinking 

water, chemicals in commerce, or pollutants in the atmosphere. 

When any agency exaggerates risk, it undermines responsible 

regulatory decision making, and it does so three ways. First, it 

scares the public, which cannot discriminate between large and 

small risks if agencies exaggerate. Second, it undermines the accurate estimation of benefits from regulation. Exaggerated risk estimates lead to exaggerated benefit estimates. Third, it usurps the 

authority of the head of the Agency, who is charged by Congress 

with making oftentimes hard choices. He can’t do that with unreliable information. And if he figures out that he’s being sandbagged 

by his own staff and decides to ignore what he’s being told, he will 

be accused of ignoring science. 

In my written testimony, I explain why EPA’s safety assessments 

are not scientific, and I use EPA’s definition of the reference dose, 

a copy of which is distributed to you, but here’s the definition. A 

reference dose is an estimate with uncertainties spanning perhaps 

an order of magnitude of a daily oral exposure to the human population, including sensitive subgroups, that is likely to be without an 

appreciable risk of a deleterious effect during a lifetime. 
38 

Now, an order of magnitude we understand. That’s a factor of 10. 

But what is perhaps an order of magnitude? Does it mean less 

than 10, more than 10, a lot more than 10? Could it be 10? It could 

be 1,000. It could be 10,000. We don’t know. What is an appreciable 

risk of a deleterious effect? How bad must an effect be to qualify 

as deleterious? What is an appreciable risk of experiencing such a 

thing? Now, these are not scientific terms. These are policy terms? 

Whose personal opinions inform these choices? This is a definition 

only a lawyer could love and quite possibly only a lawyer could justify. 

There’s a great deal of interest in transparency, and I’ve highlighted the reference dose definition because almost every issue of 

transparency arises downstream of obscure definitions like this. 

Even if every other transparency issue were solved, the most fundamental opacities in EPA risk and safety assessments would remain. 

Now, probably the most effective thing Congress can do to improve the quality of EPA science or any agency’s science is to require that they adhere to the principles and procedures set forth 

in OMB’s Information Quality Guidelines. These guidelines have 

been available—been out for 15 years, but there’s very little to 

show for it because agencies do not comply, and they don’t comply 

because no one has standing in federal court to compel them to 

comply. 

I’m happy to look forward to any questions you might have and 

expand upon this at your leisure. 

[The prepared statement of Dr. Belzer follows:] 
39 

Testimony before the U.S. House of Representatives 

Committee on Science, Space, and Technology 

Hearing: 

The Environmental Protection Agency's 

Process for Evaluating and Using Science During Its Regulatory Decision Making Activities 

Richard B. Belzer, Ph.D. 

February 7, 2017 

Chairman Smith, Ranking Member Johnson, and Members of the Committee, thank you 

for the opportunity to testify today concerning the Environmental Protection Agency's use of 

science for regulatory decision-making. My testimony in informed by 30 years of experience 

with environmental science and economics that began in earnest during my doctoral research 

at Harvard University. 

I. Background 

After completing my dissertation, in 1988 I joined the Office of Information and 

Regulatory Affairs at the Office of Management and Budget as a staff economist. I served five 

years under the administrations of Presidents Reagan and George H.W. Bush and five years 

under the administration of President Clinton. My job was to review Regulatory Impact 

Analyses prepared by Federal agencies in support of regulations expected to have annual costs 

exceeding $100 million. Many of the RIAs I reviewed concerned regulations with estimated 

costs of many billions of dollars. I reviewed RIAs from several agencies including the Food and 

Drug Administration, the U.S. Departments of Agriculture, Commerce, Interior arid Labor, but 

mostly the Environmental Protection Agency. Because of my dissertation work on the potential 

use of deposit-refund systems for managing hazardous waste, within EPA I focused on major 

rules developed by the Office of Solid Waste and Emergency Response and the Office of Water. 

The principles I followed during my reviews were the same under all three 

administrations: provide OMB officials and White House staff the most objective estimates 

possible of benefits, costs and other effects. My job was strictly analytical. Both Executive Order 

12291, signed by President Reagan, and Executive Order 12866, signed by President Clinton, 

clearly stated a preference that the net social benefits of federal regulation be maximized. But 

this objective is infeasible if decision-makers lack unbiased estimates of benefits and costs. 
40 

Page 2 of 11 

I left OMB in 1998, and after a stint as a visiting professor, in 20011 opened a private 

consulting practice. My testimony today is not on behalf of any client, past or present. 

II. Fundamental Characteristics of EPA Risk Assessment 

A. EPA risk assessments are, by design, not objective 

I learned during my doctoral research that EPA risk assessments did not objectively 

characterize risk. Rather, they were described as "conservative."1 This term is misleading 

because it does not make clear what it is being "conserved." EPA risk assessments are neither 

"conservative" nor liberal," but they are intended to approximate something close to the worst 

case. I have reviewed some risk assessments in which risk estimates were either practically or 

theoretically impossible. 

You need not take my word for it. In 2004, the EPA Science Advisor published a report 

on its risk assessment practices. At the time, EPA faced a chorus of criticism alleging that the 

Agency grossly exaggerated risks. EPA defended its practices by stating as follows: 

EPA risk assessments tend towards protecting public and environmental health 

by preferring an approach that does not underestimate risk in the face of 

uncertainty and variability. In other words, EPA seeks to adequately protect 

public and environmental health by ensuring that risk is nat likely to be 

underestimated.2 

In plain English, this means that whenever there is scientific uncertainty, EPA errs on the side of 

overstating human health risk. Further, when characterizing health risk in a population, EPA 

looks for individuals who faces the highest potential risk and uses those persons to describe the 

population. 

These are not sensible practices. If we were characterizing the risk to Americans posed 

by peanuts, we would not say that the risk of death from anaphylactic shock from peanut 

ingestion is 50%, even though it is conceivable that there is someone for whom this is true. 

Similarly, if we were concerned about obesity in the United States, we would not say that 

Americans weigh 1,036 pounds- the reputed weight of the heaviest person in the United 

1 Another descriptor EPA uses for its risk assessments is "protective," but that term also 

begs the question what is being protected. Precautionary efforts to protect the public from risk 

in one area necessarily exposes them to risk in another. 

2 U.S. Environmental Protection Agency Office of the Science Advisor (2004), p. 11 

(emphasis in original). 

Richard B. Belzer, Ph.D. 

Regulation, Risk, Economics & Information Quality 

Strategy & Analysis Consulting 

rbbe/zer@post.harvard.edu •:•121703. 780.18SO 
41 

Page 3 of 11 

States.3 When thinking about the health risk posed by PM2.5, we do not assume that everyone 

is elderly, infirm, or suffers from chronic obstructive pulmonary disease. 

We know not to assume the worst when we make routine decisions in almost every 

avenue of life. For some reason, however, we do not practice common sense in environmental 

health policy. And it is EPA policy not to use common sense. Quoting again from the 2004 

report of the EPA Science Advisor (p. 13): 

[S]ince EPA is a health and environmental protective agency, EPA's policy is that 

risk assessments should not knowingly underestimate or grossly overestimate 

risks. This policy position prompts risk assessments to take a more "protective" 

stance given the underlying uncertainty with the risk estimates generated. 

In plain English, this means EPA will strive for the highest estimate of risk that does not bring 

upon the Agency unbearable ridicule. You simply cannot rely on EPA risk assessment to give you 

an unvarnished perspective. When given an EPA risk assessment, all you know is risk can't be 

any worse. 

These practices undermine responsible regulatory decision-making at least three ways. 

First, they needlessly and irresponsibly scare the public about the hazards of life. 

Exaggerating risk is an excellent tactic for gaining the most attention from Congress, the White 

House, the press and the public, and for increasing one's budget and delegated legislative 

authority to regulate. 

Second, they undermine the responsible estimation of benefits from regulation. If I'm 

given a worst-case risk assessment, I cannot use it to estimate public health benefits. I need, at 

a minimum, a central tendency estimate, like an average or median. Ideally I would have much 

more information than this, but I can use a central tendency estimate risk estimate to 

approximate health benefits to the population. I can't do anything useful or informative with a 

"conservative" or "protective" risk estimate. 

Third, it usurps the authority of the EPA Administrator, who is charged by Congress with 

making oftentimes hard choices. When EPA staff give the Administrator an exaggerated risk 

estimate, the Administrator cannot make a fully informed decision. He faces extraordinary 

pressure to ratify the policy preferences the staff have hidden away. If the Administrator learns 

that EPA staff are sandbagging him and looks elsewhere for more objective information, he will 

be accused of "ignoring science." Indeed, EPA staff produce so-called "conservative" risk 

3 My source for this is Wikipedia, which though often inaccurate is accurate enough for 

present purposes. 

Richard B. Belzer, Ph.D. 

Regulation, Risk, Economics & Information Quality 

Strategy & Analysis Consulting 

rbbelzer@post.harvard.edu •!•IZI703. 780.1850 
42 

Page 4 of 11 

assessments to tie the Administrator's hands. This enables Agency staff to make critical policy 

decisions secretly through the back door. 

B. Nontransparency about uncertainty 

Much of EPA risk assessment inevitably consists of extrapolating to humans from 

animals, such as rats and mice, and from very high exposure levels in a laboratory to 

comparatively very low exposures in the environment. These may be reasonable practices for 

some purposes, but often they are not reasonable at all. Rats and mice are not little people, 

and effects that occur when biological systems are overloaded, as they are by design in 

laboratory experiments, generally are not expected to occur under normal conditions. 

When I began reviewing EPA cancer risk assessments in the mid-1980s, the Agency's 

conventional practice was to report risk estimates in a way that accounted for these key 

uncertainties. A common way this was done was to say, "We estimate lifetime excess cancer 

risk to be as high as x, but it could be as low as zero." And zero was understood to be the best 

risk estimate if, for example, extrapolating from rats or mice was biologically incorrect, or if 

there was a human exposure threshold below which carcinogenesis was not reasonably 

expected to occur. About 20 years ago, EPA abandoned the practice of qualifying its cancer risk 

estimates this way. Now, EPA reports them in ways that do not reveal uncertainty. 

The difference between these two approaches can be seen in Figure A below. The 

traditional description of a cancer risk estimate told decision-makers and the public that there 

was substantial uncertainty, and that the true (but unknown) risk could be as low as zero. The 

modern description does not communicate this uncertainty. 

Richard B. Belzer, Ph.D. 

Regulation, Risk, Economics & Information Quality 

Strategy & Analysis Consulting 

rbbelzer@post.harvard.edu •:•!Z1703. 780.1850 
43 

Page 5 of 11 

Figure A 

Alternativel\lnterpretations[Jbfl:IPA!llnit.OCanceJilRiskll:stimates~ 

Tradltlonalfllescriptlon 

- NeW[[lescription 

Zero Ufetimetlxcessl£ancerlll:isk 

Ill. Fundamental Characteristics of EPA Safety Assessment 

A lot of what the public understands to be "risk assessment" actually isn't risk 

assessment at all. The correct term is "safety assessment" because its purpose is to identify a 

"safe" level of exposure, not to estimate risk. But a safety assessment isn't science; it's a policy 

decision draped in scientific clothing. The reason it isn't science is science has no definition for 

"safety." Science is about ascertaining facts, not divining policies or making philosophical 

judgments. 

In EPA world, the primary example of a safety assessment is the Reference Dose, often 

abbreviated "RfD."4If you are exposure below the RfD, you're said to be "safe." Except in truly 

extraordinary cases, you are likely to agree because the methods used to derive Reference 

Doses are very, very "conservative." 

4 The Reference Concentration (RfC) is an analogous tool for the inhalation pathway. 

Richard B. Belzer, Ph.D. 

Regulation, Risk, Economics & Information Quality 

Strategy & Analysis Consulting 

rbbelzer@post.harvard.edu •!•IZJ703.780.1850 
44 

Page 6 of 11 

A. EPA safety assessments are, by design, controlled by undisclosed policy 

judgments 

Nonscientific considerations are spread throughout the RfD process. To see this, let's 

look at EPA's definition: 

An estimate (with uncertainty spanning perhaps an order of magnitude) of a 

daily oral exposure to the human population (including sensitive subgroups) that 

is likely to be without an appreciable risk of deleterious effects during a lifetime. 

It can be derived from a NOAEL, LOAEL, or benchmark dose, with uncertainty 

factors generally applied to reflect limitations of the data used. Generally used in 

EPA's noncancer health assessments.s 

I have highlighted in bold terms within the definition that are substantially or exclusively policy, 

not science. It's useful for Members to understand that EPA acknowledges that a Reference 

Dose is uncertain by a factor of 10. But wait. EPA says Reference Doses are uncertain by 

perhaps a factor of 10. Does that mean they might be uncertain by a factor of 100? A factor of 

1,000? We don't know. 

What is a "sensitive subgroup"? Is a subgroup containing a single person in the United 

States too small? How about 100 persons? How large must it be? One percent of the U.S. 

population- clearly a small fraction- means 3.25 million people. How sensitive must these 

people be? Twice as sensitive? Ten times as sensitive? 

"Likely" means a probability greater than 50%. To what does that probability apply? 

According to the definition, it applies to risk of "deleterious" effects? How bad must they be to 

qualify? They must be "appreciably" "deleterious." Only a lawyer could tell you what it means 

to experience "an appreciable risk of deleterious effects." There are no scientific answers to 

these questions; only policy judgments. When lawyers rule, science does not. 

Ambiguity in the definition of the Reference Dose goes on and on and on. It's no wonder 

that an EPA Administrator, trying to play it straight, does not know how to interpret this 

information. 

s U.S. Environmental Protection Agency (2017). There are also Reference Dose 

definitions that apply to different durations of exposure (e.g., "acute," "subchronic," "chronic") 

and pathways (e.g., "oral"). 

Richard B. Belzer, Ph.D. 

Regulation, Risk, Economics & Information Quality 

Strategy & Analysis Consulting 

rbbelzer@post.harvard.edu •!•1'11703. 780.1850 
45 

Page 7 of 11 

A. Nontransparency about uncertainty 

Therefore, it's for good reason that the EPA Administrator may not know how to use a 

Reference Dose to inform decision-making. let's assume for simplicity that uncertainty is 

exactly a factor of 10. Figure B below shows many ways the RfD might be interpreted. 

Row 1 shows what EPA conventionally reports to the public.6 It's what is called a "point 

estimate," meaning that no uncertainty about the estimate is communicated. Row 2 shows 

what the EPA staff author of the RfD probably intends; uncertainty lies above the RfD. But 

because this information is poorly communicated, and EPA Administrators have limited 

knowledge about the derivation process and are inclined to be worrisome when public health is 

involved, they may think the 10-fold uncertainty contained in the definition is below the RfD. 

Rows 4-6 show other ways this 10-fold uncertainty might be understood, and none of these 

interpretations is necessarily incorrect.? 

While it is sometimes possible to use an EPA risk assessment to estimate the benefits of 

a regulation, it is impossible to use an EPA safety assessment for that purpose. The definition of 

the Reference Dose tells us nothing about how much risk is reduction is obtained by any 

reduction in exposure. That means we can't estimate health benefits. 

Finally, I want to add that nothing I have just testified to is new. On behalf of OMB, in 

1990 I wrote a chapter for the Regulatory Program of the United States Government. Most of 

that chapter, titled "Current Regulatory Issues in Risk Assessment and Risk Management," s 

remains valid 27 years later. 

6 U.S. Environmental Protection Agency (2016). 

7 Alternative interpretations of different players in the drama are described by Felter 

and Dourson (1998). 

s Office of Management and Budget (1990). 

Richard B. Belzer, Ph.D. 

Regulation, Risk, Economics & Information Quality 

Strategy & Analysis Consulting 

rbbelzer@post.harvard.edu •:•liJ703. 780.1850 
46 

Figure B 

Alternative!1.nterpretationsllbffi_PA'~eferenceiDose!fi 

1.rJI.eported])yfl:PA 

2.11ntendediby1IPA~taff 

3.1Jeared!Jly£1PA!Dfficials 

4.!£quai!Divislo~linear} 

5.!£quai[ID1visio~logarithmic) 

6.1J'Iaus1blyr!rue 

Risk 

IV. Implications for Benefit-Cost Analysis 

Page 8 of 11 

EPA uses risk assessments as inputs to its benefit-cost analyses. "Conservatism" in risk 

assessment is therefore propagated into the Agency's estimate of regulatory benefits.9 So, all 

other things being equal, EPA will not be "knowingly underestimate" benefits. But that means 

they will overestimate benefits. Whether they "grossly" overestimate benefits depends on how 

"conservative" the risk assessment is, whether EPA has disclosed enough detail to permit third 

parties to figure it out, and whether there is a venue in which errors can be corrected 

Sometimes, a single "conservative" assumption is enough.to 

g This was the key point in Office of Management and Budget (1990), and it is the reason 

why OMB guidance on benefit-cost analysis requires agencies to estimate benefits objectively. 

See Office of Management and Budget (2003). OMB lacks the tools to enforce this requirement. 

10 EPA's "central estimate" of the present value of benefits from regulations 

promulgated under the Clean Air Act from 1990 to 2020 at $12 trillion. See U.S. Environmental 

Protection Agency (2011). Estimated annual benefits, $1.3 trillion, are 7% of U.S. Gross 

Domestic Product. Almost all benefits vanish if EPA's assumed causal relationship between low 

Richard B. Belzer, Ph.D. 

Regulation, Risk, Economics & Information Quality 

Strategy & Analysis Consulting 

rbbelzer@post.harvard.edu •!•lil703. 780.1850 
47 

Page 9 of 11 

A typical Agency benefit-cost analysis includes benefit estimates derived from these 

unreliable inputs. You should not be surprised if benefit estimates in these analyses are highly 

overstated. And you should pay no attention to OMB's Reports to Congress on the benefits and 

costs of federal regulation.u OMB does not report objective benefit or cost estimates, or their 

own estimates based on independent review. OMB merely summarizes what the agencies said 

in their published benefit-cost analyses, even if the OMB staff know that these estimates are 

wrong. Congress faces a similar problem with respect to reports submitted to the Comptroller 

General pursuant to the Congressional Review Act (5 U.S.C. § 8012(a)(1). These reports are 

generally unreliable, and GAO lacks the expertise and time to critically review them. 

V. Implications for Congress 

Consistent with the policy set forth in the 2004 EPA Staff Paper, wherever you see a 

nonscientific, policy term in the definition of a putative scientific concept such as a risk or safety 

assessment, you can be confident that EPA staff have chosen to be "conservative"- that is, 

they have made assumptions that do not "knowingly underestimate or grossly overestimate" 

the factor of interest. Risk and safety assessments are constructed using multiple 

"conservative" assumptions. So, while we can be quite sure that actual cancer risk is likely to be 

less than an EPA cancer risk estimate, and that exposures to noncarcinogens below the 

Reference Dose poses essentially zero risk, these risk and safety assessments are unreliable for 

use in benefit-cost analysis.12 

The House recently passed H.R. 26, the "Regulations from the Executive in Need of 

Scrutiny Act of 2017." This is not the time or place to debate the merits of this bill. However, if 

the bill were enacted into law, it is certain that Members will be poorly informed about the 

benefits and costs of major regulations intended to reduce human health risk. Benefit estimates 

based on "conservative" EPA risk assessments will be exaggerated and unreliable, so Members 

who rely on such estimates will be misled. 

PM2.5 concentrations and premature mortality is relaxed. Unsurprisingly, EPA's causality 

assumption is controversial. See, e.g., Cox, Popken and Ricci (2013). 

11 These Reports are mandated by the Regulatory Right-to-Know Act of 2000, Pub. L. 

106-554 (title VI, Sec. 624; 114 Stat. 2763A-161) The most recent draft Report to Congress was 

published in draft form on December 23, 2016. See Office of Management and Budget {2016). 

12 A group of 19 experts recently published a listicle identifying 10 things non-experts 

should look out for in benefit-cost analysis. Number 6 on the list warns against relying on risk 

assessments that are not transparent or objective. See Dudley, Belzer, Blomquist, Brennan, 

Carrigan, Cordes, Cox, Fraas, Graham, Gray, Hammitt, Krutilla, Linquiti, Lutter, Mannix, Shapiro, 

Smith, Viscusi and Zerbe (2017). 

Richard B. Belzer, Ph.D. 

Regulation, Risk, Economics & Information Quality 

Strategy & Analysis Consulting 

rbbelzer@post.harvard.edu •:•1'11703. 780.1850 
48 

Page 10 of 11 

Probably the most effective way Congress could improve the quality of the scientific 

information on which regulatory decision-making depends is to require all agency science and 

economics to adhere to the principles set forth in OMB's Information Quality Guidelines.B 

These Guidelines have been in place for 15 years, but there is little to show for it because 

agencies simply do not comply. And the main reason they do not comply is no one has standing 

in federal court to compel them to do so. Agency performance would improve dramatically if 

this loophole in the law were corrected.14 

Thank you for the opportunity to testify today. I look forward to answering any 

questions you might have. 

VI. References 

Cox T, Popken D, Ricci PF. 2013. Temperature, not fine particulate matter (PM2.s), is causally 

associated with short-term acute daily mortality rates: results from one hundred United States 

cities. Dose-Response 11. 

Dudley S, Belzer R, Blomquist G, Brennan T, Carrigan C, Cordes J, et al. 2017. Consumer's Guide 

to Regulatory Impact Analysis: Ten Tips for Being an Informed Policymaker. Washington DC: 

George Washington University Regulatory Studies Center. 

Felter SP, Dourson ML. 1998. The Inexact Science of Risk Assessment (and Implications for Risk 

Management). Human and Ecological Risk Assessment 4:245-251. 

Information Quality Act. 2000. 114 Stat. 2763A-153. 

Office of Management and Budget. 1990. Current Regulatory Issues in Risk Assessment and Risk 

Management. In: Regulatory Program of the United States Government, April1, 1990 --March 

31, 1991. Washington, DC: OMB, 13-26. 

Office of Management and Budget. 2002. Guidelines for Ensuring and Maximizing the Quality, 

Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies; Notice; 

Republication. Federal Register 67:8452-8460. 

Office of Management and Budget. 2003. Circular A-4: Regulatory Analysis. Available: 

http://www.whitehouse.gov/omb/circulars a004 a-4 [accessed April 6, 2011,]. 

13 Office of Management and Budget (2002). 

14 The Information Quality Act (2000),, 114 Stat. 2763A-153-154, says the government 

must "establish administrative mechanisms allowing affected persons to seek and obtain 

correction of information maintained and disseminated by the agency that does not comply" 

with OMB guidelines (emphasis added). Agencies have implemented the law so that the public 

may seek correction all it wants, but cannot obtain them. 

Richard B. Belzer, Ph.D. 

Regulation, Risk, Economics & Information Quality 

Strategy & Analysis Consulting 

rbbelzer@post.harvard.edu •!•l:ll703. 780.1850 
49 

Page 11 of 11 

Office of Management and Budget. 2016. 2016 Draft Report to Congress on the Benefits and 

Costs of Federal Regulations and Agency Compliance with the Unfunded Mandates Reform Act. 

Washington DC: OMB. 

U.S. Environmental Protection Agency. 2011. The Benefits and Costs of the Clean Air Act from 

1990 to 2020; Final Report- Rev. A. Washington DC: US EPA, 238. 

U.S. Environmental Protection Agency. 2016. IRIS Assessments. Available: 

https://cfpub.epa.gov/ncea/iris2/atoz.cfm [accessed February 4, 2017]. 

U.S. Environmental Protection Agency. 2017. IRIS Glossary; Terms and Acronyms; 'Reference 

Dose'. Available: 

https://ofmpub.epa.gov/sor internet/registry/termreg/searchandretrieve/termsandacronyms/ 

search. do? search=& term= refe re n ce%20d ose& match Criteria= Co nta i ns&checke dAcro nym =true 

&checkedTerm-true&hasDefinitions=false; Terms ands [accessed February 2, 2017]. 

U.S. Environmental Protection Agency Office of the Science Advisor. 2004. An Examination of 

EPA Risk Assessment Principles and Practices; Staff Paper, EPA/100/B-04/001. 

Richard B. Belzer, Ph.D. 

Regulation, Risk1 Economics & Information Quality 

Strategy & Analysis Consulting 

rbbe/zer@post.harvard.edu •!•IZI703.780.1850 
Definition of the Reference Dose 

(Substantially or wholly nonscientific, policy terms in bold) 

An estimate (with uncertainty spanning perhaps an order 

magnitude) of a daily oral exposure to the human populat

(including sensitive subgroups) that is likely to be withou

appreciable risk of deleterious effects during a lifetime. It 

be derived from a NOAEL, LOAEL, or benchmark dose, 

with uncertainty factors generally applied to reflect 

limitations of the data used. Generally used in EPA's 

noncancer health assessments. 

Source: U.S. Environmental Protection Agency. 2017. IRIS Glossary; Terms and Acronyms; 'Referenc

Dose'. 
51 

Richard B. Belzer 

EDUCATION 

• Ph.D. 1989 (Harvard University) 

• M.P.P. 1982 (John F. Kennedy School of Government) 

• M.S. Agricultural Economics 1980 (University of California, Davis) 

• B.S. Agricultural Economics 1979 (University of California, Davis) 

• Hometown: Torrance, California 

EMPLOYMENT 

• President, Regulatory Checkbook (2001-present) 

o Regulatory oversight 

o Information quality 

o Organization of scientific peer review 

• Managing Editor, NeutralSource.Org (2006-present) 

• Independent Consulting in Regulatory Policy, Economics and Risk Analysis 

(2001-present) 

• Visiting Professor of Public Policy, Washington University in St. Louis (1998-

2001) 

• Economist, Office of Information and Regulatory Affairs, Office of 

Management and Budget (1988-1998) 

o Division Performance Awards (1991, 1992) 

o Special Achievement Award (1990) 

ACHIEVEMENTS RELEVANT TO TODAY'S HEARING 

• Prepared the final version of OMB first guidance on how to prepare 

Regulatory Impact Analyses (1990) 

• Authored OMB white paper on risk assessment and risk management (1990) 

• Contributed significantly to the text of Executive Order 12,866 (1993) 

• Contributed significantly to OMB's Principles for Risk Analysis (established 

by the Clinton Administration in 1995 and ratified by the George W. Bush 

Administration in 2007) 

• Fellow, Cecil and Ida Green Center for the Study of Science and Society 

(1995) 

PROFESSIONAL AFFILATIONS 

• Society for Risk Analysis 

o Elected Treasurer (1998, 2000) 

o Outstanding Service Award (2003) 

• Society for Benefit Cost Analysis 

o Elected Secretary/Treasurer (2008, 2010, 2012) 

o Elected Treasurer (2014) 

• American Association of Wine Economists 
52 

Chairman SMITH. Thank you, Dr. Belzer. 

Before we get to questions, I’d like to ask unanimous consent 

that the gentleman from California, Jerry McNerney, be allowed to 

participate in today’s hearing. He has been selected to serve on the 

Science Committee, and his official appointment is imminent. So, 

Jerry, we welcome you back. 

Mr. MCNERNEY. Well, I thank the Chairman. I’ve spent two 

terms on this Committee, so it’s a great thing to be back, and I look 

forward to our work together. 

Chairman SMITH. Absolutely. I thank the gentleman from California for his comments. 

Let me recognize myself for questions. And, Dr. White, I’ll address my first question to you, and that is why would the EPA hide 

data that they say justifies regulations from the American people, 

and why should that data be made public? Turn on your—— 

Dr. WHITE. There we go. 

Chairman SMITH. Yes. 

Dr. WHITE. Transparency in the way that EPA evaluates its 

science and what data it selects to underlie its decision-making is 

extremely important. So that information should be made available 

so that folks can actually go through and evaluate those scientific 

evaluations. 

One of the things that we do need to take into consideration as 

making that data publicly available is that there are adequate protections for confidential business information to ensure that we 

keep innovation and competitiveness available for the marketplace. 

Chairman SMITH. Okay. Thank you, Dr. White. 

And, Mr. Holmstead, does the EPA use a biased modeling system 

to calculate and determine the benefits from its proposed regulations? And if so, why can that not be justified? 

Mr. HOLMSTEAD. Well, thank you for the question. I think at 

least for the last few years EPA has used methodologies that clearly overstate the benefits of their regulations. And I’m happy to submit something that would provide you a bit more detail, but let me 

just summarize very quickly. Virtually any air regulation that’s 

been done over the last eight years is based on the benefits of a 

single pollutant known as PM2.5 or fine particles. And people are 

surprised when they understand this. EPA issues a regulation for 

mercury, EPA issues a standard for ozone, EPA issues a standard 

for diesel emissions, and yet, when you look at the underlying evaluation, EPA’s claim is all of these things are justified because in 

some fashion EPA predicts that that will reduce levels of fine particles in the environment. 

There is a whole other program for regulating these chemicals. 

There’s a way of evaluating them, and yet the benefits that EPA 

claims is consistent—is—I’m sorry, is completely inconsistent with 

the way they do it in this other program. So, again, this is an issue 

that probably deserves more than a short answer, and I’d be happy 

to provide that to you. But there’s no question that EPA has started to use cost-benefit analysis as a way to promote its regulations 

rather than a tool that can be used to inform good regulatory decisions. 
53 

Chairman SMITH. Okay. Thank you for that response. And yes, 

we’ll take the details and we’ll make them a part of the record as 

well. 

Dr. Belzer, let me ask you about cost-benefit analysis and why 

should there be a cost-benefit analysis for all EPA regulations? 

Dr. BELZER. Mr. Chairman, it comes to benefit-cost I’ll say—that 

would be the appropriate term, by the way—benefit-cost I’ll say— 

I finished eight years as Treasurer—Secretary Treasurer at the Society for Benefit-Cost Analysis, and so I am an avid defender and 

advocate of practice. And the reason is fairly simple, that it is impossible for decision-makers, whether in an agency or on Capitol 

Hill, to understand what the implications are of actions that are 

being taken without benefit-cost analysis. It can’t be done without 

that. Otherwise, it’s based on emotion, it’s based on politics, based 

on cronyism. It’s based on other factors. 

But—and also I would say it can’t be based on science because 

science is the underpinning to the benefits assessment in a benefitcost analysis. So agencies should be doing that all the time, and it’s 

also cost-effective. It costs relatively little money to do a benefitcost analysis, and it can save an enormous amount of regulatory 

costs and it can also dramatically improve the benefits of a regulation. 

I’m particularly fond of one that I managed when I was at OMB, 

which the Department of Agriculture was trying to ban a product, 

and they were going about it all—in a way that made no sense scientifically and economically. With help, I was able to change the 

justification for it, and they were successfully able to ban this product because people were dying from it. Without the analysis, that 

can’t be done. That would have been litigated, and the Department 

of Agriculture would have lost. 

Chairman SMITH. Okay. Thank you, Dr. Belzer. 

And, Dr. Holt, my last question for you. The editor-in-chief of 

your publication Science recently said ‘‘Dr. Bates raises some serious concerns. After the results of any appropriate investigations, 

we will consider our options,’’ and those options ‘‘could include retracting that paper.’’ 

Dr. Bates is a well-respected scientist who was given one of 

NOAA’s highest awards for developing standards for preserving climate data records. Are AAAS and Science committed to taking his 

allegations seriously and launching a thorough investigation? 

Dr. HOLT. I think you’re referring to the Karl paper that appeared in Science a couple of years ago, and the blog that appeared 

over the weekend—this past weekend by one former scientist Mr. 

Bates—Dr. Bates from NOAA. Dr. Bates said in an article published today the issue here is not an issue of tampering with data. 

He does not—then further it is written he does not believe that 

they manipulated the data. All he is doing ‘‘is calling out a former 

colleague for not properly following agency standards.’’ This is not 

the making of a big scandal. This is an internal dispute between 

two factions within an agency. 

Chairman SMITH. Right. I—Dr. Holt, I’ve—— 

Dr. HOLT. There’s nothing in the paper, the Karl paper, that, at 

our current analysis, suggests retraction. 

Chairman SMITH. Well, I am—— 
54 

Dr. HOLT. We are always looking at our papers to see whether 

there is anything—— 

Chairman SMITH. Dr. Holt, my time—— 

Dr. HOLT. —truly erroneous. 

Chairman SMITH. My time is up. I encourage you to talk to Dr. 

Bates because everything that I have read that he has said about 

the Karl report suggests to me that NOAA cheated and got caught. 

They did falsify the data to exaggerate global warming. The Karl 

study cannot be replicated because, supposedly, the computer 

crashed maybe like the computer at the IRS and the EPA. And 

clearly, he suggests that the Karl study violated scientific integrity 

rules. To me, all that adds up to an investigation and possibly a 

retraction. I just simply ask you to look at it. It may even be a lot 

more serious than you think. 

Thank you, Dr. Holt. That concludes my questions. And the 

Ranking Member, the gentlewoman from Texas, is recognized for 

hers. 

Ms. JOHNSON. Thank you very much, Mr. Chairman. And at the 

end of my statement, I’d like to submit to the record a statement 

related to this very discussion with Dr. Bates. 

Chairman SMITH. Okay. Without objection. 

Ms. JOHNSON. We hear a lot, Dr. Holt, of artfully crafted positions from nonscientists invoking science and the scientific process 

to support positions. Unfortunately, this is the kind of tactic that 

can generate confusion and doubt about the actual state of the scientific consensus. As a scientist and a former Member of Congress, 

how should science inform policymaking? And how can we as policymakers be sure that an agency like EPA is making agency actions based on the best available science? And how should we view 

disagreements between scientists over a particular study? And is it 

dangerous to assume that any such disagreement undermines the 

overall findings of such a study? 

Dr. HOLT. Thank you, Representative Johnson. 

The approach is to ask whether the procedure, the process of 

science has been followed, not to second-guess the results or let 

one’s dislike for an outcome lead one to challenge the outcome for 

unscientific reasons or to challenge the process because sometimes 

the results are either unclear or unpalatable. 

The process, as I was saying earlier, for developing regulations 

at EPA, is based on legislation that holds the science to be paramount. And it is I think the job of oversight to make sure that the 

processes of science are working, not to try to reform them or substitute other processes. 

So, for example, whether health studies that are used for regulations are based on something other than peer-reviewed scientific 

research, not whether different standards of releasing personal information should be used. It is whether the standards of the field 

are being observed. So, for example, the Harvard health study that 

was used for regulations on air contaminants had some data, personal information about deaths, about families that would not be 

made available on the internet openly. That’s not to say that 

there’s any conspiracy here of hiding data. 

Ms. JOHNSON. Thank you very much. 
55 

Also, as a scientist and academic, you’ve had the opportunity to 

help shape the educations and careers of many young scientists 

and engineers who were interested in pursuing careers in the 

STEM fields. What type of impact would undermining the integrity 

of the scientific process by casting doubt on the accuracy or type 

of data that the EPA collects have on future scientists in the environment and health professional fields? And what impacts would it 

have on scientists and engineers broadly? 

Dr. HOLT. Well, I’ve certainly talked with scientists who find it 

uncomfortable and unattractive to work in fields where they feel 

they are constantly second-guessed by politicians and not in a 

sense allowed to operate freely in their work. That has a—sometimes a chilling effect, but at least it generates a level of discomfort 

that I think probably drives some scientists to go into areas that 

are maybe less relevant to public policy. And that of course is a 

real loss. We need the best science applied to things that effect people’s health and livelihood. 

Ms. JOHNSON. Thank you very much. My time is—— 

Chairman SMITH. Thank you, Ms. Johnson. 

The gentleman from Oklahoma, Mr. Lucas, is recognized for his 

questions. 

Mr. LUCAS. Thank you, Mr. Chairman. 

Mr. Holmstead and Dr. White, last Congress the House passed 

one of my bills, the Science Advisory Board Reform Act. I’d like to 

visit with both of you about that body and where it might be improved. I’m concerned that the body has become perhaps an echo 

chamber for the EPA. Would you both agree with that concern? 

And how else would you suggest that the EPA better utilize the 

Science Advisory Board? 

Dr. WHITE. Well, maybe I’ll start. 

Mr. LUCAS. Please. 

Dr. WHITE. So, yes, the Science Advisory Board is one mechanism that the EPA has to conduct its peer-review process. As I 

mentioned in my comments, the peer review needs to be independent and objective. That means the making sure that the peerreview process has appropriate expertise, also the depth in the 

committee to evaluate the assessments at hand. 

There also needs to be clear balance and—in the reviews and the 

peer reviews of the people participating in that process. So you 

need to make sure that there’s—if there’s conflict of interest on the 

panel or if folks have actually made specific recommendations 

about a particular chemistry that they’re evaluating on that panel, 

that that also is balanced in the review. 

What we’ve often seen most recently in the peer reviews conducted by the SAB is that EPA plays a prominent role in that peerreview process. So oftentimes, the conversations that are happening in that peer review get stymied by EPA’s input during the 

peer-review process so it’s not as independent as it should be. 

Additionally, currently in the peer-review process conducted 

under the SAB, there’s not really an adequate opportunity for the 

peer-review panel to address or respond to peer review and public 

comments. And so there will be a large set of public comments 

that’ll be submitted into the process for the peer review, and that 

information is often discounted or ignored. And there’s really no vo-
56 

lition by the peer-review panel or indication by the EPA that they 

need—that the peer review needs to address those public comments 

in any way. 

And finally, when the peer review actually generates their report 

and makes recommendations into the EPA, there’s not a volition 

that the Agency has to abide by all of the recommendations that 

have been identified by that peer review. So there seems to lack 

a check-and-balance of—the peer review makes strong recommendations for improving the scientific quality of a particular 

assessment, but that information doesn’t really get incorporated 

wholeheartedly into the final assessment. 

Mr. LUCAS. Mr. Holmstead. 

Mr. HOLMSTEAD. I’ll give just a quick answer. I do think it’s 

problematic that it’s the EPA Administrator, based on EPA staff 

recommendations, that make all these appointments. Because 

that’s the way the process works, the folks appointed often have 

views that are consistent with the views of the EPA staff. And 

there’s nothing wrong with that, but there does need to be additional views involved when there are other serious scientific points 

of view that are not represented on the panel. So I think there does 

need to be a way to ensure that there’s a little more balance and 

that the science isn’t so one-sided. 

I also want to just quickly make a—raise a particular concern. 

One of the subsidiary bodies underneath the SAB is known as the 

Clean Air Science Advisory Committee, the CASAC. By statute, 

they are supposed to look at certain issues and advise the Administrator on those issues, including the CASAC is supposed to advise 

the Administrator on the adverse economic and energy effects of 

certain regulations. CASAC has refused to do that for many years. 

Finally, the CASAC Chairman said I understand we have that obligation; the problem is we don’t have anybody on our committee 

who has expertise in that, and so I look to the Administrator to appoint someone. And a few days later the Administrator said she 

had no interest in appointing anybody to do that issue even though 

that’s supposed to be done by the statute. 

And so there needs to be a way to ensure that appointments are 

actually appropriate to look at the issues that Congress and others 

asked these committees to look at. 

Mr. LUCAS. Clearly, Mr. Holmstead and Dr. White, the general 

public that depends on us to analyze the results of these studies, 

that depends on the EPA to formulate policies based on the input 

from these various groups, it doesn’t necessarily matter so much to 

my constituency who is in charge, and that changes from Administration to Administration. But the quality of that input needs to be 

something that folks can be reassured about back home. 

So clearly, you both described a circumstance where work needs 

to be done, and I think the Committee will work legislatively on 

that. 

And with that, Mr. Chairman, I’ll yield back the balance of my 

time. 

Chairman SMITH. Thank you, Mr. Lucas. 

Without objection, I’d like to make a part of the record the entire 

comments by Dr. John Bates that were posted on February 4. The 

headline is ‘‘Climate Scientist Versus Climate Data,’’ as well as an 
57 

article by Timothy D. Clark called ‘‘Science, Lies, and Video-Taped 

Experiments.’’ 

[The information appears in Appendix II] 

Chairman SMITH. And the gentlewoman from Texas is recognized. 

Ms. JOHNSON. I’d like to also place in the record Dr. Bates today 

said that there was no manipulation. I would like to enter this article into the record, which contains his quote. 

Chairman SMITH. Without objection. 

[The information appears in Appendix II] 

Chairman SMITH. And the gentleman from Illinois, Mr. Lipinski, 

is recognized. 

Mr. LIPINSKI. Thank you, Mr. Chairman. 

Dr. Holt, I want to thank you for your work here in Congress 

previously and your work at AAAS. 

As a scientist, although some people would—I’m a social scientist 

so some people question that, but as a scientist, I understand the 

important role of science. And I think we all agree that science 

should be what guides us in making decisions where that is appropriate. Politics should not be what is trumping science. 

I want to give you an opportunity—I don’t know if there was 

anything else that you wanted to add about the controversy with 

Dr. Bates because I think you—the Chairman had run out of time. 

If there was anything you wanted to add to that or—— 

Dr. HOLT. Thank you. Of course, I don’t know the internal disagreement, and it quite clearly is largely an internal disagreement. 

Dr. Bates was not part of one research group that produced the 

Karl—the data that became the basis for the Karl, et al., paper. 

And he does not like the way the other group conducted their research. 

What’s most important is not that there was an internal disagreement within NOAA about how to handle this but that a number of other studies, including one most recently in one of AAAS’s 

other publications Science Advances, have replicated the work, 

have come up with the same conclusions so that, you know, it’s— 

it is not profitable, considering the process of science, to dwell on 

this one internal dispute about how to handle the data. If the Inspector General at NOAA wants to look at that, that’s fine. 

If there are indeed erroneous or deceptive procedures followed in 

answer—in further answer to Chairman Smith, I know our journals will consider retraction, but there just doesn’t seem to be anything like that now. There’s nothing that suggests there are problems with that work. And most importantly, even Mr. Bates says 

this does not change the policy-relevant conclusions about climate 

change. So I think that’s most important to get out there. 

Mr. LIPINSKI. Thank you. And while the—Dr. Holt, while the 

EPA is generally thought of as a regulatory agency—we’re talking 

about research here—we know that it’s research and science that’s 

led to a number of important findings such as the link between air 

pollution and cardiovascular health and also has led to research at 

EPA that’s led to innovations such as the development of low-cost 

sensors for nutrient pollution in our nation’s water. So it’s important that we understand that part of the Agency’s mission is not 
58 

just to regulate but also do research as another way to help protect 

health. 

Are we not sacrificing potentially some of this important work by 

meddling in EPA’s practices or cutting its budget or, as some in 

Congress have—would like to do is completely cutting out the EPA? 

What would we be losing in research—in research, you know, even 

take it away right now from research on climate change but other 

research and the developments that have come out of that—out of 

the EPA? 

Dr. HOLT. I think it is very important to have a scientific basis 

for the regulation for EPA to conduct research intramurally, to 

sponsor research outside of EPA because—take the National 

Science Foundation for example. They do not—you know, they do 

basic research that ultimately may affect toxicology studies and so 

forth, but they don’t do toxicology studies. They don’t do air pollution epidemiology studies. Those are things that EPA can best do 

both with internal scientists in an intramural program and 

through external sponsorship. And that would include, I would 

argue, social sciences. We at the AAAS believe that empirically 

based answers to questions that can be verified are science. That 

includes social science, as well as the earth sciences and physical 

sciences. 

Mr. LIPINSKI. Thank you. I yield back. 

Chairman SMITH. Thank you, Mr. Lipinski. 

And the gentleman from California, Mr. Rohrabacher, is recognized. 

Mr. ROHRABACHER. Thank you very much, Mr. Chairman. And 

I’d like to, first of all, thank you, Mr. Chairman, for having the political courage to actually look into something that obviously is 

going haywire in our country. I mean, we—there’s obviously something wrong in the arena, the scientific arena in our country. We 

have people telling us global warming is causing the drought in 

California. Now, it’s global warming is causing the flooding in California. And whatever malady we have we end up having people 

telling us that the amount of CO2 that’s being put into the atmosphere is the ultimate culprit. 

And then we have all of these contradicting results. And that’s 

what this is all about today is whether or not there really was a 

pause in the actual increase in the temperature of the planet since 

I think 1998 I guess it was. And the fact is, since 1998, we have 

had large increases in CO2, so if there had been a leveling off or 

it actually declined in temperature in some areas, that would mean 

the CO2 theory is wrong. Okay. That’s a basic scientific area that 

needs examination and truth associated with it. 

Because there’s been so many things that have been told to us 

that seem contradictory, we have every reason to be skeptical that 

our scientific community is maintaining its integrity. Why wouldn’t 

it? Why would there be some sort of loss of integrity among America’s scientific areas? Eisenhower warned us of that years ago. Everybody always remembers Eisenhower warning us, and this was 

exactly correct, about the military industrial complex. And I will 

tell you that I think his words of warning there are something we 

should look at today because today, it even threatens the wellbeing of our country. 
59 

But he also warned us about how government contracts become 

virtually a substitute for intellectual curiosity and warned us ‘‘The 

prospect of domination of the nation’s scholars by federal employment, project allocations, and the power of money is ever-present 

and is gravely regarded.’’ And just as he warned us against the 

military industrial complex, he warned us that our scientific—the 

integrity of our scientific endeavors could be undermined by this 

very same type of government interaction with our science community. 

And I think that you, Mr. Chairman, have been courageous in 

taking this on because there are people and very powerful forces 

at work in this world today that have their own agenda and are 

trying to justify it based on manipulating science, basically manipulating the people who did or did not get the scientific contracts to 

do specific research projects over the last 20 years. And I would 

hope that we’d actually call Mr. Bates in to testify. I hope that this 

is the beginning hearing to try to determine truth. 

What counts as truth? What counts is whether or not we have 

people whose scientific findings have actually been influenced by 

whether or not they get a government contract for research. And 

the story—there are stories of this that a number of scientists have 

come to me telling me where before a certain time period they were 

receiving government research contracts. Afterwards, after it became clear they didn’t agree with the CO2 theory, no more contracts. 

Well, this is all about the integrity of science in the United 

States of America. Dr. Holt, would you like to refute everything I 

just said? 

Dr. HOLT. In nine seconds, Mr. Rohrabacher? 

Mr. ROHRABACHER. That’s it. 

Dr. HOLT. Okay. 

Chairman SMITH. The gentleman’s—— 

Dr. HOLT. There has been no pause. It has been examined literally from many perspectives, and scientists are fiercely independent. They would resent horribly if they felt their work was 

being manipulated. It’s not. 

Chairman SMITH. The gentleman’s time has expired. 

The gentlewoman from—I was looking for the gentleman from 

California, Mr. Bera, who’s not here. The gentlewoman from Connecticut, Ms. Esty, is recognized for her questions. 

Ms. ESTY. Thank you, Mr. Chairman. I want to thank our witnesses and our former colleague, Dr. Holt, for being with us here 

today. 

I want to follow up a little bit, Dr. Holt, with the scientific process. I think there—I hope and believe there is agreement on this 

Committee that sound science is absolutely essential to good decision-making, whether it’s by the EPA, any other administrative 

agency, or Members of Congress. 

So I’d like to ask you a little bit—so there’s been a lot of criticism 

from opponents of EPA regulations that sometimes EPA gives more 

weight to some studies than to others. Now, some see that and 

have characterized that as showing bias or favoritism. Can you explain a little bit more within the scientific process whether that’s 
60 

appropriate and why that might be appropriate to give more weight 

to some studies than others? 

Dr. HOLT. Thank you, Representative Esty. 

Some studies are extensive, longitudinal, long-range, comprehensive studies. Others are more superficial. Which one do you think 

would be—which ones do you think would be more useful for making decisions about your children’s health? It is true that sometimes short, brief studies uncover things and then they should be 

explored. 

My—in my opening remarks I was saying if you want to be on 

the right side of any issue, you would do well to go with the evidence, the best understanding at the time of what it’s going to be, 

not some fringe idea. Scientists are always poking around the periphery trying to find new understanding, but we shouldn’t think 

that that is the center of gravity. 

You know, take the climate change issue we’re talking about this 

morning. This is an internal dispute about a detail of how you 

might measure land temperatures or water temperatures. It is not 

a departure from the general understanding of what’s happening to 

temperatures in our globe. And so some studies are indeed more 

worthy of trust and more—a better basis for sound regulation than 

others. 

Ms. ESTY. I’m glad you raised some of that because I am—live 

in the State of Connecticut, and we have very high rates of asthma. 

We are downwind from many of the power plants that have caused 

real problems. And we see it in higher cardiopulmonary issues; we 

see it in asthma in children. There’s a great deal of concern in my 

State that some of the longitudinal studies will not be allowed anymore under some of the proposals we’re looking at in secret science, 

the—we have—the American Thoracic Society has provided some of 

that information. 

Can you explain and flesh out a little bit more—you referenced 

the longitudinal studies, and I know in my State there’s a great 

deal concern because we’ve seen real health benefits from some of 

those changes that EPA has promulgated based on these kinds of 

studies. Can you flesh that out a little bit what that would mean 

for those studies? 

Dr. HOLT. Thank you. You know, the—I guess you’re referring 

more to the Secret Science Acts of previous Congresses. Now, I 

don’t know what the Chairman or others will be proposing in this 

Congress, but in the past, the legislation, in the name of providing 

openness, has meant that certain kinds of studies could not be considered because some of the data in those studies—and I mentioned 

this Harvard study on atmospheric pollution—well, actually on 

community health—could not be used. 

It would also, for example, hinder fast response. For example, 

some of the data from the Freedom Industries spill on the Elk 

River in West Virginia recently—I mean, a couple of years ago— 

some of that data could not or was not publicly disclosable, and 

that hindered EPA in their response. Chairman Waxman at the 

time actually contacted the manufacturer of the chemical to find 

out what it was and—but the secret science research—Secret 

Science Act probably would have prevented that fast action. 
61 

Chemically induced birth defects, no family wants the newspapers or the web pages to be listing information about their kids’ 

birth defects. But if you’re going to study birth defects you’ve got 

a look at actual kids and study the epidemiology. The Secret 

Science Reform Act presumably would have prevented that kind of 

necessary research. 

Ms. ESTY. Thank you very much, and I see my time is expired. 

Chairman SMITH. If the gentlewoman would yield real quickly, I 

think Dr. Holt knows as well as anybody that there’s such a thing 

as redactions, and if it’s personal information, that information can 

be redacted. 

I will go now to the gentleman from Florida, Mr. Posey, for his 

questions. 

Mr. POSEY. Thank you, Mr. Chairman. 

Dr. White and all of you, I think we’re all concerned when agencies promulgate rules that they say are based on sound data, and 

when they’re asked to share the data, they say no, we’re not going 

to tell you what the data is, you just have to accept our word for 

it. 

So, Dr. White, your written testimony you highlighted the peerreview process as a critical tool for insuring government policy is 

informed by sound science. I think we can all agree on the importance of objective peer reviews, which is why I’m alarmed by some 

of the reported problems with the current peer-review process at 

EPA and other agencies. Can you briefly explain why a transparent 

peer-review process is so important to ensuring the quality of scientific information? 

Dr. WHITE. Sure. Thank you very much, Congressman, for that 

question. 

So as I mentioned, peer review is a critical piece of the science 

evaluation process. Having a third-party group of scientific experts 

evaluate and look at EPA’s assessments is very important to establishing trust from the public. And making sure that you have the 

appropriate expertise, one, on that peer-review panel, that the EPA 

actually heeds the advice that they get from those peer reviewers 

and incorporates that information into their final assessment is 

critical. I think what we’ve seen is that that has not always been 

the case for the peer-review process and we’d like to see that 

change. 

Mr. POSEY. Okay. Follow-up, given what you’ve just told us, 

should agencies strive to allow the public to incorporate information from peer reviews into their comments on a proposed rule? 

Dr. WHITE. So the Agency definitely should take into consideration the public comments that it receives. One part of the peerreview process is getting in public comments. Those comments 

come from scientific experts that have evaluated the available data 

and provided that information into the Agency. Unfortunately, in 

some of the programs, while they get the information that has been 

provided by the public, they don’t incorporate or respond to that information. 

Specifically, in EPA’s Integrated Risk Information System one of 

their processes in step five is actually a disposition of the public 

comments so actually writing down how they will respond or address peer review and public comments. Unfortunately, in the last 
62 

three assessments that they put out they have not responded to 

those public comments, and so again this is kind of a violation of 

their own processes that they have currently in place. 

Mr. POSEY. Thank you very much, Dr. White. 

Dr. Holmstead, continue on the same point. Can you discuss how 

both the scientific and rulemaking process is injured when interested and qualified parties are unable to comment meaningfully on 

science supporting a proposed rule? 

Mr. HOLMSTEAD. Well, thank you for your question and for calling me doctor. Unfortunately, I’m the only one on the panel here 

who is only a lawyer and— 

Mr. POSEY. We don’t need to waste time on that. Go ahead. 

Mr. HOLMSTEAD. In both policy questions and science questions 

I’ve certainly seen that thoughtful discussions from different points 

of view leads to better outcomes. And I just think it’s—I think both 

on the regulatory side and the science side that EPA and the public 

would be better served if we actually had people who were wellqualified, who have different perspectives and different points of 

view, if they can try to resolve some of these issues instead of 

stacking some of these groups with people who share EPA’s view. 

Mr. POSEY. Okay. Well, thank you, Mr. Holmstead. 

Before my time runs out I’d just also like to thank the Chairman 

for the opportunity to explore this issue. I know a lot of us have 

been concerned about this, and a lot of our citizens are threatened 

by a torrent of bureaucratic attacks that they don’t understand 

that affects their lives and their livelihoods every day. And, Mr. 

Chairman, I just again want to applaud you for taking this issue 

on. 

Chairman SMITH. Thank you, Mr. Posey. Since you have a little 

time left, let me suggest to Mr. Holmstead that he seize on the 

juris doctor and start something new and go with the doctor. 

The gentleman from Virginia, Mr. Beyer, is recognized for questions. 

Mr. BEYER. Thank you, Mr. Chairman. 

You know, a lot of attention has been paid to the title of today’s 

hearing ‘‘Making the EPA Great Again.’’ Under President Obama, 

the EPA took aggressive action to tackle climate change, which 

most scientists agree is the major threat to the planet and to the 

human race. It will not help anyone by disputing climate science 

with stories from white nationalist websites like Breitbart.com or 

tabloids like the Daily Mail. 

Scott Pruitt, the President’s nominee to head the EPA, has sued 

the Agency 14 times over its effort to regulate the oil and gas industry and has not said whether he would recuse himself from ongoing cases against the EPA. This is not a recipe for greatness. 

The new Administration has scrubbed the EPA’s website, has frozen its grants and contracts, has placed what amounts to a gag 

order on EPA employees, and requested names of employees who 

worked on climate change. A Trump transition official suggested 

cutting the Agency’s workforce by 2/3 in a bill introduced by a fellow Republican will eliminate the EPA altogether. Mr. Chairman, 

this is not greatness. 

Alarming environmental disasters like revelations on the consequences of pesticides and cities covered in smog and rivers catch-
63 

ing on fire spontaneously drove a Republican President, President 

Richard Nixon, to create the Environmental Protection Agency. 

And since then, the EPA’s legacy has been one of great achievement. Among its most important accomplishments, the EPA 

banned the pesticide DDT. It significantly reduced levels of sulfur 

dioxide and nitrogen dioxide, the leading ingredients in acid rain. 

It mandated lead-free gasoline. It regulated toxic chemicals and established a national commitment to protecting our air and our 

water. 

The EPA’s achieved so much in recent years, often over the misguided objections of this Committee and a Congress which has 

tried to make things worse. Mr. Chairman, Members of the fellow 

Science Committee, please, please listen to the Science Committee. 

Climate change is real and environmental problems can’t be wished 

away or pretended away because they’re going to affect us, our children, and future generations. 

This Committee should be leading the charge to protect the planet and our environment for future generations and instead, it attacks the credibility of scientists, casts doubt on accepted science, 

and makes life difficult for the people trying to solve urgent crises. 

The Science Committee’s contribution now is like that of the Emperor Nero fiddling while Rome burned down around him. This is 

irresponsible and dangerous. It is not leadership and will not make 

the EPA or America great. In fact, I think we should retitle this 

hearing, that we keep the EPA great or maybe just save the EPA 

or protect our water. 

So with that, Dr. Holt, let me actually pose a question. Dr. 

Belzer said, and I quote from his written testimony, ‘‘The EPA will 

strive for the highest estimate of risk that does not bring upon the 

Agency unbearable ridicule.’’ How would you respond to Dr. 

Belzer’s comment that they always take the most extreme version 

of risk in their calculations? 

Dr. HOLT. You know, I’m not a historical scholar of EPA, but I 

can say that in the history that you touched on, because these— 

this legislation from past decades was based on science, it has 

managed to keep up with growing knowledge and improved understanding of human health and the effects on human health and on 

the environment of various kinds of water and atmospheric contaminants. And the record is very good. 

So, you know, I—I’m not a—you know, Dr. Belzer is—he is a 

scholar of benefit-cost analysis. He wants it phrased that way I 

think. I would not say that I am. But I do look at the results, and 

the results have been good. 

Mr. BEYER. Okay. Thank you. 

Juris Dr. Holmstead, in your testimony you wrote that referring 

to the Science Advisory Board Act that we took up in the last Congress, that by focusing on disclosure rather than disqualification 

that we would ensure that any potential conflicts, financial and 

otherwise, are publicly disclosed. But one of the objections we had 

to the act last time was that it in fact didn’t do full financial disclosure. Are you aware that what will be intended this year will actually prompt full financial disclosure for people that would be appointed to that board? 
64 

Mr. HOLMSTEAD. You know, I don’t know anything about the bill 

that may be introduced this year, but I do believe that full disclosure, not only of financial conflicts but other financial interests, is 

an important part of the process. So again, I’m not familiar with 

legislation that may be forthcoming from this Committee, but I do 

think that’s an important thing. 

Mr. BEYER. I hope you don’t mind us using your testimony when 

the bill comes before us later so—thank you. Mr. Chair, I yield 

back. 

Chairman SMITH. Thank you, Mr. Beyer. By the way, my cap is 

going to read ‘‘Keep the EPA Honest.’’ But I appreciate the gentleman. 

The gentleman from Texas, Mr. Weber, is recognized for his 

questions. 

Mr. WEBER. Thank you, Mr. Chairman. 

To my friend there from Virginia I want to say I’m not going to 

quote any of those websites that you did. I’ll choose an article by 

The Hill, not necessarily known as a right-wing newspaper, 

‘‘Drowning by EPA Overreach,’’ June the 17th of ’15 by Will 

Coggin. ‘‘The EPA recently found itself in hot water. The New York 

Times revealed the Agency colluded with environmentalist groups 

in a campaign to manufacture public comments in favor of a new 

rule that expands its own power. The Agency’s actions and the shenanigans of its environmentalist supporters shed light on how a 

bad rule can flow through a regulatory process,’’ not exactly your 

right-wing publication. 

But it does point to the fact of why there’s questions and why, 

with legitimate oversight, notwithstanding my friend’s comments 

about this Committee was wasting its time, why we have a legitimate oversight role to play and we should be and that doesn’t 

make us a useless Committee. 

And I applaud the Chairman on his efforts to make sure that we 

hold the EPA accountable. 

Quick, Mr. Holmstead, how did he term you? My dearest 

Holmstead? Was that what he said? 

Mr. HOLMSTEAD. Most—many people refer to me that way. 

Mr. WEBER. I’ve got that, and I’m glad to hear you all have an 

intimacy there that few do. 

The EPA will often bury—well, let me do this. Adam Peshek 

from another publication said sometime back in 2011—actually, he 

said in an article two examples of EPA overreach that ‘‘Measures 

taken to protect the environment are necessary and welcomed, but 

concerns for air quality should always be measured against the 

larger context of the economy and real-world achievability.’’ So I 

think the fact that we’re here having this discussion gives us pause 

for concern that we ought to be able to have these discussions and 

question the science. 

The EPA will often bury the cost of its regulations while inflating 

benefits. Measure what Adam Peshek said against real-world 

achievability. Do you think, Mr. Holmstead, it would improve regulations if EPA were more transparent in its cost-benefit analysis? 

Mr. HOLMSTEAD. I do believe that transparent cost-benefit analysis is important. I also believe it’s important that the Agency fairly present the results because in some cases, if you dig into the de-
65 

tails, you’ll find that EPA has been relatively transparent, but the 

way they present their conclusions gives you a very different picture than—— 

Mr. WEBER. You can understand why, when articles come out 

from The New York Times that they’ve colluded with environmentalist groups to further their own power-making authority, 

their own rulemaking authority, that the Chairman of the Science 

Committee might have cause for concern and say we might ought 

to have some oversight of that. You’d understand that? 

Mr. HOLMSTEAD. No, absolutely, and I do think that was inappropriate. I was surprised to hear about that comment. 

Mr. WEBER. Thank you. Let me move on to the next. Over the 

past eight years, based on what we just talked about, EPA has 

skewed its regulatory cost-benefit analysis to accomplish policy 

goals. Do you think this undermines public confidence in their 

analysis and could you understand why? 

Mr. HOLMSTEAD. I think that the short answer is absolutely. It 

would be better for everyone, for the public, for all of us in this 

room if there could be more confidence in some of the conclusions 

that we get from EPA and other regulatory agencies, and that’s 

why I think some of the reforms that you’re talking about are very 

important. 

Mr. WEBER. Thank you. I appreciate that. And I’m going to go 

on to Dr. White. 

Dr. White, I’m concerned that the EPA has stacked its Science 

Advisory Board and Clean Air Science Advisory Committee with 

supporters, those that we talked about. The New York Times article said they colluded with environmentalists, for example. The 

EPA rolls out a regulation that says that it’s supported by its scientists but no one is there to offer an opposing view, no one. So 

do you think there should be more balance on these advisory committees in your opinion, Dr. White? 

Dr. WHITE. Absolutely. Balance is one of the keys that’s important for having a peer review. You want to make sure that you 

have enough folks on the peer-review committee and that there’s 

appropriate balance so that you have discussion about what EPA 

has done. If you only have one side of the story being told at the 

meeting or you only have one set of views on the peer-review panel, 

then you’re not getting a full picture and you’re not really having 

a robust peer review. 

Mr. WEBER. So to you as a scientist, what does that say to you 

about those scientists that they really believe in an open and fair 

process or that they’re pretty much consigned to just those who 

support their already predetermined analyses, for example? 

Dr. WHITE. I think what it shows is that there’s just not appropriate balance on the committee—— 

Mr. WEBER. Yes. 

Dr. WHITE. —and so you really have to make sure that if you 

build a committee that only has one set of views, then you’re likely—— 

Mr. WEBER. Right. 

Dr. WHITE. —going to get a certain answer—— 

Mr. WEBER. You need a devil’s advocate. 

Dr. WHITE. Right. 
66 

Mr. WEBER. Yes. Thank you. I yield back, Mr. Chair. 

Chairman SMITH. Thank you, Mr. Weber. The gentlewoman from 

Nevada, Ms. Rosen, is recognized for her questions. 

Ms. ROSEN. Thank you, Mr. Chairman. And I want to thank everybody here today for their thoughtful testimony on what is clearly a very controversial, and all slogans aside, we need to listen to 

the scientists who create a hypothesis, go through scientific method, come up with a conclusion. No matter who we have on the 

panel, there really is a process there and that’s what we need to 

pay attention to. 

So my question is for Dr. Holt. Although we would never want 

to replicate or should we replicate natural disasters or manmade 

disasters like the Gulf oil spill, an earthquake, what have you, I 

want you to talk a little bit about how excluding one-time events, 

things that can’t be repeated nor should be, will impact the EPA 

and we won’t get the maximum scientific return if we don’t study 

them. 

Dr. HOLT. Thank you, Representative Rosen, and I’m pleased 

you’ve chosen to serve on this Committee. 

It’s not just one-time events in the cases of emergencies, for example, disasters. Many studies cannot be repeated in exactly the 

same way. The populations have changed. Those people have 

grown up or moved away or the forest that you’re studying has 

been overtaken by an invasive. Whatever it is, you sometimes cannot repeat it the same way. And the Secret Science Act is based— 

as it has previously been introduced has been based on a misunderstanding of how science works. 

You—the gold standard is to find other approaches to come up 

with the same conclusions. Rarely can you repeat an experiment in 

exactly the same way, and so this cry that you have exact data that 

somebody else will take and put it through their computer instead 

of your computer, yes, sometimes that makes sense, but what 

makes much more sense is that you approach the problem with a 

new perspective. 

And so that’s not what the—that’s not where this secret science 

legislation is heading. It’s a misunderstanding of what it means to 

replicate experiments. So I think that—well, anyway, you’ve said 

it. 

Ms. ROSEN. Well, thank you. So I guess you would say then that 

the goal of science is to take these studies, collaborate, innovate, 

and inspire the next scientist to take the foundation of what you 

have put there to look at new hypotheses and new ways to find solutions or analysis of a situation. 

Dr. HOLT. Science doesn’t make progress by doing the same thing 

over and over again. Science makes progress by looking at problems from new perspectives and testing it this way and that way 

and the other way and ultimately converge on an understanding 

that is more reliable than you have from one experiment. And 

that’s how science works. 

It also means, of course, that the conclusions might have to be 

refined. The—you know, I know it bothers some Members of this 

Committee that sometimes they hear that different standards 

might be set. Well, yes, as the science progresses, you may change. 

Ms. ROSEN. Where the data takes you. 
67 

Dr. HOLT. And you should change—— 

Ms. ROSEN. Yes. 

Dr. HOLT. —but not on the basis of political whim or personal 

preference. 

Ms. ROSEN. Thank you. I appreciate that. And I yield back my 

time. 

Chairman SMITH. Thank you, Ms. Rosen. 

And the gentleman from Arizona, Mr. Biggs, is recognized for his 

questions. 

Mr. BIGGS. Thank you, Mr. Chairman. I appreciate you addressing this issue. I thank each member of the panel for being here 

today. 

I would like to just shift discussion briefly to an issue I think we 

haven’t heard much about today and that’s the Waters of the U.S. 

or the WOTUS rule. The EPA has consistently claimed that the 

WOTUS rule would not significantly expand its jurisdiction, but I 

look upon that claim with some skepticism because of what I’m 

hearing from my constituents in Arizona. 

And so I’m asking Mr. Holmstead, right now, do you agree with 

the EPA that the Waters of the U.S. rule is not a significant expansion of their jurisdiction over waters and, in the case of Arizona, 

dry wash beds? 

Mr. HOLMSTEAD. No, I think it’s pretty clear that the WOTUS 

rule does significantly expand EPA’s jurisdiction. They—what— 

part of their claim is that they’ve cast a broad net and then you 

can come in and somehow be excluded, but that process takes a lot 

of time and effort. But the jurisdiction that EPA claims is certainly 

much broader than we’ve seen before. 

Mr. BIGGS. It seems to speak to an idea that perhaps there can 

be an institutional bias, and I would categorize it as something in 

line with something like institutional maintenance, and in this instance it’s to regulate to expand jurisdiction. And I think you’ve 

just indicated that you agree with that perhaps maybe not that 

there’s a bias but certainly the rule is. Do you see a bias there? 

Mr. HOLMSTEAD. I think institutionally, EPA—and this is not 

just EPA. I think this is regulatory agencies at federal and local 

and state levels, but that there is a tendency for them to want to 

increase their regulatory power basically because they want to 

have an opportunity to impose their own will on many of these 

choices. So I think there’s no question that we’ve seen EPA expand 

its regulatory power or at least try to expand its regulatory power 

over the years. 

Mr. BIGGS. Many manufacturers have indicated they’re going to 

be impacted by this particular rule, the Waters of the U.S. rule. 

Can you explain what areas of manufacturing that might be impacted by this and any other anticipated results that we might see 

of WOTUS being applied to them? 

Mr. HOLMSTEAD. You know, most of the concerns that I have 

heard have been with regard to not existing manufacturing plants 

but people who want to build anything that’s new, whether that’s 

a transmission line or a pipeline or a new facility. If it turns out 

you want to locate whatever you’re building in an area that’s included within this broad definition, it becomes much, much more 

difficult to do that. 
68 

Mr. BIGGS. So I guess, Mr. Holmstead, to be more direct, we’re 

talking really about permitting issues really become a major problem with this expansion of rules? 

Mr. HOLMSTEAD. Yes, that’s absolutely right. The fact that you 

need government approvals to—and in some cases many different 

approvals for many different agencies I think is a shame. It’s become very hard to build new things in this country, and I think 

that kind of permitting reform—not only reforming the WOTUS 

rule but other permitting programs to get them to function more 

efficiently would be a huge step in the right direction. 

Mr. BIGGS. Thank you. Thanks, Mr. Chairman. I yield back. 

Chairman SMITH. Thank you, Mr. Biggs. 

And the gentlewoman from Oregon, Ms. Bonamici, is recognized. 

Ms. BONAMICI. Thank you, Mr. Chair. 

Welcome back, Dr. Holt. It’s nice to see you again. We certainly 

miss you on the Committee, but I very much appreciate the work 

you do with the AAAS. And before I ask my question, I want to 

make sure that all of our colleagues know about the event that the 

AAAS helps to organize each year, the Golden Goose Awards. This 

is an award that is earned by groups of federal hardworking researchers whose seemingly obscure federally funded research has 

led to major breakthroughs in national security, public health, computing, energy, and the environment. So it’s a great annual bipartisan event and I hope all of our colleagues join us to support scientists and federally funded research. 

I want to align myself with the remarks by my colleague from 

Virginia, Mr. Beyer. Dr. Holt, as you know, our planet is facing the 

real consequence of anthropogenic climate change, and I’m deeply 

disappointed that we’re not beginning this session focusing on how 

the EPA can address that critical issue as part of its mission to 

protect human health and the environment. 

In my home State of Oregon, the renewable energy industry has 

created thousands of jobs and is growing, and I wanted you to follow up on—I know Representative Esty asked you about the 

human health effects. Can you also talk about the economic benefits of addressing climate change? 

Dr. HOLT. I know that there have been many criticisms over the 

years of the cost of environmental regulation and the cost of addressing climate change as one of the biggest environmental challenges. And most of that debate pays insufficient attention to the 

cost of not addressing it, the costs in lives and dollars. And that 

goes for climate change, as well as regulations restricting pollutants and other environmental hazards. And there’s an extensive literature on it. It’s difficult, of course, to calculate because some of 

the benefits and some of the costs are second and even third order 

and indirect. 

But it’s pretty clear, I would say, that a cleaner environment 

such as we have obtained through environmental regulation implemented by the EPA—that a cleaner environment is economically 

better in addition to being better for human health. 

And for climate, you know, it remains to be seen how hard we’re 

going to work to bring climate change under control and how expensive it will be if we don’t do a good job—— 

Ms. BONAMICI. And I want to talk—— 
69 

Dr. HOLT. —and the benefits from the industries in trying to do 

a good job. 

Ms. BONAMICI. Thank you. And I wanted to follow up on a discussion that was already brought up today about the—a piece of 

legislation that has been introduced that proposes abolishing the 

Environmental Protection Agency and basically like going back to 

the days when States would regulate clean air and water. 

So I know that under the Obama Administration there were 

partnerships between States and the EPA. This is something that’s 

going to take federal and international solutions. Can you talk a 

little bit about what would happen going back to the days when 

there was no EPA if we let States regulate clean air and clean 

water? 

Dr. HOLT. Well, Representative Beyer has touched on this. Rivers 

caught fire, people lived in levels of smog that we only see in China 

these days. Well, actually they’re probably seeing it a little bit 

worse even. But the—and so my response to that is the regulations 

by and large have worked, and that’s what we should be looking 

at. 

Ms. BONAMICI. And, Dr. Holt, also in the 114th Congress the 

House considered legislative proposals that would’ve applied the research restrictions that are on the EPA from this so-called Secret 

Science Reform Act to research efforts at all agencies. If such a proposal were to become law, what would the effect be on the research 

mission not only of the EPA but our other federal agencies as well? 

And this Committee has jurisdiction over the NSF, for example. 

Dr. HOLT. Well, there isn’t an agency in the government that 

doesn’t have significant science components. We at AAAS wrote to 

every one of President Trump’s nominees for a Cabinet position, 

and I said in the letter ‘‘I hope you realize that you are heading 

up a science agency.’’ The Attorney General, yes, he would benefit 

by having a better understanding of forensic science. The Housing 

and Urban Development, yes, there is a social science and other 

science that is done by the Department but also science that must 

be used by the Department if they’re going to make good decisions. 

The point is science-based policymaking is important in every aspect of our government. And if the scientific process, the free communication, the free collaboration, the ability to operate without intimidation is compromised anywhere, it will hurt our government’s 

functioning, it will harm the economy and human welfare. 

Ms. BONAMICI. Thank you very much. My time is expired. I yield 

back. Thank you, Mr. Chairman. 

Chairman SMITH. Thank you. Thank you, Ms. Bonamici. 

And the gentleman from Indiana, Mr. Banks, is recognized for 

his questions. 

Mr. BANKS. Thank you, Mr. Chairman. And thanks to the each 

of the four of you for being here today to educate us and discuss 

ways that we can look at reevaluating the role of the EPA in 2017 

and beyond. 

Growing up and fighting for and subsequently representing 

northeast Indiana before coming to Congress in the Indiana State 

House has made me aware of the many ways in which the EPA’s 

policies impact hardworking constituents in my district with citizens from every walk of life, from farmers to small-business own-
70 

ers, workers, and even family members. I’ve heard from many of 

these constituents over the years raising concerns directed at the 

broad, burdensome, and relatively clandestine authority exercised 

by the EPA. 

So therefore, I’ve been a firm believer that our government, 

where authorized, should implement environmental policies based 

on sound science that focus on innovation rather than regulation. 

Sound science is the foundation of sound regulatory decision-making. 

So with that, I’m fortunate to have an opportunity on this committee—and I thank the Chairman for giving me the opportunity 

on this committee—to be a part of investigating and reforming the 

EPA’s regulatory power. 

And with that, I will direct my first question to Mr. Holmstead. 

In your written testimony you highlighted the importance of both 

scientific accuracy and transparency in the EPA’s decision-making 

process. On that note, should the risk assessment process and the 

cost-benefit analysis process of the EPA be examined and reformed 

to make it more scientifically based and objective? And with that, 

what should Congress’s role be in reforming that process? 

Mr. HOLMSTEAD. I’ll give you a short answer and you might want 

to ask Dr. Belzer, who really has had much more experience in risk 

assessment. I think it is appropriate for Congress to step in and 

to institute some reforms that could improve the process. And I 

want to be clear. I think EPA does a lot of very important, good 

things, but I also know from my own personal experience that 

there are some things that really do need to be reformed. 

We do enjoy a cleaner environment in large part because of EPA, 

but in some ways we pay a lot more than we should for the protection that we get. There are better ways to do these things, and I 

think that’s one of the things that this Committee is looking at is 

are there better ways for EPA to do its job. 

Mr. BANKS. Okay. Thank you. And, Dr. White, there used to be 

a time where EPA’s Science Advisory Board and Clean Air Science 

Advisory Committee would issue reports with dissenting views. 

Now, it seems like those bodies just come together in a general consensus that almost always aligns with environmental or conservation groups. Do you think the EPA’s Scientific Advisory Boards 

could do a better job at encouraging a broader or more transparent 

range of views? 

Dr. WHITE. Absolutely. The process needs to be balanced. I mentioned that earlier in my testimony today that if you have balance 

on the committee, you’re going to get dissenting views, and that information needs to be captured and considered by the Agency. 

That’s really the value of having an open and independent peer-review process is so that you get a really robust review of EPA’s evaluation of the science and what it’s going to use to make those decisions before they are finalized. 

Mr. BANKS. Okay. Thank you. And back to you, Mr. Holmstead. 

Do you think it would help if the EPA presented its regulatory impact analysis in a more concise or easier-to-read format? 

Mr. HOLMSTEAD. You know, I’m not quite sure how to answer 

that question because when you—sometimes when you make 

things too simple, you don’t do justice to the complexities involved. 
71 

So I agree that there’s room for improvement, but again, you want 

to be sure that these documents—that this documentation gives 

people enough information so that they understand that things 

aren’t always quite as black-and-white as the Agency would make 

them appear. 

Mr. BANKS. And then one final question for you that you might 

or might not agree with. The EPA has historically, in my opinion, 

had a shoot-first style of regulating. The Agency issues a regulation 

before it is fully thought through whether it could be accomplished. 

How does this unnecessarily create burdens for businesses and 

manufacturers or would you agree that it does? 

Mr. HOLMSTEAD. Oh, sure, I mean, I think they’re—I’ve long 

been critical of one part of the Clean Air Act that imposes a legal 

requirement on States and local governments that in many cases 

is impossible to meet. It’s just not achievable. And that has all 

kinds of implications for businesses that want to locate there, and 

I just don’t think it makes any sense to give an agency authority 

to mandate things without just—without considering whether 

they’re achievable. 

Mr. BANKS. Thank you. Thank you, Mr. Chairman. I yield back 

my time. 

Chairman SMITH. Thank you, Mr. Banks. 

The gentleman from California, Mr. McNerney, is recognized. 

Mr. MCNERNEY. Well, I thank the Chairman for holding this 

hearing. 

It seems to me that the issues of contention today would be how 

science is conducted, meaning does it follow standards, does it follow standards that science usually follows, is it peer-reviewed, and 

is it influenced by politics or not, and, on the other hand, how 

science is used in rulemaking. 

So my first question will go to Dr. Holt. Would the Secret Science 

Reform Act ensure that science follows science standards that’s 

used at the EPA? 

Dr. HOLT. No, I think the problem is it’s an attempt to substitute 

a different view of what is meant by openness and a different view 

of what is meant by sharing than is the standard in the practice 

of science. And my earlier plea today was that—I’m not saying oh, 

trust the scientists. I’m saying trust the process 

Mr. MCNERNEY. Right. 

Dr. HOLT. —and don’t try to rework the process because you 

think you’ll get better results because you won’t. 

And with regard to the Science Advisory Committee, I mean, 

that is a Science Advisory Board. It will not function better by having fewer scientists on it. It is supposed to look at science. But in 

the name of balance and diversity, it—there’s an effort to make it, 

well, less scientific, and that, it seems to me, not the way to go. 

That’s not what the SAB is for. There might be other places in the 

Department where you bring in industry representatives to talk 

about the cost of the regulations to the industry, but I don’t think 

the Science Advisory Board is the place to do that. 

Mr. MCNERNEY. Well, would the Secret Science Reform Act lessen the influence of politics in the scientific process? 
72 

Dr. HOLT. No, I mean I think it’s fundamentally substituting a 

politically originated revision of the process for the scientific process that has grown up over the ages. 

Mr. MCNERNEY. Thank you. The second part is how science is 

used in rulemaking. My first question goes to you, Dr. Belzer. 

What are the standards used in the cost-benefit analysis? And I 

know you don’t like cost-benefit. I know you like benefit-cost, but 

I’ll use it anyway. What are the standards used for human health 

and human life in cost-benefit analyses? 

Dr. BELZER. We have 30 to 40 years of experience doing that. 

This is within the field of benefit-cost analysis. There isn’t any controversy, regardless of one’s political orientation, as to whether that 

ought to be done. There are technical arguments about how to do 

it, but there isn’t any dispute in the field about whether it ought 

to be done and the effort ought to be put into doing it. 

It’s the—again, think—if you think—— 

Mr. MCNERNEY. Well, I mean, what I’m asking is—— 

Dr. BELZER. —of economics as a scientific field—and I should 

point out that a lot of physicists feel the same way; they become 

economists—— 

Mr. MCNERNEY. Right, but what I’m asking is what are those 

standards? How do you include human life and human health in 

cost-benefit analysis? How do you do it? 

Dr. BELZER. The standard mechanism for dealing, let’s say, with 

premature mortality, which is the largest component of benefits for 

most environmental regulations, is to estimate the number of premature lives that are lost. And there’s premature death and then 

there are—there’s an extensive economic literature on valuing the 

premature death. And so that goes into the benefits assessment. 

And I should point out EPA is very fond of this. EPA uses this 

all the time. This is not a matter of controversy among economists. 

EPA has used this same literature to estimate the benefits of the 

Clean Air Act, and they’ve done so repeatedly. So the controversies 

with that had to do with things that are at a more technical level, 

not at a principal level. 

Mr. MCNERNEY. So what is the cost or benefit of premature 

deaths? 

Dr. BELZER. I’m sorry— 

Mr. MCNERNEY. Well, I mean, how much does it cost? What does 

a premature death cost? 

Dr. BELZER. What does it cost to—— 

Mr. MCNERNEY. Right, in your analysis. 

Dr. BELZER. I’m sorry. I’m mostly deaf, so I need clarity in—— 

Mr. MCNERNEY. So you say it’s just a technical matter. Well, 

then—— 

Dr. BELZER. Yes. 

Mr. MCNERNEY. —what is the cost of a human life in your costbenefit analysis? 

Dr. BELZER. What value—agencies differ in the values that they 

use. I believe that EPA’s figure is on the order of 9 or 10 million 

dollars per premature life—essentially, the value of saving or preventing a premature death. I think the Department of Transportation uses a number that’s quite a bit lower than that. OMB provides guidance on how to do it but doesn’t tell them what to do. 
73 

And the circumstances may well vary so there isn’t a fixed value. 

There are underlying procedures about how to estimate for a given 

situation, and those procedures are pretty much the same across 

agencies. But agencies do differ. 

Mr. MCNERNEY. So there’s leeway then? 

Dr. BELZER. There is leeway but it’s not leeway in terms of the 

methods. It’s leeway in terms of where the data and analysis lead 

you. This is the same principle Dr. Holt is talking about. Economics 

is a science in my view, and so we apply scientific tools, scientific 

method to developing these estimates. 

Mr. MCNERNEY. The gentleman’s been—— 

Dr. BELZER. There are things that are hard to estimate—— 

Mr. MCNERNEY. —generous in letting me run over, so I’m going 

to yield back at this point. 

Chairman SMITH. Thank you, Mr. McNerney. 

The gentleman from Alabama, Mr. Palmer, is recognized. 

Mr. PALMER. Thank you, Mr. Chairman. 

Dr. Holt, and Dr. White, I have very high regard for your professions. I just want to ask you, are you saying that, for instance, the 

scientists who work on the Intergovernmental Panel on Climate 

Change for those whose work has been in regard to climate change 

should be held in high regard subject to same peer review of everybody else but given respect? Is that—Dr. White? 

Dr. WHITE. Yes. 

Mr. PALMER. Okay. How about you, Dr. Holt? 

Dr. HOLT. Yes, I think so. I’m not quite sure I understood your 

question but— 

Mr. PALMER. Well, I’m just asking if these people are—— 

Dr. HOLT. —the scientists who work on the Intergovernmental 

Panel on Climate Change that I know personally I highly respect, 

and when I look at the work that has come out of that, I would 

say that’s good science. 

Mr. PALMER. And would you say that’s true of the other scientists that work on that? I mean, they have to be held in pretty 

high regard to be added to that panel. 

Dr. HOLT. Yes. 

Mr. PALMER. Okay. I just want to introduce this for the record, 

Mr. Chairman, that there’s a number of scientists who worked on 

the Intergovernmental Panel on Climate Change who now fiercely 

dispute what the panel has produced in terms of their projections 

on climate change. In fact, one of them says, ‘‘Warming fears are 

the worst scientific scandal in history. When people come to know 

the truth—what the truth is, they will feel deceived by science and 

scientists.’’ 

So if I may, I’d like to submit that for the record. 

[The information appears in Appendix II] 

Mr. PALMER. Dr. Belzer, there’s been a lot said here about asthma and health as a result of human activity. I just want to share 

some things with you. From 1980 to 2012, our gross domestic product increased 467 percent. Vehicle miles traveled went up 94 percent. Populations increased 38 percent. Energy consumption is up 

22 percent. Emissions, however, are down 50 percent. But the interesting thing is that—and these are EPA stats, these are U.S. 

government stats—is that even though our air quality and water 
74 

quality are demonstrably better, particularly air quality in regard 

to asthma, than they were in the ’60s and ’70s and ’80s, asthma 

rates have exploded. Can you give any explanation for that? 

Dr. BELZER. I am familiar with some of that literature, and I suspect but don’t know that one of the things that’s changed is the definition of asthma has expanded. And if you increase the—if you 

make the definition broader, you’re going to have more people in 

it. So that certainly could be part of it. 

But I do agree that this is a conundrum with declining air pollution that you would have increasing asthma is certainly contrary 

to the expectation and belief of many people who think or believe 

that asthma is caused by air pollution. So this is a problem. This 

is one that good science ought to be put to it, and we ought not 

to impart upon it a policy judgment first as to what the answer is. 

Mr. PALMER. Okay. Here’s what I want to point out. First of all, 

I’m not a climate denier, as some claim. I think that it is caused 

by natural variations. I think there’s enough science out there that 

indicates that that’s a viable position to have. But also in regard 

to what’s been said in this committee that it’s clear, I think, from 

particularly what some of my colleagues on the other side have 

said that they want to link asthma to human activity, pollution. 

And in every respect—I mean, lead, carbon monoxide, sulfur dioxide, the volatile organic chemicals, nitrogen dioxide, PM10, PM2.5, 

everything is down, yet they want to make it about that. And I 

think that’s a misrepresentation of science, the very thing that 

we’re trying to avoid here. 

And I also have an article, Mr. Chairman, I’d like to enter into 

the record from Scientific America to your point, Dr. Belzer. We 

don’t know what causes asthma. There’s everything—hypothesis 

from hygiene to obesity to sedentary lifestyles to more—to poor 

housing quality for lower-income families. So I really do think 

where we’re trying to go with this committee, to be able to validate 

the science, to get the politics out of it is the place we need to be. 

I yield back. 

Mr. LUCAS. [Presiding.] The gentleman yields back. 

Without objection, the gentleman from Illinois, Mr. Foster, and 

the gentlelady from Hawaii, Ms. Hanabusa, are authorized to participate in today’s hearing. They’ve been selected to serve on the 

Science Committee but have not officially yet been appointed. 

Seeing no objection, the Chair now recognizes Mr. Foster for five 

minutes. 

Mr. FOSTER. Thank you, Mr. Chairman. And if it’s—it’s my understanding that just as we speak I think I’m likely to be officially 

appointed to the committee but—— 

Mr. LUCAS. You’re right. 

Mr. FOSTER. Yes. So, Dr. Holt, you know something about 

Science magazine, and there was an interesting article a couple 

weeks ago entitled ‘‘The Polluted Brain,’’ which had a very interesting discussion of the growing evidence for the link between particulates that are about an order of magnitude’s too small to be detected by normal air-quality monitoring equipment and dementia 

and Alzheimer’s and had some actually rather alarming numbers 

in it. The—and as well as an honest discussion of the scientific uncertainty in this. 
75 

And the potential economic impacts of this are huge. About 

roughly 1/3 of all of our Medicare spending is projected to be due 

to Alzheimer’s within a couple of decades. You know, if Alzheimer’s 

did not exist, there would not be long-term financial stress on 

Medicare. And so this is a—it’s a huge issue to the extent that it 

is attributable to this. 

And there’s also difficulty that is traditionally not—these are 

things—instead of 2.5 microns and above, these are 200 nanometers and below. And so that there is a significant belief that they 

penetrate through the blood brain barrier and actually do damage 

to the brain. It’s well-documented that they cause things like asthma, cancer, and recently, heart disease. But now, the fact that they 

could be responsible for a significant amount of that is I think a 

source of growing concern and potentially a subject that this Committee will be discussing a lot and the EPA should be raising. 

And so I raise it because it’s sort of a prototypical example of 

how science is a moving target in environmental regulation. And 

it strikes me that there are two dangers that we have, one of which 

is that, as soon as this danger begins to have some scientific plausibility associated with it, large commercial interests will try to suppress that science. I mean, we saw that in cigarettes and a number 

of other areas. 

And so I was wondering if any of you could comment on the best 

way to prevent that from happening because there will be a large 

number of manufacturing products, consumer products, and so on 

that will be responsible for exposing people to these very small particulates and, you know, this may cause changes in the business 

model. And so I was wondering if you—any of you have comments 

on how to make sure that that does not happen in a way that happened with cigarettes. Or can’t it happen anymore? 

Dr. HOLT. Well, Representative Esty asked earlier why are studies some studies more creditable—credible than others? And, you 

know, one can certainly discount studies that seem to be driven by 

special interests. It’s not enough to just disclose where your funding comes from but that at a minimum should be done. 

But then, as you know in the tobacco case and in some of the 

other cases, it’s dependent on finding forensic evidence, you know, 

internal communications of collusion. That was certainly true when 

Henry Waxman exposed the collusion in the tobacco industry. So, 

you know, there’s no easy way that I know of to do that. 

Mr. FOSTER. And Dr. Belzer, to sort of follow up on Representative McNerney’s question about—you know, there’s a number—I 

think you cited $10 million per human life or some number like 

that. How do you value one year of suffering from Alzheimer’s? Is 

there a number associated with that when we do the cost-benefit 

or is it only deaths that we typically consider? 

Dr. BELZER. It is certainly true that much—or most of the research effort has gone into estimating benefits from mortality. The 

reason—logical one is that mortality is the worst health effect that 

one normally can imagine. I am hesitant to agree with that because 

Alzheimer’s is one of those things that might well be worse for a 

lot of people. My mother endured almost 20 years of it before her 

death in 2012, and it is—— 

Mr. FOSTER. As my mother did. 
76 

Dr. BELZER. —but coming—but figuring out how to estimate it, 

it requires some resources to be devoted to it. I don’t think there 

are any economists who would shy away from the effort. They 

would find it extremely interesting and challenging and would put 

forth the best possible objective effort to do so in collaboration with 

the neurobiologists who would be best equipped to help us. 

Mr. FOSTER. Would that effort to get to the real scientific and 

economic analysis be easier or harder if the size of the EPA staff 

was cut by factor of three? 

Dr. BELZER. I’m not sure that it has any correlation at all with 

the size of the EPA’s staff. 

Mr. LUCAS. The gentleman’s time is expired. 

The Chair now recognizes the gentleman from Texas, Subcommittee Chairman Mr. Babin, for five minutes. 

Mr. BABIN. Thank you, Mr. Chairman. I appreciate it. 

Risk assessment guidelines will enable the EPA to achieve consistency in the conduct of chemical risk assessments and will help 

avoid manipulation of evidence and assumptions to achieve predetermined results. It is my understanding that existing risk assessments guidelines are out of date and do not offer critical guidance that relates to current and evolving risk sciences. 

It’s also painfully apparent that the EPA risk assessors do not 

consistently follow all aspects of their existing guidelines. For example, the Science Committee’s investigation into EPA’s assessment of glyphosate found significant flaws in the scientific process 

of reregistering chemicals that warrants further examination. 

I believe that updated risk assessment guidelines should contain 

clear criteria for causal analysis so that there is as little room as 

possible for subjective judgment that reflects the policy leanings of 

the analyst. The guidelines should identify, through careful evidence integration, the conclusions that have the strongest scientific 

support. 

And I’d like to start with you, Dr. White. Do you believe the EPA 

should update and revise its risk assessment guidelines? Do you 

believe that the development of updated risk assessment guidelines 

and adherence to these guidelines will inject more objective scientific rigor into EPA’s chemical risk assessments? And will that 

make the development of risk assessments more transparent? 

Dr. WHITE. Thank you, Congressman, for that question. Transparent and consistent framework for evaluating chemicals is necessary. It needs to be transparent and, like I said, it needs to be 

consistent and science-based. As the science changed, risk evaluations need to be updated. So do the processes that are used. 

I mentioned 2016 Congress passed the Lautenberg Chemical 

Safety Act, which requires the EPA to make its decisions using the 

best available science and a weight-of-evidence process. What that 

means is a weight of evidence utilizes the—all the available 

science. It clearly identifies the criteria that it will use to identify 

the available data. It evaluates the quality of that data. 

And I think it’s important to note when we talk about the quality 

of the data, just to go back to something Dr. Holt said about studies funded by industry and whether they should be discounted, I 

do not feel that a study should be discounted solely based on the 

funding authority. It should be reviewed and evaluated based on 
77 

the merits of the actual scientific context and the study design and 

the value that that information will provide to the assessment. 

When EPA or any other agency evaluates the available scientific 

information, it needs to determine whether or not they have highquality information, looking across the board at all studies. One of 

the reasons that it’s important to do a weight-of-evidence process 

is because it allows the Agency to look across the board at all the 

available evidence to evaluate the quality of that evidence and then 

use that information to integrate and make decisions. This is information both on available human data, on the animal data that 

might be relevant to human exposures, and any mechanistic data 

that will tell us about how a chemical may be acting in the body. 

Mr. BABIN. Okay. And how would you suggest risk assessment 

guidelines account for uncertainties in the scientific evidence? 

When and how is it appropriate to use default assumptions? 

Dr. WHITE. So when scientific information is available, so when 

we actually have human data or animal data, that information 

should be used instead of a default. When we’re looking specifically 

at uncertainties, sometimes in research we have animal data and 

we may not have available human information. So we need to have 

and be able to account for how that animal data is relevant or not 

relevant to the human exposure incident. 

Mr. BABIN. Okay. And then one last thing. Do you believe that 

exposure assessments should be included in revised risk assessments guidelines? 

Dr. WHITE. Exposure is a key piece of the risk assessment process, so not only do you need to look at the toxicity information but 

also whether or not that toxicity is relevant to actual human exposures. 

Mr. BABIN. Okay. Thank you very much. Would anybody else like 

to add to that? 

Dr. HOLT. Mr. Babin, if I may—— 

Mr. BABIN. Yes. 

Dr. HOLT. Well, first of all, in your first question, it may be true 

that the risk assessment guidelines need to be updated, but I just 

wanted to make sure that it didn’t seem that I was implying that 

industrial research should be discounted on the face of it. Maybe 

Dr. White thought I said that. I don’t—I certainly didn’t mean that. 

Mr. BABIN. Okay. Thank you. Anybody else? With that, I will 

yield that the balance of my time, Mr. Chairman. Thank you. 

Mr. LUCAS. The gentleman yields back one second. 

The Chair now recognizes the gentleman—— 

Mr. BABIN. Generous. 

Mr. LUCAS. —from the big First District of Kansas, Mr. Marshall, 

for five minutes. 

Mr. MARSHALL. Thank you so much, Mr. Chairman. My first 

question is for Dr. White. 

Dr. White, I represent the largest agriculture-producing Congressional district in the country, and I’m so proud that the air we 

breathe there, the waters that my children and grandson swim in 

are cleaner today than they were when I was growing up. As you 

can imagine, our farmers and ranchers utilize quite a range of 

tools, including herbicides and pesticides. 
78 

And kind of a follow-up to Dr. Babin’s thoughts or his question 

on glyphosate, I’m new here so I was trying to understand it was 

in a report, it was out a report, but at the end of the day it looked 

like the conclusion was glyphosate Roundup was not a carcinogen. 

It’s been around since 1960 and somehow that was taken out of the 

report and I’m just trying to fill in a few loose ends. Why was it 

left out of the report or taken out of it? 

Dr. WHITE. So I can’t speak to exactly why it was included or 

taken out of the report, but what I will tell you is this is why it’s 

so important to have a consistent framework for how science evaluations are conducted is that it’s clear and transparent. You can 

see very clearly what processes EPA would have used to evaluate 

glyphosate, what science they used to make their conclusions, and 

it would be right there in front of you so there would be no question about whether or not that data was accurate or valid. 

Mr. MARSHALL. Okay. Thank you. My next question is for Dr. 

Belzer. I’m trying to understand these regulatory impact analyses. 

RIAs is another acronym for me. When they report RIAs to the 

OMB, is there any type of independent peer review before it’s submitted? 

Dr. BELZER. I would say sometimes there is an internal peer review conducted by an agency, but generally, that’s not the case. 

There’s—basically, OMB is performing the peer review. And one of 

the problems with that from my perspective is that OMB doesn’t 

disclose the results of its peer review. It works for the President 

and so its advice is proprietary to the President. I think that things 

would be a lot better if OMB actually disclosed its independent reviews of these documents, and I’ve advocated that since I worked 

there and told to mind my own business. 

Mr. MARSHALL. Do you think independent peer reviews would be 

of any help to maybe not every report but certain number of them 

randomly selected? 

Dr. BELZER. Certainly, the larger ones ought to be. I think that 

the existence of an effective peer-review process improves the quality of what agencies produce in—I mean, just simply the knowledge 

that you’re going to be peer-reviewed goes a long way toward improving quality. Then, as Dr. White’s pointing out, the panel—having a good peer-review panel is very helpful. 

I’m the only person here who is actually a member of a Science 

Advisory Board panel, and I know the processes at least for my 

panel how it goes on. It is a challenging enterprise with a panel 

of 15 members for typically only two might be knowledgeable about 

a given issue that’s actually on the table. 

So I think the peer-review process does need to be restructured 

to make sure that there really are experts in individual areas and 

that they don’t—and that they have intellectual diversity. I don’t 

care at all about who they work for, but intellectual diversity is the 

key toward teasing out the best scientific knowledge. 

Mr. MARSHALL. Okay. My last question is for Mr. Holmstead. 

You look bored there. No one’s asked you a question. It has to do 

with WOTUS, Waters of the U.S. I had the pleasure of sitting down 

with Senator Bob Dole six, seven months ago and had the conversation—he was one of the original co-authors of WOTUS, and of 

course I had to ask him, ‘‘Well, what does navigable stream mean 
79 

to you and what was your intention?’’ And of course he said, ‘‘Well, 

you know, there’s only three navigable streams in Kansas, and it’s 

very obvious what we meant.’’ 

And now there’s concern for my farmers—that WOTUS is actually managing land. If you regulate water to such a great extent 

it feels like you’re regulating land even though the Agency said it 

doesn’t want to regulate private property. So I guess my question 

is how can the Agency deny that by expanding vastly the definitions of WOTUS that it’s effectively limiting the activities that can 

occur on private property? I guess I’m trying to understand. 

WOTUS has expanded so much under the current interpretation 

that it’s impacting private property. 

Mr. HOLMSTEAD. You know, people are—I don’t think anybody at 

EPA could dispute that they’re regulating private property. I mean, 

that’s what it’s all about. So even if you own your land, even if you 

owned your land for 100 years, if somebody comes in and says 

there’s a wetland on your land, you have to get a permit to—and 

you can be prevented from using that land. That’s why there’s been 

so much debate and discussion over what really is a wetland and 

this most recent rule that’s been referred to as WOTUS really does 

significantly expand the net, expand the jurisdiction that—the 

amount of land that’s covered by that process. 

And I think that’s inappropriate, and I also think it’s the—well, 

I think it’s probably illegal. But I also think this is an area where 

Congress could really step in and provide some guidance because 

this is a debate that’s gone on for many, many years now with EPA 

making several stabs at trying to, you know, define the type of land 

over which EPA and the Corps should have jurisdiction. And that’s 

been difficult to do, but EPA has tried several times and the courts 

have sent it back saying no, you’ve gone too far. 

And I assume that this new EPA will make another stab at that 

and try to do something that’s more reasonable and that’s more understandable for people who really do care about what is a wetland. 

Mr. MARSHALL. Thank you for being so candid. 

Mr. LUCAS. The gentleman’s time is expired. The Chair now 

turns to the Ranking Member for a unanimous consent request. 

Ms. JOHNSON. Thank you very much. Mr. Chairman, I’d like to 

enter these letters, comments, and op-eds from the Asbestos Disease Awareness Organization, American Lung Association, the 

American Thoracic Association, and The National Environmental 

Health Association for—on their comments on EPA. 

The article—the Intercept article ‘‘Republicans are Using Big Tobacco’s Secret Science Playbook to Gut Health Rules,’’ and in the 

introduction to the record of the four peer-reviewed studies in the 

record that collaborate the findings of the NOAA study authored by 

Dr. Tom Karl and published by the Science magazine in 2015. 

Mr. LUCAS. Seeing no objection, so ordered. The information will 

be added to the record. Thank you. 

[The information appears in Appendix II] 

Mr. LUCAS. As we conclude this hearing, I would like to take a 

moment and note I think on behalf of myself and Chairman Smith, 

to my old colleague Dr. Holt, Rush, on the issues involving Dr. 

Bates’ concerns, I believe that it would be good for you to take the 

message back to the AAAS that they owe a fellow scientist making 
80 

such claims the honor of withholding judgment until the matter is 

fully investigated. It’s a very relevant and important issue. 

And with that, I thank the witnesses for their testimony and the 

Members for their questions. The record will remain open for two 

weeks for additional written comments and questions from the 

Members. 

This hearing is adjourned. 

[Whereupon, at 1:15 p.m., the Committee was adjourned.] 
(81) 

Appendix I 

ANSWERS TO POST-HEARING QUESTIONS
82 

ANSWERS TO POST-HEARING QUESTIONS

Responses by The Hon. Jeffrey Holmstead 
83 

and pregnant woman, but the MATS rule also set emission limits for other HAPs, primarily for 

non-mercury metals that are found in coal and emitted in very small amounts. 

When EPA did its cost-benefit analysis for MATS, it found that the human health and 

environmental benefits of reducing emissions ofthese HAPs was very small. Using a longstanding EPA methodology for estimating the benefits of reducing emissions of heavy metals 

like lead and mercury, EPA found that the benefits of the mercury reductions required by the 

MATS rule were less than $10 million a year and that the benefits of reducing the other f-lAPS 

were small but unquantifiable. EPA estimated that the MATS rule would cost the industry and 

ratepayers $9.6 billion a year (compared to $10 million in benefits for reducing mercury 

emissions). 

Dut the Obama EPA argued that MATS was a great deal for society because, according to the 

EPA website "the value of the air quality improvements for people's health alone [from MATS] 

totals $37 to $90 billion each year. That means that for every dollar spent to reduce this 

pollution, Americans get $3-9 in health benefits." (The balded words are EPA's, not mine.) 

However, almost all these benefits come.from reducing concentrations of PM2.5 below the level 

that EPA has found to be safe, because EPA found that in order to meet the emission limits for 

mercury and other HAPs, coal-fired power plants would either shut down or install controls that 

would reduce pollutants like S02 that contribute to PM2.5. So on one hand, EPA says that, when 

PM2.5 levels are 12 ug/m3 or less, public health is protected with an adequate margin of safety, 

even for people who are especially sensitive to air pollution. Then, on the other, EPA claims that 

tens of thousands of people are dying every year because they are exposed to levels of PM2.5 

below 12 ug/m3, and the MATS rule will save as many of II ,000 of them every year. 

In my view, the cost-benefit analysis that EPA has done over the last few years does not 

represent a serious effort to make good policy. The documents that were supposed to provide a 

rigorous tool for policymaking became promotional materials that EPA could use to sell its 

regulations publicly (as can be seen from EPA's website for MATS). These issues may seem 

obscure, but they do have important implications for regulatory policy, because EPA is allowed 

(or even required) to consider costs and benefits when taking certain actions. Using good science 

to understand whether and to what extent certain pollutants atTect public health is the only way to 

look at the benefits of EPA regulations. 

2. EPA has a shoot first style of regulating. The Agency issues a regulation before it has 

fully thought through whether it can be accomplished. How does this unnecessarily create 

burdens for business and manufacturing? 

EPA usually does a fairly good job of trying to understand the actions that a regulated industry 

will need to take in order to comply with proposed regulations and to estimate the cost that such 

regulations will impose on the industry. The Agency does underestimate the costs in some cases, 

but for major rules, it normally does try to analyze the impact that a rule will have on the 

regulated industry. In some important cases, however, the Agency makes no such effort. The 

most notable example is EPA's recent decision to lower the national ambient air quality 

standards (NAAQS) for ozone. 
84 

Rather than trying to evaluate the measures that will actually be needed to attain the more 

stringent ozone NAAQS, EPA simply tried to calculate the total tons of pollution reduction that 

would be needed and then assumed a relatively low "cost-per-ton" at which all these reductions 

could be achieved. This assumed cost-per-ton number is not supported by any real analysis and 

is much lower than the "cost-per-ton" already being incurred in several parts of the country. 

Moreover, states and industry have already taken the most cost effective steps to reduce 

pollution, and additional measures will be increasingly more expensive. EPA's assumed costper-ton number is simply not credible. 

Moreover, regulators in some parts of the country- those who have many years of experience 

trying to reduce ozone levels do not believe it will be possible to achieve the new ozone 

standard. EPA does a disservice to the public when it intentionally understates the cost of its 

regulations and insists that they arc achievable without any real attempt to evaluate whether this 

is the case. 

In the case of ozone and other national ambient air quality standards, the U.S. Supreme Court has 

ruled that EPA must set a standard based solely on what is requisite to protect public health with 

an adequate margin of safety- and that it cannot consider the cost of achieving the standard. 

However, refusing to do a serious analysis of the cost of achieving the standard- or even 

whether it will be achievable in certain areas- EPA docs a disservice to policymakcrs, regulated 

industry, and the public. If EPA were to disclose the true cost to society of meeting the ozone 

standard- and that it is not achievable in certain areas ofthe country- it is likely that Congress 

would amend the statute to adopt a more reasonable approach for dealing with air pollution. 

3. President Trump recently announced a review of streamlined permitting processes and 

regulatory burdens on American manufacturing. Are there issues regarding the EPA air 

permitting processes that you think should be included in this review? 

There is a Clean Air Act (CAA) permitting program that imposes a substantial and unnecessary 

burden on American manufacturing- a program known as "new source review" (NSR) that now 

makes it impossible to get a permit to build or expand a manufacturing plant in certain parts of 

the country and makes it prohibitively expensive in others. Even where it is possible to get an 

NSR permit, the permitting process is often lengthy, burdensome, and unpredictable. 

Under the current NSR program, before a company can even begin construction on a new 

industrial facility or on a project to expand an existing facility, it must first go through the NSR 

permitting process and obtain a permit that, among other things, ensures that the new or 

expanded facility will employ up-to-date pollution control technology. This requirement, 

however, is not normally an obstacle because plant owners have already identified the most 

effective pollution controls and are planning to install them. 

It is other NSR requirements that impose a large and unnecessary burden on U.S. manufacturing. 

The first is the "offset" requirement that applies in any part ofthe country that does not meet all 

the NAAQS set by EPA. Such areas are called "nonattainment areas." Anyone who wants to 

build a new major plant in such an area- even one with state-of-the-art pollution controls must 

obtain pollution "offsets" from other facilities in the same area. It is this offset requirement that 
85 

effectively prohibits new plants in some parts of the country because there are no other facilities 

in the same area. With EPA's increasingly stringent standards- especially for ozone some 

areas witb no industrial facilities do not meet the standard because of natural background 

pollution and emissions from areas that may be more than a hundred miles away. 

The idea behind offsets is that, in order to build a new industrial facility in a nonattainment area, 

a company must pay someone else to reduce emissions in that same area by an amount that 

exceeds the emissions that will come from the new facility. Depending on the area, it must obtain 

offsets that are between I 0 and 50 percent greater than the projected emissions from the new 

facility. 

Not surprisingly, offsets cannot be created on the basis of actions already required by EPA or 

state regulations. To be counted as an offset, an emissions reduction must go beyond what is 

required by law. But for more than 40 years, EPA and states have been looking for every 

conceivable way to reduce emissions related to ozone. In many areas, all the cost-effective 

emissions reductions have been mandated by regulation. Where any reductions can be made, 

they are very expensive. 

For example, the Houston area, especially near the Houston Ship Channel, has numerous 

industrial facilities, but they are generally very well controlled. Because there is so much 

industry, it is possible to purchase offsets, but they are very expensive. Houston area offset prices 

vary from $150,000 to $200,000 per ton for volatile organic compounds (VOCs) and $80,000 to 

$100,000 per ton for nitrogen oxides (NOx). 1 Even a relatively small facility with state-of-the-art 

controls will emit more than I 00 tons per year ofthese pollutants. The so-called "offset ratio" in 

the Houston area is 1.4 to I, meaning that the new facility would need to offset 140 percent of its 

projected emissions. Thus, even if the new facility will emit only 100 tons per year ofNOx and 

VOCs, the company trying to build it would need to purchase 140 tons ofNOx offsets and 140 

tons ofVOC offsets. At current offset prices, this means an upfront cost of$32 million to $52 

million just to purchase emissions offsets. 

In the South Coast nonattainment area in California, average offset prices in 2014 were $23,500 

per ton for VOCs and $63,000 per ton for NOx. 2 Moreover, the quantities involved in these 

emissions offset transactions are relatively small compared with the emissions from a new 

manufacturing plant. 3 If the applicant does not have a facility in the no~attainment area that it 

can readily control (or tear down) to provide offsets, then emissions offsets for five or more 

years in the future are reportedly hard or even impossible to find. 

The second part of the NSR program that imposes a substantial burden on American 

manufacturing is a requirement that applies in areas that do meet EPA's air quality standards. 

1 MIKE TAYLOR, SEPTEMDER 9, 2014, UPDATE ON SCARCITY OF HOUSTON-GALVESTON-BRAZORIA (HGB) EMISSION 

REDUCTION CREDITS (ERCS) ANU ALLOWANCES, AND USE OFNOx ERCS FOR VOCERCs (2014), 

http://www.awma-gcc.org/docs/Sept20 14Presn.pdf; TEXAS COMMISSION ON ENVIRONMENTAL QUALITY (TCEQ), 

TRADE REPORT (20 IS), www. !ceq .texas.gov/assets/public/imp Jementation/air/bankinglreports/ectradereport. pdf. 2 CALIFORNIA AIR RESOURCES BOARD (CARB), EMISSION REDUCTION OFFSET TRANSACTION COSTS SUMMARY 

REPORT FOR 2014 (20 IS), http://www.arb.ca.gov/nsr/erco/erc l4.pd£ 3 NSR generally applies to sources emitting I 00 tons/year of a precursor ozone pollutant. 
86 

Such areas are called "attainment areas." To obtain an NSR penn it in such areas, the penn it 

applicant must show, to the satisfaction of the permitting authority, that (I) projected emissions 

from the new plant will not result in changes in ambient air quality that would cause the area to 

exceed the NAAQS for any pollutant; and (2) even if projected emissions will not violate a 

NAAQS, they will not result in an increase in ambient concentrations of any pollutant that 

exceeds the allowable PSD "increments" set by the CAA. 4 

The requirement to show that emissions from a new facility will not "cause or contribute" to a 

violation of any NAAQS standard will be challenging or even impossible under the new ozone 

standard because many areas of the country that have always been in attainment do not meet the 

new standard. Until these areas are fonnally designated as nonattainment areas, a penn it 

applicant would need to show that the proposed plant will not "contribute to" a violation ofthe 

new standard, which would appear to be impossible in or ncar areas that are already in violation 

of the standard. 

EPA has said that it intends to create at least two options that would address this concern: (I) by 

setting certain de minimis emissions thresholds below which a new facility would be deemed not 

to "contribute" to a violation of the NAAQS; or (2) by allowing the permit applicant to purchase 

offsets. Given the history of CAA regulation, it is likely that these options, when finalized by 

EPA, will be challenged in court. Even if they pass muster in the courts, it remains to be seen 

whether either of these options will be practically viable-especially for large industrial 

facilities. 5 If not, it will not be possible to build or expand a new industrial facility in certain 

areas, even if the facility would use state-of-the-art technology to control its emissions and even 

if the local community desperately wants it to be built. 

The ideal solution to these problems would be a narrow statutory change that would allow 

manufacturing facilities to be built or expanded as long as they use the best available technology 

to control their emissions and the state or local environmental agency finds that projected 

emissions from the new or expanded facility will not pose an appreciable risk to human health or 

the environment. But even without any statutory change, EPA could take a number of steps to 

expand the pool of offsets and allow pennit applicants to use more reasonable approaches to 

show that a new plant will not cause or contribute to a violation of a NAAQS or exceed an 

applicable increment. 

4 The CAA established PSD increments for PM and S02 for the three classes of attainment areas: Class I (pristine), 

Class II (intermediate), and Class III (growth). EPA has established PSD increments for the other conventional 

pollutants through rulemaking. 5 For example, it appears that a number of rural areas may exceed the new 70 ppb ozone standard-not because of 

local emissions but because of background ozone and pollution transported from distant sources. Some areas have 

no local stationary sources and thus no way to generate offsets that can be used by new plants. In such ca,es, the 

offset requirement will impose a de facto ban on most types of industrial development 
87 

4. Many environmental laws leave regulatory decisions to the discretion of the EPA 

Administrator, but EPA bureaucrats often hijack the process to pressure their desired 

outeome. Are there ways in which the Administrator and his subordinates can reclaim the 

policy-making process at EPA- particularly for air policy? 

It is certainly true that some EPA career staffers are inclined to impose stringent requirements on 

industry whenever they can, even when the cost of such requirements far outweighs the benefits. 

But effective political leadership can overrule such actions, and this is one ofthe primary roles 

for political appointees at EPA. I do not believe that the career staff can be blamed for the 

regulatory overreach that we have seen at EPA over the last 8 years. 

Based on my experience as the head of the EPA Air Office during the George W. Bush 

Administration, I would say that most EPA career staffers are true civil servants and are 

dedicated to implementing the decisions made by the Agency's political leadership. This is 

certainly true among senior career officials in Air Office, which implements the CAA. Several 

senior career staffers worked very hard to implement and defend major regulatory reforms that 

were my highest priority as a political appointee- refonns that were strongly opposed by 

environmental activists because they reduced the regulatory burden on industry. But I know that 

many ofthese same staffers worked just as hard on the most costly and overreaching regulations 

issued during the Obama years. I am confident that they will work just as hard on the regulatory 

reforms that I hope and expect to see from the Trump Administration under Administrator Pruitt. 

I also know of several cases in which the EPA career staff made recommendations to Obama 

officials that would have reduced the cost of EPA regulations, but the Obama appointees rejected 

those recommendations in favor of more stringent and burdensome requirements. Especially 

when it came to coal-fired power plants, the Obama EPA often looked for ways to load up the 

regulatory burden as much as possible in the hope that such plants would shut down and be 

replaced by the type of generating facilities favored by the Obama Administration. 

In my view, the best way to reign in EPA would be for Congress to pass legislation to reform 

parts ofthe Clean Air Act and other environmental statutes that have proven to be particularly 

burdensome and even counterproductive. But even without new legislation, a President who is 

opposed to overregulation can appoint political officials who understand EPA's regulatory 

programs and are committed to reforming them in ways that would substantially reduce the 

costly and unnecessary regulatory burden now imposed by EPA regulations. I believe that 

President Trump has made one such appointment in Administrator Pruitt and will continue to 

make others, and I am confident that they will be able to accomplish this goal. 
88 

Responses by Dr. Kimberly White 

House Committee on Science, Space, and Technology 

Hearing Titled: Make EPA Great Again 

Hearing Date: February 7, 2017 

Questions and Responses for the Record 

Question 1: President Trump recently announced a review of streamlined permitting 

processes and regulatory burdens on American manufacturing, Are there issues regarding 

the EPA air permitting processes that you think should be included in this review? 

Response to Question 1: A review to improve and streamline the EPA's air permitting process 

will help facilitate the timely processing and implementation of necessary permits. The following 

issues should be evaluated in the review: 

A process for determining the utility and benefit achieved from fully implementing and 

attaining existing air quality standards before new standards are considered 

A process for developing and disseminating timely guidance when the Agency issues 

new National Ambient Air Quality Standards (NAAQS) 

A process and associated guidance to ensure that the science evaluation uses a weight of 

evidence process and that the findings support the revision of existing air standards 

A process for establishing guidance and polices associated with the review and granting 

of Prevention of Significant Deterioration (PSD) permits that: 

o Allows a source to make the PSD required demonstration of compliance with a 

distinct secondary standard through a demonstration of compliance with the 

primary NAAQS 

o Indicates how facilities may obtain offsets and demonstrate they do not "cause or 

contribute" to a violation of the NAAQS when obtaining a PSD permit 

Question 2: Many environmental laws leave regulatory decisions to the discretion of the 

EPA Administrator, but EPA bureaucrats often hijack the process to pressure their 

desired outcome. Are there ways in which the Administrator and his subordinates can 

reclaim the policy-making process at EPA- particularly for air policy? 

Response to Question 2: EPA's policy-making process must be firmly based on the best 

available and relevant scientific information and modeling approaches. In order to have a 

transparent, objective process for making policy decisions, the Agency must establish clear, 

consistent and systematic approaches for identifying, evaluating and integrating science 

information to support regulatory decisions. There is also a clear distinction between science and 

policy judgments. The Agency's policy judgments need to be identified, clearly discussed and 

documented in the administrative record. Notably, a 2009 report by the Bipartisan Policy Center 

titled: Improving the Use of Science in Regulatory Policy, included a recommendation that "The 

Administration needs to promulgate guidelines (through executive orders or other instruments) 

to ensure that when federal agencies are developing regulatory policies, they explicitly 

differentiate, to the extent possible, between questions that involve scient{fic judgments and 

questions that involve judgments about economics, ethics and other matters of policy." 
89 

Questions 3: I'm concerned that the EPA has stacked its Science Advisory Board and 

Clean Air Science Advisory Committee with supporters. EPA rolls out a regulation and 

says that it is supported by its scientists, but no one is there to offer an opposing view. Do 

you think there should be more balance on these advisory committees? 

Response to Question 3: The chartered Science Advisory Board (SAB), the Clean Air Science 

Advisory Committee (CASAC) and other EPA federal advisory committees are tasked with 

providing independent scientific and technical peer review, consultation, advice and 

recommendations directly to the EPA Administrator on the scientific basis for EPA actions and 

programs. The usefulness and value of this advice relies heavily on the expertise, independence 

and balance of the peer review committee. In order to ensure comprehensive, appropriate and 

effective peer review, the members of the SAB and CASAC need to be free from potential 

conflicts of interest, and should represent a cross-section of the scientific community. 

The Federal Advisory Committee Act (FACA) governs the operation of the SAB and CASAC. 

FACA requires that these advisory groups be fairly balanced in terms of points of view for the 

function to be performed by the committee. Additionally, the EPA's 2015 Peer Review 

Handbook, 4th Edition highlights the need for appropriate scientific expertise and balance of 

perspectives in all of EPA's various peer review mechanisms. Specifically, the handbooks notes 

"As part of the peer review process, the Agency (or the contractor managing the peer review) 

must select peer reviewers who have technical expertise in the subject matter that is needed to 

answer specific charge questions ...... These reviewers must not only be subject matter experts. 

but also must be independent and free from ethics issues such as potential conflicts of interest 

(CO Is) or an appearance of a loss of impartiality (~ee Sections 5.3.4 and 5.3. 7) so that the 

integrity of the peer review is not brought into question." 

To ensure a robust and balanced peer review process the below recommendations should be 

implemented consistently across all committees responsible for providing scientific and technical 

peer review to EPA. 

• The nomination and selection process for members of the peer review committee should 

be open and transparent. 

• All nominations submitted for consideration to the peer review committee should be 

subject to notice and open for public comment. 

• EPA shall adequately vet selected members of the peer review committee for potential 

bias or conflicts of interest. This would include the appearance of inherent bias based on 

past public positions taken on issues relevant to the peer review committee's charge. 

Appointments to peer review committees should be based principally on the scientific 

credentials, demonstrated accomplishments, and professional credibility of the nominee. 

It is also important to ensure that there is sufficient representation (i.e. more than one 

person for a given area of expertise). 

3 
90 

Question 4: There used to be a time where EPA's Science Advisory Board and Clean Air 

Science Advisory Committee would issue reports with dissenting views. Now it seems like 

those bodies just come to a "consensus" that almost always aligns with environmental 

groups. Do you think EPA's science advisory boards could do a better job at encouraging a 

broader range of views? 

Response to Question 4: Scientific discourse and discussion of different scientific views are 

cornerstones to an effective peer review process. They allow for objective and independent 

advice that reflects the state of the scientific debate. EPA must ensure that its peer review 

process consistently facilitates the transparent representation ofthe views discussed during the 

review process, including any disparate and dissenting views by peer review committee 

members. The recommendations below should be implemented to ensure incorporation of a 

broad range of views. 

• Peer review committee members should receive clear guidance regarding their role in the 

process, EPA's role in the process, and the role of public stakeholders. 

• The charge to the peer review committee and charge questions directing the peer review 

activities should be structured to encourage and facilitate objective consideration of all 

relevant scientific data and views. 

• A report summarizing peer review consensus and any critical or dissenting views should 

be developed and made publically available. 

• An independent process should be put in place to oversee and ensure that EPA has 

adequately addressed peer review committee recommendations. 

4 
91 

Responses by The Hon. Rush Holt 

Questions for the Record 

Before the 

Committee on Science, Space and Technology 

Hearing on "Making EPA Great Again" 

by 

Rush Holt, Ph.D. 

Chief Executive Officer 

American Association for the Advancement of Science 

Executive Publisher, Science 

March 15, 2017 

1) How do you respond to those in Congress, and within the Administration, who are 

attempting to dismantle EPA? 

The Environmental Protection Agency's mission is to protect human health and 

the environment. That is something we should all agree on. While debates occur 

about the scope of EPA's ability to issue regulations to achieve its mission, I 

would stress that what should not be under debate is the EPA's support for, access 

to, and use of the best available science to inform these regulations. Oversight of 

EPA's regulatory authority is a natural occurrence that demonstrates the sign of a 

healthy democracy and has been transpiring since EPA's founding in 1970. 

However, what is unhealthy is denying EPA access to the best available science in 

order to undercut its regulatory authority, or unnecessarily attacking research that 

EPA has used because of a disagreement on regulations that cite particular 

studies. 

In addition, to carry out its mission, the EPA funds extramural research on topics 

like toxicology, ecosystems and climate change. This empowers scientists to 

discover new ways to make our drinking water safer, breathe cleaner air, and 

protect our environment. It also provides a powerful tool for policymakers to 

utilize this scientific knowledge in setting rules and regulations through traditional 

rule-making procedures. Attempts to dismantle the funding of this research will 

only limit the ability of scientists to discover ways to help us all live in a 

healthier, cleaner society. 

2) During the hearing, one of the witnesses stressed the importance of a "benefit-cost" 

analysis in federal rulemaking, stating "that it is impossible for decision-makers, whether 

in an agency or on Capitol Hill, to understand what the implications are of actions that 

are being taken without benefit-cost analysis. It can't be done without that. Otherwise, 

it's based on emotion, it's based on politics, based on cronyism. It's based on other 
92 

factors. But- and also I would say it can't be based on science because science is the 

underpinning to the benefits assessment in a benefit-cost analysis." 

• Do you believe this statement accurately reflects the current role of science in 

decision making? 

Scientific evidence can present both positive and negative outcomes, so 

agencies can base decisions on science. That does not discount the fact 

that agencies also weigh other factors when crafting policies. It is not an 

either-or, but a balance between the two. 

• How should science be used in the decision making across the federal government 

The role of science for the federal government should be to present 

evidence, after peer review and thorough testing, to assist policymakers in 

making the best-informed decisions. The scientific process is one of 

removing bias and attempting, as best you can, to engage in evidentiarybased decision making. As a result policymakers can benefit tremendously 

from the outcomes of scientific research and input trom scientists and 

engineers across a range of fields. Science should be an integral element 

across every agency of our government so policymakers have access to 

data and experts that enable them to make decisions based on the 

evidence. That is why AAAS wrote a letter to each incoming secretary of 

the Trump Administration pointing to the insights of scientific research 

and how science relates to their department, so they can make life safer 

and more prosperous for all of us. 

3) How would you distinguish between the need to base regulations on studies that are 

reproducible as opposed to replicable? Why is this distinction important? Is it possible to 

have regulations supported by science that is reproducible and not replicable? 

The important principle of science is that the collection, analysis, and open 

communication of evidence can be verified through a deliberate process. In other 

words, it is essential that a scientist's work is subjected to verification by others in 

an unbiased way. It is often impossible to repeat an experiment exactly down to 

the last detail, and so the verification process leads to examining the procedure the 

experiment used, or determining whether the experimental results fit well with 

other experiments and the fabric of scientific understanding. 
93 

Some experiments may be fully replicable. For example, the preparation of a 

potential drug compound may be completed over and over given that the 

experimental methods are described adequately and with access to the same 

materials. However, consider experiments that involve a collection of samples 

from a large forest over the duration of many years or a clinical trial involving 

I 0,000 patients. In these cases, it would be essentially impossible to exactly 

repeat these experiments for logistical reasons or because the same population no 

longer exists. Nonetheless, these experiments could be checked if the data were 

collected and described fully so that another research team could examine the 

methodologies, the data and repeat the analysis. 

In the context of policy deliberations and regulations, it is important that the 

scientific process has been applied to the data. The principal point I tried to make 

to the committee is that it should not be a political judgment to perfect the 

scientific review process. Elected officials are not well suited to perfecting the 

scientific process and should not try. Changing the composition of scientific 

advisory committees or how studies are conducted, as legislation introduced 

before this committee in the past two congresses has tried to do, are political 

attempts to perfect the scientific process which are therefore misguided. 

4) How do principles of transparency and scientific integrity help build public trust in 

government regulations? Why is that important? 

Trust and accountability are integral to the research enterprise and the sharing of 

scientific information. AAAS has been a leader on this subject, collaborating with 

the National Academy of Sciences to promote informed discussions as well as 

resources on the standards for ethical conduct in all fields of study. If science is 

cited in the promulgation of government regulations, the principles of 

transparency within the limitations imposed by national security, privacy, and 

proprietary interests -- alongside scientific integrity help build public trust 

because the public will know that the science used in regulations was conducted 

in an appropriate manner. 

I mentioned in my testimony, scientists-- whether in industry, academia, or the 

government -- must have confidence that they can conduct their work in an 

atmosphere free of intimidation or undue influence. Policymakers should never 

dictate the conclusions of a scientific study or disparage scientists because the 

results of a study contradict a previously held belief. The integrity of the process 

must be upheld so that the public can trust in both the scientists conducting the 

research and the policymakers utilizing the research. 
94 

5) In your testimony you stated that "removing concepts like reproducibility and 

independent analysis from the hands of scientists and into a legislative chamber or a court 

room" impacts the scientific process. Can you please describe what those impacts would 

be? 

As mentioned in my response to question 3, scientists have processes and 

procedures to verify research results through reproduction, replication, and 

independent analysis. These are terms and practices of the scientific enterprise 

which scientists undertake to review their work and test their theories. 

Legislatures and courts can ask whether these principles are upheld in the practice 

and use of science by an agency, but legislatures and courts should not attempt to 

redefine the scientific process to suit their wishes. If they do, these principles will 

then be taken out of the scientific community and thrust into an inappropriate 

forum. This risks individuals or groups not familiar with the process of science 

altering the scientific process and turning science into a political combat zone. 

This would have a chilling effect on any scientist whose research is used by the 

EPA that their work could be litigated. This is not the type of message we should 

be sending the scientific community. We should be encouraging more science and 

scientists to be engaged in the policy process, not subjecting their work to 

lawsuits. 

6) How would undermining the scientific integrity of research undertaken at a federal 

agency affect the draw for immigrants to pursue research in this country, and more 

importantly, to stay and work in the US after receiving their degrees? 

If the scientific integrity of research is compromised or undermined at a federal 

agency, it would send a dangerous message to scientists both nationally and 

internationally. If scientists can't trust a society or government to allow them to 

follow the scientific method without coercion or undue influence, it stymies 

progress. As I mentioned in my testimony, scientific progress depends on 

openness, transparency, and the free flow of ideas and people. These are the 

principles that have helped the United States attract and richly benefit from 

scientific talent. However, should there be interference of these principles at a 

federal agency, there is no doubt that scientists will not want to work there, 

whether a US citizen or immigrant. This will thwart the United States' role as the 

global leader on innovation and scientific advancement. 
95 

Responses by Dr. Richard Belzer 

U.S. House Committee on Science, Space, and Technology 

Hearing: "Making EPA Great Again" 

February 7, 2017 

Questions and Responses for the Record 

Submitted February 23, 2017 

1. "EPA will often bury the costs of its regulations while inflating benefits. Do you think it 

would improve regulations if EPA were more transparent in its benefit-cost analyses?" 

Every agency, including but not limited to EPA, has powerful incentives to understate costs 

and overstate benefits. I saw this routinely during my tenure at the Office of Management and 

Budget, and I have seen it routinely since I left OMB in 1998 nineteen years ago. 

The reason is very simple. Agency analysts (and their contractors) work for the program 

offices that have decided to propose or promulgate a regulation. Regardless of their education, 

training, skill or intellect, the analysts' job is to provide an economic justification for what 

program managers want to do. If analysts fail to support program managers, they will be 

denied promotion and encouraged to resign (and if they are contractors, they will not be 

rehired). Program managers may not have to work very hard to secure analysts' cooperation, 

either. Analysts (and contractors) are recruited with a keen eye for candidates who are 

comfortable with this role. 

There is a "market" for benefit-cost analysis in regulatory decision-making. On the demand 

side, there are agency program managers, agency lawyers, agency appointed leadership, OMB 

and the rest of the Executive Office of the President, Congress and the public. Agency program 

mangers hire the analysts who prepare regulatory benefit-cost analyses, so they have 

functional control over supply as well as demand. 

From the perspective of agency analysts, by far the most important of these "customers" 

are agency program managers. They want benefit-cost analyses showing substantially greater 

benefits than costs, and often they do not care what analysts must do to achieve this. An 

agency economist once told me that his job was to find three times as much in benefits as he 

found in costs. His work was not complete until he had done so, and if he failed to accomplish 

Richard B. Belzer, Ph.D. 

Regulation, Risk, Economics & Information Quality 

Strategy & Analysis Consulting 

rbbelzer@post.harvard.edu •!-1'11703. 780.1850 
96 

2 

this objective, he'd be relieved of his duties and replaced with someone who could succeed, by 

hook or by crook. Needless to say, this environment is not conducive to honest work. 

In short, there is a market failure in the production of regulatory benefit-cost analysis. For 

many market failures, government can, through regulation, play a decisive role by providing 

what the market will not. But the market for regulatory benefit-cost analysis is unlike all private 

markets. The government cannot solve by regulation a problem of its own creation. An agency 

is both the sole buyer and the sole seller of a regulatory benefit-cost analysis. This enables 

agency program managers to choose whatever level of quality they want. If they expect to have 

to defend a regulation in court and high-quality analysis is required by law, program managers 

will set and ruthlessly enforce high quality standards. But if they expect not to be challenged in 

court, or if they expect courts will defer to them on matters of analytic quality, then program 

managers will choose the lowest level of quality sufficient to satisfy an undemanding judge. 

Transparency is a key attribute in regulatory benefit-cost analyses, but whether it's a 

desirable or undesirable attribute depends on whose interests are considered. It's certainly 

ironic that agencies practice transparently rarely, but they demand it without reservation from 

those they regulate. Transparency is a highly undesirable attribute from the perspective of 

many agency program managers, and oftentimes agency appointed leadership as well. During 

my years at OMB, I sought to improve transparency at every opportunity. However, I 

encountered implacable resistance from agency program managers. I also encountered 

resistance from agency lawyers, for whom transparency threatened to undermine their ability 

to defend agency actions in court. 

Congress can try to mandate transparency through legislation, but it appears to be 

impossible to directly overcome the myriad internal agency incentives arrayed against it. An 

information quality approach is more likely to be successful. That is, Congress can act to reward 

agencies that rely on transparent regulatory benefit-cost analyses. Congress would need to 

create a straightforward legal pathway whereby an agency's lack of transparency (and failure to 

adhere to substantive information quality standards, such as objectivity) is justiciable in federal 

court. To further reward agencies that are transparent, Congress should consider making 

prospective litigants eligible to recover their attorneys' fees if they prevail. And, if Congress 

really wants to encourage transparency, it could direct that attorneys' fees be paid from the 

agency's budget rather than the Treasury Department's Judgment Fund. (Agencies do not care 

about the award of attorneys' fees if they are paid by the Judgment Fund.) 

Agencies can be rewarded for practicing transparency other ways, as well. For example, an 

agency seeking constructive input from the public would publish an Advanced Notice of 

Propose Rule making well before it is ready to publish a proposal. That ANPRM would be 

accompanied by a transparent preliminary analysis of benefits and costs. If public review led to 

Richard B. Belzer, Ph.D. 

Regulation, Risk, Economics & Information Quality 

Strategy & Analysis Consulting 

rbbelzer@post.harvard.edu •!•r11703.780.1850 
97 

a consensus that effects are unlikely to exceed $100 million in any one year, then Congress 

could exempt the agency from performing a benefit-cost analysis along with a proposed rule 

(unless, of course, the proposed rule was materially different). 

3 

Agencies that fully disclose all their data, assumptions, models and the like could be 

rewarded even more. First, they would save money by enabling interested parties to prepare 

benefit-cost analyses for them. While competing interests might produce significantly different 

work products, they would be held to the same transparency and information quality 

requirements. Agencies could summarily discard third-party analyses that are not transparent 

or fail to comply with key information quality principles, such as objectivity. (Agencies that 

relied upon substandard third-party analyses would face serious legal risk.) 

Second, agencies would be able to promulgate higher-quality regulations- regulations that 

achieved their statutory purposes cost-effectively. This would significantly reduce public 

controversy over rulemaking, which would benefit almost everyone. That includes a reduced 

propensity to litigate final rules and a greater likelihood that agency actions would withstand 

judicial review. 

2. "OMB issues guidance like "Circular A-4" on how agencies should perform benefit-cost 

analyses for regulations. How well has EPA adhered to such guidance? Do you have 

any other advice and thoughts?" 

Circular A-4 is generally not a helpful guidance document. It provides too little guidance for 

agency analysts who need help, and it imposes only a minimal barrier to agency analysts who 

willfully violate the established norms of benefit-cost analysis. 

To give an obvious example where many agency analysts likely need help, Circular A-4 

mentions "opportunity cost" 13 times. This is the fundamental principle of cost assessment. 

Circular A-4 correctly states that cost is properly measured in terms of the value of benefits that 

must be foregone in order to pay for a regulatory requirement. But the Circular provides little 

assistance to agency analysts who want to know how to do this. Guidance that correctly 

identifies the destination but omits any discussion about which highway to take (and where the 

potholes are located) is not particularly helpful. 

Experienced agency analysts understand the opportunity cost principle, but they are free to 

ignore it without penalty. That's because nothing in Circular A-4, no matter how elementary or 

critical the provision, is actually required. Three hundred seventeen times, OMB says agencies 

"should" do something. As every regulator knows, "should" is hortatory and "must" is 

mandatory. 

Richard B. Belzer, Ph.D. 

Regulation, Risk, Economics & Information Quality 

Strategy & Analysis Consulting 

rbbelzer@post.harvard.edu •Nil703.780.1850 
98 

4 

Because Circular A-4 is so limited, some agencies have developed their own guidance. EPA, 

for example, has its own guidance, and itis a much richer and more complete work. 1 That's not 

to say the EPA guidance is technically superior on key principles; rather, the EPA guidance 

simply provides EPA analysts with much more guidance. (Problems arise where Circular A-4 and 

EPA's guidance conflict. In those cases, EPA analysts can be expected to treat the Agency 

guidance as mandatory and Circular A-4 as hortatory.) 

Circular A-4 is out of date, having been published in 2003, and a case can be made that it 

needs a thorough updating. Whether this should be done deserves a benefit-cost analysis. On 

the cost side, the OIRA staff is very limited and fully occupied with other tasks, most notably 

implementing Executive Order 13771. Right now, what agencies need most from OMB is 

guidance on the estimation of cost savings from the deregulatory actions agencies are required 

to undertake pursuant to§ 2(c), and which§ 2(d) appears to direct OMB to prepare and issue. 

Redirecting scarce OIRA resources to revising the existing text of Circular A-4 therefore has high 

opportunity costs. On the benefit side, the potential value of revision to regulatory decisionmaking is unclear. OMB's authority to enforce Circular A-4 is delegated by the President. If 

President Trump wants OMB to rigorously enforce Circular A-4, OMB will do so. 

When OMB began providing agency guidance on benefit-cost analysis in 1990, the available 

literature was not nearly as extensive as it is today. Agency analysts seeking high-quality 

guidance may be better served by consulting this literature. There are dozens of textbooks 

available for general use, and a rich scholarly literature is available to offer valuable insights on 

complicated issues or unique applications. On behalf of the Society for Benefit-Cost Analysis 

(benefitcostanalysis.orgl. Cambridge University Press publishes a scholarly journal (cleverly 

titled Journal of Benefit-Cost Analysis). The Society hosts scholarly meetings where agency 

analysts can learn from experts and present their own work. (The 2017 annual conference will 

be held March 15-17 at George Washington University, so few agency analysts are impeded 

from attending due to limited travel funds.) Another useful source of advice comes from a 

group of 19 (!)experienced analysts who recently published a Consumer's Guide to Regulatory 

Impact Analysis, which is undergoing peer review by the Journal of Benefit-Cost Analysis. 

1 See U.S. Environmental Protection Agency. 2016. Guidelines. for Preparing Economic Analyses. 

Available: https :1/yos emite.epa .goy/ee/epa /eerm.nsf&wA N IE E -0568-50.pdf/Sfile/E E -0568-SO.pdf 

[accessed February 23, 2017]. While the length of a document is not always a good proxy for quality, 

it is worth noting that Circular A-4 is 48 pages long and EPA's guidance is 302 pages long. 

Richard B. Belzer, Ph.D. 

Regulation, Risk, Economics & Information Quality 

Strategy & Analysis Consulting 

rbbelzer@post.harvard.edu <•IZI703. 780.1850 
99 

Agency adherence to Circular A-4, or more generally to accepted principle and practices in 

benefit-cost analysis, has been problematic for the reasons I explained in my answer to 

Question 1, and because presidential authority for enforcement has waxed and waned. To the 

extent that EPA has not complied with Circular A-4, it is because OMB has not enforced it 

administratively. Because it is guidance, it would be inappropriate for the courts to enforce it 

legally. 

5 

Congress can accomplish a great deal by requiring agencies to perform benefit-cost analysis, 

and creating incentives for them to comply. Congress does not necessarily need to legislate 

benefit-cost principles, and it should not legislate by reference a particular external authority 

such as Circular A-4 or ask the National Academy of Sciences to opine. The most important 

thing Congress can do is to incentivize compliance by making noncompliance a legally 

vulnerable position. Agencies should not be legally able to rely on analyses that are not 

transparent (i.e., they cannot be reproduced by qualified third parties) or objective (i.e., free of 

embedded policy preferences). 

Finally, without reforms like this Congress cannot be assured of having before it the best 

available record of the benefits, costs and other effects of regulation. Congress needs this 

information, when it considers Joint Resolutions for Disapproval under the Congressional 

Review Act, of which by my count 48 have been introduced so far this session. Congress would 

need accurate information much more if it enacted H.R. 26 ("Regulations from the Executive in 

Need of Scrutiny Act of 2017"). Otherwise, Congress would be ill-advised to rely on the benefitcost analyses that agencies would submit pursuant to proposed§ 801(a)(1)(B)(i). 

Richard B. Belzer, Ph.D. 

Regulation, Risk, Economics & Information Quality 

Strategy & Analysis Consulting 

rbbelzer@post.harvard. edu •!•ID703. 780.1850 

(101) 

Appendix II 

ADDITIONAL MATERIAL FOR THE RECORD
102 

DOCUMENTS SUBMITTED BY COMMITTEE CHAIRMAN

LAMAR S. SMITH
103 

2/21/2017 Climate scientists versus climate data I Climate Etc. 

climate data archive, would not follow the policy of his O\o\'ll Agency nor the guidelines in Science magazine for 

dataset archival and documentation. 

I questioned another co-author about why they choose to use a go% confidence threshold for evaluating the 

statistical significance of surface temperature trends, instead of the standard for significance of 95%- he also 

expressed reluctance and did not defend the decision. A NOAA NCEI supervisor remarked how it was eye-opening 

to watch Karl work the co-authors, mostly subtly but sometimes not, pushing choices to emphasize \varming. 

Gradually, in the months after KlS came out, the evidence kept mounting that Tom Karl constantly had his 'thumb 

on the scale'-in the documentation, scientific choices, and release of datasets-in an effort to discredit the notion of 

a global warming hiatus and rush to time the publication of the paper to influence national and international 

deliberations on climate policy. 

Defining an Operational Climate Data Record 

For nearly two decades, I've advocated that if climate datasets are to be used in important policy decisions, they 

must be fully documented, su'Qject to software engineering management and improvement processes, and be 

discoverable and accessible to the pnblic with rigorous information preservation standards. I was able to implement 

such policies, with the help of many colleagues, through the NOAA Climate Data Record policies (CDR) []ink]. 

Once the CDR program was funded, beginning in 2007, I was able to put together a team and pursue my goals of 

operational processing of important climate data .records emphasizing the processes required to transition research 

datasets into operations (known as R20). Figure 1 summarizes the steps required to accomplish this transition in 

the key elements of software code, documentation, and data. 

CDR R20 Process Diagram 

Figure 1. Research to operations transition process methodology from Bates et al. 2016. 

Unfortunately, the NCDC/NCEI surface temperature processing group was split on whether to adopt this pmcess, 

with scientist Dr. Thomas C. Peterson (a co-author on K15, now retired from NOAA) vigorously opposing it. Tom 

Karl never required the surfaee temperature group to use the rigor of the CDR methodology, although a document 

https:f~ud!thcurry.com/2017f02104Jdlmate-scientists-versus-climate-datal 2/164 
104 

2/21/2017 Climate scientists versl.IS climate data! Climate Etc. 

was prepared identifying what patts of the surface temperature processing had to be improved to qualify as an 

operational CDR. 

Tom Karl liked the maturity matrix so much, he modified the matrix categories so that he could claim a number of 

NC.EI products were "Examples of"Gold" standard NCEI Products (Data Set Maturity Matrix Model Level6)." 

See his NCEI overview presentation all NCEI employe-es fncci-overviev.•-2015nov-2] were told to usc, even though 

there had never been any maturity assessment of any of the products. 

NCDC/NCEI surface temperature processing and archival 

In the fall of 2012, the monthly temperature products issued by NCDC were incorrect for 3 months in a row llink]. 

As a result, the prc..<;s releases and datasets had to be withdrawn and reissued. Dr. Mazy Kicza, then the NESDIS 

Associate Administrator (the parent organization ofNCDC/NCEI in NOAA), noted that these repeated enors 

reflected poorly on NOAA and required NCDC/NCEI to improve its .<;oftware management processes so that such 

mistake..;; would be minimized in the future. Over the next several years, NCDC/NCEI had an incident report 

conducted to tmce these enors and recommend corrective actions. 

Follov.ring those and otherreeommendations, NCDN/NCEI began to implement new software management and 

process management procedures, adopting some of the elements of the CDR R20 process. In 2014 a NCDC/NCEI 

Science Council was formed to review new science activities and to review and approve new science products for 

operational release. A draft operational readiness review (ORR) was prepared and used for approval of all 

operational product releases, which was finalized and formally adopted in Janua1y 2015. Along '\\ith this process, a 

contractor who had worked at the CMMI Institute (CMMI, Capability Maturity Model Integration, is a software 

engineering proce.<>s level improvement training and appraisal program) was hired to improve software processes, 

with a focus on improvement and code re}uvenation of the surlace temperature processing code, in particular the 

GHCN-M dataset. 

The first NCDCjNCEI surlace temperature software to be put through this rejuvenation was the paiiVlise 

homogeneity adjustment portion of processing for the GHCN-Mv4 beta release of October 2015. The incident repmt 

had found that there were unidentified coding errors in the GIICN-M processing that caused unpredictable results 

and different results evecy time code was mn. 

The generic flow of data used in processing of the NCDC/NCEI global temperature product suite is shown 

schematically in Figure 2. There are three steps to the processing, and two of the three steps are done separately for 

the ocean versus land data. Step 1 is the compilation of observations either from ocean sources or land stations. Step 

2 involves applying various adjustments to the data, including bias adjustments, and provides as output the 

adjusted and unadjusted data on a standard grid. Step 3 involve..<> application of a spatial analysis technique 

(empirical orthogonal teleconnections, .EOTs) to merge and smooth the ocean and land smface temperature fields 

and provide these merged fields as anomaly fields for ocean, land and global temperatures. This is the product used 

in K15. Rigorous ORR for each of these steps in the global temperature processing began at NCDC in early 2014. 

https:/{Judithcurry.com/2017/02/04/climate-scientists-versus-cllmate-dal:a/ 31164 
105 

2/21/2017 Climate scientists versus climate data I Climate Etc. 

Figure 2. Generic data flow for NCDC/NCEI surface temperature products. 

In Kls, the authors describe that the land surface air temperature dataset included the GHCN-M station data and 

also the new ISTI (Integrated Surface Temperature Initiative) data that was nm through the then operational 

GHCN-Mbias correction and grldding program (i.e., Step 2 ofland airtemperatureprocessing in Figure 2). They 

further indicated that this proce..<;sing and subsequent corrections were 'essentially the same as those used in 

GHCN-Monthly version 3'. This may have been the case; however, doing so failed to follow the process that had 

been initiated to ensure the quality and integrity of datasets at NCDC/NCEI. 

The GHCN-MV4 beta was put through an ORR in October 2015; the presentation made it clear that any GHCN-M 

version using the ISTI dataset should, and would, be called version 4. This is confirmed by parsing the file name 

actually used on the FIT site for the K15 dataset llink]; NOTE: placing a non-machine readable copy of a dataset on 

an FrP site doe..<; not constitute archiving a dataset). One file is named 'box.12.adj-4.a.1.20150119', where 'adj' 

indicates adjusted {passed throu~h step 2 of the land processing) and '4.a.1' means version 4 alpha runt; the entire 

name indicating GHCN-Mversion 4a tun 1. That is, the folks who did the processing forKlS and saved the file 

actually usL.>d the correct naming and versioning, but KlS did not disclose this. Clearly labeling the dataset would 

have indicated this was a highly experimental eady GHCN-Mversion 4 run rather than a routine, operational 

update. As such, according to NOAA sci('ntific integrity guideline.<;, it would have required a disclaimer not to use 

the dataset for routine monitoring. 

In August 2014, in response to the continuing software problems v.rith GHCNMv3.2.2 (version of August 2013), the 

NCDC &ience Council was briefed about a proposal to subject the GHCNMv3 software, and particularly the 

pairvvise homogeneity analysis portion, to a rigorous software rejuvenation effort to bring it up to CMMI level2 

standards and resolve the lingering software errors. All software has errors and it is not surprising there were some, 

but the magnitude of the problem was significant and a rigorous process of software improvement like the one 

proposed was needed. However, this effort was just beginning when the Kts paper was .submitted, and so Kis must 

have used data \-vith some experimental processing that combined aspects ofV3 and V 4 with known flaws. The 

GHCNMv3.Xuscd in K15 did not go through any ORR process, and so what precisely was done is not documented. 

https:/~udithcurry.com/2017/02/04/c\imate-scientists~versus.climate-da!al 4/164 
106 

2121/2017 Climate scientists versus climate data! Climate Etc. 

The ORR package for GHCNMv4 beta (in October 2015) uses the rejuvenated software and also includes two 

additional quality checks versus version 3. 

Which version of the GHCN-M software K15 used is further confounded by the fact that GHCNMv3.3.0, the upgrade 

from version 3.2.2, only went through an ORR in April2015 (i.e., after the K15 paper was submitted and revised). 

1be GHCN-Mv3.3.0 ORR presentation demonstrated that the GHCN-Mversion changes between V3.2.2 and V3.3.0 

had impacts on ran kings of wannest years and trends. The data flow that was operational in June 2015 is shown in 

figure 3. 

Figure 3· Data flow for surface temperature products de..;;cribed in K15 Science paper. Green indicates operational 

datasets having passed ORR and archived at time of publication. Red indicates e..xperimental datasets never subject 

to ORR and never archived. 

It is clear that the actual nearly-operational release of GHCN-Mv4 beta is significantly different from the version 

GHCNM3.Xused in Kts. Since the version GHCNM3.Xneverwent through any ORR, the resulting dataset was also 

never archived, and it is virtually impossible to replicate the result in K15. 

At the time of the publication of the K15, the final step in processing the NOAAGlobalTP.mpV 4 had been approved 

through an ORR. but not in the Kts configuration. It is significant that the current operational version of 

NOAAGlohalTempV 4 uses GHCN-M V3.3.0 and does not include the ISTI dataset used in the Science paper. The 

Kts global merged dataset is also not archived naris it available in machine-readable form. This is why the two 

boxes in figure 3 are colored red. 

The lack of archival of the GHCN-M V3.X and the global merged product is also in violation of Science policy on 

making data available llink]. This policy states: "Climate data. Data should be archived in the NOAA climate 

repository or other public databases". Did Karl et aL disclose to Science Magazine that they would not be fo11owing 

the NOAA archive policy, would not archive the data, and would only provide access to a non-machine readable 

version only on an FTP server? 

https:/[judithcurry.comf2017/02104/climate-scientists-versus-climate-data! 51164 
107 

2/21/2017 Climate scientists versus climate data I Climate Etc. 

For ocean temperatures, the ERSSfversion 4 is used in the Kts paper and represents a major update from the 

previous version. The bias correction procedure was changed and this re..~ulted in different S...'rr anomalies and 

different trends during the last 15+ years relative to ERSSTversion 3· ERSSTV 4 beta, a pre-operational release, was 

briefed to the NCDC Science Council and approved on 30 September 2014. 

The ORR for ERSSTV 4, the operational release, took place in the NCDC Science Council on 15 January 2015. The 

ORR focused on process and questions about some of the controversial scientific choices made in the production of 

that dataset will be discussed in a separate post. The review went well and there was only one point of discussion on 

process. One slide in the pre..c;entation indicated that operational release was to be delayed to coincide with Karl et 

al. 2015 Science paper release. Several Science Council members objected to this, noting the Kis paper did not 

contain any further methodological infonnation-all of that had already been published and thus there was no 

rationale to delay the dataset release. After discussion, the Science Council voted to approve the ERSS1\q ORR and 

recommend immediate release. 

The Science Council reported this recommendation to the NCDC Executive Council, the highest NCDC 

management board. In the NCDC Executive Council meeting, Tom Karl did not approve the release ofERSSI'v4, 

noting that he wanted its release to coincide with the release of the next version of GHCNM (GHCNMv3.3.0) and 

NOAAGlohalTemp. Those products each went through an ORR at NCDC Science Council on 9 April2015, and were 

used in operations in May. The ERSS1\r4 dataset, however, was still not released. NCEI used these new analyses, 

including ERSSTv4, in its operational global analysis even though it was not being operationally archived. The 

operational version ofERSSTv4 was only released to the public following publication of the Kis paper. The 

withholding ofthe operational version of this important update came in the middle of a major ENSO event, thereby 

depriving the public of an important source of updated information, apparently for the sole purpose of Mr. Karl 

using the data in his paper before making the data available to the public. 

So, in every aspect of the preparation and release of the datasets leading into Kis, we find Tom Karl's thumb on the 

scale pushing for, and often insisting on, decisions that maximize warming and minimize documentation. I finally 

decided to document what I had found using the climate data record matmity matrix approach. I did this and sent 

my concerns to theNCE! Science Council in early February 2016 and asked to be added to the agenda of an 

upcoming meeting. I was asked to tum my concerns into a more general presentation on requirements for 

publishing and archiving. Some on the Science Council, particularly the younger scientists, indicated they had not 

known of the Science requirement to archive data and were not aware of the open data movement. They promised to 

hegin an archive reqne..<;t fort he K15 datascts that were not archived; however I have not been able to confirm they 

have been archived. I later learned that the computer used to process the software had suffered a complete failure, 

leading to a tongue-in-checkjoke by some who had worked on it that the failure was deliberate to ensure the result 

could never be replicated. 

Where do we go from here? 

I have wrestled for a long time about what to do about this incident. I finally decided that there needs to be systemic 

change both in the operation of government data centers and in scientific publishing, and I have decided to become 

an advocate for such change. First, Congress should re-introduce and pass the OPEN Government Data Act. The 

Act state..~ that federal datasets must be archived and made available in machine readable form, neither of which 

was done by Kls. The Act was introduced in the last Congress and the Senate passed it unanimously in tbe lame 

duck session, but the Honse did not. This bodes well for re-introduction and passage in the new Congress. 

However, the Act will be toothless without an enforcement mechanism. For that, there should be mandatory, 

independent certification of federal data centers. As I noted, the scientists working in the trenches would actually 

https;/~udithcurry.com/2017/02104/dimate-scientists-versus-climate-data/ 61164 
108 

2/2112017 Climate scientists versus climate data 1 Climate Etc. 

welcome this, as the problem has been one of upper management taking advantage of their position to thwart the 

existing executive orders and a lack of process adopted 'Nithin Agencies at the upper levels. Only an independent, 

outside body can provide the needed oversight to ensure Agencies comply 'Nith the OPEN Government Data Act. 

Similarly, scientific publishers have formed the Coalition on Publishing Data in the Earth and Space Sciences 

(COPDESS) 'Nith a signed statement of commitment to ensure open and documented datasets are part of the 

publication process. Unfmtnnately, they, too, lack any standard checklist that peer reviewers and editors can use to 

ensure the statement of commitment is actually enforced. In this case, and for assessing archives, I would advocate 

a metric such as the data matmity model that I and colleagues have developed. This model has now been adopted 

and adapted by several different groups, applied to hundreds of datasets across the geophysical sciences, and has 

been found useful forensming information preservation, discovery, and accessibility. 

Finally, there needs to be a renewed effort by scientists and scientific societies to provide training and conduct more 

meetings on ethics. Ethics needs to be a regular topic at major scientific meetings, in graduate classmoms, and in 

continuing professional education. Respectful discussion of different points of view should be encouraged. 

Fmtunately, there is initial progress to report here, as scientific societies arc now coming to grips with the need for 

discussion of and guidelines for scientific ethics. 

There is much to do in each of these areas. Although I have retired from the federal government, I have not retired 

from being a scientist. I now have the luxury of spending more time on these things that I am most passionate 

about. I also appreciate the opportunity to contribute to Climate Etc. and work vv:ith my colleague and friend Judy 

on these important issue.'>. 

Postlude 

A couple of examplr.s ofhowthe public can find and use CDR operational products, and what is lacking in a nonoperational and non-archived product 

1. NOAA CDR of total solarirradiance- this is the highest level quality. Start at web sitehttps: 1/data.nodc.noaa.gov /cgi-bin /iso ?id- gov.noaa ncdc:Coo828 

Here you will see a fully documented CDR At the top~ we have the general description and how to cite the data. 

Then below, you have a set of tabs vv:ith extensive information. Click each tab to see how it's done. Note, for example, 

that in 'documentation' you have choices to get the genera] documentation, processing documents including source 

code, data flow diagram, and the algorithm theoretical basis document ATBD which includes all the info about how 

the product is generated, and then associate.d resources. This also includes a permanent digital object identifier 

(doi) to point uniquely to this dataset. 

2. NOAA CDR of mean layer temperature- RSS- one generation behind in documentation but still quite goodhttps: //wv-.'v; .ncdc.noaa.gov /cdr/fundamental/mean-layer-temperature-rss 

Here on the left you vv:ill find the documents again that are required to pass the CDR operations and archival. Even 

though it's a slight cut below TSI in example 1, a user has all they need to use and understand this. 

3. The Karl hiatus paper can be found on NCEI here- https: //www.ncdc.noaa.gov /news/recent-global-surface: 

waiming-hiatus 

https:h'judithcurry.com/2017/02/04/climate-scientists-versus-climate-datal 7/164 
109 

2121/2017 Climate scientists versus climate data I Climate Etc. 

If you follow the quick link 'Download the Data via l<lP'you go hereftp://ftp.ncdc.noaa.gov/pub/data/scpub20IS06/ 

The contents of this FTP site were entered into the NCEI archive following my complaint to the NCEI Science 

Conncil. Ilowever, the artifacts for full archival of an operational CDR are not includf'_d, so this is not compliant 

V¥-ith archival standards. 

Biosketch: 

John Bates received his Ph.D. in Meteorology from the University of Wisconsin-Madison in 1986. Post Ph.D., he 

spent his entire career at NOAA, until his retirement in 2016. He spent the last 14 years of his career 

at NOAA's National Climatic Data Center(now NCEI) as a Principal Scientist, where he served as a Supervismy 

Meteorologist until2012. 

Dr. Bates• technical expertise lies in atmospheric sciences, and his interests include satellite observations of the 

global water and energy cycle, air-sea interactions, and climate variability. His most highly cited papers are in 

observational stndie.s oflong term variability and trends in atmospheric ,~ . .rater vapor and douds. 

NOAA Administrator's Award 2004 for "outstanding administration and 1eadership in developing a new division to 

meet the challenges to NOAA in the area of climate applications related to remotely sensed data". He was awarded a 

U.S. Department of Commerce Gold Medal in 2014 for visionary work in the acquisition1 production, and 

preservation of climate data records (CDRs). He has he1d elected positions at the American Geophysical Union 

(AGU), inclnding MemberoftheAGU Council and Member oftheAGU Board. He has played a leadership role in 

data management fortheAGU. 

He is currently President of John Bates Consulting Inc., which puts his recent experience and leadership in data 

management to use in helping clients improve data management to improve their preservation, discovery, and 

exploitation of their and others data. He has developed and applied techniques for assessing both organizational 

and individual data management and applications. These techniques help identify how data can be managed more 

cost effectively and discovered and applied by more users. 

David Rose in the Mail on Sunday 

David Ro.se of the UK Mail on Sunday is working on a comprehensive expose of this issue llink]. 

Here are the comments that I provided to David Rose, some of which were included in his article: 

!Jere is what I think the broader implications are .. FOllowing ClimateGate~ I made a public plea for greater 

transparency in climate data sets, including documentation. In the U.S., John Bates has led the charge in 

developing these data standards and implementing them. ~t::;o it is very disturbing to see the institution that is the 

main U.S. custodian of climate data treat this issue so cavalierly, violating its own policy. The other concern that 

Iraisedfollowing ClimateGate was overconfidence and inadequate assessments of uncertainty. Large 

adjustments to the raw data, and substantial changes in successive data set versions, imply substantial 

uncertainties. The magnitude of these uncertainties influences how we interpret observed temperature trends, 

'warmest year' claims, and how we interpret differences between observations and climate model simulations. I 

also raised concerns about bias; here we apparently see Tom Karl's thumb on the scale in terms of the 

metlwdologies and procedures used in this publication. 

https:/~udlthcurry.com./2017/02/04/dimate..scientists.versus·dimate-data/ 81164 
110 

2/2112017 Climate scientists versus climate data I Climate Etc. 

Apart from the above issues, how much difference do these issues make to our overall understanding of global 

temperature change? All of the global surface temperature data sets employ NOAA's GJIG"N land surface 

temperatures. The NASA GJSS data set also employs the ERSST datasetsfor ocean surface temperatures. There 

are global surface temperature datasets, such as Berkeley Earth and HadCRUTthat are relatively independent of 

the NOAA data sets, that agree qualitatively with the new NOAA data set. Hawever, there remain large, 

unexplained regional discrepancies between the NOAA land surface temperatures and the raw data. Further, 

there are some very large uncertainties in ocean sea swface temperatures, even in recent decades. Efforts by the 

global numerical weather prediction cente1•s to produce global reanalyses such as the European Copernicus effort 

is probably the best way forward for the most recent decades. 

Regarding uncertainty, 'wannest year', etc. there is a good article in the WSJ: Change would be healthy at U.S. 

climate agencies (hockeyshtick has reproduced the full article). 

I also found this recent essay in phys.org to be very germane: Certainty in complex scientific research an 

unachievable goal. Researchers do a good job of e..'>timating the size of errors in measurements but underestimate 

chance of large errors. 

Backs tory 

I have known John Bates for about 25ycars, and he seiVed on the Ph.D. committee..<; of two of my graduate students. 

There is no one, anywhere, that is a greater champion for data integrity and transparency. 

When I started Climate Etc., John was one of the few climate scientists that contacted me, sharing concerns about 

various ethical issues in our field. 

Shortly after publication ofK15, John and !began discussing our concerns about the paper. I encouraged him to 

come forward publicly with his concerns. Instead, he opted to try to work within the NOAA system to address the 

issues -to little effect. Upon his retirement from NOAA in November 2016, he decided to go public with his 

concerns. 

He submitted an earlier, shorter version of this essay to the Washington Post, in response to the 13 December article 

(climate scientists frantically copying data). The WaPo rejected his op-ed, so he decided to publish at Climate Etc. 

In the meantime, David Rose contacted me about a month ago, saying he would be in Atlanta covering a story 

about a person unjustly imprisoned !Jin..k]. He had an extra day in Atlanta, and wanted to get together. I told him I 

wasn't in Atlanta, but put him in contact with John Bates. David Rose and his editor were excited about what John 

had to say. 

I have to wonder how this would have played out if we had issued a press release in the U.S., or if this sto:rywas 

given to pretty much any U.S. journalist working for the mainstream media. Under the Obama administration, I 

suspect that it would have been vety difficult for this story to get any traction. Under the Trump administration, I 

have every confidence that this will be investigated (but still not sure how the MSM v.:ill react). 

Well, it \vill be interesting to see how this story evolves, and most importantly, what policies can be put in place to 

pre\'ent something like this from happening again. 

I will have another post on this topic in a few days. 

https:f{judithcurry.comf2017f02/04/climate-sdentists-versus-dimate-dataf 9/164 
111 

2121/2017 Climate scientists versus climate data 1 Climate Etc. 

Being retired sure is liberating ... 

Moderation note: As 'vith all guest posts, please keep your comments civil and relevant. 

SHARE THIS: 

*Like 

30 bloggers like this. 

RELATED 

JC oped: the politics 

surrounding global temperature 

data 

In "Ethics" 

•• The latest climate 1

Conspiracy 

theory' 

In "Sociology of science" 

•••• [1.11 

Understanding adjustments to 

temperature data 

In "Data and obsen·ations" 

This ent:lywas posted in Data and obsen-ations, Ethics. Bookmark the permalink. 

700 RESPONSES TO "CLIMATE SCIENTISTS VERSUS CLIMATE DATA" 

Pingback: Climate scientists versus climate data- Enjeux energies et environnement 

beththeserf1 Februarv4 2017 at6:34omi&Jlli 

Incredible ain't it, non-archiving of cdtical evidence-? 

Faith-based eli-sci- not for general exhibition. 

Nick Stokes 1 Februarvs 2017 at2:58 am 1 &Jlli 

incredible ain't it, non-archiving of critical evidence-?" 

And just not tmc. There is an extensive archive. Bates even linked to it. It is hill:g. 

Bates complaints seem to be 

1. The archiving wasn't complete until six months after the paper appeared 

2. Data is in ascii format which is not "machine readable". Of course it is, it just requires a format 

statement. 

i johnfpitbnanl Februarv 5 2017 at?:OO am 1 

Nick you left out complaints: 

1. Karl made administrative decisions contrary to data integrity; 

2. Karl used 90% rather than 95% standard; 

3. The use of non standardized data set implied a greater uncettainty to the data that was not, 

could not be addressed; 

https:f~udithcurry.com/2017/02104/dimate-sdentists-versus-climate-dataf 10/164 
112 

Science, lies and 

video-taped experiments 

Too many researchers make up or massage their data, says 

Timothy D. Clark. Only stringent demands for proof can stop them. 

Late last month, a US physicist began a jail sentence for scientific fraud. Darin Kinion took funds for research on quantum 

computing but did not carry out the work he claimed; instead, 

he invented the data that the research supposedly produced. 

Scientists like to think that such blatant dishonesty is rare, but I 

myself have witnessed several serious cases of scientific misconduct, 

from major data manipulation to outright fabrication. Most have gone 

unpunished- in fact, it has been disheartening to see the culprits 

lauded. It makes little sense for fraudsters to fabricate mediocre data. 

Their falsehoods generate outstanding stories, which result in high-profile publications and a disproportionately large chunk of the funding pie. 

I have noticed a lesser-known motive for bad science in my field, 

experimental biology. As environmental change proceeds, there is great 

demand from the public and policymakers for 

simple stories that show the damage being done to 

field: using a tank of flowing water to expose fish to environmental 

perturbations and looking for shifts in behaviour. It is trivial to set up 

a camera, and equally simple to begin each recorded exposure with a 

note that details, for example, the trial number and treatment history 

of the organism. (Think of how film directors use clapper boards to 

keep records of the sequence of numerous takes.) This simple meas~ 

ure would make it much more difficult to fabricate data and 'assign' 

animals to desired treatment groups after the results are known. 

My colleagues and I are currently using this approach to record 

studies of how coral-reef fish respond to dissolved carb-on dioxide. 

There would also be benefits for other disciplines, including socialpsychology studies based on direct observations. 

Sharing visual evidence is straightforward, Video files can be compressed and transferred without excessive loss of 

resolution. Files can then be uploaded to free data 

wildlife. I occasionally meet scientists who argue 

that the questions we ask and the stories we tell 

are more important than the probity of our investigations: the end justifies the means, even if the 

means lead to data fabrication. That view is alarmingly misguided and has no place in science. The 

undeniable anthropogenic impacts on wildlife 

must be investigated with strict scientific rig our, 

SCIENTISTS HAVE 

LITTLE EXCUSE 

NOT TO 

repositories (such as figshare or Zenodo) before 

manuscripts are submitted for publication. Notably, the online supplementary material of most 

journals allows for 10-lSOMB of storage to 

accommodate images and detailed descriptions 

of methodology. 

One reason some scientists can get away with 

questionable practices is that the scientific system is based on trust. The burden of pro-of is 

on those who suspect and report misconduct, 

Unless there is overwhelming evidence to the 

contrary, scientists are believed to have done 

RECORD WHAT 

GOES ON 

There is more to this than preventing misconduct. Visual evidence can help reviewers (before 

and after publication) to spot problems that are 

not obvious from written descriptions and diagrams. Software could help to quantify behavioural features in recorded experiments and 

mitigate experimenter biases. Plus, scientists who 

IN LAB AND FIELD 

STUDIES. 

what they say they did. If the community is 

serious about tackling misconduct, this must change. It is time to shift 

the burden of proof onto those who produce the results. 

In some fields, this proof is often implicit in how scientists collect 

and report data. Detailed evidence may be provided hy the outputs 

of mostly autonomous equipment. Access to all the raw, non-manipulated data files- as increasingly demanded hy journals and peers 

aero&.;; disciplines- may be enough. 

Science that relies on human observation of remote field work 

and trials that arc difficult to replicate precisely-such as studies in 

the field of animal behaviour- needs a different approach. Simply, 

researchers should routinely film their experiments and present the 

f-ootage to journal editors, reviewers and colleagues alongside their 

data and analyses. In some disciplines (such as ornithology), photo 

or audio files may provide better evidence than video. 

If extreme athletes can use self~ mounted cameras to record their 

wildest adventures during mountaintop blizzards, scientists have little 

excuse not to record what goes on in lab and field studies. 

Yes, visual evidence can be faked, but a few simple safeguards 

should be enough to prevent that Take a typical experiment in my 

know that their equipment and techniques will be 

on display will try harder to improve them. 

The best way to implement these changes is for academic journals 

to start mandating visual (and audio) evidence to support a submitted 

paper. As far as I am aware, no journals routineJy do this, Journals must 

also ensure that their stated requirements are adhered to. 

Surveys suggest that I am not unusual in witnessing fraud: some 

14% of scientists say that they have witnessed it, too. Although it would 

be simpler to turn a blind eye to this issue and move on, this situation 

inhibits so many aspects of scientific progress that I feel compelled to 

try to fix: it. The added logistical difficulties of providing visual evidence arc a small price to pay to tackle dishonesty and greatly reduce 

the number of irreproducible (and often poorly conducted) studies. 

Mandatory visual evidence will undoubtedly help to reconcile the tens 

ofbillions of dollars wasted on irreproducible research every year. In 

short, show us your science. • 

Timothy D. Clark is a senior research fellow nt the University of 

1hsmania and the Commonwealth Scientific and Industrial Research 

Organisation in Hobart, Australia. 

e-mail: timothy.clark.mail@gmail.com 

9 FEBRllARY 2017 i VOL 542] NATURE) 139 

(rl:t:117 Macm1!lan Pub!ishl"r5 Limited, rart o! 'lpringN NaturE.\ All nghts rE>sPrvc>d 
113 

DOCUMENTS SUBMITTED BY COMMITTEE RANKING MEMBER

EDDIE BERNICE JOHNSON
114 

Patient Privacy Must Continue to Be Protected. Physicians and researchers have a clear 

legal and ethical obligation to maintain patient privacy. Researchers who evaluate the 

health impacts of air pollution, for example, must collect sensitive data from participants 

such as family medical history, geographic location, and personal medical history. 

Researchers who collect information about tobacco use can aggregate the data to 

determine patterns, but personal information about specific individuals must remain 

confidential. Scientists and institutions build in systems to protect this information while 

still maintaining open access to the collective data. The studies themselves are peerreviewed and published in transparent processes. However, no way exists to protect 

patient privacy if the raw patient data are released. The federal government must 

continue to protect patient privacy by ensuring that patients' sensitive information is 

never made public - but that does not negate the use of such data to inform policy. 

Public Funding of Science is Essential. The federal government must continue to fund 

core scientific efforts to ensure that public health is protected, as the Clean Air Act and 

Tobacco Control Act require. Those core efforts must include research investigating the 

health impacts of air pollution and climate change; the public health impact of tobacco 

products; epidemiological surveillance data regarding the prevalence and severity of 

diseases such as asthma, lung cancer, influenza, COPD and pneumonia; research into 

effective measures to reduce pollution and prevent and treat diseases; and data 

collection and evaluation of air quality and pollution levels in communities across the 

nation. These funds must include funding to state and local governments and tribes to 

collect air quality data affecting their residents and to develop plans to clean up the 

pollution to protect their health, as well as funding for state and local departments of 

health to monitor and implement public health interventions. The Federal government 

must also continue to fund research at NIH and CDC that will continue to lead to lifesaving breakthroughs for lung health. 

Politics must not be allowed to undermine the integrity of, or access to, scientific data needed to 

protect public health. The American people depend on our leaders to maintain public access to 

scientific data and enforce the nation's public health laws based on accurate science. Censoring 

science will have devastating impacts on the health of the communities our organizations serve. We 

urge you to embrace the above listed principles to protect the health of all Americans. 

Sincerely, 

Harold P. Wimmer 

National President and CEO 

Cc: Committee on Science, Space and Technology 
115 

Written Comments of the American Thoracic Society 

Submitted to the House Science Committee 

For the hearing: "Make EPA Great Again" 

Tuesday, February 7, 2017 

The American Thoracic Society (ATS) is pleased to submit the following comments to the House 

Science Committee hearing: "Making EPA Great Again". By the title of this hearing, we have 

high expectations for the bold ideas that will be discussed by the committee to improve the 

ability of the US Environmental Protection Agency (EPA) to protect and improve our nation's 

air and water quality. However, previous actions taken by this Congress raise concerns that this 

hearing, and subsequent legislative activity taken by the committee, will only serve the 

interests of polluters, and will in fact seek to weaken, delay, and deter EPA's authority to 

protect our nation's children and adults from harmful health effects of air and water pollution. 

As background, the ATS is a medical professional organization with over 16,000 members 

dedicated to prevention, detection, treatment, cure and research of pulmonary disease, critical 

care illness and sleep disordered breathing. Our members include physicians and scientists 

who are world-renowned experts on the health effects of air pollution. Our peer-reviewed 

journals publish cutting-edge science that examines effects of exposure to air pollution on the 

heart, lung, and human survival. Most importantly, our members treat patients with 

respiratory conditions like asthma and COPD- who are among most vulnerable to harm caused 

by air pollution. 

Our society of over 16,000 medical professionals recognizes and accepts medical findings once 

they are supported by multiple studies of different designs, conducted in different settings, and 

published in reputable peer-reviewed journals. We know that air pollution is harmful for 

human health because this finding is supported by decades of research, consisting of 

thousands of peer-reviewed scientific studies conducted around the world, 

We agree that we live in a society that is increasingly technology-based, and that a dear process 

for judging and interpreting scientific findings to inform our nation's health policies, is 

essentiaL But as indicated by the title of previous legislation considered by this committee, the 

"Secret Science Reform Act", many members of Congress appear to have a fundamental 

misunderstanding of how science works and progresses. This misunderstanding severely 

undermines the ability of Congress to govern this issue wisely and effectively. The aim of this 

submission is to help clarity the nature of sound science, such as that funded and employed by 

the US EPA, and to make constructive recommendations to make the science relied upon by the 

US EPA even stronger. 

in :-to 

~ ••. ,,,) We help the world breathe· ATS' PI!\!HI'I'"". Cf<!HCA< CARE. $ittf 

1150 18th Street, N.W., Suite 300 

Washington, D.C. 20036 U.S. 

T. 202~296-9770 F. 202 296·9776 I thoracic,org 

ATS2017 lnterhational Conference 

May 19-14, 2017 

Washington, DC 

conference.thoradc.org 
116 

February 6, 2017 

PageZ 

American Science is Conducted in an Open Process 

The phrase "secret science" bears no semblance to reality. To achieve quality and transparency, 

any science conducted or published in the US requires three levels of external scrutiny before 

results are considered part of the evidence base, as follows: 

External Review Step 1: Study Design Peer Review- The first level of review occurs when an 

investigator applies for funding, be it public funding or private funding. The investigator's 

proposal is subjected to peer review, which asks basic questions like: 

• Is the question the investigator is seeking to answer scientifically valid? 

• Will the answer provide useful knowledge? 

• Does the investigator have the research infrastructure and scientific expertise to actually 

conduct the study? 

• Are the methods the investigator is using actually capable of answering the question? 

• Are the study sample sizes large enough of provide statistically reliable answers? 

And probably most importantly: 

• How does the proposed study, compared to other proposed studies, give us the best 

opportunity to advance our scientific understanding of the world? 

Who asks these questions and reviews the answers? Panels of investigators who have specific 

expertise in that relevant field of study provide peer review. In the case of EPA, this review 

comes from independent, balanced expert review panels to ensure a broad range of subject 

matter expertise .. Why is having expertise important? Because experts know the field, so they 

can best assess whether the proposed study adds significantly to the field, uses the best 

research tools to answer the questions, and is well suited to answer important questions to 

advance our understanding of the world. 

Simply said, the pre-review is comprehensive, competitive, and, especially in the case of 

federally funded research such as by the EPA, conducted in a transparent process. And this is 

only Step 1. 

If the study involves human participants, there is another level of scrutiny to ensure the safety 

and well-being of the study participants. Scientists must submit their research proposal to an 

institutional review board (IRB), which is an independent ethics committee consisting of 

scientific professionals and community members to ensure that any risks to humans of the 

research are outweighed by the expected human health benefits of the research project. The 

IRB asks questions like: 

• What are the benefits and/or risks to the volunteers? 

• Does the benefit of the study outweigh the risk to volunteers? 

• How should the risk to volunteers be communicated? 

We help the li'Or!d !Jreathe P\JL .. (l>fA!'IY • CR<H<;Al CARl: • St~Et 

1150 18th Street, N.W., suite 300 

Washington, D.C. 20036 U.S. 

T. 202~296-9770 F. 202 296-9776 I thoradc.org 

ATS2017 lnterhational Conference 

May19·14,1017 

Washington, DC 

conference.thorac:ic.org 
117 

February 6, 2017 

Page 3 

• Is there appropriate monitoring during the study to ensure participants' safety? 

• Will the privacy of the participants' medical records be maintained? 

All scientific studies must protect the privacy of human subjects and their families to ensure 

that the Information gained in the study is not somehow used against them (e.g., by employers, 

health insurance companies selling policies, or by individuals via social media, for example). 

External Review Step 2: Publication Peer Review- the next step taken in peer-reviewed 

science follows when the results of the study are submitted for publication in scientific or 

academic journals. In many ways, Step 2 is a quality check on Step 1, including: 

• Did you execute the research project as you initially proposed? 

• Did you follow and record at the data end points that you collected? 

• Did you apply the correct statistical tools to analyze your data? 

• Are the conclusions from your data justified? 

• Are there alternative conclusions from your data? 

• How conld have this study been done better? 

These are all examples of questions that are applied in the publication peer view process. For 

any study published in the peer~reviewed literature, scientists are typically asked to return to 

their data and provide additional analyses during the peer review process to address reviewer 

concerns and to convince them that the findings are robust. The publication peer review 

process is also conducted by experts who know the field and are best able to judge the scientific 

methods of the study. 

External Review Step 3: Broad Community Review- After a study has been completed and 

published, its finding are then subject to review by the entire scientific community and the 

public. People who disagree with a research finding can publically comment on it, point out its 

potential weaknesses and present alternative interpretations of the study findings, and submit 

letters of disagreement for publication and response in the journal. Further, researchers from 

other institutions frequently conduct subsequent independent studies that either validate or 

refute the conclusions of fellow researchers. In this way, scientists compete with one another 

to advance our understanding of the world. ln the practice of medicine and puhlic health, a new 

scientific finding is generally not accepted as truth until it has been replicated in multiple 

studies. 

We hope these comments help the committee better understand the multiple requirements for 

peer review that ensure transparency in the scientific process. It is this scientific process that 

drives EPA's science program, and informs EPA's approach of comprehensively reviewing the 

peer~reviewed, published scientific studies when setting standards to protect human health. 

1150 18th Street, N.W., Suite 300 

Washington, D.C. 20036 U.S. 

T. 202~296-9770 F. 202 296*9776 I thoracic.org 

ATS2017 lnterhational Conference 

May 19-2~ 2017 

Washington, DC 

conference.thoracic.org 
118 

February 6, 2017 

Page 4 

Beyond the multi-stage peer review process that insures the quality and transparency of the 

scientific process in the US, it is also important for members of the House Science Committee to 

consider the criteria that the medical community (and Federal agencies such as the Institute of 

Medicine and EPA) uses to weigh the evidence from multiple studies. 

Consistency Across Studies: rarely does one study or one finding alone radically change our 

understanding of the world. Multiple independent studies are required to test the consistency 

and coherence of the results across studies. For example, the National Ambient Air Quality 

Standards for pollutants like ozone and particulate matter pollution are based on hundreds of 

studies. Each study adds to the overall understanding and assessment of the health effects of 

each pollutant 

Consistency Across Study Designs: Scientists use a wide range of scientific tools, approaches, 

and disciplines to advance our medical knowledge, with each enhancing, refining, and 

occasionally challenging the findings of sister scientific methods. In some cases, a scientific 

finding found using one type of study design (e.g. an observational questionnaire study) may 

not be replicated in a different study design, such as a randomized controlled trial, thus 

challenging the conclusion. The evidence used by the EPA to reach conclusions about the 

adverse health effects of ozone and particulate matter pollution are based on studies using a 

wide range of research approaches including: epidemiology studies, toxicology studies, animal 

exposure studies, human challenge studies, case control studies, and natural studies. 

Consistency of findings across multiple study designs substantially adds to the scientific 

plausibility of these findings. 

Secret Science Reform Act 

The ATS opposes legislation that the committee has previously considered, like the Secret 

Science Reform Act, that are not intended to improve the scientific process, but rather to 

suppress key studies from EPA's consideration, create a mandate on researchers to share 

information beyond what is already required in the peer review process, and to intimidate the 

research community. Measures such as these may also make it more challenging to conduct 

medical and environmental health research, because potential study participants will more 

likely refuse to participate when their private medical records are no longer secure, and might 

be made public in the name of transparency. It is our hope that to day's hearing will not be used 

as an avenue to discuss the Secret Science bill or other obstructionist legislation, but rather to 

explore ways to further improve the already excellent science programs at EPA. 

In the spirit of the hearing title, "Making EPA Great Again", the ATS offers the following concrete 

steps that Congress can take to improve operations of the EPA; 

Expand the EPA Science Program 

The EPA science program has funded high quality and relevant research that provides 

immediate benefit to the American tax payer by optimizing our nation's environmental 

protection efforts to focus only on the most health· relevant compounds. 

1150 18th Street, N.W., Suite 300 

Washington, D.C. 20036 U.S, 

T. 202·29&-.9770 F. 202 296-9776 I thoracic.org 

ATS2017 lnterhational Conference May 19-24, 2017 

Washington, DC 

conference.thoradc.org 
119 

February 6, 2017 

Page 5 

Research on the health effects of air pollution, in particular ozone and particulate matter 

poUution, have shown that reducing these pollutants provides immediate health benefits that 

far exceed implementation costs. However, we still do not know which types of particles are 

the most health damaging and, therefore, which should he the most efficient focus of any risk 

mitigation measures. Additional funding should therefore be provided to further our 

understanding of the effects specific environmental exposures on human health, and to thereby 

optimize approaches to reduce human health effect."! of po1lution. 

Expand EPA's Monitoring Network 

EPA's air pollution network is concentrated in large urban areas and at sites nearby expected 

sources of air pollution (major roads, power plants, industry facilities etc.). However, the air 

pollution levels in many small and rural communities are not monitored at all. These 

communities are also exposed to air pollution, including unique sources that do not affect urban 

communities, and they deserve the same level of public information and protection the EPA 

offers larger communities. We recommend Congress provide the funds needed to expand EPA's 

monitoring network to better protect both urban and rural populations. 

Expand Use of New Monitoring Technology 

Advances in satellite and personal monitoring technologies have the potential to both enhance 

and expand EPA's monitoring capability. Technological advancement has significantly lowered 

cost and improved the capability and reliability of small personal monitors that can be easily 

worn by an individual. These personal monitors have been used in many research studies 

addressing environmental exposures. EPA should take the next step in sponsoring research to 

better understand how to validate and incorporate data from personal devices into its air 

monitoring network, and to help doctors protect their most vulnerable patients from the health 

effects of pollution. 

Similarly, environmental satellite data have advanced significantly, and can allow EPA to 

remotely collect data on air pollution exposure levels, on both a local and global scale, including 

areas that are left out of the EPA's ground monitoring network. Additional EPA research 

funding is needed to understand the potential of personal and sateHite monitoring, and to 

develop protocols to validate and standardize data collected with different monitoring 

technologies into EPA's regulatory framework. 

We urge the Congress to provide the US EPA with the funds needed to explore and incorporate 

these new technologies into their science and regulatory programs1 so as to ultimately make 

their surveillance efforts and regulatory actions more focused and efficient in protecting the 

health of American children and adults. 

We help the world breathe· ~(JUHlNAitY • CRITICAl C~Rl • SHf~ 

1150 18th Street, N.W., Suite 300 

Washington, D.C. 20036 U.S. 

T. 202~296~9170 F. 202 296-9776 I thoracic.org 

AT52017 lnterhational Conference 

Mayl9·14,1017 

Washington, DC 

conference.thoracic,org 
120 

Board of Dirutors 

Linda Reinstein 

President 

Ellen Costa 

Secretary 

Doug Larkjn 

Member at Large 

\.auric Rice 

Member -at Large 

E!len Tunke!rott 

Trcasw:er 

National 

Spokesptrson 

Jordan Zevon 

ArbriSM)' Boards liaisoN 

Celeste Monforton, DrPH, 

!vfPH 

Sduue Adtlisory BotJrtl 

Arthur L. Frank, MD, PhD 

Co~Chair 

R1chard Lemen, PhD, MSPH 

Cn-Chair 

IkBradBlack 

Dr. Barry Castleman 

Dr. RajaF!ores 

Dr.Michaclliarbut 

Dr. Hedy Kindler 

Dr. Chrisrinl'" Oliver 

Brent Kynod1 

Co-Chair 

Omstlne\'t'inter 

Co-Chair 

lviarkCatlin 

Claire Deacon 

Tom Laubenthal 

John Kewquist 

Tony Rich 

Mark\\'imer 

'!tl1,fll'~· J,~ji o~no ,. AlWtos DimutAwamlm: fk.gllllimtion"' Voice of the Victims 

The Honorable Lamar Smith 

Chairman, Science, Space, and Technology Committee 

U.S. House of Representatives 

Via Fax: 202-225-8628 

The Honorable Eddie Johnson 

Ranking Member, Science, Space, and Tcchnolot,'Y Committee 

U.S. House of Representatives 

Via Fax: (202) 225-3895 

Dear Chairman Smith and Ranking }.1ember Johnson, 

February 7, 2017 

On behalf of the Asbestos Disease Awareness Organization (ADAO), the largest independent 

nonprofit dedicated to preventing asbestos exposure to eliminate deadly asbestos-caused 

diseases we are deeply concerned about today's hearing entitled "Making EPA Great Again." 

Instead of a hearing to discuss the dismantling the U.S. Environmental Protection Agency 

(EPA), you should be discussing ways to increase the agency's funcUng and staffing to protect 

public health and the environment. 

Since 2004, ADAO has been working with the White House, Congress, and Agencies on 

prevention and policy to end the asbestos man-made disaster. Recently, we worked with the 

I louse and Senate to pass TSCA reform legislation, which President Obama signed into law. 

I personaUy know the devastation of asbestos., as mesothelioma, an asbestos cancer, claimed 

my husband's life. Each year, 15,000 Americans die from preventable diseases caused from thi 

knmvn carcinogen, yet asbestos imports continue. LSGS reported that in 2015, asbestos 

consumption in the United States was estimated robe about 360 tons. Even more alarming, 

the chloralkali industry accounted for an esdmated 90% of U.S. ashcstos consumption. 

In addition, we urge you to stand strong against biUs such as II.R.861, a bill to terminate the 

Environmental Protection Agency, introduced by Representative ~Iatt Gaetz. 

American deserve to have air, water, and soil free from toxins and look to you and your 

committee to ensure the EPA can protect puhlic health and our environment today and the 

future 

Sincerely, 

Linda Reim;tein, President/CEO, Asbestos Disease Awareness Organization (ADAO) 

Asbestos Disease Awareness Organi7atlan is a registered 501(c) (3) nonprofit org~nization 

"U~~.ittJ j6f' A!but6t Di.f~au A11arr11,.r:t, Edtll:ation, AJuo&ary, at~d Co••Mnily SMppo,.t" 

1525 AYi~tion Boulevard. Suite 318 ·Redondo Beach · C~!ifornia · 90278 · (310) 251-7477 

·www./\sbc~to-,Di~easeA,>:areness.org 
121 

The Intercept_ 

REPUBLICANS ARE 

USING BIG TOBACCO'S 

SECRET SCIENCE 

PLAYBOOK TO GUT 

HEALTH RULES 

Sharon Lerner 

February 5 2017, 9:03 a.m. 

1 

MUCH OF THE COUNTRY has been watching in horror as 

Donald Trump has made good on his promises to eviscerate the 

Environmental Protection Agency- delaying 30 regulations, severely 

limiting the information staffers can release, and installing Scott 

Pruitt as the agency's administrator to destroy the agency from 

within. But even those keeping their eyes on the EPA may have 

missed a quieter attack on environmental protections now being 

launched in Congress. 

On Tuesday, the House Committee on Science, Space, and 

Technology is expected to hold a hearing on a bill to undermine 

health regulations that is based on a strategy cooked up by tobacco 

industry strategists more than two decades ago. At what Republicans 

on the committee have dubbed the "Making EPA Great Again" 

hearing, lawmakers are likely to discuss the Secret Science Reform 

Act, a bill that would limit the EPA to using only data that can be 

replicated or made available for "independent analysis." 

https://theintercept.com/2017/02/05/republicans-want-to-make-the-epa-great·again-by-gutting-health-regulations/ 
122 

2 

The proposal may sound reasonable enough at first. But because 

health research often contains confidential personal information that 

is illegal to share, the bill would prevent the EPA from using many of 

the best scientific studies. It would also prohibit using studies of onetime events, such as the Gulf oil spill or the effect of a partial ban 

of chlorpyrifos on children, which fueled the EPA's decision to 

eliminate all agricultural uses ofthe pesticide, because these eventsand thus the studies of them - can't be repeated. Although it is 

nominally about transparency, the bill leaves intact protections that 

allow industry to keep much of its own inner workings and skewed 

research secret from the public, while delegitimizing studies done by 

researchers with no vested interest in their outcome. 

The top-billed witness scheduled to provide testimony at the House 

hearing on Tuesday is a lawyer named Jeffrey Holmstead, who has 

has worked to block the EPA's efforts to limit mercury pollution while 

representing coal companies including Duke Energy, Progress 

Energy, and Southern Company. Meanwhile, Lamar Smith, the Texas 

Republican chair of the House Science Committee who has been 

zealously promoting the"secret science" bill, is also in the pocket of 

the energy companies. Though he's also received funding from Koch 

Industries and iHeartMedia (formerly Clear Channel 

Communications), Smith's biggest contributors are oil and 

gas companies, according to the Center for Responsive Politics. Also 

testifying on Tuesday will be Kimberly Smith of the American 

Chemistry Council, the chemical industry trade group. 

This bald industry bid to subvert public health-based regulations that 

can cut into profit isn't new. What's new is that this upside-down 

environmental attack, in which those who benefit directly from 

polluting industries are policing the independent scientists who can 

show the harms of their products, could now succeed. Although the 

House passed the secret science bill in 2014 and 2015, it never made 

it to the Senate floor. After it passed the House in 2015, Barbara 

Boxer called the bill "insane," Bernie Sanders called it "laughable," 

and President Obama promised to veto it. This time, it's not a joke. 

With a Republican majority in both houses and Trump in the White 

House, the secret science act could easily become law. 

https://theintercept.com/2017/02/05/republicans-want-to-make-the-epa-great-agaln-by-guttlng·health-regulatlons/ 
123 

THE EPA 

WRECKING 

CREW 

DAVID SCHNAR! 

Trump EPA transltion team;poeral 

counsel of EneiJY & Envirorvne-nt 

Legal Institute; harassed dimate 

scientists with FOIA requests and 

lawsufts 

Graphic: The Intercept 

MYRONEBELL 

He.ad at Trump EPA tn~onsition team; 

director or Competit!Ye Enterprt:se 

Institute; former member American 

Petrollllt.lm Institute Global Climate 

Setence Corntnooicatlons Team: 

fonnerpolicydirtctor, Frontiel'$ of 

Freedom, which has ties to 

Tobacco lnstthrt.e. Lori liard 

Tobacco, Philip Morris 

STEVEMIUOY 

Senior policy feUow, Enerav& 

Envin;mll'le"nt Leaallnstitute; 

Jorrneor member American 

Petroleum Institute Global Climate 

Setence Communtcatlons Team; 

former head of the Advancement 

of Sound Science Coalition, 11 front 

poup created and funded by aPR 

firm DCti!'IA for Phifip Moni:s; 

Breitbart author 

CRAIG RICHARDSON 

President, El'lergy & Environment 

Lepllnstftute; fonnerconsressional 

stafterfor Rep. Tom DeLay; ties to 

Philip Morris, Tobacc.o lnstltute,R.J. 

Reynolds Tobacco 

JAMES ENSTROM 

Former research fellow, Enet&Y & 

Environment le&al Institute; 

consulted for Ph~lp Morris,Cooncil 

for Tobacc:;o Research,Centetfor 

Indoor Air Research 

3 

CHAISTOPH!RHORNER 

Trump EPA tran4ttlon team; senior 

leJal fellow,Ener&Y &. Environment legal Institute; attorney 

reprenntina RJ. Reynolds Tobacco 

at Bracewell & Patterson,llP; 

harassed climate scl8ntists;former 

lobbvfst for Chemical MBnufacturers 

Association 

AMYOUVERCOOJ(E 

Trump EPA transition team; senior 

ITiftlia fellow, Ene~JY & Environment 

lepllnatitute:;dlrectorofthe 

EOf!lll' Policy Center for the 

Independence tnstrtute; founder of 

Mothers In love with A'acklng 

The small group oflawyers and PR strategists orchestrating the secret 

science effort are closely tied to those attacking the EPA from within. 

All have connections to either big tobacco, oil, or both- and almost 

all have been affiliated with a small, right-wing group called 

the Energy & Environment Legal Institute. It's interesting that E&E 

should fixate on transparency since the group has gone to great 

lengths to conceal its donors. Nevertheless, public records document 

some of the group's ties to big coal companies, including the now 

bankrupt Alpha Natural Resources, Peabody Coal, and Arch Coal. 

E&E senior policy fellow Steve Milloy, a former tobacco industry 

attorney, has perhaps written the most - at least publicly- about 

the secret science strategy, both in an ebook and for Steve 

Bannon's Breitbart News. Milloy calls Myron Ebell, who oversaw 

Trump's EPA transition team, his "friend and hero." In the late 

1990s, Milloy and Ebell were both members of the American 

Petroleum Institute's Global Climate Science Communications Team, 

https://thelntercept.com/2017 /02/05/republicans-want-to-make-the-epa-great-agaln-by-guttlng-health-regulatlons/ 
124 

which laid out the oil industry's strategy to undermine the science of 

global warming. Meanwhile, three of Milloy's colleagues from E&E 

are also members of the EPA landing team. Among them are David 

Schnare, E&E's general counsel, who is perhaps best known for 

harassing Michael Mann and other environmental scientists with 

FOIA requests, and Amy Oliver Cooke, an energy industry think 

tanker who created MILF, Mothers In Love with Fracking. 

Amy Oliver Cooke describes her love for fracking. 

4 

Two other E&E associates have been wrapped up in the secret science 

strategy for years. The first is Christopher Horner, a senior fellow at 

both E&E and the Competitive Enterprise Institute, who is also a 

member of Trump's EPA landing team. Back in the 1990s, Horner 

worked for Bracewell LLP, the law firm (formerly known as Bracewell 

& Giuliani) supplying the top witness at Tuesday's hearing. The 

dawning awareness of the dangers of second-hand smoke was putting 

tobacco companies on the defensive, including Horner's client, the 

R.J. Reynolds Tobacco Company. In a 1996 memo, which seems to be 

the earliest known reference to the secret science strategy, Horner 

laid out a plan to fight back. 

https://theintercept.com/2017/02/05/republlcans-want-to-make-the-epa-great-agaln-by-gutting-health-regulations/ 
125 

RJR7028305 

70090 0686 

From: Christopher Homer 

To: Hyde, Timothy N.: Tompson, Randy 

CC: 

BCC: 

Subject: Federal Agency Science 

Date: 12123/19961:56:01 PM 

Attachments: 

Gentlemen: The following Is the document we discussed. Have a happy 

holiday.CCH 

MEMORANDUM 

TO: Mr. Tim Hyde 

Mr. Randy Johnson 

RJ Reynolds Tobacco Company 

FROM: Mr. Christopher C. Homer 

Bracewell & Patterson, L.L.P. 

DATE: December23, 1996 

RE: Background and Proposed Program to Address Federal 

Agency Science 

Per our earlier conversations, the following sets forth what needs to be 

done to reform agency science, focusing on the need based upon your 

interests, and how you are positioned to take a behind the scenes 

leadership position. It provides an overview of the issues relevant to this 

goal, and details a program taking advantage of the Increasingly flagrant 

way regulators have perverted the scientifiC process. hiding behind a 

wall of selecled scientists to essentially cow industry and Congress into 

accepting fringe scientific conclusions. 

Summary 

We propose creating, beginning with congressional oversight and a goal 

of enacting legislation, required review procedures which EPA and other 

federal agencies must follOw In developing "extra-judicial" documents 

{i.e., those documents produced as guidance, science or other 

government products issued by regulatory agencies which are not 

necessarily at limo of publication ripe for judicial review). This is 

important to your organization because, at some point in the near future, 

EPA will most likely be ordered to re-examine ETS. Tho only way to do 

htlp:lllegacy.library.uasteduJiitllr,zb7;7aoolpdl!cwn0nts ilbrary uc;;l czlur1obaccoldocs,jh•,e\',00 1 ~~ . ''""t-t;a;11 ,, ,'l::,;;,c:1 ... :c.t.-... .:....,,,i ... v~ ,·1'V4..T .:.~tr ~t:r ... uo, .. o; •. rv-ari"P .. v-·,, ib",;..,..,, •c:-iot.-o""&>c:o· .. -agu .. ;-,; ~e.u,u• •e.-• d:Q'"' ,-, c:&u•ouv• ,~, • 

5 
126 

Hor~ner to fUR Hrynolds 199b f1r'dCP,Y1l G!ulla!lJ 

"We propose creating, beginning with congressional oversight and a 

goal of enacting legislation, required review procedures which EPA 

and other federal agencies must follow," Horner wrote in his memo. 

"This is important to your organization because, at some point in the 

near future, EPA will most likely be ordered to re-examine ETS 

[environmental tobacco smoke]." Horner's plan? "To construct 

explicit procedural hurdles the agency must follow in issuing 

scientific reports. Because there is virtually no chance of affecting 

change on this issue if the focus is ETS." 

6 

Horner already saw that the secret science approach could subvert far 

more than the imminent regulations based on the science about 

second-hand smoke. "Our approach is one of addressing process as 

opposed to scientific substance, and global applicability to industry 

rather than focusing on any single industrial sector," he wrote, going 

on to explain how the strategy could be used to interfere with the 

EPA's efforts to address mercury emissions, hazardous waste, and 

dioxins as well as restrictions on air pollution. 

The Attack on Air Pollution 

Protections 

By 1998, Powell Tate, a lobbying firm that represented R.J. Reynolds, 

had helped organize a secret science working group to look at 

questions of "data access," according to one internal memo. The 

memo clarified that its intention was to "focus public opinion on the 

importance of requiring the disclosure of tax-payer funded analytical 

data." 

Though it was the brainchild of tobacco strategists, the energy 

industry soon followed Horner's advice and adopted the secret 

science approach as a way to hamper air quality improvement efforts. 

In the 1990s, the EPA began efforts to reduce the amount of tiny 

https://theintercept.com/2017/02/05/republicans-want-to-make-the-epa-great-agaln-by-guttlng-health-regulatlons/ 
127 

particles in the air, a kind of pollution known as PM 2.5, that are 

produced by combustion from power plants, cars, and 

manufacturing. The clearest evidence of the need to limit such 

particles came from the "Six Cities" study, in which a team of 

Harvard researchers clearly tied higher levels of PM 2.5 particles to 

increased mortality, as well as lung cancer, asthma, and sudden 

infant death syndrome. 

7 

While the new limits were designed to save lives - preventing 15,000 

premature deaths annually, according to EPA projections - the rules 

would also increase costs in some sectors by, for instance, making 

energy companies install pollution equipment. In response, a group 

funded by the Koch brothers rose up to challenge the EPA and the 

scientists on the grounds that scientists were hiding their data from 

the public. Citizens for a Sound Economy, a forerunner of the Koch 

brothers' current Freedom Works, demanded that the Harvard 

researchers provide their original data so an "independent" scientist 

could analyze it. 

At first the researchers refused to share the data, which they had 

collected from individuals with the promise that their health 

information would remain confidential. Eventually, after an elaborate 

and expensive pressure campaign, the Six Cities researchers agreed 

to allow their data to be reanalyzed by two separate teams of 

researchers. Both confirmed the group's findings that rates of PM 2.5 

were correlated with increased mortality. 

The EPA went on to institute the changes. And scientists throughout 

the world have since come to recognize the dangers posed by small 

particle air pollution, which accounted for "over 2.1 million 

premature deaths and 52 million years of healthy life lost in 2010," 

according to the 2010 Global Burden of Disease report. The report 

drew on research by more than 450 experts from around the world 

and was led by the Institute of Health Metrics and Evaluation at the 

University of Washington; the World Health Organization; the 

University of Queensland, Australia; Johns Hopkins University; and 

Harvard University. 

https://theintercept.com/2017/02/05/republlcans-want-to-make-the-epa-great-again-by-gutting-health-regulations/ 
128 

Despite the scientific consensus, a small group of extremists has 

continued to fixate on the idea that the science on the dangers of air 

pollution is somehow a sham. Even more disturbingly, this small 

extreme group now holds sway in key parts of the U.S. government. 

Not least among them is Rep. Lamar Smith, who in 

2013 subpoenaed the EPA in yet another effort to obtain the data 

from the Six Cities study. 

In an op-ed that ran in the Wall Street Journal shortly afterward, 

Smith noted that "the data in question have not been subjected to 

scrutiny and analysis by independent scientists." Smith pressed his 

point in a House Science Committee hearing a few days later, 

insisting that independent scientists were being denied access to the 

air pollution data. When Democrat Donna Edwards pressed Smith 

about who these scientists were, he mentioned the name Jim 

Enstrom. 

Enstrom, you may not be surprised to learn, has been a research 

fellow at E&E and has received money from the Council for Tobacco 

Research, the Tobacco Institute, Philip Morris, and R.J. Reynolds. In 

part because he didn't disclose his tobacco industry ties in a study he 

did on the connection between second-hand smoke and mortality 

(which he found to be inconclusive), he was widely criticized by the 

scientific community, including the American Cancer Society, and 

was subsequently dismissed from UCLA. 

https://theintercept.com/2017/02/05/republicans-want-to-make-the-epa-great-again-by-gutting-health-regulations/ 

8 
129 

SCIENCE 

'Whistle blower' says protocol was breached but no data fraud 

Scott Waldman, E&E News reporter 

House Science Space and Technology Chairman Lamar Smith (R-Texas). Photo by Bill Clark, 

courtesy of AP Images. 

The federal climate scientist hailed by conservatives as a whistleblower for allegedly revealing 

manipulated global warming data said yesterday he was actually calling out a former colleague 

for not properly following agency standards for research. 

In an interview with E&E News yesterday, former National Oceanic and Atmospheric 

Administration principal scientist John Bates had a significantly more nuanced take on the 

controversy that has swirled since a top House Republican hailed his blog post as proof that the 

agency "played fast and loose" with temperature data to disprove the theory of a global warming 

"pause." 

Bates accused former colleagues of rushing their research to publication, in defiance of agency 

protocol. He specified that he did not believe that they manipulated the data upon which the 

research relied in any way. 

"The issue here is not an issue of tampering with data, but rather really of timing of a release of a 

paper that had not properly disclosed everything it was," he said. 

Bates, who recently retired from NOAA's National Climatic Data Center, claimed in his post that 

the agency rushed research disproving the global warming pause to publish in Science magazine 

before the December 2015 Paris climate talks. Climate skeptics have called that proof of massive 

fraud among federal climate researchers and said it allowed world leaders to be "duped" into 

signing the Paris climate agreement to reduce carbon emissions from fossil fuel use. 
130 

Bates said the NOAA study relied on land data that were "experimental." Typically, NOAA 

officials can publish research that relies partially on experimental data, as long as the data are 

properly identified, especially if there is an urgent situation that requires something to go out 

quickly because it is related to human health, safety and the environment. 

The publishing safeguards are important, he said, because they help protect federal research 

against lawsuits. Bates added that science suffers !fits results cannot be reproduced. 

Yesterday, the House Science, Space and Technology Committee portrayed Bates' allegations as 

a bombshell that required immediate investigation. 

Former National Oceanic and Atmospheric Administration meteorologist John Bates says the 

agency broke its own rules for scientific integrity when it published a study debunking the 

"hiatus" in global warming. Photo courtesy of the American Geophysical Union. 

Committee Chairman Lamar Smith (R-Texas) has issued subpoenas and has for nearly two years 

attempted to obtain scientists' emails involved in the global warming pause research. A Science 

Committee aide yesterday said Bates' revelation was evidence that NOAA needed further 

investigation because its own employees were identifying significant policy breaches. 

The aide said the committee would again seek the em ails of federal researchers, and if a formal 

request were ignored, another round of subpoenas could be issued or scientists might be forced to 

testify in front of the committee. 

"I think the brushback that the committee received, and the chairman received consistently, about 

how science is capable of policing itself and doesn't need anyone outside asking questions, even 

when the science being discussed is paid for and performed by scientists paid for with the 

taxpayer's money and used to implement far-reaching federal policies or justify implementation 

of far-reaching federal policies, doesn't really work," the aide said·. 

'An incredibly bizarre claim' 

2 
131 

Bates laid out his claims, which are largely technical and related to the sharing of data, on the 

blog run by Judith Curry, a climate scientist who has broken with many colleagues and called 

into question the actual extent of humanity's influence on the planet. 

The report's authors, Bates wrote, put a "thumb on the scale- in the documentation, scientific 

choices, and release of datasets- in an effort to discredit the notion of a global warming hiatus 

and rush to time the publication of the paper to influence national and international deliberations 

on climate policy." 

The NOAA administrator under former President Obama, Kathryn Sullivan, refused to turn over 

the emails because she said doing so could chill the scientific process by making it harder for 

researchers to communicate openly while they were actively engaged in research. Smith's 

committee threatened her with criminal charges. The issue is expected to become part of the 

committee's hearing today into the use of scientific research in crafting federal regulations. 

Yesterday, a NOAA spokesman did not directly address the specific allegations, other than to say 

that they are currently under review. 

"NOAA is charged with providing peer-reviewed data to the American public and stands behind 

its world-class scientists," said the spokesman, who declined to be named. "NOAA takes 

seriously any allegation that its internal processes have not been followed and will review the 

matter appropriately." 

Whether the research was published to influence the Paris climate talks is a moot point, said 

Andrew Light, a senior member of the State Department's climate talks negotiating team in 2015. 

He said the talks had already been underway for about four years when the paper was published 

and that 188 nations were relying on a tremendous amount of research to support their goal of 

reducing humans' carbon emissions to slow the warming of the planet. They had also already 

crafted proposed reductions by the time the research was published, he said. 

"I never heard it discussed once, let alone this one NOAA report, discussed in Paris, the run-up 

to Paris or anything after Paris, so this is really just an incredibly bizarre claim," Light said. 

Bates: Be careful of bias 

For many years, climate scientists were puzzled by an apparent plateau in global temperature rise 

from 1998 to 2012 as ocean temperatures stayed consistent. The 2015 research paper addressed 

the issue when it found there was no pause because the method to collect ocean temperatures was 

flawed. 

Since then, multiple independent studies have confirmed NOAA's findings, including one 

published last month in Science Advances. 

3 
132 

Buoys and satellites support NOAA record 

G 

l"~ 

I~ 

F 

I~ 

jo 

--- --- ---

J~ '---.-----.,-----,.-•:::.-=.;c-=· :.::"""';.:.;;:..,-;.=.. 2000 111005 2010 2015 

l±LA study earlier this year using data from buoys, satellites and Argo floats backs up a 

challenge of the so-called global warming pause by NOAA. Graph courtesy of the University of 

York Department of Chemistry. 

That study replicated NOAA's findings by accounting for different methods of temperature 

collection over time. For instance, data collected in the engine rooms of ships show slightly 

elevated levels of warming compared with those collected by buoys. When researchers 

accounted for that discrepancy, the so-called global warming pause disappears, researchers 

found. 

The American Geophysical Union, which represents thousands of scientists who study climate, 

pointed out that the results of the 2015 study had been discussed in peer-reviewed journals and 

that multiple studies had independently backed up the findings. 

The reports do not change the fundamental understanding of climate change science, AGU 

President Eric Davidson wrote in his blog yesterday. 

"These types of statements by policymakers that attempt to take one study/dispute and blow it 

out of proportion are both unhelpful and misleading," he wrote. "We will be working with the 

science committee to demonstrate the scientific consensus on climate change and to encourage 

them not to interfere with the scientific process." 

Yesterday, Bates said he was contacted by the Science Committee for the first time only after the 

story broke. He said he has not communicated with anyone there before and was not a 

whistleblower for the committee previously but that he expected to be invited to Washington to 

testify at a future hearing. 

He said he would accept such an invitation, but cautioned scientists against advocating policy. 

"You really have to provide the most objective view and let the policymakers decide from their 

role," Bates said. "I'm getting much more wary of scientists growing into too much advocacy. I 

think there is certainly a role there, and yet people have to really examine themselves for their 

own bias and be careful about that." 

This story also appears in E&E Daily. 

4 
133 

Twitter: @scottpwaldman Email: swaldman@eenews.net 

5 
134 

The EPA Is Already Great 

The effort on Capitol Hill to undermine the EPA is at once a national tragedy and deeply 

puzzling. 

{•\ ~" :t'r;c;!"'J ~,;, i'>'i d~ ""i;di;;t .. il",: !.;:: • 

By David Dyjack I Contributor 

Feb. 7, 2017, at 12:00 p.m. 

This is not the time to limit the Environmental Protection Agency's scope or throttle back its 

efforts. Americans should empower the agency to do more on behalf of our families and our 

standard of living. The agency's successes are worthy of celebration; the cost of retreat is 

staggering. 

Did you know we could save an estimated 5.5 million IQ points by permanently removing lead 

paint and dust from homes? The economic return on this investment is estimated at $68 billion. 

For every $1 invested in lead reduction, there is up to an estimated $221 return on that dollar. 

Were you aware that the EPA-administered Clean Air Act is a public health success story? In its 

first 20 years, more than 200,000 premature deaths and 18 million cases of respiratory illness in 

children were prevented. Estimates suggest that total benefits of the Clean Air Act amount to 

more than 40 times the costs of regulation. 

Americans enjoy some of the safest and most reliable drinking water in the world because of 

EPA's Safe Drinking Water Act. More than 90 percent of water customers enjoy drinking water 

that meets all health standards all the time. Alternately, Flint, Michigan, is a case study, which 

demonstrates why the absence of informed, credentialed environmental health professionals in 

decisions regarding water safety can lead to disaster. 
135 

The EPA Clean Water Act regulates outdoor water quality in rivers and streams. It has been a 

remarkable success since 1972 and is arguably the crown jewel of the EPA, resulting in water 

quality improvements and preventing water pollution where people boat, swim and fish. 

These opportunities and successes showcase the essence of a government agency that is essential 

to life in our society. The current effort on Capitol Hill to undermine the EPA is at once a 

national tragedy and deeply puzzling. 

We understand a comically awful, recycled piece of legislation, the Secret Science Reform Act, 

is about to take center stage once again. The proposed law is based on a bogus proposition, that 

somehow the EPA is exploiting secret studies to support its rule-making, and is intended to 

undermine the EPA. 

In full disclosure, this is personal for me. The proposed law's intent is to publicize the medical 

records of people involved in health studies. You read that correctly. If you participate in 

government-funded research, your private medical information could become fodder for public 

consumption. 

For the record, I am part of a National Cancer Institute-sponsored longitudinal study, which 

examines the relationship between lifestyle and disease. Do I want my 100-plus page lifestyle 

questionnaire made public? Of course not. So the net effect of the Secret Science Act will be that 

fewer people will elect to participate in research, and the opportunity to discover innovative 

solutions to today's problems will be diminished. This is a page taken directly from the tobacco 

industry's playbook and is shameful. 

Environmental health issues are profoundly local. These issues flow from your kitchen faucet, 

are found in the meat and produce sections of your neighborhood grocery store, present 

themselves at your local elementary school and appear in your beaches and streams. The nation's 

90,000 environmental scientists are generally modestly paid, humble civil servants, who live in 

local communities throughout our great country. They rely on EPA science and guidance 

documents in their efforts to ensure the health, safety and security of you and your family. 

The EPA does not, and has not, suppressed or concealed scientific information in its rulemaking. Congress should fully fund the EPA and reject sham legislation. The EPA is already 

great. Let's keep it that way. 

Tags: EPA, environment, water, pollution, science, public health, Congress 

David Dyjack is executive director of the National Environmental Health Association. 
136 

SCIENTIFIC REPORTS 

0 

OPEN On the definition and identifiability 

of the alleged "hiatus" in global 

warmmg Received:12 May2015 

Accepted: 20 October 2015 Stephan Lewandowsky1

'

2 James 5, Risbey3 & Naomi Oreskes4 

Pub!ished:2.1,November2015 

Recent public debate and the scientific literature have frequently cited a "pause" or "hiatus" in global 

warming. Yet, multiple sources of evidence show that climate change continues unabated, raising 

questions about the status of the "hiatus". To examine whether the notion of a "hiatus" is justified 

by the available data, we first document that there are multiple definitions of the "hiatus" in the 

literature, with its presumed onset spanning a decade. For each of these definitions we compare 

the associated temperature trend against trends of equivalent length in the entire record of modem 

global warming. The analysis shows that the "hiatus" trends are encompassed within the overall 

distribution of observed trends. We next assess the magnitude and significance of all possible trends 

up to 25 years duration looking backwards from each year over the past 30 years. At every year 

during the past 30 years, the immediately preceding warming trend was always significant when 17 

years (or more) were included in the calculation, alleged "hiatus" periods notwithstanding. If current 

definitions of the "pause" used in the literature are applied to the historical record, then the climate 

system "paused" for more than 1/3 of the period during which temperatures rose 0.6 K. 

"lhere was no such thing as the Scientific Revolution, and this is a book about it:' 

--Steven Shapin, 1996, The scientific revolution. 

University of Chicago Press. 

In the public sphere, the claim that global warming has "stopped" has long been a contrarian talking 

pointU. After being confined to the media and internet blogs for some time, this contrarian framing 

eventually found entry into the scientific literatureM, which is now replete with articles that address a 

presumed recent "pause" or "hiatus" in global warming5• The «hiatus" also featured as an accepted fact in 

the latest assessment report of the IPCC6 • Despite its widespread acceptance in the scientific community, 

there are reasons to be skeptical of the existence of the "hiatus"5. 

Recently, possible artifacts iu the global surface temperature record have been noted which, when 

corrected, suggest that there is little evidence for a "hiatus" relative to the long-term trend used by the 

!PCC7

• In addition, multiple other indicators such as ocean heat content point to continued warming8- 10• 

In this article, we show that even putting aside possible artifacts in the temperature record, there is 

no substantive evidence for a "pause" or "hiatus" in warming. Vole suggest that the use of those terms 

is therefore inaccurate. Because this conclusion appears to contradict the IPCC's explicit endorsement 

of the "hiatus': it is important to differentiate between the different ways in which the term "pause" or 

"hiatus" has been motivated and used in the recent climatological literature. 

1Schoo! of Experimental Psychology and Cabot Institute University of Bristol, 12A Priory Road, Bristol, BSB 1 TU, 

United Kingdom. 2School of Psychology, University of Western Australia, 35 Stirling Highway, Crawley, WA 6009 

Australia. 3CSIRO Oceans and Atmosphere, Box 1538, Hobart, Tasmania 7001 Australia. "Department of the 

History of Science, Science Center 371 Harvard University, 1 Oxford Street, Cambridge, Massachusl"tts a2138, USA. 

Correspondence and requests for materials should be addressed to S.L. (email: stephan.lewando>NSky@bristol.ac.uk) 

SCIENTIFIC REPORTSI5:16784! 001: 10.1038/srep16784 
137 

Research on the "hiatus" has been couched within at least 4 distinct research questions: (I) Is there a 

"pause" or "hiatus~ in warming? (2) Has warming slowed significantly compared to the long-term trend? 

{3) Has warming lagged behind model-derived expectations? {4) \Vhat are the physical mechanisms 

responsible for the "hiatus''? Here, we are exclusively concerned with the first question: Is there, or has 

there recently been, a "pause" or "hiatus" in warming? \Ve focus on this question because it is ineluctably tied to the contrarian claim that global warming has "stopped': which has dem(mstrably affected 

the political and media landscape" as well as, arguably, the scientific community1• "lhe question whether 

there is a "pause" in global warming can be readily tested: Standard dictionary definitions of the words 

"pause'' or "hiatus" imply that a process has been suspended or interrupted. It follows that for the notion 

of a "hiatus" in global warming to be scientifically well-founded, there must either be a demonstrable 

and statistically-relevant absence of any trend in global mean surface temperature {GMST) during the 

time period that is considered relevant or, minimally, the observed trend must differ in a statistically 

identifiable way from the historical record. 

Our focus on the question whether there is a "hiatus" or "pause" implies that we do not address two 

related issues: First, we arc not concerned with the differences, if any, between climate model projections 

and observed GMST trends, We have addressed the issue whether or not warming has lagged behind 

model-derived expectations elsewhere 11 , and this issue has no bearing on the existence of a "hiatus", 

Second, we are not concerned with the underlying physical processes that may ex:plain fluctuations, 

whether positive or negJ.tive, in GMS'L This is again a different question, which is interesting in its own 

right but has no bearing on the existence of a "hiatus", 

We examine the status of the "hiatus" in three steps; First, we compile an inventory of operationaHzations of the "hiatus" in the existing scientific literature and ask whether they converge on a consistent 

defmition. Second, we ask whether the rate of temperature change during the "hiatus': as it is operationalized in the literature, differs meaningfully from the set of rates for equivalent trend lengths observed 

during the era of modern climate change. 1his comparison is essential because any trend will exhibit 

periods of statistical insignificance when the sample size (i.e., number of years considered) is small: 'lhc 

existence of the presumed "hiatus" thus cannot be ascertained without a historical comparison to other 

comparable trend durations at earlier times during which warming was consensually thought to be present. Finally, for the same reason, we ask whether the duration of periods in which there is no significant 

warming has changed during the presumed "hiatus" relative to the rest of the modern period. 

Results 

There is no agreed "hiatus" period in the scientific literature. We catalogued a corpus of peer-reviewed articles published between 2009 and 2014 that specifically addressed the presumed "hiatus" in 

global warming. Table I shows that the term "hiatus" was used more than 550 times in this corpus, and 

the word "pause" in excess of 70 times. 

Many articles assumed that the "hiatus" commenced around 1998, at which time temperature anomalies were considerably above the long-term trend. There is, however, considerable heterogeneity in published onset times, with the range spanning a decade (1993-2003}. Similarly, there is considerable 

heterogeneity in the presumed duration of the "hiatus" across the same corpus of articles, with a range 

10-20 (median I3 years, m = 13.5, s 2.86), For each article, we took the duration to be the number 

of years since the assumed onset of the "hiatus" to the end of the period being analyzed. This constitutes 

a lower bound on the presumed duration of the ''hiatus" as some authors may have presumed that the 

"hiatus" was ongoing at the time they published an article. Figure 1 shows the modern global temperature 

data together with a histogram of the distribution of presumed onset times of the "hiatus" derived from 

the corpus, 

The heterogeneity in onset and duration raises the possibility that the use of the term "hiatus" departs 

from normal scientific practice, which strives to define phenomena on the hasis of dear and generally 

acCt-'Pted criteria. The heterogeneity may he explained by the supposition that authors defmed the "hia~ 

tus" retrospectively, via an ad hoc analysis of the recent trend leading up to the time of writing, rather 

than on the basis of a priori criteria. This apparent lack of clear and a priori criteria must be of concern 

in the statistical environment in which the "hiatus" has unfolded, which is known to be sensitive to the 

particular choice of start and end points that define short-term trends and the comparison base!ine11• 

The "hiatus" is an unexceptional fluctuation. If the definitions of the presumed hiatus are highly 

variable, wilh many differenl time periods proposed in the literature, how can we determine whether 

or not there is one? In order to !lnswer this question, we compared the distribution of decadal warming 

trends during the "hiatus" --as defined by the articles in the corpus--against the distribution of all possible trends that have been observed during the period of modern global warming. The results are shown 

in Fig. 2, using three different onset dates for global warming. 

The question of when, precisely, greenhottse-driven warming began to be observable against background natural variability is itself contested. An early review13 that examined the literature back to 1824 

finds that scientific concern about global warming arose as early as 1938. Every decade since then has 

seen increased scientific attention and concem13, although no consensual onset date for global v.'arming 

has been identified. Figure 2 therefore uses three different onset dates for Lhe computation of al! possible 

SCIENTIFIC REPORTSI5:16784I DOL 10,1036/srep16784 
138 

SC!E"'T!F!C REPORTS I 5:16784 j DO!: 10.103B/srepl67Bio 
139 

SCIENTIFIC REPORTS j5:16784j DO!: 10.1038/srep16784 
140 

Table I. Summary of literature on the "hiatus'~ 

Figure I. Global mean surfa(;e temperature (GMST) anomalies estimated by NASA's Goddard Institute 

for Spa(;e Studies (GISS) data set (1° http:!/data.giss.nasa.gov/gistemp/, all analyses based on dataset 

downloaded on 17 January 2015), '!he histogram at the bottom represents the distribution of presumed 

start years for the presumed "hiatus" in the corpus of arlic\es (N = 40; see Table I) considered for this 

analysis. lhe venicallincs represent the 5th (1993) and 95th (2001) percentile, respectively, of presumed 

starting years for the "hiatus~ The small inset shows the overall historical temperature anomalies recorded 

since 1880. 

trends. Panel A uses the period 1951-2012, which was used by the IPCC in AR5 as the long term trend 

against which to define the "hiatus"j). Panel Buses 1964 as the onset of modern global warming, whereas 

Panel C uses l976. "!hose two years are two standard deviations (s = 3) below and above, respectively, 

of the best estimate (l970) of the onset of modern global warming in the GISS data set reported in a 

recent change-point analysis 14, Panels Band C therefore approximate the lower and upper bound, respectively, of the 95% confidence interval for the onset of modern global warming hy the change-point 

measure, All panels include data through 2012 because many of the articles in the corpus were written 

when the latest available data were for 2012 (or even earlier). (See the Online Supplementary Material 

for an extension of our analysis to the entire instrumental record.) 

To permit a commensurable comparison, in all panels the distribution of all possible trends has the 

same propensity of trend durations as the ''hiatus" in the corpus. 'lhus, each possible 10-year trend is 

replicated 8 times (as 8 articles in the corpus presumed the "hiatus~ to extend over 10 years), each ll-year 

trend 5 times, and so on as determined by the propensity of trend durations in the corpus, 'I he distribution of trend durations is therefore identical between the two histograms in each panel. 

Figure 2 demonstrates that, although the distribution of trends during the "hiatus" is shifted downward compared to the overall distribution of trends of the same durations, the "hiatus" distribution falls 

within the overall envelope of historically observed trends. For the IPCC base period (1951-2012; Panel 

A) there is liule discernable difference between the two distributions. For the two years that brackel the 

SC!ENTlF!C REPORTS! 5·1678"1 DOl: 10.1038/srepl6784 
141 

A 

B 

c 

Figure 2. (A) distribution of observed decadal temperature trends (GTSS) within the "hiatus" windows 

defined hy the corpus of articles considered for this analysis (blue), compared to the distnbution of a!! 

possible temperature trends from 1950 till2012, the reference period used by the IPCC to establish the longterm warming trend (pink). (B) same distribution of temperature trends within the "hiatus" windows (blue) 

compared to the distribution of all possible temperature trends from 1964 till 2012 (pink). 'Ihc year 1964 is 

the lower bound fnr the 95% confidence il1tcrval of a recent change-point analysis that sought to identify the 

SCIENTIFIC REPORTS j 5:16784 j DO!: 10.1038/SrEpl6784 
142 

(C) same distribution of temperature trends v.rithin the "hiatus" windows 

(blue) comparet! to the of all possih!c temperature trends from 1976 till 2012 (pink). 'I he year 

1976 is the upper bound for the 95% confidence interval of a recent change-point analysis that sought to 

identify the onset of modern global warming. In all panels, the distribution of all possible trends is obtained 

by computing all trends of a given duration from all possible years within the time period considered. 1he 

duration of trends is weighted by the propensity of presumed "hiatus" durations in the corpus. '!hus, each 

10-year trend is replicated 8 times (a;, 8 articles in the corpus presumed the "hiatus" to extend over lO 

years), each 11-year trend 5 times, and so nn. Sec Table I for details of the distribution of presumed '"hiatus" 

durations in the corpus. ]he vertical red lines in each panel represents the long-term trend (1951-2012) that 

was used by the IPCC in their Fifth Assessment Report as a bencl;mark for comparison with the ''hiatus:' 

1he solid line is for the GISS dataset40 analyzed here, and the dashed line is the same long-term trend using 

the UK Met Office's HadCRUT4 data setl9 used by the IPCC. 

most likely change-point onset of the modern warming period (Panels B and C), the "hiatus" distribntion 

is more clearly offset towards the lower end but it is by no means unusual or extreme. 

Moreover, for nearly 15% of imputed "hiatus" trends (5 out of 40 articles in the corpus), the warming 

exceeded the long-term trend used by the IPCC (1951-2012; vertical red lines in Fig. 2). Similarly, nearly 

20% of operationalizations (7/40) referred to a period during which temperatures increased significantly 

(i.e., p<.OS in OLS regression), which is not consistent with a "hiatus:' 

lhe results in Fig. 2 show that all operationalizations of the "hiatus" in the literature are unexceptional 

in the context of equivalent-length trends in the record of modern global warming. At most, the operationalizations in the literature support the conclusion that the rates of warming over some recent intervals have been toward the lower end of the historically-observed surface temperature record. However, 

they do not support the conclusion that there is a "pause" or "hiatus" in the warming. 

The "hiatus" has always been there when sample size is small. We next analyzed the GMST 

data from all possible dilfeTent vantage points (end years looking back in time) to examine whether a 

scientist in, say, 2014 or 2010 would have been justified in accepting the existence of a "hiatus" in warming relative to what would have been detectable at any other prior point in time. 

Figure 3(A) shows the warming trends that were observable, given the available data at the time, for 

any vantage point between 1984 and 2014 (horizontal axis). For each vantage point, between 3 and 25 

years were included in the trend calculation (vertical axis). lhe Online Supplementary Material extends 

this analysis to even longer time scales. Timescales of at least 17 years are known to be necessary for 

noise reduction and detection of a signal 15• 

Figure 3(A) shows that at every year (vantage point) during the past 30 years, Lhe immediately preced~ 

ing warming trend was always significant when 17 years (or more) were included in the calculation (dots 

denote p < .05). figure 3{B) presents the same data using a ternary classification of p-values for the 

linear trend into non-informative (beige), partially informative but not conventionally significant (gray), 

and significant (terracotta). This panel also includes three diagonal lines that identify the earliest calendar 

year included in the analysis. lhus, any ob..~ervation to the Southeast of the line labeled "1975" only 

includes observations later than that, and so on for the other two Hnes. The observations to the Northwest 

of "1965" go back to 1960 (top-left corner; looking back 25 years from 1984 inclusive). 

"lhe large beige area in Panel H highlights the well-known fact that when sample size is small, statistical power is often insufficient to differentiate signal from noise. Conversely, the large terracotta area 

highlights the fact that when power is sufficient, the warming signal has been detectable at any point 

during the last 30 years, irrespective of vantage point. When one extends the period looking backwards 

in time, lhe warming trend is always signiftcant, and the most recent vantage point{s) do not differ systematically from earlier wntage points. It follows that the data do not permit identification of a "pause" 

or "hiatus" during tbe last 10-20 years. Signiftcantly, this conclusion is unaffected by the choice of year 

taken to represent the onset of modern warming (Le., areas to the Southeast of all 3 diagonal lines in 

Figure 3(B) permit the same conclusion). 'I he conclusion is also unafiectcd by the choice of the year 

during which the "pause" was examined (i.e., the vantage point). 

Conversely, if one uses shorter time periods of analysis, one can find many "pauses:' Using the operationalizations found in the corpus (mean duration 13.5 years), and a null hypothesis of no warming, we 

f1nd that the climate "paused" strikingly often during the last 30 years. During that period, the 14-year 

trend escaped significance tO times and the B-year trend l3 times, sugbresting that a "pause" occupied 

between 30% and 43% of a time period during which the climate warmed 0.6K overall (Fig. l). 1f the 

duration of the defined "hiatus" drops to below 12 years-which applies to 13 out of 40 articles (i.e., 

32.5%) in the corpus-then almosl everything is a "hiatus'; as signifi.ed by the preponderance of beige for 

trends of this duration in Fig. 3(B). Anyone making a "hiatus" claim of this duration will almost always 

find one, not because something new and different is happening, but because of the fundamental fact 

that smaU sample sizes provide insutlkient statistical power for the detection of trends. Thu~. a third of 

the articles in the corpus either presumed that the climate has nearly always "paused" during the last 30 

SctENT!FIC REPORTS! 5:16784! 001: 10.1038{srepl6784 
143 

A 

B 

Figure 3. (A) Observed magnitude of temperature trends (GISS, K/decade) as a function of vantage year 

and the number of years included in the computation of the trend. Trends are capped at ± l K for plotting. 

For each vantage year, trends are computed for all possible windows between 3 and 25 years duration, all of 

which end with the particular vantage year. ]be dots indicate whidl trends are significant (p < .05) in an 

ordinary least squares analysis of annual means, and the horiwntal dashed line indicates the number of 

years that must be included (N=l7) for the trend to be signiftcant from all vantage points. lhe open circles 

identify combinations of onset and duration that have hecn used to identify the "hiatus" by articles in the 

corpus. Mulliplc articles may contribute to a given circle. '!he Online Supplementary Material shows that the 

basic conclusions are unaffected hy consideration of autocorrdations, although an additional 2 years are 

required to reach significance for all vantage points across the entire 30-ycar period. (B) Level of statistical 

significance for trends (GISS, K/decade) as a function of vantage year and the number of years included in 

the computation of the trend. Trends that arc clearly non-significant (p > .10) are shown in beige, those that 

approach significance (.05 < p < .10) are shown in shades of gray, and significant trends (p < .05) are 

shown in shades of terracotta. '!he diagonal lines identify calendar years that contribute to the analysis. Any 

observation in the grid that lies to the Southeast of a given line includes only observations from the stated 

year onward, and any observation to the Northwest also includes earlier years. TilC observation in the topleft corner is 1960 (Le., lookmg backward 25 years from 1984). 

years (rendering the term meaningless), or they inconsistently highlighted only one of many events that 

would qualify with their definition. 

These results have been replicated using a variety of additional methods that incorporate autocorrelations in the time series (see the Online Supplementary Material). The results are not sensitive to the 

trend detection methods employed, and they are also not sensitive to the choke of GMST data set (see 

the Online Supplementary Material). 

We conclude that the evidence does not support the notion of a "pause" or "hiatus" as an identifiable 

phenomenon that is implied by standard dictionary definitions and common understandings of these 

terms. 

SCIENTIFIC REPORTS [ 5.16784 [DO!: 10.1038/srep16784 
144 

Discussion 

We recognize that our claim that there is no "hiatus" will be controversial, particularly in light of the 

widespread embrace of the "hiatus" in public and scientific discourse. Therefore, it is important to clarify what we are not claiming. First, and perhaps most important, we do not argue against the merit 

of research on decaJal-scale variation in the climate, On the contrary, the numerous articles on the 

"hiatus" have contributed to our understanding of what drives decadal fluctuations in climate, including 

for example its seasonal aspects16, Notably, none of the articles in our corpus indicate that they expect 

the "hiatus" to continue indefinitely, implying: that they do not support some public interpretations that 

recent fluctuations in the GMST undermine the scientific basis for understanding anthropogenic climate 

change 17• 

Second, our exclusive focus on GMST relative to the null hypothesis of no trend was mandated by our 

goal to examine the notion of a "pause" or "hiatus" with respect to the observations alone. It docs not 

follow that global trends constitute the only-or even preferable--level of analysis for the climate system. 

Third, we do not explicitly address the question whether warming has slowed significantly during 

the presumed "hiatus" period, although we have suggested elsewhere that it has not5

• In confirmation, 

a recent change· point analysis of GMST has shown that there is no statistically-identifiable change in 

warming trend after the 1970s14• 

Fourth, our analysis does not speak to the apparent or presumed discrepancy between model projections and GMST trends. Research on this question has identified several effects and variables that can 

recoud!e apparent differences between modeled and observed temperatures during the recent fluctua~ 

tion, such as model-versus·observed differences in the phasing of internal variability1US,!9, systematic 

errors i!l some of the external forcings used in CMIPS simulations20•

21 , and incomplete coverage and 

quality of observations7• 

FinaHy, our demonstration that the "hiatus" is statistically indistinguishable from previous fluctuations has no bearing on the question of the physical causes of fluctualions in surface temperature trends, 

Such fluctuations can be due to internal variability alone12•

22·n, or they may involve variations in external 

fordngs on the climate system such as solar cycles or volcanic eruptions, or both24- 26• \Ve have no commitment to a particular causal model of those fluctuations. 

Conclusions 

We have shown that there is a wide range of different operationalizations of the "hiatus" in the literature. 

For none of these operationalizations is the rate of temperature change meaningfully different from the 

set of rates of equivalent trend lengths over the modern period. That is, the "hlatns': however defined, is 

not unusual or unprecedented2i. Further, the duration of periods over which trends must be extended 

to generate signiCtca:nt warming trends has not changed noticeably in the "hiatus" periods relative to the 

rest of the modern "'<trming period. We conclude that there is no "hiatus': and neither has the climate 

system "paused." 

Our conclusion raises at least two questions. First, why has so much research been directed at the 

"hiatus" when it does not exist? We have addressed the likely reasons for this in detail elsewhere·\ ]he 

notion of a "pause" or "hiatus" demonstrably originated outside the scientific community\ and it likely 

found entry into the scientific discourse because of the constant challenge by contrarian voices that arc 

known to affect scientific communication and conduct4

•

2

s, 29• 

'lhe second question pertains to the broader implications of this apparent discord between data and 

the discussion in the literature. We suggest that discussing climate change using the terms "pause" or 

~hiatus" creates notable hazards for the scientific community. 

Adoption of the terms "hiatus'' or "pause'' is not inconsequential because the way in which environmental issues arc linguistically and semantically framed contributes crucially to publk (mis-)under· 

standing'0

• Scientists may argue that when they usc the terms "pause" or "hiatus" they know-and their 

colleagues understand-that they Jo not mean to imply that global warming has stopped. Indeed, the 

use of scare quotes in some articles (Table l) is clearly intended to imply this. "lhe problem, however, 

is that these terms have vernacular meanings, and when scientists use a term from the public vernacular to describe a feature of science, confusion results when the vernacular term is not an appropriate 

description of that feature. 'lhis misunderstanding may be particularly acute in this instance because 

the terms "pause" and "hiatus" originated as contrarian talking pointsM. Hence, we argue that scientists 

should use the term that most appropriately describes what they are studying. In the present case, that 

implies the use of "fluctuation': not "hiatus;' because when scientists use the term "hiatus': this sends 

a confusing and potentially misleading message to the public. Scientists might tacitly understand that 

global warming continues notwithstanding the alleged "hiatus~ or they may intend the "pause" to refer 

to ditferences lwtwcen observed temperatures and expectations from theory or models, but the public is 

not privy to that tacit understanding. 

Moreovc1; acceptance and use of scientific propositions carries ethical implications and responsibilityn·32. Some philosophers argue that holding a belief without sufficient Warrant"-i.e., without support 

by strong evidence-engenders a moral hazard ~. An important clement of this argument is that any 

belief, no matter how innocuous or inconsequential, creates the enabling conditions for similar and 

related beliefs. Any belief or opinion thus contributes to shaping an epistemological landscape, which 

SCIENTIFIC REPORTS j 5:16784 j DO!: 10.1038/srep16784 
145 

in turn implies a responsibility-or when the belief is unwarranted, a moral hazard-for "downstream'' 

intellectual consequences. Specifically, if unwarranted acceptance of a "hiatus" in global warming contributed to the delay of political action to mitigate climate change, with potentially adverse consequences 

on innocent parties, then the scientific S!atus of the "hiatus" could become a malter not just of science 

and philosophy, but also ethics and even law. test one consider such a potential hazard remote, the legal 

aftermath of tht' earthquake in I .:Aquila, Italy, which embroiled scientists in charges of manslaughter for 

their alleged failure to warn the community}i·35, vividly illustrates the legal and moral hazards that are 

incurred when !he public is not adequately informed of the ful! envelope of identifiable risks arising from 

scientific findings. In this context, it is notable that in a blind expert test, the notion that global warming 

has "stopped" was found to be misleading in light of the data5

. 

'lhose hazards can be largely avoided in this case by clear communication, which includes (although 

to be sure is not limited to) avoiding the unsubstantiated use of "pause" or "hiatus" when referring to 

fluctuations of GMST about the longer-term warming rate. 

Methods 

Corpus of articles. Table 1 summarizes the corpus of 44 articles that explicitly addressed the "hiatus': 

either by seeking an explanation or by reconciling it with model output Only articles addressing global 

(as opposed to regional) temperatures were included. Articles were sourced by the authors with the help 

of a number of other researchers and climate experts who arc conversant with the current literature. 

For each article, the table records the number of times that keywords such as "slowing': "pause': or 

"hiatus" occurred in the text. Occurrences in the reference section, in running heads, or in metadata 

were not counted. All forms of the stem were accepted; e,g., "slow': "slowed", "slowing': and so on. Note 

that Crowley et af.36 used another term, namely "plateau': 13 times. In addition, the word "stop" appeared 

4 times in two articles37·

38. Wherever a number is put into quotation marks (e.g., "I") this refers to 

the number of times the term was put into "scare quotes;' implying that the term was not necessarily 

accepted by the author. When scare quotes were used together with unquoted occurrences, those other 

occurrences are provided after the"+" symbol. 

Where applicable, the table also presents a quotation (usually from the abstract or first paragraph) that 

was judged to be indicative of the "framing" of the article. Citations or acronyms (or clauses not relevant 

to the meaning) in the quotation are omitted and replaced by .... When the quotation is absent for an 

article, a dear identification of framing was not possible. The Focus column indicates whether the "hiatus" was discussed primarily with respect to the observations (O) or with respect to the match between 

models and observations (M), or both (OM). 'I he Data column indicates which data set was used by the 

study, where H = HadCRUT439; G = GISS10; N = NCDCa; CW = Co'Wtan & Wa/2

; C3 and CS refer to 

CMJP343 and CMIP544 model ensembles, respectively; and "o" refers to other data sets. 

'lhe table also records the presumed onset of the "hiatus" as stipulated in each article (column labeled 

From) and the end of the "hiatus" (To). Concerning onset, articles sometimes use fuuy terminology such 

as "first decade of 21st century" (interpreted to mean 2000··2009) or "2000s" (also taken to mean 2000-

2009), or they contain several explicit and mutually incompatible onset times (in which case the first or 

more explicit one was taken as the article's declaration of onset). Similarly, the presumed end of the 

"hiatus" sometimes remained unclear as it was often (hut not always) the "present" or time of writing of 

the article. It was not always possible to unambiguously identify the last observation in the data set. 

Because of those potential ambiguities, a second independent reader who was blind to the purpose of 

the study audited, and confirmed, the values derived by the 6rst author. Unambiguous identification of 

onset and duration proved impossible for 4 articles, and the main analyses are therefore based on N =- 40. 

The corpus reported in Table I does not claim to he exhaustive; note, however, that the inclusion of 

further articles cannot reduce the range of onset times-it could only extend it. 

The Trrnd column indicates if the trend in the observations (NASA's GISS data set;4°) was significant 

for the time period specified(* denotes p<.OS) and whether it exceeded the IPCCs long-term reference 

trend (1951-2012), denoted by >I. Entries in this column that are labeled NA are not included in the 

quantitative analysis because computation of the trend was prevented by ambiguity in the operationali~ 

zation of the "hiatus:' 

The table omits articles Lhal did not address global mean surface temperature (GMST) hut exclusively 

focused on other indicators such as ocean heat content or temperature9•

15,46; sea level rise47; or wind~ • 

SCIENTIFIC REPORTS j5:16784l DOI.l0.1038/srepl67B4 
146 

SCIENTIFIC REPORTS!5:16784! DOl: 10.1038/srep16784 
147 

Acknowledgements 

We thank Didier Monsclesan for helpful comments and suggestions on the analysis. S.L receives funding 

from the Royal Society through a Wolfson Research Merit Fellowship. J.S.R. receives funding from the 

Grains Research and Development Corporation. 

Author Contributions J.S.R. provided the initial idea for the data analysis and S.L. provided the initial idea for the quantitative 

literature review. S.L. and j.S.R. conceived and designed the study and co-wrote a first draft the paper, 

S.L. conducted the principal analysis. All authors contributed to the conceptual framing, editing, and 

revision of the manuscript. 

Additional Information Supplementary information accompanies this paper at http:/lwmv.nature.com/srep 

Competing financial interests: ]he authors declare no competing financial interests. 

How to cite this article: Uwandowsky, S. et al. On the definition and identifiability of the alleged 

"hiatus" in global warming. Sci. Rep. 5, 16784; doi: l0.1038/srepl6784 (2015). 

SCIENTIFIC REPORTS j5:J6784j DOl: 10.1038/srepl6784 
148 

Commons license, in the credit line~ if the material is not included under the 

Creative Commons license, users will need to obtain permission from the license holder to reproduce 

lhe material. To view a copy of this license, visit http:/lcreativecommons.org/licenses/by/4.0/ 

SCIENTifiC REPORTS 15:167841 DOL 10.1038/srep16784 
149 

SCIENCE ADVANCES I RESEARCH ARTICLE 

CLIMATE CHANGE 

Assessing recent warming using instrumentally 

homogeneous sea surface temperature records 

2:017 ©The Authors, 

some rights reserved; 

exclusivelicem.ee 

American Association 

fortheAdv~ncement 

ofSdence.Distributed 

under a Creative 

Commons Attribution Zeke Hausfather,1

'

2

* Kevin Cowtan/ David C. Clarke,4 Peter Jacobs/' 

Mark Richardson,6 Robert Rohde2 

NonCommercial 

License4.0(CCBY-NC). 

Sea surface temperature {SST) records are subject to potential biases due to changing instrumentation and 

measurement practices. Significant differences exist between commonly used composite SST reconstructions 

from the National Oceanic and Atmospheric Administration's Extended Reconstruction Sea Surface Tempera~ 

ture (ERSST}, the Hadley Centre SST data set (HadSST3}. and the Japanese Meteorological Agency's Centennial 

Observation~Based Estimates of SSTs (COBE~SST) from 2003 to the present. The update from ERssr version 3b 

to version 4 resulted in an increase in the operational SST trend estimate during the last 19 years from 0.07" to 

0.12"( per decade, indicating a higher rate of warming in recent years. We show that ERSST version 4 trends 

generally agree with largely independent, near~global, and instrumentally homogeneous SST measurements 

from floating buoys, Argo floats, and radiometer~based satellite measurements that have been developed 

and deployed during the past two decades. We find a large cooling bias in ERSST version 3b and smaller 

but significant cooling biases in HadSST3 and COBE~SST from 2003 to the present, with respect to most series 

examined. These results suggest that reported rates of SST warming in recent years have been underestimated 

in these three data sets. 

INTRODUCTION 

Accurate sea surface temperature (SST) data are necessary for a wide 

range of applications, from providing boundary conditions for numerical 

weather prediction, to as.sessing the performance of climate modeling, to 

understanding drivers of marine ecosystem changes. However, 1n re~ 

cent years, SST records have been hampered by large inhomogeneities 

due to a marked increase in the use of buoy~based measurements and 

changing characteristics of ships taking measurements (1, 2). Up until 

the last two decades, most SST measurements were taken by ships, 

first with buckets thrown over the side and increasingly through engine room intakes (ERis) after 1940. Since 1990, the number of buoybased SST measurements has increased around 25-fold, whereas the 

number of observations from ships has fallen by around 25% (3, 4). In 

the last 25 years, SST assay methods have changed from 80% shipbased in 1990 to 80% buoy-based in 2015. Modem ship~based mea~ 

surements (primarily ERI, although hull contact sensors and other 

devices are also used) tend to generate temperature readings around 

O.l2°C higher than those of buoys, wha._ore sensors are directly in contact 

with the ocean's surface {I, 5, 6). As the number of ships actively taking 

measurements available in the International Comprehensive OceanAtmosphere Data Set (ICOADS) database (4) has fallen, a growing 

portion of ships are also using non-ERI systems that may introduce 

further changes in the combined record (1). Although buoy records 

are widely considered to be more accurate than ship-based measurements, their integration with ship records into longer SS'l' series poses a 

number of challenges (3). 

The National Oceanic and Atmospheric Administration's (NOAA) 

Extended Reconstruction Sea Surface Temperature (ERSST) (5), the 

Hadley Centre SST data set (HadSST3) (1), and the Japanese Metooro-

~Esn:r~:e:;!:~:~~r~~::,~n~~~:u~~~~;n~%:;~iih:~~~~~t~~~~ York. York, UK 4

1ndependent Researcher, Montreal, Quebec, Canada. ~Department of 

Environmental Sdence and Policy, George Ma5011 University, Fairfax, VA 12030, USA. 

~ASA Jet Proptdslon LaboratOI)', california Institute of Technology, Pasadena, CA 

91109,USA 

~corresponding author. Email: hausfath@lberkeley.edu 

Hausfather "t a/. Sci. Adv. l017;3:e1601107 4 January 2017 

logical Agency's Centennial Observation-Based Estimates of SSTs 

(COBE~SST) (7) are composite SST series that assimilate data from 

multiple different instrument platforms (ships and buoys from 

ICOADS and some satellite data in the case of COBE-SST) and mea~ 

surement methods (wood buckets, canvas buckets, engine intake 

valves, etc.) to create consistent long-term records. These three 

composite ocean SST series are used by the primary groups reporting 

global temperature records: NASA's GISTEMP (Goddard Institute for 

Space Studies Surface Temperature Analysis) (B), the Met Office Hadley 

Centre's and the University of East Anglia's Climatic Research Unit's 

HadCRUT (9), NOAA's Globo!Temp (IO, I I), the japan Meteorological 

Agency (12), Berkeley Earth (13), and Cowtan and Way (14). Because 

the oceans cover 71% of Earth's surface, changes to SST series have large 

impacts on the resulting global temperature records. 

ERSST was recently updated from version 3b (ERSSTv3b) to ver~ 

sian 4 (ERSSTv4), adding corrections to account for the increasing use 

of buoy measurements and incorporating adjustments to ship-based 

measurements based on nighttime marine air temperature (NMAT) 

data from the Met Office Hadley Centre and the National Oceanography Centre's HadNMAT2 (5, 15-17). ERSSTv3b did not include 

any SST bias adjustment<> after 1941, whereas ERSSTv4 continues 

these adjustments through the present. Although the largest changes 

to the ERSST record occurred during World War II, ERSSTv4 also 

indicated a higher rate of warming after 2003. This led Karl et al. 

(18) to conclude that the central estimate of the rate of global mean 

surface temperature change during the 1998-2012 period was compa~ 

rable to that dudng the 1951-2012 period, in contrast to the Intergovernmental Panel on Climate Change characterization of the 

recent period as a "hiatus" (19). These updates also created a notable 

divergence behveen ERSSTv4, HadSST3, and CO BE-SST from 2003 to 

the present and raise the question of which composite SST series provides the most accurate record in recent years. 

Over the past two decades, reasonably spatially complete, instrumen~ 

tally homogeneous SST (IHSST} measurements are available from 

drifting buoys, Argo floats (20), and satellites (see Materials and Methods 

for details on each HISST series}. To assess how well the composite SST 

1 of13 

0 

0 

~ 0 

" o_ 

iii 

a 

3 
150 

SCIENCE ADVANCES I RESEARCH ARTICLE 

records correct for biases due to the changing instrumentalion, we compare each of them in turn to JHSST series that were created using only 

drifting buoys, only Argo floats, and only satellite infrared radiometer 

data. Because these IHSST series are created from relatively homogeneous measurements from a single type of instrument, they should 

be less subject to bias due to changing measurement methods, although 

other factors, such as differences in spatial coverage or instrumental 

drift (in the case of satellites), need to be carefully accounted for. 

Each of the three IHSST series (buoys, Argo floats, and satellites) 

spans a different period of time. Buoy data have reasonably complete 

spatial coverage of the oceans from the late 1990s to the present. Argo 

floats achieve sufficient coverage for analysis from January 2005, 

whereas reliable satellite data span from 1996 to the present Two 

sources of infrared radiometer-based satellite sea skin temperature 

are considered: the ARC [ATSR (Along Track Scanning R..1.diometer) 

Reprocessing for Climate] SST product (21) from ATSR data, whid1 

provided data only through the end of 20ll, and the European Space 

Agency Climate Change Initiative experimental record (hereafter CCJ) 

(22), which combines ATSR and Advanced Very High Resolution 

Radiometer (AVHRR) data to obtain a continuous record for the 

whole period. The experimental version of the CCI record is not strictly instrumentally homogeneous and is not fully independent from in 

situ buoy SST observations but closely matches the independent ARC 

SST record during the period of overlap; the next official release of the 

CCI containing AVHRR and ATSR data should be fully independent 

of in situ observations. Three different Argo-based ncar-surface temperature data sets-from the Asia-Pacific Data Research Center 

(APDRC) (23), the Japan Agency for Marine-Earth Science and Technology (hereafter H2008) (24, 25), and Roemmich and Gilson 

(hereafter RG2009) (26)-are examined, with a number of different 

data scts chosen to reflect the uncertainty introduced by attempting 

to reconstmct near-SSTs using Argo data, 

RESULTS 

From January 1997 through December 2015, ERSSTv3b has the lowest 

centrnl trend estimate of the operational versions of the four composite 

SST series assessed, at 0.07°C per decade. HadSST3 is modestly higher 

at 0.09°C per decade, COBE-SST is at O.OSOC per decade, whereas 

ERSSTv4 shows a trend of O.l2°C per decade over the region of common coverage for all four series. We find that ER.()STv3b shows significantly less warming than the buoy-only record and satellile-ba<ied IHSSTs 

over the periods of overlap [P < O.Ql, using an ARMA(l, 1) (autoregressive moving average) model to correct fur autocorrelation], as shown in 

Fig. L ER5STv3b is comparable to ERSSTv4 and the buoy and satellite 

record<; before 2003, but notable divergences are apparent thereafter. 

Both the buoy-only and CCI series are very similar to ERSSTv4 

during their respective periods of overlap; trends in differences are insignificant in all cases. This strongly suggests that the improvements 

implemented in ERSSTv4 removed a cooling bias in ERSSTv3b. The 

ERSSTv4 record is expected to show good agreement with the collocated buoy record, because of new ship-buoy bias corrections and the 

increased weight attached to buoy observations in ERSSTv4. Thus, this 

agreement represents a replication of the ERSSTv4 result from the 

same data uslng a substantially different methodology. The CCI data 

are not used in the ERSSTv4 record and therefore represent an 

independent validation of the ERSSTv4 record. 

In addition to ER.'iST, 've also examine how the other two commonly 

used composite sea surface records, HadSST3 and COBE-SST, comHausfather etQ/.Sci.Adv. 2017;3:e1601207 4 January 2017 

pare with the buoy-only and satellite-based IHSST records. Both show 

significant cool biases in the period from 2003 to the present relative to 

the buoy-only record, although the magnitude of this cool bias is 

smaller than that found in ER..')STv3b. Difference series betw-een all four 

composite records and the buoy-only and satellite-based lHSST records 

are shown in Fig. 2. Each difference series is constructed by restricting 

all four composite SST series to common grid cells for each month and 

by comparing all grid cells where the composite records and the IHSST 

in question have data available. Our conclusions are similar when we 

consider aU~product common coverage or interpolating products to 

global coverage; details of the spatial coverage approach and lU1certalnty 

calculations can be found in Materials and Methods. 

Two of the three Argo near-SST records assessed, APDRC and 

H2008, agree well with the buoy-only and satellite-based records 

and suggest a cool bias in ERSSTv3b during the 2005-2015 period, 

when sufficient Argo data are available (Fig. 3). The RG2009 series 

is more ambiguous, with trends that are not significantly different 

(P > 0.05) from either ERSSTv3b or ERSSTv4, Similarly, both APDRC 

and H2008 suggest cool biases in HadSST3 and COBE-SST, whereas 

RG2009 docs not show a significant trend in the difference series 'Vt"':ith 

any of the composite temperature records (see Fig. 4). Differences bet\veen the Argo series emerge through different interpolation techniques and additional data incorporation; APDRC uses Aviso satellite 

altimetry for sea surface height estimates, H2008 uses a small amount 

of data from the Triangle Trans-Ocean Buoy Network and conductivitrtemperature-depth profilers (mostly befure 2005) (25), whereas RG2009 

relies solely on Argo data. 

To assess the significance of differences between composite series 

and JHSSTs, we examined whether trends in differences between the 

data sets were statistically different from 0 (that is, P < 0.05), as shown 

in Fig. 4. We looked at two periods: 1997-2015 (where buoys, CCI, 

and the four composite series have records) and 2005-2015 (buoys, 

CCI, three Argo series, and four composite series). When comparing 

ERSSTv4 to all six IHSSTs during both periods, Lhere are no significant 

Yeac 

Fig. 1, Comparison of the different ERSSTv3b, ERSSTv4, buoy-only, and CQ SST 

monthly anomalies from January 1997 to December Xl15, restricting aQ series 

to common coverage. ERSSTv4 is shown as a broad band for visualization purposes; 

this band does not represent an uncertainty range. The series are aligned on the 

1997-2001 period for comparison purposes. Spatial trend maps are also available 

in fig. 51, and a similar comparison with Argo data is shown in Ag. Sl. 

2 of 13 
151 

SCIENCE ADVANCES I RESEARCH ARTICLE 

trends in differences between the data sets except in the case of H2008, 

which showed slightly greater wanning over the 2005-2015 period. 

ERSSTv3b, HadSST3, and COBE-SST show a significantly lower 

warming trend over the period since 1997, compared to the buoy-only 

and CCI records (ARC SST shows nearly identical trends to CCI during its period of coverage from 1997 to 2012, as shown in fig. 53). 

During 2005-2015, ERSSTv3b, HadSST3, and CO BE-SST have significantly lower warming trends than the H2008 Argo record, and 

ERSSTv3b and HadSST3 have significantly lower trends than the 

APDRC Argo record. For the RG2009 Argo record, no significant 

trend difference can be found for any of the composite temperature 

series during 2005-2015. 

Both ERSSTv4 (15) and HadSST3 (1) incorporate detailed assessments of fully correlated (parametric) and partially correlated (sampling 

0.06 

~ -0.03 

c 

~ -0.06 

~ '6 

~ 

B 

., 

~ 

c. E -0.03 ., 

~ -0.06 

-0.06 

ERSSTV4 

and me'asurement) uncertainties into their respective composite SST series, ERSSTv4 assesses these combined "bias" uncertainties vlll an ensemble of SST reconstructions, incorporating a range of parametric 

setting combinations, most recently in an expanded 1000-member ensemble (16). HadSST3 provides a lOO~member ensemble to assess parametric uncertalnty but separately treats sampling and measurement 

uncertainty. We derived a 1000-member ensemble from the HadSST3 

ensemble, vvith each member expanded to l 0 members by adding an 

ARI time series with SD and autocorrelation scaled to match the 

missing partially correlated uncertainty. We repeat the buoy-only and 

CCI IHSST comparisons on each of the realizations masked to common coYerage (Fig. 5). 

The ERSSTv4 ensemble is not symmetric around the operational 

"best" estimate, which is based on the most empirically justified 

- ERSSTv4 - buoy 

- ER55Tv4 - ARC 

-- ER55Tv4 - CCI 

ERSSTv3b - buoy 

EI\SSTv3b- ARC 

ERSSTv3b- CCI 

1998 2000 2002 2004 2006 2008 2010 2012 2014 

Year Fig. 2, Twelve-month centered moving average of temperature difference series between composite and buoy-only, CCI, and ARC SST anomalies. Values be!ow 0 

lncllcate that the composite series has a cool bias relative to the IHSST record. 

Hausfather eta/. Sd. Adv. 2017;3;el601207 4 January 2017 3 of 13 
152 

SCIENCE ADVANCES J RESEARCH ARTICLE 

combination of parameter settings (5); most of the realizations have 

lower trends, with the lower bound of the ensemble encompassing 

ERSSTv3b, Only 16 of the 1000 ERSSTv4 realizations have a trend 

greater than that of the buoy-only IHSST record. The HadSST3 en~ 

semble, in contrast, is largely symmetric around the operational 

estimate, which is based on the median of the ensemble. All of the 

100-member and 1000-memher HadSST3 ensemble realizations 

have lower trends than the buoy~only record. The increased spread 

of the difference between the HadSST3 ensemble members and CCI, 

compared to the corresponding differences with the buoy record, 

may arise from the interaction of the greater regional variability in 

the difference between HadSST3 and CCI, coupled with the timevarying coverage of HadSST3. 

The structural uncertainty in the buoy record, estimated by com· 

paring two subsets of the buoy data, is about 0.05"'C in 1997, dropping 

0,06 ERSSTV4 

Q03 

noo 

-0.03 

-0.06 

0.06 

0.03 

G 0.00 -2..-

-0.03 

-0.06 

to 0.027"C for the 2005-2015 period (fig. 54) as the number of observations increases. The structural uncertainties estimated, using Eq. 8 

(sec Materials and Methods), from an intercomparison of the IHSST 

records are 0.024°, 0.020°, and O.OlrC for the buoy, Argo-H2008, and 

CCI records, respectively. 1be structural uncertainties in the trends 

over the 2005-2015 period using Eq. 10 are 0.012"', 0.014", and 

0.009°C per decade for the buoy, Argo-H2008, and CCI records, respectively. If the Argo-RG2009 data are used in place of the ArgoH2008 data, the trend uncertainties are 0.0140, 0.020°, and 0.012"C, 

respectively, representing a small increase in the uncertainties for 

the buoy and CCI records and a larger increase in the uncertainty 

for the Argo data. 

The trend uncertainties estimated from Eq. 8 are very similar to the 

uncertainty ofO.Ol3°C per decade estimated from the ERSSTv41000-

member ensemble. This represents a useful validation of the F.RSST 

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 

Year fig. 3. Twelve-month centered moving aver,ge of temper~~oture difference series between composite and Argo near--SST anornalie$. 

Hausfather eta/. Sci. Adv. 2017;3:el601207 4 January 2017 4 of 13 
153 

SCIENCE ADVANCES I RESEARCH ARTICLE 

ensemble, because the methods are independent: The ERSST ensemble relies on a bottom-up estimation of uncertainty from the different 

uncertainties in the methodology, whereas Eq. 8 yields a top-down 

eslimate based on the differences between independent data sources. 

The trend uncertainties estimated from Eq. 8 are 10 to 20% of the 

linear trend uncertainties in the corresponding temperature trends, 

which include the effect of internal variability. The uncertainties are 

based on the region of common coverage, and inclusion of poorly 

sampled regions will increase the structural uncertainty. The limited 

time span means that uncertainties are somewhat determined by a few 

outliers in each temperature series; however, the results show that 

linear trend uncertainty should not be used as an estimate of the structural uncertainty in the trend. 

The resulting d..iil:erence series and trends in the all of the figures 

\viii differ modestly on the basis of how spatial coverage is handled. 

For each IHSST difference series, we restrict coverage for each month 

to the coverage shared in common between the IHSST series in question and the four composite records. This not only serves to maximize 

the spatial overlap between tl1e data sets and provide a more accurate 

global estimate of differences for each individual IHSST but also 

results in difference series and trends that are not strictly comparable 

between IHSSTs due to coverage differences. This is particularly 

pronounced in the 1997-2005 period, when the buoy-only record 

has less coverage than the more spatially complete ARC and CCI 

satellite radiometer-based records. Some coverage diflerences also arise 

in the 2005-2015 period between Argo-based records and buoy/CCI 

records, because Argo data are largely unavailable north of 60°N, 

south of 60°S, or in the Malay Archipelago. 

To ensure that our results are robust regardless how spatial coverage is handled, we performed two additional tests to account for both 

spatial and temporal-spatial consistency across the series. In the first 

test, we restricted all series examined for the hvo time periods in question (1997-2015 and 2005-2015) to only 1 x llatitude/Jongitude grid 

cells containing records from all series examined over those time 

frames. During the 1997-2015 period, we only looked at grid cells 

with common coverage across the four composite series, buoys, and 

CCI, whereas during the 2005-2015 period, we examined only grid 

cells with common coverage between the composite series, buoys, 

CCI, and all three Argo-based series. This results in a record that is 

less spatially complete for any g.iven IHSST-composite series compar~ 

ison but is strictly comparable between IHSSTs. Difference series and 

trends for this common coverage approach are shown in figs. SS to $7. 

Results are largely comparable to those in the main paper, with a 

slightly higher trend in CCI difference series during the 1997-2015 

period and a lower CCI trend during the 2005-2015 period as the only 

notable differences. 

In the second coverage test, we applied a kriging spatial inleipOlation 

approach to the two series (buoys and HadSST3) that contain large gaps 

in spatial coverage for all months to create fully spatially and temporally 

complete records (the three Argo series and the other three composite 

series have their own interpolation provided, whereas satellite records 

are largely spatially complete apart from high latitudes). We then 

restricted all series to common coverage over the 1997-2015 and 

2005-2015 periods, following the approach of the common coverage 

test. This introduces some additional uncertainty due to the kriging 

but ensures that the spatial coverage represented by the difference series 

and trends does not change from month to month and that all series 

have nearly complete coverage over the period of overlap. The results 

for the kriged series are shown in figs. SS to SIO. Here, the cooling bias 

in ERSSTv3b, COBE-SST, and HadSST3 is more pronounced with 

e ERSSTY4 e ERSSTv3b e HadSSr.l e COBE·SST 

I~ f ,~~·r~ l·•c,.j··•m•••••••···········••• : ···•·• mm 

' ..,#' o:-' 

:g 

ci 

Fig. 4. Trends and 95% confidence intervals ec pet decade} In difference series for eoch IHSST and composite SST series, masked to common composite SST 

co"erage. Each difference series represents a com]Xlsite series minus an IHSST series. Confidence intervals for trends are cakulated using an ARMA(1, 1} autocorrelation 

model. Values below 0 indicate th<lt the com]XlSite series has a lower trend than the IHSST series over the period examined, The two trend periods examined are January 1997 

to December 2015 and January 2005 to December 2015. 

Hausfather et af. Sci. Adv. 2017;l:el601207 4 January 2017 5 of 13 
154 

SCIENCE ADVANCES I RESEARCH ARTICLE 

respect to the buoy-only and CCI records, although the overall 

results are comparable. Interpretation of the Argo records is largely 

unchanged for any of the spatial coverage approaches examined. 

In addition, the collocated buoy and CCI records show a spatial 

disagreement (not apparent in Figs. 2 and 4) that is only apparent 

when the CCI coverage is reduced to match the buoy coverage (see 

Buoy 

flgs. Sll and Sl2). This arises from regional differences between the 

CCI record and other records, particulady before 2001. CCI shows 

greater warming than ERSSTv4 in the Southern Ocean but less 

warming in the northern mid-latitudes. The Southern Ocean is 

consistently cloud-covered; thus, CCI might be expected to be less accurate in these regions. Winds can also affect skin temperature retrievals 

CCI 

Y~c 

Fig. 5. Twelve-month centerli!d moving average of temperature difference series between collocated ERSSTv4/Hi!ldSST3 ensemble rlii!aliutions and IHSST 

anomalilii!S. The left co!umn shows the difference ~rles with the buoy,onJy record. The right column shows the difference series with the CC! re<:ord. The top row shows 

1000 ERSSTv4 ensemble members, with operational versions of ERSSTv3b and v4 highlighted in black (note that the ERSSTensE'mble runs only go through 2014). The middle 

row shows the 100 ptJblished HadSST3 ensem~e members, with the operational version in black The bottom row displays the 1000 expanded ensemble members, as 

discussed in the text. 

Hausfather eta!. Sci. Adv. 2017;3;el601207 4 January 2017 6 of13 
155 

SCIENCE ADVANCES I RESEARCH ARTICLE 

relative to those at depth. In situ observations are prevalent in the 

Northern Hemisphere and so may be more reliable. In the Southern 

Ocean, in situ observations are sparse and so temperature trends rem.-tin 

tmcertain. The regional deviations from the in situ records and their 

impact on trends mean that comparisons with CCI should be treated 

\vith GJ.ution. 

Coverage biases are also affected by the choice of baseline for 

geographical map series. The results presented use a 19*year (1997-

2015) baseline for both the ERSSTv3b data to which the other series 

are then matched and the high-resolution climatology used in constructing the buoy record. Changing either of these to a 30-ycar (1986-

2015) baseline has no perceptible effect on the results. 

DISCUSSION 

Trends in lHSSTs constructed from buoy and satellite data agree ¥lith 

ERSSTv4 over the 1997-2015 period but are significantly higher (P < 

0.01) than the ERSSTv3b trend, supporting the conclusions of Karl 

et al. (18). Both huoys and satellites also suggest a significant (P < 

0.05) cooling bias in HadSST3 and COBE-SSI'. Over 2005-2015, four 

of five lHSST series agree with ERS..-;Tv4 or suggest that il might be 

slightly cool-biased. By contrast, four of five IHSST series suggest cool 

hiases in both ERSSTv3b and HadSST3, whereas three of five IHSST 

series suggest a cool bias in CO BE-SST. One of the three Argo series 

(RG2009) is statistically indistinguishable from all four of the composite SST products during the 2005-2015 period. 

The difference in IHSST records relative to HadSST3 is particularly 

noteworthy, because HadSST3 includes expHdt buoy-ship offset adjustments comparable to those used by ERSSTv4 and continues ship 

SST corrections through the present (1). The source of the apparent 

cooling bias in recent years in HadSST3 is unclear, although it is likely 

related to biases in ship records introduced by the changing composition 

of shipping fleets and a general decline in the number of available 

ship-based SST measurements (4). When comparing IHSSTs to a 

ship-only SST record (restricting to common coverage), we have identified a strong cool bias in the ship record, particularly since 2010. Not 

only are ship temperatures higher than buoy temperatures at the start 

of the study period (due to an approximately O.l"C offset), the ship 

record substantially underestimates the rate of warming over the later 

part of the period as well (fig. 513). This result is supported hy the 

satellite observations of skin temperature, the buoy measurements in 

the top meter of the ocean, and Argo observations from three different 

methodologies over depths spanning 2,5 to 20 m (fig. Sl4). ERSSTv4 

mostly avoids this potential bias in ship records by assigning an 

increased weight to buoys in recent years (5), although the slightly 

higher trends in buoys, CCI, and two of the three Argo series vis-avis ERSSTv4 during 2005-2015 (Fig. 4, bottom) might be driven by 

some residual ship· related bias. 

The difference in trend between ERSSTv3h and ERSSTv4 is smaller 

than the difference in trend between the buoy and ship records, because 

ERSSTv3b also incorporates data from buoys but does not account for 

the offset between the ship and buoy temperatures or assign the huoys 

more weight than ship-based measurements. HadSST3 falls between 

the two versions, incorporating an offset adjustment between ships 

and buoys and some corrections to the ship observations but equally 

weighting ships and buoys. NMATs (HadNMAT2), which are used 

as part of the ERSSTv4 homogenization, also appear to show a cool 

bias comparable to, if not larger than, that of HadSST3 relative to the 

lHSSTs in the period after 2003 (fig. SIS), possibly because of the 

Hausfather et af. ScL Adv. 2017;3 :el601207 4 January 2017 

residual inhomogeneities in NMAT records. Whereas COBE-SST 

is also significantly cooler in recent years than the buoy-only record 

and CCl, a new version (COBE-SST2) incorporates buoy adjustments and shows better agreement \-vith the lHSST records but does 

not extend up to the present and is not yet in operational use in the 

Japanese Meteorological Agency global land/ocean temperature 

product (fig. 516) (27). 

Interpreting the Argo results 

The Argo records cover a shorter period (11 years rather than 19), and 

their results are less clear-cut than the buoy and CCI IHSSTs. The 

H2008 and APDRC records support ERSSTv4 (and even suggest that 

it might be a bit too cool), although APDRC results are somewhat 

sensitive to the choice of start year (fig. S17). RG2009 falls betw-een 

RRSSTv3h and ERSSTv4 in trend and does not reject either, Similarly, 

H2008 and APDRC suggest a cool bias in HadSST3 and (to a lesser 

extent) in COBE-SST over the 2005-2015 period, whereas the results 

of RG2009 are ambiguous and do not allow any differentiation betw-een composite record trends. 

The brevity of the Argo records and their divergence from other 

records limit the weight that can be placed on them. If the faster 

warming 112008 and APDRC records are accurate, then all of the 

IHSSTs (buoys, satellites, and Argo floats) are in basic agreement in 

rejecting the slower wam1ing ERSSTv3b record. However, if the slower 

wanning RG2009 record is correct, then this would imply either that 

the buoy and CCI fHSSTs are too warm during 2005-2015 or that 

there may be a variation in temperature trend with depth: The skin 

record and the top meter show faster warming, whereas the deeper 

ship and Argo records show slower warming. Different observational 

platforms sample sea "surface" (or near~surface) temperature at different depths in the mixed layer, with satellites, buoys, ships, and Argo 

floats observing the temperature at increasing depths. If H2008 or 

APDRC records are more accurate, it seems unlikely that depth plays 

a role in the differences between temperature trends, because the 

slower warming ship record is bracketed in depth by the satellite/buoy 

records and the Argo records. This would also suggest that measurement depth does not explain any part of the slower warming found in 

the ship record. However, if the RG2009 record is correct, it may suggest that the slower warming ship record arises from a combination of 

both depth and the bias in the ship record (because the ship record 

exhibits less warming than even RG2009, as shown in fig, SI4). 

Argo instruments have temperature profiles at depths throughout 

the mixed layer (and below), with the shallowest observations in any 

of the Argo products in the range of 2.5 to 7.5 m. Although the Argo 

records show no discernable reduction in trends between depths of 5, 

IO, and 20 m (fig. Sl8), they cannot exclude a difference with the top 

meter measured by the buoys. If there is a significant difference in 

temperature trend between the top meter and the remainder of the 

mixed layer, this would present a problem in the construction of a 

homogeneous SST product from the combination of ship and buoy 

records. Similarly, most of the Coupled Model Intercomparison 

Project Phase 5 (CMIPS) climate models have a top layer spanning 

0 to 10 m and so may not resolve the top meter of the ocean. This 

could present a challenge both in testing for the depth effect in 

models and in comparing the models to observations. However, because two of the three Argo-based records analyzed show no sig~ 

nificant difference with buoy and CCI surface records and the Argo 

series is short, any conclusions about depth-related effects appear 

to be premature. 

7 of13 
156 

SCIENCE ADVANCES I RESEARCH ARTICLE 

Concluding remarks 

Adjustments to correct for inhomogeneities in SSTs in recent years 

have a large impact on the resulting decadal-scale global temperature 

trends. Assessing the efiix:tiveness of these adjustments is critical to 

improving our understanding of the structure of modern climate 

changes and the extent to which trends in recent periods may have 

been anomalous with respect to longer-term warming. Using 

independent IHSST series, we find that NOAA's new ERSSTv4 

effectively corrects a significant cooling bias present in ERSSTv3b during the past two decades without introducing any detectable residual 

trend bias. We also conclude that t\vo other widely used composite 

SST series, HadSST3 and COBE-SST, likely suffer from spurious 

cooling biases present in ship-based records in recent years. 

Some llilcertainty remains, particularly in Argo-based near-SST reconstructions. Two of the three Argo reconstructions examined agree 

well globally 'With the buoy and radiometer-based IHSSTs, whereas the 

third does not allow for any effectlve differentiation between composite 

SST series. Similarly, although CCI and ARC-SST radiometer-based estimates agree quite well globally \•:ith the buoy-only record, there are 

significant zonal differences. The time period considered is relatively 

short, with most of the divergence between composite SST records 

occurring after 2003, and sufficient Argo data are only available after 

2005. Nonetheless, SST time series from drifting buoys. satellite radiometers, and two of the three Argo series strongly suggest a cool bias 

present in ERSSTv3b, HadSST3, and COBE-SST. Overall, these results 

suggest that the new ERSSTv4 record represents the most accurate 

composite estimate of global SST trend~ during the past two decades 

and thus support the fmding (14) that previously reported rates of 

surface wanning in recent years have been underestimated, 

MATERIALS AND METHODS 

We compared composite SST records including ERSSTv3b, ERSSTv4, 

HadSST3, and COBE-SST to three separate IHSST records constructed from ICOADS-reporting buoys, near-surface measurements 

from Argo floats, and radiometer-based satellite SST records. We obtained existing spatially gridded fields for each SST series (and created 

novel ones in the case of buoy-only and ship-only records) and 

converted each to standardized 1"' latitude by 1"' longitude uniform 

grid (hereafter 1" x 1" grid). 

Temperature averaging in the presence of varying geographical coverage requires that all of the temperature series be aligned on a common baseline. It is common practice to apply an offset to each cell and 

month of the year to bring the mean of that cell and month to 0 over a 

30-year baseline period; however, this is impractical for the short buoy 

record. Fixing the baseline for an incomplete record is problematic in 

the case where the months for which obseiTations are present are unusually hot or cold; howt'Ver, the problem may be addressed by aligning the data to a more complete record containing the same weather 

signal. The spatially complete ERSSTv3b record was therefore aligned 

to 0 during the 1997-2015 period, and then the other data sets are 

aligned to the normalized ERSSTv3b map series. This method is a conservative choice in attempting to detect a bias in the ERSSTv3b record, 

as it may bias the compared series slightly toward it. 

Data series were carefully alif,med to ensure accurate intercomparisons of SST series. The process wa,<; as follows: Optimum Interpolation 

SST (OISST) was used to construct high-resolution daily climatology 

on the baseline period (1997-2015)-yielding 365 fields, one for each 

day (leap days are also treated). 1he buoy series was then calculated 

Hausfather eta/. Sci. Adv. 2017;3 :el601207 4 January 2017 

using this high-resolution daily climatology, yielding 228 monthly 

fields (19 years x 12 months). ERSSTv3b was also aligned to the 

1997-2015 baseline, All of the composite series and IHSSTs (including 

the buoy series) were then aligned to the baselined ERSSTv3b on the 

basis of whatever months are available for each grid cell These were 

then masked to common coverage and plotted 1n Fig. L This made use 

of the spatial completeness ofERSSTv3b to avoid artifacts due to base~ 

lining temporally incomplete cells on an incomplete baseline period; 

we used ERSSTv3b for this purpose to avoid biasing our results toward 

ERSSTv4. Pairwise difference map series were calculated between the 

aligned maps. The study was restricted to the 1997-2015 period, with 

the start date determined by buoy coverage and a data break in the 

ATSR~based SST data. Details of how each data set was obtained 

and processed are provided below. 

ERSST, HadSST3, and COBE-SST 

Both ERSSTv3b (10) and ERSSTv4 (11) were produced on a 2° x 2" 

grid, with sea ice cell<> recorded as -l.B"C. The ice cells were set to 

missing, and then the data were expanded to a 1 o x 1"' grid, repeating 

each value from the original grid to the four corresponding cells in the 

finer grid. HadSST3 (1) v.o.s produced on a 5° x so grid with no values 

for sea ice cells and was expanded to the l 0 x l" grid by repeating each 

value from the original grid to the 25 corresponding cells in the finer 

gtid. COBE-SST (7) and COBE~SST2 (27) were distributed as a 1 x 

l gridded product; cells with sea ice were recorded as -l.8°C, similar 

to ERSST, and were set to missing. Because both HadSST3 and 

ERSSTv4 included ensembles of realizations vvith different parameter~ 

izations, for the main analysis in the paper (for example, Figs. 1 to 4), 

the operational version of each series was used. This is the ensemble 

median in the case of HadSST3, whereas ERSSTv4 provides a 

preferred realization. 

Different approaches were used in the construction of the gridded 

SST products. In the HadSST3 record, observatioru; only contributed to 

the grid cell and month in which they occurred leading to some cells 

for which no temperature estimate was available. In the COBE-SST 

records, optimal interpolation was used in both space and time to create a spatially complete field from the available data. The ERSST and 

COBE-SST2 data sets combined a low-resolution reconstruction with 

the fitting of empirical orthogonal teleconnections to the observations 

to produce a spatially complete field, in which local temperatures 

could be inferred from distant observations {up to a specified distance) 

through teleconnections. All the records included data from ICOADS 

(albeit some from different releases of the database); however, in addition to differences in the processing methods, ERSSTv4 attached an 

increased weight to buoy observations on the basis of their lower estimated uncertainty. 

Because some of the composite SST series included interpolation of 

observations into prmdmate grid cells with missing data, all composite 

SST series were restricted to grid cells common to the HadSST3, 

ERSSTv4, and COBE-SST data sets for any given month. Because 

HadSST3 included no explicit interpolation (apart from that implicit 

in its use of relatively large 5° x 5" grid cells), this should remove any 

differences benveen series due to interpolation. Failing to account for 

interpolation could lead to difficulty in cross-comparison of difference 

seiies between lliSST and different composite SST records. 

Buoys 

The buoy data were obtained from the ICOADS Release 2.5 data (4), 

Drifting buoys were selected by the World Meteorological Organization 

B of13 
157 

SCIENCE ADVANCES I RESEARCH ARTICLE 

(WMO) buoy identifier and the presence of a value in the SST field 

(thus excluding Argo buoys with WMO identifiers). Moored buoys 

were excluded from the analysis because of an offset in temperatures 

between drifting and moored buoys (perhaps due to measurement 

depth; see fig. $19), which would introduce a bias as the proportion 

of moored and drifting buoys changes over the period of interest. A 

large majority of measurements in recent years come from drifting 

rather than moored buoys, and the use of drifting buoys only has no 

major impact on the results. The temperalure field was determined by 

averaging buoy observations over the span of a month for each cell in a 

global grid. The grid consisted of cells of equal area, with equatorial cells 

spanning a range of 5" in hath longitude and latitude. At higher latitudes, the longitudinal width of a cell in degrees was increased by 

calculating the area of the latitude band, dividing by the area of <I 5 X 

5 cell at the equator, and using that many cells in the latitude band to 

maintain a constant area. 

The data were processed l month at a time. For each buoy, data 

were divided into days. The (typically hourly) temperature, latitude, 

and longitude data for that day were averaged. Buoys that showed 

temperature variations with an SD exceeding 1 "C or positional varia~ 

tion with an SD exceeding 0.5" oflatitude or longitude during a single 

day were excluded for the whole month: This can occur jf a buoy is 

beached or picked up by a ship. The temperature was then converted 

to an anomaly using climatology calculated from OISST version 2 

(OISSTv2) (28) for that day of the year and for the corresponding latitude and longitude on a fmer 0.5" grid. This mitigated the biasing 

effects of temperature observations at the beginning or end of a month 

or the northern or southern edges of a 5° latitude hand. The daily 

mean temperature anomaly for the buoy was then added to a list 

for the corresponding grid celL After all buoy records were processed, 

all temperature anomalies for a given cell were averaged to produce a 

final anomaly value for that cell, 

This method for constructing the buoy-only temperature record 

was chosen for simplicity, with the aim of reducing the possibility of 

methodological artifacts, such as infilling distorting the result: A consequence of this is that the resulting temperature reconstruction is 

limited to regions where observations are available. However, simplicity does not in itself preclude bias: An overly simple method 

might, for example, fail to detect some faulty observations. This pos~ 

sibility will be addressed tluough internal consistency checks on the 

buoy data. 

Another possible source of bias is miscalibration of the tempernture sensor, leading to systemalicatly lower or higher readings. Nor~ 

mally, these would contribute noise rather than a bias in the trends as 

the miscalibrated buoy moves into more or }ess sampled regions and 

so receives a different weight in the temperature calculation. However, 

if new buoys are introduced, which are systematically different in cal~ 

ibration relative to older buoys, a bias in the trends could result. There 

was no sign of such a bias in the comparisons between different 

IHSSTs, and the cross-validated uncertainties were lowest for the recent period where the composite records show most difterence. 

Additional interbuoy comparisons were performed to address this 

possibility. For each grid cell and month -where at least three buoys 

contributed observations, a bias estimate was calculated from the 

difference berween the mean anomaly for the buoy and the mean 

of the anomalies for all the remaining buoys in that cell. All the bias 

estimates for a buoy were collected, and buoys for which the magnitude of the mean bias or SD of the bias estimates exceeded 1 oc were 

eliminated, reducing the total number of buoys by about 10%. Jn a 

Hausfather et a/. Xi. Adv. 2017;3: e1601207 4 January 2017 

further test, the temperature record was recalculated, applying the resulting bias adjustment to the readings from each buoy in tum. 

Four versions of the buoy record were prepared to evaluate the 

potential impact of buoy biases, a~ follows: 

(1) Using all of the data, omitting the test for daily variability. 

(2) Filtering on the basis of daily variability only (the default perbuoy filter, described at the start of this section). 

(3) Filtering on the basis of daily variability and interbuoy variability (that is, the additional filter described in the previous paragraph). 

( 4) Filtering on the basis of monthly and interbuoy variability and 

application of the bias correction [as in (3) bul then recalculating the 

huoy record after applying a correction to each buoy on the basL~ of its 

mean difterence with passing buoys]. 

The resulting temperature series are shown in fig. $20, along v.rith 

the differences of the other methods from the default method. The 

largest difference arose from using all of the data without filtering 

[or daily variability. lnterbuoy variability and bias correction made a 

smaller difference. The differences between the methods were small 

compared to the differences between the composite records. The 

default method using a per-buoy filter showed the lowest trend during 

1997-2015 and was therefore a conservative choice. 

The buoy coverage was limited, particularly in the 1990s, and comparisons to other data sets may have been affected by coverage bia~. To 

produce an unbiased comparison to other data sets, all the data sets 

\'ICTe expanded onto a 1° x 1" grid. Comparisons were made using only 

the cells for which the data sets being compared had values. The areaweighted mean temperature was then calculated for each record using 

the common coverage cells. The percentage of global ocean covered by 

buoy measurements varies from around 40% in the mid-1990s to 

around 70% in recent years. 

Ships 

The ship rerord was constructed in the same way as the buoy record, 

\vith one exception: Many ships only report once per day, and from 

2007, some ship identifiers were masked for security reasons (although 

this has been improved in Release 3 of ICOADS). The test to detect 

excessive motion or variation within a single day was therefore omitted. 

The only quality control applied to the ship record therefore arose 

from the calculation of the global mean of the SST field, which excluded observations that fell in land areas. The ship observations were 

subject to significant quality issues, and the limited quality control implemented in this record therefore provided no more than a general 

indication of the presence and scale of any bias in the ship record. 

Argo floats 

Three different gridded Argo data provided online by the International 

Pacific Research Center APDRC (23), the Japan Agency for MarineEarth Science and Technology (H2008) (24, 25), and Roemmich and 

Gilson (RG2009) (26) were used. These data were produced on a 

monthly 1° x 1° grid and were smoothed and infilled by the data provider using a variational analysis technique to provide global coverage 

over all cells unaffected by seasonal sea ice. Sea surface height was used 

as part of the interpolation process in APDRC, whereas cells 

containing sea ice were represented by missing data. The data did 

not require regridding and were aligned to the ERSSTv3b data, as described previously. 

The RG2009 Argo product had temperature values at 2.5, 10, and 

20 dbar and deeper levels; the H2008 product had temperatures at 10, 

20, and 30 dbar and deeper levels, and the APDRC product had 

9of13 
158 

SCIENCE ADVANCES I RESEARCH ARTICLE 

temperature values at 0, 5, and 10m and deeper levels. We used the 

S~m level for the APDRC product,. the 10-dbar (10 m) level for the 

H2008 product, and the 2.5-dbar level for the RG2009 product (which 

represented measurements ranging from 2.5 to 7.5 dbar with a mean 

level of 5 dhar/rn) to provide the most comparable and highest available depths; estimated 0-m temperatures from APDRC were not used 

because they resulted from interpolation (because no Argo floats 

sampled sea skin temperatures). 

Throughout the paper, we refer to the record derived from Argo 

floats as "near-SST," because the highest level of the ocean measured 

by most Argo floats is approximately 5 m below the surface (26), 

However, with the exception of satellite radiometer-based estimates, 

all of the instmments used in this analysis recorded ocean temperatures at depths between 0 and 20m For example, ships tend to measure temperatures through ERI valves at depths of 7 to 11 m for large 

ships and l to 3 m for small ships (3). Moored buo~ typically measure SSTs at a depth of 3 m, whereas drifting buoys measure SSTs at 

around 0.5 m. Recent work (29) found no long-term difference in 

warming rates betw-een depths of 0 to 4 m and depths of 4 to 9 m 

in a CMIPS model; similarly, we have established that our results 

are robust when using the next deeper level of each Argo data set 

(fig. Sl9). The different depths sampled by the different observational 

systems provide a basis to assess whether depth plays a role in the rate 

of recent warming. 

Argo data have been used to create SST analogs in the past; for 

example, Roemmich and Gilson (30) compared ARGO "near" -SST 

to NOAA's OISSTvl, whereas Roemmich et al.(31) compared a 5-m 

Argo-based SST record to OISSTv2. Here, we performed a similar 

analysis using the Argo-based fields provided by RG2009, APDRC, 

and 112008. 

Satellites 

The ATSR instruments provided infrared images of Earth, from which 

skin temperatures may be derived. ATSR data were incorporated into 

two gridded data sets, the ATSR ARC (21) spanning the 1996-2012 

period and the experimental National Center for Earth Observation/ 

European Space Agency SST CCI Analysis L3S version EXP-1.2 (ESA~ 

CCI or CCI) (22), which also incorporates data from the A VHRR and 

spans the period from 1996 to the present (end of 2015). Coverage 

between 60°$ and 60°N was largely complete (except for a fe<.v cells 

each month in the ATSR record, which were affected by cloud, typically in the Southern Ocean or North Atlantic). Both the ATSR-only 

(through mid-2012) and ATSR + AVHRR (through present time) 

CCI data were analyzed, and the CCI data were used in the paper 

because they extend to the present (and differences between the two 

were minor during the period of overlap, as shown in Fig. 6). 

Spatial coverage 

The main figures in the paper were generated by limiting difference 

series to common spatial coverage between the four composite SST 

series and the IHSST in question. For example, a difference series between ERSSTv4 and the buoy-only record would show the difference 

for all grid cells for each month, where all four composite SST series 

and the buoy-only record had data available. The requirement that all 

four composite series share the same coverage is intended to remove 

the effects of interpolation on the results, because all largely rely on the 

same ICOADS data. 

Two additional tests described in the Discussion were undertaken 

to ensure that the results were robust to choices of how coverage was 

handled. In the first test, the analysis was carried out for the two 

periods of interest (1997-2015 and 2005-2015), restricting the analysis 

to only grid cells, where all series available for those periods had 

coverage. During the 1997-2015 period, this means that only 1 x 

1 latitude/longitude grid cells (where the four composite series, the 

buoy-only record, and the CCI record all had coverage for any given month) were used. During 2005-2015, grid cells required coverage by the four composites, buoys, CCI, and all three Argo 

records to be used. 

In the second test, we created fully spatially and temporally 

complete fields to control for both difference in coverage for any given 

time period and changes in coverage over time. Infilling was performed on the gridded data using the original grid sampling for that 

record: For the buoy record, this was on the 550-km equal area grid, 

and for the HadSST3, tills \vas on the 5° x 5° grid, The resulting infilled 

field was then copied onto a 1° x 1° grid as before. Infilling was perfomled using the methOO of kriging (32), by which the values at unobserved locations were inferred from the observed values. Each 

observation was weighted on the basis of distance from the target loca~ 

tion using a variogram, relating the expected variance between PNo grid 

cells to the distance between them, which was determined from grid 

cells for which observations were available, fitted with an exponential 

model controlled by a single range parameter, which was the e~folding 

distance of the variance. The kriging calculation also used the covariance bet\veen locations where observations were present to estimate 

the amount of independent information in each observation. The buoy 

record showed longer range autocorrelation than the HadSST3 data, 

\Vith respective e-folding distances of 1400 and 900 km, suggesting that 

the buoy record showed more spatial autocorrelation. 

0.06 - CCI(ATSR + AVHRR)- ARC(ATSR) 

E 

- 0.03 

<I> 

~ 0.00 

~ 

~ -0.03 

i5 -0.06 

1998 2000 2002 2004 2006 2008 2010 2012 2014 

Year 

Fig. 6. Twelve-month tentered moving average of differences between CCI ATSR + AVHRR and ATSR-only ARC SST records dUJing the period of overlap. The earlier 

lHSST ARC shows small differences to the newer combined version; however, the differences are minor compared to the differences relative to the composite SST rewrds. 

Hausfather eta/. Sci. AcN. 2017;3:el601207 4 January 2017 10 of 13 
159 

SCIENCE ADVANCES I RESEARCH ARTICLE 

Infillcd temperature observations will therefore be a weighted com+ 

bination of the nearest obsenrations if there are observations within a 

small multiple of thee-folding distance. Locations that are very distant 

from any observation will tend toward an optimal estimate of the 

global mean of the temperature field. 

Uncertainty estimation 

SST reconstructions include uncertainties due to limitations of both 

the data and the methods. Differences between reconstructions may 

arise because of random errors in the data or errors introduced during 

processing or because of uncorrected biases in the obsenrational data. 

Identification of a bias requires that the difference betv.reen reconstructions 

must be shown to be larger than can be accounted for by random errors 

alone. To that end, we now examined different methods for the determination of the uncertainty in a reconstruction. Two approaches were 

used. First, collocated temperature difference series were used to 

estimate the significance of the differences. Second, a method was outlined for the use of independent temperature series to directly estimate 

the structural uncertainty in each series. 

Significance of the temperature difference series trends. 

To assess the significance of differences in trends between temperature 

series, we first calculated the difference temperature series from the 

difference map series to eliminate differences in coverage. The trend in 

the difference series was then rompared to the uncertainty in that trend 

estimated using an appropriate autoregression model and used lo determine whether the trend difference was significantly different from 0. 

The trend in the difference series is identical to the difference in the 

trends betv.reen the two series, assuming that both map series are reduced to common covemge. However, calculation of the trend in the 

difference series offers a benefit when determining the uncertainty in 

that trend (33). If the trend difference is calculated from the trends of 

the individual series, the uncertainty in the trend difference requires the 

determination of the covariance between the model residuals. The respective residuals contain common internal variability and so are 

strongly correlated; therefore, the covariance term is positive. Omission 

of the covariance term leads to the uncertainty in the trend difference 

being markedly overestimated. With the covariance term included, estimates of the uncertainty in the trend difference from either the 

difference series, or from the two individual series, give identical results. 

The difference series linear trends were estimated with ordinary 

lea1.1. squares (OLS), with SE correction to account for serial correlation 

of the residuals (34-36). The general approach is to estimate the effec~ 

tive sample length (and, thus, the eiTective degrees of freedom) from an 

estimate of the positive autocorrelation of the residuals 

Then, the autocorrelation function (ACF) of an ARMA(l, 1) series is 

given by 

Qo= 1 

Q1 ~ ($ + 6)(1 + ¢6)/(1 + 2$6 + 62

) (3) 

Qj Ql~~l, j22 

where !j) and 8 are the respective AR and MA coeffidents. 

Because the assessed trends cover only 11 to 19 years (132 to 

228 months), a bias correction was also applied to the global 

difference series trends to account for the underestimate of autocorrelation in these short series (35, 37). The original TjGstheim and 

Paulsen correction for the AR(l) estimated coefficient !j) is given by 

<1>oc ~ $ + (1 + 4~)/n, (4) 

The bias correction of ARMA(l, I) estimated ACF coefficients Ql, 

and$ generalizes (7) by also accounting for the positive difference between $and Q1• Note that the AR(l) bias correction in Eq. 4 then 

becomes a special case where 8 0 and Q1 = ¢ [AR(l) is employed 

in the few cases where this results in more conservative uncertainties}. 

$,, ~ l (1 +4(2$ Q,))/n, 

Q1oc = Q, + (1 + 4(2~ Q1))fn, (5) 

'The ARMA coefficient estimates <Jlbc and Qu,o can then be substituted 

into the appropriate specific form ofEq. 1. "fl1e ARMA(l, l) formulation in Eq. 3 can then be simplified as (36) 

(6) 

IHSST uncertainty esrimadon. 

The methods presented so far allowed us to estimate the significance 

of the differences between temperature series. However, the ability to 

estimate the uncertainty in each individual IHSST series would also be 

useful. Two methods will be used, the first based on the internal 

consistency of the buoy data and the second based on the intercomparison of the JHSST tempemture data sets. 

The uncertainty in the buoy data may be estimated by dividing the 

(l) buoys into two random subsets and calculating g.ridded temperature 

data from each subset of the data. Global temperature series were then 

calculated from the collocated values from each map series. A 120-month 

moving root mean square difference between the two temperature series 

provides an estimate of the uncertainty in the global temperature for the 

region of common coverage (after scaling by lt'h). This uncertainty 

estimate includes the effects of random measurement errors, as well 

as a sampling error that increases v.ith decreasing coverage; however, 

it does not include coverage uncertainty or systematic biases affecting 

all of the buoys. 

where nt is the original series length, n, is the effective sample length, and 

Q_j is the autocorrelation at lagj of an autoregressive (AR) or AR.MA noise 

model estimated from the OLS residuals. An ARMA(l, 1) model "WaS 

used fur all gridded and global difference series (for example, ERSSiv4-

buoys). 'The ARMA model coefficients were estimated v.rith maximum 

likelihood for global series and Yule-Walker (moments) for gridded series 

trends. An ARMA(l, l) series X1, with white noise series E1 satisfies 

X, 

In the second approad1, an estimate of the uncertainties in each of 

the IHSST series is obtained from the difference temperature series for 

(2) the overlap period 2005-201 L The uncertainty in the difference series 

Hausfather eta/. Sci. Adv. 2017;3;e1601207 4 January 2017 11 of 13 

0 

~ 

0 " 

"-"' 

~ 

a 

3 
160 

SCIENCE ADVANCES I RESEARCH ARTICLE 

between the buoy and Argo data arises from the sum of the variances 

of the two series, assuming that the series are independent 

fig. $4. Cros~·val!dated uncenaml>es for the buoy record, whethe! with no climatology or with 

dailydimatologiesderivedfrom theOISSTv2dailyrNnalysisdata. 

fig.SS.Twelve-monthcenteredmovingaverageoftemperaturedifferern:eseriesbetween 

cornposite and buoy-only, CCI, and ARC SST anomalies restricted to common coverage acros> 

al!seriesshown(fourcompa5ites,buoys,andARUCCl), 

(7) frg. 56. Twelw·month centered mcwmg awrage of temperature difference series between 

composite and Argo near· SST anomahes restricted to common coverage across all series with 

records from 2005 to201S(fourcomposites,threeArgos, buoy-only, and CCI). 

and assuming similar expressions for the remaining two series, 

where cr2 is the squared uncertainty in the given temperature series, 

The squared uncertainty in the difference temperature may be esti" 

mated from the variance of the difference series, adjusting the number of degrees of freedom to account for the removal of the annual 

cycle from the difference series. 

The uncertainty in a given series may then be estimatt>d using 

equations of the following form 

The resulting uncertainty estimates include the effects of random 

measurement errors and any biases in the independent data sources, 

which are not correlated across the data sources; however, as before, 

they do not include coverage bias. This is similar to the approach outlined in O'Carroll et al. (38). 

The uncertainty in the trend in an IHSST series may be estimated 

from the uncertainty in 1he monthly temperatures obtained from Eq. 

8 using the equation 

f1g. $7, Trendo and 95% confidence intervals ('C per decade) in differem;e series for NCh IHSST 

andcompositeSSTseries,maskedtocommoncoverngeforallseriesavai!able 

fig.S8.Twelve·monthcenteredmovingaver<rgeoftemperaturedifferenceseriesbetween 

composite and buoy--only, CO, and ARC SST anomalies, with the buoy and HadSST3 series 

kriged and all s€'nes reduced to common nwerage to ensure conslst~nt complete spatial and 

tempor~l coverage. 

fig. 59. Twelve-month centered ITl(JYing average of temperature difference ~ries bet>Neen 

composite and Argo near-SST anomalies with the buoy and HadSST3 serie5 kMged and all series 

redlJCed to common coverage to ensure consistent complete spatial and temporal coverage. 

fig,S10.Trendsand9S%confldenceimervals("Cperdecade)indifferencesenesforea{h 

IHSSTandcompositeSSTseries,withthebuoyandHadS$T3serieskrigedandal!series 

reduced to common coverage to ensure consistent complete spat1al and temporal cover~ge 

fig.S12.Dlfferences betweenERSSTv4and CCibylatitude;rone. 

frg. 513. Buoy·only and ship-only temperature anomalies from January 1997 to December 

2015,wrtllno matching of coverage 

frg. 514. D1fference betw~n sh!p-only reconl and the three Argo seoes using a 12·month 

centered moving average 

fig.S15.Comparison ofCOBE·SSTand COBE·SST2 totheiHSSTsusinga 12·monthcentered 

mov1ngaverage. 

frg. 516. Comparison of HadSST3 and HadNMAT2 to the !HSSTs iJSing a 12-month centered 

movrng aver~ge 

fig_ 517. Trends in drfferences for ERSSTv4 records versus IHSST records, with common 

coverage from 1997(buoysandCCionlyasdashedlines)andcommoncoveragefrom2005 

(buoys, CCI, and Argos assolidhnes) 

f1g. 518. Differences between Argo series at minimum reponed depth, artd differences within 

(9) ~~~~1~g~o:~::i:n~;~:~;r::~rd~e:~! ;o::: ~~·~ :::~s(drifting +moored) and onty 

where cr~ is the variance of the trend, o is the SD of the time series 

values, t; is the date of the ith value in fractional years, and v is the 

number of months of data per effective degree of freedom (36). Note 

that this differs from the ordinary equation for the uncertainty in a 

trend in the use of the SD of the time series in place of the SD of 

the residuals-this is because the difference in trends between a pair 

of series also contributes to the uncertainty. For the trend of a set of 

contiguous monthly values, this simplifies to 

(10) 

where M is the length of the period in years. v is about 2 for the buoy 

series or about 8 for the smoother Argo or CCI series. 

driftiflgbuoys 

flg.S20.Compansonofdriftingbuoy-based!HSSTrecordsfordlfferentqualitycontroland 

homogenization choKes. 

flg.S21.Twelve-monthcenteredmovingaverageofdlfferencesbetweeniHS5Tseriesfrom 

January 199710 De<:embo2r 2015 when reduced to common covet<~ge for each separate painng.. 

fig S22.Trenddlffe.-encemapsduring2005-201Sforthecompositerecordsvef'iuSBuoy,CCI, 

and Argo (H200S) 

flg.S23. Trends in cl1fferencesforcompositeversusbuoy(sohd lines}andCCI(dashed lifles) 

IHSSTrecordsWlthcommonccwerage. 

fig. 524. Number of observations over time by inmument type in the lCOADS (verston 25) 

f1g S25. Similar to fig. $24, but showing the percentage of ICOADS observations in e~ch year 

fromeach!nstrumenttype. 

REFERENCES AND NOTES 1. J. J. Kennedy, N, A. Rayner, R. 0. Smith, D. E. Parker, M. Saunby, Reassessing biases ~nd 

other uncertaimie> m sea surface temperature observations me~sured 1n situ since 

1850:2.llla:>esandhomogenization.J. Geophy;.. Rei. A/mos.ll6,01410/1.(2011), 

2. J.J.Kennedy,Areview ofuncenainzyinln situmeasurementsanddatasets of sea 

swfacetemperature.R<N.Geophys.52,1-32{1014). 

3. E. C. Kent, J. J. Kennedy, 0.1. Berry, R. 0. Smith, Effects of mstrumentation changes on sea 

SUPPLEMENTARY MATERIALS surface temperature measured in situ. Wiley lnterdi:odp. Rev. Gim. Change 1. 718-728 

Supplementary matNial for this ~rtide is available ~~ http:J/advances.sciencemag.org/cgV (2010). 

conterrt/full/3/1/e1001207/DC1 4, S, D. Woodruff, S, J. Worley, S.J. Lubker, Z.J1, E. Freeman, D.l, Berry, P. Brohan, E C. Kent. 

fig. 51. Trend maps on the 2005-2015 period for all of the composite records, and for the R W. Reynolds, S. R Smith, C. W1lkiru011, tcOADS Release 25: Extensi()(ls and enh<lr~cements 

buoy, ArgD, and CCI records. to the surface manne meteomlogica! archive. Int. J. Cfimatol. 31. 951-967 {2011). 

frg_ Sl. Comparioon of ERSSTv3b and ERSSTv4 with three different Arge>"based near-SST 5. B. Huang, V. f. Bai\ZOn, E. freeman, J. Lawrimore, W. Uu, T, C. Peterson, T. M, Smith, 

records, using the o;ame spatlal restrictions as In fig. l, but with ERSSTv4 al!gned to 1997-·2001 P. W. Thome, S. D. Woodruff, H.·M. Zhang, Extended reconstruaed sea surface 

{mdmive), with all other senes aligned onto ERSSTv4 using the 1005-2007 perkld !N!caus!"! of temperature version 4 (ERSST.v4). Part 1: Upgrades and !ntercompansons. J. Cl!m. 28, 

the limited time span with Argo data. 911~930 (2015). 

fig. 53. Trends and 95% confidence intervals ("C per decade} for the 1997 ·2012 period for 6. W. J, Emery, D. J. Baldwin, P. Schliisse!, R. W. Reynolds, Accuracy of in situ sea surface 

buoy, ARC. and CCIIHSSTs and each composite SST series, masked to common compO<;ite SST temper~tures used to 'ahbrate infrared satellite measurements.. J. Geoph~. Re5. Oceans 

coverage. 106,2387-1405(1001}. 

Hausfather eta/. Sd. Adv. 2017;3:e1601207 4 January 2017 12 of 13 

~ " 0 

., 

"-" "-

a 

3 
161 

SCIENCE ADVANCES j RESEARCH ARTICLE 

7. M. !shl!, A. Shouj:, S. Sugimoto, T. Matsumoto, Objective analyses of sea-surface 29. M. R~ehardson, K. Cowtan, E. Hawl<.in>, M. E. Stolpe, Reconciled d!mate respoose 

tMJperawre and marine meteorological v.niables for the 20th century using ICOADS and estimates from climate models and the energy budget of Earth. Nat. Cfim Change 6. 

the Kobe Co!ledJon.lm. J. Oima/JJI. 25, Bli5~879 (2005). 931~935 {2016). 

8, J. Han'ioen, R. Ruedy, M. 5.ato, K.to, Global surface temperatur" change. Rev. Geophys. 48, 30. D. Roemmich, J. Gilson, The glob;, I ocean imprmt of Ef\150. Goophys, Res. Lert. 38, L 13606 

fl(o4004{2010). (2011). 

9. C. P. Morice, J. J. Kennedy, N. A. Rayner, P. D. Jones, Quantifying uncenainties in global 31. D. Roemmich, J. Olurch, J. Gilson, 0. Monse\esan, P. Sutton, 5. Wijffe!s, Unab.J.ted 

and regional temperature change using an ensemble of obS€rvational estimates: The planetary warming and it> ocean structure smce 2006. Nat. Gim. Changes, 240.--245 

HadCRUT4 data set, J. Geophys. Res. Atmos. 117 10.1029/2011JD017187 (2012}, (2015). 

10. T. M. Smlth, R. W. Reynolds, T. C Peterson, J. Lawnmore, !mprovemMts to NOAA's 32. N. Cress~e, The origms of kriging. Math. Grol. 22, 239--251 (1990). 

h1stoncal merged la11d-ocean swfao'' tempetature analysis (1800--2006). J. C/im. 21, 33. P.). Klotzbach, R. A. Pielke Sr., R. 11.. Plelke Jr., J. R Christy, R T McNider, A11 alternative 

2283-ll%(2008). expl<mationfordifTerentialtemperaturetrel1.dsatthesurfaceandinth€'1awer 

11. R. S. Vose, D. Arndt, V. F. Banzon, D. R. Easterling, B. Glwson, B. Huang, E. Kearns, troposphere.). ~oph~ Res. Atmos_ 114, D21102 (2009). 

J. H. lawrimore, R. W. Reynolds, T. M. Smith, C N. Wtll!ams, D. 8. Wu€'flz, M. J. Menne, 34. B. D. Saotf!f, T. M. L Wigley, J. s. Boyle, D.). Gaffen, J. J. HnHo, D. Nychka, D. E. L. Pi!rker, 

T. C Peterwn, NOAA's merged land-ocean surf<~ce temperature analysis. Bull. Am E. li!ylor, Staustical Significance of trends and trend differences in layer·average 

Meteowl. Soc. 513.11577-1685 (2012). atmos~eric temperature time series.J. Geophys. Res. Almos.105, 7337-7356 (2000). 

12. K. Ishihara, Cakulation of global surface temperature ;momalies with CO BE· SST. Weatht-r 35. J. Lee, R. Lund, Revisiting stmple lmear regression with autocom;olated error<;. Btometrika 

Service Bu!letin73, 519-525 (2006) 91, 2-10·245 (2004). 

13. R Rohde, R. A.. Muller, R. Jacobsen, E. Muller, S. Perlmutter, A. Rosenfeld, J. Wwtele, 

0. Groom, C. Widham, A new estimate of the a~rage Earth surface !and temperature 

s~nning 1753to2011.Geoinfor.Geosta!.Afl0verview 1.1-7 {2013). 

14. K. CowJ:an, R. G. Way, Coverage bias in the HadCP.tJT4 temperature series and its im~C! 

onrecenttemperaturetrends.Q.J. R.Met/?()ro/.Soc.l40, 1935-1944 {2014) 

15. 8. Hvang, P. W. Thorne, T. M. Smith, W. Uu, J.lawrimore, V. F. Banzon, H.·M, Zhang, 

T.C.Peterson, M.Menne,Furtherexploringandquantifying UflCertaintiesforextended 

remnstr1..1cted sea surface temperature (ERSST) version 4 (v4). J. Clim. 29, 3119--3142 

(2015). 

16. W. tiu, B. Huang, P. W. Thorne, V. F. &mzon, H.·M. Zhang, [Freeman, J.lawrimort>, 

T. C Peterson, T. M. Smith, S. D. Woodruff, Extended reconstru(ted sea surfac:e 

Wmperaturever5ion4(ERSST.v4):Part!I.Parametricandstruc11.lraluncert.linty 

estimations.J.Ciirn.28,931 .. g51{2015). 

17. E. Client, N. A. Rayner, D. I. Berry, M. Saunby, B. I. Moat, J. J. Kennedy, D. E. Parker. Global 

analysis of nlght martne air temperature and its un,ertalnty since 1880: The HadNMAT2 

dataset.J.Geopfly.>.Res.AtmoS-118,1281-i29B(1013). 

18. T. R. Karl, A. Arguez. 8. Huang, J. H.l.Jwrimore, J. R McMahon, M. J. Menne, T. C. Petefl;on, 

RS.VOSJ'!:,H.·M.Zhang,PoSS!bleanifactsofdata biasesintherecentglob.J.Jsurf<Ke 

warrn1ng hlaws. Sdence 348, 1469-1472 {2015), 

19. G. Flato, J. Marot>:ke, B. Abiodun, P. Braconnot, S, Chou, W. Collins, Evalu<~tion of climate 

models,. in Climate Change 2013: The Phywo/XIence Basis, T. F. Stocker, D. Qin, 

G, K. P!attller, M. Tignor, S. K. Allen, J. Bosdmng, A- Nauels, Y. Xia, V. Bex, P. M. Medley, 

Eds.(U.mbridgeUniv. Press,.2013),pp. 741-866. 

20. Argo, Argo float data and metadata from Global Dilta AS5Emb!y Centre (Argo GDAQ. 

SEANOf, 10.17882(42182 [2002). 

21. C J. Men:hant, 0. Embury, N. A. Rayner, D.!. ~rry, G. K_ Corlett, K.tean, K. L Veal, 

E. C. Ken~ D. T. Uewoo!!yn-Jones, J. J. Remedios, A 20 ~ar independent record of S€a 

surface temperature f(lrchmate from Along·Track Scanning Radiometers. J. Geophy>. Ref 

Ckl'am117,C12013(2011). 

22. C. J. Men:hant, 0. Embury, J. Rober1S Jones, E. Fiedler, C. E. Bulgin, G. K. Corlett, S. Good, 

A. Mclaren, N. Rayner, S. Marak-Bozzo, C. Donlon, Sea surface temperature dat~sets for 

dimate apphcations from Ph.l!Se 1 of the European Space Agency Chmate Change 

lnittatlvc{SST CCI). Geosn. DaraJ. 1, ;79--191 (2014} 

23. W, Tang, S. H. Yueh. A. G. fwe, A. Hayashi, Val!dat1011 of Aquarius s~a surface salinity with 

in situ measurements from Argo 00<115 and moored buoys. J. Geophy>. Res. Oceans 1151, 

6171-6189(1014). 

24. S.Hoo;oda, T.Ohira, K.Sato, T.Suga,lmproveddescriptionofglobil.!IT'ixed,!ayerdepth 

using Argo profiling floats.J. Oceanogr. 66, 773~787 {2010). 

2S. S. Hosoda, T. Ohira, T. Nakamura. A monthly mean dataset of global oceanic 

temperature and sahnity derived from Argo float observations. JAMSTEC Rep. ReS- Dev. 

36. G.Foster,S.Rahmstorf,Giobaltemperatureevo!ution1979-2010.fnviron. Res. Letr,6, 

44022(2011). 

37. D. Tj.,5theim, J. Paulsen, Bias of s.ome commooly·used time s~ies ~limates. BiometrikCJ 

70,389-399{1983). 

3B. A. G. O'Carroll, J. R. Eyre, R. W. Saunders, Three·way error analysis betm>en AATSR, 

AMSR·E. and in situ Sei! surface temperature observations. J. Almos. OLean Tech. ll, 

1197-1207(2008) 

Acknowledgments: We thank four ar10nymous reviewers fcr feedback oo the amcle, 

0. Embury for assistance with ESA·CCI SST data, M. Ishii for information regardl!lg COBE·SST 

data, W. E~henbach for help in locating APDIK Argo data, and B. Huang for providing 

griddedERS5Tv4ensembledata,Argodatausedinthestudywerecol!ectf!dandmadefree!y 

available byth.e!nt.ernationaiArgo Programandtherrationatprograms that contribute to 

It (www.argo.ucsd.edu). Th~ Argo program !S pan of the Global Ocean ObserYing S~tem. 

Funding:Z.H.andR.R.werefundedby8erkeleyEarth.M.R.'sresean:hwa$carriedoutatthe 

JetPropuklofl L!boratory,Ci!lifornia !nstituteofTechnology,underacontractwith NASA. 

P J. was funded by George Mll~on University. No specific go.mts were al!oc:ated to support 

thispruject.Authorconttibutions:K.C.andZ.Hconceivedtheproject;K.C.i!ndR.R.produced 

the buoy records. Z.H. ~nd K.C analyzed the Argo data. M.R. analyzed the CCI dalil. K.C., 

D.CC, and ZJt analyzed time seri~s and trends. Z.H., K.C, PJ .. D.C.C, and M.A. wrote 

the pi!p~. with all authors providing input. Competing interests: All authors <Rdare 

that they hav~ no competing interests, Data and materials availability: fluoy and ship 

data are available from !COAOS at http://icoads.noaa.gov/producti.html; ESA-<:Cl d<lta are 

available from the Centre for EnVironmental Oi!ta Analy5ls at http;//gws-access.ceda.ac. 

uk/pub!ic2Jnceo_uor/ss11L3Sil=:XP1.2/; ARC-SST data are av~ilable at http://catalogue.ceda.ac 

uk/uUid/ff8a7f27b827c10Sdd97$6adffaaa942;ArgodataareavailablefromtheAPDRCat 

hllp://apdrooest,hawaii.edufproject;./Argofdata/grldded!On_stand!rdjevelslindex·l.html; 

Argodataareavallab!efromtheJa~nAgencyforMarine-EanhScil!nceandTechno!ogy 

(H2008) at http:/lwww~amst<i'C.go.jp/ARGO/argo_weblargo/?pa~_id"'83&1ang"'en; Argo data 

are avaiJilble fwm R()l'mlll!ch and Gilson (RG2009) at http;//;fo.argo.oc§d.edu~G_Ciimatology. 

html; ERSSTv4 dat.:l are avat!able at http-J/www1.ncdc.noaa.gov/pub/data/cmb/ersstlv4/netcdf/; 

ERSSTv3bdataareavailab!eathttp://www1.11Cdc.noaa.govfpub/data/crnb/ers:stlv3blnetcdf/; 

HadSST3 data are available at http:ltwww.metorflce.gov.ul<lhadobs!hadsst3/datd!download. 

html;CDBE-SSTdataareava!lableathttp;!/ds.datajma.gojp/te<:Jtcc/prcducts/elnino/cobessV 

eobe-ssthlml; and the Python code u~d in this analySis 1s available at http;//www·users.york. 

ac.ul<l-kck3/papers/Ihsst2016/.A!Idatasetsusedweremostr<i'Centlyaccessedinearty 

July2016.A1Idataneededtoeva!uatethecoodmlol'l.sin thepap<i!rarepresent!nthepaper 

~nd/or the Supplementary Materials. Additional dat.l related to this paper miljl be reque:<ited 

from the authors 

8,47-59{2008). Submittf!d27May2016 

26. D. RoemmiCh, J. Gils(l(l, The 2004-2008 mean and annual cyde of temp<i!rature, salinity, Accepted 22 November 2016 

andstericheightinthegloba!oceanfwmtheArgoprogram,Prag. OreanDgr. 8]., Publlshed4January2017 

81-100(2009). 10.1126fscladv.1501207 

27. S.Hirah.ara, M, lshll, '(. Fu~uda,Centennia!-scale sea surface temperature analysis and its 

uncertainty. J. C/im. 27, 57-75 (1014). Citation: Z. Hausfather, K Cowtan, D. C. Clarke, P.Jacobs, M. Richardson, R Rohde, Assessing 

28. R. W. Reynolds, T. M. Smlth, C Uu, D. 6. Chelton, K. S. (aS€y, M. G. Schklx, Daily high- recent warming using instrum~ntally homogeneous sea surface tem~rature r<i'Cords. Sci. Adv. 

resolutioo-blendedanalyS€sforseasurface temperature.J. Cllm.20,S473-5496(2007). 3,e1601207 (2017). 

Hausfather rt a!. Sci. Adv. 2017;3 :e1601207 4 January 2017 13 of 13 
162 

ScienceAdvances &1MAS 

Assessing recent warming using instrumentally 

homogeneous sea surface temperature records 

Zeke Hausfather, Kevin Cowtan, David C. Clarke, Peter Jacobs, 

Mark Richardson and Robert Rohde (January 4, 2017) 

Sci Adv 2017, 3:. 

doi: 10.1126/sciadv.1601207 

This article is publisher under a Creative Commons license. The specific license under which 

this article is published is noted on the first page. 

For articles published under CC BY licenses, you may freely distribute, adapt, or reuse the 

article, including for commercial purposes, provided you give proper attribution. 

For articles published under CC BY-NC licenses, you may distribute, adapt, or reuse the article 

for non-commerical purposes. Commercial use requires prior permission from the American 

Association for the Advancement of Science (AAAS). You may request permission by clicking 

here. 

The following resources related to this article are available online at 

http://advances.sciencemag.org. (This information Is current as of February 7, 2017): 

Updated information and services, including high-resolution figures, can be found in the 

online version of this article at: 

http://adva nces.sciencemag. org/contenU3/1/e 1601207. full 

Supporting Online Material can be found at: 

http://advances.sciencemag.org/contenUsuppl/2016112/30/3.1.e1601207.DC 1 

This article cites 34 articles, 3 of which you can access for free at: 

http://advances.sciencemag.org/contenU3/1/e1601207#BIBL 

Science Advances (ISSN 2375-2548) publishes new articles weekly. The journal is 

published by the American Association for the Advancement of Science (AAAS), 1200 New 

York Avenue NW, Washington, DC 20005. Copyright is held by the Authors unless stated 

otherwise. AAAS is the exclusive licensee. The title Science Advances is a registered 

trademark of AAAS 

0 

0 

" :;, 

0 

w 

c. 

(!) 

c. 

0 

3 

::r 

-6' 

~ c. 

< 

w 

@ 

II> 

in 

0 

c;;· 

:;, 

£ 

3 

Ql 

1.0 

0 

cg_ 

0 

:;, 

;p 

CT 

2 

Ql 

-< 

_.., 

N 

0 

"' 
163 

Climatic Change (2015) 133:129-140 

DOl 10.1007/sl0584-015-1495-y <I> CrossMark 

Debunking the climate hiatus 

BaJa Rajaratnam1•

2,3 ·Joseph Romano3•4 • 

Michael Tsiang1 ·Noah S. Diffenbaugh1•

2 

Received: 12 May 2015/ Accepted: 1 August 2015/ Published online: 17 September 2015 

©The Author(s) 2015. This article is published with open access at Springerlink.com 

Abstract The reported "hiatus" in the warming of the global climate system during this 

century has been the subject of intense scientific and public debate, with implications ranging from scientific understanding of the global climate sensitivity to the rate in which 

greenhouse gas emissions would need to be curbed in order to meet the United Nations 

global warming target. A number of scientific hypotheses have been put forward to explain 

the hiatus, including both physical climate processes and data artifacts. However, despite 

the intense focus on the hiatus in both the scientific and public arenas, rigorous statistical 

assessment of the uniqueness of the recent temperature time-series within the context of the 

long-term record has been limited. We apply a rigorous, comprehensive statistical analysis 

of global temperature data that goes beyond simple linear models to account for temporal 

dependence and selection effects. We use this framework to test whether the recent period 

has demonstrated i) a hiatus in the trend in global temperatures, ii) a temperature trend that is 

statistically distinct from trends prior to the hiatus period, iii) a "stalling" of the global mean 

temperature, and iv) a change in the distribution of the year-to-year temperature increases. 

Electronic supplementary material The online version of this article 

(doi:l0.1007/sl0584-015-1495-y) contains supplementary material, which is available to authorized 

users. 

["] BaJa Rajaramam 

brajarat@stanford.edu 

Department of Earth System Science, Stanford University, 473 Via Ortega, Rm 140, Stanford, CA 

94305-4216, USA 

Woods Institute for the Environment, Stanford University, 473 Via Ortega, Starrford, CA 

94305-4216, USA 

Department of Statistics, Stanford University, Sequoia Hall, Stanford, CA 94305-4065 USA 

Department of Economics, Stanford University, 579 Serra Mall, Stanford, CA 94305-6072, USA 

~Springer 
164 

!30 Climatic Change (2015) 133:129-140 

We find compelling evidence that recent claims of a "hiatus" in global warming lack sound 

scientific basis. Our analysis reveals that there is no hiatus in the increase in the global mean 

temperature, no statistically significant difference in trends, no stalling of the global mean 

temperature, and no change in year-to-year temperature increases. 

1 Introduction, Motivation and Approach 

The international debate on the "hiatus" in the warming of the global climate system over the 

last 15 years has intensified (e.g., Meehl eta!. (2011), IPCC (2013), Otto eta!. (2013), Fyfe 

eta!. (2013), Kosaka and Xie (2013), Santer eta!. (2014), Trenberth and Fasullo (2013), 

Smith (2013), Guemas eta!. (2013), Chen and Tung (2014), Boykoff (2014), Hawkins eta!. 

(2014), England eta!. (2014), Karl et al. (2015), Cowtan et al. (2015)). The implications 

of the purported hiatus (also referred to as a "pause" or "slowdown") are far reaching. 

First, contradictory scientific conclusions have emerged regarding the relationship between 

climate change and anthropogenic global warming, especially during a period of heightened 

carbon emissions (Kosaka and Xie 2013). Second, the discrepancy between climate model 

projections and observations appear to point to an overestimation of climate sensitivity to 

anthropogenic forcings (Otto et al. 2013; Fyfe eta!. 2013). 

The perceived hiatus has led to a myriad of resources being expended on trying to better 

understand the geophysical mechanisms that lead to a possible hiatus (including, among others, volcanic activity (Santer eta!. 2014), Pacific Ocean variability (Kosaka and Xie 2013; 

Trenberth and Fasnllo 2013), and increased ocean heat uptake (Smith 2013; Guemas eta!. 

2013; Chen and Tung 2014)), as well as spurious artifacts of the global climate observing 

system (Durack eta!. 2014; Cowtan and Way 2014; Karl eta!. 2015; Cowtan et al. 2015). 

The purported hiatus has therefore inspired valuable scientific insight into the processes that 

regulate decadal-scale variations of the climate system. However, the perception of a hiatus has important repercussions for public decision making, as the implications that global 

warming has paused or slowed down (Boykoff 2014; Hawkins eta!. 2014), and that climate 

models have overestimated the rate of warming (e.g., Fyfe eta!. (2013)), both influence the 

perceived level of mitigation action that is needed to obtain particular policy targets (Otto 

et al. 2013). 

Fundamental to any work on the hiatus is to ascertain whether there is sufficient empirical evidence in support of its existence. Surprisingly, to our knowledge, a rigorous statistical 

analysis has not been undertaken, at least not one which incorporates temporal dependencies without making strong assumptions about the underlying process. Without empirical 

evidence in support of the hiatus claims, any further conclusions stemming from the 

assumption should be called into question. 

As a part of our investigation to better understand the hiatus, we develop a comprehensive scientific framework that is intended to systematically test hypotheses that have been 

implied in statements claiming a hiatus in global warming. We first identify a typology of 

the scientific assertions that have been put forward, including i) thal there has been a hiatus 

in the trend in global warming, ii) that there is a difference in trends before and during the 

hiatus, iii) that there has been a hiatus in the change in mean global temperature, and iv) 

that there is a difference in warming before and during the hiatus (when accounting for possibly non-linear increases without explicit reference to a linear trend). (See Supplementary 

Section 3.4 for more detail on the typology.) We next connect these scientific claims 

with four classes of distinct testable statistical hypotheses, with each hypothesis focusing 

~Springer 
165 

Climatic Change (2015) 133:129-140 131 

on different aspects of the underlying (unknown) temperature process. We then identify and develop appropriate statistical tools in order to test each of these hypotheses in 

a principled manner, and under progressively less restrictive - and therefore more generally applicable-modeling assumptions, thereby allowing for a deeper understanding of 

the nuances of the global temperature time series. In particular, we attempt to properly 

account for temporal dependence, we use less restrictive resampling methods to assess 

statistical significance, and we employ a flexible nonparamctric modeling approach. By 

applying these progressively more general techniques in a cascading approach, we are able 

to test the extent to which invalid statistical assumptions can lead to erroneous scientific 

conclusions. 

Our analysis is first undertaken using the NASA-GISS global mean land-ocean temperature index. It is subsequently also repeated on the NOAA and HadCRUT4 datasets for 

comparison purposes (see Supplemental Tables I and 2). The analysis is also undertaken 

on the recently released ERSSTv4 (Karl eta!. (2015)) datascts (see Supplemental Tables II 

and 12). Plots of the NASA-GISS raw and smoothed global mean land-ocean temperature 

index from 1880 to 2013, with the base period 1951-1980, are given in in Fig. 1 (top). As 

there is a clear underlying trend, a moving average is superimposed on the time series. A statistical analysis of the serial correlation in the residuals after fitting a regression line is also 

given in Fig. I (top). The autocorrelation in the temperature time series is non-negligible. 

Fig. 1 Top panel global mean 

land-ocean temperature index 

from 1880 to 2013, with base 

period 1951-1980 and moving 

average superimposed. The table 

provides Durbin-Watson and 

Ljung-Box p-values for the 

residuals from three OLS fits 

between 1950-2013. The LjungBox test here considers residual 

autocorrelation in the first 20 

Jags. The 1950--2013 Full OLS 

model fits a single regression line 

to all observations from 1950 to 

2013. The 1950--2013 Separated 

model fits a separate regression 

line to the 1950--1997 and 

1998-2013 periods. Bottom 

panel plot of the global mean 

land-ocean temperature index, 

from 1998 to 2013, with the 

ordinary least squares regression 

line superimposed 

Temporal Dependence 

Ot.S"""" 

1950-1997 O.D18 

1950-2013FUH 0 

1950-2013$eparaled e: 0.50 +====:___:::::.:__-=.:.:c._j 

J 

i 

~ -0.50 

0.00 

e: 0.85 

I 

~ 

0.80 

0.75 

J 0.70 

0.85 

0.60 

-Raw Data - Moving Average of Order 5 

1880 1900 1920 1940 1960 1980 2000 

Hypothesis 1 

-Raw Data 

- OLS Line 1998-2013 

(2-sided) 

0.051 (1-sided) 

2000 2005 2010 

Signficance: >10% 5% 1% 0.1% 

~Springer 
166 

132 Climatic Change (2015) 133:129-140 

The presence of autocorrelation motivates the need to use less naive statistical methods to 

understand the evolution of temperature over time (see also Supplemental Section 2.2). 

2 Methods 

Datasets The datascts of global surface temperature anomalies used in our analysis come 

from three sources: the NASA Goddard Institute for Space Studies (GISS) Surface Temperature Analysis (GIS TEMP) Data, the NOAA National Climatic Data Center (NCDC) data, 

and the HadCRUT4 data, produced from the Met Office Hadley Centre in collaboration 

with the University of East Anglia Climatic Research Unit (CRU). Each source combines 

monthly land and sea surface temperature measurements into spatial grids that are then 

averaged into a single global temperature series. Temperature anomalies are computed from 

a baseline period, which differs by dataset. The differences in the three datasets largely 

come from the adjustment/infilling methods for sparse temporal/spatial coverage (Hansen 

eta!. 2010; Morice eta!. 2012). (See the Supplemental Section for more details). Note that 

given the global mean temperature data that is available, the main goal of our analysis is 

to understand the possible mischaracterizing of hiatus claims as compared to understanding 

the source of observational errors of the temperature process. 

Temporal dependence and uncertainty quantification The global temperature record 

exhibits temporal correlation. Standard statistical methods tend to ignore this important 

feature, which in tum can lead to incorrect statistical modeling assumptions and incorrect 

statistical significance, which can in turn lead to erroneous scientific conclusions. For the 

purposes of uncertainty quantification when. testing each of the four statistical hypotheses, 

we either model the temporal dependence in the global temperature time series explicitly 

through a parametric autoregressive model, or account for it through the nonparametric 

circular block bootstrap, stationary block bootstrap, or subsampling. (See the Supplemental 

Section for more details.) 

Statistical hypothesis testing The various scientific assertions regarding the global 

warming hiatus arc collected into four groups and then formulated as four testable statistical hypotheses. These four hypotheses are specified rigorously, in a principled statistical 

framework, and are given in Supplemental Sections 3.1, 3.2, 3.3 and 3.4. The Wald test is 

used to test slope parameters in the linear regression model in Hypotheses I and II leading 

to Normal or !-distribution based p-values. Moreover, p-values based on the bootstrap and 

subsampling are also calculated as alternatives to the Wald test whenever appropriate. When 

comparing two distributions, the Kolmogorov-Smirnov test is used, together with the bootstrap or subsampling, to account for temporal dependence. (See the Supplemental Section 

for more details.) 

Observational uncertainties It is important to recognize that the temperature data that 

is used in our analysis are estimates of an unobserved process and is thus subject to observational errors and the implied uncertainties. Observational uncertainties could arise due to 

various factors, including instrumental error, changes in the observing network configuration and observing technology, and also due to uncertainties in adjustments made to the data. 

The HadCRUT4 dataset allows an analysis that incorporates observational uncertainties. 

The single time-series used for the analysis of the HadCRUT4 data is actually derived from 

~Springer 
167 

Climatic Change (2015) 133:129-140 133 

multiple time series which are constructed in order to reflect observational uncertainties. 

This analysis is provided in Supplemental Section 4. 

3 Results 

3.1 Hypothesis 1: hiatus in temperature trend during 1998-2013 

A basic assertion regarding the hiatus is that the steady increase in global surface· temperature around a linear positive trend has stopped, or "paused" (Guemas et al. 2013). 

This sentiment is reflected in statements that "Despite a sustained production of anthropogenic greenhouse gases, the Earth's mean near-surface temperature paused its rise 

during the 2000-2010 period" (Guemas et al. 2013), and that "climate skeptics have 

seized on the temperature trends as evidence that global warming has ground to a halt" 

(Tollefson 2014). These scientific claims can be turned into a precise statistical null hypothesis: the slope in the regression line of global temperature on time is zero during the hiatus 

period. 

We use three methods with increasing levels of generality to test the above hypothesis. Specific details of the methodology are provided in Supplementary Section 3.1. First, 

beginning with the 1998-2013 period we fit a standard regression to the response variable 

global temperature on time during 1998-2013, with errors assumed to be independently and 

identically distributed (see Fig. I for the fit). A two-sided hypothesis test yields a p-value 

of 0.102 (a one-sided test yields a p-value of 0.051). Thus, the claim of a zero warming trend during the hiatus period cannot be rejected at the 5 % significance level. The 

second method fits a linear regression with autocorrelated errors that follow a parametric autoregressive model with lag I. This model aims to directly address the year-to-year 

temporal dependency present in the global temperature record. Estimating the autoregression and regression parameters using the method of Cochrane and Orcutt ( 1949), a p-value 

of 0.075 is obtained for the regression slope coefficient by the bootstrap method (with 

one-sided p-value less than 5 % ). Taking temporal dependence into account, there is now 

more evidence against the null hypothesis of a climate hiatus. The third method is completely nonparametric, and instead of using the parametric AR(l) approach to model the 

temporal dependency, a block bootstrap is used which allows for quite general forms of 

temporal dependence, and yields a two-sided p-value of 0.019. There is now compelling 

evidence to reject the claim of no warming trend during the 1998-2013 period at the 5 % 

significance level (and even at the I % level for a one-sided test). Moreover, the p-values 

corresponding to starting years 1999 and 2000 are 0.005 and 0.017 respectively, yielding 

even lower p-values - and stronger evidence against a hiatus - than when using a starting 

year of 1998. The sensitivity analysis highlights the fact that choosing the year 1998 had a 

priori favored the hiatus claim. Moreover, assuming the hiatus as the null makes it harder 

to conclude otherwise. Regardless, the assertion of a climate hiatus is nevertheless rejected 

at the 5 % level. We therefore conclude that there is "overwhelming evidence" against 

the claim that there has been no trend in global surface temperature over the past ~ 15 

years. 

Note also that, in applying progressively more general statistical techniques, the scientific conclusions have progressively strengthened from "not significant," to "significant at 

the 10 % level," and then to "significant at the 5 % level." It is therefore clear that naive 

statistical approaches can possibly lead to erroneous scientific conclusions. Methods that 

rely upon a strong modeling assumption of no temporal dependence, or that of a specific 

~Springer 
168 

134 Climatic Change (2015) 133:129-140 

fonn, are less reliable than methods that capture dependence without assuming structural 

knowledge of the type of dependence. 

3.2 Hypothesis II: difference in temperature trends 

Otto et al. (2013) state that: "the rate of mean global warming has been lower over the 

past decade than previously." This statement encompasses a second interpretation of the 

purported hiatus: that the hiatus represents a "slowdown" of global warming (Chen and 

Tung 2014), in which the rate of warming is less during the hiatus compared with the 

warming prior to the hiatus (Chen and Tung 2014; Otto et al. 2013; Smith 2013). This 

claim can be formulated as a testable statistical hypothesis, where the null hypothesis is 

that the regression slope before the hiatus period minus the regression slope during the 

hiatus period is zero or negative, versus the alternative hypothesis that this difference 

is positive. 

We employ three different methods with increasing levels of statistical sophistication 

to test this hypothesis. Specific details of the methodology are provided in Supplementary Section 3.2. First, a standard regression of global temperature on time is fitted to both 

the 1998-2013 hiatus period and the period 1950-1997, with errors assumed to be independently and identically distributed (see Fig. 2 top left panel). The first method yields a 

p-value of 0.210. Thus, there is no evidence of a difference in warming trends even at the 

10 % significance level. The second method accounts for the temporal dependency in the 

global temperature record by using a block bootstrap approach, yielding a p-value of 0.323. 

The evidence for a difference in trends is further weakened when temporal dependency 

is accounted for. The third approach uses the method of subsampling (Politis et al. 1999; 

Rajaratnam et al. 2014) to determine how the current 16-year trend during 1998-2013 compares against all the previous 16-year trends observed between 1950 and 1997. A p-value of 

0.3939 is obtained and evidence for the hiatus is further weakened. From the plots in Fig. 2 

(bottom panel), observe that during the \950-1997 period, there are several 16-year periods 

with both higher and lower linear trends. Therefore the observed trend during 1998-2013 

does not appear to be anomalous in a historical context. 

Sec Fig. 2 (top right panel) for a summary of results of hypothesis II. Varying the 

cut-off year from 1998 to either 1999 or 2000 yields p-values of 0.214 and 0.348, respectively, for the bootstrap method. Even after properly accounting for temporal dependence, 

and undertaking a sensitivity analysis, there is no compelling evidence to suggest that the 

slopes are significantly different. We therefore conclude that the rate of warming over the 

past :::::: 15 years is not appreciably different from the rate of warming prior to the recent 

period. 

3.3 Hypothesis III: hiatus in the mean global temperature 

Some claims have simply asserted that the annual mean global temperature has remained 

constant since 1998 (versus slowing of the trend in global warming). For example, Kosaka 

and Xie (2013) state that "Despite the continued increase in atmospheric greenhouse gas 

concentrations, the annual-mean global temperature has not risen in the twenty-first century", while Tollefson (2014) states that "Average global temperatures hit a record high in 

1998 -and then the warming stalled." This claim can also be precisely formulated as a 

testable statistical hypothesis. The statistical model can be written as x 1 = !Lt + s1, where t 

denotes time (in years), x 1 is the 1998-2013 global mean temperature anomalies series, ~t 1 

is the mean parameter and t:1 is the random noise component(with E(t:1 ) = 0, Var(s1) 

~Springer 
169 

Climatic Change (2015) 133:129-140 

Hypothesis 2 

e: -A<lwData 

2:- 0.8 - OLS Une 1950-1997 

E "'0.6 0 

c: 

<( 0.4 

re 0.2 '§ " 

" 0.0 c. 

E -0.2 "' ,__ 

1950 1970 1990 2010 

16-Year UnearTrend 

Estimates 

0.025 -;::- G o.o2o 

';:; 0.015 c: 0.010 ~ <;; 0.005 

" Observed 

c: 0.000 _ ..... 1998-2013 

::J Trend 

-o.oo5 

1950 1960 1970 1980 

Starting Year 

Difference between porklds: 

Method 

Classical approach 

(assumes independence) 021 

The nonparametric 

block boo !strap 0.32 

Subsampling 0.39 

Signficance· 

>10% 5~{: 1'%. Q_1% 

60 

Histogram of 16· Year 

Linear Trend Estimates 

smothed 

density 

1 Observed 

I 1998·2013 

I Trend---

1 

I 

I ~ I 40 c: 

0 " 20 

-0.005 0.005 0,015 0.025 

135 

Fig. 2 Top panel (left) plot of the global mean land-ocean temperature index, from 1950 to 2013, with the 

base pedod of 1951-1980. The regression fits for the two time pedods (1950--1997 and 1998-2013) are 

supedmposed. Top panel (right) summary table of results for Hypothesis II Bollom panel (left) time sedes 

plot of 16-year observed trends. Bottom panel (right) histogram of 16-year observed trends 

a 2 ). The corresponding null hypothesis and alternative are given as Ho : E(XJ99s) = 

E(XJ998+tl for t = 1, 2, · · · , 15 versus HA : E(XJ99S) f= E(XJ998+1). 

Specific details of the methodology are provided in Supplementary Section 3.3. Hypothesis III is tested in four different ways. There are two options for determining the value of 

E[XJ998l = J.l.I998 :to directly use the observed 1998 temperature record XJ998 as a substitute for J.l.J998, or to alternatively estimate J.l.l998 from the regression line from the period 

1950--1997. Figure 3 (top panel) illustrates this concept. As the two approaches for specifying J.l.!998 yield fixed values, the inherent variability therein can be explicitly accounted for 

by using the bootstrap. Doing so propagates the variability in a rigorous manner. The table 

in Fig. 3 (bottom panel) summarizes the results of testing hypothesis III. 

For Method A, when XJ998 is used as a substitute for J.l.l998, the statistical test concludes that the mean has decreased during the hiatus, and thus strongly favors the hiatus 

claim. However, since this one single observed value is not a consistent estimate of J.l.I998, 

the conclusion is not reliable. In Metiiod B when J.l.I998 is estimated from the 1950--1997 

regression line, the null hypothesis is rejected in the opposite direction, suggesting that the 

mean temperature has actually increased during the hiatus period. Thus, tiie selection effect 

from choosing 1998 as the reference cut -off year has a tremendous impact on the statistical 

conclusion. Method C, which specifically incorporates the variability inherent in estimating 

J.l.I998 as XJ998 leads to a different conclusion than in Method A. In particular, as soon as the 

variability in estimating J.l.l998 to be XJ998 is incorporated, one can no longer reject the null 

hypothesis that the mean has remained constant - even when the high value X!998 is used. 

Method D uses a value for J.l.l998 which is estimated from the 1950--1997 regression and 

~Springer 
170 

136 

Fig. 3 Top panel figure 

illustrating how the mean lll998 

can be estimated. Bottom panel 

summary table of results for 

Hypothesis III with !998 as start 

of hiatus period 

e: 

.2-

E "'

0 

c: 

-<: 

~ ::J 

1ii 

(;; 

c. 

E 

"' .... 

Climatic Change (2015) 133:129-140 

Hypothesis 3 

0.8 

0.6 

0.4 

0.2 

0.0 

·0.2 

• x199S 

1\ 

0 ~1998 

1950 1970 1990 

Raw Data 

1950·1998 

OLS Line 

2010 

Hypothesis 3 with 1998 as start of ~hiatus" period: 

E(Xm11) Variability of X1998 Result Remark 

x,s~ Assume fixed Reject H0 deC<"ease in mean 

A Assume fixed Reject Ho increase in mean 1-11M8 

X199s Simulate by bootstrap Retain H0 no chonge in mean 

A Simulate by bOOtstrap Reject fie !ncrea.seinmeara ].119'9$ 

also incorporates the variability of this estimate. Here the assertion that the mean is either 

zero, or has decreased, is rejected. 

Given the results of this nuanced analysis, we conclude that claims that the global mean 

temperature has not changed in recent decades are not supported by evidence. In addition, 

our nuanced analysis gives much needed rigor to the claim that using 1998 as a reference 

year amounts to "cherry picking" (Leber2014; Stover 2014), see also Supplemental Section 

for detailed discussions). The results are further validated when the analysis is repeated 

with 1999 and 2000 as the starts of the hiatus period (sec Supplemental Section 3.3). Note 

furthermore that since 2014 was the warmest year on record Karl et aL (2015), ignoring 

2014 in our analysis can be viewed as being even more conservative, similar to using J 998 

as the starting point. 

3.4 Hypothesis IV: difference in year-to-year temperature changes 

It is also instructive to extend the analysis above without relying on a linear model to 

understand trends or means. One such approach is to assess whether the distribution of 

year-to-year temperature changes is markedly different between the hiatus period and the 

prior periods. Such analysis is inherently less reliant on a statistical model of temperature 

on time, and hence makes fewer assumptions. The scientific assertion here is that yearto-year changes in global mean temperature during 1998-2013 are different from those 

during 1950-1997. Under the null hypothesis, these year-to-year changes are assumed to 

come from a common underlying distribution, though we do not assume that the observations of differences are independent. This framework also allows for testing of specific 

features of the distribution, including changes in the mean, median and variance. The empirical distribution of annual changes in the global temperature can be constructed by taking 

first differences: the global mean temperature during a given year is subtracted from the 

global mean temperature in the previous year. The first differences during 1998-2013 give 

rise to a IS-year times series of temperature changes. Differences in distribution (using the 

~Springer 
171 

Climatic Change (2015) 133:129-140 

Fig. 4 Top panel time series plot 

of 15-year observed KS 

differences. Bottom panel 

summary table of results for 

Hypothesis IV using bootstrap 

and subsampling S! 0.20 

j 

oll 

,!! 

j 0.15 

~ :; 

~ 

~ 0.10 

137 

Difference between 1998-2013 and 

the pre-1998 15-year periods 

distribution 

Observed 1998-2013 

K·SS!atislic 

1950 1955 1960 1965 1970 1975 1980 

Starting Year 

Hypothesis 4 using bootstrap and subsampling: 

Test Bootstrap Subsampling 

p-.value p-va!ue 

Difference in distribution 0.295 

Difference in mean 0.362 0.118 

Difference in medfan < 0.029 

Difference in variance 0.496 0.265 

Difference in log variance 0.483 

Signficance: >10% 5oo 1% 0.1% 

Kolmogorov-Srnimov (K-S) statistic), in means, medians and variances are tested using 

the block bootstrap and subsampling, thus taking temporal dependency fully into account. 

Specific details of the methodology are provided in Supplementary Section 3.4. 

The results of this analysis are given in Fig. 4. Using either bootstrap or subsampling 

there is no evidence at the 5 % significance level to suggest that the distribution of changes 

during the hiatus period is different from the previous period 1950-1997. The same applies 

to the mean and variance of the distributions, The difference in medians is not statistically 

significant at the 5 % level using the block bootstrap approach, but is significant when 

using subsampling. However this difference in medians completely disappears when the 

starting year of the hiatus is changed to either 1999 or 2000, hence the result is not robust 

(see Table S8 in Supplemental Section 3.4). Given these results, we conclude that the distribution of annual changes in global temperature has not been different in the past 15 years 

than earlier in the global temperature record. 

3.5 Re-analyzing recently-updated global temperature observations 

We have also implemented our methodology on the recently released ERSSTv4 dataset to 

compare our results to the results obtained in a recent paper by Karl et al. (2015). Unlike 

the study by Karl et al. (2015), we do not indirectly impose Gaussianity on the temperature data (in the most general approach that we propose for each hypothesis). We also do 

not impose an autoregressive structure for modeling the temporal dependence. Instead we 

account for the temporal dependency more flexibly and non-parametrically using the circular block bootstrap and related methods. The increased sophistication allows one to have 

more confidence in the results' general validity as our approach makes fewer assumptions. 

~Springer 
172 

138 Climatic Change (2015) 133:129-140 

The end result is also compelling. First, the results in Karl eta!. (20 15) show a positive slope 

during the hiatus period (Hypothesis I) only at the 10 % significance level. Our analysis 

shows however that removing the arbitrary and parametric autoregressive structure on the 

residuals and using the block bootstrap yields significance at the 0.1 %level. The p-value 

stemming from our approach is less than 0.0005. The implication of the much stronger conclusion is that the warming trend observed during 1998-20141 arising from a model of no 

warming is less than 1 in 2000 (as compared to less than 1 in 20 from Karl eta!. (2015)). 

Thus the conclusion is made stronger by a factor of 100 using the methodology we have 

developed. 

Now consider hypothesis II which compares the warming trend during the hiatus period 

to that in the previous period (1950-1997). Karl et a!. (2015) assert that the analysis on the 

corrected NOAA global temperature shows that the 90 %confidence interval for the trend in 

the hiatus period encompasses that of the previous period. Note that this confidence interval 

is based on the period 1998-2012 and is thus calculated on only 15 years of data. Since the 

theoretical justification of such confidence intervals is valid for large sample sizes, it is not 

clear how reliable the conclusion really is. On the other hand, our subsampling methodology 

for comparing the trends in the two periods is applicable even when the sample size in the 

hiatus period is small. In particular, the validity of the subsampling approach here does not 

rely on asymptotic arguments (i.e., increasing sample sizes) during the hiatus period. Details 

ofthe analysis are given in Tables S 11 and S 12 in Supplementary Section 6. 

Recall that the analysis by Karl eta!. (2015) requires the use of the corrected NOAA 

dataset to reject the claim of a hiatus. We note that our analysis rejects the hiatus claim even 

when using the older NOAA temperature dataset (that is, even without correcting for the 

data biases). The use of methodology with far fewer restrictive assumptions appears to be 

more robust to errors in the data. This may not be unexpected since biases in the data tend 

to violate basic parametric assumptions, whereas the less restrictive techniques, such as the 

ones we develop, can handle a variety of data generating mechanisms simply by their very 

non-parametric nature. 

Note that, by and large, the conclusions reached by Karl et al. (2015) and our conclusions 

agree. However, it is important to mention that an approach based on stringent or unrealistic 

assumptions which agrees with our conclusions for this dataset may fail to do so on another 

dataset. 

3.6 Summary 

We summarize the overall results from all four hypothesis tests I, II, III and IV in 

Tables 5 and 6 in Supplementary Section 4. These two tables also analyze the sensitivity 

of the results to two important factors: first when the cut-off year is changed from 1998 to 

either 1999 or 2000; and second when the NOAA or HadCRUT4 datasets are used instead 

of the NASA-GISS dataset. As there are four hypotheses being tested, using a battery of rigorous test procedures, the number of hypothesis being tested are numerous. Hence the issue 

of multiple hypothesis testing surfaces. In particular, a certain number of these hypotheses are expected to be falsely rejected by chance alone, casting further doubt on any of the 

hiatus claims. 

1

Though we consider the hiatus period as 1998-2013 elsewhere in the analysis, we consider the hiatus period 

as 1998-2014 here in order to compare directly with Karl eta!. (2015) 

~Springer 
173 

Climatic Change (2015) 133:129-140 139 

Our rigorous statistical framework yields strong evidence against the presence of a global 

warming hiatus. Accounting for temporal dependence and selection effects rejects - with 

overwhelming evidence - the hypothesis that there has been no trend in global surface temperature over the past "='15 years. This analysis also highlights the potential for improper 

statistical assumptions to yield improper scientific conclusions. Our statistical framework 

also clearly rejects the hypothesis that the trend in global surface temperature has been 

smaller over the recent"=' 15 year period than over the prior period. Further, our framework 

also rejects the hypothesis that there has been no change in global mean surface temperature over the recent"=' 15 years, and the hypothesis that the distribution of annual changes in 

global surface temperature has been different in the past "=' 15 years than earlier in the record. 

Taken together, these results clearly reject the presence of a hiatus, pause, or slowdown in 

global warming. In rejecting all four hiatus hypotheses, our results instead demonstrate that 

the evolution of global surface temperature over the past 1-2 decades is not abnormal or 

unexpected within the context of the long-term record of variability and change. 

Without empirical evidence in support of the hiatus claims, the assumption that there has 

been a hiatus/pause/slow-down in global warming should be called into question. That being 

said, recent work investigating the geophysical causes of the recent temperature time series 

have provided valuable insights into the processes that create decadal-scale variability in 

global temperature within a long-term trend of global warming. Moreover, it is also useful 

that errors in data aggregation have been corrected in the recent work of Karl eta!. (2015). 

Author Contributions B.R. and N.D. conceived the initial scope of the study; B.R. and J.R. conceived 

details of the overall investigation; B.R. designed the scientific framework. B.R. and J.R. translated the 

scientific hypothesis into statistical hypothesis and designed the statistical framework; M.T. analyzed the data 

and implemented the analytical tools; B.R. and J .R. developed and supervised the statistical analyses; B.R. 

synthesized the paper. All authors discussed the results and commented on the draft. B.R. was partially funded 

by the US National Science Foundation under grants DMS-CMG 1025465, AGS-1003823, DMS-1!06642, 

DMS-CAREER-1352656 and the US Air Force Office of Scientific Research grant award FA9550-13-l0043. J.R. was partially funded by the US National Science Foundation under grants DMS-1307973 and 

DMS-1007732. N.S.D. was partially funded by NSF-CAREER-0955283. 

Acknowledgments The authors thank Ben Santer, Page Chamberlain, Francis Zwiers, and an anonymous 

referee for useful comments and discussions 

Conflict of interests The authors declare that they have no conflict of interest. 

Open Access This article is distributed under the tenns of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses!by/4.0/), which permits unrestricted use, distribution, 

and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, 

provide a link to the Creative Commons license, and indicate if changes were made. 

References 

Boykoff MT (2014) Media discourse on the climate slowdown. Nat Clim Chang 4:156-158. 

doi: 10.1 038/nclimate2156 

Chen X, Tung KK (2014) Varying planetary heat sink led to global-wanning slowdown and acceleration. 

Science 345(6199):897-903. doi:lO.ll26/science.l254937 

Cochrane D, Orcutt G (1949) Application of least squares regression to relationships containing autocorrelated error terms. JAm Stat Assoc 44(245):32-<il 

Cowtan K, Way RG (2014) Coverage bias in the HadCRUT4 temperature series and its impact on recent 

temperature trends. Q J R Meteorol Soc. doi: I 0.1 002fqj.2297 

i} Springer 
174 

140 Climatic Change (2015) 133:129-140 

Cowtan K, Hausfather Z, Hawkins E, Jacobs P, Mann ME, Miller SK, Steinman BA, Stolpe MB, Way RG 

(2015) Robust comparison of climate models with observations using blended land air and ocean sea 

surface temperatures. Geophys Res Lett (in print) 42. doi: 10.1002/2015GL064888 

Durack PJ, GleckJer PJ, Landerer FW, Taylor KE (2014) Quantifying underestimates of long-term upperocean warming. Nat Clim Chang 4:999-1005. doi:IO.l038/nclimate2389 

England MH, McGregor S, Spence P, Meehl GA, Timmermann A, Cai W, Gupta AS, McPhaden MJ, Purich 

A, Santoso A (2014) Recent intensification of wind-driven circulation in the pacific and the ongoing 

warming hiatus. Nat Clim Chang 4:222-227. doi:IO.I038/nclimate2106 

Fyfe JC, Gillett NP, Zwiers FW (2013) Overestimated global warming over the past 20 years. Nat Clim 

Chang 3:767-769 

Guemas V, Doblas-Reyes FJ, Andreu-Burillo I, Asif M (2013) Retrospective prediction of the global 

warming slowdown in the past decade. Nat Clim Chang. doi: I 0.1 038/nclimate 1863 

Hansen J, Ruedy R, Sato M, Lo K (2010) Global surface temperature change. Rev Geophys 48(RG4004). 

doi: l 0.1 029/201 ORG000345 

Hawkins E, Edwards T, McNeal! D (2014) Pause for thought. Nat Clim Chang 4:154-156. doi:IO.I038/ncli 

mate2150 

IPCC (2013) Climate Change 2013: the physical science basis. In: Stocker TF, Qin D, Plattner G-K, Tignor M, AJlen SK, Boschung J, Nauels A, Xia Y, Bex V, Midgley PM (eds) Contribution of Working 

Group I to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge 

University Press, Cambridge 

Karl TR, Arguez A, Huang B, Lawrimore JH, McMahon JR, Menne MJ, Peterson TC, Vase RS, Zhang 

HM (2015) Possible artifacts of data biases in the recent global surface warming hiatus. Science 

348(6242): 1469-1472. doi: 10.1126/science.aaa5632 

Kosaka Y, Xie SP (2013) Recent global-warming hiatus tied to equatorial pacific surface cooling. Nature. 

doi: I 0.1 038/nature 12534 

Leber R (2014) We're stuck with the global warming "hiatus" myth for years to come. New Republic. http:// 

www.newrepu blic.com/article/119212/study -atlantic-ocean-role-global-warming- hiatus 

Meehl GA, Arblaster JM, Fasullo JT, Hu A, Trenberth KE (2011) Model-based evidence of deep-ocean 

heat uptake during surface-temperature hiatus periods. Nat Clim Chang 1:360-364. doi:IO.l038/ncli 

mate1229 

Morice CP, Kennedy JJ, Rayner NA, Jones PO (2012) Quantifying uncertainties in glohal and regional temperature change using an ensemble of observational estimates: The HadCRUT4 data set. J Geophys Res 

117:008,101. doi: 10.1029/201JJD017187 

Otto A, Otto PEL, Boucher 0, Church J, Heger] G, Forster PM, Gillett NP, Gregory J, Johnson GC, Knutti 

R, Lewis N, Lohmann U, Marotzke J, Myhre G, Shindell D, Stevens B, Allen MR (2013) Energy budget 

constraints on climate response. Nat Geosci 6:415-416. doi:IO.l038/ngeo 1836 

Politis ON, Romano JP, Wolf M (1999) Subsampling. Springer, New York 

Rajaratnam B, Romano J, Tsiang M (2014) Fixed b subsampling with application to testing for a change. 

Technical Report, Department of Statistics, Stanford University 

Santer BD, Bonfils C, Painter JF, Zelinka MD, Mears C, Solomon S, Schmidt GA, Fyfe JC, Cole JNS, 

Nazarenko L, Taylor KE, Wentz FJ (2014) Volcanic contribution to decadal changes in tropospheric 

temperature. Nat Geosci. doi:10.1038/ngeo2098 

Smith D (2013) Oceanography: has global warming stalled? Nat Clim Chang 3:618-619 

Stover D (2014) The global warming "hiatus". Bulletin of the Atomic Scientists. 

http://thebulletin.org/global-warming-%E2%80%9Chiatus%E2%80%9D7639 

Tollefson J (2014) The case of the missing heat. Nature 505:276-278 

Trenberth KE, Fasullo JT (2013) An apparent hiatus in global warming? Earth's Future 1:19-32. 

doi: 10.!002/2013EF000165 

~Springer 
175 

THE "PAUSE" IN GLOBAL 

WARMING 

Turning a Routine Fluctuation into a Problem for Science 

BY STEPHAN LEWANDOWSKY, jAMES 5. RISBEY, AND NAOMI 0RESKES 

M

any indicators confirm that Earth continues to 

warm from greenhouse gases (Abraham et al. 

2013; Bahnaseda eta!. 2013; Durack et al. 2014). 

Notwithstanding, climate contrarians have been 

claiming for nearly a decade that global warming has 

"stopped" (Carter 2006). Boykoff (2014) showed how, 

over time, those repeated contrarian claims entered 

the discourse in the media and among policy makers 

and politicians. In consequence, climate change has 

frequently been framed around the presumed fact 

AMERICAN METEOROl OG!CAL SOCIETY 

that global warming-measured by global mean surface temperature (GMST)-has "stalled," "stopped," 

"paused," or entered a "hiatus." Evidence for the widespread adoption of this frame is provided by a Coogle 

Trends analysis (conducted on 21 October 2014), 

which reveals that the search term "global warming 

stopped" has been used nearly continuously since 

February 2008, with distinct spikes ahead of the climate meetings in Copenhagen, Den mark (December 

2009), and Doha, Qatar (November 2012). 

This frame has also found explicit uptake in the 

peer-reviewed literature, with two special issues of 

Nature journals devoted to the "pause" or "hiatus" in 

early 2014, and a total of more than 40 articles having 

appeared in print on the pause by 2014. Moreover, 

the Intergovernmental Panel on Climate Change 

(IPCC), which reflects the scientific consensus on 

climate change, adopted the term hiatus in its Fifth 

Assessment Report, and even gave it a definition "as 

the reduction in GMST trend during 1998-2012 as 

compared to the trend during 1951-2012" (IPCC 

2013, Box TS.3). 

Is there a meaningful pause or hiatus in global 

warming? If not, what has caused the scientific community to devote such intense activity to analyzing 

something that does not exist? This article presents 

MAY2Dib BAn> 1 m 
176 

evidence that there has been no meaningful pause 

in global warming and offers an account of why this 

notion has become so widespread in the scientific 

community. 

There has been ongoing analysis and commentary 

arguing against the existence of a statistically meaningful pause for several years (e.g., Foster and Rahmstorf2011). Two analyses of the GMST time series have 

failed to find any statistical evidence for a slowdown 

(Foster and Abraham 2015), or a distinct changepoint 

in the rate of warming (Cahill et al. 2015). There have 

also been questions about biases in some datasets 

used to identify a potential pause (Cowtan and Way 

2014; Karl ct a!. 2015). Most recently, a bias-corrected 

release of the National Oceanic and Atmospheric 

Administration's (NOAA) National Centers for Environmental Information (NCEI) dataset (Karl eta!. 

2015) assessed the rate of warming during the hiatus 

period identified by the Intergovernmental Panel on 

Climate Change (IPCC; 1998-20I2) to differ little 

from the longer-term trend considered by the IPCC 

for comparison (1951-2012). Although those bias corrections were unavailable at the time when the pause 

gained entry into the literature, we show below that 

our conclusions do not depend on those corrections. 

Accordingly, there are other indications of longstanding disquiet with the presumed pause. For example, the IPCC's use of the term hiatus (without scare 

quotes) came under critical scrutiny during review of 

the Fifth Assessment Report. In a high-priority comment on the Summary for Policy Makers, the German 

government noted that the term hiatus was strongly misleading and recommended against its usc.' Although 

the German delegation's suggestion was not adopted, 

it points to a fundamental problem surrounding the 

pause: what exactly is meant by a pause or hiatus? 

1 The full comment reads as follows: "the underlying report 

and the TS label the recent reduction in surface warm~ 

ing as 'hiatus'. The web site http://thesaurus.com gives as 

definition of this expression 'pause, interruption', www 

.merriam~webster.com gives 'la: a break in or as if in a 

material object, 2a: an interruption in time or continuity; 

break; especially: a period when something (as a program or 

activity) is suspended or interrupted.' All these definitions 

do not appropriately describe the recent temperature evolution as there is not a pause or interruption, but a decrease 

in the warming trend, and the first decade of this century 

has been the warmest since preindustrial times, see Figure 

SPMl. (a), lower figure, Hence, the expression 'hiatus' is 

strongly misleading and should not be used throughout 

the report" (www.climatechange2013.orgfimages/report 

IWGIAR5 _FGD _Final DraftS PM Comments .pdf). 

m 1 BAn~ MAnol6 

WHAT IS A PAUSE? By definition, a "pause" involves the interruption or suspension of a process. The 

presence of a pause or hiatus in global warming would 

thus mean what contrarians say it means (e.g., Carter 

2006), namely, that warming had stopped, at least for 

a time. Determining whether warming has stopped is 

nontrivial because greenhouse-driven global warming 

is expressed on multidecadal and longer time scales 

(i.e., 30 yr and longer), whereas on shorter time scales 

(10-20 yr) the rate of warming speeds up and slows 

down relative to the longer-term average trend (IPCC 

1996; Risbey 2015). At one point or another, there may 

therefore be periods oflimited duration during which 

surface temperatures do not increase significantly. 

In this article, we consider the period since 1970 

to provide a representation of the "longer term" rate 

of greenhouse warming that is characteristic of the 

modern period. The choice of period marking the 

longer-term trend is necessarily somewhat arbitrary. 

The year 1970 has been statistically identified as an 

approximate marker of an upsurge in the rate of 

global warming on multidecadal time scales (Cahill 

et al. 2015). This longer-term trend (1970-2014) has 

been estimated at 0.17 K decade-1 (Cowtan and Way 

2014; Karl et al. 2015) or 0.16 K decade-' [National 

Aeronautics and Space Administration (NASA) Goddard Institute for Space Studies Surface (GISS) Temperature Analysis (GISTEMP; Hansen et al. 2010) and 

the Met Office's Hadley Centre/Climatic Research 

Unit, version 4 (HadCRUT4; Morice et al. 2012)]. 

By contrast, we refer to decadal scale (10-20 yr) 

variations in the rate of warming about a longerterm trend as .fluctuations. Here, we focus on 15-yr 

trends to reflect the duration of the presumed hiatus 

employed by the IPCC (2013, Box TS.3). Those fluctuations may be driven by internal variability (ocean 

circulation and its coupling to the atmosphere), or 

they may involve variations in external forcings of 

the climate system (such as solar irradiance and 

aerosol concentrations), or both. These fluctuations 

are "routine'' in the sense that they occur commonly 

and are caused by different combinations of the same 

set of processes. 

A given fluctuation is defined by a start year and 

an end year, and its magnitude is highly dependent 

on the choice of start and end years. For example, 

the decadal rate of warming during the 15 yr centered around 2005 was 0.11 K; for the 15 yr centered 

on I999 it was 0.31 K-a nearly threefold difference 

resulting from a shift of the temporal window by 6 yr 

[data from Cowtan and Way (2014)]. Fluctuations 

can therefore display warming rates that are greater 

than or less than the greenhouse-driven longer-term 
177 

trend. There may also be short-term periods of cooling embedded within a longer-term warming trend 

(Easterling and Wehner 2009). 

Any claims of a pause or hiatus in the recent rate 

of warming must therefore be assessed against the 

overall pattern of fluctuations in the temperature 

record. A claim to find a pause or hiatus is a different assertion than a fluctuation and implies that the 

fluctuation is extraordinary in a particular way: the 

meaning of the terms pause and hiatus implies that 

the normal fluctuations in warming rate have been 

surpassed such that warming has stopped. We next 

show that no such stoppage has occurred. 

GLOBAL WARMING CONTINUES: THE 

STATISTICAL EVIDENCE. Detailed analyses 

of temperature trends have been reported previously 

(Easterling and Wehner 2009; Santer et al. 2011; Karl 

et al. 2015). Here, we generalize and update those 

results. The top panel in Fig. 1 shows all possible 

15-yr trends in GMST for the period 1970-2014 (i.e., 

1970-84, 1971-85, and so on; N = 31) for four different 

datasets. It is clear that the short-term trend sometimes falls above the longer-term trend (indicated by 

the gray band) and sometimes below it. It is also clear 

that warming has continued throughout the 45 yr as 

none of the trends are zero (dashed horizontal line). 

The linear trend in GMST (established by ordinary 

least squares on annual global means) is statistically 

significant for the last 15-yr period (ending in 2014) 

for three of the four available datasets: G!SS (trend, 

b = 0.08 K decade·•; test statistic, t = 2.20; level of 

significance, p < 0.05), the dataset of Cowtan and Way 

(b = 0.10 K decade· I, t = 2.41, p < 0.05), and the most recent NOAA dataset by Karl et al. (b = 0.11 K decade·', 

t= 3.25, p < 0.007). Only HadCRUT4, which does not 

cover parts of the Arctic where warming is known 

to be most rapid, fails to yield a significant trend 

for this 15-yr period (b = 0.07 K decade·', t 1.70, 

p > 0.10). When a further year is included in the 

analysis, HadCRUT4, too, yields a significant trend 

(b = 0.09 K decade·', t = 2.48, p < 0.03). 

Although the most recent 15-yr trend is significant 

for most datasets~ there have been six occasions since 

1970 when a 15-yr trend would have failed to reach 

significance (using G!SS); namely, in the years 1986, 

1993, 1994, 1995, 2011, and 2012. At all those times, 

the preceding 15 yr failed to show significant warming. And at all those times, the inclusion of further 

years renders the trend significant. The claim that 

global warming uniquely "stopped" during any recent 

15-yr period is therefore not sustainable. Conversely, 

any argument about a pause, hiatus, or stoppage 

AMERICAN METEOROLOGICAL SOCIETY 

0.3 

g 0.2 c 

~ 

"' ro 0.1 

0 

~ 0.0 

1980 1985 1990 1995 2000 2005 

c 2.5 - Cowtan&Way 

~ G1SS 

"' 2.0 HadCRUT4 

'0 ... NOM 

f 

1.5 

1.0 

~ 0.5 

~ 0.0 

1980 1985 1990 1995 2000 2005 

Mid-Year of 15-year window 

F1c. I. Summary of all possible 15-yr trends in GMST 

between 1970 and 2014 inclusive. (top) The trend 

(K decade'1

) for the I 5-yr window centered on the 

plotted year for four datasets: NASA's GISS (Hansen 

et at. 2010; http://data.giss.nasa.gov/gistemp/, accessed 17 

Jan 2015), the Met Office's HadCRUT4 (Morice et al. 

2012; www.metoffice.gov.uk/hadobslhadcrut4/data/current 

/time_series/HadCRUT.4.3.0.0.annual_ns_avg.txt, accessed 

2 Feb 2015), the coverage-bias~corrected version of 

HadCRUT4 reported by Cowtan and Way (2014) (http:// 

www-us ers .yor k.ac.u k/-kdc3/pape rs/cover age20 t 3/series 

.html, accessed 2 Feb 2015), and the latest NOAA dataset (Karl et at lOIS; www.ncdc.noaa.gov/cag/time-series 

/global/globelland_oceanlytdll2/1880-2014.csv, accessed 12 

Aug 2015). The GISS dataset is based on sea surface 

temperature data [Extended Reconstructed SST version lb (ERSSTvlb)]. The decadal temperature increase 

is greater than zero (dashed horizontal line) in all datasets at all times. The gray horizontal band represents the 

average of the trends between 1970 and 2014 across the 

four datasets. The longer-term trend is represented as 

a band to capture some of the uncertainty from dataset 

to dataset, but also to indicate that this is an inherently 

imprecise quantity because it varies with the exact pew 

riod that is chosen to represent a longer-term trend. 

(bottom) The same data as in the top panel, but 15-yr 

trends are converted into absolute z scores, by expressw 

ing each observed trend as the absolute difference in 

standard deviation units from the mean of all trends 

since 1970. Originally positive z scores (representing 

greater than average warming) are plotted in red, and 

originally negative z scores are shown in blue. 

could have been made with equal justification (or lack 

thereof) repeatedly during the past 45 yr. 

Nor does the most recent fluctuation constitute a 

uniquely large deviation from Lhe longt::r-tt::rm trt::nd. 

MAY2ol6 BAni· 1 m 
178 

This is shown in the bottom panel of Fig. I, which plots 

the same 15-yr trends but converted into absolute z 

scores. The advantage of z scores is that they reexpress 

each data point as a deviation from the overall mean 

of a sample in units of standard deviation, thereby 

providing an indication of the extremity of the observations. To compute z scores, the mean of all possible trends was first subtracted from each individual 

trend, and each such difference was in turn divided 

by the standard deviation of all trends. To permit a 

comparison of decelerating (z < 0) and accelerating 

(z > 0) fluctuations, the z scores were converted to absolute values for plotting. For clarity, z scores that were 

originally negative are plotted in blue in Fig. I, and 

those that were originally positive are shown in red. 

For a pause to be distinctive, it must deviate below 

the longer-term trend more than previous periods 

deviated above the longer-term trend; otherwise, it 

can be considered to be just a fluctuation like oth· 

ers observed in the past. The bottom panel in Fig. 

1 shows that this criterion for distinctiveness is not 

met: for all datasets bar HadCRUT4, the pause is less 

anomalous than the accelerated period of warming 

that took place during the 15 yr spanning 1999 (i.e., 

1992-2006). That is, the absolute magnitudes of 

the z scores associated with the recent deceleration 

(whichever recent year is picked as the point on which 

the pause is centered) are consistently smaller-some· 

times by a considerable margin-than those for the 

1999 acceleration. Only for HadCRUT4, and only for 

the 15-yr period centered on 2005, are the z scores 

for the pause and the maximum warming virtually 

indistinguishable (1.86 vs -1.90). 

Taken together, the statistical evidence presented 

here and elsewhere (Cahill et al. 2015; Foster and 

Abraham 2015) shows that the pause period is 

comparable in statistical terms with other recent 

fluctuations. Any exceedance of the z score of the 

pause period compared to other fluctuations, if it ex· 

ists, is marginal and depends on the details of which 

dataset is used and precisely what time window is 

used to assess the pause. The pause is not unusual 

or extraordinary relative to other fluctuations and it 

does not stand out in any meaningful statistical sense. 

Note that these conclusions are not dependent on 

the choice of baselines used to represent longer-term 

greenhouse warming. For example, a longer baseline 

such as the IPCC's 1951-2012 period yields a lower 

longer-term trend, thus rendering any fluctuations 

with slower rates of warming even less unusual. Our 

conclusions are also qualitatively unaffected by the 

modeling of auto correlations and by the choice of 

window size for the short-term trend. 

726 I BAM~ MAY20 16 

We next show that experts fail to detect evidence 

for a pause in a blind test. 

GLOBAL WARMING CONTINUES: THE 

BLIND EXPERT TEST. The forecasting of time 

series data is central not only to climatology, but 

also to economics, finance, and allied disciplines. 

Forecasting techniques have therefore attracted 

considerable research attention, and the last 25 years 

have seen a striking reevaluation of the role of human 

judgment in forecasting. Whereas human judgment 

used to be given little if any credence in forecasting, 

today it is "recognised as an indispensable component 

offorecasting" (Lawrence et al. 2006, p. 493). 

People are known to be able to learn smooth 

functions with considerable precision (DeLosh et aL 

1997; Lewandowsky et al. 2002), People are also able 

to extract information from noisy data presented in 

graphical form (Lewandowsky and Spence 1989). In 

forecasting studies, participants across a broad range 

of expertise are now generally thought to perform 

well (Harvey and Bolger 1996; Harvey et al. 1997; Du 

and Budescu 2007), and domain experts outperform 

statistical models in some circumstances (Forrest et al. 

2005), although this is becoming increasingly less com· 

man in weather forecasting (Baars and Mass 2005). 

Here, we are interested in human forecasting not 

because people's predictions might constitute a viable 

alternative to the projections of climate models, but 

because forecasting judgments reveal people's percep· 

tions ofthe trend in a dataset. People's extrapolations 

of visually presented temperature data can therefore 

reveal whether people believe that global warming 

has stopped. 

To assess the claim that global warming has indeed 

stopped, Lewandowsky (2011) presented naive par· 

ticipants with a graph of the historical temperature 

record, which either identified the data as global tern· 

peratures or as a fictitious share price. Figure 2 shows 

the results ofLewandowsky (2011) for the condition in 

which the data were identified as global temperatures. 

Respondents clearly did not perceive a pause or hiatus 

in the GMST data,' as revealed by the fact that their 

extrapolations (large squared plotting symbols) had a 

statistically significant positive slope. Extrapolations 

did not differ notably from a condition (not shown in 

the figure) in which the stimulus data were presented 

as fictitious share prices. In the eyes of naive observ· 

ers, therefore, global warming has not stopped but is 

2 It must be noted that at the time of the study, the time series 

ended in 2009. However, at that time the idea of a pause had 

already been established in contrarian discourse. 
179 

set to continue. People's extrapolations were, however, 

conservative, falling consistently below the linear 

extrapolation of the long-term trend. The tendency to 

underestimate a long-term trend is a well-established 

phenomenon in judgmental forecasting known as 

trend damping (Harvey and Bolger 1996). This observation merits further exploration because it raises 

the possibility that people are overly sensitive to any 

slowing in warming. 

This possibility was explored in a blind test involving professional economists, who were asked specifically to comment on the presence of a pause or hiatus 

in GMST. The sample of economists (N = 25) was 

tested online and was recruited by a survey firm (Qualtrics.com). All experts held at least a master's degree or 

a Ph.D. in economics or an allied discipline, with all 

but four experts reporting five or more years of professional experience. Participants were shown the GMST 

data through 2010, but presented as "world agricultural 

output" (see Fig. 3). The graph was accompanied by 

the following statement that experts had to evaluate in 

light ofthe plotted data: "A prominent Australian critic 

of conventional economics, Mr. X., publicly stated in 

2006, that 'There IS a problem with the growth in 

world agricultural output-it stopped in 1998.' A few 

months ago, Mr. X. reiterated that ' ... there's no trend, 

2010 is not significantly more productive in any way 

than 1998: " This statement is an exact translation, 

into the economic terms of world agricultural output, 

of a series of public statements about the putative pause 

or stoppage of global warming (Carter 2006, 2011). 

The experts responded to six test items, which are 

shown in Table 1.3 Table I also shows the responses 

of the experts on a six-point scale that ranged from 

"strongly disagree" (I) to "strongly agree" (6). Any 

mean response above 3.5 therefore represents agreement. and any mean response below 3.5 indicates 

disagreement, respectively, with the test item (there 

was no "neutral'' response category). It is clear that 

the experts disagreed with the invocation of a pause: 

experts rejected the idea that the data confirm the 

statement and instead find that the data contradict the 

statement. The experts also found the statement to be 

misleading and ill-informed. The experts were divided 

on whether or not the statement is fraudulent, although 

nearly 'h of them endorsed that possibility as welL The 

experts were also divided on whether the statement 

might be compatible with the data in a "narrow sense.'' 

3 The experiment involved additional statements by contrarians, pertaining to other climate variables, such as glaciers 

and Arctic ice, thal are not relevant to the present article and 

are not reported here. 

AMERICAN METEOROLOGICAL SOCIETY 

100 

80 

- 60 

! 40 

f 20 

~ 

• 

1-20 

!'-

-40 

-60 

100 

80 

60 

40 

20 

-20 

-40 

-80 

1sao 1900 1920 1940 1960 1980 2000 2020 

Yaar 

FIG. 2. Stimuli and data from an experiment by 

Lewandowsky (2011). Gray circles show actual global 

mean land surface air temperature anomalies from 

1880 to 2009. Extrapolations of the trend by the respondents are represented by large red squares. When 

the graph was presented as a stimulus, the three ques .. 

tions marks(!) at the top identified the three columns 

in which participants marked their predictions. Tern .. 

perature data are from the GISTEMP (http://data.giss 

.nasa.govfgistemp/, accessed 4 Feb 20 I 0; see also 

Hansen et al. 2010). 

These results from our experiment are consistent 

with an earlier informal study conducted by the Associated Press with a small sample of statisticians who 

were blind to the data source (Borenstein 2009). Those 

experts) too, saw no evidence for a decline in the temperature trend and instead decried the cherry-picking 

of observations on which that claim was based. 

In summary, in two blind tests, experts and novice 

observers alike consider the evidence of continued 

global warming to be clear. By contrast, statements 

endorsing the pause were identified by experts in 

forecasting and time series analysis to be misleading 

and at odds with the data. 

WHERE DID THE PAUSE COME FROM? Our 

preceding analyses show that the entrenchment ofthe 

pause concept in the literature is incommensurate 

with the lack of evidence supporting it, and that it 

does not pass a blind expert test. Despite that, large 

segments of the climate science community, including 

the IPCC (2013, Box TS.3), have adopted the notion 

of a pause or hiatus in global warming. 

This is not to say that interpretations of the pause 

are entirely uniform. A few articles addressing the 

pause question its existence. For example, Seneviratne 

MAY2016 BAni· I 721 
180 

---------- .. .. ;;; 

! 50 !-._,.... ............ , 50 

~ 40 .., 

~ 

1 "" 30 

6 

i "' "' 

I 10 ,. 

1880 1000 1920 f940 1960 1980 2000 

FIG. 3. Stimulus data shown to expert economists in a 

blind test of contrarian statements invoking the pause. 

See text for details. Data are actually global land-sea 

surface temperature anomalies from the GISTEMP 

dataset (http://data.giss.nasa.gov/gistempl, accessed 

3 Mar 20 II; see also Hansen et al. 20 I 0). 

et al. (2014) call the term misleading and conclude 

that "not only is there no pause in the evolution of 

the warmest daily extremes over land but. .. they 

have continued unabated over the observational 

record" (p. 163). Risbey et al. (2014) show that recent 

fluctuations are not unusual and do not constitute 

meaningful evidence against climate model projections. Santer et al. (2014) refer to the pause or hiatus 

in quotation marks (i.e., scare quotes), thereby implying skepticism or disagreement with the phrase. 

However, the majority of the more than 40 articles on 

the pause that we know of start from the premise that 

the pause is meaningful, and present it as a significant development requiring explanation. Moreover, 

some researchers (albeit a minority) have taken the 

pause to imply that the climate system may be Jess 

sensitive to greenhouse gas emissions than previously 

thought (Otto et al. 2013; Curry 2014). But any use of 

the term-except in a clearly refutative context-is 

problematic because it reinforces, both in scientific 

and public debate, the belief that there has been a 

statistically meaningful cessation of warming when 

there has not. 

How did this occur? We have shown in detail elsewhere 

(Lewandowsky et al. 2015) that there are several psychological and cognitive reasons why climate scientists may have been susceptible to the meme of a pause 

728 I BAnl· MAY2016 

that demonstrably originated in contrarian discourse 

on the Internet and in the media (Boykoff2014). Here, 

we suggest that a contrarian me me can find entry into 

the scientific community simply by exploiting scientists' commitment to explanation and to responding 

to intellectual challenges. Scientists generally strive 

to emphasize factual information and deemphasize 

value judgments. Indeed, "disinterestedness" has long 

been identified as one of the core norms of science 

(Merton 1942). 

In a world in which contrarian claims in the media 

and other public arenas are overrepresented (Boykoff 

and Boykoff2004; Elsasser and Dunlap 2013; Boykoff 

2013), scientists may feel the need to respond to these 

claims. This may occur informally, as when friends, 

neighbors, or family members ask questions about 

contrarian claims they encountered online, or formally, when journalists, editors, or policy makers 

seek answers to contrarian talking points. If these 

encounters involve loaded questions, such as <<What 

about the ·pause' in warming?," then climate scientists 

may inadvertently accept the biasing terms in which 

those questions are framed. 

Frames are rhetorical and communicative structures that select and highlight certain aspects of a 

perceived reality over others (Dirikx and Gelders 

2010). Because frames are rarely made explicit-for 

example, few people know that the use of the term 

"climate change" rather than "global warming" was 

advocated by Republican strategist Frank Luntz 

(Mooney 2005; Lakoff 2010)-frames can shape in 

a hidden manner the way in which people discuss 

an issue (de Boer et al. 2010). Would voters be more 

likely to support a price on carbon if it were framed 

as an "additional tax burden,') "insurance premium 

for your grandchildren's well-being," or "putting a 

fair price on the true cost of oil and gas?" Even simple 

choices of wording, such as «tax" versus "offset" can 

have large effects on people's endorsement of policy 

options (Hardisty et al. 2010). 

Simply by being exposed to the pause meme for 

over a decade, and by explaining short-term fluctuations from a longer-term trend in the terms posed to 

them, scientists have accepted a contrarian frame, 

and this acceptance may in turn have subtly changed 

scientists' way of thinking (Lewandowsky et al. 2015). 

To illustrate, we provide citations from some recent articles on the pause in Table 2. None of those 

articles questioned the fundamental fact that Earth is 

warming from greenhouse gas emissions, and some 

authors even underscored the likelihood of future 

warming, for example by suggesting that the "present 

hiatus will be short-lived" with "rapid warming set to 
181 

TAIIIJ! f;: Test Items and ...,.,....,,.,~given by expert economists to contrarian statements endorsing 

tile pa.111e<that -re evaluated in lltlht of the data. 

Test item Agreement' Meanb t< p' 

The data -confirm the claim made by Mr. X. 0.36 2.84 -2.72 <0.02 

The data contradict the claim made by Mr. X. 0<68 4.12 2<58 <0<02 

The claim made about the data by Mr. X is misleading. 0.76 4.28 3.67 <0.002 

The claim made about the data by Mr. X is ill·informed. 0.76 4.04 2.38 <0.03 

If incompetence is ruled out, the claim made about the 0.64 3.84 1.49 n.s. 

data by Mr. X is fraudulent. 

The statement by Mr. X is compatible with the data in a 0.52 3.60 0.34 n.s. 

narrow sense, but the data do not support the implication of his statement, which is that world agricultural 

output is no longer growing. 

Proportion of experts out of 25 who agreed (ratmg > 3) w1th the test Item. 

b Mean response on the six-point scale. Any value> 3.5 represents agreement. 

c Single-sample t statistic (dj::::: 24) comparing the mean response to the null hypothesis that the mean is equal to 3.5 (neutrality on the 

six-point scale). 

d The p value of the t test in the previous column: n.s. means nonsignificant. 

resume" once the present decadal variation comes to 

an end (England eta!. 2014, p. 225). Nonetheless, the 

majority of articles accepted the framing of a pause 

and sought to explain its cause. Furthermore, the 

citations in Table 2-typically from the opening paragraph of an article-show that authors often framed 

the article by juxtaposing the continuing increase of 

atmospheric C02 levels with the presumed lack of 

warming on a decadal scale as though this presented 

a notable scientific problem at odds with expectations 

from greenhouse theory. 

The statements in Table 2-and similar but often tacit implications of many other articles-are 

at variance with long-established knowledge that 

muJtidecadal natural variations in climate are superimposed on a longer-term C02 warming trend. 

These variations demonstrate that whereas C02 may 

increase year after year, surface temperature need 

not. More than 20 years ago, the JPCC's Second 

Assessment Report pointed to the importance of 

decadal and longer time-scale variability (IPCC 1996, 

329-330), as did a U.S. National Research Council 

report (Martinson 1995). The JPCC summary for 

policy makers in the 1995 report cautioned that future 

decadal-scale changes would include considerable 

natural variability despite the longer-term warming. 

If this knowledge had been foremost on scientists' 

minds, rather than the contrarian pause meme, the 

framing of many recent research articles arguably 

would have been different. Instead of opening an article with "Despite ongoing increases in atmospheric 

greenhouse gases, the Earth's global average surface 

AMERICAN METEOROLOGICAL SOCIETY 

air temperature has remained more or less steady 

since 2001," we suggest that scientists might have 

adopted a more appropriate framing such as "It has 

long been known that the longer-term greenhouse 

warming trend is punctuated with decadal and longer 

fluctuations. In this article we show that the most 

recent fluctuation during which warming fell below 

the longer-term trend was due to ... '' 

THE MERITS OF RESEARCH ON THE 

PAUSE. The body of work on fluctuations in warming rate has clearly contributed to our understanding 

of decadal variations in climate. For example, studies have shown that the negative radiative forcing 

from stratospheric loadings of volcanic aerosol has 

increased in recent years and is larger than previously 

thought (Solomon eta!. 20ll; Neely eta!. 2013; Ridley 

eta!. 2014; Santer eta!. 2014). Research has also highlighted processes whereby the ocean can vary the rate 

at which heat is taken up from the surface (Kosaka 

and Xie 2013; England eta!. 2014). 

Research on decadal fluctuations has also highlighted differences in expectations between climate 

projections that tend to average out decadal variations and the actual transient response of the climate 

system (Schneider and Thompson 1981) that includes 

such variation. Research has shown that differences 

in expectation between averages of projections and 

the actual transient response are related to modelversus-observed differences in the phasing ofinternal 

variability (Meehl and Teng 2014; Risbey eta!. 2014), 

systematic errors in some of the external forcings used 

MAY2ol6 BAns- 1 m 
182 

TABLE l. Representative quotations from peer-reviewed articles that frame the pause or hiatus as a 

problem for climate science. 

Quotation Source 

"Reconstructions of global mean surface temperature [Hansen eta!. 2010; (Drijfhout et al. 2014, p. 7868) 

Morice et aL 2012] show rising values after the 1960s but a slowing of the 

warming in the 2000s, even though atmospheric greenhouse gas concentraM 

tions continued to increase. This hiatus in warming may have been exagger~ 

ated by sampling errors [Cowtan and Way 2014], but a significant slowdown 

is evident." 

"Despite ongoing increases in atmospheric greenhouse gases, the Earth's (England et al. 2014, p. 222) 

global average surface air temperature has remained more or less steady 

since 2001.'' 

"The warming of the climate system is unequivocal as evidenced by an increase (Estrada et al. 2013, p. 1050) 

in global tempe raw res by 0.8°C over the past century. However, the attribuM 

tion of the observed warming to human activities remains less clear, particu~ 

larly because of the apparent slow~down in warming since the late 1990s." 

"Despite a sustained production of anthropogenic greenhouse gases, the (Guemas et al. 2013, p. 649) 

Earth's mean near-surface temperature paused its rise during the 2000-

2010 period." 

"Given the widely noted increase in the warming effects of rising greenM (Kaufmann et al. 2011, p. 11,790) 

house gas concentrations, it has been unclear why global surface temperatures did not rise between 1998 and 2008." 

"Despite the continued increase in atmospheric greenhouse gas concentra- (Kosaka and Xie 2013, p. 403) 

tions, the annuaiRmean global temperature has not risen in the twenty-first 

century, challenging the prevailing view that anthropogenic forcing causes 

climate warming." 

"Despite continued growth in atmospheric levels of greenhouse gases, (Santer et al. 2014, p. 185) 

global mean surface and tropospheric temperatures have shown slower 

warming since 1998 than previously." 

"Despite increasing radiative forcing, the observed globally averaged annual (Sillmann et al. 2014, p. I) 

mean surface temperature (Tmean) has only increased very slowly since the 

late 1990s (e.g., IPCC AR.5 2013)." 

in Coupled Model lntercomparison Project phase 5 

(CMIPS) simulations (Fyfe et al. 2013; Schmidt et al. 

20!4), incomplete coverage and quality of observations 

(Karl et al. 2015), and use of incommensurate measures 

between models and observations (Cowtan et al. 20!5). 

the notion of a pause (with all the connotations of that 

term) in the literature as well as in the public's mind. 

In addition, the statistical properties of many 

different examples of decelerating fluctuations are 

very similar in observations and in models (Risbey 

et al. 2014; England et al. 2015; Marotzke and Forster 

2015). Other research has highlighted that there will 

be similar fluctuations (in both directions; faster as 

well as slower warming) in the future, a point about 

which policy makers perhaps need to be reminded 

(Easterling and Wehner 2009; Hawkins et al. 2014; 

England et a!, 2015). 

Research on the pause has thus ultimately reaffirmed the overall reliability of climate models for projecting temperature trends. However, by accepting the 

framing of a recent fluctuation as a pause or hiatus, that 

research has, ironicaUy and unwittingly, entrenched 

no 1 BAni- MAna 16 

PUTTING THE PAUSE TO FUTURE 

"PAUSES." To avoid misframing in the future does 

not mean that scientists should necessarily avoid an 

issue simply because it has gathered public prominence or is being used by contrarians. Scientists have 

previously responded to contrarian memes with success, for example by showing that appeals to the sun 

or galactic cosmic rays fail to explain global warming 

(Benestad 20!3; Sloan and Wolfendale 20!3). Concerning the recent fluctuation, we have shown that its 

framing as a pause or hiatus that constitutes a problem for greenhouse warming is incorrect, because it is 

not meaningfully different from other fluctuations in 

warming rate. If the fluctuation were instead framed 

as an instance of decadal variation, then scientists 

would be able to put the pause to misleading contrarian claims that global warming has stopped. 
183 

It bears remembering that the point of contrarian 

memes is to "keep the controversy alive" (Oreskes and 

Conway 2010). Accepting contrarian linguistic frames 

helps maintain the fiction that the science is still too 

uncertain to form a reliable basis for public policy. 

Moreover, it should be noted that the remaining uncertainties often provide a greater, rather than lesser, 

impetus for mitigation (Lewandowsky et al. 2014a,b). 

ACKNOWLEDGMENTS. Preparation of this paper 

was facilitated by a Wolfson Research Merit Award from 

the Royal Society to SL and by Linkage Grant LPI20100224 

from the Australian Research Council to Ben R. Newell and 

SL. JSR is supported by CSIRO, ACCSP, and GRDC. We 

thank Richard Somerville for comments on an earlier draft 

of this article that substantially improved the final product. 

REfERENCES 

Abraham, J. P., and Coauthors, 2013: A review of global 

ocean temperature observations: Implications for 

ocean heat content estimates and climate change. 

Rev. Geophys., 51,450-483, doi:IO.l002/rog.20022. 

Baars, J. A.1 and C. F. Mass, 2005: Performance ofNational Weather Service forecasts compared to operational, 

consensus, and weighted model output statistics. Wea. 

Forecasting, 20, !034-!047, doi:!O.li75/WAF896.l. 

Balmaseda, M.A., K. E. Trenberth, and E. Kallen, 

2013: Distinctive climate signals in reanalysis of 

global ocean heat content. Geophys. Res. Lett., 40, 

!754-!759, doi:l0.!002/grl.50382. 

Benestad, R. E., 20!3: Are there persistent physical 

atmospheric responses to galactic cosmic rays? 

Environ. Res. Lett., 8, 035049, doi:IO.l088/1748 

-9326/8/3/035049. 

Borenstein, S., 2009: Statisticians reject global cooling: 

Some skeptics claim Earth is cooling despite contrary 

data. MSNBC, accessed on 18 August 20!0. [Available online at www.msnbc.msn.com/id/33482750/.] 

Boykoff, M. T., 2013: Public enemy no. I? Understanding media representations of outlier views 

on climate change. Amer. Behav. Sci., 57, 796-817, 

doi: !0.1!77/00027642!3476846. 

-. 2014: Media discourse on the climate slowdown, 

Nat. Climate Change, 4, 156-158, doi:IO.l038 

/nclimate2!56. 

-, and J. M. Boykoff, 2004: Balance as bias: Global warming and the US prestige press. Global Environ. Change, 14, 125-136, doi:IO.l016/j 

.gloenvcha.2003.!0.00!. 

Cahill, N., S. Rahmstorf, and A. C. Parnell, 2015: Change 

points of global temperature. Environ. Res. Lett., 10, 

084002, doi:l0.!088/1748-9326/I0/8/084002. 

AMERICAN METEOROLOGICAL SOCieTY 

Carter, B., 2006: There IS a problem with global 

warming .. .it stopped in !998. The Telegraph, 

accessed !8 August 2010. [Available online at www 

.telegraph.co.uk/comment/personal-view/3624242 

/There-IS-a-problem-with-global-warming ... -it 

-stopped-in-!998.html.] 

-, 201l: 2010 one ofthe hottest on record. ABC Australia, accessed 2 November 20!4. [Available online 

at www.abc.net.au/am/content/20!!/s3117917.htm.] 

Cowtan, K., and R. G. Way, 20!4: Coverage bias in the 

HadCRUT4 temperature series and its impact on 

recent temperature trends. Quart.]. Roy. Meteor. Soc., 

140, !935-!944, doi:IO.I002/qj.2297. 

-, and Coauthors, 2015: Robust comparison of climate models with observations using blended land 

air and ocean sea surface temperatures. Geophys. Res. 

Lett., 42, 6526-6534, doi:!O.l002/20!5GL064888. 

Curry, )., 20!4: The global warming statistical meltdown. Wall Street Journal, accessed 2 November 

20!4. [Available online at http:l/online.wsj.com 

/articles/judith-curry-the-global-warming-statistical 

-meltdown-14!290!060.] 

de Boer, J., J. A. Wardekker, and J.P. van der Sluijs, 2010: 

Frame-based guide to situated decision-making on 

climate change. Global Environ. Change, 20, 502-5!0, 

doi:l0.1016/j.gloenvcha.20!0.03.003. 

DeLosh, E. L., J. R. Busemeyer, and M. A. McDaniel, 

!997: Extrapolation: The sine qua non of abstraction 

in function learning. f. Exp. Psycho/. Learn. Mem. 

Cogn., 23, 968-986, doi:I0.1037/0278-7393.23.4.968. 

Dirikx, A., and D. Gelders, 20!0: To frame is to explain: 

A deductive frame-analysis of Dutch and French 

climate change coverage during the annual UN Conferences of the Parties. Public Understanding Sci., 19, 

732-742, doi:!O.II77/0963662509352044. 

Drijfhout, S. S., A. T. Blaker, S. A. Josey, A. J. G. Nurser, 

B. Sinha, and M. A. Balmaseda, 2014: Surface 

warming hiatus caused by increased heat uptake 

across multiple ocean basins. Geophys. Res. Lett., 41, 

7868-7874, doi: !0.1002/20!4GL06!456. 

Du, N., and D. V. Budescu, 2007: Does past volatility 

affect investors' price forecasts and confidence judgements? Int. f. Forecasting, 23, 497-Sll, doi:IO.!OI6/j 

.ijforecast.2007.03.003. 

Durack, P. J., P. J. Gleckler, F. W. Landerer, and K. E. 

Taylor, 2014: Quantifying underestimates of longterm upper~ ocean warming. Nat. Climate Change, 

4, 999-1005, doi:!O.l038/nclimate2389. 

Easterling, D. R., and M. F. Wehner, 2009: ls the climate 

warming or cooling? Geophys. Res. Lett., 36, L08706, 

doi:!O.I029/2009GL037810. 

Elsasser, S. W., and R. E. Dunlap, 2013: Leading 

voices in the denier choir: Conservative columnists 

MAnol6 BAnt 1 731 
184 

dismissal of global warming and denigration of 

climate science. Amer. Behav. Sci., 57, 754-776, 

doi: 10.1177/0002764212469800. 

England, M. H., and Coauthors, 2014: Recent intensification of wind-driven circulation in the Pacific and 

the ongoing warming hiatus. Nat. Climate Change, 

4, 222-227, doi:IO.I038/nclimate2106. 

-, J. B. Kajtar, and N. Maher, 2015: Robust warming 

projections despite the recent hiatus. Nat. Climate 

Change, 5, 394-396, doi:IO.I038/nclimate2575. 

Estrada, F., P. Perron, and B. Martinez-Lopez, 2013: 

Statistically derived contributions of diverse human 

influences to twentieth-century temperature changes. Nat. Geosci., 6, 1050-1055, doi:IO.I038/ngeol999. 

Forrest, D., J. Goddard, and R. Simmons, 2005: Oddssetters as forecasters: The case of English football. 

Int. f. Forecasting, 21,551-564, doi:IO.IOI6/j.ijforecast 

.2005.03.003. 

Foster, G., and S. Rahmstorf, 20ll: Global temperature 

evolution 1979-2010. Environ. Res. Lett., 6, 044022, 

doi: 10.108 8/1748-9326/6/4/044022. 

-,and J. Abraham, 2015: Lack of evidence for a 

slowdown in global temperature. Variations, Vol. 13, 

No. 3, U.S. Climate and Variability and Predictability 

(CLIVAR) Program, Washington, DC, 6-9. [Available online at https:/ /usclivar.org/sites/default/files 

/documents/2015/Variations2015Summer.pdf.] 

Fyfe, J., K. Salzen, J. Cole, N. Gillett, and J.-P. Vernier, 

2013: Surface response to stratospheric aerosol 

changes in a coupled atmosphere-ocean model. 

Geophys. Res. Lett., 40, 584-588, doi:IO.I002 

/grl.50!56. 

Guemas, V., F. J. Doblas-Reyes, I. Andreu-Burillo, and 

M. Asif, 2013: Retrospective prediction of the global 

warming slowdown in the past decade. Nat. Climate 

Change, 3, 649-653, doi:IO.I038/nclimatel863. 

Hansen, j., R. Ruedy, M. Sato, and K. Lo, 2010: Global 

surface temperature change. Rev. Geophys., 48, 

RG4004, doi:I0.1029/2010RG000345. 

Hardisty, D. J., E. ). johnson, and E. U. Weber, 2010: 

A dirty word or a dirty world? Attribute framing, 

political affiliation, and query theory. Psycho/. Sci., 

21,86-92, doi:IO.ll77/0956797609355572. 

Harvey, N., and F. Bolger, 1996: Graphs versus tables: 

Effects of data presentation format on judgmental forecasting. Int. f. Forecasting, 12, 119-137, 

doi: I 0.1016/0169-2070(95)006 34- 6. 

-, 1'. Ewart, and R. West, 1997: Effects of data noise 

on statistical judgement. Think. Reason., 3, lll-132, 

doi:IO.I080/135467897394383. 

Hawkins, E., T. Edwards, and D. McNeal!, 2014: Pause 

for thought. Nat. Climate Change, 4, 154-156, 

doi: I 0.103 8/nclimate2150. 

m 1 BAn~ MAna t6 

IPCC, 1996: Climate Change 1995: The Science of Climate Change. Cambridge University Press, 572 pp. 

-, 2013: Summary for policymakers. Climate Change 

2013; The Physical Science Basis, T. F. Stocker et al., 

Eds., Cambridge University Press, 33-ll5. 

Karl, T. R., and Coauthors, 2015: Possible artifacts of 

data biases in the recent global surface warming 

hiatus. Science, 348, 1469-1472, doi:IO.ll26/science. 

aaa5632. 

Kaufmann, R. K., H. Kauppi, M. L. Mann, and ). H. 

Stock, 20ll: Reconciling anthropogenic climate 

change with observed temperature 1998-2008. Proc. 

Nat/. A cad. Sci. USA, 108, II 790-11 793, doi:IO.I073 

/pnas.ll02467108. 

Kosaka, Y., and S.-P. Xie, 2013: Recent global-warming 

hiatus tied to equatorial Pacific surface cooling. Nature, 501,403-407, doi:IO.I038/naturel2534. 

Lakoff, G., 2010: Why it matters how we frame 

the environment. Environ. Commun., 4, 70-81, 

doi: I 0.108 0/17524030903529749. 

Lawrence, M., P. Goodwin, M. O'Connor, and D. Onkal, 

2006: Judgmental forecasting: A review of progress 

over the last 25 years. Int. f. Forecasting, 22,493-518, 

doi: 10.10 16/j .ijforecast.2006 .03.007. 

Lewandowsky, S., 2011: Popular consensus: Climate 

change set to continue. Psycho/. Sci., 22, 460-463, 

doi: 10.1177/0956797611402515. 

-,and I. Spence, 1989: Discriminating strata in scatterplots. /. Amer. Stat. Assoc., 84, 682-688, doi:l 0.108 

0/01621459.1989.10478821. 

-, M. L. Kalish, and S. Ngang, 2002: Simplified learning in complex situations: Knowledge partitioning in 

function learning. f. Exp. Psycho/. Gen., 131, 163-193, 

doi:IO.I 03 7/0096-3445.131.2.163. 

-, J. S. Risbey, M. Smithson, B. R. Newell, and). Hunter, 

2014a: Scientific uncertainty and climate change: 

Part I. Uncertainty and unabated emissions. Climatic 

Change, 124,21-37, doi:I0.1007/s10584-014-1082-7. 

-,and-, 2014b: Scientific uncertainty and climate change: Part II. Uncertainty and 

mitigation. Climatic Change, 124,39-52, doi:IO.l007 

/sl0584-014-1083-6. 

-, N. Oreskes, J. S. Risbey, B. R. Newell, and 

M. Smithson, 2015: Seepage: Climate change denial 

and its effect on the scientific community. Global 

Environ. Change, 33, 1-13, doi:IO.IOI6/j.gloenvcha 

.2015.02.013. 

Marotzke, J., and P.M. Forster, 2015: Forcing, feedback 

and internal variability in global temperature trends. 

Nature, 517,565-570, doi:IO.I038/naturel4ll7. 

Martinson, D., Ed., 1995: Natural Climate Variability on 

Decade-to-Century Time Scales. National Academy 

Press, 644 pp. 
185 

Meehl, G. A., and H. Teng, 2014: CMIPS multi-model 

hindcasts for the mid-1970s shift and early 2000s 

hiatus and predictions for 2016-2035. Geophys. Res. 

Lett., 41, 1711-1716, doi:IO.l002/2014GL059256. 

Merton, R. K., 1942: The normative structure of science. 

The Sociology of Science: Theoretical and Empirical 

Investigations, N. Storer, Ed,, University of Chicago 

Press, 267-278. 

Mooney, C., 2005: The Republican War on Science. Basic 

Books, 376 pp. 

Morice, C. P., J. J. Kennedy, N. A. Rayner, and 

P. D. Jones, 2012: Quantifying uncertainties in 

global and regional temperature change using an 

ensemble of observational estimates: The HadCRUT4 data set. f. Geophys. Res., 117, D08101, 

doi:10.1029/2011JD017187. 

Neely, R. R., and Coauthors, 2013: Recent anthropogenic 

increases in S02 from Asia have minimal impact 

on stratospheric aerosoL Geophys. Res. Lett., 40, 

999-1004, doi:10.1002/grL50263. 

Oreskes, N., and E. M. Conway, 2010: Merchants of 

Doubt. Bloomsbury Publishing, 368 pp. 

Otto, A., and Coauthors, 2013: Energy budget constraints on climate response. Nat. Geosci., 6, 415-416, 

doi:10.1038/ngeo1836. 

Ridley, D. A., and Coauthors, 2014: Total volcanic 

stratospheric aerosol optical depths and implications 

for global climate change. Geophys. Res. Lett., 41, 

7763-7769, doi:IO.l002/2014GL06154L 

Risbey, J, S., 2015: Free and furced climate variations. 

Nature, 517,562-563, doi:IO.l038/517562a. 

-, S. Lewandowsky, C. Langlais, D. P. Monselesan, 

T. ). O'Kane, and N. Oreskes, 2014: Well-estimated 

AMERICAN METEOROLOGICAL SOCIETY 

global surface warming in climate projections 

selected for ENSO phase. Nat. Climate Change, 4, 

835-840, doi:10.1038/nclimate2310. 

Santer, B. D., and Coauthors, 2011: Separating signal and 

noise in atmospheric temperature changes: The importance of timescale. f. Geophys. Res., 116, D22105, 

doi:10.1029/2011JD016263. 

-,and Coauthors, 2014: Volcanic contribution to 

decadal changes in tropospheric temperature. Nat. 

Geosci., 7, 185-189, doi:l0.1038/ngeo2098. 

Schmidt, G. A., D. T. Shindell, and K. Tsigaridis, 

2014: Reconciling warming trends. Nat. Geosci., 7, 

158-160, doi:10.1038/ngeo2105, 

Schneider, S. H., and S. Thompson, 1981: Atmospheric 

C02 and climate: Importance of the transient response. f, Geophys. Res., 86, 3135-3147, doi:I0.1029 

/JC086iC04p03135. 

Seneviratne, S. 1., M. G. Donat, B. Mueller, and L. V. 

Alexander, 2014: No pause in the increase of hot temperature extremes. Nat. Climate Change, 4, 161-163, 

doi:10.1038/nclimate2145. 

Sillmann, J., M.G. Donat, J, C. Fyfe, and F. W. Zwiers, 

2014: Observed and simulated temperature extremes 

during the recent warming hiatus. Environ. Res. Lett., 

9, 064023, doi:l0.1088/1748-9326/9/6/064023. 

Sloan, T., and A. W. Wolrendale, 2013: Cosmic rays, 

solar activity and the climate. Environ. Res. Lett., 8, 

045022, doi:10.1088/1748-9326/8/4/045022. 

Solomon, S., J. S. Daniel, R. R. Neely, J.-P. Vernier, E. G. 

Dutton, and L. W. Thomason, 2011: The persistently 

variable "background" stratospheric aerosol layer 

and global climate change. Science, 333, 866-870, 

doi: 10.1126/science. 1206027. 

MAY2016 BAMf I )3] 
186 
187 

DOCUMENTS SUBMITTED BY REPRESENTATIVE RANDY K. WEBER

Drowning by EPA 

overreach 

BY WILL COGGIN- 06/17/15 05:30PM EDT 1 

The Environmental Protection Agency (EPA) recently found itself in hot 

water. The New York Times revealed the agency colluded with 

environmentalist groups in a campaign to manufacture public comments in 

favor of a new rule that expands its own power. The agency's actions and the 

shenanigans of its environmentalist supporters shed light on how a bad rule 

can flow through the regulatory process. 

The Waters of the United States (WOTUS) rule extends the reach of EPA to 

regulate ponds, ditches and even large puddles under the Clean Water Act 

(CWA). That's bad news for farmers, ranchers, small businesses or anyone 

else who wants to use land under CWA jurisdiction: It costs an average of 

$270,000 to obtain the special permit required to do so, according to the 

National Federation of Independent Businesses. 

The downsides are clear, and the EPA's judgment was murky even before the 

rule. Last year, the agency threatened to fine a Wyoming man $75,000 a day 

for building a pond on his own property without a permit. 

Almost immediately after its proposal, the rule prompted a wide opposition 

urging the EPA to "ditch the rule," from small businesses, farmers and 

ranchers, energy producers and others. 

The EPA needed support for its water grab. While the EPA failed to consult 

with those harmed by the WOTUS rule, documents obtained by The New York 

Times show the EPA worked with environmentalist groups including the Sierra 
188 

Club and National Resources Defense Council to manufacture public 

comments in its favor. 

EPA Administrator Gina McCarthy later testified at a Senate subcommittee 

hearing that 87 percent of the approximately 1 million public comments her 

agency received were supportive. By omitting mention of the efforts (or money 

spent) to solicit the comments, McCarthy attempted to make it look like there 

was a spontaneous groundswell of support for her rule. 

And that wasn't the only subterfuge behind the EPA's power grab. 

A number of left-wing groups camouflaged as sportsmenfriendly organizations, including the Theodore Roosevelt Conservation 

Partnership {TRCP), Backcountry Hunters and Anglers (BHA) and Trout 

Unlimited, were also helping the EPA to foist the water rule onto an 

unsuspecting public. 

In July 2014, TRCP called for "broad public involvement," setting the table for 

the EPA's campaign to gather public comment in support. This despite the 

fact that the organization's support had already been touted by the EPA in an 

effort to make it look like a broad coalition was in favor. 

These groups claim to represent sportsmen's interests-giving the rule 

seemingly conservative support-but they are tangled in a web of money from 

left-wing foundations with anti-gun and anti-agriculture agendas. BHA gets 

most of its donations from three environmental groups, according to tax 

records, while TRCP gets its money from a handful of Big Labor and Big 

Green groups. Trout Unlimited, meanwhile, has taken tens of millions from 

fringe environmental groups. 

A bipartisan bill to send the rule back to the EPA's drawing board has already 

passed the House and a similar measure introduced by Sen. John 

Barrasso (R-Wyo.) has gained cosponsors from both sides of the aisle in the 

Senate. In response BHA labeled the congressional effort "un-American." 

Meanwhile, attorneys general in three states have said that state challenges 

to the rule are likely. 

And the EPA may have violated federal law that prohibits using appropriated 

funds for lobbying in creating the comments. Sen. Pat Roberts (RKan.) called it "a political grassroots lobbying campaign with environmental 
189 

groups to manipulate the process and disregard legitimate concerns from rural 

America." 

The EPA's brazenness in ramming through a rule with camouflaged and 

concocted support is concerning even for a town where there's no shortage of 

dirty tricks. Congress would be wise to wash away the water rule before it 

does lasting damage. 

Coggin is the director of research at the Environmental Policy Alliance, a 

project of the non-profit Center for Organizational Research and 

Education. CORE is supported by a wide variety of businesses and 

foundations, including those in the hospitality, agriculture, and energy 

industries. 
190 

DOCUMENTS SUBMITTED BY REPRESENTATIVE GARY PALMER

Former UN Lead Author: Global Warming 

Caused By 'Natural Variations' In Climate 

MICHAELBASfASCH 

Global temperature change observed over the last hundred years or so is well within the natural 

variability of the last 8,000 years, according to a new paper by a former Intergovernmental Panel 

On Climate Change (IPCC) lead author. 

Dr. Philip Lloyd, a South Africa-based physicist and climate researcher, examined ice core-based 

temperature data going back 8,000 years to gain perspective on the magnitude of global 

temperature changes over the 20th Century. 

What Lloyd found was that the standard deviation of the temperature over the last 8,000 years 

was about 0.98 degrees Celsius- higher than the 0.85 degrees climate scientists say the world has 

warmed over the last century. 

"This suggests that while some portion ofthe temperature change observed in the 20th century 

was probably caused by greenhouse gases, there is a strong likelihood that the major portion was 

due to natural variations," Lloyd wrote in his study. 

The United Nations' IPCC claims there's been 0.85 degrees Celsius of warming since the late 

1800s, and concludes that most of this warming is due to human activities- mainly, the burning 

of fossil fuels and changes in land use. The IPCC ~that "more than half ofthe observed 

increase in global average surface temperature from 1951 to 20 I 0" have been caused by human 

activity. 

If Lloyd's results hold, the IPCC may have to revise how much warming it attributes to mankind 

In any case, the IPCC's estimate of man-made and natural warming (0.85 degrees) is still below 
191 

the standard deviation for the last 8,000, according to Lloyd's results. This means that warming 

is not very significant within the context of the Earth's recent climate history. 

Lloyd arrived at his conclusion after the "differences in temperatures between all records which 

are approximately a century apart were determined, after any trends in the data had been 

removed." Lloyd noted the "differences were close to normally distributed." 

But Lloyd's study hits at a larger debate within climate science: how much warming is 

attributable to mankind or nature. Clearly, Lloyd and the IPCC he once contributed to now 

represent different ends of the spectrum. 

"The key challenge in understanding climate change is to assess the natural climate variability," 

Dr. Judith Curry, a climate scientist at Georgia Tech, told The Daily Caller News Foundation in 

April. 

At the time, Ronald Bailey, a science write for Reason magazine, wrote that there has still not 

been enough observed warming to meet the JPCC's standard of"enhanced warming"- that is, 

warming above natural levels. 

In his article, Bailey noted that there has not been enough temperature rise since the IPCC set its 

benchmark for "enhanced warming" in 1990. Curry noted that there was a big jump in 

temperature between 1993 and 1998, but that was basically because of the latter year's El Nino. 

"The magnitude of natural climate variability over the past I 000 years and even the past I 00 

years is hotly debated," Curry added. "Personally, I think the role of natural climate variability 

has been substantially underestimated in our interpretation of recent climate change." 

But not all scientists agree with Bailey's article, and some argue that signs of human influence 

on the Earth's climate were evident in the 1970s. Indeed, by 1995 the IPCC stated that the 

"balance of evidence suggests a discernible human influence on global climate." The 

international body has only made stronger statement on man's climatic influence ever since. 

"I would not pin anything on what was said by IPCC in 1990," Dr. Kevin Trenberth, a climate 

scientist with the National Center for Atmospheric Research, told TheDCNF in April. "In the 

reports since then there have been thorough evaluations of past IPCC projections and whether 

they were out of line." 

Human influence on the climate may have been observable in the 1970s, but scientists have had 

trouble explaining why satellite data shows that average global temperatures have been virtually 

flat for more than 18 years. Satellites measure the troposphere the lowest few miles of the 

atmosphere- in contrast, to surface temperature measurements, which most climate bodies rely 

on for estimates of global average temperature average. 

But even surface temperature data showed a hiatus in warming for about 15 years or so. 

Scientists have offered up dozens of explanations for why global temperatures have been flat 

since the late 1990s. The most prominent explanation is that oceans have been absorbing most of 
192 

the "heat" from increased greenhouse gas emissions, meaning surface temperatures show less 

warming than they otherwise would. 

"What is evident now is that the signal of global warming emerged from the noise of natural 

variability about the mid 1970s," Trenberth added. "There are fluctuations in global mean 

temperatures: from year to year with El Nifios, etc., and from decade to decade, so that trends 

reflecting global warming need to be taken over at least 20 years." 
193 

Guest essay by Philip Lloyd, 

The raw data that is fed to NASA in order to develop the global temperature series is subjected to 

"homogenization" to ensure that it does not suffer from such things as the changes in the method of 

measuring the mean temperature, or changes in readings because of changes in location. However, 

while the process is supposed to be supported by metadata- i.e. the homogenizers are supposed to 

provide the basis for any modification of the raw data., For example, the raw data for my home city, 

Cape Town, goes back to 1880:, clip_image002, 

http:ljdata.giss.nasa.gov/tmp/gistemp/STATIONS/tmp 141688160000 0 0/station.txt, The warmest 

years were in the 1930's, as they were in many other parts of the globe. There was then a fairly steep 

decline into the 1970's before the temperature recovered to today's levels, close to the hottest years of 

the 1930's., In NASA's hands, the data pre-1909 was discarded; the 1910 to 1939 data was adjusted 

downwards by 1.1deg C; the 1940 to 1959 data was adjusted downwards by about 0.8 deg Con average; 

the 1969 to 1995 data was adjusted upwards by about 0.2 deg C, with the end result that GISS Ver 2 

was:-, clip_image004, Being curious, I asked for the metadata. Eventually I got a single line, most of 

which was obvious, latitude, longitude, height above mean sea level, followed by four orfive 

alphanumerics. This was no basis for the "adjustments" to the raw data., Which should I believe? The 

raw data showed a marked drop from the 1940's to the 1970's, which echoed similar drops elsewhere. 

Time magazine covers showed the 1970's were indeed cold., The raw data is probably accurate. The 

homogenized data is certainly not. It is difficult to avoid the conclusion that "homogenization" means 

"revise the story line" and "anthropogenic global warming" really means "humans changed the figures"., 

Prof Philip Lloyd, Energy Institute, CPUT, SARETEC, Sacks Circle, Bellville 

Sent from my iPad 
194 

Researchers once blamed a cleaner world. Now they are 

not so sure 

Asthma rates have been surging around the globe over the past three decades, and for a 

long time researchers thought they had a good idea of what might be fueling the 

increase: the world we live in is just a little too clean. According to this notion-known 

as the hygiene hypothesis-exposure in early childhood to infectious agents programs 

the immune system to mount differing highly effective defenses against disease-causing 

viruses, bacteria and parasites. Better sanitary conditions deprive the immune system of 

this training, so that for reasons that are still unclear, the body pounces on harmless 

particles-such as dust and ragweed-as if they were deadly threats. The resulting 

allergic reaction leads to the classic signs of asthma: chronic inflammation or swelling of 

the airways and acute spasms of those passageways. 

Or so the thinking went. Although a lot of data support the hygiene hypothesis for 

allergies, the same cannot be said for asthma. Contrary to expectations, asthma rates 

have skyrocketed in urban areas in the U.S. that are not particularly clean. Moreover, 

the big increase in asthma rates in developed countries did not kick off until the 1980swell after general sanitary conditions in the richer parts of the world had improved. And 

some studies are beginning to show that far from protecting children from asthma, 

respiratory infections in early childhood may actually be a risk factor for it. 

The collapse of the hygiene hypothesis as a general explanation for the startling jump in 

asthma rates has led physicians and scientists to a new realization: asthma is a much 

more complex condition than anyone had truly appreciated. Indeed, it may not be even 
195 

be a single disease. Studies now suggest that only half of asthma cases have an allergic 

component. 

The prevention and treatment implications are significant. If, for instance, it is true that 

allergy is not a fundamental cause of asthma in many people, then an alternative mix of 

treatments may be more effective for those individuals. To root out asthma's cause (or 

causes) and properly treat the burgeoning number of people who are affected-300 

million globally at last count-scientists will have to come to grips with the biology of its 

various forms. 

Balancing Act 

The hygiene hypothesis was first described in 1989 by David P. Strachan, a British 

epidemiologist who was studying hay fever. The more children in a family, he noticed, 

the lower the rates of hay fever and eczema, an allergic skin condition. Children in large 

families tend to swap colds and other infections more often than children with fewer 

siblings. Could it be that increased exposure to pathogens from their many siblings was 

protecting children from large families against allergies? 

That same year Erika von Mutius, an epidemiologist at Munich University, was looking 

into the effect of air pollution on asthma in what was then East and West Germany. 

Children from dirtier East Germany, she was shocked to find, had dramatically less 

asthma than their West German counterparts living in cleaner, more modern 

circumstances. The East German children, unlike their Western counterparts, had spent 

more time in day care and thus had likely been exposed to many more viruses and 

bacteria. "That was astonishing," she recalls, and led to "a major shift" in thinking. 

These findings sparked intense debate among scientists. What is it about unhygienic 

living that might protect against asthma? One of the more popular explanations in the 

following decades entailed a balance between the immune cells that are involved in the 

body's reaction to most viruses and bacteria and those that are involved in the reaction 

to most parasites and allergens. These two groups of cells produce chemicals that inhibit 

each other. Early-cllildhood exposure to bacteria and viruses would cause the infectionrelated cells to become active, keeping the allergy- and parasite-related cells in check. 

Without that interplay, the allergy-related cells would later become overreactive, 

starting an allergic cllain reaction that became chronic and ended in constricted airways, 

asthmatic spasms and labored breathing. 

[break] 

Inconvenient Facts 

There was only one problem. As more data came in, they failed to tell the same story as 

the hygiene hypothesis. Children in Latin America with high rates of supposedly 

protective infection have even higher rates of asthma than children in western Europe. 

Inner-city children in Chicago and New York have quite high rates of asthma, despite 

unhygienic living. And the rates of asthma varied among countries with very similar 

histories of cleanliness-indicating that there was more to it than tidiness. For example, 

by 2004 Sweden's asthma cases had increased to 10 percent, according to one 

international study, while the number of cases in the U.K. had soared to 20 percent. 

In addition, research showed that the relation between asthma and allergy is not at all 

straightforward. Some cases of asthma are indeed triggered by allergies, although the 

consensus among researchers over the past decade is that the connection is probably not 

as clear-cut as the hygiene hypothesis would suggest. Still other layers of immune 

regulation must be involved. Maria Yazdanbakhsh, a parasitologist at Leiden University 
196 

in the Netherlands, has shown that people infected with parasitic worms have very high 

levels of the allergy-related immune cells but very low rates of asthma, disproving a 

direct connection between allergy and asthma in these cases at least. 

What is more, a landmark review of asthma studies in 1999 by Neil Pearce, now at the 

London School of Hygiene and Tropical Medicine, demonstrated that at least half of 

asthma cases in the general population have no connection to allergic reactions at all. 

These could never be explained by the hygiene hypothesis. 

In fact, the same factors that the hygiene hypothesis suggests protect people from 

developing allergic asthma may cause them to develop nonallergic asthma. "We think 

that dirt protects against allergic asthma, as foretold by the hygiene hypothesis, but 

increases the risk of having a nonallergic form," says Laura Rodrigues of the London 

School of Hygiene and Tropical Medicine, who studies asthma in Latin America. 

Pollutants in the air can irritate the airways and cause inflammation that leads to 

constricted breathing. Childhood colds, which the hygiene hypothesis suggested might 

help prevent development of asthma, can actually be a risk factor for asthma, especially 

if severe, says James E. Gern, a pediatrician who studies colds and asthma at the 

University of Wisconsin-Madison. "Early-life infections are an indicator of asthma risk 

rather than protective in any way," he says. 

Besides the hygiene hypothesis, what can explain the increase in asthma rates? Other 

suggested causes include a rise in sedentary lifestyle, which could affect lung strength, 

and the rise in obesity, which increases inflammation throughout the body. A reworking 

of the hygiene hypothesis that focuses on changes in the normal nondisease-causing 

bacteria that live inside and on the body (in the intestines or the airways or on the skin) 

has promise. Studies by von Mutius and others have shown that children who live on 

farms where cows or pigs are raised and where they drink raw milk almost never have 

asthma, allergic or otherwise. Presumably because the children drank unpasteurized 

milk and handled livestock, they have different strains of normal bacteria in their 

airways that are somehow more protective than those found in city kids. 

But the short answer to the question of why asthma has increased, according to Pearce, 

von Mutius, Rodrigues and many others, is, "We don't know." Pearce, in particular, 

wonders whether modernization in general or westernization in particular may play a 

role. "There is something about westernization that means people's immune systems 

function in a different way," he says. "But we don't know what the mechanism is." 

[break] 

Getting at the true underlying cause of the climb will require better ways of 

distinguishing among various possible types of asthma. Major asthma research 

networks supported by the National Institutes of Health have begun recording the 

details of thousands of individuals' symptoms and treatments. As the results are 

gathered and analyzed, researchers hope to identify clusters of asthma cases that have 

different causes and respond to different treatments. The hope is that "if you come in 

with these characteristics in asthma, we can anticipate what the prognosis is going to be 

and what the most effective treatment for you is going to be," says William W. Busse of 

the University of Wisconsin School of Medicine and Public Health, who is part of one 

such network. 

It will take years to understand fully whether microbial exposure, lifestyle changes or 

the obesity epidemic is more important in explaining the continuing increase in asthma 

rates. But one thing is clear: the hygiene hypothesis was just the beginning. 
197 

CliMATE DEPOT 

UN Scientists Who Have Turned on the UN 

IPCC & Man-Made Climate Fears- A 

Climate Depot Flashback Report 

Warming fears are the "worst scientific scandal in the history ••• When people come to know what the 

truth is, they will feel deceived by science and scientists."- UN IPCC Japanese Scienti~t Dr. Kiminori 

ltolr, an award-winning PhD environmental phy,vical chemist 

By: Marc Morano- Climate DepotAugust 21,2013 9:34PM 

Here is a very small sampling of what current and former UN scientists have to say about the UN's 

climate claims and its scientific methods. 

Wanning fears are the "worst scientific scandal in the history ... When people come to know what the tmth is, 

they will feel deceived by science and scientists." UN IPCC Japanese Scientist Dr. Kiminori Itoh, an 

award-winning PhD environmental physical chemist. 

"The lPCC has actually become a closed circuit; it doesn't listen to others. It doesn't have open minds ... I am 

really amazed that the Nobel Peace Prize has been given on scientifically incorrect conclusions by people who 

are not geologists."- Indian geologivt Dr. Arun D. Alrluwalia at Punjab University and a board member of 

the UN-supported International Year of the Planet. 

"Temperature measurements show that the [climate model-predicted mid-troposphere] hot zone is nonexistent. This is more than sufficient to invalidate global climate models and projections made with 

them!"- UN IPCC Scientist Dr. Steven M. Japar, a PhD atmospheric chemist who was part of 

Intergovernmental Panel on Climate Change's (IPCC) Second (1995) and Third (2001) Assessment 

Reports, and has authored 83 peer-reviewed publications and in tire areas of climate change, atmospheric 

chemistry, air pollutions and vehicle emission,v. 

UN IPCC Scientist Kenneth P. Green Declares 'A Death Spiral for Climate Alarmism' September 30, 2009-

'We can expect climate crisis industry to grow increasingly shrill, and increasingly hostile toward anyone who 

questions their authority'- Dr. Kenneth Green was a Working Group I expert reviewer for the United 

Nations' Intergovernmental Panel on Climate Change (IPCC) in 2001 
198 

'The whole climate change issue is about to fall apart- Heads will roll!' -South African UN Scientist Dr. 

Will Alexander, April12, 2009- Profe.fsor Alexander, is Emeritus of the Department of Civil and 

Biosystems Engineering at the University of Pretoria in South Africa, and a former member of the United 

Nations Scientific and Technical Committee on Natural Disasters. 

"! was at the table with three Europeans, and we were having lunch. And they were talking about their role as 

lead authors. And they were talking about how they were trying to make the report so dramatic that the United 

States would just have to sign that Kyoto Protocol," Christy told CNN on May 2, 2007. -Alabama State 

Climatologist Dr. John Chri.fty of the Univer.fity of Alabama in Huntsville, served as a UN 1PCC lead 

author in 2001 for the 3rd assessment report and detailed how he personally witne.fsed UN scientistf 

attempting to distort the science for political purposes. 

"Gore prompted me to start delving into the science again and I quickly found myself solidly in the skeptic 

camp ... Climate models can at best be useful for explaining climate changes after the fact." -Meteorologist 

Hajo Smit of Holland, who reversed flis belief in man-made warming to become a skeptic, i.f a former 

member of the Dutch UN IPCC committee. 

"The quantity of C02 we produce is insignificant in terms of the natural circulation between air, water and 

soil. .. I am doing a detailed assessment of the UN IPCC reports and the Summaries for Policy Makers, 

identifYing the way in which the Summaries have distorted the science."- South African Nuclear Physici.ft 

and Chemical Engineer Dr. Philip Lloyd, a UN IPCC co-coordinating lead author who has authored over 

150 refereed publications. 

"The claims of the IPCC are dangerous unscientific nonsense" declared IPCC reviewer and climate 

reset~rcher Dr Vincent Gray, of New Zealand in 2007. Gray was an expert reviewer on every single draft of 

the IPCC reports going back to 1990, author of more than 100 scientific publications. (LINK) & (LINK) 

"After reading [UN IPCC chairman] Pachauri's asinine comment [comparing skeptics to] Flat Earthers, it's 

hard to remain quiet."- Climate statistician Dr. William M. Briggs, who specializes in the statistics of 

forecast evaluation, serves on the American Meteorological Society's Probability and Statistics Committee 

and is an As.mciate Editor of Monthly Weather Review. 

UN IPCC Lead Author Tom Tripp Dissents on man-made warming: 'We're not scientifically there yet'- July 

16,2009 

The UN IPCC's Kevin Trenberth's claim that the UN !PCC is an "very open" also needs examining. The IPCC 

summary for policymakers is used to scare politicians and goad the public into action. The UN is all about 

politics. 

UN special climate envoy Dr. Gro Harlem Brundtland declared "it's completely immoral, even, to 

question" the UN's alleged global wanning "consensus," according to a May I 0, 2007 article. Sounds 

scientific, doesn't it? 

Dr. John Brignell, a UK Emeritus Engineering Professor at the University of Southampton who held the 

Chair in Industrial Instrumentation at Southampton, accused the UN of"censorship'' on July 23. 2008. 
199 

"Here was a purely political body posing as a scientific institution. Through the power of patronage it rapidly 

attracted acolytes. Peer review soon rapidly evolved from the old style refereeing to a much more sinister 

imposition of The Censorship. As Wegman demonstrated, new circles of like-minded propagandists formed, 

acting as judge and jury for each other. Above all, they acted in concert to keep out alien and hostile opinion. 

'Peer review' developed into a mantra that was picked up by political activists who clearly had no idea of the 

procedures of science or its learned societies. It became an imprimatur of political acceptability, whose 

absence was equivalent to placement on the proscribed list," Brignell wrote. 

Research by Australian climate data analyst John McLean revealed that the IPCC's peer-review process 

for the Summary for Policymakers leaves much to be desired. (LINK) (LINK) (LINK) & (LINK) McLean's 

research revealed that the UN IPCC peer-review process is "an illusion." McLean's study found that very few 

scientists are actively involved in the UN's peer-review process. The report contained devastating revelations 

to the central IPCC assertion that 'it is very highly likely that greenhouse gas forcing has been the dominant 

cause of the observed global warming over the last 50 years." The analysis by McLean states: "The IPCC leads 

us to believe that this statement is very much supported by the majority of reviewers. The reality is that there is 

surprisingly little explicit support for this key notion. Among the 23 independent reviewers just 4 explicitly 

endorsed the chapter with its hypothesis, and one other endorsed only a specific section. Moreover, only 62 of 

the IPCC's 308 reviewers commented on this chapter at all." Repeating: Only four UN scientists in the IPCC 

peer-review process explicitly endorsed the key chapter blaming mankind for wanning the past 50 years, 

according to this recent analysis. 

Here is a small sampling of what current and former UN scientists have to say about the UN IPCC's 

"very open'' process. 

(Below are excerpts from various U.S. Senate reports which Climate Depot's Morano authored during his 

years at the U.S. Senate's Environment and Public Works Committee.) 

One former UN IPCC scientist bluntly told the Senate Environment and Public Works (EPW) committee how 

the UN IPCC Summary for Policymakers "distorted" the scientists work. "I have found examples of a 

Summary saying precisely the opposite of what the scientists said," explained South African Nuclear 

Physicist and Chemical Engineer Dr. Philip Lloyd, a UN IPCC co-coordinating lead author who has 

authored over ISO refereed publications. 

In an August 13, 2007 letter, UN IPCC Scientist Dr. Madhav Khandekar, a retired Environment 

Canada scientist, lashed out at those who "seem to naively believe that the climate change science espoused in 

the [UN's]lntergovernmental Panel on Climate Change (IPCC) documents represents 'scientific consensus."' 

Khandekar continued: "Nothing could be further than the truth! As one of the invited expert reviewers for the 

2007 IPCC documents, I have pointed out the flawed review process used by the IPCC scientists in one of my 

letters. 1 have also pointed out in my letter that an increasing number of scientists are now questioning the 

hypothesis of Greenhouse gas induced warming of the earth's surface and suggesting a stronger impact of solar 

variability and large-scale atmospheric circulation patterns on the observed temperature increase than 
200 

previously believed." "Unforhmately, the IPCC climate change documents do not provide an objective 

assessment of the earth's temperature trends and associated climate change," Khandekar concluded. 

Paul Reiter, a malaria expert formerly of the Centers for Disease Control and Prevention, participated 

in a past UN JPCC process and now calls the concept of consensus on global warming a "sham." Reiter, a 

professor of entomology and tropical disease with the Pasteur Institute in Paris, had to threaten legal action to 

have his name removed from the IPCC. "That is how they make it seem that all the top scientists arc agreed," 

he said on March 5, 2007. "It's not true," he added. 

Hurricane expert Christopher W. Landsea of NOAA's National Hurricane Center, was both an author a 

reviewer for the JPCC's 2nd Assessment Report in 1995 and the 3rd Assessment Report in 2001, but 

resigned from the 4th Assessment Report after charging the UN with playing politics with Hurricane science. 

Landsea wrote a January 17, 2005 public letter detailing his experience with the UN: "I am withdrawing [from 

the UN] because I have come to view the part of the IPCC to which my expertise is relevant as having become 

politicized. In addition, when I have raised my concerns to the IPCC leadership, their response was simply to 

dismiss my concerns." "I personally cannot in good faith continue to contribute to a process that I view as both 

being motivated by pre-conceived agendas and being scientifically unsound," Landsea added. 

In addition, a Greenpeace activist co-authored a key economic report in 2007. Left unreported by most of the 

media was the fact that Bill Hare, an advisor to Greenpeace, was a lead co- author of a key economic report in 

the JPCC's 4th Assessment. Not surprisingly, the Greenpeace co-authored report predicted a gloomy future for 

our planet unless we follow the UN's policy prescriptions. 

The UN IPCC's own guidelines explicitly state that the scientific reports have to be "change[ d)" to "ensure 

consistency with" the politically motivated Summary for Policymakers. 

In addition, the IPCC more closely resembles a political party's convention platform battle- not a scientific 

process. During an JPCC Summary for Policymakers process, political delegates and international bureaucrats 

squabble over the specific wording of a phrase or assertion. 

Steve Mcintyre of Climate Andit, one of the individuals responsible for debunking the infamous 

"Hockey Stick" temperature graph, slammed the JPCC Summary for Policymaker's process on January 24, 

2007. 

Mcintyre wrote: "So the purpose ofthe three-month delay between the publication of the (IPCC) Summary for 

Policy-Makers and the release of the actual WG I (Working Group I) is to enable them to make any 

'necessary' adjustments to the technical report to match the policy summary. Unbelievable. Can you imagine 

what securities commissions would say if business promoters issued a big promotion and then the promoters 

made the 'necessary' adjustments to the qualifying reports and financial statements so that they matched the 

promotion. Words fail me." 

Former Colorado State Climatologist Dr. Roger Pielke Sr. also detailed the corruption of the UN IPCC 

process on September 1, 2007: "The same individuals who are doing primary research in the role of humans on 

the climate system are then permitted to lead the [IPCC] assessment! There should be an outcry on this 
201 

Æ 

obvious conflict of interest, but to date either few recognize this conflict, or see that since the 

recommendations of the JPCC fit their policy and political agenda, they chose to ignore this conflict. In either 

case, scientific rigor has been sacrificed and poor policy and political decisions will inevitably follow," Pielke 

explained. He added: "We need recognition among the scientific community, the media, and policymakers that 

the IPCC process is obviously a real conflict of interest, and this has resulted in a significantly flawed report." 

Andrei Kapitsa, a Russian geographer and Antarctic ice core researcher: "The Kyoto theorists have put 

the cart before the horse. It is global wanning that triggers higher levels of carbon dioxide in the atmosphere, 

not the other way round ... A large number of critical documents submitted at the 1995 U.N. conference in 

Madrid vanished without a trace. As a result, the discussion was one-sided and heavily biased, and the U.N. 

declared global warming to be a scientific fact."