U.S. GOVERNMENT PRINTING OFFICE

WASHINGTON : 

For sale by the Superintendent of Documents, U.S. Government Printing Office

Internet: bookstore.gpo.gov Phone: toll free (866) 512–1800; DC area (202) 512–1800

Fax: (202) 512–2250 Mail: Stop SSOP, Washington, DC 20402–0001

92–340PS 2004

FISCAL YEAR 2005 ENVIRONMENTAL

PROTECTION AGENCY BUDGET

HEARING

BEFORE THE

SUBCOMMITTEE ON ENVIRONMENT, TECHNOLOGY,

AND STANDARDS

COMMITTEE ON SCIENCE

HOUSE OF REPRESENTATIVES

ONE HUNDRED EIGHTH CONGRESS

SECOND SESSION

MARCH 11, 2004

Serial No. 108–46

Printed for the use of the Committee on Science

(

Available via the World Wide Web: http://www.house.gov/science

(II)

COMMITTEE ON SCIENCE

HON. SHERWOOD L. BOEHLERT, New York, Chairman

RALPH M. HALL, Texas

LAMAR S. SMITH, Texas

CURT WELDON, Pennsylvania

DANA ROHRABACHER, California

KEN CALVERT, California

NICK SMITH, Michigan

ROSCOE G. BARTLETT, Maryland

VERNON J. EHLERS, Michigan

GIL GUTKNECHT, Minnesota

GEORGE R. NETHERCUTT, JR.,

Washington

FRANK D. LUCAS, Oklahoma

JUDY BIGGERT, Illinois

WAYNE T. GILCHREST, Maryland

W. TODD AKIN, Missouri

TIMOTHY V. JOHNSON, Illinois

MELISSA A. HART, Pennsylvania

J. RANDY FORBES, Virginia

PHIL GINGREY, Georgia

ROB BISHOP, Utah

MICHAEL C. BURGESS, Texas

JO BONNER, Alabama

TOM FEENEY, Florida

RANDY NEUGEBAUER, Texas

VACANCY

BART GORDON, Tennessee

JERRY F. COSTELLO, Illinois

EDDIE BERNICE JOHNSON, Texas

LYNN C. WOOLSEY, California

NICK LAMPSON, Texas

JOHN B. LARSON, Connecticut

MARK UDALL, Colorado

DAVID WU, Oregon

MICHAEL M. HONDA, California

BRAD MILLER, North Carolina

LINCOLN DAVIS, Tennessee

SHEILA JACKSON LEE, Texas

ZOE LOFGREN, California

BRAD SHERMAN, California

BRIAN BAIRD, Washington

DENNIS MOORE, Kansas

ANTHONY D. WEINER, New York

JIM MATHESON, Utah

DENNIS A. CARDOZA, California

VACANCY

VACANCY

VACANCY

SUBCOMMITTEE ON ENVIRONMENT, TECHNOLOGY, AND STANDARDS

VERNON J. EHLERS, Michigan, Chairman

NICK SMITH, Michigan

GIL GUTKNECHT, Minnesota

JUDY BIGGERT, Illinois

WAYNE T. GILCHREST, Maryland

TIMOTHY V. JOHNSON, Illinois

MICHAEL C. BURGESS, Texas

VACANCY

SHERWOOD L. BOEHLERT, New York

MARK UDALL, Colorado

BRAD MILLER, North Carolina

LINCOLN DAVIS, Tennessee

BRIAN BAIRD, Washington

JIM MATHESON, Utah

ZOE LOFGREN, California

BART GORDON, Tennessee

ERIC WEBSTER Subcommittee Staff Director

MIKE QUEAR Democratic Professional Staff Member

JEAN FRUCI Democratic Professional Staff Member

OLWEN HUXLEY Professional Staff Member

MARTY SPITZER Professional Staff Member

SUSANNAH FOSTER Professional Staff Member

AMY CARROLL Professional Staff Member/Chairman’s Designee

ADAM SHAMPAINE Majority Staff Assistant

MARTY RALSTON Democratic Staff Assistant

(III)

C O N T E N T S

March 11, 2004

Page

Witness List ............................................................................................................. 2

Hearing Charter ...................................................................................................... 3

Opening Statements

Statement by Representative Vernon J. Ehlers, Chairman, Subcommittee on

Environment, Technology, and Standards, Committee on Science, U.S.

House of Representatives .................................................................................... 31

Written Statement ............................................................................................ 31

Statement by Representative Brad Miller, Member, Subcommittee on Environment, Technology, and Standards, Committee on Science, U.S. House

of Representatives ................................................................................................ 32

Written Statement ............................................................................................ 33

Witnesses:

Mr. Clay Johnson, III, Deputy Director of Management, Office of Management and Budget

Oral Statement ................................................................................................. 35

Written Statement ............................................................................................ 36

Biography .......................................................................................................... 38

Dr. Paul Gilman, Assistant Administrator for Research and Development,

United States Environmental Protection Agency

Oral Statement ................................................................................................. 39

Written Statement ............................................................................................ 40

Biography .......................................................................................................... 71

Mr. Paul L. Posner, Managing Director, Federal Budget and Intergovernmental Relations, General Accounting Office

Oral Statement ................................................................................................. 71

Written Statement ............................................................................................ 73

Dr. Genevieve Matanoski, Professor, Department of Epidemiology, Johns Hopkins University; Chair, EPA Science Advisory Board Review of EPA’s Fiscal

Year 2005 Budget; Former Chair, EPA Science Advisory Board

Oral Statement ................................................................................................. 82

Written Statement ............................................................................................ 83

Biography .......................................................................................................... 86

Dr. Costel D. Denson, Professor, Department of Engineering, University of

Delaware; Member of the National Research Council Panel Review of The

Measure of STAR: Review of the EPA’s Science to Achieve Results (STAR)

Research Grants Program

Oral Statement ................................................................................................. 87

Written Statement ............................................................................................ 88

Biography .......................................................................................................... 92

Discussion

Rationale for Science to Achieve Results (STAR) Research Grants Program

Budget Cuts ...................................................................................................... 92

Incorporation of STAR Scientific Research into Decision-making ................... 95

Effect of Ecological Research Programs Budget Cuts on Human Health

Research Programs ........................................................................................... 96

Immediate Effects of Science to Achieve Results (STAR) Grants Program

Budget Cuts on EPA Research ........................................................................ 97

Page

IV

Application of the Program Assessment Rating Tool (PART) and Alternative

Review Methods Within EPA’s Office of Research and Development .......... 99

Termination of EPA’s Building Decontamination Research Program ............. 102

Peer Review and Evaluation of R&D Programs ................................................ 103

Evaluating Short-term vs. Long-term Research ................................................ 105

Concluding Remarks ............................................................................................ 106

Appendix: Answers to Post-Hearing Questions

Mr. Clay Johnson, III, Deputy Director of Management, Office of Management and Budget .................................................................................................. 108

Dr. Paul Gilman, Assistant Administrator for Research and Development,

United States Environmental Protection Agency .............................................. 111

(1)

FISCAL YEAR 2005 ENVIRONMENTAL

PROTECTION AGENCY BUDGET

THURSDAY, MARCH 11, 2004

HOUSE OF REPRESENTATIVES,

SUBCOMMITTEE ON ENVIRONMENT, TECHNOLOGY, AND

STANDARDS,

COMMITTEE ON SCIENCE,

Washington, DC.

The Subcommittee met, pursuant to call, at 10:00 a.m., in Room

2318 of the Rayburn House Office Building, Hon. Vernon J. Ehlers

[Chairman of the Subcommittee] presiding.

2

3

1The proposed reductions to the STAR program described here and in detail below are compared to the President’s FY04 budget request. EPA’s final FY04 spending will not be available

until the agency’s operating plan is completed.

HEARING CHARTER

SUBCOMMITTEE ON ENVIRONMENT, TECHNOLOGY, AND

STANDARDS

COMMITTEE ON SCIENCE

U.S. HOUSE OF REPRESENTATIVES

Fiscal Year 2005 Environmental

Protection Agency Budget

THURSDAY, MARCH 11, 2004

10:00 A.M.–12:00 P.M. 2318 RAYBURN HOUSE OFFICE BUILDING

1. Purpose

On Thursday, March 11, 2004 at 10:00 a.m. the House Science Subcommittee on

Environment, Technology, and Standards will hold a hearing to examine the Environmental Protection Agency’s (EPA) fiscal year (FY) 2005 budget request that proposes steep cuts in the Science to Achieve Results (STAR) grants program. Managed

by EPA’s Office of Research and Development (ORD), the STAR program supports

research at colleges and universities on a wide array of environmental science

issues.

The hearing will examine why the Administration has proposed a $35 million (or

35 percent) reduction in the grant program and the consequences of the reduction.1

Specifically, the proposed reduction would reduce EPA-sponsored external (also

called extramural) research on pollution prevention and eliminate STAR funding for

research in the areas of ecological systems, endocrine disruptors, hazardous substances and mercury. The hearing will also examine the extent to which the Office

of Management and Budget’s (OMB) effort to assess the performance of government

programs under OMB’s Program Assessment Rating Tool (PART) led the Administration to propose these reductions.

The Subcommittee plans to explore the following questions:

1. Why does EPA’s FY05 budget request propose to eliminate funding for EPA’s

STAR research grants to colleges and universities for research on ecological

systems, pollution prevention, endocrine disruptors, hazardous substances,

and mercury? What are the consequences of these reductions?

2. To what extent did the OMB’s PART review of EPA’s ecological and pollution

prevention research programs drive these reductions?

3. How can the performance of environmental research programs best be measured? How do OMB’s efforts to assess the performance of EPA’s program differ from those of the National Academy of Sciences and EPA’s Science Advisory Board?

2. Witnesses

• Mr. Clay Johnson III, Deputy Director for Management, Office of Management and Budget.

• Dr. Paul Gilman, Assistant Administrator, Office of Research and Development, Environmental Protection Agency.

• Mr. Paul Posner, Managing Director, Strategic Issues, U.S. General Accounting Office (GAO); Managed GAO’s recent report, Performance Budgeting:

Observations on the Use of OMB’s Program Assessment Rating Tool for the

Fiscal Year 2004 Budget, GAO–04–174.

• Dr. Genevieve Matanoski, Professor, Johns Hopkins University, Chair,

EPA Science Advisory Board’s Review of EPA’s FY 2005 Budget Request;

former Chair, EPA’s Science Advisory Board (SAB).

4

2 A large portion of the STAR grants in ecological research support the measurement of ecological conditions and most of ecological condition research is carried out through the Environmental Monitoring and Assessment program (EMAP) program, a program that the Committee

has supported in recent years.

• Dr. Costel Denson, Professor, University of Delaware, member of the National Academy of Sciences panel that authored The Measure of STAR: Review of the U.S. Environmental Protection Agency’s Science to Achieve Results

(STAR) Research Grants Program (2003).

3. Brief Overview

The President’s budget for FY05 proposes to cut Science and Technology (S&T) at EPA by $92 million, or about 12 percent. The most significant percentage cut to

S&T (other than the elimination of earmarks) would reduce funding for EPA’s external grants program by almost 35 percent.

These proposed cuts to the grant program, known as Science to Achieve Results

(STAR), are especially noteworthy because in the last decade a number of outside

experts, including the National Academy of Sciences (NAS), have urged EPA to fund

research outside of its own laboratories. Most notably, the NAS praised the STAR

grant programs in its 2003 report, The Measure of STAR: Review of the U.S. Environmental Protection Agency’s Science to Achieve Results (STAR) Research Grants

Program. The report argues that the external STAR grants are a critical means for

the agency to have access to expertise that it does not have in-house, and to respond

quickly to emerging issues. The external grants have also often been favored by critics of EPA who view EPA’s in-house scientists as too likely to come up with research

results that would favor a pre-existing regulatory agenda.

The cuts to STAR have also attracted attention because they can be seen as a

test case of how the Administration is using a new system the Office of Management and Budget (OMB) has created to evaluate the effectiveness of federal programs and inform spending decisions. The system is known as the Program Assessment Rating Tool (PART). Each year, OMB is selecting programs at each agency to

evaluate using PART. It hopes to have reviewed all programs within the next four

years.

The proposed cuts to STAR raise two sets of questions about PART: How fairly

does the PART tool evaluate programs? And how does the Administration apply

PART evaluations in making budget decisions? The General Accounting Office

(GAO) recently weighed in on the first question with a report, Performance Budgeting: Observations on the Use of OMB’s Program Assessment Rating Tool for the

Fiscal Year 2004 Budget (see more detail below).

The proposed cuts to STAR raise particularly thorny questions about the use of

PART. Most problematically, STAR grants are part of larger programs targeted at

particular environmental issues. The STAR grants targeted for cuts are part of five

separate research programs: ecological systems, pollution prevention, endocrine

disruptors, mercury, and hazardous substances. The PART was applied to the first

two programs as a whole; the external grants were not evaluated as a separate element. The other three programs were not evaluated at all. Other issues about the

way PART relates to the STAR program and EPA’s research and development

(R&D) programs are described below.

The Five Programs—What They Do And What Would Be Cut

Ecological Research. EPA characterizes the $110 million ecological research program as a core or basic research program. Its goal is to develop the scientific understanding to determine ecosystem conditions and trends, diagnose impairments, forecast ecosystem vulnerability and, ultimately, restore degraded ecosystems. For example, recent STAR funded research has been instrumental in developing scientifically grounded ways to measure water and ecosystem quality2 along the Nation’s

coastal areas and in the mid-Atlantic region.

According to EPA, the $22 million cut to STAR would eliminate 50 grants in FY05

across all areas of the ecological research program. The proposed cuts would slow

research on water quality in FY05 in the Ohio and Mississippi River basin and

eliminate grants for research in such areas as western rivers and streams, the

Great Lakes, and the Gulf of Mexico.

Pollution Prevention. EPA’s pollution prevention research program develops tools,

technologies, and new systems for preventing pollution. Most of the $50 million program funds applied research, such as the development of innovative technologies for

reducing the use of hazardous solvents. However, a small portion of the program

supports extramural grants for more basic research on sustainable technologies. For

example, in work that EPA carries out in partnership with the National Science

5

3 If this were the case, and ORD were to be funded at $572, the level requested in the President’s FY05 budget request, STAR funding would then need to be between $86 million and $114

million. However, the President’s request for STAR in FY05 appears to be no more than $65

million, and may be even less than that.

Foundation, EPA supports research in so-called ‘‘green chemistry,’’ which promotes

safer chemicals and chemical manufacturing processes.

The proposed budget would redirect $5 million from research to a pollution prevention outreach program in another part of the agency. Redirecting these funds

would eliminate $3 million in STAR funding, which is EPA’s contribution to the

EPA–NSF partnership.

Endocrine Disruptors. EPA’s $13 million research program focuses on providing

a better understanding of the effects of endocrine disruptors and on humans, wildlife, and the environment. EPA also is developing new methods to screen chemicals

for their potential endocrine effects. The program invests in both basic and applied

research.

The proposed budget would cut $4.9 million, which would eliminate the entire

STAR grant research program on endocrine disruptors. The funds would otherwise

have supported research on the extent to which humans and wildlife are exposed

to endocrine disruptors, an area that the NAS and the World Health Organization

have identified as an important research gap.

Mercury. The goals of EPA’s $7 million mercury research program are to reduce

and prevent the release of mercury into the environment and to understand how

mercury moves through the environment. This research supports a variety of the

agency’s air and water regulatory programs by developing control technologies,

measuring mercury deposition, and attempting to understand the effect of mercury

on wildlife. The proposed $1.9 million reduction to the STAR grant portion of the

mercury research program would eliminate STAR research in FY05 on how and

where mercury moves through the environment.

Hazardous Substances. EPA established five university-based centers affiliated

with 22 universities to address concerns about hazardous substances in the environment. Each center has developed a research program to meet regional needs. Centers are based at Johns Hopkins University, Louisiana State University at Baton

Rouge, Purdue University, Colorado State University at Fort Collins and Oregon

State University at Corvallis. The proposed $2.3 million dollar reduction in STAR

funding would eliminate ORD’s contribution to these centers.

What is the Value of Extramural Research at EPA?

The NAS reviewed EPA’s STAR program in 2003. The report, The Measure of

STAR: Review of the U.S. Environmental Protection Agency’s Science to Achieve Results (STAR) Research Grants Program, reaffirmed earlier National Academy recommendations that EPA should maintain an appropriate balance between its intramural and extramural research programs. (See the Attachment A for a summary

of the report.) The report gave the STAR program a strong endorsement, calling it

‘‘EPA’s pre-eminent program that solicits independent scientific and technologic research from the Nation’s best academic and non-profit research institutions.’’ It also

described STAR as enabling EPA to have access to the broad research community,

fund research at the cutting edge of science, respond quickly to new issues, and address research gaps when EPA lacks the appropriate in-house expertise. It specifically praised the unique contributions that the STAR program is making to endocrine disruptors and ecological indicators research. The report concluded by recommending that ORD maintain STAR funding at a level somewhere between 15 and

20 percent of ORD’s total budget.3

What is the Program Assessment Rating Tool (PART) and what did GAO

conclude in its recent assessment of the PART?

The Program Assessment Rating Tool (PART) is a new evaluation tool developed

by OMB to assess the performance of federal programs and to link that performance

to spending decisions. It judges programs on their purpose and design, strategic

planning, management and results, heavily weighting the results portion of the review. Based on the review, OMB rates a program as either effective, moderately effective, adequate, ineffective, or ‘‘results not demonstrated.’’ (See Attachment B for

a more detailed description of the PART.) OMB plans to apply the tool to all federal

programs within the next four years.

The Government Accounting Office (GAO) recently reviewed the PART process,

though it did not specifically review its application to R&D programs (Performance

Budgeting: Observations on the Use of OMB’s Program Assessment Rating Tool for

the Fiscal Year 2004 Budget, GAO–04–174). (See Attachment C for a summary of

6

4 EPA has evaluated subprograms that cut across all of its research programs, such as the

STAR program, and elements of the ecological research program, such as Environmental Monitoring and Assessment Program. 5See OMB’s PART summaries, where it specifies reductions to these programs at http://

www.whitehouse.gov/omb/budget/fy2005/pdf/ap¥cd¥rom/part.pdf 6 EPA also disagrees with OMB’s conclusion that the ecological research program failed to coordinate its research agenda within EPA or with outside agencies or researchers (suggesting

redundancies with other programs), and that previous evaluations of the ecological research program were too focused on process measures.

the GAO report.) According to GAO, the PART process has reinvigorated the Executive Branch’s focus on performance budgeting. However, GAO also concluded that

OMB must do more to ensure fair and consistent application of the PART. GAO described the PART as ‘‘a work in progress’’ and needing ‘‘[a]dditional guidance and

considerable revisions. . .to meet OMB’s goal of an objective, evidence-based assessment tool.’’ GAO concluded that there are inherent challenges when applying the

PART’s restrictive yes/no format to programs with multiple purposes and goals, and

recommended, among other things, that OMB clarify subjective terminology, provide

flexibility in judging complex programs, clarify when output and outcome measures

are appropriate, and increase dialogue with agency staff on such things as the definition of the program that will be reviewed.

The way OMB applied the PART to EPA’s S&T programs may illustrate some of

GAO’s concerns. For example, many of EPA’s R&D programs have multiple goals

and purposes, such as combining basic and applied, as well as intramural and extramural research, in one program. In addition, OMB’s decision to evaluate the overall

ecological research program may have led to challenges for EPA, which had never

evaluated the entire program.4

What did the PART evaluation conclude about the ecological research and

pollution prevention programs and was the PART applied fairly?

OMB used the PART to evaluate two of the five programs with extramural research elements that were cut in the President’s budget. OMB concluded that the

ecological and pollution prevention research programs could not ‘‘demonstrate results,’’ because neither had adequate standards to measure the progress of the programs.5

EPA has said it disagrees with how OMB applied the PART and the conclusions

it reached. The primary area of disagreement appears to be over how to measure

the performance of EPA’s R&D programs.6 For example, EPA views the ecological

research program as more of a basic research program, making it similar to research supported by the National Science Foundation (NSF). In OMB’s PART review

of NSF programs, it measured the performance of those programs using process indicators, such as whether the agency has conducted the appropriate peer reviews

and how quickly it processes research grants. EPA maintains that its program

should be similarly evaluated.

OMB, however, seems to want more than process measures to evaluate EPA’s

R&D programs. According to both EPA and the public PART review documents,

OMB seems to want EPA to measure its programs less on process and more on output and outcome measures, such as the degree to which others used the products

of the research, and how much pollution the research reduced or might reduce in

the future. EPA agrees that its research strategic plans should make the connection

between research and eventual reductions in pollution. However, it does not believe

that either its basic or applied research programs should be held accountable for the

actions of others who are outside of EPA’s control and who may or may not use

EPA’s research products.

EPA seems to believe that its R&D programs should be evaluated through a peer

review process that considers the degree to which the research reflects the state of

the science, adds knowledge to the field, and creates tools and methods that others

could use. This view seems consistent with the views of EPA’s SAB and various

NAS reports, such as the STAR report, on the proper way to evaluate R&D. These

reports have also praised EPA’s for developing high quality basic, extramural research programs that develop knowledge, but are not tied to regulatory results.

Why was the ecological research and pollution prevention program cut and

who decided to cut it? Was the decision related to the PART?

The Administration’s proposed budget clearly shows that OMB decided to cut the

ecological and pollution prevention research programs because of their low PART

scores. OMB’s specific mention of the amount of the proposed reduction in the PART

summaries makes this readily apparent. However, what is less clear is why these

programs were cut when many other EPA programs could not demonstrate results

7

either. OMB evaluated a total of 20 EPA programs in FY04 and could not determine

results for 13 of them. Of those 13, some are proposed for decreases, some for increases and others for flat funding. The rationale for the uneven treatment is unclear.

Why was the STAR portion of these programs cut and who decided to cut

it?

OMB’s PART review did not address the STAR program, other than a brief mention of NAS’s positive review of the STAR-funded ecological research. As a result,

it appears that the STAR cuts may have emerged from EPA as it decided how to

allocate the overall reductions required by OMB. The ultimate reason, however, is

difficult to know because final decisions on reductions are usually made in negotiations between the agency and OMB, which are not made public.

4. Witness Questions

Questions for Clay Johnson III, Deputy Director for Management, Office of Management and Budget

In your testimony, please describe the justification for the proposed reductions to

EPA’s STAR grant research on ecological systems, pollution prevention, endocrine

disruptors, and mercury. In particular, please focus your testimony on the following

questions:

1. To what extent is the proposed reduction in each of these research areas

based on the evaluation of EPA’s research programs that OMB undertook

with the PART?

2. Given that the PART review did not specifically assess the extramural portion of the research programs, why was the extramural portion of the program cut?

3. Why does OMB’s PART review of the ecological research program tend to

treat it as an applied research program when EPA characterizes it more as

basic research? Is a single performance score under the PART tool appropriate for reviewing a diverse research program, such as EPA’s endocrine

disruptor research, that combines basic, applied, intramural and extramural

research? To what extent, should EPA’s R&D performance measures be

based on specific regulatory program outcomes or environmental outcomes?

4. Does OMB agree with the Government Accounting Office’s recommendations

for improving the PART process and its content? How will you implement

those recommendations?

Questions for Paul Gilman, Assistant Administrator, Office of Research and Development, U.S. EPA

Please give a brief description of the STAR extramural research program and how

it fits into EPA’s overall R&D program. In addition, please answer the following

questions:

1. Given the positive review by the National Academy of Sciences of the STAR

program last year, why was the STAR program cut?

2. Given the elimination of EPA’s STAR grants for research on ecological systems, pollution prevention, endocrine disruptors and mercury, does EPA now

believe it no longer important to seek out the expertise of university researchers in these fields.

3. How does EPA’s ecological research compare to research supported by the

National Science Foundation and other federal research programs? To what

extent does EPA coordinate its research with those agencies?

4. Does EPA characterize its research on ecological systems, pollution prevention, endocrine disruptors and mercury as basic or applied research?

5. What performance measures are most appropriate for evaluating these programs? To what extent does EPA believe that the performance measures for

these programs should be tied to the outcomes of specific regulatory programs or environmental outcomes?

6. What research would not be done as a result of the proposed reductions and

what impact would this have on our scientific understanding and EPA’s regulatory programs?

8

Questions for Paul Posner, Managing Director, Strategic Issues, U.S. Government Accounting Office.

In your testimony, please describe GAO’s findings and recommendations concerning the PART and answer the following questions:

1. What unique problems do research programs raise for evaluation tools like

PART? What types of evaluation techniques and performance measures are

most appropriate for reviewing basic research programs and what types are

most appropriate for applied research? How should OMB decide whether a

research program should be evaluated as a basic or applied program?

2. How should the PART deal with programs that have several distinct elements, for instance a single research program that funds both basic and applied, and intramural and extramural research?

Questions for Gene Matanoski, Professor, Department of Epidemiology, Johns Hopkins University; Chair, EPA Science Advisory Board FY05 Budget; Former

Chair, EPA Science Advisory Board

In your testimony, please describe the Science Advisory Board’s views on the proposed cuts to the STAR grant program and OMB’s PART review of the ecological

and pollution prevention research programs. In addition, please answer the following questions:

1. What has the SAB recommended to EPA in terms of balancing its research

and development (R&D) investments between intramural and extramural research and between basic and applied research? Are SAB’s recommendations

consistent with recommendations from other reviews of EPA’s science programs?

2. What performance measures are most appropriate for evaluating EPA’s research programs on ecological systems, pollution prevention, endocrine

disruptors and mercury? To what extent should performance measures differ

for basic and applied research programs at EPA? Should EPA’s R&D performance measures be tied to the outcomes of specific regulatory programs

or environmental outcomes?

Questions for Costel Denson, Professor, Department of Engineering, University of

Delaware; Member of the National Research Council panel for the report, The

Measure of STAR

1. How important are the extramural portions of EPA’s research efforts, including those for research on ecological systems, pollution prevention, endocrine

disruptors, and mercury? What are the likely effects the elimination of these

grants will have on our scientific understanding and EPA’s regulatory programs?

2. How important is it that EPA ensures that some portion of its environmental

research funding support extramural research? What portion is an optimal

amount?

3. What performance measures are most appropriate for judging the performance of EPA’s STAR grant program? To what extent should the STAR program be evaluated as basic research or applied? How well does the STAR

program perform relative to other federal research programs of similar design?

4. What actions should EPA take to strengthen its STAR research grant program?

5. Attachments:

Attachment A. Summary of the NAS Report on STAR

Attachment B. Summary of the PART program and process.

Attachment C. Summary of the GAO Report on PART

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

7There is no standard definition of program, though it is intended to capture a set of activities

clearly recognized as a program, having a discrete budget, or related to the level at which budget decisions are made.

Attachment B

Summary of the PART program and process

What is the Program Assessment Rating Tool (PART)?

The PART is the latest executive branch initiative designed to

better align spending decisions and program performance, often

called ‘‘performance budgeting. The current statutory framework

for performance budgeting is the Government Performance and Results Act, which became law in 1993. President George W. Bush

announced his strong support for linking performance and budget,

when he made it a major goal of the ‘‘President’s Management

Agenda.’’ A key element in accomplishing this objective is the Office of Management and Budget’s PART. The Administration views

the PART as enabling more effective implementation of GPRA, and

as aligned more closely with budgeting decisions than GPRA.

According to OMB, the PART is a diagnostic tool meant to provide a consistent approach to evaluating federal programs7 as part

of the budget formulation process. The PART, which is implemented by OMB’s budget examiners, requires an examiner to answer 25 yes/no questions under four overarching categories. Each

of the four categories is given a specific weight for determining an

overall numerical score for each program. The categories and their

weightings are: (1) program purpose and design (e.g., is the design

clear and purpose sensible?) (20 percent); (2) strategic planning,

(e.g., has the program set appropriate annual and long-term goals?)

(10 percent); (3) program management, (is there sound financial

and management oversight?) (20 percent); and (4) program results,

(has the program is met its annual and long-term goals?) (50 percent). OMB asks a few supplementary questions, which vary depending on what type of program is under review, such as block

grant, regulation, or R&D. Based on the total score, programs are

rated as either effective, moderately effective, adequate, ineffective,

or results not demonstrated.

OMB applied the PART to 234 programs in FY 2004 across all

federal agencies, and plans to rate nearly 100 percent of all remaining programs over the next four years. Of the 234 programs,

over 100 programs received ratings ‘‘results not demonstrated.’’ According to a recent GAO study, discretionary programs that received effective scores tended to see budget increases and programs

that received ineffective scores tended to receive budget decreases.

For programs that received ‘‘results not demonstrated’’ scores, the

budget story was more mixed. According to GAO, programs that received this score tended to indicate programs for which OMB and

the Agency could not agree on appropriate performance measures.

24

25

26

27

28

29

30

31

Chairman EHLERS. Good morning. Welcome to today’s hearing on

the Environmental Protection Agency’s Fiscal Year 2005 Science

and Technology Budget. I will try to be brief with my opening

statement so we can get right to the business of the day, and I will

also note that Congressman Miller is here filling in for Congressman Udall, who is the Ranking Member.

The role of science in technology at EPA is more important than

ever. EPA is being asked to address increasingly complex technical

questions in its regulatory processes and in its search for emerging

environmental problems. That is why I am surprised and concerned

at the Administration’s proposal to cut $22 million, which is nearly

12 percent, from EPA’s science budget. It is hard to understand

why, even in this tight budget time, the science budget would deserve such a substantial cut.

Today, the Subcommittee hopes to learn why the Administration proposes a cut of $35 million or 35 percent to EPA’s competitive external research grants to colleges, universities and other researchers, as well as find out the impact of those reductions. This proposed cut is in the Science to Achieve Results grant program, affectionately referred to as STAR grant. It would mean less extramural

research on ecological systems, pollution prevention, endocrine

disrupters and mercury, among other topics.

Today, we are also interested in the understanding of the extent

to which the Office of Management Budget’s effort to assess the

performance of governing programs under its new Program Assessment Rating Tool called PART led to these proposed cuts. As a

strong supporter of linking performance to funding decisions, I look

forward to a healthy discussion on how we can best evaluate environmental research and developmental programs. The PART is

playing an increasingly important role in the Administration’s

budget decisions. It has been applied to hundreds of federal programs in the last two years and will be applied to the remainder

over the next four. With this in mind, it is critically important for

this Committee and the Congress overall to understand how this

assessment tool is being used.

I would also like to mention my concern about the 8.3 million reduction the Administration proposes that would eliminate EPA’s

research on building decontamination. EPA has a unique role in

cleaning contaminated buildings and it is not clear why this work

is being cut, or whether another agency, such as the Department

of Homeland Security, will pick it up. The full Science Committee

plans to address this question further at our joint hearing with a

select Committee on Homeland Security on March 25.

We have an excellent panel of witnesses today, and I expect that

we will learn a great deal. I am looking forward to the testimony

that we will hear.

The Chair now recognizes Congressman Brad Miller from North

Carolina for an opening statement.

[The prepared statement of Mr. Ehlers follows:]

PREPARED STATEMENT OF CHAIRMAN VERNON J. EHLERS

Good morning. Welcome to today’s hearing on the Environmental Protection Agency’s fiscal year 2005 Science and Technology budget.

The role of science and technology at EPA is more important than ever. EPA is

being asked to address increasingly complex technical questions in its regulatory

32

processes and in its search for emerging environmental problems. That is why I am

surprised and concerned about the Administration’s proposal to cut $92 million,

which is nearly 12 percent, from EPA’s Science budget. It is hard to understand

why, even in this tight budget time, the science budget would deserve such a substantial cut.

Today, the Subcommittee hopes to learn why the Administration proposes a cut

of $35 million, or 35 percent, to EPA’s competitive, external research grants to colleges, universities and other researchers, as well as the impact of those reductions.

This proposed cut is in the Science to Achieve Results grant program, affectionately

referred to as STAR grants. It would mean less extramural research on ecological

systems, pollution prevention, endocrine disruptors and mercury, among other topics.

Today, we are also interested in understanding the extent to which the Office of

Management and Budget’s effort to assess the performance of government programs

under its new Program Assessment Rating Tool, called PART, led to these proposed

cuts. As a strong supporter linking performance to funding decisions, I look forward

to a healthy discussion of how we can best evaluate environmental research and development programs.

The PART is playing an increasingly important role in the Administration’s budget decisions. It has been applied to hundreds of federal programs in the last two

years and will be applied to the remainder over the next four. With this in mind,

it is critically important for this committee, and the Congress overall, to understand

how this assessment tool is being used.

I would also like to mention my concern about the $8.3 million reduction the Administration proposes that would eliminate EPA’s research on building decontamination. EPA has a unique role in cleaning contaminated buildings and it is not

clear why this work is being cut, or whether another agency, such as the Department of Homeland Security, will pick it up. The Full Science Committee plans to

address this question further at our joint hearing with the Select Committee on

Homeland Security on March 25th.

We have an excellent panel of witness today and I expect that we will learn a

great deal. I’m looking forward to the testimony.

Mr. MILLER. Thank you, Mr. Chairman.

I am pleased to be here this morning to open this hearing in Mr.

Udall’s absence. I am also very pleased to hear how much common

ground there is between my views and those just expressed by the

Chair.

I represent many of EPA’s scientific staff who work at the Research Triangle Park in North Carolina. My interest, however, is

more than parochial. I think I have the same interest in EPA’s research and development funding that every American has or should

have. In the 1960s, the Federal Government moved several air pollution-related environmental research facilities to the Research Triangle Park in North Carolina. This later evolved into a campus for

the EPA, consisting of 400 laboratories. The laboratories focus on

a wide range of research issues facing our nation, including air,

health, exposure and environmental information. The Research Triangle is home to the EPA’s largest operation outside of Washington, with a labor force of more than 2,500, and a $250 million

annual contribution to North Carolina’s economy.

You do not have to spend a lot of time on environmental issues

to realize that they are very complicated and often very contentious. We have made great strides in balancing and recognizing the

need to balance and in balancing economic development and environmental protection, and the ability to strike that balance is

largely because of our investment in research and development.

Protection of the environment is central to the protection of public

health. Clean air and clean water are not luxuries. They are necessities. As a former member of the North Carolina legislature, I am

very much aware of the problems caused by air pollution. Every

33

summer, air pollution causes 240,000 asthma attacks, 6,300 emergency room visits, and 1,900 hospital admissions in North Carolina.

Almost one in five emergency room admissions in North Carolina

is for pediatric asthma. That is a terrified parent taking their child

to the emergency room because their child cannot catch their

breath.

I will continue to work to improve that environmental health

problem to protect North Carolina’s children and North Carolina

citizens against air pollution. I cannot reconcile the right to clean

air and clean water and the constant call to base regulations and

environmental policy on sound science with the Administration’s

budget request for this year for research and development at EPA.

The request essentially guts R&D funding. From her written testimony, I understand that Dr. Matanoski of EPA’s Science Advisory

Board shares that opinion.

The Administration has requested cuts in research and development funding for EPA to about 12 percent below the appropriated

level for this year. Now last year’s request proposed a cut to EPA’s

Science and Technology accounts by almost five percent. The Administration is obviously going in the wrong direction.

The specific program cuts contained in this budget are unacceptable and difficult to understand. EPA’s extramural grant program,

which I believe the Chairman just mentioned, the Science to

Achieve Results program, which has received high marks in external reviews by the Science Advisory Board and the National Academy of Sciences, is rewarded with a disproportionate cut of about

one-third of its budget. During the same month, the Senate was

closed due to contamination from ricin; this Administration released this budget terminating its research on building decontamination.

This budget request does not serve the needs of my constituents,

I think the constituents of many on the Committee or in the Congress, or maintain a healthy research and development program

within the EPA. If this is the result of applying the Office of Management and Budget’s new budget evaluation tool, the PART, clearly, the tool is deeply flawed.

I do welcome our panel of experts, our witnesses, and I look forward to hearing your testimony today. Thank you.

[The prepared statement of Mr. Miller follows:]

PREPARED STATEMENT OF REPRESENTATIVE BRAD MILLER

I am pleased to be here this morning to open this hearing. I have the privilege

of representing many of EPA’s fine scientific staff that works at Research Triangle

Park in North Carolina. As such, I have not only a general interest in EPA’s research and development funding, but a hometown one as well.

In the 1960’s, the U.S. government moved several air pollution-related environmental research functions to Research Triangle Park, North Carolina. This later

evolved into a campus for the EPA consisting of 400 laboratories. The laboratories

focus on a wide range of research issues facing our nation including air, health, exposure and environmental information. The Research Triangle is home to the EPA’s

largest operation outside of Washington with a labor force of more than 2,500, and

a $250 million annual contribution to North Carolina’s economy.

You don’t have to spend much time on environmental issues to recognize they are

contentious and complex. We have made great strides in balancing economic development and environmental protection because we have invested in research and development. Protection of the environment is central to the protection of public

health. Clean air and clean water are not luxuries. They are necessities. As a former

34

member of the North Carolina legislature, I am acutely aware of the problems

caused by air pollution. Every summer, air pollution causes 240,000 asthma attacks,

6,300 ER visits, and 1,900 hospital admissions in our state. I will continue to fight

this environmental and health problem.

I cannot reconcile the right to clean air and water and the constant call to base

regulations and environmental policy on sound science with the Administration’s FY

2005 request for research and development at EPA. The request guts R&D funding.

I note from her written testimony, that Dr. Matanoski of EPA’s Science Advisory

Board shares my opinion.

The Administration’s request cuts research and development funding for EPA by

about 12 percent below the appropriated level for this year. Last year’s request proposed a cut to EPA’s Science and Technology accounts by almost five percent. The

Administration is going in the wrong direction.

The specific program cuts contained in this budget are unacceptable and difficult

to understand. EPA’s extramural grant program—the Science to Achieve Results

program—which has received high marks in external reviews by the Science Advisory Board and the National Academy of sciences is rewarded with a disproportionate cut of about one third. During the same month the Senate was closed due

to contamination from ricin, the Administration released this budget terminating its

research on building decontamination.

This budget request does not serve the needs of our constituents or maintain a

healthy research and development program with the EPA. If this is the result of

applying the Office of Management and Budget’s new budget evaluation tool—the

PART—clearly this tool is deeply flawed.

I welcome our panel of witnesses and I look forward to hearing your testimony.

Chairman EHLERS. If there is no objection, all additional opening

statements submitted by the Subcommittee Members will be added

to the record. Without objections, so ordered.

I have to issue a correction. My staff said that I referred to the cut as $92 million in my opening statement. I referred to it as $22

million. It is actually $92 million. I hope that will be my only senior moment today, but I wanted to make clear that it is a very

large amount of money.

At this time, it is my pleasure to introduce our witnesses. We are

honored today to have with us Mr. Clay Johnson, the Deputy Director of Management at the Office of Management and Budget.

We welcome you. It is good to see you in this role. I spent many

times on the telephone with you in your first job in the personnel

office, trying to recruit good scientists to work in the government

and the Bush Administration, and it is a pleasure to see you in this

role now, so thank you for coming.

Next, we have Dr. Paul Gilman, who is no stranger in this room.

He has appeared here frequently. He is the Assistant Administrator for Research and Development at the United States Environmental Protection Agency, of course, always known by its acronym

EPA.

Next, Mr. Paul Posner. Is it Posner or Posner? Posner. Posner.

Sorry. Mr. Paul Posner is the Managing Director for Budget and

Intergovernmental Relations at the General Accounting Office,

which we all know by GAO. He offers—oversaw GAO’s recent report on how PART is being implemented.

Next, we are pleased to have Dr. Matanoski, who is a Professor

in the Department of Epidemiology at Johns Hopkins University.

She also chaired the EPA Science Advisory Board Review of EPA’s

Fiscal Year 2005 Budget. While I introduce Dr. Matanoski, I would

also like to recognize Dr. William Glaze, who is the current chair

of EPA’s Science Advisory Board, who has helped lead the budget.

And finally, Dr. Denson is a Professor in the Department of Engineering at the University of Delaware. He is also a Member of

35

the National Academy of Science’s Panel that reviewed the EPA’s

Science to Achieve Results Research Grants Program, better known

by its acronym STAR. And now I would like to introduce not only

the witnesses, but all the acronyms. We can soon proceed to the

testimony. Our witnesses I presume have been informed that spoken testimony is limited to five minutes each, after which the

Members of the Committee will have five minutes each to ask

questions. If you cannot complete your statement, obviously, your

written statement will go in the record. If you do not feel you have

had a chance to say everything you wanted to say in your opening

oral statement, you will have ample opportunity during the questions to raise any points you believe should be raised.

At this point, we will open our first round of questions. The

Chair recognizes himself for five minutes. Oh, I am sorry. That is

my second senior moment. It is a bad day today. I was ill last

night, but that is no excuse. A Member of the Minority wants me

to lose count. I suppose he would like me to forget who is the minority here too. Okay. We will start with Mr. Johnson.

STATEMENT OF MR. CLAY JOHNSON, III, DEPUTY DIRECTOR

OF MANAGEMENT, OFFICE OF MANAGEMENT AND BUDGET

Mr. JOHNSON. Mr. Chairman, Mr. Miller, thank you for having

me.

We, all of us, are interested and working very hard to make the

Federal Government results oriented, and I would suggest, Mr.

Miller, that in fact, we are hitting in the right direction, not the

wrong direction. The right direction is let us pay attention to the

results we are getting for the money we are spending, and not how

much money we are spending. And I think this is your interest and

our interest as well, as I think this is the direction that in fact the

Federal Government is heading.

EPA is one of the leaders in the Federal Government at focusing

on results. They are particularly strong in the financial management area in the way we keep score, the score card. They are one

of the few agencies that have achieved green, which just means

they are using accurate and timely financial information with

which to make regular, frequent management decisions. So they

are one of the agencies we are very, very proud of in terms of the

accomplishments they have made.

We are here to talk about two things, as I understand it. One

of them is the PART and one of them is the specific decision that

has been with regard to ecological research and the Pollution Prevention Program. The PART helps agencies look at their programs

with consistency. I would suggest that it is a valuable tool now. I

believe this is GAO’s assessment of the program, and it is a tool

that will get—it is a device that will get better every year. There

is nothing magic about the PART. There is nothing sacrosanct

about the 25 or 30 questions we ask. What is magic about it is it

is a device now that will get better over time that—with which we

can look at programs and ask ourselves key management structure

and results, questions about programs in a consistent fashion, and

focus more and more with increasing proficiency as to what we are

getting for our money. Nothing happens automatically, as a result

of a PART evaluation, but the information that comes out of the

36

PART assessment is used to help inform decisions about how to

better manage, better structure, and better fund programs, including research programs.

The two programs in particular that are the primary focus of this

hearing, the Ecological Research and Pollution Prevention Programs, were assessed with the use of the PART. They were considered to be less results-oriented than they could be or than other

programs of a similar nature. In fact, EPA’s Prevention, Pesticides

and Toxics Program were considered to be most results-oriented,

and in fact, the money that is recommended not be spent, the reductions that were recommended for these two programs in fact is

being put in this other program, where the belief is that the return

for the taxpayer on the $30 million will be greater than it would

be if it was spent in the other programs. These two programs are

well-funded at the recommended 2005 levels. They are reasonably

well-funded programs. Therefore, we believe we do not significantly

impair the focus of these research programs with these reductions,

but we do produce a greater return for the taxpayer.

The PART was used to inform this budget proposal, but there

was nothing automatic that flowed out of this PART assessment.

These programs were not as results-oriented as they could be, or

we thought should be. The other obviously major part that went in

this funding recommendation was the very tight fiscal situation we

find ourselves in, trying to produce very, very tight budgets, not related to Homeland Security and Defense. But I look forward to

your questions later on. Thank you.

[The prepared statement of Mr. Johnson follows:]

PREPARED STATEMENT OF CLAY JOHNSON, III

Introduction

Thank you, Mr. Chairman, Members of the Committee, for inviting me to testify

this morning. I want to discuss with you our assessment of the Environmental Protection Agency’s (EPA) research programs and describe how the President’s Management Agenda helps federal agencies get greater results on behalf of the American

people.

We, all of us, are in the process of making the Federal Government results-oriented. We here in Washington tend to focus on the amount of money we’re spending

as a validation for how much the Federal Government is committed to an objective.

As a part of becoming results-oriented, however, we are now focusing more heavily

on the results we achieve on behalf of the American people. With just a little help

from OMB, agencies are asking whether they are achieving their objectives as effectively and efficiently as possible. EPA is a leader in this effort.

EPA is as advanced as any agency in government in having and using accurate

financial information to make day-to-day decisions about program management. For

example, EPA negotiates performance commitments with its grantees and provides

resources based on those commitments. EPA regularly monitors grantees’ performance and expenditures and, if a grantee isn’t meeting its commitments, EPA may

withhold resources from the non-performers and redirect those resources to grantees

that are meeting their commitments.

The Program Assessment Rating Tool

Applying the Program Assessment Rating Tool (PART) is one of the ways we are

becoming results-oriented. The PART is a series of questions that assesses the purpose, strategic planning, management, and performance of individual programs.

Programs must demonstrate that their purpose is clear, that they set aggressive,

outcome-oriented long- and short-term goals, that they are well managed, and that

they achieve results. With this tool, we are assessing the performance of every federal program, and if it is not working as intended, we are trying to do something

about it.

37

The Administration has used the PART to assess 400 programs so far, representing approximately $1 trillion in federal spending. We are using these assessments not only to guide our budget decisions, but also to improve the performance

and management of the government’s programs. The purpose of asking whether programs are working is to figure out how to fix them, not whether to spend more or

less on them.

Ecological Research and Pollution Prevention PARTs

As you know, OMB and EPA assessed EPA’s Ecological Research program and

Pollution Prevention and New Technologies program using the PART. According to

the assessment, the Ecological Research program:

• did not adequately coordinate the expenditure of resources with other EPA

offices or other agencies;

• lacks adequate annual measures of its performance; and

• does not have sufficient evaluations of its performance.

Like nearly 40 percent of the programs evaluated using the PART, the principal

finding for the program was the lack of adequate performance measures. Therefore,

EPA has committed to finding the right measures for this important program. The

President has requested $110 million for this program in his FY 2005 Budget, down

from $132 million in FY 2004.

According to the assessment, the Pollution Prevention and New Technologies program:

• has not addressed findings made by independent evaluations; and

• has not developed adequate measures of its performance.

As a result of these findings, EPA has committed to developing adequate performance measures and addressing findings made in previous independent evaluations.

The President has requested $36 million for this program in his FY 2005 Budget,

down from $42 million in FY 2004.

Why reduce funding for these programs?

Both the Ecological Research and Pollution Prevention programs were ‘‘unable to

demonstrate results,’’ which clearly influenced funding decisions related to the programs. Especially in a year like this one, when resources are constrained, we should

be directing resources to those programs that can achieve the most for the money.

EPA and OMB used the PARTs for the Ecological Research and Pollution Prevention programs as one factor in making budget decisions about those programs and

to focus resources on the programs most effective in helping EPA accomplish its

mission.

As I’ve mentioned, the Pollution Prevention research program could not show

whether the tools it is developing are used by industry, and, if so, to what extent

they are used. Also, previous independent evaluations of the Pollution Prevention

research program concurred with the PART review, especially in the areas of strategic planning and measurable results. On the other hand, a similar program in the

EPA’s Office of Prevention, Pesticides, and Toxics was able to show that industry

reduced its use and emissions of toxic chemicals through the use of tools and methods developed by the program. We consider reductions in pollution to be one of the

highest-level outcomes of an environmental program’s performance. Therefore, we

redirected funds to the pollution prevention program so EPA can continue to achieve

pollution reduction, thereby positively impacting the quality of public health and the

environment. Despite redirection of a small amount of funds from pollution prevention research to OPPTS’s program, the Administration maintained a large amount

of funding for the pollution prevention research program to assist it, among other

things, in developing performance measures.

This is our rationale for funding decisions related to EPA’s research programs. I

will leave to Dr. Gilman a more robust discussion of how these funding decisions

were applied to specific components of the research programs.

Research and Development and the Investment Criteria

The Government’s investment in research & development, not only in the environmental arena but elsewhere, is substantial. But in a time of constrained resources,

it is imperative that we invest in R&D wisely. In recognition of the special challenges that measuring R&D programs present, and leveraging work done by the National Academies of Science, the Administration developed its R&D Investment Criteria, which were incorporated into the PART. These criteria are some of the things

we look at when assessing the value of particular R&D programs:

38

• Relevance. Programs must be able to articulate why they are important, relevant, and appropriate for federal investment;

• Quality. Programs must justify how funds will be allocated to ensure quality;

and

• Performance. Programs must be able to monitor and document how well the

investments are performing.

As noted in our PART evaluations, the programs we assessed could improve the

ways they measure their performance. The three EPA programs we assessed cover

important issues, and receive funding totaling approximately $210 million. We

strongly believe that programs with federal funding of this magnitude should be

able to monitor and document how these investments are performing. There are

other equally important programs that are receiving similar levels of funding, but

whose results are more measurable. For example, the Department of Energy’s Wind

Energy program, with proposed FY 2005 funding of $42 million, can demonstrate

its contributions to the commercial success of wind energy use throughout the

United States. The Federal Aviation Administration’s Research, Engineering and

Development program, with proposed FY 2005 funding of $117 million, has set a

long-term goal to produce turbulence forecasting products that allow pilots to avoid

hazardous flight conditions while improving safety and ensuring efficient airspace

use.

The Future of the PART

The PART is a vehicle for improving program performance. It is just a tool to

achieve the goals laid out by Congress in the Government Performance and Results

Act (GPRA). GAO has made a number of recommendations for improving the PART,

the vast majority of which we agree with and are addressing. For example:

• With respect to centrally monitoring PART recommendations, we have provided a simple format for agencies to follow when reporting the status of recommendation implementation to OMB and I receive these reports semi-annually. We will continue to refine this process so that sufficient attention is

given to recommendation follow-up.

• As the PART relies on separate evaluations of evidence of a program’s success, we agree with GAO that the judgment about what constitutes a sufficient evaluation should be based on the quality, in addition to the independence, of the evaluation.

• One of the greatest opportunities for the PART is to compare the performance

of, and share best practices among, like programs across government. We will

continue to use the PART for that purpose.

• We will continue to improve agency and Executive Branch implementation of

GPRA by insisting GPRA plans and reports meet the requirements of this important law and the high standards set by the PART.

• We are clarifying the PART guidance so that it is well understood by those

who have to use it, as well those who have to administer it. We will continue

to assess completed PARTs to ensure they are completed consistently by

agencies and OMB.

Conclusion

The PART is a valuable tool now, as the General Accounting Office and others

have asserted, and it will get better each year. As more and more program assessments are conducted, the vast majority of budget and management decisions will

be significantly influenced by information about how programs are performing.

Agencies, including EPA, will be better able to describe to Congress and the taxpayer what his or her funding is purchasing and will be managing so that each year

improvements in efficiency and service delivery can be documented.

BIOGRAPHY FOR CLAY JOHNSON, III

Clay Johnson is the Deputy Director for Management at the Office of Management and Budget. The Deputy Director for Management provides government-wide

leadership to Executive Branch agencies to improve agency and program performance. Prior to this he was the Assistant to the President for Presidential Personnel,

responsible for the organization that identifies and recruits approximately 4,000

senior officials, middle management personnel and part-time board and commission

members.

39

From 1995 to 2000, Mr. Johnson worked with Governor George W. Bush in Austin, first as his Appointments Director, then as his Chief of Staff, and then as the

Executive Director of the Bush-Cheney Transition.

Mr. Johnson has been the Chief Operating Officer for the Dallas Museum of Art

and the President of the Horchow and Neiman Marcus Mail Order companies. He

also has worked for Citicorp, Wilson Sporting Goods and Frito Lay.

He received his undergraduate degree from Yale University and a Master’s degree

from MIT’s Sloan School of Management. In Austin, he helped create the Texas

State History Museum, and was also an Adjunct Professor at the University of

Texas Graduate School of Business. In Dallas, he served as President of the Board

of Trustees for St. Marks School of Texas, and as a Board Member of Equitable

Bankshares, Goodwill Industries of Dallas, and the Dallas Chapter of the Young

Presidents Organization.

Chairman EHLERS. Thank you.

Dr. Gilman.

STATEMENT OF DR. PAUL GILMAN, ASSISTANT ADMINISTRATOR FOR RESEARCH AND DEVELOPMENT, UNITED

STATES ENVIRONMENTAL PROTECTION AGENCY

Dr. GILMAN. Thank you, Mr. Chairman, Members of the Committee. Thanks for the opportunity to be here. I am here as the Assistant Administrator for the Office of Research and Development.

I also serve as Science Advisor for the Agency.

The Office of Research and Development is a service element to

the various programs of the Agency. It does both basic and applied

research. Roughly half basic, half applied. It does research in

human health, public health and environmental and ecological

sciences. Again, approximately a 60/40 split, in terms of human

health and environmental health. There is a great emphasis on

quality in the research program. It ranges from the very earliest

stages of planning. We have instituted a planning process that develops multi-year plans for each of the major research areas. Those

plans are developed in collaboration with our customers, if you will,

the various programs of the agency; water, air, land, and coordinated with other federal agencies and outside entities, as well.

We have gone from an Agency that was viewed as a laggard in

the use of peer-review to, I think today, an agency viewed as a

leader in the use of peer-review, and the Extramural Grants Program, as you will hear later from the witness representing the National Research Council, is a well-respected program as well. We

place a great emphasis on collaborative research with other federal

agencies in order to stretch our dollars and to avail ourselves of

their expertise. Probably a very good example of that collaborative

effort is a report we just released two days ago on the health of

our nation’s coastal waters, where we collaborated with 28 different

states, a number of federal agencies; NOAA, Department of Interior, Agriculture, to really produce the first scientifically supportable assessment of the quality of coastal waters that we have had

in a comprehensive way.

The programs of particular matter, ecological research and pollution prevention all have tough goals set for them by our Agency,

and embedded in our multi-year plans. Like providing the data and

tools to predict, measure, reduce and meet the standards for particulate matter, as well as producing that report I just mentioned on

the coastal health of our ecosystems. Yet, we are still challenged

to provide measures that truly get into the performance of those

40

programs. I am committed to working with the Office of Management and Budget and others to create long-term annual and efficiency measures that capture the important work our programs are

doing. In the end, these measures will help advance our programs

by demonstrating the value of our achievements very clearly.

Let me say something about some recent accomplishments to

demonstrate the breadth of the work that we are doing. These

goals and accomplishments all draw on the STAR program that is

the subject of this hearing. We have recently done work that is directly applicable to protecting water quality, a mission for the

Agency, that also has Homeland Security implications towards better understanding of how water distribution systems actually work,

not the pre-treatment but the post-treatment side of the water system. We are working to improve our air models, to improve their

performance and their accuracy. We are working to develop DNAbased technologies for the identifications of things like common

household molds that are deleterious to our health. And a development like that is the kind of thing we are very proud of because

we are in the process of licensing it to a number of companies, with

over 15 companies licensing that particular technology.

Again, Mr. Chairman, it is a challenge for us to relate the results

of our research on inherently long-term research to measures for

environment and public health outcomes. We are committed to

doing it. We have been trying to develop some approaches in collaboration with our Inspector General. We are also very interested

in continuing our discussions with the Office of Management and

Budget to utilize some different approaches to the evaluation of

these programs, and I think they will prove to be effective.

Thank you, Mr. Chairman.

[The prepared statement of Dr. Gilman follows:]

PREPARED STATEMENT OF PAUL GILMAN

Introduction

Good morning Mr. Chairman and Members of the Subcommittee. I am honored

to appear before you today to discuss the Fiscal Year (FY) 2005 budget request for

the U.S. Environmental Protection Agency’s (EPA) Office of Research and Development (ORD), and to share with you the uniqueness and success of ORD’s research

program from my perspective as both the Assistant Administrator for ORD and the

EPA Science Advisor.

The President’s FY 2005 budget request for ORD is $572.2 million. This includes

funding for ORD’s in-house program carried out by 1,975 employees, who account

for 11 percent of EPA’s workforce. In addition, the budget request supports our

Science to Achieve Results (STAR) research grants program. Together, our in-house

and STAR programs allow our nation’s brightest scientists to apply their talents and

knowledge to solve environmental science problems. My testimony highlights the

contributions we and our partners have made and describes changes to the Agency’s

research budget for STAR research in FY 2005.

ORD’s Unique Contributions

ORD conducts leading-edge research and fosters the use of science and technology

in environmental decisions in support of EPA’s mission to protect human health and

safeguard the environment. This research tackles problems to which solutions will

have both immediate and long-term public health and environmental benefits. The

advancement of science and the development of answers to questions posed by environmental issues makes ORD unique among federal research agencies. No other federal agency has a comprehensive research program devoted to improving our understanding of both public health and environmental impacts. No other agency is researching these issues in an integrated fashion. In addition, no other agency can

claim as large an impact on ensuring EPA’s decisions are informed by the strongest

possible science. To further strengthen our science program, EPA has been impleVerDate 11-MAY-2000 11:30 Dec 21, 2004 Jkt 092340 PO 00000 Frm 00044 Fmt 6633 Sfmt 6621 C:\WORKD\ETS04\031104\92340 SCIENCE1 PsN: SCIENCE1
41

menting the National Research Council (NRC) recommendations in its 2000 report,

‘‘Strengthening Science at the U.S. Environmental Protection Agency: Research

Management and Peer Review Practices,’’ as I will describe below. In sum, ORD is

conducting leading-edge research that informs the risk-based environmental decision-making of EPA’s program and regional offices and helps States and Tribes decide how best to implement these policies.

Ensuring these decisions are based on sound science requires relevant, high quality, integrated, leading-edge research in human health, ecology, pollution prevention

and control, and socio-economics. To maintain both short- and long-term relevance

to EPA’s mission, ORD’s scientific research activities are mainly focused on applied

research, which is problem-driven and, to a lesser extent, basic research. To ensure

the quality of our research program, ORD uses a coordinated, cooperative research

planning process; rigorous, independent peer review; and interagency partnerships

and extramural grants to academia that complement EPA’s own in-house scientific

expertise. We have a uniquely integrated research program in that we address both

human and ecological endpoints, conduct research across the risk assessment/risk

management paradigm, have expertise across scientific disciplines and within the

different environmental media, and draw from expertise in other agencies, organizations, and academia. Lastly, ORD keeps a leading edge in research by focusing our

efforts and resources on those areas where EPA can add the most value toward reducing uncertainty in risk assessments and enhancing environmental management.

The following are a few examples of our more recent accomplishments. ORD researchers:

• Collaborated with the Department of Homeland Security, Department of Energy, Department of Defense, and Centers for Disease Control to strengthen

water security, develop rapid risk assessment techniques, and develop building decontamination methods.

• Partnered with 24 marine coastal States, four territories, and other federal

agencies through the Environmental Monitoring and Assessment Program’s

National Coastal Assessment, to conduct sampling of estuaries using probabilistic methods.

• Collaborated with EPA’s Office of Environmental Information to deliver the

draft Report on the Environment, the first-ever national picture of U.S. environmental quality and human health using science-based indicators.

• Developed the Computational Toxicology Program, which has moved EPA to

the leading edge in the use of genetics, genomics, and computation for environmental protection.

• Completed an evaluation of Superfund clean-up technologies citing 143 successfully demonstrated technologies and $2.6 billion in total inflation-adjusted

cost savings.

• Continued our tradition of leadership in the use of external scientific expertise to enhance the quality and relevance of our scientific products, by relying

on the processes of peer participation and peer review.

I am proud of these accomplishments and the others I will identify later. They

are the direct result of careful research planning that relies on the active involvement of the Agency’s program and regional offices, as well as outside peer input.

Research Planning

The President’s budget request for FY 2005 will allow us to build upon these accomplishments by continuing a research program that directly serves EPA’s mission.

EPA’s science and technology efforts are aligned with the Agency’s strategic goals,

and we now have gone a step further by including science objectives within each

of EPA’s five strategic goals. ORD created these science objectives in collaboration

with EPA’s program and regional offices, to ensure that we produce the right scientific and technical information to meet EPA’s programmatic needs and thereby

advance the Agency’s mission.

The alignment of our science and technology program with EPA’s strategic goals

is carried forward into ORD’s planning of our research and development program.

We have divided our R&D program into topical areas, each of which is guided by

a multi-year research plan (the plans can be found at www.epa.gov/osp/myp.htm).

Each multi-year plan contains long-term research and development goals for the

next 5–10 years that tie back to EPA’s strategic goals, and are supported by annual

performance goals and measures. Every multi-year plan, and the goals and measures that comprise the plan, is developed in concert with colleagues across EPA and

in consultation with our stakeholders and the broader scientific community. The

plans also undergo expert, external peer review by EPA’s Science Advisory Board

42

(SAB) and ORD’s Board of Scientific Counselors (BOSC). Both groups have endorsed

this research planning process.

The multi-year plans are ‘‘road maps’’ that mark the progress our research programs have already made, as well as lay out the new directions we are taking to

adjust as changes occur in the complex scientific landscape ahead. Developing this

road map requires identifying a logical progression of scientific research to be contributed by EPA and its partners. This progression is defined in each multi-year

plan using ‘‘logic models’’ that demonstrate how research results contribute to EPA’s

desired long-term outcomes of improved human and ecosystem health. By following

the logic diagram, one can begin to see how each research project contributes to the

achievement of the long-term outcome. For illustration purposes, I have attached

the logic diagram from our Particulate Matter (PM) multi-year plan. I discuss logic

models in greater detail later in this testimony.

The multi-year research plans help EPA maintain its focus on high-priority

science issues. They also assist in coordinating research efforts across the environmental science community, including other federal entities; State, tribal, and local

governments; international organizations; and academia. Such coordination is essential. EPA’s science and technology budget is only a small fraction of the total annual

expenditures on environmental research, so leveraging our efforts with others—and,

most important, identifying the appropriate niche for EPA’s science and technology

programs—is necessary for our doing the right science in a fiscally responsible manner.

Independent scientific bodies have lauded EPA’s process for planning its research

efforts. In its 2000 publication, Strengthening Science at the U.S. Environmental

Protection Agency, the National Research Council stated, ‘‘Our committee expects

that ORD’s recent efforts in multi-year planning will contribute greatly to research

program continuity and the achievement of strategic goals, and ORD merits commendation for these initiatives.’’ Four years later, I can state with confidence that

our research planning process is meeting—and perhaps exceeding—the NRC’s expectations.

I wish to discuss two of our research programs—airborne particulate matter and

ecosystem protection—to illustrate how EPA’s science complements the scientific

work of others, to advance scientific understanding and inform the decisions that

solve environmental problems. Both of these research programs were evaluated

using the Program Assessment Rating Tool (PART). The principles and practices applied in the particulate matter and ecosystem protection programs are those used

in each of EPA’s research and development programs.

Particulate Matter

Among the most serious environmental problems affecting the health of Americans is exposure to airborne particulate matter. Based on the best science available

to us, these exposures contribute to the premature deaths of tens of thousands of

Americans annually, as well as the hospitalization of children and adults for diseases such as asthma. This has been documented in the Office of Management and

Budget’s (OMB) ‘‘Thompson Report’’ (68 Fed. Reg. 5492, 5499 (2003) ). To protect the

public against these effects, the Clean Air Act calls for the promulgation and periodic review of National Ambient Air Quality Standards, or NAAQS. In the late

1990s, after such a review yielded new standards for fine PM (particles less than

2.5 microns in diameter), Congress authorized and appropriated funds to EPA for

a greatly expanded PM research program, to be guided by advice from the National

Research Council. I would like to describe how we have organized this program and

share what we have learned.

To deliver the best science needed to inform sound public policy decisions, we

have worked with our Agency partners in the Office of Air and Radiation and the

regions to develop a multi-year plan for PM research that looks forward a little

more than a decade. This plan, which will be peer reviewed by the EPA Science Advisory Board later this year, describes research activities in two major areas: (1) PM

health effects and exposure, to guide future reviews of the NAAQS to refine the type

and amount of PM that needs to be controlled to protect public health; and (2) implementation tools, so that EPA, the States and the tribes, and the private sector

can ensure that these standards are met.

The PM multi-year plan integrates the strengths of our in-house scientists with

those of the external scientific community, through the extensive use of our STAR

research grants program, including the support of five PM Research Centers. In addition, EPA’s researchers are coordinating their efforts with others in the public and

private sectors, both domestically and internationally. For example, health research

is being conducted overseas by several organizations, while in the United States,

studies are being supported by industrial organizations including the Electric Power

43

Research Institute and the Coordinating Research Council through their support of

the Health Effects Institute (co-funded by EPA). Recently, EPA, the National Institute of Environmental Health Sciences, and the National Heart, Lung, and Blood

Institute co-sponsored a workshop on the cardiovascular effects of environmental

pollutants, and planning is now underway to develop joint Requests For Applications in the area of cardiovascular effects of PM exposure. Through these and other

mechanisms, EPA contributes to and keeps abreast of the scientific advancements

and initiatives in the PM area.

What have we learned since the setting of the 1997 NAAQS? Here are a few examples:

• In 1997, questions were raised about the legitimacy of findings showing associations between centrally-monitored PM and health effects. We now understand that these monitors actually do a good job at estimating population exposures, which has lent further credence to the health associations found in

epidemiologic studies.

• While we knew of these associations between PM and increased mortality in

1997, we were at something of a loss to explain them biologically. Due to

work done by both ORD in-house and STAR-supported extramural scientists,

we now have several plausible hypotheses for the biological mechanisms leading to those associations, including recent findings showing an effect of PM

directly on the heart.

• In 1997, we had a poor understanding of the chemical composition and size

distribution of PM that correlated with health effects. Today, we have detailed profiles of the PM associated with many significant sources and geographic areas, and we continue to refine our understanding about the specific

types of sources responsible for these public health risks.

While EPA’s PM research program has been a success, there continues to be more

to learn, as described in the PM multi-year plan. One focus of the program in the

coming years will be to integrate the methods of diverse disciplines to determine the

specific types of PM, and their sources, that have the greatest effect on public

health. This will allow future standards and control strategies to focus attention

only on those sources of pollution that need to be addressed. Another major focus

will be on understanding the effects of long-term exposures to PM, through the

funding of a long-term epidemiologic study to be conducted as part of our STAR research program. Lastly, EPA will evaluate new technologies for reducing air pollution, examining the ability of controls to reduce emissions of many pollutants at

once. The results of these efforts will inform EPA’s future PM policies, to ensure

these policies protect human health in the most effective ways.

Ecological Research

Current ecological management approaches have made important contributions to

improved environmental quality through greatly reducing emissions of pollutants

from point sources and waste disposal sites, and reducing the mishandling of toxic

or hazardous chemicals and pesticides. Future ecological problems, however, will

likely be more subtle, potentially more far-reaching, and require very different solutions. Examples include non-point source pollution control, regional-scale effects of

air pollutants on aquatic ecosystems, dislocations in ecologically and economically

important species due to invasions by non-indigenous species, and the cumulative

effects and synergistic interactions of multiple stressors on the health of aquatic

species and communities.

To deliver sound science for informed decision-making, EPA has focused its ecological research program to assess and compare risks to ecosystems, to protect and

restore them, and to demonstrate progress in terms of ecological outcomes. The ecological research program also reflects the growing ethic of environmental stewardship and the recognition that the implementation of these ecological management

approaches will be largely community and sector-based, place-based, and performance-based.

Environment and natural resource research is coordinated government-wide

through the Committee on Environmental and Natural Resources (CENR). EPA is

an active member on this committee, whose goal is to increase the overall effectiveness and productivity of federal research and development in environmental issues.

Given the current fiscal constraints, EPA believes it is more important than ever

for federal agencies to collaborate and coordinate research activities. EPA has a long

history of collaborating ecosystem research with the National Science Foundation.

EPA plans to continue, and wherever appropriate enhance, its coordination with

NSF and other agencies.

44

ORD’s Ecological Research Strategy underwent interagency peer review by the

CENR in June 1997, and external review by the Science Advisory Board’s Ecological

Processes and Effects Committee in July 1997. The final Strategy, published in June

1998, formed the basis for ORD’s Ecological Research Multi-Year Plan, which describes how the Agency plans to align its resources to achieve the plan’s goals, including the integration of ORD’s in-house research efforts with those conducted by

our STAR research grants program.

The Ecological Research Multi-Year Plan lays out four critical scientific questions

to be addressed and their associated research emphases and programmatic goals.

These questions are:

• What is the current condition of ecosystems and what are the trends in their

condition over time? (Assessing condition)

• How do natural ecological disturbances and human activities affect ecosystems? How can we most accurately diagnose the causes of ecosystem deterioration? (Diagnosis)

• How can we reliably predict the vulnerability of ecosystems to harm from current resource development and management practices? How can we predict

the most probable responses of ecosystems to best management and sustainable development practices? (Forecasting)

• How can we most effectively control risks and manage to protect ecosystems

once they have been degraded? (Restoration)

The PART evaluation on the ecological research program found that the program

addresses a clear and continuing need and that it is generally well-managed, with

adequate grantee and resource oversight. Its work has lead to accomplishments such

as the Environmental Monitoring and Assessment Program (EMAP) National Coastal Assessment accomplishments that I mentioned earlier. Additional examples include:

• Research methods and findings from ORD’s EMAP have enabled State and

tribal water monitoring programs to obtain more reliable data on the ecological condition of their streams and rivers, at significantly lower cost than the

methods they had been using.

• ORD produced national guidelines on assessing ecological risks. For the first

time, these guidelines extend the principles of EPA’s risk assessment paradigms to assessing and comparing risks to ecosystems.

• STAR researchers have developed and applied integrated methods to model

and evaluate the effect of stressors on water quality. These include development of models to: (1) estimate annual nutrient loading to Lake Tahoe from

atmospheric deposition, precipitation, stream discharge, overland runoff,

groundwater and shoreline erosion; (2) estimate how ‘‘build-out’’ in urbanizing

watersheds affects nutrient cycling, water quality, and the ecological health

of rivers and streams in Gwynns Falls, Maryland; (3) evaluate the effects of

agricultural best management practices on stream flow, sediment, and nitrate

loadings in the lower Minnesota River; and (4) contaminant loading and bioaccumulation in Lake Erie.

As described in the Ecological Research multi-year plan, however, we are committed to building upon these achievements, and in the future, the ecological research program will focus heavily on diagnosis, forecasting, and restoration research. This research will enable the Agency to implement performance oriented,

place-based protection of ecological systems. Our challenge now is to translate these

successes into performance measures that demonstrate the utility of the tools and

other protocols that we develop. In particular, long-term goals are difficult for any

environmental program to develop, even more so for an environmental research program. I am committed to working with OMB and others to create long-term, annual,

and efficiency measures that capture the important work our program is doing. In

the end, these measures will help advance our program by demonstrating the value

of our achievements.

Science Quality Across EPA

While our comprehensive and collaborative research planning process guides EPA

to do the right science, as EPA’s Science Advisor, I believe EPA’s integrated approach to scientific quality makes sure that we also do the science right, not only

in ORD but across the Agency. The three pillars of this approach are our Quality

System, Information Quality Guidelines, and Peer Review Policy.

EPA’s Quality System is the means by which we manage our scientific information in a systematic, organized manner. It provides a framework for planning, imVerDate 11-MAY-2000 11:30 Dec 21, 2004 Jkt 092340 PO 00000 Frm 00048 Fmt 6633 Sfmt 6621 C:\WORKD\ETS04\031104\92340 SCIENCE1 PsN: SCIENCE1
45

plementing, and assessing the scientific work performed by EPA and for carrying

out quality assurance and quality control activities. Each EPA organization develops

a quality management plan that describes its quality system in terms of the organizational structure, policy and procedures, functional responsibilities of management

and staff, lines of authority, and necessary interfaces for the planning, implementing, documenting, and assessing of all activities conducted. At the individual

project level, we develop quality assurance project plans that describe the necessary

quality assurance, quality control, and other technical activities that must be implemented to ensure that work outputs will satisfy the stated performance criteria. The

goals of the EPA Quality System are to ensure that environmental programs and

decisions are supported by data of the type and quality needed and expected for

their intended use, and that decisions involving environmental technology are supported by appropriate quality-assured engineering standards and practices.

EPA recognizes that the Office of Management and Budget’s Information Quality

Guidelines, together with our own Information Quality Guidelines issued in October

2002, are an important step forward in the quest for quality. The OMB guidelines

call for all federal agencies to develop quality performance goals, including procedures to assure quality before information is disseminated. In response to these

guidelines, EPA has established a system for addressing complaints about the quality of information that the Agency has disseminated. We now have more than a

year’s worth of experience in addressing challenges to EPA information under the

guidelines, and this experience has validated our belief that ensuring the quality of

our scientific information is paramount to maintaining the integrity of, and the

public’s confidence in, EPA’s policies and decisions.

Consistent Agency-wide application of independent, expert scientific peer review

has been an EPA priority for many years. Since issuing our peer review policy in

1993, we have taken several major steps to support and strengthen the policy. But

proof of a policy’s value lies in its implementation, and here also EPA has been very

active to ensure that our peer review policy is not only understood across the Agency, but is applied rigorously across EPA’s program and regional offices. EPA has in

place a strong and extensive program for peer reviewing our scientific and technical

work products.

EPA’s approach to peer review is articulated in our policy, Peer Review and Peer

Involvement at the U.S. Environmental Protection Agency. In addition to the policy,

EPA has published a handbook that provides detailed guidance for implementing

the policy. The Peer Review Handbook can be found at www.epa.gov/osp/spc/

2peerrev.htm. We believe this is one of the most advanced treatments of peer review

for intramural research and scientific/technical analysis of any federal agency.

Most of EPA’s scientific and technical work products now undergo peer review. In

1995, the Agency identified 120 work products for peer review. In 2002, of 859 work

products generated by or for EPA, only 111 were deemed, usually because of their

repetitive or routine nature, not to be candidates for peer review. So, we see that

nearly 90 percent of our scientific and technical work products receive internal or

external peer review. And 90 percent of those peer-reviewed products received independent, external review.

We were confident enough in the strength of our peer review program that we

made it a cornerstone of our Information Quality Guidelines. Since issuing our policy ten years ago, peer review has become a part of EPA’s culture, and its use is

widespread across the Agency. Our challenge for the future is to continue the significant progress we have achieved to date and, not being content with the status

quo, to look for ways to enhance the use of peer review as a tool for ensuring that

EPA’s decisions are supported by a firm foundation of scientific and technical information.

Doing the right science through forward-looking collaborative research planning,

and doing the science right by adherence to information quality and peer review

standards, have given EPA policy-makers relevant, timely, and credible scientific information to guide Agency decisions.

ORD—Making a Difference

ORD scientists are committed to generating products of the highest quality to ensure sound science informs Agency decision-making. Our successes have been numerous, and we continue to build upon them. I have highlighted below a sampling

of such successes, to illustrate the depth, breadth, and relevance of our research programs’ contributions to environmental science generally and to EPA’s mission in

particular. As these examples demonstrate, ORD’s research program—as a major

part of the entire EPA scientific endeavor—plays a critical role in protecting human

health and safeguarding the environment.

46

• In July 2003, EPA conducted an important drinking water distribution field

study to map the movement of contaminants in a water system. This research

is helping water system managers and emergency responders better predict

how a biological or chemical contaminant would react in a drinking water system. This study ties directly into EPA’s community support and homeland security efforts.

• ORD, working with academia, developed the first air quality model (Models3/CMAQ) to use a ‘‘one atmosphere’’ approach to simulate the interactions

among many air pollutants, which is necessary to achieve truly cost-effective

air pollution control strategies. This work is critical for local air pollution

forecasting, as well as supporting the Agency’s multi-pollutant control strategies.

• Working with the Department of Energy and the National Oceanic and Atmospheric Administration, ORD is researching exposures to air pollutants in

complex terrains, such as urban canyons created by high-rise buildings and

complex traffic patterns. This research combines field monitoring with wind

tunnel studies to refine exposure models that can be applied to different U.S.

cities.

• ORD developed toxicity methods for determining acute and chronic toxicity to

plants, invertebrates, and vertebrates, using several different end points.

ORD also participated in the development of the Whole Effluent Toxicity Test

Methods Rule, which allows these methods to be used as a basis for decisionmaking in the National Pollutant Discharge Elimination Systems Program.

• ORD developed analytical methods for Cryptosporidium and evaluated technologies that could be used for removing Cryptosporidium from drinking

water sources. ORD worked with the Office of Water to use these results in

promulgating the Long-Term 2 Enhanced Surface Water Treatment Rule.

This rule will protect drinking water consumers, including sensitive sub-populations such as children, by avoiding Cryptosporidium incidents that have resulted in health impacts and even death in the past.

• EPA’s cancer risk assessment prompted industry decisions to phase-out the

use of chromated copper arsenate (CCA) treated wood in home settings, due

to concerns of exposure to kids from decks and play equipment. ORD is working with other EPA scientists to analyze exposures to homeowners and children from CCA-treated decks and play equipment and to evaluate coatings

and sealants that can be used to reduce risk from exposure to CCA-treated

wood.

• ORD developed a DNA-based system that allows rapid identification and

quantification of molds in a matter of hours, as opposed to current methods

that require days or even weeks. The new technology can be used to detect

the mold Stachybotrys, commonly known as ‘‘black mold,’’ and more than 50

other possibly harmful molds. The new method has been licensed to 13 companies for use in detecting mold, and four additional licenses are pending.

• EPA chairs the coordination of endocrine disruptor research across federal

agencies through an interagency working group under the Committee on Environment and Natural Resources, under the President’s National Science and

Technology Council. Through this interagency working group, EPA and its

partners issued two joint solicitations for research proposals to address the

critical data gaps of understanding the impact of endocrine disruptors on humans and wildlife.

Linking Research Results to Outcomes

EPA recognizes that research findings—no matter how insightful or cuttingedge—cannot of their own accord achieve environmental outcomes. Achieving environmental outcomes depends on decisions made and actions taken by the Agency’s

program and regional offices, as well as by our State and tribal partners. We are

working with our Office of Inspector General (OIG) to develop better ways to describe the link between our research program and environmental and public health

outcomes. Our efforts are focused on the use of a logic model that was developed

by the OIG.

The OIG, in collaboration with the ORD, piloted using the logic model to determine if the design of the Pollution Prevention and New Technology research program was conducive to achieving desired environmental outcomes. The pilot was

successful, and we now employ logic models to clearly identify the outputs of our

research and their associated near-term outcomes. Logic model techniques are parVerDate 11-MAY-2000 11:30 Dec 21, 2004 Jkt 092340 PO 00000 Frm 00050 Fmt 6633 Sfmt 6621 C:\WORKD\ETS04\031104\92340 SCIENCE1 PsN: SCIENCE1
47

ticularly useful for identifying outputs and methods for transferring research results

to our clients, helping them to achieve environmental outcomes.

The logic model also emphasizes that there are factors outside the realm of

science that may help or hinder the success of the program and the accomplishment

of its results. ORD scientists, EPA program offices, and our State, tribal, and local

clients each have their respective roles for helping to achieve environmental outcomes. In light of this, ORD believes that research programs are most appropriately

evaluated with respect to the soundness of the research strategy, the significance

of the research findings, and the usefulness of the resulting scientific tools or policies for their intended applications. We also believe there is an important role for

independent, expert peer review for accomplishing such evaluations.

ORD is moving forward with its plans to conduct reviews of its research programs

by external independent experts. These expert panels will review our research in

accordance with the Administration’s investment criteria for research and development; namely, quality, relevance, and performance. These reviews will provide valuable input for determining that ORD is managing its programs to ensure scientific

quality, and is providing relevant results for achieving the Agency’s mission.

It is a challenging task to relate research, especially inherently long-term research, to specific environmental and public health outcomes. However, as I mentioned earlier, I am committed to moving ORD in that direction. The PARTs conducted last year have provided up with valuable experience that will help us demonstrate the value of our programs, and we are working with OMB to develop recommendations to improve program performance.

Science to Achieve Results Research

Mr. Chairman, your letter of invitation asked me to specifically address the reduction of EPA’s STAR grants for research on ecological systems, pollution prevention,

endocrine disruptors, and mercury. While I will address the specific reductions later

in my testimony, I want to share at this point some of my thoughts and the

thoughts of others about our STAR program and how it continues to be a vital part

of ORD’s research portfolio.

In 1995, ORD created the Science To Achieve Results extramural research program. This program was created for the purpose of providing ORD swift, flexible access to nationally and internationally acclaimed scientists who could conduct independent and original research to complement the efforts of ORD’s intramural research program.

Since the program’s inception, all or parts of the STAR program have been reviewed three times by the EPA SAB and twice by the NRC. These reviews have

been very favorable, but have also noted areas for improvement. As the NRC is also

a witness today, I will leave it to them to describe the findings of their 2003 review

of the STAR program, The Measure of STAR.

EPA has developed an in-house staff capability to address environmental research

needs. In some cases, EPA lacks a critical mass of in-house expertise that can devote itself full-time to new research issues, and the STAR program enables ORD to

quickly deploy resources to access nationally and internationally acclaimed scientists to conduct independent and original research where the Agency lacks capacity or specialized expertise.

The STAR program remains strong and is aligned to most effectively support

EPA’s priority research needs. For example, STAR research efforts will be funded

consistent with previous years’ investments in important areas including children’s

health, particulate matter, safe food, and drinking water. In those areas where

STAR will be eliminated in FY 2005 (ecological systems, pollution prevention, endocrine disruptors, and mercury), EPA will continue to conduct in-house research as

well as look to increase its ongoing research partnerships with university researchers and initiate new ones. STAR currently leverages its resources through joint solicitations with 12 federal and private sector research partners, enabling EPA to enhance its research portfolio by about 30 to 50 additional grants.

FY 2005 President’s Budget

The President’s FY 2005 budget request continues the tradition of ORD research

excellence by emphasizing cutting-edge science and technology, collaboration with

other agencies, and an orientation on results.

Mr. Chairman, in your letter of invitation you asked me to identify what research

would not be done as a result of the proposed reductions in the STAR grants program in the President’s Budget request and the associated impacts. The following

are areas of decreased STAR research.

48

Ecological Protection Research Grants (-$22.2M)

EPA would no longer fund STAR grants in the area of ecological protection, a reduction of about 50 grants. In response to PART findings, EPA is working to develop

long-term, annual, and efficiency performance measures for the program. Key areas

of research at academic institutions across the Nation would no longer be conducted,

affecting Agency efforts to assess ecosystem condition, diagnose ecosystem impairment, and forecast ecosystem health.

Hazardous Substance Research Centers (HSRCs) (-$2.25M)

Eliminate most of the research in the fifth and final year of planned funding for

the HSRCs, as well as the technical support and outreach efforts of the centers that

directly support EPA regional, State, and tribal efforts to evaluate and manage risk

at clean-up sites.

Mercury Research (-$2.0M)

Eliminate STAR-supported university research in support of understanding the

atmospheric processes that affect the transport, transformation, and deposition of

mercury emissions from natural and anthropogenic sources.

Endocrine Disruptors (-$4.7M)

Eliminate funding for the STAR portion of the Endocrine Disruptors research program. However, the President’s Budget provides a $3.5 million increase for EPA’s

computational toxicology program, which uses computational chemistry and molecular biology to more accurately predict health effects from chemicals, thereby improving linkages between potential exposure and disease. Our computational toxicology program offers more promising and timely application for our Endocrine

Disruptors Screening Program.

Pollution Prevention and New Technologies (-$6.0M)

Transfer funding of the research program to the Office of Pesticides, Prevention,

and Toxic Substances pollution prevention program, which the PART analysis has

shown a reduction in the use of chemicals and pollution. In response to PART findings, the program is working to develop long-term, annual, and efficiency performance measures.

Conclusion

By uniquely combining human health and ecological research in one federal agency, ORD has made significant contributions to developing a better understanding of

environmental risks to both human health and ecosystems. The results of this research have consistently and effectively informed EPA’s environmental decisionmaking, leading to environmental policies based on sound science at the federal,

State, tribal, and local level.

The President’s FY 2005 budget request for ORD continues this tradition of excellence, by emphasizing cutting-edge science and technology, collaboration with other

agencies, and an orientation on results.

Thank you.

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

BIOGRAPHY FOR PAUL GILMAN

In April 2002, Dr. Gilman was sworn-in to serve as the Assistant Administrator

for the Office of Research and Development which is the scientific and technological

arm of the Environmental Protection Agency. In May 2002, he was appointed the

Agency Science Advisor. In this capacity, he will be responsible for working across

the Agency to ensure that the highest quality science is better integrated into the

Agency’s programs, policies and decisions.

Before his confirmation, he was Director, Policy Planning for Celera Genomics in

Rockville, Maryland. Celera Genomics, a bio information and drug discovery company, is known for having decoded the human genome. In his position Dr. Gilman

was responsible for strategic planning for corporate development and communications.

Prior to joining Celera, Dr. Gilman was the Executive Director of the life sciences

and agriculture divisions of the National Research Council of the National Academies of Sciences and Engineering. The National Research Council is the operating

arm of the National Academies which were chartered to provide independent advice

to the government in matters of science and engineering. Dr. Gilman’s divisions focused on risks to health and the environment, protection and management of biotic

resources, and practical applications of biology including biotechnology and agriculture.

Before joining the National Research Council, Gilman was the Associate Director

of the Office of Management and Budget (OMB) for Natural Resources, Energy, and

Science. There he coordinated budget formulation, regulatory, and legislative activities between agencies such as the Environmental Protection Agency, National

Science Foundation, Agriculture, and Energy with the Executive Office of the President.

Dr. Gilman served as Executive Assistant to the Secretary of Energy for technical

matters before joining the OMB. His responsibilities included participating in policy

deliberations and tracking implementation of a variety of programs including the

Department’s environmental remediation and basic science research.

Gilman has 13 years of experience working on the staff of the United States Senate. He began that time as a Congressional Science Fellow sponsored by the American Association for the Advancement of Science in the office of Senator Pete V.

Domenici. Later, as the Staff Director of the Subcommittee on Energy Research and

Development, he was involved in the passage of the Nuclear Waste Policy Act of

1982 and oversight of energy technology and environmental research. Later he

served as the chief-of-staff for Senator Domenici.

Dr. Gilman matriculated at Kenyon College in Ohio and received his A.B., M.A.,

and Ph.D. degrees in ecology and evolutionary biology from Johns Hopkins University, Baltimore, Maryland.

Chairman EHLERS. Thank you.

Mr. Posner.

STATEMENT OF MR. PAUL L. POSNER, MANAGING DIRECTOR,

FEDERAL BUDGET AND INTERGOVERNMENTAL RELATIONS,

GENERAL ACCOUNTING OFFICE

Mr. POSNER. Thank you, Mr. Chairman.

My purpose at this hearing is to talk about the report we did on

the OMB PART and performance budgeting more in general. PART

is really a new chapter in a long history, some would argue checkered history of performance budgeting, trying to apply performance

to budgeting which is inherently in political process. One of the lessons we have learned from the failures of PPB, ZBB and OMB is

it is a fool’s errant. In fact, however, the fact that we keep trying

suggest we won’t accept failure for an answer because it is too important to try to link what we are trying to get out of these programs with our resource allocation. This is really the essence of

what government should be about. GPRA has, in fact, stayed

around much longer than its predecessors. We just issued a report

today on that, showing that in fact for 10 years, it has sustained

72

itself and built a credible supply for information and improved

planning.

What PART does is take that supply and attempt to more actively prompt decision-makers to use the information. Our review

of the process in fiscal year 2004 suggests that PART succeeded in

more explicitly developing linkages between performance and budgeting. It more transparently informed budget decisions than we

have seen before. There is more public transparency about the results of the process. Having said that, there is not a direct link

with funding, nor should there be, because budgeting is an exercise

in political choice and priorities of which performance is but one

input. In fact, what we really should expect performance budgeting

to do is not change the answers in some mechanical way, but to

change the kind of questions we ask. And in fact, this hearing is

a very good example of what we are talking about, and I would like

to see this in other areas. The focus we should have in our budget

debate is what do we know about programs? What more should we

know? What should we change in the metrics and the kinds of

goals that we are setting for these programs? This is actually exactly what performance budgeting should be all about, what you

are doing today in this hearing.

Having said this, doing this is not easy. Developing a bottom line

of federal programs is not easy. We are not a private sector business, where we have one profit and loss statement. We have multiple goals for every program, and it is really difficult to assign a

single rating to various kinds of balanced portfolios or programs.

So judgment is always required, regardless of what the number

says. OMB has tried, in the development of the PART instrument

and the application, to provide more consistency among its raters,

among OMB staff and agency people. Some terms will inherently

be subjective and require judgment; things like what constitutes an

ambitious performance goal. We found inconsistencies in such

things as defining what is an outcome versus an output in evaluating agencies.

The format of the PART tool itself, with the yes, no kinds of answers for most questions, force some standardizations for areas

where ultimately considerable judgment was required to balance

answers across multiple criteria. The chronic lack of performance

in evaluation across many programs, which we have long identified, remains a considerable barrier in doing this exercise, and one

thing that we think PART may in fact instigate and instill is a

greater incentive to get more of this done. Particular challenges, as

you will note today in research and development, in terms of the

basic nature of research, makes it more difficult. It takes time to

really assess what we are getting out of research. Research itself,

particularly basic research, is an uncertain enterprise. But these

challenges are being addressed in agencies across the board, and

we think the National Academy of Sciences has provided useful criteria.

One important issue that remains—and when you try to judge

multiple programs—is defining what a program is. It is arguably

seemingly something that we should all know, but there is no uniform definition of what a program is, and so this question of what

is the unit analysis for defining what a program is, is critical in

73

1Pub. L. No. 103–62 (1993).

2 U.S. General Accounting Office, Performance Budgeting: Past Initiatives Offer Insights for

GPRA Implementation, GAO/AIMD–97–46 (Washington, D.C.: Mar. 27, 1997). 3See Pub. L. No. 103–62 §2 (1993), 5 U.S.C. §306 (2003), and 31 U.S.C. §§1115–1116 (2003). 4 In addition to budget and performance integration, the other four priorities under the PMA

are strategic management of human capital, expanded electronic government, improved financial performance, and competitive sourcing.

terms of defining what we are getting out of government. This was

an issue with GPRA, frankly, and the National Academy of Science

report identified this, and it is clearly an issue with PART. Programs were defined one way for GPRA, for example, in more strategic ways. Under PART we are redefining programs for budget

purposes in a more discrete, granular way. These are two different

competing kinds of accountability mechanisms, and one issue we

surfaced is the need to better synchronize between the two.

Ultimately, we recommended continued improvement in the

guidance and the rating tool by OMB. We recommended a more

targeted selection process so that, in the future, related programs

can be grouped together so we can look across different programs

and see what we are getting. We recommended early consultation

with the Congress, and most importantly, we suggested that Congress develop a process to better identify its performance issues

and oversight priorities to enable it to better communicate with

OMB and move this process forward.

Thank you.

[The prepared statement of Mr. Posner follows:]

PREPARED STATEMENT OF PAUL L. POSNER

Mr. Chairman and Members of the Subcommittee:

I am pleased to be here today to discuss performance budgeting and the Office

of Management and Budget’s (OMB) Program Assessment Rating Tool (PART).

Since the 1950s, the Federal Government has attempted several government-wide

initiatives designed to better align spending decisions with expected performance—

what is commonly referred to as ‘‘performance budgeting.’’ The consensus is that

prior efforts—including the Hoover Commission, the Planning-Programming-Budgeting-System, Management by Objectives, and Zero-Based Budgeting—did not succeed in significantly shifting the focus of the federal budget process from its longstanding concentration on the items of government spending to the results of its

programs. However, the persistent attempts reflect a long-standing interest in linking resources to results.

In the 1990s, Congress and the executive branch laid out a statutory and management framework that provides the foundation for strengthening government performance and accountability, with the Government Performance and Results Act of

19931 (GPRA) as its centerpiece. GPRA is designed to inform congressional and executive decision-making by providing objective information on the relative effectiveness and efficiency of federal programs and spending. A key purpose of the act is

to create closer and clearer links between the process of allocating scarce resources

and the expected results to be achieved with those resources. We have learned that

this type of integration is critical from prior initiatives that failed in part because

they did not prove to be relevant to budget decision-makers in the executive branch

or Congress.2 GPRA requires both a connection to the structures used in congressional budget presentations and consultation between the executive and legislative

branches on agency strategic plans; this gives Congress an oversight stake in

GPRA’s success.3

This administration has made the integration of performance and budget information one of five government-wide management priorities under the President’s Management Agenda (PMA).4 Central to this initiative is PART. OMB developed PART

as a diagnostic tool meant to provide a consistent approach to evaluating federal

programs and applied it in formulating the President’s fiscal years 2004 and 2005

74

5There is no standard definition for the term ‘‘program.’’ For purposes of PART, OMB described the unit of analysis (program) as (1) an activity or set of activities clearly recognized

as a program by the public, OMB, or Congress; (2) having a discrete level of funding clearly

associated with it; and (3) corresponding to the level at which budget decisions are made. 6The seven major categories are competitive grants, block/formula grants, capital assets and

service acquisition programs, credit programs, regulatory-based programs, direct federal programs, and research and development programs.

budget requests. PART covers four broad topics for all ‘‘programs’’ 5 selected for review: (1) program purpose and design, (2) strategic planning, (3) program management, and (4) program results (i.e., whether a program is meeting its long-term and

annual goals) as well as additional questions that are specific to one of seven mechanisms or approaches used to deliver the program.6

GPRA expanded the supply of performance information generated by federal agencies, although as the PART assessments demonstrate, more must be done to develop

credible performance information. However, improving the supply of performance information is in and of itself insufficient to sustain performance management and

achieve real improvements in management and program results. Rather, it needs

to be accompanied by a demand for that information by decision-makers and managers alike. PART may mark a new chapter in performance-based budgeting by

more successfully stimulating demand for this information—that is, using the performance information generated through GPRA’s planning and reporting processes

to more directly feed into executive branch budgetary decisions.

My statement today focuses on seven points:

• PART helped structure OMB’s use of performance information for its internal

program and budget analysis, made the use of this information more transparent, and stimulated agency interest in budget and performance integration. Moreover, it illustrated the potential to build on GPRA’s foundation to

more actively promote the use of performance information in budget decisions.

• The goal of PART is to evaluate programs systematically, consistently, and

transparently. OMB went to great lengths to encourage consistent application

of PART in the evaluation of government programs, including pilot testing the

instrument, issuing detailed guidance, and conducting consistency reviews.

Although there is undoubtedly room for continued improvement, any tool is

inherently limited in providing a single performance answer or judgment on

complex federal programs with multiple goals.

• Performance measurement challenges in evaluating complex federal programs

make it difficult to meaningfully interpret a bottom-line rating. The individual section ratings for each PART review provided a better understanding

of areas needing improvement than the overall rating alone.

• As is to be expected with any new reform, PART is a work in progress and

we have noted in our work where OMB might make improvements. Any tool

that is sophisticated enough to take into account the complexity of the U.S.

government will require some exercise of judgment. Therefore it is not surprising that we found some inconsistencies in OMB staff interpreting and applying PART.

• PART provides an opportunity to more efficiently use scarce analytic resources, to focus decision-makers’ attention on the most pressing policy issues,

and to consider comparisons and trade-offs among related programs by more

strategically targeting PART assessments based on such factors as the relative priorities, costs, and risks associated with related clusters of programs

and activities. The first year PART assessments underscored the long-standing gaps in performance and evaluation information throughout the Federal

Government. By reaching agreement on areas in which evaluations are most

essential, decision-makers can help ensure that limited resources are applied

wisely.

• The relationship between PART and its process and the broader GPRA strategic planning process is still evolving. Although PART can stimulate discussion on program-specific performance measurement issues, it is not a substitute for GPRA’s strategic, longer-term focus on thematic goals and

department- and government-wide crosscutting comparisons. PART and

GPRA serve different but complementary needs, so a strategy for integrating

the two could help strengthen both.

• Federal programs are designed and implemented in dynamic environments

where competing program priorities and stakeholders’ needs must be balanced continually and new needs must be addressed. While PART clearly

75

7 U.S. General Accounting Office, Performance Budgeting:Observations on the Use of OMB’s

Program Assessment Rating Tool for the Fiscal Year 2004 Budget, GAO–04–174 (Washington,

D.C.: Jan. 30, 2004). 8The 234 programs assessed for fiscal year 2004 contained a total of 612 recommendations.

serves the needs of OMB in budget formulation, questions remain about

whether it serves the various needs of other key stakeholders. If the President or OMB wants PART and its results to be considered in the congressional debate, it will be important for OMB to (1) involve congressional stakeholders early in providing input on the focus of the assessments; (2) clarify

any significant limitations in the assessments as well as the underlying performance information; and (3) initiate discussions with key congressional committees about how they can best take advantage of and leverage PART information in congressional authorization, appropriations, and oversight processes. Moreover, Congress needs to consider ways it can articulate its oversight priorities and performance agenda.

My statement is based on our recently published report on OMB’s PART7 in

which we reviewed the first year of the PART process—fiscal year 2004—and

changes in the PART process initiated for fiscal year 2005. We have not reviewed

or analyzed the PART results for the fiscal year 2005 budget request. For this testimony, this subcommittee asked us to discuss our overall findings and recommendations concerning PART to help frame today’s hearing. We conducted our work in accordance with generally accepted government auditing standards.

Strengths and Weaknesses of PART in Its First Year of Implementation

Through its development and use of PART, OMB has more explicitly infused performance information into the budget formulation process; increased the attention

paid to performance information and program evaluations; and ultimately, we hope,

increased the value of this information to decision-makers and other stakeholders.

By linking performance information to the budget process, OMB has provided agencies with a powerful incentive for improving both the quality and availability of performance information. The level of effort and involvement by senior OMB officials

and staff clearly signals the importance of this strategy in meeting the priorities

outlined in the PMA. OMB should be credited with opening up for scrutiny—and

potential criticism—its review of key areas of federal program performance and then

making its assessments available to a potentially wider audience through its Web

site.

As OMB and others recognize, performance is not the only factor in funding decisions. Determining priorities—including funding priorities—is a function of competing values and interests. Accordingly, we found that while PART scores were

generally positively related to proposed funding changes in discretionary programs,

the scores did not automatically determine funding changes. That is, for some programs rated ‘‘effective’’ or ‘‘moderately effective’’ OMB recommended funding decreases, while for several programs judged to be ‘‘ineffective’’ OMB recommended additional funding in the President’s budget request with which to implement changes.

In fact, the more important role of PART was not its use in making resource decisions, but in its support for recommendations to improve program design, assessment, and management. Our analysis of the fiscal year 2004 PART found that 82

percent of the recommendations addressed program assessment, design, and management issues; only 18 percent of the recommendations had a direct link to funding

matters.8

OMB’s ability to use PART to identify and address future program improvements

and measure progress—a major purpose of PART—depends on its ability to oversee

the implementation of PART recommendations. As OMB has recognized, following

through on these recommendations is essential for improving program performance

and ensuring accountability. Currently, OMB plans to assess an additional 20 percent of all federal programs annually. As the number of recommendations from previous years’ evaluations grows, a system for monitoring their implementation will

become more critical. However, OMB does not have a centralized system to oversee

the implementation of such recommendations or evaluate their effectiveness.

The goal of PART is to evaluate programs systematically, consistently, and transparently. OMB went to great lengths to encourage consistent application of PART

in the evaluation of government programs, including pilot testing the instrument,

issuing detailed guidance, and conducting consistency reviews. Although there is undoubtedly room for continued improvement, any tool is inherently limited in providing a single performance answer or judgment on complex federal programs with

multiple goals.

76

OMB recognized the complexity inherent in evaluating federal programs by differentiating its rating tool for seven mechanisms or approaches used to deliver services, ranging from block grants to research and development. However, judgment is

involved in classifying programs by these categories since many programs fit into

more than one of these groupings. OMB guidance, for instance, acknowledges that

some research and development programs can also be evaluated as competitive

grants and capital assets.

Performance measurement challenges in evaluating complex federal programs

make it difficult to meaningfully interpret a bottom-line rating. OMB published both

a single, bottom-line rating for PART results and individual section scores. It is

these latter scores that are potentially more useful for identifying information gaps

and program weaknesses. For example, in the fiscal year 2004 PART, one program

that was rated ‘‘adequate’’ overall got high scores for purpose (80 percent) and planning (100 percent), but poor scores in being able to show results (39 percent) and

in program management (46 percent). In a case like this, the individual section ratings provided a better understanding of areas needing improvement than the overall

rating alone. In addition, bottom-line ratings may force raters to choose among several important but disparate goals and encourage a determination of program effectiveness even when performance data are unavailable, the quality of those data is

uneven, or they convey a mixed message on performance.

Any tool that is sophisticated enough to take into account the complexity of the

U.S. government will always require some interpretation and judgment. Therefore

it is not surprising that OMB staff were not fully consistent in interpreting complex

questions about agency goals and results. Many PART questions contain subjective

terms that are open to interpretation. Examples include terminology such as ‘‘ambitious’’ in describing sought-after performance measures. Because the appropriateness of a performance measure depends on the program’s purpose, and because program purposes can vary immensely, an ambitious goal for one program might be

unrealistic for a similar but more narrowly defined program. Without further guidance, it is unclear how OMB staff can be expected to be consistent.

We also found inconsistencies in how the definition of acceptable performance

measures was applied. Our review of the fiscal year 2004 PART surfaced several

instances in which OMB staff inconsistently defined appropriate measures—outcome

versus output—for programs. Agency officials also told us that OMB staff used different standards to define measures as outcome-oriented. Outputs are the products

and services delivered by the program whereas outcomes refer to the results of outputs. For example, in the employment and training area, OMB accepted short-term

outcomes, such as obtaining high school diplomas or employment, as a proxy for

long-term goals for the Department of Health and Human Services’ Refugee Assistance program, which aims to help refugees attain economic self-sufficiency as soon

as possible. However, OMB did not accept the same employment measure as a proxy

for long-term goals for the Department of Education’s Vocational Rehabilitation program because it had not set long-term targets beyond a couple of years. In other

words, although neither program contained long-term outcomes, such as participants gaining economic self-sufficiency, OMB accepted short-term outcomes in one

instance but not the other.

The yes/no format employed throughout most of the PART questionnaire resulted

in oversimplified answers to some questions. Although OMB believes it helped

standardization, the yes/no format was particularly troublesome for questions containing multiple criteria for a ‘‘yes’’ answer. Agency officials have commented that

the yes/no format can oversimplify reality, in which progress in planning, management, or results is more likely to resemble a continuum than an on/off switch. Our

review of the fiscal year 2004 PART found several instances in which some OMB

staff gave a ‘‘yes’’ answer for successfully achieving some but not all of the multiple

criteria, while others gave a ‘‘no’’ answer when presented with a similar situation.

For example, OMB judged the Department of the Interior’s (DOI) Water Reuse and

Recycling program ‘‘no’’ on whether a program has a limited number of ambitious,

long-term performance goals, noting that although DOI set a long-term goal of

500,000 acre-feet per year of reclaimed water, it failed to establish a time frame for

when it would reach the target. However, OMB judged the Department of Agriculture’s and DOI’s Wildland Fire programs ‘‘yes’’ on this question even though the

programs’ long-term goals of improved conditions in high-priority forest acres are

not accompanied by specific time frames.

The lack of program performance information also creates challenges in effectively

assessing program performance. According to OMB, about half of the programs assessed for fiscal year 2004 lacked ‘‘specific, ambitious long-term performance goals

that focus on outcomes’’ and nearly 40 percent lacked sufficient ‘‘independent, quality evaluations.’’ Nearly 50 percent of programs assessed for fiscal year 2004 reVerDate 11-MAY-2000 11:30 Dec 21, 2004 Jkt 092340 PO 00000 Frm 00080 Fmt 6633 Sfmt 6621 C:\WORKD\ETS04\031104\92340 SCIENCE1 PsN: SCIENCE1
77

9 U.S. General Accounting Office, Program Evaluation: Agencies Challenged by New Demand

for Information on Program Results, GAO/GGD–98–53 (Washington, D.C.: Apr. 24, 1998). 10 U.S. General Accounting Office, Transportation Research: Actions Needed to Improve Coordination and Evaluation of Research, GAO–03–500 (Washington, D.C.: May 1, 2003). 11 U.S. General Accounting Office, Program Evaluation: An Evaluation Culture and Collaborative Partnerships Help Build Agency Capacity, GAO–03–454 (Washington, D.C.: May 2, 2003). 12OMB Circular A–11, Preparation, Submission, and Execution of the Budget. 13 U.S. General Accounting Office, Results-Oriented Government: GPRA Has Established a

Solid Foundation for Achieving Greater Results, GAO–04–38 (Washington, D.C.: March 10,

2004).

ceived ratings of ‘‘results not demonstrated’’ because OMB decided that program

performance information, performance goals, or both were insufficient or inadequate. While the validity of these assessments may be subject to interpretation and

debate, our previous work9 has raised concerns about the capacity of federal agencies to produce evaluations of program effectiveness as well as credible data.

In our report on PART, we note that several factors have limited the availability

of performance data and evaluations of federal programs, including the lack of statutory mandates and funding to support data collection and analysis. Our work has

recognized that research programs pose particular and long-standing challenges for

performance assessments and evaluations.10 For instance, in both applied and basic

research, projects take several years to complete and require more time before their

meaning for the field can be adequately understood and captured in performance reporting systems. These challenges can and have been addressed by federal and private research organizations. One evaluation approach we have identified in our review of leading practices is the use of peer review to evaluate the quality of research

outcomes.11 For example, the National Science Foundation (NSF) convenes panels

of independent experts as external advisers—a Committee of Visitors (COV)—to

peer review the technical and managerial stewardship of a specific program or cluster of programs periodically. The COV compares research plans with progress made,

and evaluates outcomes to determine whether the research contributes to NSF mission and goals.

The Relationship between GPRA and PART

PART was designed for and is used in the executive branch budget preparation

and review process. As a result, the goals and measures used in PART must meet

OMB’s needs. By comparison, GPRA—the current statutory framework for strategic

planning and reporting—is a broader process involving the development of strategic

and performance goals and objectives to be reported in strategic and annual plans

and reports. OMB said that GPRA plans were organized at too high a level to be

meaningful for program-level budget analysis and management review. OMB acknowledges that GPRA was the starting point for PART, but as I will explain, it

appears that OMB’s emphasis is shifting such that over time the performance measures developed for PART and used in the budget process may also come to drive

agencies’ strategic planning processes.

The fiscal year 2004 PART process came to be a parallel competing structure to

the GPRA framework as a result of OMB’s desire to collect performance data that

better align with budget decision units. OMB’s most recent Circular A–11 guidance

clearly requires both that each agency submit a performance budget for fiscal year

2005 and that this should replace the annual GPRA performance plan.12 These performance budgets are to include information from the PART assessments, where

available, including all performance goals used in the assessment of program performance done under the PART process. Until all programs have been assessed

using PART, the performance budget will also include performance goals for agency

programs that have not yet been assessed. OMB’s movement from GPRA to PART

is further evident in the fiscal year 2005 PART guidance stating that while existing

GPRA performance goals may be a starting point during the development of PART

performance goals, the GPRA goals in agency GPRA documents are to be revised,

as needed, to reflect OMB’s instructions for developing the PART performance goals.

Lastly, this same guidance states that GPRA plans should be revised to include any

new performance measures used in PART and that unnecessary measures should

be deleted from GPRA plans. In its comments to another recently issued GAO report, OMB stated that it will revise its guidance for both GPRA and PART to clarify

the integrated and complementary relationship between the two initiatives.13

Although there is potential for complementary approaches to GPRA and PART,

the following examples clearly illustrate the importance of carefully considering the

implications of selecting a unit of analysis, including its impact on the availability

of performance data. They also reveal some of the unresolved tensions between the

President’s budget and performance initiative—a detailed budget perspective—and

78

14 U.S. General Accounting Office, Agencies’ Strategic Plans Under GPRA: Key Questions to

Facilitate Congressional Review (Version 1), GAO/GGD–10.1.16 (Washington, D.C.: May 1997). 15 5 U.S.C. §306(d) (2003).

GPRA—a more strategic planning view. Experience with PART highlighted the fact

that defining a ‘‘unit of analysis’’ useful for both program-level budget analysis and

agency planning purposes can be difficult. For example, disaggregating programs for

PART purposes could ignore the interdependence of programs recognized by GPRA

by artificially isolating programs from the larger contexts in which they operate.

Agency officials described one program assessed with the fiscal year 2004 PART—

Projects for Assistance in Transition from Homelessness—that was aimed at a specific aspect of homelessness, that is, referring persons with emergency needs to

other agencies for housing and needed services. OMB staff wanted the agency to

produce long-term outcome measures for this program to support the PART review

process. Agency officials argued that chronically homeless people require many services and that this federal program often supports only some of the services needed

at the initial stages of intervention. GPRA—with its focus on assessing the relative

contributions of related programs to broader goals—is better designed to consider

crosscutting strategies to achieve common goals. Federal programs cannot be assessed in isolation. Performance also needs to be examined from an integrated, strategic perspective.

One way of improving the links between PART and GPRA would be to develop

a more strategic approach to selecting and prioritizing areas for assessment under

the PART process. Targeting PART assessments based on such factors as the relative priorities, costs, and risks associated with related clusters of programs and activities addressing common strategic and performance goals not only could help ration scarce analytic resources but also could focus decision-makers’ attention on the

most pressing policy and program issues. Moreover, such an approach could facilitate the use of PART assessments to review the relative contributions of similar programs to common or crosscutting goals and outcomes established through the GPRA

process.

The Importance of Congressional and Other Stakeholder Involvement

We have previously reported14 that stakeholder involvement appears critical for

getting consensus on goals and measures. In fact, GPRA requires agencies to consult

with Congress and solicit the views of other stakeholders as they develop their strategic plans.15 Stakeholder involvement can be particularly important for federal

agencies because they operate in a complex political environment in which legislative mandates are often broadly stated and some stakeholders may strongly disagree

about the agency’s mission and goals.

The relationship between PART and its process and the broader GPRA strategic

planning process is still evolving. As part of the executive branch budget formulation process, PART must clearly serve the President’s interests. Some tension about

the amount of stakeholder involvement in the internal deliberations surrounding

the development of PART measures and the broader consultations more common to

the GPRA strategic planning process is inevitable. Compared to the relatively openended GPRA process, any budget formulation process is likely to seem closed.

Yet, we must ask whether the broad range of congressional officials with a stake

in how programs perform will use PART assessments unless they believe the reviews reflect a consensus about performance goals among a community of interests,

target performance issues that are important to them as well as the administration,

and are based on an evaluation process in which they have confidence. Similarly,

the measures used to demonstrate progress toward a goal, no matter how worthwhile, cannot serve the interests of a single stakeholder or purpose without potentially discouraging use of this information by others.

Congress has a number of opportunities to provide its perspective on performance

issues and performance goals, such as when it establishes or reauthorizes a new

program, during the annual appropriations process, and in its oversight of federal

operations. In fact, these processes already reflect GPRA’s influence. Reviews of language in public laws and committee reports show an increasing number of references to GPRA-related provisions. What is missing is a mechanism to systematically coordinate a congressional perspective and promote a dialogue between Congress and the President in the PART review process.

In our report, we have suggested steps for both OMB and the Congress to take

to strengthen the dialogue between executive officials and congressional stakeholders. We have recommended that OMB reach out to key congressional committees early in the PART selection process to gain insight about which program areas

and performance issues congressional officials believe warrant PART review. EngagVerDate 11-MAY-2000 11:30 Dec 21, 2004 Jkt 092340 PO 00000 Frm 00082 Fmt 6633 Sfmt 6621 C:\WORKD\ETS04\031104\92340 SCIENCE1 PsN: SCIENCE1
79

ing Congress early in the process may help target reviews with an eye toward those

areas most likely to be on the agenda of Congress, thereby better ensuring the use

of performance assessments in resource allocation processes throughout government.

We have also suggested that Congress consider the need to develop a more systematic vehicle for communicating its top performance concerns and priorities; develop

a more structured oversight agenda to prompt a more coordinated congressional perspective on crosscutting performance issues; and use this agenda to inform its authorization, appropriations, and oversight processes.

Concluding Observations

The PART process is the latest initiative in a long-standing series of reforms undertaken to improve the link between performance information and budget decisions. Although each of the initiatives of the past appears to have met with an early

demise, in fact, subsequent reforms were strengthened by building on the legacy left

by their predecessors. Prior reforms often failed because they were not relevant to

resource allocation and other decision-making processes, thereby eroding the incentives for federal agencies to improve their planning, data, and evaluations.

Unlike many of those past initiatives, GPRA has been sustained since its passage

10 years ago, and evidence exists that it has become more relevant than its predecessors. PART offers the potential to build on the infrastructure of performance

plans and information ushered in by GPRA and the law’s intent to promote the use

of these plans in resource allocation decision-making. GPRA improved the supply of

plans and information, while PART can prompt greater demand for this information

by decision-makers. Enhancing interest and use may bring about greater incentives

for agencies to devote scarce resources to improving their information and evaluations of federal programs as well.

Increasing the use and usefulness of performance data is not only important to

sustain performance management reforms, but to improve the processes of decisionmaking and governance. Many in the United States believe there is a need to establish a comprehensive portfolio of key national performance indicators. This will raise

complex issues ranging from agreement on performance areas and indicators to getting and sharing reliable information for public planning, decision-making, and accountability. In this regard, the entire agenda of management reform at the federal

level has been focused on shifting the attention of decision-makers and agency management from process to results. Although PART is based on changing the orientation of budgeting, other initiatives championed by Congress and embodied in the

PMA are also devoted to improving the accountability for performance goals in agency human capital management, financial management, competitive sourcing, and

other key management areas.

In particular, we have reported that human capital—or people—is at the center

of any serious change management initiative. Thus, strategic human capital management is at the heart of government transformation. High-performing organizations strengthen the alignment of their GPRA strategic and performance goals with

their daily operations. In that regard, performance management systems can be a

vital tool for aligning an organization’s operations with individual day-to-day activities, but they are currently largely unused. As we move forward to strengthen government performance and accountability, effective performance management systems can be a strategic tool to drive internal change and achieve desired results.

The question now is how to enhance the credibility and use of the PART process

as a tool to focus decisions on performance. In our report, we make seven recommendations to OMB and a suggestion to Congress to better support the kind of

collaborative approach to performance budgeting that very well may be essential in

a separation of powers system like ours. Our suggestions cover several key issues

that need to be addressed to strengthen and help sustain the PART process. We recommend that the OMB Director take the following actions:

• Centrally monitor agency implementation and progress on PART recommendations and report such progress in OMB’s budget submission to Congress. Government-wide councils may be effective vehicles for assisting OMB

in these efforts.

• Continue to improve the PART guidance by (1) expanding the discussion of

how the unit of analysis is to be determined to include trade-offs made when

defining a unit of analysis, implications of how the unit of analysis is defined,

or both; (2) clarifying when output versus outcome measures are acceptable;

and (3) better defining an ‘‘independent, quality evaluation.’’

• Clarify OMB’s expectations to agencies regarding the allocation of scarce evaluation resources among programs, the timing of such evaluations, as well as

the evaluation strategies it wants for PART, and consider using internal

80

agency evaluations as evidence on a case-by-case basis—whether conducted

by agencies, contractors, or other parties.

• Reconsider plans for 100 percent coverage of federal programs and, instead,

target for review a significant percentage of major and meaningful government programs based on such factors as the relative priorities, costs, and

risks associated with related clusters of programs and activities.

• Maximize the opportunity to review similar programs or activities in the

same year to facilitate comparisons and trade-offs.

• Attempt to generate, early in the PART process, an ongoing, meaningful dialogue with congressional appropriations, authorization, and oversight committees about what they consider to be the most important performance issues

and program areas that warrant review.

• Seek to achieve the greatest benefit from both GPRA and PART by articulating and implementing an integrated, complementary relationship between

the two.

In its comments on our report, OMB outlined actions it is taking to address several of these recommendations, including refining the process for monitoring agencies’ progress in implementing the PART recommendations, seeking opportunities

for dialogue with Congress on agencies’ performance, and continuing to improve executive branch implementation of GPRA plans and reports.

Our recommendations to OMB are partly directed at fortifying and enhancing the

credibility of PART itself and the underlying data used to make the judgments. Decision makers across government are more likely to rely on PART data and assessments if the underlying information and the rating process are perceived as being

credible, systematic, and consistent. Enhanced OMB guidance and improved strategies for obtaining and evaluating program performance data are vital elements.

The PART process can be made more sustainable if the use of analytic resources

at OMB and the agencies is rationalized by reconsidering the goal of 100 percent

coverage of all federal programs. Instead, we suggest a more strategic approach to

target assessments on related clusters of programs and activities. A more targeted

approach stands a better chance of capturing the interest of decision-makers

throughout the process by focusing their attention on the most pressing policy and

program issues and on how related programs and tools affect broader crosscutting

outcomes and goals. Unfortunately, the government-wide performance plan required

by GPRA has never been engaged to drive budgeting in this way.

Improving the integration of inherently separate but interrelated strategic planning and performance budgeting processes can help support a more strategic focus

for PART assessments. GPRA’s strategic planning goals could be used to anchor the

selection and review of programs by providing a foundation to assess the relative

contribution of related programs and tools to broader performance goals and outcomes.

Finally, refining the PART questionnaire and review process and improving the

quality of data are important, but the question of whose interests drive the process

is perhaps paramount in our system. Ultimately, the impact of PART on decisionmaking will be a function not only of the President’s decisions, but of congressional

decisions as well.

Much is at stake in the development of a collaborative performance budgeting

process. Not only might the PART reviews ultimately come to be disregarded absent

congressional involvement, but more important, Congress will lose an opportunity

to use the PART process to improve its own decision-making and oversight processes.

This is an opportune time for the executive branch and Congress to carefully consider how agencies and committees can best take advantage of and leverage the new

information and perspectives coming from the reform agenda under way in the executive branch. Ultimately, the specific approach or process is not important. We face

a long-term fiscal imbalance, which will require us to re-examine our existing policies and programs. It is all too easy to accept ‘‘the base’’ as given and to subject

only new proposals to scrutiny and analysis. The norm should be to reconsider the

relevance or ‘‘fit’’ of any federal program, policy, or activity in today’s world and for

the future.

Mr. Chairman, this concludes my prepared statement. I would be pleased to answer any questions you or the other Members of the Subcommittee may have at this

time.

81

82

Chairman EHLERS. Dr. Matanoski.

STATEMENT OF DR. GENEVIEVE MATANOSKI, PROFESSOR, DEPARTMENT OF EPIDEMIOLOGY, JOHNS HOPKINS UNIVERSITY; CHAIR, EPA SCIENCE ADVISORY BOARD REVIEW OF

EPA’S FISCAL YEAR 2005 BUDGET; FORMER CHAIR, EPA

SCIENCE ADVISORY BOARD

Dr. MATANOSKI. Good morning, Mr. Chairman and Members of

the Subcommittee on Environment, Technology and Standards.

My name is Genevieve Matanoski. I am honored to appear before

you today to present the views of the Science Advisory Board for

EPA regarding their Fiscal Year 2005 Science and Research Budget

Request. The Board will complete its full report by March 19, 2004,

and with the permission of the Chairman, we will submit that report for the record then.

Over the last year, the Board, under the leadership of its Chairman, Dr. William Glaze, has reorganized to make its advisory function as apparent as its peer-review function. With this reorganization, the Board moved the EPA Science and Research Budget Review from a small standing committee to the entire larger Board,

and this has provided great visibility to the review, and it also has

increased the expertise available to conduct the evaluation. This

Board, in the past years, has noted its deep concern over the constant erosion of EPA’s research budget programs. The fiscal year

2005 budget we now observe not only continued the erosion but actually cut funding for research. This continuous decrease in research funding severely constrains the ability of the agency to provide the necessary science to guide informed decision-making, and

to assist in reaching the Nation’s goals for human health and environmental risk reduction.

In particular, the Board notes, as did the Chairman, substantial

cuts in their fiscal year 2005 budget request for EPA’s Science to

Achieve Results program, STAR. STAR is recognized by this Board

as the science program of major importance to the agency. That

view is consistent with the views of the National Academy of

Science in its review of STAR. STAR provides many benefits to

EPA, one of which is the necessary flexibility to obtain critical scientific expertise in a wide range of disciplines that are essential in

addressing emerging issues that are outside EPA’s current areas of

expertise. EPA could never maintain the same large base of scientific expertise that is available on an as-needed basis to carry out

specific research. STAR also enhances EPA’s collaborations with

outside researchers and academic institutions, and in the process

actually stimulates additional resources for the agency and its

science needs.

STAR also benefits and strengthens scientific research throughout the United States by providing training for graduate students

who will reinforce the declining base of engineers and scientists in

the United States. Even though STAR is largely focused on EPA’s

core research, it has begun to accrue a record of early success. Serious budget cuts in STAR are as follows. Ecosystems protection research got a $22 million loss, with a loss of some 50 STAR grants

based in many states. That is a serious reduction. Endocrine

disruptors’ research was cut by five million. Pollution prevention

83

research, which is—a focus of that research is to avoid future problems. That was cut by five million. And mercury research was cut

by another two million. Even the STAR Graduate Fellowship Program, despite its increases, is still $4 million short of what Congress recommended for fiscal year 2003.

The Board believes that these cuts will have a negative impact

on the balance research portfolio that EPA has developed over the

last decade. In that time, EPA has developed a program that balances its problem-driving, short-term applied research, whatever

you want to call it, with its core long-term basic research. Here, the

STAR program helps EPA balance its internal and its extramural

research portfolios. The result is that science from many different

institutions is integrated into a total research program that compliments the scientific niche filled by EPA’s own excellent scientists. This provides a more nimble resource that is available to

work on existing and emerging environmental issues than would be

available with only an intramural program.

Another surprising change as already noted in the budget was

EPA’s decrease in its commitment to Homeland Security. The

Board believes that EPA must play a continuing role in Homeland

Security in several areas, and one of these is building contamination, which was cut.

I have run out of time. There are several other issues that we

have in our written testimony, one regarding PART and the other

regarding the future of research at EPA.

[The prepared statement of Dr. Matanoski follows:]

PREPARED STATEMENT OF GENEVIEVE MATANOSKI

Good morning Mr. Chairman and Members of the Subcommittee on Environment,

Technology, and Standards. My name is Genevieve Matanoski. I am honored to appear before you today to present the conclusions of the EPA Science Advisory Board

(SAB) on the EPA’s FY 2005 Science and Research Budget Request. My comments

today will summarize the views of the Board about that request for funding EPA’s

science and research programs during FY 2005. The Board will complete its full report on this issue by March 19, 2004 and with the permission of the Chairman, will

submit that report for the record then.

1. Introductory Remarks

Over the last year the Board, under the leadership of its Chairman, Dr. William

Glaze, has reorganized to make its advisory function as apparent as its peer review

function. To take advantage of the benefits of this reorganization, the Board moved

the EPA Science and Research advisory function from a small standing committee

to the larger Board. This enhances the visibility of the review and increases the expertise available to conduct the evaluation. We will also be working with EPA to

obtain information on the Agency’s science and research programs on a systematic

and continuous basis. The Agency now presents information to the Board in ways

that correlate with their new Strategic Plan and which will ultimately include all

science and research programs regardless of their funding source or where they are

conducted within EPA.

2. Specific Comments on EPA’s FY 2005 Science and Research Program and

Budgets

a. Erosion of the EPA Research Budget

First, I want to mention that in past years, the Board has noted its deep concern

with the constant erosion of EPA’s research program budgets. For the FY 2005

budget, we now observe not just continued erosion, but a substantial cut to research

funding. This continuous decrease in science and research funding severely constrains the ability of the Agency to provide the necessary science to guide informed

decision-making and to assist in reaching the Nation’s goals for human health and

environmental risk reduction. The EPA has an outstanding group of scientists who

conduct excellent basic and applied research focused on reducing both recognized

84

problems such as air pollution, and the continually emerging problems arising from

the past misuse of our environment.

b. Cuts to the STAR Program

In particular, the Board notes the substantial cuts in the FY 2005 budget request

for EPA’s Science to Achieve Results program (STAR). STAR is recognized by this

Board as a science program of major importance to the Agency. That view is consistent with the views of the National Academy of Science in its 2003 review entitled ‘‘The Measure of STAR.’’ STAR provides many benefits to EPA, including the

necessary flexibility to obtain critical scientific expertise in the wide range of disciplines that are essential in addressing emerging issues that are outside EPA’s current areas of expertise. EPA could never maintain the same large base of scientific

expertise that is available on an as needed basis to carry out specific research.

STAR enhances EPA’s collaboration with outside researchers and academic institutions and in the process actually stimulates additional resources for Agency science

needs. STAR also benefits and strengthens scientific research throughout the United

States by providing training for graduate students who will reinforce the declining

base of engineers and scientists in the U.S. Even though STAR is largely focused

on EPA’s core research, it has begun to accrue a record of early success. Evidence

of this can be found in the NAS’s ‘‘Measure of STAR’’ report in which it states that

STAR research on endocrine disruptors, and ecologic assessment have already resulted in ‘‘. . .peer-reviewed publications that are of immediate use in understanding causes, exposures, and effects of environmental pollution.’’ By any measure, STAR is an excellent investment.

To emphasize the seriousness of this situation, we note a number of cuts to STAR

research which are a part of the FY 2005 budget request:

i. Ecosystems Protection Research is reduced by over $22 million with a loss

of some 50 STAR grants based in many states. The Board is particularly

concerned about this cut given the critical need for ecosystems research,

which the Board feels is generally under-funded across EPA.

ii. Endocrine Disruptors Research is reduced by about $5 million. This is an

area of research that investigates the effects that could be associated with

use of many chemicals used in large quantities in our society.

iii. Pollution Prevention Research is reduced by $5 million even though the

focus of this research is on avoiding future problems and reducing the expensive cleanup costs that we face today.

iv. Mercury Research is reduced by $2 million, just at the time when more information is needed on this ubiquitous contaminant.

In addition, even though the STAR Graduate Fellowship program increases by $1.2 million over the FY 2004 level, it is still nearly $4 million below the level enacted by Congress for FY 2003 ($9.8 million). This program’s aim is to educate the

future environmental scientists that will be needed to replace the currently aging

population of such scientists. Thus, adequate funding of this program continues to

be essential.

The Board believes that these cuts will have a negative impact on the balanced

research portfolio that EPA, especially ORD, has developed over the last decade. In

that time, EPA has developed a program that balances its problem-driven (shorterterm, applied) research with its core (longer-term, basic) research. Though components of the core program are not always easy to identify in the budget, EPA appears to have a balanced research program, in this regard, with about half being

‘‘core’’ and half being ‘‘problem-driven.’’ This seems to be appropriate.

Further, the STAR program helps EPA balance its internal research portfolio with

its extramural research portfolio. The result is that science from many different institutions (government, academia, non-governmental organizations, and industry) is

integrated into a total research program that complements the scientific niche filled

by EPA’s own scientists. This provides a more nimble resource that is available to

work on existing and emerging environmental issues than would be available with

only an intramural or an extramural program. Changes in this budget, especially

to STAR, will significantly impair the balance of this integrated research program

in both core vs. problem driven and intramural vs. extramural research dimensions.

c. Building Decontamination and Homeland Security

Another surprising change for which no satisfying explanation was offered was

EPA’s decrease in its commitment to Homeland Security. The Board believes that

EPA must play a continuing role in Homeland Security in several areas. EPA’s

building decontamination research is one of EPA’s contributions to Homeland Security and it is being eliminated in an $8.3 million dollar cut prior to its completion.

85

The Board is unsure as to why research on this important issue is being cut when

it appears that further research is needed to provide rapid, on-target responses to

contamination episodes. The Board is aware that other agencies have substantial resources devoted to Homeland Security, so perhaps other groups have taken on this

role for the future. EPA has the special expertise to carry out this research. However, if it is judged that this is not a research direction for EPA, it is still important

to ensure that this work be conducted somewhere.

2. Program Planning and Measurement

a. Program Assessment

Each year, the Board tries to evaluate EPA’s research priorities and their role in

meeting the Agency’s goals. As part of the current review, the Board was given information resulting from the application of a new survey tool, the Program Assessment Rating Tool (PART) that was used to evaluate selected EPA programs. The

Board is concerned that decisions are being made about research program funding

on the basis of the application of this new tool.

To be clear, the Board did not receive or review information on the rating instrument itself; however, after evaluating PART summaries for several research programs, our conclusion is that PART may, at this time, have a limited capacity to

inform budget decisions on research programs. The Board’s is concerned with the

manner in which the weighting formula in PART seems to influence the full analysis and thus favor programs with short-run results over those having long-term results. There is also concern that an evaluator’s subjective considerations might be

able to bias those weights and the rating itself.

Specifically, it appears that the weighting formula in the PART favors programs

with near-term benefits at the expense of programs with long-term benefits. Since

research inevitably involves more long-term benefits and fewer short-term benefits,

PART ratings serve to bias the decision-making process against programs such as

STAR ecosystem research, global climate change research, and other important subjects. The PART seems to be intended as a formula for predictions about likely program success. However, the weights that the PART assigns to different program

characteristics do not seem to have been validated systematically against the contribution of each program characteristic to any independent objective measure of

program success. If the weights in the tool are arbitrarily assigned, the PART may

have characteristics that could lead to biases in evaluation that are related to the

subjective judgments of its designers. We believe that the tool should be reviewed

to determine its adequacy for its use in supporting budget decisions.

As the Board observed significant decreases in science and research funding, it

also noted a substantial resource increase in the State and Tribal Assistance Grant

account (STAG) for an initiative for retrofitting school busses. The Board does not

challenge the worthiness of this program, rather it notes that it has no information

on the science supporting this initiative. The Board trusts that the benefits of this

program have been rigorously reviewed.

The real issue here is how research programs (and others) are to be evaluated

and whether a different metric is necessary for basic vs. applied research programs.

Also, of interest is whether research results should be evaluated separately from the

outcomes of programs they are intended to support? Although the Board did not directly evaluate the PART itself, it is of obvious difficulty to conceive of a simple

quantitative metric that could be applied across the broad areas of ecosystem quality, human health effects, endocrine effects, and technology development. The question is even more complex when you consider that some research is intended to develop limited data in the short-run to fill a specific knowledge gap and other research is intended to provide an understanding of whole systems in the long-term.

Research program measurement is even more difficult because the knowledge and

methods developed by EPA, especially ORD’s researchers, are not usually directly

applied by ORD, rather they are often used by others to support decisions on a

broad suite of diverse statutory mandates. Thus, we believe that evaluations of the

performance of research programs will need to consider the specific factors of each

program that the research is intended to support. Further, it is unlikely that simple

formulas will be able to handle this task well. It is more likely that realistic research program performance assessment will need to be a combination of quantitative metrics and other information and analyses which is then evaluated by

groups of experts with relevant knowledge.

I note that the NAS, in its review of STAR, also had concerns with quantitative

routines used in performance assessments and noted that ‘‘The Committee judges

that expert review by a group of people with appropriate expertise is the best method of evaluating broad research programs, such as the STAR program.’’

86

b. Multi-Year Plans (MYPs)

Multi-Year Plans (MYPs) are an important innovation in EPA’s research planning

process. The SAB has reviewed a limited number of these plans and the process

used in their development and we believe that they will become more useful to the

Board’s evaluation of EPA’s science and research, and its funding, in the future.

MYP’s are tools that identify knowledge and methodology gaps needed to support

EPA’s mission areas and the body of research that would address those needs. Further, they provide a basis for identifying annual performance goals and measures

for efforts that become a part of EPA’s research budget. Finally, MYPs are very useful in providing focus on long-term progress toward research goals, especially on

cross-cutting subjects such as pollution prevention where coordination across the

Agency is essential. The Board supports the continued refinement of Multi-Year

plans and is available to continue its review of EPA’s progress in this regard.

3. EPA Science

My final comment will address an issue of great concern to the Board. Our analyses of EPA’s science and research budgets, today and in the past, convince us that

the Agency is in danger of underestimating the pace of large scale changes that are

now occurring in our society. If so, EPA and the Nation are at risk of repeating the

mistakes of the past that force us to spend huge sums of public funds to reduce and

to clean up the pollution brought on by the first industrial revolution.

The evidence before us suggests that we are now in a new, high velocity technological revolution that will yield great economic gains, but at the same time, will

offer new environmental challenges. Nanotechnology and biotechnology, to name

only two innovations, are proceeding with breathtaking speed, and are compounded

by forces such as global transfer of pollution and disease, and possible climate

change. EPA must carefully examine all of its science and research programs and

ask whether the Agency is conducting research that will help us protect human

health and the environment while encouraging innovation and growth.

This is not to say that EPA should neglect the ‘‘legacy’’ issues of the past; rather,

the Agency must continue to resolve those problems, and at the same time, work

with citizen’s groups, industry, and academia in creative ways to ensure that the

Nation avoids a new legacy of human health and environmental problems.

The Board would be pleased to work with EPA to explicitly address ways in which

EPA science and research might be focused to help EPA develop, and use to the fullest, knowledge that will be instrumental in avoiding a negative legacy.

I want to express my gratitude to the Members of the Subcommittee for inviting

me to testify about EPA’s science and research. I would be pleased to answer your

questions.

Thank you.

BIOGRAPHY FOR GENEVIEVE MATANOSKI

Dr. Matanoski is a Professor of Epidemiology at the Johns Hopkins University

School of Hygiene and Public Health in Baltimore, MD. For a time after medical

school she pursued a career in pediatrics and general preventive medicine. After

earning a Doctor of Public Health Degree, she was appointed to the faculty of Johns

Hopkins University and has been a professor since 1976. In addition to teaching and

research, Dr. Matanoski has had appointments in a number of teaching and training programs in the U.S. and abroad and is a frequent advisor to legislative and

policy-making groups. She is a member of several scientific advisory bodies both for

governmental agencies and for industry. She is a past Chair of the EPA Science Advisory Board, as well as a past Chair of the SAB Radiation Advisory Committee.

She now serves as a member of the Board of the EPA SAB. During her tenure on

the EPA SAB, Dr. Matanoski has been involved in the writing of several documents

produced to provide advice to EPA including the ‘‘Beyond the Horizon: Using Foresight to Protect the Environmental Future’’ document and the Integrated Risk

Project report ‘‘Toward Integrated Environmental Decision-making,’’ and was Chair

of the latter Committee. She is the author or co-author of over 80 publications.

Dr. Matanoski’s work has focused on the epidemiology of cancer, including bladder, lung, skin and uterine cancers, and leukemia. Her research studies have examined the risks associated with occupational and environmental exposures to such

agents as radiation, electromagnetic fields, and chemical substances as styrene, butadiene, arsenic and environmental tobacco smoke. Her recent research has emphasized reproductive effects and congenital malformations from environmental exposures. Her early work involved infectious diseases and illnesses in infants and children. Dr. Matanoski received a B.A. degree in chemistry at Radcliffe College and

a M.D. at the Johns Hopkins School of Medicine. She also earned a Doctor of Public

87

Health Degree from the Johns Hopkins University School of Hygiene and Public

Health (11/2003).

Chairman EHLERS. Thank you for your testimony.

Dr. Denson.

STATEMENT OF DR. COSTEL D. DENSON, PROFESSOR, DEPARTMENT OF ENGINEERING, UNIVERSITY OF DELAWARE;

MEMBER OF THE NATIONAL RESEARCH COUNCIL PANEL

REVIEW, THE MEASURE OF STAR: REVIEW OF THE EPA’S

SCIENCE TO ACHIEVE RESULTS (STAR) RESEARCH GRANTS

PROGRAM

Dr. DENSON. Good morning, Mr. Chairman and Members of the

Committee.

My name is Costel Denson. I am a Professor of Chemical Engineering at the University of Delaware, where I have been employed

since 1977. I have also served as Vice Provost for Research. In that

position, I had oversight of and was responsible for all aspects of

the research enterprise at the University. Recently, I served as a

member of the National Research Council, the NRC Committee, to

review EPA’s Research Grants Program. The NRC is the operating

arm of the National Academies of Sciences and Engineering. I am

pleased to be here to discuss the unanimous findings and recommendations of that committee.

The Environmental Protection Agency is a mission agency established to protect human health and to safeguard the natural environment. EPA’s regulatory and decision-making role requires that

the agency have access to the best available science that is relevant

to its mission. In an effort to improve the scientific foundation of

its decision-making process, the agency’s Office of Research and

Development established the Science to Achieve Results program,

the STAR program, as part of its Research Grants Program in

1995.

STAR is a competitive, peer-reviewed, extramural research

grants program created to encourage interagency collaboration and

to increase EPA’s access to the Nation’s best scientists and engineers in academic and other non-profit research institutions. It

supports research pertaining to human health and the environment

and it is designed to maximize the independence of the researchers

it supports and to provide an equal opportunity for all researchers

to qualify for that support.

Over the past six years, a number of occasions have arisen where

I have had the opportunity to review the STAR program. When I

was chair of EPA’s Office of Research and Development’s Board of

Scientific Counselors, I oversaw the review that BOSC conducted

in 1997 of EPA’s National Center for Environmental Research and

Quality Assurance, and the STAR program, which was operated by

that center. Again, in 1999/2000, while still chair of BOSC, I

oversaw a review of the STAR program that BOSC and EPA’s

Science Advisory Board conducted jointly. Now as I mentioned earlier, I served on the NRC Committee to Review EPA’s Research

Grants Program. The findings and conclusions from this most recent report, along with those from the previous BOSC reports, lead

me to conclude the following.

88

First, environmental regulatory decisions must be informed by

the best science. The STAR program is judged to be the best mechanism that we have for providing the very best science through extramural sources. Second, research in STAR is focused on EPA’s

and the country’s greatest environmental needs. And finally, the

STAR program has an exceptional process for the peer-review of

proposals. The NRC Committee stated in its conclusions that the

STAR program ‘‘compares favorably with and in some cases exceeds that in place at other agencies that have extramural research

programs, such as the National Science Foundation and the National Institute of Environmental Sciences Health,’’ which is part

of NIH.

The Committee, in its deliberations, developed its own metrics,

and in that case, developed a series of nine questions, and for each

question, there was a finding and a recommendation. Those are delineated in my written testimony, and also as a part of this report,

which is called The Measure of STAR: Review of the U.S. Environmental Protection Agency’s Science to Achieve Results Research

Grants Program.

Mr. Chairman, that concludes my testimony. I would be pleased

to answer any questions. Thank you and the Members of your

Committee for the opportunity to participate in this hearing.

[The prepared statement of Dr. Denson follows:]

PREPARED STATEMENT OF COSTEL D. DENSON

Good morning, Mr. Chairman and Members of the Committee. My name is Costel

Denson. I am a Professor of Chemical Engineering at the University of Delaware

where I have been employed since 1977. I have also served there as Vice Provost

for Research. In that position I had oversight of, and was responsible for, all aspects

of the research enterprise at the university. Recently, I served as a member of the

National Research Council (NRC) Committee to Review EPA’s Research Grants Program. The NRC is the operating arm of the National Academies of Sciences and Engineering. I am pleased to be here to discuss the unanimous findings and recommendations of that committee.

The Environmental Protection Agency (EPA) is a mission agency established to

protect human health and to safeguard the natural environment. EPA’s regulatory

and decision-making role requires that the agency have access to the best available

science that is relevant to its mission. In an effort to improve the scientific foundation of its decision-making process, the agency’s Office of Research and Development

established the Science to Achieve Results (STAR) research grants program in 1995.

STAR is a competitive, peer-reviewed, extramural research grants program created to encourage interagency collaboration and to increase EPA’s access to the Nation’s best scientists and engineers in academic and other nonprofit research institutions. It supports research pertaining to human health and the environment and is

designed to maximize the independence of the researchers it supports and to provide

an equal opportunity for all researchers to qualify for support.

Over the past six years a number of occasions have arisen where I have had the

opportunity to review the STAR Program. When I was chair of EPA’s Office of Research and Development’s Board of Scientific Counselors (BOSC), I oversaw the review that BOSC conducted in 1997 of EPA’s National Center for Environmental Research and Quality Assurance, and the STAR Program, which was operated by that

center. Again, in 1999/2000, while still chair of BOSC, I oversaw a review of the

STAR program that BOSC and EPA’s Science Advisory Board conducted jointly. And

now, as mentioned earlier, I served on the NRC Committee to Review EPA’s Research Grants Program.

The findings and conclusions from this most recent report, along with those from

the previous BOSC reports, lead me to conclude the following:

1. Environmental regulatory decisions must be informed by the best science:

the STAR program is judged to be the best mechanism that we have for providing the very best science through extramural sources.

89

2. Research in STAR is focused on EPA’s and the country’s greatest environmental needs.

3. The STAR Program has an exceptional process for the peer-review of proposals. The NRC committee stated in its conclusions that the STAR program

‘‘compares favorably with and in some cases exceeds that in place at other

agencies that have extramural research programs, such as NSF and NIEHS

(NIH).’’

In 2000, EPA asked the NRC to conduct an independent assessment of the STAR

program. In response, the NRC established a committee and gave it the following

tasks:

Assess the program’s scientific merit, its demonstrated or potential impact on

the agency’s policies and decisions, and other program benefits that are relevant

to EPA’s mission;

Recommend ways to enhance the program’s scientific merit, impact, and other

benefits; and

In the context of other relevant research conducted or funded by EPA, and in

comparison with other basic and applied research grants programs, address the

STAR program’s research priorities, research solicitations, peer-review process,

ongoing research projects, and results and dissemination of completed research.

In 2003, the committee completed its report, The Measure of STAR, Review of the

U.S. Environmental Protection Agency’s Science to Achieve Results (STAR) Research

Grants Program.

In its evaluation, the committee developed a series of nine specific questions that

it considered were of greatest importance to the research program. I will present

each of these questions, along with the committee’s overall findings and recommendations for the STAR program.

1. Should the STAR program continue to be part of EPA’s research program?

Finding. EPA requires a strong and balanced science and technology research program to fulfill its mission properly. The STAR program is an important part of the

overall EPA research program.

The STAR program is EPA’s pre-eminent program that solicits independent scientific and technologic research from the Nation’s best academic and other nonprofit

research institutions. The program has established and maintains a high degree of

scientific excellence. Through broadly advertised, competitively awarded, peer-reviewed grants, the STAR program provides the agency access to independent information, analyses, and perspectives.

The STAR program provides the agency access to a broad community of researchers, allows it to fund research at the cutting edge of science, and assists it in addressing information gaps that it does not have the internal resources to address

properly. The STAR program also encourages its grantees to disseminate their research results widely in peer-reviewed scientific journals.

Recommendation. The STAR program should continue to be an important part of

EPA’s research program.

2. What is the unique contribution of the STAR program?

Finding. The STAR program funds important research that is not conducted or

funded by other agencies.

For instance, the STAR ecologic-indicators program is the primary source of support of research on the development of water-quality indicators for biologic monitoring. The interdisciplinary centers that STAR has funded also represent an innovative approach to supporting research that will be most relevant for environmental

decision-making.

The STAR program has also made commendable efforts to leverage funds by establishing research partnerships with other agencies and organizations that have

similar or complementary research interests, including the National Science Foundation, the Department of Energy, the Office of Naval Research, the National Aeronautics and Space Administration, the National Oceanic and Atmospheric Administration, the U.S. Department of Agriculture, the National Institute of Environmental Health Sciences, the Department of Interior, and the American Waterworks

Research Foundation.

Recommendation. STAR should continue to partner with other government and

non-government organizations to support research of mutual interest and of relVerDate 11-MAY-2000 11:30 Dec 21, 2004 Jkt 092340 PO 00000 Frm 00093 Fmt 6633 Sfmt 6621 C:\WORKD\ETS04\031104\92340 SCIENCE1 PsN: SCIENCE1
90

evance to EPA’s mission, explore innovative approaches for carrying out this research, and sponsor a diverse portfolio of research that alerts the agency to emerging issues and provides independent analyses of issues that the agency is currently

addressing.

3. Does the STAR program have adequate processes to ensure that it is

sponsoring high-quality and relevant research?

Finding. The STAR program has developed a grant-award process that compares

favorably with and in some ways exceeds that in place at other agencies that have

extramural research programs, such as the National Science Foundation (NSF) and

the National Institute of Environmental Health Sciences. An unusually high degree

of planning goes into identifying the specific research subjects to be supported. The

agency also puts considerable time and thought into preparing effective research solicitations and into funding projects that are relevant to its mission and program

needs. Furthermore, the STAR program has established a rigorous peer-review process.

Recommendation. The STAR program should continue to improve the focus of its

Request for Applications, and when the agency does not have the capacity internally

to adequately define the state of the science in a particular research field, STAR

should consider greater use of external experts to assist in identifying the highestpriority research and data gaps.

4. Is the STAR program producing high-quality research results?

Finding. Although it is still too early for comprehensive evaluations of the research

results of the STAR program, some STAR research efforts have already substantially improved the scientific foundation for decision-making.

Many STAR projects have resulted in articles in highly respected, peer-reviewed

journals a traditional measure of research quality. These STAR research results

have already helped to improve our understanding of the causes, exposures, and effects of environmental pollution information critical to improving the scientific foundation for decision-making. For instance, STAR-funded research on particulate matter has helped to improve our understanding of the biologic mechanisms by which

inhaled ambient particles cause health effects and the nature of some of these effects. These data are critical to future regulatory decisions regarding our nation’s

ambient air quality.

Recommendation. EPA should continue its efforts to attract ‘‘the best and the

brightest’’ researchers to compete for STAR funding.

5. Are the STAR program results useful for EPA decisions and processes?

Finding. The STAR portfolio effectively supports EPA’s mission and research and

development strategic plans and GPRA goals. Specific STAR research projects have

yielded significant new findings and knowledge critical for regulatory decision-making.

The STAR program is too young to be able to document fully the extent to which

its research results are being used to inform development of new regulations and

environmental-management decisions. However, some STAR projects have already

yielded information important for environmental decision-making. For example,

STAR-sponsored research in endocrine disruptors, particulate matter, and ecologic

assessment has resulted in groups of peer-reviewed publications of immediate use

in understanding causes, exposures, and effects of environmental pollution. Those

results are directly relevant to EPA’s mission to ‘‘protect human health and to safeguard the natural environment—air, water, and land—upon which life depends.’’

For instance, research on ecologic indicators has led to the development of a dynamic, economically linked model to evaluate the driving forces and ecologic consequences of land-use change.

Recommendation. The STAR program and EPA’s Office of Research and Development should develop mechanisms for documenting the extent to which its research

is being used to support the agency’s environmental decision-making, should consider using outside experts to help document systematically the ‘‘state of the

science’’ before research is initiated, and should synthesize the results of the research when it is completed to identify the specific contributions that STAR and

other EPA research has made to providing critical information.

91

6. Is the STAR program effective in providing results relevant to the appropriate audiences?

Finding. The STAR program has been commendably aggressive in experimenting

with innovative approaches to communicating the results of its funded research to

a wide variety of users and audiences, but its success in these efforts has been uneven.

The STAR program supports research of potential value to a variety of users and

audiences, both in and outside EPA. Much of the research is aimed primarily at the

scientific community and those responsible for providing technical support for environmental-management decisions.

The program, however, also has other potential users, at least for some of its research results, including other federal agencies; industry; State, tribal, and local

governments; nonprofit environmental organizations; and international environmental agencies.

The STAR program has experimented aggressively with a wide variety of communication mechanisms, including EPA’s web site and publication of reports. Through

these efforts, the program appears to substantially exceed the dissemination efforts

of most other research-sponsoring organizations, both in and outside the Federal

Government. Nevertheless, the STAR program could substantially improve its dissemination efforts by directing them more effectively to specific users and audiences.

Recommendation. The STAR program should clearly identify the intended audiences for proposed research results as early in the process as possible and indicate

them in the Request for Applications. When appropriate, EPA should consider involving representatives of the intended audiences from outside the agency to help

define the relevant research results and a strategy for their dissemination.

7. Should the fellowship program continue to be part of EPA’s research

program?

Finding. The STAR fellowship program is a valuable mechanism for enabling a

continuing supply of graduate students in environmental sciences and engineering

to help build a stronger scientific foundation for the Nation’s environmental research and management efforts.

It is the only federal fellowship program exclusively designed for students pursuing advanced degrees in environmental sciences.

Recommendation. Given the Nation’s continuing need for highly qualified scientists and engineers in environmental research and management, the STAR fellowship program should be continued and funded.

8. Are the STAR program’s funds adequate to achieve its objectives?

Finding. STAR is only able to fund less than 15 percent of the proposals received

for its individual investigator and center grants, and its funding has not kept pace

with the rate of inflation.

NIH and NSF strive to fund, on the average, 25–30 percent of the proposals received. STAR’s budget allows it to fund only 10–15 percent of the proposals it receives and only about 60 percent of those rated ‘‘excellent’’ or ‘‘very good’’ by its independent quality peer-review panels. By that measure, STAR does not have sufficient

funds to recognize all the best proposals received.

Although the STAR program’s budget grew rapidly in its first three years, it has

not kept pace with general inflation in the last few years. That is particularly true

of the STAR fellowship program. The effect of that budgetary situation is exacerbated by the fact that costs of research have outpaced general inflation for more

than a decade. Therefore, at present, STAR funds buy less research than the same

amount of money could have bought several years ago.

It is appropriate to consider the funding of the STAR program in the context of

the overall funding for all of EPA’s Office of Research and Development, which also

has not kept pace with inflation. STAR currently represents about 18% of EPA’s Office of Research and Development total funding. The committee considers that percentage to be a reasonable recognition of the value of independent peer-reviewed research to the agency.

Recommendation. STAR program funding should be maintained at 15–20 percent

of the overall research and development budget, even in budget-constrained times.

However, budget planners should clearly recognize the constraints of not having inflation escalators to maintain the level of effort of the entire program.

92

9. How should the STAR program be evaluated?

Finding. There are no easy answers when it comes to identifying metrics for evaluating research programs, and the best approach for evaluating the STAR program

is to establish a structured system of reviews by panels of experts.

The committee assessed the quality, relevance, and performance of the STAR program, as set forth in recent OMB research and development criteria, by using qualitative and quantitative metrics. That is one approach for reviewing the STAR program and similar programs. Several examples of qualitative and quantitative

metrics that were used for evaluating the STAR program are: Does the STAR program have a clearly defined plan for regular, external reviews of its research quality, and has this plan been effectively carried out? Has the program made significant contributions to advancing the state of the science in particular research topics? Does the program award grants expeditiously? Does the program have a schedule for the products it intends to produce and how well is it adhering to the schedule?

The committee’s judgment is that quantitative metrics, although outwardly simpler to use, are not necessarily more informative than qualitative metrics. In some

cases, quantitative metrics can be misleading, and emphasizing inappropriate

metrics can distort the research outputs of a program. Qualitative metrics are less

likely to have such effects, but they need to be interpreted carefully.

The committee judges that expert review by a group of people with appropriate

expertise is the best method of evaluating broad research programs, such as the

STAR program. Expert review is appropriate for evaluating both the processes and

the products of the STAR program. Both qualitative and quantitative metrics can

provide valuable support for such expert reviews.

Recommendation. STAR and EPA’s Office of Research and Development should

establish a structured program of reviews by panels of independent experts and

should collect the appropriate information to support these reviews.

Mr. Chairman, that concludes my testimony. I would be pleased to answer any

questions. Thank you and the Members of your committee for the opportunity to

participate in this hearing.

BIOGRAPHY FOR COSTEL D. DENSON

Costel Denson is a professor of chemical engineering at the University of Delaware. He received a B.S. from Lehigh University, a Master’s from Rensselaer Polytechnic Institute and his Ph.D. from the University of Utah, all in chemical engineering. His research has focused on the rheology and processing of polymeric materials and he is a fellow of the Society of Plastics Engineers. Dr. Denson has served

as Vice Provost for Research at the University of Delaware where he was responsible for the administration of all aspects of the research enterprise. He has served

as chair of EPA’s Board of Scientific Counselors and is a past chair of the Materials

Task Force at the Military Engineering Center of Excellence. Dr. Denson has served

as a member of the National Science Foundation Advisory Committee for Environmental Research and Education and on the National Research Council’s Committee

on Air Quality Management in the United States. He has also served on the Ford

Foundation Minority Predoctoral Review Panel on Engineering.

DISCUSSION

Chairman EHLERS. Thank you very much, and thank you to all

of the witnesses for being here and for your testimony.

We will now begin with the questions. The Chair will recognize

himself first for five minutes, and then each Member, in turn, will

receive a five-minute question period. If there are sufficient questions, we will continue with a second round, as well.

RATIONALE FOR SCIENCE TO ACHIEVE RESULTS (STAR)

RESEARCH GRANTS PROGRAM BUDGET CUTS

The first question is for Mr. Johnson and Dr. Gilman. You

have—been interesting in my career in the Congress. Because I am

a scientist, Members are constantly coming to me to ask my opinVerDate 11-MAY-2000 11:30 Dec 21, 2004 Jkt 092340 PO 00000 Frm 00096 Fmt 6633 Sfmt 6601 C:\WORKD\ETS04\031104\92340 SCIENCE1 PsN: SCIENCE1
93

ion on scientific issues and questions, and probably the most often

repeated phrase that I have heard is I want some good science on

this. Frequently, they will also say I want sound science, in which

case I say I don’t know anything about acoustics. But I do know

something about good science, and I have been actively involved in

trying to improve the scientific effort of the Nation in every agency,

including the EPA, and I am very pleased the EPA has made considerable progress.

It is very disappointing to see the recommendations for the big

cuts that we are seeing recommended here. The 92 percent I referred to earlier and the 35 percent in STAR, and the question for

Mr. Johnson and Dr. Gilman is why the cuts? Where does this

come from? This is not just a reallocation. This is really a cut, unless you can show me that you are transferring money to some

other agency to do the same type of research. Particularly in the

environment, Members of Congress are very interested in good

science. They want it done right. They want to know the right answer so they can make the right decision, so I would appreciate

comments from both of you on this issue.

Mr. JOHNSON. The more substantive answer will come from Dr.

Gilman, I suspect, but let me just give you my general view of this.

As we mentioned, we think the focus should be on what we are

getting, what the result is, what we are learning and what—how

we are advancing our usual pursuit of a cleaner, healthier environment. The feeling was that the 20-some odd million dollars would

be better spent in other programs than in the Ecological Research

and Pollution Prevention Programs because we believe that there

are more specific deliverables that could be resulting from those

programs. We think those programs are well-funded now, and we

believe that there would be little detriment to the output of those

research programs with those cuts. We believe we could produce a

greater return for the 20-some odd million dollars if we put those

monies in the Prevention of Pesticides and Toxics Program.

I think this President has demonstrated his commitment to all

the things that EPA stands for and all of these environmental objectives that we all share. This is—should not be considered to be

a back-off in the President’s or the Administration’s commitment to

a healthy, clean environment. It should be considered to be a focus

on what we should be—what we are trying to get for our money,

not how much money we are trying to spend. Now for the real answer, Dr. Gilman.

Dr. GILMAN. Well, there is no question there is competition for

resources. The Agency is putting forward a number of aggressive

efforts proposing a $150 million increase in Superfund, proposing

a School Bus Diesel Retro-Fit Program on the order of $60, $65

million. There is also an initiative in leaking underground storage

tanks. You know, the list goes on. These are the kinds of things

that create a competition for funds within the Agency. So in terms

of where do we place the dollars, that is certainly an element for

consideration. For the ecological research, the program still has a

substantial effort beyond the cuts, as is the case in pollution prevention. So we think we can still drive those programs to have good

work done and good outcomes from that work, even with these reductions.

94

Chairman EHLERS. But I am really puzzled because science is

the basis for what you do, and it seems to me you are transferring

money out of scientific research and into programs, and I just can’t

believe that the science has been done. That you don’t have to do

it anymore, and that it should be cut by that. I am particularly

concerned about the STAR program, which has received high ratings. You heard two panelists here today make good comments

about it, good reviews, and it is generally acknowledged as being

one of the strongest environmental programs. Why cut STAR by 35

percent? And I know there is always a tendency of managers to say

well, we have got to protect our own employees, and therefore, we

will cut the outside. But you get more for your money with STAR

than I think you get almost anywhere else.

What is the justification for cutting STAR?

Dr. GILMAN. Again, the decision wasn’t made on the basis that

STAR is a bad program. Decisions were made on the basis of the

program reviews and the elements of the ecological and pollution

prevention programs.

Chairman EHLERS. What are these program reviews you are referring to?

Dr. GILMAN. The program reviews that we did in collaboration

with the Office of Management and Budget’s PART exercise.

Chairman EHLERS. So you are saying the cuts are a result of applying PART?

Dr. GILMAN. Yes, for these two programs, that is the case, but

not as it relates to performance of the STAR grant program per se.

The reductions were made in the STAR grant programs because,

just as those programs are the place where we can easily ramp up,

it is also the place where, in terms of managing the entire research

enterprise for the agency, we look to ramp down when we have to

make a reduction. It is the place where we do the least disruption

to our infrastructure for research when we have to make those reductions.

Chairman EHLERS. But that implies that you are doing it because it is easier to ramp down, even though some of the best

science is done there.

Dr. GILMAN. Well, I wouldn’t say it necessarily is better than,

though it sometimes is different than, the work that is going on inside the Agency. But the alternative of cutting back on the resources for the Intramural Program or reducing the Intramural

Program is for the longer run. When we are very hopeful of re-engineering these programs and getting them their better measures

and their better performance—to begin to reduce the internal infrastructure when we hope to be able to turn these things around

doesn’t make sense in our opinion.

Chairman EHLERS. Well, it doesn’t make sense to me when the

President is trying to freeze the discretionary non-defense, non-security budget that we are cutting a major effort on your part. I

mean, all of EPA is being cut, and that also doesn’t make sense to

me. If we are freezing, let us freeze everything where it is, but let

us not cut to this extent.

My time has expired. Let me just add one point, and after spending a fair amount of my life in science and part of it in managing

science, I can assure you that managing science is one of the most

95

difficult enterprises around because it is very difficult to develop

measures for measuring science because you really literally do not

know what the outcome is going to be. And it takes a special type

of expertise and I am very, very suspicious of instruments and

methodologies that are applied across the board by any agency,

whether OMB or something else, that just says okay, fill in the

blanks and then we will make a decision. That does not do justice

to the enterprise, and I think we have to be very cautious about

that.

Mr. Johnson, did you want to respond before I go on?

Mr. JOHNSON. Yes, I would. I think we recognize that R&D needs

to be looked at differently than an operating program, and so the

questions are different. I have gotten to know Eli Zerhouni in the

last year plus, and he talks about the opportunity to better manage

our investments in research better than we are now, and he understands way better than I and comparable to your understanding as

to how difficult it is to measure this, but these can be managed.

Shame on us if we don’t continually try to deal with all of its difficulty, try to get a better understanding of what we are getting

and are we on track, and is there accountability? And we understand that the end product, especially of basic research, is by definition an open question. But like every dollar we spend, we must

continue to try to find better ways to make sure that the money

is well spent.

Chairman EHLERS. I don’t agree with your goal. I disagree with

the result you have come up with. I have—my time has expired.

I would like to—who came first? All right. Zoe, you were first.

Please recognize Congresswoman Lofgren from California.

INCORPORATION OF STAR SCIENTIFIC RESEARCH INTO

DECISION-MAKING

Ms. LOFGREN. Thank you, Mr. Chairman. I thank you for holding

this hearing today. I think it is an important one, and although the

testimony has been a little bit dense, I think we can really distill

what is going on here so that it is easily understood.

I remember years ago a Member of the House giving me a phrase

about OMB. That they know the cost of everything and the value

of nothing, and I think we have got that kind of fight brewing

here—I don’t mean disrespect, but it is a word we use often in the

House, the bean-counters versus the scientists, and when it comes

down to good science, I am going to rely more on the scientists than

I am on the CPAs. And it does disturb me that the basic research

that we are—have gotten through the STAR program, which has

been evaluated by the scientists, and Dr. Denson, I found your testimony, written testimony particularly helpful because as you went

through the analysis that has been undertaken of the program, it

is not without rigor, and it—but it is not mechanistic. It has been

an analysis that actually is thoughtful. It is not, you know, check

off the boxes, and I think that is the kind of analysis that we need

to have.

And that doesn’t mean that changes won’t sometimes result because I know—I am not a scientist, as the Chairman is, but I have

certainly—I come from Silicone Valley. I live surrounded by scientists, and there is not a more vicious critic of science than anVerDate 11-MAY-2000 11:30 Dec 21, 2004 Jkt 092340 PO 00000 Frm 00099 Fmt 6633 Sfmt 6601 C:\WORKD\ETS04\031104\92340 SCIENCE1 PsN: SCIENCE1
96

other scientist. So that can certainly result. But I am just sort of

wondering, when you look at these nine specific results, on page

eight of your testimony, really the—one of the things that you discovered was not that the science that was being developed was

flawed or defective, but there were big questions about the extent

to which the research was actually being used, I am quoting you,

‘‘to support the agency’s environmental decision-making.’’ So the

question, as I understood it, was really not about the science delivered through the STAR program, but the agency’s inability or questioning their ability to actually incorporate the data in their decision-making.

Is that a fair analysis of that testimony? I don’t want to misquote

you, but that is the way it seemed to me.

Dr. DENSON.I think that is an accurate assessment. I would add

one other thing, and that is that it takes a while for the research

results to percolate through the system and to apply the results to

some known situation. These results, the research results in many

ways are autocatalytic in that they feed on each other, and so once

you have a set of results, a scientist someplace reads those results

and has another idea that yet takes it to another level. And finally,

it gets to a critical mass where it becomes very obvious how these

results can be used to implement a particular program.

My own sense, and this is my personal reflection since I have

been involved with it since nearly the start, is that the STAR and

the agency have made great strides in improving the program and

improving the way it implements the results. They have a way to

go yet. I think the STAR program has some improvements to make,

and we have mentioned what those are. But in every step along the

way that I have been involved in the evaluation of STAR, when

those comments have been made, the agency has responded very

promptly and forthrightly in trying to make the changes and the

improvements.

EFFECT OF ECOLOGICAL RESEARCH PROGRAMS BUDGET

CUTS ON HUMAN HEALTH RESEARCH PROGRAMS

Ms. LOFGREN. Well, listening to your testimony and really the

testimony of all the witnesses, it seems that the extramural portions of EPA’s research efforts, including the research on ecological

systems, I am particularly interested in the mercury analysis, are

basically just not going to happen if the STAR program sustains

the kind of cuts that are recommended. You are a prominent scientist, and I thank you very much for volunteering to do this for

so many years. Is that your assessment as well?

Dr. DENSON. I believe that—I believe personally that it would be

detrimental to cut the ecological programs for several reasons. One

is that you have to recognize that EPA and ORD is a regulatory

agency, and conducting research in a regulatory agency is different

than conducting research in other kinds of agencies. One feeds on

the other. It is a coupling effect. What happens in research on

human health informs what happens in research and the results on

the ecology, and vice-versa. So they are intimately coupled. And if

you cut one, then in an agency of this sort, I think you seriously

undercut the ability to do research across the board.

97

My second point is this, is that the National Science Foundation

conducts, in the environmental area, research on ecological problems. I mean, they don’t do human health research.

Ms. LOFGREN. Right.

Dr. DENSON. The National Institute of Health does human health

research. So we have two compartments. And what EPA, ORD is

doing is to bring those two together because they are in a regulatory agency and they have to conduct research, one arm which

informs the other. So to cut the ecological part, impairs EPA’s and

ORD’s ability to carry out its mission. See, because that is part of

their mission, and they need that, as we said in this report, they

need a strong research arm.

Ms. LOFGREN. I know my time is over, Mr. Chairman, but I

would just note that at a time when there has been a wide concern

about proposals to increase the amount of mercury that can be permitted in the environment, I think the American people are going

to be very suspicious of a proposal to cut research into the mercury

element, while, you know, polluters are being promised relief for

more mercury in the environment. I think it is a terrible message

to give to the American people who I don’t think ever asked for

more pollution in their children.

Chairman EHLERS. The gentlewoman’s time has expired. Pleased

to recognize the gentleman from Texas, Mr. Burgess.

IMMEDIATE EFFECTS OF SCIENCE TO ACHIEVE RESULTS

(STAR) GRANTS PROGRAM BUDGET CUTS ON EPA RE- SEARCH

Mr. BURGESS. Thank you, Mr. Chairman, and I also thank you

for calling this hearing today.

Let me first ask Dr. Gilman if the Science to Achieve Results, if

the cuts in that program are implemented as written, what do we

lose? What research are we going to—what is going to be the immediate effect of that?

Dr. GILMAN. There are a set of requests for proposals that we

would have otherwise put out on the street that we will not put out

on the street, but none of the cuts that we are proposing to the Extramural Program would, in and of themselves, terminate research

overall. So there is still a substantial ecological research effort underway. It behooves us to do more towards collaborating with some

of our other federal agencies to stretch the dollars we have there.

In the case of pollution prevention, we have a substantial Pollution

Prevention Program remaining. We are in the process of re-engineering that program, and there is a true connection between the

results of the PART process and our efforts to re-engineer that

process and change its focus. So none of the cuts to the program

will terminate initiatives in either ecological research or in the pollution prevention research.

Mr. BURGESS. Well, then taking the last point from the

gentlelady from California, will the mercury research and the research on human endocrine disruptors, will that continue?

Dr. GILMAN. That research does continue. The mercury research

is a good example. Actually, we received a cut from the Congress

in our proposed budget for 2004, which made us cut back on an initiative we had proposed in mercury-related research. In effect, the

98

proposal for 2005 would take that off the table, as the Congress did

as well, but we still have an intramural program. The big focus of

our intramural program at this point is trying to better understand

how we can do continuous monitoring of emissions for mercury so

that we can put in place the regulatory program that has been proposed. The regulatory program that has been proposed for mercury

reduction does constitute the largest reduction that has ever been

proposed, and it is the first time that power companies will be required to reduce their mercury emissions.

Mr. BURGESS. Okay. Let me be clear on that then, that program

is going to continue to go forward. Is that correct?

Dr. GILMAN. Yes.

Mr. BURGESS. All right. Then just to give us a flavor of what is

on the shelf in those requests for proposals, what are some of the

things that will not be brought forth as a result of the reduction

in funds?

Dr. GILMAN. In substance, there is no one particular body of research that won’t go forward. It is the supplemental work that

would be done by extramural researchers. There is no question that

that supplemental research would be of value to us, but it is not

anything that we can’t make up for in the future, in my opinion.

Mr. BURGESS. Very well. Dr. Denson, and too, I thank you for

providing such a comprehensive but readable paper for us. You allude in here to that you do research that is not funded by other

agencies, but there is also some discussion about leveraging your

funding and partnering either with I guess other agencies or with

private, nongovernmental organizations. Can you talk to us about

that just a little bit? If the proposed cuts go through, how are you

prepared to carry forward to make sure to maximize your research

dollars and to leverage those dollars so that they are spent effectively?

Dr. DENSON. I would speak to that question, sir, as a Member

of the NRC Committee. If I understand the question correctly, my

response would be this. That one of the areas where we gave STAR

program very high marks was in the area of partnerships and collaboration. The partnerships allowed STAR program to greatly leverage the funds that they have, and in particular, in terms of the

ecology, I am thinking of leveraging the funds to the National

Science Foundation and the Program for Technology for Sustainable Environment. So areas where there are not a great deal of resources in the STAR program, they have been very diligent with

the money they do have in leveraging it with other organizations.

As I said, NSF is a good example because they only work on ecological problems, on environmental problems.

And the STAR program has been diligent enough to take what

resources they have to leverage those with NSF. If those monies

are cut, then one would have to look at the leveraging effect because not only do you lose the money that is cut, you lose what you

could have gotten in that leveraging effect.

Mr. BURGESS. Very well. I see my time has expired. I take your

point that we will lose the multiplier effect. I want to thank the

panel for doing this today, being so well-prepared. It has been very

informative to me, and I will yield back.

99

APPLICATION OF THE PROGRAM ASSESSMENT RATING TOOL

(PART) AND ALTERNATIVE REVIEW METHODS WITHIN

EPA’S OFFICE OF RESEARCH AND DEVELOPMENT

Chairman EHLERS. The gentleman’s time has expired. Mr. Davis

has stepped out of the room temporarily, so I will proceed with my

next questions and we will recognize him, should he return.

I just wanted to note, Dr. Matanoski, you in your testimony said

that PART appears to favor short-term R&D programs with nearterm benefits at the expense of programs of long-term benefits. I

also noticed Mr. Posner has also had some criticisms of PART, and

the real question is what is a good evaluation program? Mr. Gilman, you said that EPA is committed to developing good standards

by which to measure EPA’s R&D programs. A question, Mr. Johnson, is what standards would OMB find acceptable? It seems to be

the thinking of the scientists here and GAO that PART is not applicable as it has been used to the scientific programs. I am wondering, and I am sympathetic to good management of science. I always have been.

The question is can we do it right? I remember we worked very

hard in this Committee when GPRA came out to modify GPRA as

it is applied to science so that it made sense, and after some stumbles, I think that has begun to work. The question is, Mr. Johnson,

what standards would OMB find acceptable if we are going to improve this? How can we make PART fit science, rather than the

present situation where we have at least expressed opinions here

that PART doesn’t really fit in this situation? The Floor is yours.

Mr. JOHNSON. I can’t speak, maybe Dr. Gilman can, on what specific opportunities we have to improve the R&D criteria that are

addressed in the PART R&D, R&D PART. But I do know Dr.

Zerhouni is supportive of use of the PART in assessing various research programs, and the PART will get better and better each

time. There is nothing magic about the PART, these 25 questions.

We think these are good questions now, and they will get better

every year as we get smarter and smarter, and the goal here is to

look at, on a regular basis, everything we spend money on, including science, and ask ourselves is this consistent with our priorities?

Do we think we are getting something for our money? Is the research money of these long-term projects being well-managed, and

if not, what can we do about it?

But I don’t know the answer to your specific question.

Chairman EHLERS. Okay. I will make a comment, but let me let

the others comment first, and we will go through the whole panel.

Dr. Gillman?

Dr. GILMAN. Okay. Mr. Chairman, EPA is not GPRA-phobic. We

are not afraid to be measured. We, the science arm have tried

hard. We need to do better. One of the things we have done is a

pilot with the Office of the Inspector General, trying to develop a

way of planning our program and measuring our program that

really does look to long-term outcomes. That is a pilot effort that

we have brought before the Office of Management and Budget and

hope to continue the discussions whether we can look at the program slightly differently? To try and sum it up for you, when you

are developing research in science in a mission-oriented Agency

100

like our own, it is difficult to look at the results of that work in

the absence of asking yourself how well has the program integrated

the results that were to achieve the long-term outcomes?

So what we are, in short, proposing is an integrated process,

looking at both the program and the research arm to ask the question how is the Agency doing in getting the outcomes it wants with

the research it is engaged in?

Chairman EHLERS. As you know, we have had a number of discussions about integration and I am very interested in strengthening that effort within the EPA by increasing the scientific component of it. Mr. Posner.

Mr. POSNER. Thank you, Mr. Chairman.

I want to make sure we are not saying that PART shouldn’t be

applied to science programs. I think recognizing that there are particularly unique challenges, given the long-term nature of how results unfold and the uncertainties with science programs, means

you have to be careful, and OMB does have particular questions for

R&D. That may be one area that needs to be focused on. I want

to add, thanks to the National Academy of Sciences, the National

Science Foundation and others, there are developed criteria for

doing this.

One of the things that we typically find when we look at research, such highway research, is it the success story model, which

is really not the way necessarily to go, where agencies reach for individual good examples to justify their programs. But what we

want to see and what has been developed in the National Science

Foundation and others is a way for scientists to get peer-reviewed

retrospective evaluations of portfolios of research. This is something that has been developed in other agencies and it is the kind

of thing that I think OMB has endorsed in their research criteria.

The National Academies of Science really laid this out as a model.

So when we are talking about having leading scientists review

the results of science programs, that is the model of accountability

that makes sense here.

Chairman EHLERS. Well, thank you, and what we are specifically

interested in is what specific measure should OMB use to evaluate

EPA’s R&D programs, or for that matter, other R&D programs? So

we will continue with Dr. Matanoski.

Dr. MATANOSKI. I think our committee, the Science Advisory

Board, in looking at the actual outcomes, suggested that the shortterm outcome would fare much better in the PART review than

would any long-term outcome. Now if you add on top of the longterm outcome something like ecological research, which is so complex, it has so many ramifications to it that when you go back and

start looking at one basic piece of it and then carry it all the way

to its completion at the end, which we don’t even do to be truthful—we don’t even know what is the best ecological end for most

of what we are looking at, take that and try to produce an outcome

with it becomes extremely difficult.

It is easy to say I have to model something and I go in and I

model that, like whether jellyfish are doing well in the Chesapeake

Bay, and if you can do that, you can do that very well and you can

do certain pieces of research and figure it out. But when you try

to put the whole thing together, the entire fish population, the enVerDate 11-MAY-2000 11:30 Dec 21, 2004 Jkt 092340 PO 00000 Frm 00104 Fmt 6633 Sfmt 6601 C:\WORKD\ETS04\031104\92340 SCIENCE1 PsN: SCIENCE1
101

tire growth population, the entire water systems of all of these, it

becomes a very complex not only research arm, but a very complex

outcome arm. And so the long-term outcome is you want something

better in your coastal waters, for example. But all the pieces that

have to go into the research to get there and then the application

become too complicated to measure simply with a tool.

I agree wholeheartedly that everybody has to be accountable for

their money, in science as well as anywhere else. It is just the

same as if we were in a business. But by the same token, when

you get a complex program like that, it is very difficult, and our

recommendation was that they begin to think about instead looking

at a committee that looked at the tool along with an actual science

review group, and then gradually, you could work aware, as you

began to figure out what was the best way to do this. You could

begin to work your way back and evaluate each piece. Right now,

it is such a complex issue; it would be very difficult to do that.

Chairman EHLERS. Dr. Denson.

Dr. DENSON. Yes, sir. I am going to answer the question in two

ways, from two perspectives. One is that from the perspective of

the Committee, the NRC Committee. That Committee attempted to

do guidelines at that time, those guidelines required the Committee

to look at metrics of performance, quality and relevance of the program. And the Committee judged the programs on those bases. The

metrics they used were both product and process metrics. The

Committee discarded bibliometric types of analysis, quantitative

types of analysis in evaluating the program, and instead concluded

that complex research programs, and I use the word programs very

carefully, require a different kind of evaluation. And that evaluation means bringing in a panel of experts to peer-review the program in all of its aspects. To bring in a panel, peer-review all of

the aspects of the program.

And as we point out in the report, we call this a level three or

a level four evaluation. Level one evaluations, which look at individual grants, the progress of an individual research project can be

graded quantitatively, and you can use bibliometric analysis for

that. But the Committee concluded that that was inappropriate for

a program as complex as the STAR program, which we consider to

be a level three program. The ORD program in general would be

a level four program, and you would need a much bigger, much

broader panel of experts to review that.

I am going to change for just a moment, sir, and bring out the

fact that for four years, I served on the National Science Foundation’s Advisory Committee for the Engineering Directorate. I also

served for two years on the National Science Foundation’s Directorate Advisory Committee for Environmental Research and Education. So I have a perspective of how it is done in one of the very

best research agencies. And there, while they follow the GPRA and

have the tools, ideas and people concern, there it is also difficult

to do a metric on some of the programs until they have had a

chance to fully develop. And my point about the STAR program is

that it just needs a little bit more time, and I think that given that

time and more results and having a review of a complex program

by a peer-review panel would be the metric to use, and we say that

in our report.

102

Chairman EHLERS. My time has expired. We will recognize the

gentlelady from California, Ms. Lofgren.

TERMINATION OF EPA’S BUILDING DECONTAMINATION

RESEARCH PROGRAM

Ms. LOFGREN. I wanted to explore a specific program and—because I serve on the Homeland Security Committee as well, and I

am on the Cybersecurity and Science Subcommittee, and we are obviously spending a lot of time in that area. And I was interested

in the termination of the Building Decontamination Research Implementation Plan, and I was wondering, Dr. Gilman, it is my understanding that last year, the National Academy of Sciences, at

EPA’s request, reviewed that plan. They indicated, the National

Academy did, that it was not possible really for EPA to get the job

done in the timeframe remaining. I am interested in why EPA

wants to terminate the program, rather than extending it and is

that your judgment that building decontamination is now a lesser

threat, or is that because this decision was made in conjunction

with the Department of Homeland Security, Mr. Ridge, and is it

going to be transferred to Homeland Security? Because we haven’t

seen it surface yet in that department.

Could you give me some report on——

Dr. GILMAN. Sure.

Ms. LOFGREN.—that?

Dr. GILMAN. Let me start by giving you a little background of the

research effort at EPA, which we are very proud of. We set out to

establish a center there. It is something of a virtual center. We

gave it a time, a sunset in creating the center, a three-year window

to do the work. Why did we do that? We did that because we wanted to really drive the folks involved in delivering products to first

responders and to planners. So we created the center. Its mission

was to work heavily through partnerships with Homeland Security,

the Army, Air Force, Food and Drug Administration, all of the different groups where we might find technologies and approaches to

utilize in the mission areas for the EPA—they include water infrastructure, both drinking and sanitary. They include buildings, and

it includes the use of tools for rapidly assessing risks for different

factors.

Those are the three areas the research center is focused on. We

are in the process right now of working with those various agencies

who view themselves as our customers to determine what we

should do about that three-year sunset, whether there should be a

further research effort within a center embedded elsewhere in the

Agency. The decontamination program, that again was meant to be

a three-year program. That is what the National Research Council

reviewed for us. We are in the process now of trying to identify

those things that were in the third year of our program that we

would like to move up into the second year. We haven’t expended

all the resources that have been appropriated for the first two

years yet, so we do have the flexibility to change our priorities, and

as I said, we are in consultation with Department of Homeland Security and others now about what their desires are for our future

work.

103

I think the products that we have turned out so far are quite important and we are very, very proud of the productivity of that

group to this point in time.

Ms. LOFGREN. It looks to me that the sun set on the thing. It is

not really a work in progress, and it may be that we want to transfer the whole thing over to Homeland, but it is just I don’t think

that has happened, and with so many things in the Homeland Security area, not just environmental issues but where you have an

activity going on by another agency and it never really gets picked

up and it leaves the United States exposed and bare, I mean, in

many respects, not just in the environmental arena, I am concerned that this is another one of them.

My time is just about up, Mr. Chairman. I would just say that

I think this has been a very useful hearing. One thing I think has

become very clear in the questioning is that the mechanistic application of PART two science is not going to serve the country well,

and I really appreciate your leadership, not only as Chairman, but

as a scientist in helping this committee, and then later, I think the

Congress to understand that that will not well serve our country

and our future. So I yield back the time, and thank you once again.

PEER REVIEW AND EVALUATION OF R&D PROGRAMS

Chairman EHLERS. The gentlelady yields back her time. Let us

continue with some of the previous discussion. There seems to be

a general agreement among three of the panel that peer-review is

the best way to review R&D programs. I just want to establish, Mr.

Johnson, Dr. Gilman, do you agree that peer-review is the best

method to review the science programs, the R&D programs?

Mr. JOHNSON. Well, I will let, again, Dr. Gilman answer in a

more detailed fashion, but I believe the PART has a very substantial role in the assessment of R&D programs. There is nothing

mechanistic, not one thing that is mechanistic about the use of

PART. Nothing happens automatically as a result of a PART score.

I would like to just add one other comment—to elaborate on something Dr. Gilman said about the building contamination research.

The balance—the majority of the funds appropriated for this research in the last two years are un-obligated. They have not been

spent or committed. There is already plenty of money available for

this effort. Additional monies are not needed in the fiscal year 2005

to continue this effort in an aggressive fashion.

Chairman EHLERS. All right. Thank you. Dr. Gilman.

Dr. GILMAN. The pilot that we worked with the Inspector General and the work that we would hope to do for the future does involve the use of peer-reviewers in evaluating both the research outputs and the long-term outcomes. The PART process envisions the

use of that kind of peer-review. The National Science Foundation

has availed itself of that. Those are the kinds of things we hope to

be discussing with the Office of Management and Budget for the

future.

Chairman EHLERS. Let me just comment. There are really two

aspects, and I have seen these get interwoven here, but not clearly

distinguished. It seems to me part of it is simply evaluating the

science per se. Is the EPA doing good science? Is it judged by peers,

it is really good work and should it be done? That is one aspect.

104

The other aspect is, in a sense, a management aspect. Does the research fit with the mission and the program of the EPA? And I

think it is important to clearly distinguish those. Both of those are

difficult to do, but they are of a different nature, and what is most

difficult is to mesh the two. But isn’t that what you are trying to

do, Dr. Gilman, with your work and Mr. Johnson, your primary interest, I believe, is does the science fit the program? Does this

make sense for the government to do? Is that a fair statement?

Mr. JOHNSON. Well, I would say more than that. Does it make

sense for what our priorities are, is it promising fruit? Is it on

track? Is the money being well-spent?

Chairman EHLERS. Right.

Mr. JOHNSON. Is it being spent in a responsible fashion? But I

would also say that with regard to this—we are talking about generalities, and we also need to talk about specifically this program,

the Ecologic Research Program. I am not a scientist, but as I understand it, Ecologic Research, the primary opportunity in this

area of inquiry has to do with the opportunity for us as a country

is better monitoring and better diagnosing. It is a pretty specific

area. Therefore, the primary benefit or the opportunities to be pursued with research is to develop tools and databases to help informed decision-making.

There is, for this kind of research, a much more specific kind of

end product, if you will, which is something you don’t—it is not the

right term for research, but it can—it is more purposeful than most

research programs, and therefore, I think it lends itself more to

PART assessment than basic research.

Chairman EHLERS. Absolutely, and I don’t argue that. However,

that is fairly low-level science, which makes it easy to analyze. I

mean, monitoring and diagnosing is—well, I don’t want to get into

Pasteur’s Quadrant here. Some of the scientists will know what

this is. But that is in the lowest quadrant. The difficult part comes

in, as one of the scientists mentioned here, synthesizing the information after you have monitored it, after you have diagnosed it,

and then synthesizing it with other results from elsewhere.

So you are looking at organism X and organism Y and organism

Z and many others and seeing how they are interacting and what

the net effect is, and you find some very surprising things there,

which you don’t anticipate, some of which create greater environmental problems, some of which actually solve your problems. For

example, depending—developing microbes that will consume toxic

materials or oil. That is very basic research, but yet it is also related to the program. If you are trying to clean up underground

storage tanks, as we have been trying to do for over a decade, that

is a very important issue, if you can clean those up with microbial

action rather than digging it up, hauling it somewhere, treating it

at a very high cost.

So it all interrelates, and that is where my concern is. How are

we evaluating how everything interrelates? And just without really

knowing as much as I should about the evaluation program, it

seems to me the PART program is ideal for what you just described, Mr. Johnson. But I don’t think it is—from what I have

seen, it is not the appropriate instrument for the broader picture,

and the question is can we all, working together, develop someVerDate 11-MAY-2000 11:30 Dec 21, 2004 Jkt 092340 PO 00000 Frm 00108 Fmt 6633 Sfmt 6601 C:\WORKD\ETS04\031104\92340 SCIENCE1 PsN: SCIENCE1
105

thing that is more appropriate for the broader picture? We are not

here to condemn EPA or OMB. We are all part of the same team.

We are trying to help develop something that really does the job

and does it in the best way for the country, and I hope everyone,

both panelists and audience, understand this. Our goal is, as I

often tell my people back home when I give a speech, I am here

from the Federal Government, and I am here to help you——

Mr. JOHNSON. Yeah.

Chairman EHLERS.—which usually brings a laugh. But that—I

am serious about that. We really want to try to find out what the

problems are and what we can do to help. Mr. Johnson.

Mr. JOHNSON. But as Dr. Poser said, or Mr. Poser said, one thing

I believe with all kinds of programs, including research and development, basic and applied, the PART is causing us to ask really

good questions, and that is more important than the kind of answers that we are getting, but it is causing us questions that we

weren’t asking before.

Chairman EHLERS. And Mr. Johnson, I have no objection to that.

I think that is an important function. The question is how those

answers are evaluated and how they are used, and what the appropriate use is. Did you have additional questions, Ms. Lofgren.

EVALUATING SHORT-TERM VS. LONG-TERM RESEARCH

Ms. LOFGREN. I think I do, but I think we are going to be having

a vote soon, and the only thing I would note is Dr. Matanoski, in

your testimony, you describe research that is targeted to fill a research gap in the short run versus other types of research intended

to provide an understanding of large systems in the long run, and

I am just wondering if it is possible to have programs that have

only short-term or long-term goals, and not both. And as I look at

the PART questions themselves, it appears that really, you are

going to get penalized, and you can’t get an approval if you have

done that, and if your understanding is similar to mine.

Dr. MATANOSKI. I think it is very similar to what you have said.

You have hit it right on the nose. If you have something that is

very simple that I am going to evaluate, an operative procedure or

something, and I know what the long—what the outcome is going

to be and they are going to come off the table alive or dead, that

is my outcome. That is easy. And I can tell you how many operations I do and how many are going to take place, you know, and

who is going to do them.

When you get into a program as complicated and integrated as

ecological research, then you really begin to have to look at the

independent parts, part of which was STAR. And so STAR was at

the very basic end of it, and some of the more applied and the monitoring, which is fairly easy, you know. You put how many monitors

out there. How much do they show us, and whether they have improved the quality of the water. But when you begin to look at the

whole picture that is a very difficult thing to do. So what do we

need to do? Separate it out so I have a product that I can say

comes out next week, and if I get it out next week, great. I have

just met my goal. And, you know, that is—if you get that simple,

like putting a car on the line and coming out at the other end, then

you can easily use a——

106

Ms. LOFGREN. Right.

Dr. MATANOSKI.—very simple metric system. The more complicated you get, the more complicated any metric system would become, and right now, we are probably stuck with not having a very

good metric system for a complicated problem.

Ms. LOFGREN. Right. In just looking at the questions, it is really

biased against basic, complex research, it seems to me. I remember,

Mr. Chairman, when I was first elected to the House and the late

Congressman Steve Schiff was on the Science Committee and I was

new to the Committee, and he actually gave an education to me

and other new members about the value of basic research, and

really its role in not just solving problems that are immediately before us, but really in building the economy of the country. And I

think we would be ill-advised to forget Steve’s lessons to us, and

I yield back the balance of my time.

CONCLUDING REMARKS

Chairman EHLERS. I thank the Congresswoman for her comments, and I think we have kept you here long enough and tortured you enough, but I certainly want to thank the panelists for

being here. You have been very, very helpful to me in helping me

learn more about it from both sides of the issue, and I appreciate

you being here. I would like to just summarize a few thoughts I

have. So far, I have not heard any convincing reason why the

STAR program should be cut so drastically, by all accounts. Not

just what we have heard here, but other accounts I have heard. It

is a well-run, competitive peer-reviewed program that produces

high-quality research, and I don’t think the proposed reduction

should take place, and so we will work with the appropriators on

that and see what we can achieve.

I also learned this morning the PART program was at least partially responsible for the funds being cut from Ecological and Pollution Prevention Research Programs, but it doesn’t appear to me

that PART is related at all directly to the cuts in the STAR program, and again, I don’t quite understand why the cut, but we will

have further discussions. And finally, I believe that PART has been

a valuable tool for OMB in a number of different areas, and I am

just not yet convinced that it is the appropriate one as it is structured to use for EPA and particularly I would be interested in how

conclusions are drawn from the PART review and how they are

used. It seems to me that the results of the PART program were

not applied uniformly across the agency. That certain areas were

cut more than they should have been, others were cut less than

they should have been, if you look strictly at the PART review. And

that is something else that we would like to investigate. Thank you

again for being here. If there is no objection, the record will remain

open for additional statements from the Members, and for answers

to any follow-up questions the Subcommittee may ask of the panelists, and I hope you will be willing to answer in writing if any Subcommittee Members wish to send you communications. Without objection, so ordered.

Hearing is now adjourned.

[Whereupon, at 11:30 a.m., the Subcommittee was adjourned.]

(107)

Appendix:

ANSWERS TO POST-HEARING QUESTIONS

108

ANSWERS TO POST-HEARING QUESTIONS

Responses by Clay Johnson, III, Deputy Director of Management, Office of Management and Budget

Questions submitted by Democratic Members

Q1. GAO’s review of the PART program indicated that PART and Government Performance and Results Act are not well integrated. GAO recommends these programs be integrated to be more complementary. Does OMB intend to adopt this

recommendation? Will OMB alter PART to conform to GPRA or will agencies

be instructed to alter their GPRA performance plans to conform to PART? Since

GPRA mandated an open process for the development of agency strategic plans

and performance measures, what process will OMB use to include Congress and

program stakeholders in the effort to conform PART and GPRA performance

plans?

A1. Responding in part to GAO’s recommendation that PART and GPRA be better

integrated, OMB clarified its PART guidance so agencies understood that PART is

simply a tool to ensure GPRA is implemented as intended. The guidance now includes the following:

RELATIONSHIP OF THE PART TO THE GOVERNMENT PERFORMANCE AND RESULTS ACT (GPRA)

The GPRA statute provides a framework under which agencies prepare strategic

plans, performance plans, and performance reports that set goals and report on

the extent to which they are achieved. The PART is a systematic method of assessing the performance of program activities, focusing on their contribution to

an agency’s achievement of its strategic and program performance goals.

The PART strengthens and reinforces performance measurement under GPRA by

encouraging careful development of performance measures according to the outcome-oriented standards of the law and by requiring that agency goals be appropriately ambitious. Therefore, performance measures included in GPRA plans

and reports and those developed or revised through the PART process must be

consistent.

The PART also extends the usefulness of performance information by defining

programs that comprise activities about which management and budget decisions are made. As a matter of sound manage practice, agencies will integrate

operational decisions with strategic and performance planning by:

• improving performance measures over time through the PART review,

• aligning budgets with programs, and

• aligning programs and measures with GPRA goals.

If agency goals are revised as a result of the PART process, agencies must engage

in whatever consultation with stakeholders, including Congress, is required by

GPRA.

Q2. In cases where OMB and the Agency disagree about the performance measure

used to evaluate a program with PART, how is the disagreement resolved?

Please provide documentation of the EPA and OMB positions on the PART evaluation of the Ecosystem Research Program. Please include documentation of the

negotiations that led to the final resolution of the differing positions and the

awarding of the final PART evaluation score for this program.

A2. OMB and agencies complete PART assessments together. We work hard to

come to agreement on every answer to PART questions. Agencies can ‘‘appeal’’ OMB

answers to PART questions to OMB policy officials, but this process is rarely used.

The give and take between OMB and agencies in PART assessments is considered

pre-decisional and therefore cannot be disclosed. However, OMB and EPA would be

happy to discuss with you and/or your staff the process EPA and OMB went through

to complete its PART assessments and, more importantly, what we are doing subsequently to improve the programs we assessed.

Q3. Under GPRA, program goals and performance matrices for programs are supposed to be adopted in consultation with Congress and outside stakeholders.

However, it appears that PART-defined program goals and performance metrics

are superseding those established under GPRA. Please provide a side-by-side

comparison of the PART goals and performance metrics for the EPA programs

109

that have been assessed under PART in the 2004 and 2005 budget and the goals

and performance metrics for these same programs developed under GPRA.

Please provide an explanation of all instances where PART and GPRA program

goals and performance metrics differ and OMB’s rationale for deviating from

GPRA to develop PART.

A3. As discussed above, the PART is simply a tool to ensure GPRA is implemented

as intended. The PART is used to ensure, among other things, that program goals

meet the high standards of GPRA. Where we find that existing goals do not meet

those standards, agencies may be required to revise the goals associated with the

programs being assessed.

I am assured that EPA conducts extensive outreach to Congress and consults with

its stakeholders when developing goals, objectives, and performance measures under

GPRA. In its Strategic Plan and Annual Performance Plans, EPA seeks to use the

highest-quality and most outcome-oriented goals and measures for which valid performance information is expected to be available. OMB and EPA consider EPA’s

GPRA goals and measures to be the definitive set for Congress and the public to

use to understand the agency’s budget in performance terms and to hold the agency

accountable for results.

The PART is not a substitute for GPRA. It is an assessment tool that informs

management decisions, enhances organizational learning, and promotes effective

strategies and program results. The PART is used to ensure, among other things,

that program goals meet the high standards of GPRA. The PART process has been

one of several drivers for improving the agency’s performance measures. Some of the

new measures identified during a PART assessment will become GPRA measures.

In many cases, newly identified measures are not ready immediately for inclusion

in GPRA documents, since it will take some time for the agency to conduct appropriate monitoring and data collection, to establish a baseline, and to set relevant

targets. In these cases the agency develops specific plans, with milestones, for putting the necessary elements in place to use it as a GPRA measure. Each of the agency’s program offices vets these measures with stakeholders, just as they do with any

newly proposed measures.

Some performance measures, notably efficiency measures, which are identified

under the PART process, are more appropriately used as management tools. In

other cases, PART assessments have been conducted on a relatively small set of

agency activities that are not readily identifiable as a distinguishable unit in GPRA

documents. In these cases, new measures identified under the PART process may

complement GPRA measures, but they are not necessarily in GPRA documents.

EPA included a discussion about the PART and performance measures in its most

recent Annual Report, which we are including with this package (see pp. 12–13,

Using Program Evaluation and Improving Environmental Indicators, Performance

Measurement, and Data Quality and pp. 109–110, Support for Environmental Results, Fiscal Year 2003 Annual Report, U.S. Environmental Protection Agency). Also

included with this package are summaries of all EPA programs assessed with the

PART. EPA, in full consultation with stakeholders, has increased the percentage of

externally reported annual performance measures based on environmental outcomes

(as opposed to activity-based outputs) from seven percent in the FY 1999 Annual

Performance Plan to 36 percent in the FY 2004 Annual Performance Plan.

Q4. OMB established two groups to consult with regarding the development of

PART, the Performance Measurement Advisory Council and an Interagency Review Panel. What recommendations of these groups were adopted as revisions to

PART for the 2005 budget presentation? What is the current status of these two

groups and what role will they play in the making adjustments to PART for the

2006 budget? Please provide documentation of the date of meetings held between

OMB and the PMAC and the IRP, a list of persons attending these meetings,

and a brief description of the issues discussed.

A4. The Performance Measurement Advisory Council (PMAC) was established in

June 2002 pursuant to the Federal Advisory Committee Act (FACA) to provide OMB

with independent, expert advice relating to performance measurement. Pursuant to

the statute, the PMAC’s duration was limited to nine months and all the meetings

were public. Minutes from those meetings are available on the OMB website at

http://www.whitehouse.gov/omb/budintegration/pmac¥index.html. The PMAC’s

recommendations are detailed in the meeting minutes.

In fall 2002, OMB established an interagency panel to review a sample of draft

PARTS completed the first year, judge their consistency with the PART guidance,

and make recommendations for how consistency could be improved. The panel also

reviewed a selected number of disagreements between agency and OMB staff on

110

how particular questions in the PART had been answered. The panel’s work reviewing draft PARTs is considered pre-decisional and therefore cannot be disclosed. In

the two subsequent years of PART implementation, similar quality control exercises

have been performed. In summer 2003, the National Academy of Public Administration convened a working group to assist OMB with the PART consistency check.

This summer, OMB conducted the consistency check on its own. We would be happy

to brief you in greater detail on the substance of the groups’ work and the improvements that resulted from it.

111

ANSWERS TO POST-HEARING QUESTIONS

Responses by Paul Gilman, Assistant Administrator for Research and Development,

United States Environmental Protection Agency

Questions submitted by Representative Zoe Lofgren

Q1. In relation to the proposed rule on mercury emissions from coal-fired electric

utilities (Fed. Reg. Vol. 69, No. 2: Friday, January 30, 2004; pp. 4651–4752):

Q1a. Provide a list and a brief description of the studies relied upon by EPA to develop the proposed mercury cap-and-trade program prior to the release of the

proposed rule in January. Please indicate the author/s of these studies and indicate where EPA has made the studies available for examination by the general public and Members of Congress.

A1a. EPA places all studies relied upon by EPA to develop the proposed and final

rules in the docket. The Clean Air Mercury Rule docket can be found at http://docket.epa.gov/edkpub/index.jsp by searching for the docket ID OAR–2002–0056.

Q1b. Administrator Leavitt has recently announced his intention to initiate and/or

obtain additional analysis on the proposed mercury regulation. What additional analyses will be undertaken? What aspects of the proposed mercury capand-trade program for electric utilities will be examined by these new analyses?

Will the analyses be undertaken by EPA’s Office of Air and Radiation or by

EPA’s Office of Research and Development? What is the anticipated date of

completion for the additional analyses?

A1b. As Administrator Leavitt has clearly stated, EPA will make sure that we have

all the analysis necessary to make the right decision about how to address mercury

emissions from power plants.

During the comment period on the proposal, EPA received a number of relevant

analyses from various groups, including both industry and environmental groups.

These analyses address many of the key issues faced by EPA and will help inform

the Agency’s ongoing mercury work. They are available in EPA’s docket, and the

Agency is seeking public comment on these analyses in its Notice of Data Availability (NODA) released on November 30, 2004. The NODA is part of the EPA process toward delivering a final mercury rule by March 15, 2005.

EPA received a number of modeling analyses from various groups, including both

industry and environmental groups. In some cases, EPA and commenters modeled

the same or similar policy scenarios, sometimes using the same model, but obtained

substantially different results due to differences in the assumptions employed. In

these cases, model-input assumptions can be better understood by comparing and

contrasting the modeling performed. The NODA shares these analyses and seeks additional comment on the models and assumptions used.

Administrator Mike Leavitt has outlined five guiding principles that provide context for additional inquiry and that narrows the focus of the Agency’s deliberations.

The five principles will ensure that the final mercury rule: (I) concentrates on the

need to protect children and pregnant women from the health impacts of mercury;

(2) stimulates and encourages early adopters of new technology that can be adequately tested and widely deployed across the full fleet of U.S. power plants utilizing various coal types; (3) significantly reduces total emissions by leveraging the

$50 billion investment that the Clean Air Interstate Rule (CAIR) will require; (4)

considers the need to maintain America’s competitiveness; and (5) comprises one of

many agency actions to reduce mercury emissions.

Q1c. What study or combination of studies led to EPA’s assertion that the cap-andtrade mercury control program would reduce mercury emissions by 70 percent

by 2015? What do EPA’s analyses show in regard to the anticipated change in

the geographic distribution of U.S. mercury emissions from coal-fired electric

utilities if the cap-and-trade mercury control program were adopted as compared to the reductions anticipated if the MACT mercury control program were

adopted?

A1c. The proposed mercury cap-and-trade program would place an emissions cap on

mercury emissions from coal-fired electricity generating units. This cap would be implemented in two phases. The second phase of the program would begin in 2018,

with a cap of 15 tons for emissions from these units. When this cap is fully implemented, emissions from affected units would be reduced by approximately 70 percent. The 15 ton cap represents a 70 percent reduction from the estimated 1999

level of 48 tons of mercury emissions from coal-fired electricity generating units.

112

Once the 15 tons cap is fully implemented, emissions would not exceed that level,

because the cap and trade mechanism places an absolute limit on allowable emissions. Because the cap is lowered in 2018, cumulative emissions reductions under

the proposed cap-and-trade program would exceed cumulative emissions reductions

under the proposed MACT. Furthermore, EPA analyses suggest that the proposed

cap-and-trade program attains more reductions from larger and higher emitting facilities than the proposed MACT.

An important feature of the proposed mercury cap-and-trade program is the ability for sources to bank emissions allowances. Sources can over-comply with the program in one period and bank remaining allowances for use in a later period. Banking of allowances provides flexibility to sources, encourages earlier or greater reductions than required, stimulates the market, and encourages efficiency. By encouraging early reductions, banking provides early human health and environmental

benefits relative to what would occur otherwise, though it results in extending the

time until the cap is reached. We expect that sources will take advantage of the

banking provision under the mercury cap-and-trade program by beginning to reduce

emissions soon after the program takes effect. Allowing banking should not affect

the cumulative mercury reductions achieved under the program.

Questions submitted by Democratic Members

Q1. EPA’s budget presentation includes program funding request changes for FY

2005 relative to the Administration’s FY 2004 budget request. Please provide the

changes for the following programs relative to the FY 2004 enacted appropriations for EPA including the funding level received in the FY 2004 enacted appropriations law and the requested funding level for FY 2005. In cases where a decrease in funding is proposed include a description of the activities proposed for

termination. In cases where and increase in funding is proposed, include a description of the new or expanded activities that will be undertaken.

Q1a. Office of Research and Development (ORD) research and development funding

for mercury.

A1a. Mercury Research, EPA’s Office of Research and Development.

EPA will maintain in-house research in this area, but plans to award five fewer

Science to Achieve Results (STAR) grants for mercury research in FY 2005.

Q1b. Office of Air and Radiation (OAR) research and development funding for mercury

A1b. In the Science and Technology Appropriation in the FY 2004 Operating Plan,

OAR has $361,700 for mercury activities; OAR has requested $361,700 in the FY

2005 President’s Budget. This funding supports the regulatory and guidance programs within OAR for mercury.

Q1c. ORD research and development funding on fine particulates.

A1c. Particulate Matter Research, EPA’s Office of Research and Development

The increase in resources will provide funding for a recently awarded 10-year, $30

million Science to Achieve Results (STAR) grant to study the connection between

113

long-term exposure to air pollution and cardiovascular disease. The balance of the

resources will support STAR research examining emerging biological mechanisms

that can explain observed health effects associated with exposure to PM.

The findings from these studies will assist policy-makers in understanding the

public health consequences of long-term exposure to air pollutants, and will contribute important information for PM standard setting by increasing our understanding of the biological plausibility of reported effects; the effects of PM and copollutants; who is most susceptible to these effects; and the effects of long-term exposure to PM. All these subjects are highlighted in the 2001 NRC Report IV on PM

research priorities.

Q1d. Office of Air and Radiation (OAR) research and development funding for fine

particulates.

A1d. In the Science and Technology Appropriation in the FY 2004 Operating Plan,

OAR has $33,009,800 for fine particulates activities; OAR has requested $42,762,500

in the FY 20005 President’s Budget, This funding supports the regulatory and guidance programs within OAR, particularly the Mobile Source program.

Q1e. ORD research and development funding on ozone.

A1e. Tropospheric Ozone Research, EPA’s Office of Research and Development

The decrease is due to a realignment of research support resources, and there is

no programmatic impact to this research effort.

Q1f. Office of Air and Radiation (OAR) research and development funding for ozone

A1f. In the Science and Technology Appropriation in the FY 2004 Operating Plan,

OAR has $34,167,3000 for ozone activities; OAR has requested $36,224,400 in the

FY 2005 President’s Budget. This funding supports the regulatory and guidance programs within OAR, particularly the Mobile Source program.

Q2. The FY 2005 Budget Request describes the Green Suppliers Network (GSN) activity within the pollution Prevention Program. How much funding does EPA intend to provide in FY 2005 to support this partnership program and meet the

goals of expanding the GSN partnerships to include four additional industry

sectors? How much funding did EPA supply to support the GSN pilot effort in

2002? What were the funding levels for this activity in 2003 and 2004?

A2. The promise of the Green Suppliers Network (GSN) is best illustrated by the

response to it by industry. The program is not yet a year old and already countless

Original Equipment Manufacturers and their suppliers have joined the program. It

is truly a win-win for all involved. The Original Equipment Manufacturers benefit

because the program is designed to improve their supply chains both economically

and environmentally. Small businesses benefit because they are being shown opportunities to lean their operations and improve their environmental footprints, while

at the same time achieving significant cost savings. The states benefit because the

small businesses and their clients located within their jurisdictions remain competitive, leading to job retention and viability within the state’s economy. EPA and the

environment benefit because the GSN outlines direct opportunities for environmental improvement from energy savings to waste minimization to the reduction in

the use and emissions of toxic chemicals.

When the program was launched, EPA did not expect the response from industry

to be so strong and so immediate. Currently, The Green Suppliers Network is

partnering with the Automotive Industry, the Aerospace Industry, the Office Furniture Industry and the Healthcare/Pharmaceutical Industry. However, EPA has

been approached by several other industries which have shown an interest in participating in the program. The Transportation Industry represented initially by AMTRAK is already engaged in a pilot program with GSN. The Farm Machinery Industry, represented by John Deere and others have written EPA with an interest in

participating. The Trucks and Buses industries have also registered interest and

most recently, the Retail Industry is opening a dialogue with EPA on this program.

Additional resources are needed to ensure that the technical information the GSN

114

program offers is well packaged and effectively delivered to these new industries.

Each industry that participates in this program is different and must first identify

a list of suppliers within their supply chains which will benefit most from the program. Consequently, for each of these four industrial sectors, we could be addressing

as many as 100 small to medium sized enterprises located across the United States.

EPA, in partnership with NIST and the State Technical Assistance Providers, must

make sure that they can deliver the GSN program effectively and consistently to

all of these suppliers. This will require cross-training, more extensive communication and planning and the tailoring of existing lean and clean tools to these new

suppliers. Additional resources are needed to ensure that EPA can successfully

merge lean technologies and environmental technical assistance for delivery to this

extended supply chain. Moreover, GSN has attracted the attention of foreign governments such as Canada and China, arid resources will be used to build a foundation

for GSN implementation there.

With additional resources one should expect to see measurable results from numerous small to medium sized enterprises and their Original Equipment Manufacturers that are aggregated and ready to be reported in the near-tern. The results

are collected from individual suppliers by the Manufacturing Extension Partnership

program of NIST and aggregated for EPA. The results are measured through the

tools that the NIST MEPs use in conducting an assessment. This includes, for example, the use of a tool called Value Stream Mapping which creates a material and

information flow map of a product or process. This powerful tool allows companies

to map the flow of products and information from order to cash as well as through

the supply chain. Kaizen is another tool that is used to monitor and measure continuous improvement. These lean manufacturing tools and the environmental considerations that are being built into them through GSN are designed to measure improvements as the process changes they recommend are implemented. Results measured by these systems include pollution reductions, crater and energy savings, and

company cost savings associated with those environmental benefits. These results

will contribute to achievement of new results-oriented Pollution Prevention annual

performance measures incorporated into EPA’s FY 2003 President’s Budget Request

anal related Goal 5 Strategic Targets in EPA’s Strategic Plan for 2003–2008.

Please see the table below for the requested funding levels.

Q3. Do the goals, program definitions, and performance measures for research programs that EPA has developed and included in its strategic plan align with the

goals, program definitions, and performance measures associated with OMB’s

PART analysis? How will the discrepancies be addressed?

A3. The three EPA research programs that were reviewed through OMB’s Program

Assessment Rating Tool (PART) analysis for the FY 2005 budget represent research

programs as defined by EPA, each with its own multi-year research plan. These

three programs (Particulate Matter Research, Ecosystems Protection Research, and

Pollution Prevention Research) received PART scores of ‘‘Results Not Demonstrated’’

because the Agency and OMB were unable to reach agreement regarding appropriate outcome-oriented goals and measures. For this reason, the PART worksheets

for these programs note that goals and measures are under development. EPA is

continuing to work with OMB to develop and identify appropriate outcomes for environmental research.