BBooaarrdd ooff SScciieennttiiffiicc CCoouunnsseelloorrss 

OOffffiicceeooffRReesseeaarrcchhaannddDDeevveellooppmmeenntt

UUnniitteeddSSttaatteessEEnnvviirroonnmmeennttaallPPrrootteeccttiioonnAAggeennccyy

Ecological Research 

Program Review 

Report of the Subcommittee on 

Ecological Research 

April 1, 2005 

Revised August 16, 2005
This report was written by the Ecological Research Subcommittee of 

the Board of Scientific Counselors, a public advisory committee 

chartered under the Federal Advisory Committee Act (FACA) that 

provides external advice, information, and recommendations to the 

Office of Research and Development (ORD). This report has not been 

reviewed for approval by EPA, and therefore, the report’s contents and 

recommendations do not necessarily represent the views and policies of 

the EPA, or other agencies of the federal government. Further, the 

content of this report does not represent information approved or 

disseminated by EPA, and, consequently, it is not subject to EPA’s 

Data Quality Guidelines. Mention of trade names or commercial products does not constitute a recommendation for use. Reports of the 

Board of Scientific Counselors are posted on the Internet at http:// 

www.epa.gov/osp/bosc. 
REPORT OF THE

BOARD OF SCIENTIFIC COUNSELORS

ECOLOGICAL RESEARCH PROGRAM REVIEW

Office of Research and Development 

U.S. Environmental Protection Agency 

APRIL 1, 2005 

REVISED AUGUST 16, 2005 

SUBCOMMITTEE ON ECOLOGICAL RESEARCH

Dr. Michael T. Clegg–University of California, Irvine, Chair 

Russel Frydenborg–Florida Department of Environmental Protection 

Dr. John P. Giesy–Michigan State University 

Dr. Richard Lowrance–USDA Agricultural Research Service 

Dr. Sue A. Thompson–Pennsylvania Biodiversity Partnership 

Dr. R. Eugene Turner–Louisiana State University 

Dr. Jianguo Wu–Arizona State University

EPA CONTACT

Greg Susanke, Designated Federal Officer 

U.S. Environmental Protection Agency 
Ecological Research Program Review Report i

TABLE OF CONTENTS

I. EXECUTIVE SUMMARY................................................................................................................. 1 

Structure of the Program Review.................................................................................................. 1 

Objectives of the Review.............................................................................................................. 1 

Conclusions and Recommendations ............................................................................................. 2 

II. INTRODUCTION ............................................................................................................................. 3 

III. RELEVANCE ................................................................................................................................... 6 

A. Charge Questions .................................................................................................................... 6 

B. Specifics for Program Elements.............................................................................................. 6 

IV. SCIENTIFIC AND COMPETITIVE QUALITY................................................................................. 11 

A. Charge Questions .................................................................................................................... 11 

B. Specifics for Program Elements.............................................................................................. 11 

V. PERFORMANCE.............................................................................................................................. 16 

A. Charge Questions .................................................................................................................... 16 

 B. Specifics for Program Elements.............................................................................................. 16 

VI. LEADERSHIP................................................................................................................................... 19 

A. Charge Questions .................................................................................................................... 19 

 B. Specifics for Program Elements.............................................................................................. 19 

VII. COLLABORATIONS ........................................................................................................................ 22 

A. Charge Questions .................................................................................................................... 22 

 B. Specifics for Program Elements.............................................................................................. 22 

VIII. RESOURCES.................................................................................................................................... 25 

 A. Charge Question......................................................................................................................... 25 

IX. INTEGRATION AND FUTURE DIRECTIONS ................................................................................. 26 

X. FINDINGS AND RECOMMENDATIONS.......................................................................................... 29 

APPENDIX A—BOSC ECOLOGICAL RESEARCH SUBCOMMITTEE MEMBERS ............................. 32

APPENDIX B—BOSC ECOLOGICAL RESEARCH SUBCOMMITTEE MEETING DATES.................. 33 

APPENDIX C—LIST OF ACRONYMS..................................................................................................... 35 
Ecological Research Program Review Report 1 

I. EXECUTIVE SUMMARY

Structure of the Program Review 

The Executive Committee of the Board of Scientific Councilors (BOSC) of the Office of 

Research and Development (ORD) within the U.S. Environmental Protection Agency (EPA) has 

agreed to undertake a series of reviews of major ORD research programs. This report addresses 

the review of the Ecological Research Program (ERP). The evaluation is based on the review of 

a large volume of written materials describing the ERP Multi-Year Plan (MYP), individual 

research projects, and budgetary information. 

The Ecological Research Subcommittee, chaired by Dr. Michael Clegg from the University of 

California–Irvine, conducted this review. Other members included Russel Frydenborg, Florida 

Department of Environmental Protection; Dr. John P. Giesy, Michigan State University; Dr. 

Richard Lowrance, U.S. Department of Agriculture (USDA) Agricultural Research Service; Dr. 

Sue A. Thompson, Pennsylvania Biodiversity Partnership; Dr. R. Eugene Turner, Louisiana State 

University; and Dr. Jianguo Wu, Arizona State University. A list of the Subcommittee members 

is included in Appendix A. 

The review was conducted from February through April 2005. The Subcommittee members met 

twice via conference call prior to a site visit for orientation to the Federal Advisory Committee 

requirements and to discuss the review procedures. A 3-day site visit was conducted at the EPA 

facility in Research Triangle Park (RTP), North Carolina, from March 7-9, 2005. The site visit 

featured direct presentations by ERP program leaders and scientists and it provided time for the 

BOSC Ecological Research Subcommittee to question EPA staff on program details. 

Subsequent to the RTP meeting, a conference call was held on April 1, 2005, to finalize the draft 

report. 

Objectives of the Review 

The objectives of the review were to evaluate: (1) program relevance and quality; (2) program 

design and implementation; (3) progress achieved towards meeting long-term goals (LTGs); 

(4) stakeholder involvement and the degree to which research is consistent with needs articulated 

at regional and local levels; and (5) the degree to which research “outputs” are being used by 

stakeholders. The ERP’s research is organized into three LTGs and this report provides an 

analysis of the ERP in the context of each of the LTGs. The report then considers the ERP in 

holistic terms and provides specific conclusions and recommendations to improve the program’s 

effectiveness and impact. A major purpose of this review, therefore, is to assist the ERP to adapt 

to new requirements and expectations as it works to meet the needs of the nation. 
2 Ecological Research Program Review Report

Conclusions and Recommendations 

As noted above, the review was focused on assessing progress towards meeting the three LTGs 

of the ERP: 

 LTG 1: By 2010, national policy makers will have the tools and technologies to develop 

scientifically defensible assessments of the state of our nation’s ecosystems and the 

effectiveness of existing national programs and policies. 

 LTG 2: By 2010, states and tribes apply improved tools and methods to protect and restore 

their valued ecological resources. 

 LTG 3: By 2012, decision-makers understand the importance of ecosystem services and 

make informed, proactive management decisions that consider a range of alternative 

outcomes. 

In general, the Subcommittee found the ERP to be a high-quality scientific program that is 

providing essential technical information to the regulatory offices within EPA as well as to state, 

local, and tribal governments to assist these entities in addressing novel problems of 

environmental management. In particular, the Subcommittee concluded that results of the 

program’s research are relevant and of direct use to states and tribes in protecting and restoring 

ecological resources. The program is developing major tools for measuring environmental 

health, and these tools are being adopted in the field. The quality of the ERP science is high and 

compares favorably with ecological research in academic, non-governmental, and private 

sectors. Program leadership appears to be very good. 

There is a need for improved integration among the three LTGs. To gain additional investment 

leverage, more emphasis should be placed on collaboration between EPA scientists and scientists 

outside the Agency. Moreover, processes designed to facilitate communication between the ERP 

and its stakeholders need to be improved. There also is a need for a formal performance 

evaluation to assess the integration of the various research projects in the context of the LTGs of 

the ERP. The articulation of research goals and priorities is not always clear to those inside or 

outside the organization. In this context, it is crucial that a new MYP be developed that aligns 

with current resource constraints and that better integrates the three LTGs. 

The ERP is undergoing substantial budget reductions and short-term approaches to budget 

reductions are being successfully implemented. The long-term impacts of these short-term 

solutions, however, threaten to disproportionately reduce the ERP’s effectiveness. In particular, 

a budget reduction strategy focused almost entirely on extramural programs will have 

disproportionate costs because the leveraging of outside resources may be compromised and 

productive connections with academic, non-governmental, and private sector researchers may be 

undermined. Plans need to be developed for a long-term equilibrium that balances the research 

portfolio against expected resource constraints. This will entail a difficult priority-setting 

process, but if done effectively, it will maximize the effectiveness of the ERP over the long term. 
Ecological Research Program Review Report 3 

II. INTRODUCTION

EPA relies on expert external review to assess the scientific quality and performance of its 

research programs. This report communicates the findings of a review of EPA’s Ecological 

Research Program. The ERP is one of nine research programs within ORD and it represents one 

of ORD’s largest budgetary components. ORD performs and funds research that provides 

technical standards and scientific information to support EPA’s broad mission of protecting and 

restoring the environment and specifically to support EPA’s various regulatory functions. 

Many customers and stakeholders utilize ORD’s research products. Foremost among these are 

the regulatory offices of EPA that are mandated to provide technical standards for regulatory 

decisions. These technical standards must be grounded in well-documented scientific knowledge. 

Other important customers include state regulatory entities, tribes, and local governments. The 

stakeholder population also includes non-governmental organizations (NGOs) that have a stake 

in environmental conservation; the private sector that must adapt to and implement regulatory 

decisions which in turn require clear and understandable technical standards; and finally, the 

general public that has a strong stake in EPA’s mission of protecting, sustaining, and restoring 

the health of the nation’s ecosystems. To achieve this broad goal, EPA must devise scientifically 

valid measurements of the health of the environment and this requires an intensive program of 

ecological research. 

The BOSC’s Ecological Research Subcommittee conducted this review (see Appendix A for a 

list of the Subcommittee members). Prior to the review, the Subcommittee met twice via 

conference call (February 17 and March 3, 2005) for orientation to the Federal Advisory 

Committee requirements and to discuss the review procedures. The review was conducted at the 

EPA facility in RTP, North Carolina, from March 7-9, 2005. Subsequent to the RTP meeting, a 

teleconference was held on April 1, 2005, to finalize the draft report. The review meeting at 

RTP and the three teleconferences were conducted as open meetings under the guidelines of the 

Federal Advisory Committee Act (FACA). A list of the Subcommittee conference calls and 

meetings is provided in Appendix B. 

The format of this review is relatively new. The BOSC has just begun an intensive review 

process for the various ORD research programs and this is only the third review in that series. 

An impetus for the BOSC review is the Office of Management and Budget’s (OMB) Program 

Assessment Rating Tool (PART) evaluation system. The ERP received a “Results Not 

Demonstrated” rating in its recent PART evaluation. Based on this rating, an external review 

appeared in order to more fully assess the strengths and weaknesses of the ERP and to identify 

ways to improve the program. A major purpose of this review, therefore, is to assist the ERP to 

adapt to new requirements and expectations as it works to meet the needs of the nation. 

The charge to the Subcommittee was to review the ERP’s work and products using the following 

eight general questions as guidelines: 

1. Is the focus of the program relevant to and consistent with EPA’s strategic goals? Does it 

develop a scientific foundation that will lead to attainment of the program’s stated 
4 Ecological Research Program Review Report

environmental outcomes? Are potential public benefits of the program clearly articulated? 

What would be the minimum research program that would be both effective and successful? 

2. Does the program have a logical and comprehensive design with clear goals, priorities, and 

schedules to track progress toward those goals? 

3. Do the design and implementation of the program’s structure facilitate attainment of 

outcomes through integration of research across the program? 

4. Has the program made significant progress toward each of its long-term goals? 

a. Do the research results address the key research questions? 

b. Is the rationale to address the questions clearly articulated? 

c. Does the program follow a long-term plan to address the logical sequence of the 

questions? 

d. Is progress to address the questions being made in a timely fashion? 

e. Does the research reflect the current state-of-the-science, and meet the current and future 

needs of the Agency, science, and program customers? 

5. What is the scientific quality of the program’s research products? Does the program ensure 

high quality research through competitive, merit-based funding? If funds are not 

competitively awarded, what process does the program use to allocate funds? Does this 

process ensure that quality is maintained? 

6. Is the stakeholder involvement in the development of the program clearly and adequately 

articulated? 

a. Does the program effectively engage stakeholders in its planning? 

b. Does the program have a process for using the results of the research, along with 

stakeholders’ feedback, to identify key research gaps and to update the program’s 

research agenda? 

c. Are potential public benefits clearly articulated? 

 d. What are the impediments, if any, to collaboration with other organizations? 

7. Are the program’s research results being used by clients and stakeholders? Are these 

research results consistent with the needs articulated by the Agency’s program and regional 

offices? 

8. Will the program’s completed and planned outputs lead to the intended outcomes that are 

protective of our ecological resources? 

This report discusses the above charge questions as they relate to the ERP’s three revised LTGs: 

 LTG 1: By 2010, national policy makers will have the tools and technologies to develop 

scientifically defensible assessments of the state of our nation’s ecosystems and the 

effectiveness of existing national programs and policies. 

 LTG 2: By 2010, states and tribes apply improved tools and methods to protect and restore 

their valued ecological resources. 
Ecological Research Program Review Report 5 

 LTG 3: By 2012, decision-makers understand the importance of ecosystem services and 

make informed, proactive management decisions that consider a range of alternative 

outcomes. 

Each of the above LTGs is reviewed in the context of the charge questions, followed by a 

discussion of the integration of all LTGs to assess the program holistically, and finally a list of 

conclusions and recommendations. 
III. RELEVANCE

A. Charge Questions 

1. Is the focus of the ERP relevant to and consistent with EPA’s strategic goals? Does it 

develop a scientific foundation that will lead to attainment of the ERP’s stated environmental 

outcomes? Are potential public benefits of the program clearly articulated? What would be 

the minimum research program that would be both effective and successful? 

4e. Does the research reflect the current state-of-the-science, and meet the current and future 

needs of EPA, science, and the ERP’s customers? 

B. Specifics for Program Elements 

LONG-TERM GOAL 1 

By 2010, national policy makers will have the tools and technologies to 

develop scientifically defensible assessments of the state of our nation’s 

ecosystems and the effectiveness of existing national 

programs and policies. 

Question 1: The focus of the ERP is entirely relevant and consistent with EPA’s strategic goals. 

The Agency’s strategic goals are: (1) clean air, (2) clean and safe water, (3) land preservation 

and restoration, (4) healthy communities and ecosystems, and (5) compliance and environmental 

stewardship. To achieve these strategic goals, it is critically important for national policy makers 

to have accurate, reliable, and timely information on the status and trends of the nation’s 

environmental conditions, so that proper policies can be made and actions taken accordingly. To 

this end, scientifically sound and practically effective tools and technologies must be made 

available to national policy makers, and this will be accomplished through LTG 1. These tools 

and technologies include a scientifically sound design framework, reliable ecological indicators, 

and effective monitoring protocols and actions. Products from LTG 1 are most relevant to 

environmental problems and policy issues at the national level. 

Through LTG 1, the ERP has developed a probability-based design and sampling framework that 

is solid in theory and efficient in practice. This framework has been successfully used for 

measuring, assessing, and monitoring the status and trends of ecosystem conditions at the 

national, regional, and local scales. A comprehensive scientific foundation that emphasizes the 

integration between pattern and process across systems, scales, and levels of biological 

organization continues to be built and refined. These salient characteristics of the scientific 

foundation are reflected in a number of projects, including the Environmental Monitoring and 

Assessment Program (EMAP), the Mid-Atlantic Integrated Assessment (MAIA), and landscape 

6 Ecological Research Program Review Report
studies. This developing scientific foundation is appropriate for achieving the ERP’s expected 

environmental outcomes. 

The potential benefits to the public of the ERP’s research are evident and clearly articulated. 

The products of LTG 1 are crucial for national policy makers and also for public awareness of 

the nation’s environmental conditions. 

It is difficult to define the minimum research program for protecting the environment. All of the 

ERP’s present research projects seem necessary to understand and deal with the nation’s 

different environmental issues across different scales. Removing any part of the program would 

certainly not enhance the functionality and outputs/outcomes of the ERP. 

Question 4e: The research of LTG 1 is evidently of high scientific quality, and reflective of the 

state-of-the-art in broad-scale environmental research, particularly in developing statistical 

design and sampling strategies for measuring and monitoring ecosystem conditions. Because 

landscapes are large enough to incorporate multiple interacting ecosystems and major regimes of 

human influences on them, while still small enough for insightful mechanistic explorations, 

landscape-level indicators provide critically important information for legislators and policymakers at the national level and decision-makers and stakeholders at the local level. The ERP 

has developed a suite of such indicators in recent years, and this research represents cutting-edge 

science. Empirical testing of these indices as well as additional core research in this extremely 

important area, however, are clearly needed. 

LONG-TERM GOAL 2 

By 2010, states and tribes apply improved tools and methods to protect 

and restore their valued ecological resources. 

Core research and problem-driven research undertaken by ERP staff, cooperators, and grantees 

form the basis for many of the tools and methods employed by states and local entities, both 

public and private, to protect and restore ecological resources. This function of the ERP is not 

only consistent with, but is essential to, EPA’s core mission to protect human health and 

safeguard the natural environment—air, water, land—upon which life depends. Safeguarding 

the nation’s environment is intrinsically linked to activities at the state, tribal, and local levels. 

The scientific research produced and supported by the ERP in support of LTG 2 ranges in quality 

from good to excellent. Results from these projects have answered key questions on processes in 

natural systems and have resulted in a greater understanding of the affects of anthropogenic 

activities on the physical, chemical, and biological integrity of the nation’s waters and air. The 

ERP research represents major contributions to ecosystem science. Examples of this research 

include stressor identification techniques for specific biological impairment determination, 

advanced watershed stormwater models for pollutant reduction, and novel microbial and 

genetically based ecological assessment procedures. 

Results of the ERP research are relevant and of direct use to states and tribes in protecting and 

restoring ecological resources. Model case studies by collaborators (for example, Maryland’s 

Green Infrastructure assessment) have had “on-the-ground” impacts in land conservation. Most 

of the ERP’s research, however, has not yet translated into direct conservation and restoration 

Ecological Research Program Review Report 7 
activities, other than the handful of collaborative examples. These outcomes are expected to 

occur in more states and tribes over time. 

Although there are numerous potential public benefits of the ERP’s research, these benefits are 

not always clearly articulated, even at the program level, and are rarely explicitly included as a 

goal in individual research projects. This lack of an outcomes-based research paradigm hampers 

the program from producing research that fully meets both the future and current needs of 

clients, especially those outside of EPA (e.g., states and tribes). Although examples were given 

in the onsite presentations of states routinely and effectively applying ERP-derived tools, no 

formalized tracking system that quantifies the extent of this application was evident and no 

specific uses by tribal entities were highlighted. 

LONG-TERM GOAL 3 

By 2012, decision-makers understand the importance of ecosystem 

services and make informed, proactive management decisions that 

consider a range of alternative outcomes. 

LTG 3 is the ERP’s outreach component with the goal of developing sophisticated tools to 

enable decision-makers to make decisions that enhance ecosystem services. Although this goal 

has elements of more basic and applied research, scientists working on LTG 3 seek 

people/institutions with specific problems that can be used as case studies for critical resources 

management issues. To date, these site-specific projects have been used as proof-of-concept 

exercises, coordinated through either regional or program offices, such as the Office of Water 

(OW) and the Office of Air Quality Planning and Standards (OAQPS). 

The research, tools, and analytical technologies developed under the ERP represent the most 

comprehensive federal government research program examining the provision of ecosystem 

services and the communication of these to decision-makers. LTG 3 is a highly relevant activity 

that is central to EPA’s mandate of improving environmental quality and protecting and restoring 

the health of the nation’s ecosystems. ORD and particularly the ERP are uniquely suited and 

positioned to address these issues. 

In addition to being relevant to the federal role, the major projects under LTG 3 address critical 

complex questions facing local and state governments. The major projects described under LTG 

3 and the questions framed for future research will provide both new scientific knowledge about 

the provision and value of ecosystem services and new decision tools to allow decision-makers 

and the public to address alternative futures with varying levels of ecosystem services. 

The projects presented during the site visit demonstrate an excellent balance between 

development of new information and conceptualizing new analytical methods for the future, 

while also providing clients with tools for addressing current issues. Quantifying and 

demonstrating future scenarios with varying levels of ecosystem services, will provide the 

scientific foundation needed to make informed decisions. Many of these decisions are made at a 

local level with tools that currently are inadequate to consider cumulative effects of the loss or 

gain of ecosystem services. The tools developed under LTG 3 will provide the scientific 

foundation for better decision-making and thus help in attainment of environmental benefits. 

8 Ecological Research Program Review Report
All projects presented under LTG 3 have the specific objective of providing information on 

ecosystem services and ecosystem functions to either specific or hypothetical decision-making 

groups. For some major projects and tools, potential public benefits are clearly articulated. For 

example, the Willamette Basin alternative futures research has lead to an atlas of the basin and to 

a clear focus among regional governments on using the tools developed under LTG 3 to increase 

the level of future ecosystem services within the large river (see Box 1). 

 Box 1: The Willamette River Basin Case Study 

The ERP worked with the Oregon Governor’s office, EPA Region 10, Oregon State 

University, University of Oregon, and numerous stakeholders to develop an 

alternative futures analysis of the Willamette River Basin, a 30,000 km2

 basin 

undergoing urbanization and intensification of agriculture and forestry in a waterlimited environment. The scientists used a reconstruction of historical land use and 

new analytical tools to develop three alternative future scenarios: plan trend (current 

state and local policies and practices extrapolated); development (current state and 

local land use regulations relaxed to spur development); and conservation (land and 

water allocation to conservation increased). 

The evaluation criteria were indicators of either the need for or provision of 

ecosystem services: population density in urban growth areas; urbanized area; rural 

developed area; prime farmland; water availability for urban, industrial, and 

agricultural uses; water consumed; changes in streamflow; area in conifer forest 

> 80 years old; habitat for amphibians, birds, mammals, and reptiles; percent of 

riparian areas in natural vegetation; population densities of 17 birds and mammals; 

cut-throat trout habitat; and fish species abundance and richness. 

Results showed that population densities in urban growth areas doubled under both 

the current trend and the conservation scenarios, but increased much less under the 

development scenario. Urbanized areas and rural growth areas increased under all 

scenarios, but increased most under the development scenario. Prime farmland 

decreased in all scenarios, but decreased most in the development scenario. In 

general, the conservation scenario conserved ecosystem services as measured by 

the indicators. Of particular interest in this semi-arid water-limited region, water 

consumption increased in all scenarios, but increased least with the conservation 

scenario. 

Information from the Willamette Project has been used by the Oregon Governor’s 

office, state agencies, and stakeholder groups to frame policy discussion about 

future growth in the Willamette Basin and the state. The successful project was the 

impetus for state establishment of the Institute for Natural Resources to provide 

science-based information on ecosystem services and growth to Oregon decisionmakers. 

In some cases, projects at an earlier stage of development have not yet manifested public 

benefits, but the potential benefits are articulated. The goal and sub-questions form a body of 

work that should proceed as a whole. Because the ongoing work focuses on the delivery of tools 

to understand societal benefits of ecosystem services, new research on the provision of these 

ecosystem services is essential. Conversely, the important outcomes of LTG 3 will not be 

Ecological Research Program Review Report 9 
10 Ecological Research Program Review Report

realized unless the decision support systems and analytical techniques to project and 

communicate alternative future development scenarios are available and applied. Thus, the 

entire research and technology development program is necessary. 
IV. SCIENTIFIC AND COMPETITIVE QUALITY

A. Charge Questions 

2. Does the ERP have a logical and comprehensive design with clear goals, priorities, and 

schedules to track progress toward those goals? 

3. Do the design and implementation of the ERP’s structure facilitate attainment of outcomes 

through integration of research across the ERP? 

4a. Do the research results address the key research questions? 

4b. Is the rationale to address the questions clearly articulated? 

4c. Does the ERP follow a long-term plan to address the logical sequence of the questions? 

4d. Is progress to address the questions being made in a timely fashion? 

5. What is the scientific quality of the ERP’s research products? Does the ERP ensure high 

quality research through competitive, merit-based funding? If funds are not competitively 

awarded, what process does the ERP use to allocate funds? Does this process ensure that 

quality is maintained? 

8. Will the ERP’s completed and planned outputs lead to the intended outcomes that are 

protective of our ecological resources? 

B. Specifics for Program Elements 

LONG-TERM GOAL 1 

By 2010, national policy makers will have the tools and technologies to 

develop scientifically defensible assessments of the state of our nation’s 

ecosystems and the effectiveness of existing national 

programs and policies. 

Question 2: LTG 1 activities are organized in terms of three related, but distinctive, 

components: (1) statistically valid and scientifically defensible design frameworks, (2) a suite of 

reliable ecological indicators for different ecosystems, and (3) environmental monitoring. Goals 

for each component are clearly defined. The priorities of LTG 1 activities are clearly placed at 

the national level, which is appropriate because LTG 2 and LTG 3 deal mainly with the state and 

local levels, respectively. In other words, each goal has its own focal level of organization and 

corresponding spatial scale, and this scheme is, in principle, both scientifically sound and 

practically feasible. 

LTG 1 is particularly important because it provides an overall conceptual framework and 

technical guidelines for the other LTGs. To assure both high scientific quality and relevance to 

EPA’s mission, LTG 1 projects should encourage close collaborations between EPA’s own 

researchers and external high-caliber scientists who have broad visions and pertinent knowledge 

and skills. Such collaborations can be accomplished through open grant competition with an 

explicit requirement for collaboration with EPA researchers. In addition, although the design 

Ecological Research Program Review Report 11 
12 Ecological Research Program Review Report

principles of LTG 1 are sound, improved planning and better-coordinated cross-level integration 

are needed. Also, more systematic plans and mechanisms for increased tracking of the outputs 

and outcomes of research projects must be developed as part of the research cycle. 

Question 3: The new structure of the ERP with three LTGs seems scientifically sound and 

practically feasible. Products of LTG 1 are most relevant to environmental problems and policy 

issues at the national level, whereas LTG 3 deals primarily with local-scale environmental issues 

that may frequently involve local ecosystems (e.g., lakes, streams, or forest habitats) or local 

landscapes (watersheds or mosaics of different ecosystems) (see Figure 1). LTG 2 is mainly

targeted at environmental problems and policy issues at the state level, often spanning over a 

region. This suggests that in order to achieve desirable outcomes, each LTG must emphasize 

interactions and collaborations with relevant agencies and stakeholder groups at its focal level. 

For example, the expected outcomes of LTG 3 are unlikely to be accomplished without directly 

involving local decision-makers and stakeholders. Having a focal level or scale for each LTG 

facilitates use-inspired research, which is consistent with the mission of EPA, but does not, and 

should not, imply that the three LTGs can be achieved separately without interactions or 

coordination. On the contrary, a major advantage of such a hierarchical framework is both to 

highlight the primary targeted user group and to provide pathways to effectively integrate across 

scales and programs. Products of LTG 1 should influence and have influenced activities of LTG 

2, which in turn have bearings on LTG 3. The same is true the other way around. To date, the 

ERP has demonstrated such two-way interactions to some extent with some excellent examples. 

The ERP has a logical and comprehensive design, which is adequate for ORD’s planning process 

and for demonstrating progress toward its overall goals. 

Questions 4a and 4b: The results of LTG 1 are quite relevant to its key research questions. The 

probability-based design framework has served well for multiple-scale sampling and monitoring 

activities at the national, state, and local levels. A suite of indicators for landscape integrity and 

ecosystem health has been developed. These methods and tools have produced valuable 

information on the nation’s ecosystems that is critically important to policy makers and decisionmakers. 

Question 4c: The three LTGs are well designed and follow a multiple-scale, hierarchical 

framework that facilitates addressing environmental issues at the national, state, and local levels. 

LTGs are well articulated and are not only relevant, but also crucial, to the overall mission of 

EPA. The realization of these LTGs requires continuation of high-quality research conducted by 

EPA researchers and outside scientists collaboratively as well as secure long-term funding. 

Question 4d: Research under LTG 1 has made tremendous progress in terms of both research 

output and environmental outcomes. Timely progress also has been made to address important 

research questions. For example, a statistically sound, scientifically defendable, and practically 

feasible design and sampling framework was developed early through EMAP, and has been 

widely used by and provided guidance to a variety of projects at the national and regional scales 

ever since. Based on this framework, extensive efforts have been made to develop methods and 

tools to measure and monitor conditions of different ecosystems and landscapes across the nation 

(e.g., EMAP-WEST, Coastal Ecosystem Assessment, West Rivers). 
Figure 1. Subcommittee Interpretation of the Organizational Levels and 

Spatial Scales at Which Each LTG Operates 

LONG-TERM GOAL 2 

By 2010, states and tribes apply improved tools and methods to protect 

and restore their valued ecological resources. 

Questions 5 and 8: Although the implementation of the ERP’s structure has demonstrated the 

capability to achieve beneficial environmental outcomes, the formal integration between 

individual research components should be better. Cooperation exists among many projects, but 

the need for integration between some projects, although implicit, should be better articulated by 

ORD management. This would improve synergy between investigators and enhance the overall 

body of research, which is generally of excellent quality. 

Ecological Research Program Review Report 13 
Based on the quality of research, substantial advances toward achieving LTG 2 are apparent. 

Examples of these efforts include the Modeling and Stressor Identification procedures, which 

have played key roles in watershed restoration through the Total Maximum Daily Load (TMDL) 

program and the Multimedia Pollutant studies, which have led to mercury reduction in the 

Everglades and elsewhere. Several other projects, such as the recent development of microbial 

and genetic tools, should reasonably expect to achieve outcomes when more mature. A key to 

the success of all the LTG 2 research efforts will be educating state and local decision-makers on 

the benefits of the tools, as well as training appropriate end users. 

The research questions are clearly formulated to address factors most responsible for degrading 

our environment. The rationale behind the research is scientifically sound and appropriate, 

including physical, chemical, biological, landscape, and modeling components. The research is 

structured in a logical sequence to allow results from projects to build on others, although the 

formal integration of individual projects should be better articulated and incorporated into the 

strategic planning process. 

The overall body of research generated by the ERP, in terms of quality and ability to produce 

beneficial environmental outcomes, is superior. Without the ERP’s leadership in this arena, 

many important advances in environmental assessment and the development of strategies for 

effective ecological management would not exist. Stringent quality assurance procedures are 

followed throughout the ERP’s research process, from sampling to data analysis. Peer review 

and feedback, including input from academic and government scientists, is routinely followed, 

resulting in a solid scientific program. 

ORD’s Science To Achieve Results (STAR) grants program is an excellent example of the 

successful implementation of a competitive funding process, which when coupled with the 

ERP’s guidance and understanding of complex environmental issues, can lead to research results 

that are directly applicable to environmental problems at the state and tribal level. 

When funds are not competitively awarded, the ERP appears to use a “best professional 

judgment” approach to allocate funds, coupled with a post-award assessment of the project’s 

success. Based on the successful results associated with these projects, quality appears to have 

been maintained, although a more formal evaluation is warranted. 

LONG-TERM GOAL 3 

By 2012, decision-makers understand the importance of ecosystem 

services and make informed, proactive management decisions that 

consider a range of alternative outcomes. 

Although the initiatives included as elements of LTG 3 are very ambitious, the goals are clearly 

stated and achievable. Because LTG 3 has not been the subject of an MYP, the specifics of 

schedules are difficult to evaluate. The three component questions around which the future 

activities will be developed are clear and should lead to measurable outcomes. Some 

connections between LTG 3 components and other LTGs are not clear. Although the potential 

for these connections is obvious and necessary, integration of all three LTGs should be made 

explicit in the revised MYP. For example, a clear connection between EMAP and the Regional 

14 Ecological Research Program Review Report
Ecological Research Program Review Report 15 

Vulnerability Assessment (ReVA) Program was not evident, although the two are clearly parallel 

projects. The program description would be improved by providing clear linkages with the other 

two LTGs. 

 

The research results that were presented have done an excellent job of addressing the key 

research questions. The ERP research has made numerous advances in both integrative 

ecosystem science that explicitly includes socioeconomic constraints (Willamette Basin, MAIA) 

and in developing general tools for integrated assessments (ReVA). The rationale for the key 

questions is clear, but it is not always clear how individual projects are initiated. As there are 

many ways that projects might be initiated, some of which are beyond the control of the ERP, 

clear documentation of the criteria for project selection is desirable. 

It is understood that a revised MYP is now being developed. Because LTG 3 is proposed as a 

completely new element in the overall research strategy, the Subcommittee can make only an 

overall assessment and give general guidance on program direction. The research is clearly 

long-term and will need to incorporate feedback from clients who help implement the pilot 

programs. As the MYP is revised, an integrated program structure should be developed that 

explicitly states the relationships between program elements and packages them in a way that 

allows the tracking of results and progress toward longer-term goals and the reporting of shortterm outcomes. LTG 3 was not part of the 2003 MYP, but includes elements of the earlier LTG 

3 and LTG 4 as well as new elements that focus on the development of forecasting models and 

analytical tools that will allow the assessment of ecosystem services. The Subcommittee 

understands the rationale for this proposed LTG and endorses the goals and approach. 

The research currently being conducted and that which is proposed to be conducted under LTG 3 

represents state-of-the-science in assessing complex systems and developing tools to understand 

and enhance ecosystem services. The scientific quality of the program is very good to excellent. 

We received varied information on how peer review is used within the organization. Peer review 

and quality assurance for the STAR Program were mostly good, although some inconsistencies 

occurred. If there is a program for internal peer review of projects, it was not evident. It would 

strengthen the internal research program to have external peer review of projects. 
V. PERFORMANCE

A. Charge Questions 

2. Does the ERP have a logical and comprehensive design with clear goals, priorities, and 

schedules to track progress toward those goals? 

3. Do the design and implementation of the ERP’s structure facilitate attainment of outcomes 

through integration of research across the ERP? 

4d. Is progress to address the questions being made in a timely fashion? 

6b. Does the ERP have a process for using the results of the research, along with stakeholders’ 

feedback, to identify key research gaps and to update the ERP’s research agenda? 

7. Are the results of ERP research being used by clients and stakeholders? Are these research 

results consistent with the needs articulated by EPA’s program and regional offices? 

8. Will the ERP’s completed and planned outputs lead to the intended outcomes that are 

protective of our ecological resources? 

B. Specifics for Program Elements 

LONG-TERM GOAL 1 

By 2010, national policy makers will have the tools and technologies to 

develop scientifically defensible assessments of the state of our nation’s 

ecosystems and the effectiveness of existing national programs and 

policies. 

Question 2: LTG 1 activities are organized in terms of three related, but distinctive, 

components: (1) statistically valid and scientifically defensible design frameworks, (2) a suite of 

reliable ecological indicators for different ecosystems, and (3) environmental monitoring. Goals 

for each component are clearly defined. The priorities of LTG 1 activities are clearly placed at 

the national level, which is appropriate because LTG 2 and LTG 3 are more focused on the state 

and local levels, respectively. 

Question 3: LTG 1 program maturation has been accompanied by standardization of the 

conceptual approaches and the experienced solicitation and involvement of stakeholders. These 

positive developments have drawn strong support from a core group of statisticians and 

geographic information system (GIS) experts (among others) within the ERP that has an 

institutional memory, experience, and professional skills that are nationally significant. Their 

activities have been a positive and significant influence on LTG 2 and LTG 3 activities, as well. 

Question 4d: Progress to address the research questions is being made in a timely fashion, but 

articulation of those goals and the planning involved in the processes leading to these goals need 

to more clearly permeate the science culture. The idea is not to homogenize an individual’s 

scientific talents and perspectives so communication skills are emphasized over science skills, 

16 Ecological Research Program Review Report
but that the programmatic elements are masterfully and more completely articulated on behalf of 

the whole. 

Question 6b: The Subcommittee believes that more effective processes and formal mechanisms 

are needed that allow better communications between the ERP and its clients. The experience of 

some members of the Subcommittee is that dissemination of important research results is slow to 

reach the end users. 

Question 7: The stakeholders and clients are using the results in a variety of ways, some of 

which are obvious and others that are not so obvious. The National Coastal Assessments have 

led to an increasing number of states either issuing their own reports using the protocols and 

sampling strategies developed by the ERP, or joining together to issue regional summaries. 

These are in addition to the EPA documents produced by the ERP staff and their collaborators. 

This tangible outcome is monitored. For example, in 2000, no states were directly involved in 

these reports (as state or regional reports), but by 2005, a minimum of 12 states will participate. 

At this rate, the goal of reaching all coastal states should be nearly completed in a few years. 

Question 8: In the opinion of the Subcommittee, the ERP has resulted in the desired outcomes, 

and its present activities and future (planned) outputs are consistent with the conclusion that 

protection of the nation’s ecological resources is enhanced. 

LONG-TERM GOAL 2 

By 2010, states and tribes apply improved tools and methods to protect 

and restore their valued ecological resources. 

Performance measures of the ERP’s planning process and measures for demonstrating progress 

toward overall program goals do not appear to be strictly established, although they can be 

extracted from the body of research. 

There is a need to develop a formal performance evaluation that assesses the degree of 

integration between various research projects to help improve cooperation between individual 

research investigators. 

Clear and substantial progress toward LTG 2 was demonstrated in much of the mature research, 

although this could not readily be evaluated for research at earlier stages of development. Each 

research component should issue a periodic report that more clearly details the incremental 

progress made toward expected outcomes. This would allow an adaptive management strategy 

to keep projects “on track” and internally integrated. 

Stakeholder input to identify research gaps does occur, but the process of briefing decisionmakers on research results as well as on applicability of the research currently is informal or 

often lacking. The effectiveness of this aspect of the program could be improved by establishing 

timely and regular communications with a broad array of stakeholders using an established 

procedure in conjunction with appropriate EPA program offices. 

The ultimate use of the products is dependent upon factors beyond the control of the ERP, but 

the above outreach process would improve the utilization of pertinent results. States, tribes, and 

Ecological Research Program Review Report 17 
local governments should be urged to seek the ERP tools appropriate for carrying out their 

specific environmental protection missions. For example, nonpoint source pollution can be 

mitigated through judicious land use decisions, which almost exclusively are made at the level of 

local government. If utilized by local government resource managers, the ERP products, such as 

landscape and predictive models, would more directly achieve desired outcomes. 

The results of the ERP are thoroughly consistent with the expressed needs of EPA program and 

regional offices. EPA program offices present the ERP with a prioritized list of research issues 

developed through interactions with stakeholders. There is a reasonable expectation that the 

environmental protection tools generated by the ERP will translate into beneficial outcomes 

when the research matures and the tools are implemented at appropriate levels of government. 

LONG-TERM GOAL 3 

By 2012, decision-makers understand the importance of ecosystem 

services and make informed, proactive management decisions that 

consider a range of alternative outcomes. 

It is not clear how progress is tracked, but it is assumed that this is the primary responsibility of 

program managers and lead scientists on the projects. Very good progress in this area has 

occurred during the past 5 years and it is expected that this progress will continue in the future 

with adequate funding and scientific resources. 

The component of the ERP aggregated under LTG 3 is well positioned to make significant 

contributions in the near- and long-term. While it is understood that science is a continuing 

process, it is evident that research products are being adopted and used. The review process 

should be structured in such a manner that useful products and technology can be transferred to 

clients in a timely and efficient manner while the tools are further refined and adapted to 

additional situations and applications. To identify key research gaps and to update the projects, 

the Subcommittee suggests reviews of individual projects by external scientists and stakeholders. 

Currently, LTG 3 does not have a plan for internal periodic assessment, but it is assumed that 

this will be part of the MYP. 

The results, products, and tools generated by LTG 3 are being used by clients. At this point, the 

main clients are involved with the pilot projects. Other clients are being developed through 

online use of the ReVA and Restoration Plus (RePlus) tool kits. Research results, such as the use 

of ReVA with the Sustainable Environment for Quality of Life (SEQL) program and the use of 

research results to inform decision-making by state agencies, such as the Wisconsin Department 

of Natural Resources, demonstrate that the ERP is meeting needs of clients and stakeholders. 

The research program will lead to the intended outcomes if the research products and tools are 

applied by the regions, states, tribes, and local governments. There is good evidence that this is 

occurring and with continued involvement of clients in the research through pilot projects, it 

should continue. As with ReVA and some of the new technologies to enhance water 

management for ecosystem services, there should be specific plans to make all tools and 

technologies available to users such as state governments, planning agencies, and private 

consultants. This will require commitment of resources to technology transfer through both inperson and online training. 

18 Ecological Research Program Review Report
VI. LEADERSHIP

A. Charge Questions 

4e. Does the research reflect the current state-of-the-science, and meet the current and future 

needs of EPA, science, and the ERP’s customers? 

6a. Does the ERP effectively engage stakeholders in its planning? 

6b. Does the ERP have a process for using the results of the research, along with stakeholders’ 

feedback, to identify key research gaps and to update the ERP’s research agenda? 

6c. Are potential public benefits clearly articulated? 

6d. What are the impediments, if any, to collaboration with other organizations? 

7. Are the ERP’s research results being used by clients and stakeholders? Are these research 

results consistent with the needs articulated by EPA’s program and regional offices? 

B. Specifics for Program Elements 

LONG-TERM GOAL 1 

By 2010, national policy makers will have the tools and technologies to 

develop scientifically defensible assessments of the state of our nation’s 

ecosystems and the effectiveness of existing national 

programs and policies. 

Question 4e: The research of LTG 1 is evidently of high scientific quality and reflective of the 

state-of-the-art in broad-scale environmental research, particularly in developing statistical 

design and sampling strategies for measuring and monitoring ecosystem conditions. Because 

landscapes are large enough to incorporate multiple interacting ecosystems and major regimes of 

human influences on them, while still small enough for insightful mechanistic explorations, 

landscape-level indicators provide critically important information for legislators and policy 

makers at the national level and decision-makers and stakeholders at the local level. The ERP 

has developed a suite of such indicators in recent years, and this research represents cutting-edge 

science. Empirical testing of these indices, as well as additional core research in this extremely 

important area, however, are clearly needed. 

Question 6a: The LTG 1 program initially tended to develop methodologies and approaches that 

were independent of stakeholder involvement once the basic national needs were identified. 

Stakeholders increasingly are more involved in the process, from implementation of the 

conceptual designs and sampling protocols to field-testing and regional-application, and then to 

the stage where there is a firm engagement of LTG 1 products within a formal framework. The 

LTG 1 sub-program is now at a point where stakeholders from many regions are interested in 

adopting the methodologies, in some cases before EPA has resources to assist them. 

Ecological Research Program Review Report 19 
Question 6b: More effective processes and formal mechanisms appear to be needed that allow 

better communication between the ERP and its clients. These processes and mechanisms may be 

sufficient, but were not evident to the Subcommittee because of time constraints. 

Question 6c: The public benefits are well appreciated within the ERP science culture, and 

among many state agencies. The Subcommittee members heard several verbal assessments and 

read others that led to the conclusion that the public benefits were appreciated by the states. The 

Subcommittee was not able to assess whether the public benefits were articulated well across the 

entire spectrum of stakeholders, probably because of the time constraints involved in this 

assessment. The Subcommittee recognizes that the public benefits are numerous and that they 

are articulated within the culture of EPA and state agencies, but perhaps not clearly articulated to 

all stakeholders, including the general public, in the same way. 

Question 6d: There are no obvious impediments to collaboration with other organizations. The 

three LTGs, each focusing mainly, but not exclusively, on a relatively explicit level of 

organization and corresponding geographic scales, should facilitate the process of identifying 

and working with collaborative organizations outside EPA. 

Question 7: The Subcommittee believes that research results are being used by clients and stakeholders as illustrated in the boxes included in this report. 

LONG-TERM GOAL 2 

By 2010, states and tribes apply improved tools and methods to protect 

and restore their valued ecological resources. 

The ERP scientists and collaborators frequently are leaders in their respective scientific fields 

and ORD has clearly supported outstanding research in many areas of ecosystem science. The 

ERP is a valuable and integral part of EPA and essential to carrying out EPA’s mission. The 

ERP’s leadership within the scientific community is evidenced by the participation of its 

scientists on editorial boards, review panels, conferences, and other academic or information 

exchange gatherings. There is evidence that the ERP scientists take a leadership role in 

disseminating research results in many states, but this is not uniformly true for a variety of 

reasons, some of which are beyond their control. 

The process for stakeholder engagement in research planning is unclear. In many cases, it 

appears to be ad hoc with fortuitous partnerships formed based on requests from entities or 

similar interests. Although these partnerships have produced results that may translate into 

useful products for states and tribes in general, implementation of a strategic planning process 

that involves broad stakeholder input will increase these opportunities. 

20 Ecological Research Program Review Report
LONG-TERM GOAL 3 

By 2012, decision-makers understand the importance of ecosystem 

services and make informed, proactive management decisions that 

consider a range of alternative outcomes. 

ORD scientists and collaborators working on LTG 3 are among the leaders in this research in the 

United States. No other federal research agency has as extensive or advanced program in 

transferring tools to assess the provision of ecosystem services. Although numerous 

collaborators and stakeholders already are engaged, the process of identifying and engaging them 

could be more transparent. It would be very useful if resources were available to fund 

cooperative development of the pilot projects and if these could be focused on areas with a wide 

variety of resource management and environmental quality issues. In particular, when dealing 

with rural areas and insuring the provision of ecosystem services as rural areas develop, 

stakeholders representing agriculture and forestry should be more clearly involved as there is a 

significant amount of terrestrial ecosystem assessment, landscape ecology, and ecosystem 

modeling being conducted to understand how agriculture and forest practices affect ecosystem 

services. Although the Subcommittee assumes that some level of coordination exists among 

these activities, this was not apparent in the information presented to the Subcommittee. 

Increased interactions among LTG 3 research scientists with other elements of ORD focusing on 

socioeconomic research may result in significant opportunities to further leverage resources. In 

addition, there was no evidence of interactions with the international community. Programs 

addressing the issues outlined in LTG 3 are the subject of research programs in other countries. 

The Subcommittee recommends that review of other ongoing programs will strengthen elements 

of LTG 3. 

Ecological Research Program Review Report 21 
VII. COLLABORATIONS

A. Charge Questions 

5. What is the scientific quality of the ERP’s research products? Does the ERP ensure high 

quality research through competitive, merit-based funding? If funds are not competitively 

awarded, what process does the ERP use to allocate funds? Does this process ensure that 

quality is maintained? 

6a. Does the ERP effectively engage stakeholders in its planning? 

6b. Does the ERP have a process for using the results of the research, along with stakeholders’ 

feedback, to identify key research gaps and to update the ERP’s research agenda? 

6c. Are potential public benefits clearly articulated? 

6d. What are the impediments, if any, to collaboration with other organizations? 

7. Are the results of the ERP being used by clients and stakeholders? Are these research results 

consistent with the needs articulated by EPA’s program and regional offices? 

8. Will the ERP’s completed and planned outputs lead to the intended outcomes that are 

protective of our ecological resources? 

B. Specifics for Program Elements 

LONG-TERM GOAL 1 

By 2010, national policy makers will have the tools and technologies to 

develop scientifically defensible assessments of the state of our nation’s 

ecosystems and the effectiveness of existing national 

programs and policies. 

Question 5: The quality of the science is adequate to high, depending on the standards used and 

the subject area. The results from the congressionally mandated atmospheric deposition program 

are exceptionally good, and meet internationally recognized science quality standards among 

independent scientists active in the field, and have grounded EPA’s policies in science. The 

monitoring and assessment program has an excellent statistical design and more than adequate 

sampling protocols, and is well on its way to accomplishing its leadership and implementation 

goals. Because it also is in the first decade of what is a very long-term monitoring program, the 

scientific quality of the monitoring and assessment program is not yet fully evident. The 

program has very effectively engaged some state agencies, but with an anticipated focus on 

Agency-style documents (primarily reports) and less so on peer-reviewed literature that is more 

widely distributed and which receives greater scrutiny (and concurrently, expanded opportunities 

for improvement). The emphasis on fuller disclosure of the science products and their 

implementation will eventually gain greater prominence as monitoring continues beyond the 

alpha phase and generates results extending over several decades. Much of the research is done 

through noncompetitive funding directly to stakeholders, which are primarily states, as a direct 

and intentional result of the integration of stakeholders into the field program. 

22 Ecological Research Program Review Report
Question 6a: LTG 1 research can be further improved by more systematically and proactively 

engaging decision-makers and stakeholders at local, state, and national levels. 

Question 6b: More effective processes and formal mechanisms appear to be needed that allow 

better communication between the ERP and its clients. These processes and mechanisms may be 

sufficient, but were not evident to the Subcommittee because of time constraints. 

Question 6c: The potential benefits to the public and stakeholders are clearly articulated. 

Question 6d: There are no obvious impediments to collaboration with other organizations. The 

three LTGs, each focusing mainly, but not exclusively, on a relatively explicit level of 

organization and corresponding geographic scale, should facilitate the process of identifying and 

working with collaborative organizations outside EPA. 

Question 7: The research results of LTG 1 have been widely used by clients and stakeholders. 

Numerous examples exist in the documents and presentations provided to the Subcommittee. 

More formal mechanisms for facilitating communications with regional offices and keeping 

track of outreach activities, however, appear to be needed, although the Subcommittee is willing 

to accept that these mechanisms exist, but were not revealed to the members during the course of 

the Subcommittee’s deliberations. 

Question 8: LTG 1 addresses some of the most pressing and challenging environmental research 

questions today, and is extremely relevant to EPA’s mission. The research activities under LTG 

1 so far have been quite productive, resulting in important outcomes. Maintaining the strength of 

LTG 1 is imperative for the continued success of the ERP. The accomplishments of the ERP are 

critical building blocks to achieve EPA’s overall goals and objectives of protecting and restoring 

the nation’s ecosystems. The ERP, however, cannot by itself accomplish all of EPA’s stated 

goals and objectives because of the size and complexity of the problems at hand. High-quality 

involvement makes enhancement more likely, and demonstrably so where there has been enough 

time to move from program beginnings to maturation and implementation. 

LONG-TERM GOAL 2 

By 2010, states and tribes apply improved tools and methods to protect 

and restore their valued ecological resources. 

ORD has a superb track record of collaboration with a variety of partners at the level of specific 

research projects (see Box 2 and Box 3). Such collaborations, especially for LTG 2 projects, 

seem typically to be the result of “willing and able” partners expressing an interest in working 

with EPA. Targeted outreach to nontraditional partners should be pursued as part of a strategic 

communication process involving stakeholders. 

Ecological Research Program Review Report 23 
Box 2: Mercury Contamination in the Everglades 

During EPA-sponsored monitoring of a hazardous waste site in Florida, mercury was 

detected in fish tissue at unacceptable levels, even though mercury was not a contaminant of concern at the site. The ERP scientists were instrumental in providing 

intellectual energy and funding to diagnose this mystery, which involved complex multimedia dynamics. These included understanding the sources of mercury in air, atmospheric deposition patterns, and biological magnification of mercury through the 

Everglades food web. This culminated in diagnosing the causes behind the fish consumption advisories and adverse effects on the Florida panther, an endangered species. 

In approximately 10 years, the ERP-assisted research was directly connected to a beneficial environmental outcome—reductions in regional mercury emissions and subsequent improvements in biological health and continued safeguarding of human health.

LONG-TERM GOAL 3 

By 2012, decision-makers understand the importance of ecosystem 

services and make informed, proactive management decisions that 

consider a range of alternative outcomes. 

Collaboration on specific pilot 

projects has been excellent and the 

Subcommittee expects this to 

continue. Collaboration would be 

enhanced by the ability to enter into 

cooperative research projects that 

address specific resource management 

issues. Outcomes would be enhanced 

by identifying and collaborating with 

representatives of expected user 

groups at the outset of pilot projects, 

in addition to the specific project 

collaborators. This could be 

accomplished by convening broadbased user groups to comment on 

specific pilot projects at an early stage 

of the projects. 

Box 3: Biological Assessment and TMDLs 

Florida routinely employs a probabilistic sampling design to conduct statewide bioassessments of streams and lakes, both of which are 

based upon the ERP LTG 1 products for assessing environmental health. When degradation is 

detected in the biological communities, staff 

utilizes the principles of Stressor Identification 

(an ERP LTG 2 tool) to reasonably determine 

the pollutant(s) responsible for the adverse biological effects. Various ERP-based models then 

are used to establish Total Maximum Daily 

Loads (components of LTG 3) for pollutant reduction throughout localized watersheds. Continued monitoring over a 5-year cycle will establish the degree of biological recovery, which is 

the desired environmental outcome. This entire 

process would not be possible without the various assessment procedures and applications 

produced by the ERP. 

24 Ecological Research Program Review Report
Ecological Research Program Review Report 25 

VIII. RESOURCES

A. Charge Question 

Question 1: What would be the minimum research program that would be both effective and 

successful? 

Resources, both intramural and extramural (i.e., STAR), are adequate for measured progress, but 

are clearly inadequate to undertake other than a handful of extensive projects, such as the 

Willamette Basin study and ReVA/SEQL. Currently, there are many issues that are important to 

clients, both intramural and extramural, that cannot be addressed because of resource limitations. 

The ERP has done an excellent job of using cooperation with the programs of various groups 

within and outside of government to extend the resources available. The ERP scientists often 

provide expertise and obtain the necessary information for proof-of-concept or model 

parameterization by their participation in ongoing projects. 

The ERP has done an excellent job of leveraging resources, but a fairly poor job of documenting 

the magnitude of these leveraged funds. The ERP should institute a mechanism to track these 

contributions so that they can be articulated and reported in a quantitative manner. Additional 

project funds are needed to develop more pilot projects. 

The loss of the STAR Program would hurt the integrity of the entire ERP. Something similar to 

STAR must be maintained in accordance with the three LTGs to assure the quality of the ERP’s 

collaboration, efficiency, leveraging of funds, and intellectual capital. 
26 Ecological Research Program Review Report

IX. INTEGRATION AND FUTURE DIRECTIONS

EPA must perform scientific research to provide the baseline knowledge to establish technical 

standards for regulatory decisions and to provide appropriate tools and methodologies to those 

who are charged with measuring and conserving the health of our national environment. This 

section addresses the appropriateness of the ERP’s three LTGs and their integration; the ERP’s 

effectiveness in meeting its responsibilities to its customers; and, to the extent possible, an 

assessment of the future directions of ERP research. In evaluating the program and its LTGs, it 

is important to reiterate that ORD has a research mission with the primary responsibility of 

developing scientific knowledge, tools, and methodologies to serve the mission of the regulatory 

offices of EPA as well as to meet the technical and scientific needs of state governments and 

tribes. In this sense, these EPA regulatory offices are the customers of ORD and its constituent 

research programs. Other important customers include other federal agencies, states, tribal 

governments, and decision-makers. 

The ERP has many substantial accomplishments to its credit and the BOSC Ecological Research 

Subcommittee rates the overall quality of the research as good to excellent. Variation exists 

among the projects as is inevitably the case within any large and complex organization, but 

overall, EPA and the nation are well served by the research accomplishments within the ERP. In 

many cases, there is ample evidence of demonstrated outcomes, for example the Willamette 

River Basin project (see Box 1) and the wide implementation of the results of EMAP across the 

nation. Other programs that deserve specific mention include the successful adoption of bioindicators as measures of environmental health, the MAIA research program, and the various 

programs focused on developing tools to assess the water and air quality of our nation. 

Following the disappointing PART review, the ERP’s LTGs were revised and reduced from four 

to three. The new LTGs thus supersede, in part, the current MYP. This review took place in the 

context of both budgetary changes and changes in the formulation of the ERP goals. Moreover, 

the review contains elements that are new to the BOSC, so the situation is quite fluid and this 

review needs to viewed within this very dynamic context. In responding to the charge questions, 

the appropriateness of both the reformulated LTGs and the budget reduction strategy in the 

context of the goals was discussed. Because the PART review challenged the ERP to explain 

more effectively how its research leads to concrete outcomes in the field, this question received 

considerable attention. Standard questions about program quality and performance that are a 

normal review component also were included in this review. 

This is a challenging time for the ERP in terms of budget. The past 2 years have witnessed 

substantial budget reductions (see Figure 2) and the ERP recently has experienced a 

disappointing PART review. The budget for Fiscal Year (FY) 2005 is not yet finalized (as of 

March 2005 when the review materials were provided to the Subcommittee) and Congress has 

not acted on the FY 2006 budget request and thus the trajectory may change, but current data 

suggest a reduction of approximately 25 percent over a 2-year period (see Figure 2). Despite 

major reductions in budget, almost no changes have occurred in full-time equivalent (FTE) 

employees supporting the ERP (see Figure 3). This disjunction reflects the fact that virtually all 
Figure 2. Total Ecological Research Program Resources, 

2001-2006 (total dollars in millions) 

$102.2

$106.5 $107.8 $107.3

$94.1

$84.1

$0.0

$20.0

$40.0

$60.0

$80.0

$100.0

$120.0

2001 Enacted

2002 Enacted

2003 Enacted

2004 Enacted

2005 Request

2006 Request

Figure 3. Total Ecological Research Program Resources, 

2001-2006 (full-time equivalents) 

329.3 338.1 338.2 333.1 331.7

294.6

0.0

50.0

100.0

150.0

200.0

250.0

300.0

350.0

FT

E

2001 Enacted

2002 Enacted

2003 Enacted

2004 Enacted

2005 Request

2006 Request

Ecological Research Program Review Report 27 
28 Ecological Research Program Review Report

of the budget reductions were in extramural programs, particularly in the STAR Program. The 

present and future budgetary uncertainties make it difficult to comment on future directions. 

Maintaining the current research portfolio and extending it into the future is both essential and of 

high priority in meeting EPA’s mission of protecting, sustaining, and restoring the health of the 

nation’s ecosystems. Nevertheless, current budgetary constraints dictate a full re-evaluation of 

the priorities of the ERP. Future directions must be re-evaluated in light of current and expected 

future budgetary resources. 

As interpreted by this Subcommittee based on the presentations, the three LTGs have a 

consistent logic (see Figure 1). The LTGs appropriately capture the mandate to develop 

scientific knowledge, tools, and methodologies that serve the mission of EPA’s regulatory 

offices and meet the information needs of state governments and tribes. LTG 3 imposes the 

special challenge of reaching local and regional decision-makers. The ERP must depend on 

other EPA offices to implement programs in the field and to work directly with decision-makers, 

and the program should not be asked to take on an extension role when other components of 

EPA are better equipped to assume such a responsibility. Thus, the ERP clearly is dependent on 

other EPA offices to create demonstrated outcomes in the field. It certainly is appropriate to 

expect the entire Agency to function as a seamless organization even though the BOSC reviews 

are piece-meal and are not intended to look at the total organizational context. 
Ecological Research Program Review Report 29 

X. FINDINGS AND RECOMMENDATIONS

The ERP is of good to high quality and program leadership is very good. The Subcommittee is 

impressed with the substantial accomplishments of the ERP as evidenced by the variety of 

excellent tools, methods, and research outcomes documented in the materials provided for the 

review. The findings and recommendations below suggest areas for improvement, but it should 

be emphasized that the ERP is fulfilling its mission as well as providing good value to the nation. 

The Subcommittee’s findings and recommendations are presented below by LTG: 

 The integration of LTG 1 with the other LTGs can be further improved through designing 

research projects specifically for cross-level integration and by reinforcing rules set by 

research programs for close collaborations between EPA and outside researchers at the 

national, regional, and local levels. This integration can be facilitated by working with other 

federal agencies, such as the National Science Foundation, U.S. Department of Agriculture, 

National Aeronautics and Space Administration, and National Oceanic and Atmospheric 

Administration, to develop multiple-scale, interdisciplinary, place-based, and use-inspired 

research programs. 

 Although the overall quality of research under LTG 1 is excellent, high transparency in 

research design, implementation, and evaluation and close collaborations with external 

scientists must be maintained to assure that this high-quality research will persist. 

 Research for all three LTGs would be improved by collaborations with international 

scientific communities. This is important because many environmental problems are either 

physically connected or ecologically similar worldwide and because such scholarly exchange 

among countries will help improve the global environment in which we are embedded. 

 Clear and substantial progress toward LTG 2 was demonstrated in much of the mature 

research, although this could not be readily evaluated for research at earlier stages of 

development. Stakeholder input to identify research gaps does occur, but the process to brief 

decision-makers of results and accept feedback currently is informal or often lacking. The 

effectiveness of this aspect of the program could be improved by establishing timely and 

regular communications with a broad array of stakeholders using an established procedure. 

 LTG 3 is a newly reorganized program element, and there is an opportunity to develop an 

explicit research plan. The Subcommittee suggests that the plan include specific programs 

and projects with specific deliverables and timelines such that, in the future, progress can be 

tracked and the quality, efficiency, and impact of the program elements can be evaluated. 

Furthermore, the BOSC should review the MYP when it is developed. 

 LTG 3 requires better integration with, and articulation of, outcomes at local levels. This is 

essential to achieve EPA’s mandate, but the Agency must be aware of the dangers of asking a 

very good research organization to take on responsibilities that it is not structured to 

accomplish. Responsibilities for communication and dissemination of results certainly rest 
30 Ecological Research Program Review Report

with the ERP, but other elements of EPA also have responsibilities for client and stakeholder 

communication. It is important to recognize that ORD has a primary research mission. 

There is a danger in assigning other priorities to ORD because the research mission may be 

compromised. The time and talents of ORD’s research scientists need to be focused on the 

research mission. At the same time, careful tracking of outcomes is essential to assure that 

the research conducted by the ERP is appropriate and that it addresses customer priorities. 

 Currently, there is no plan for an extramural component to LTG 3. The Subcommittee 

members believe that, historically, partnering with other federal, state, and tribal agencies; 

academic institutions; and NGOs has been very successful. This has increased the 

productivity of the program by leveraging the resources and creativity of these partners. The 

Subcommittee understands that difficult decisions need to be made relative to resource 

allocations, but suggests that the elimination of extramural programs will result in 

disproportionately less productivity and creativity. The Subcommittee recommends that 

some form of extramural cooperation be re-established to leverage resources and continue to 

provide flexibility in the research program. 

 More rigorous program-wide mechanisms should be in place to maximize collaborations 

between EPA researchers and external scientists. This assures that the ERP research is of the 

highest scientific quality and of utmost relevance to EPA’s mission. Close collaborations 

between EPA researchers and external scientists are vital. Mechanisms for forging and 

maintaining such collaborations need to be in place to assure intellectual vitality and 

openness to new ideas. For example, open grant competitions, such as STAR, should be 

slightly modified with an explicit requirement for collaboration with EPA researchers. Such 

collaborations make the best use of intellectual and logistic resources within and outside 

EPA; facilitate high-quality scientific output that is intimately tied with the goals of the ERP; 

and promote scholarly exchanges and information sharing that enhance EPA’s research 

capacities. For the ERP to be most productive and relevant, it is crucial to create a research 

environment in which the ERP’s own researchers can feel excited about, and rewarded by, 

external collaborations rather than feeling pressured for competing resources with outsiders. 

 EPA and ORD must increase involvement of stakeholders (especially external stakeholders) 

in setting research priorities and targeting research efforts (adaptive research management). 

 Effective communication is vital and must be viewed as an essential step in turning research 

results into outcomes. Moreover, those charged with administering the organization must 

receive timely and appropriate measures of research effectiveness. 

• Clearly articulate specific outcomes of research projects, including linked performance 

indicators. 

• Institute a formal process for sharing and disseminating research results to stakeholders. 

 Critical scientific peer review is the standard measure of quality in judging among potential 

research investments and this standard has served our nation very well. The ERP utilizes 

peer review in judging many, but not all, aspects of its research portfolio. There is room for 

improvement in the application of peer review to potential projects within the organization. 
Ecological Research Program Review Report 31 

At the same time, EPA should be generally complimented on its dedicated efforts to 

implement peer review across the organization. 

 The ultimate concern of EPA is to measure and improve the health of the nation’s 

environment, yet some things are missing from what might be viewed as a well-balanced 

research portfolio. 

• The ERP program is heavily oriented towards aquatic ecosystems and EPA’s mission of 

conserving the nation’s water resources justifies this focus. Moreover, many terrestrial 

issues are indirectly addressed through the aquatic program. Nevertheless, a balanced 

research portfolio also requires attention to impacts on terrestrial ecosystems, especially 

relative to clean water and nonpoint source pollution. It is important to better integrate 

ERP research with assessments of terrestrial ecosystems done by other entities, especially 

other federal agencies. 

• Improving the health of the environment clearly requires a better understanding of human 

motivations and behavior. Some projects such as the MAIA involve a social science 

component, yet this dimension of science is largely absent from the ERP portfolio. It 

may be that these responsibilities lie elsewhere in ORD and the Subcommittee’s review 

failed to see the full picture. Nevertheless, we urge the ERP to be mindful of the full 

spectrum of research required to meet the goals of EPA. 

 More could be done to improve the tracking of outcomes. Specifically, EPA, ORD, and the 

ERP need to work harder to foster better communication among EPA offices, with the other 

elements of the Executive Branch of government, and with external stakeholders. The 

Subcommittee urges that more be done to integrate new research paradigms into an 

“outcomes-based mentality” and that these efforts include specific provisions for tracking onthe-ground outcomes. Somewhere within EPA, but not necessarily within ORD, there needs 

to be a focus on better integration of scientific results into stakeholder education and 

decision-making. 

 The recent budget reductions have been difficult for the ERP, and the decision to take these 

reductions in extramural programs is understandable when viewed from the perspective of 

salvaging an effective research organization. This may appear to be a reasonable short-term 

response to weathering a storm, but it is dangerous. EPA and ORD should take a broader 

view and consider how to achieve maximum results in a new permanent budgetary 

environment. Clearly, abandoning extramural research grants is not a cost-effective strategy 

in this circumstance. Competitive research grants, as embodied in STAR and other 

programs, leverage considerable resources as well as bring new thinking and approaches into 

the organization. They also connect the organization to one component of the external 

customer base. The lost leveraging capability, reduced connection to other sources of 

innovation, and weakening of communication channels, over the long term, will 

disproportionately diminish the cost-effectiveness of EPA’s research.
APPENDIX A: BOSC ECOLOGICAL RESEARCH

SUBCOMMITTEE MEMBERS

Dr. Michael T. Clegg1

Department of Ecology & Evolution 

University of California 

Irvine, CA 92697-1010 

Tel: 949-824-4490 or 949-824-4489 

E-mail: mclegg@uci.edu 

Russel Frydenborg 

Florida Department of Environmental Protection 

2600 Blair Stone Road 

Tallahassee, FL 32399-2400 

Tel: 850-245-8063 

E-mail: russel.frydenborg@dep.state.fl.us 

Dr. John P. Giesy2

Department of Zoology 

Michigan State University 

East Lansing, MI 48824-1222 

Tel: 517-353-2000 

E-mail: jgiesy@aol.com 

Dr. Richard Lowrance 

Agricultural Research Service 

USDA-ARS 

P.O. Box 748 

Tifton, GA 31794 

Tel: 229-386-3894 

E-mail: lorenz@tifton.usda.gov 

Dr. Sue A. Thompson 

Pennsylvania Biodiversity Partnership 

16 Terminal Way 

Pittsburgh, PA 15219-1209 

Tel: 412-481-4100 

E-mail: thompson@pabiodiversity.org 

Dr. R. Eugene Turner 

Coastal Ecology Institute 

Louisiana State University 

Baton Rouge, LA 70803 

Tel: 225-578-6454 

E-mail: euturne@lsu.edu 

Dr. Jianguo (Jingle) Wu 

School of Life Sciences 

Arizona State University 

P.O. Box 874501 

Tempe, AZ 85287-4501 

Tel: 480-965-1063 

E-mail: jingle.wu@asu.edu 

1

 Member of BOSC Executive Committee and Chair of the Ecological Research Program Subcommittee. 

2

 Member of BOSC Executive Committee. 

32 Ecological Research Program Review Report
Ecological Research Program Review Report 33 

APPENDIX B: BOSC ECOLOGICAL RESEARCH

SUBCOMMITTEE MEETING DATES

17 February 2005 Teleconference 

Participants: 

Subcommittee Members: Michael T. Clegg, Russel Frydenborg, John P. Giesy, Richard Lowrance, Sue 

A. Thompson, R. Eugene Turner, Jianguo Wu 

EPA Attendees: Steven Hedtke, Jennifer Robbins, Kevin Summers, Greg Susanke 

Other Participants: Stefanie Nelson, The Scientific Consulting Group (SCG) 

3 March 2005 Teleconference 

Participants: 

Subcommittee Members: Michael T. Clegg, Russel Frydenborg, John P. Giesy, Richard Lowrance, Sue 

A. Thompson, R. Eugene Turner, Jianguo Wu 

EPA Attendees: Deb Gonima, Kevin Summers, Greg Susanke 

Other Participants: Stefanie Nelson, SCG 

7-9 March 2005 Site Visit at EPA Facilities, Research Triangle Park, North Carolina 

Participants: 

Subcommittee Members: Michael T. Clegg, Russel Frydenborg, John P. Giesy, Richard Lowrance, Sue 

A. Thompson, R. Eugene Turner, Jianguo Wu 

EPA Attendees: Rochelle Araujo, Mark Bagley, Joan Baker, Thomas Barnwell, Allen Basala, William 

Benson, Roger Blair, Benjamin Blaney, Michael Blum, David Bolgrien, Patricia Bradley, Arden 

Calvert, Timothy Canfield, Robin Dennis, Robert Dyer, Virginia Engle, Gary Foley, Tome Fontaine, 

Jonathan Garber, Deborah Gonima, Iris Goodman, Richard Greene, Steven Hedtke, Jeffrey Herrick, 

Robert Hetes, Virginia Houk, Laura Jackson, William Jenkins, John Johnston, Bruce Jones, Janet 

Keough, Daniel Kluza, Lorelei Kowalski, Gail Lacy, James Lazorchak, Timothy Lewis, Deborah 

Mangis, Michael McDonald, Megan Mehaffey, Bruce Mintz, Marirosa Molina, Jewel Morris, Wayne 

Munns, Anne Neale, Susan Norton, Robert Olexsey, Tony Olsen, John Paul, Steve Paulsen, Gina 

Perovich, Drew Pilant, Robert Puls, Jack Puzak, Mary Reiley, Denise Sailstad, Victoria Sandiford, 

Gregory Sayles, Stephen Schmelling, Nathan Schumacker, Victor Serveiss, Mark Shanis, Michael 

Slimak, John Stoddard, Elise Striz, Kevin Summers, Greg Susanke, Ginger Tennant, Jo Ann 

Thompson, Paul Wagner, Randall Waite, Thomas Wall, David Walters, Jordan West, Joseph 

Williams, Dorsey Worthy 

Other Participants: Tom Atkeson (Florida Department of Environmental Protection), Roger Batterman 

(Wisconsim Department of Environmental Protection), Vicki Bott (Bowman) (University of North 

Carolina–Charlotte Urban Institute), Stefanie Nelson (SCG), Kathryn Saterson (Duke University), 

Robert Stevens (Florida Department of Environmental Protection), Scott Urquhart (Colorado State 

University), Lexia Weaver (University of North Carolina–Chapel Hill) 
34 Ecological Research Program Review Report

1 April 2005 Teleconference 

Participants: 

Subcommittee Members: Michael T. Clegg, Russel Frydenborg, John P. Giesy, Richard Lowrance, Sue 

A. Thompson, R. Eugene Turner, Jianguo Wu 

EPA Attendees: Kevin Summers, Greg Susanke 

Other Participants: Stefanie Nelson, SCG 
Ecological Research Program Review Report 35 

APPENDIX C: LIST OF ACRONYMS

BOSC Board of Scientific Counselors 

EMAP Environmental Monitoring and Assessment Program 

EPA U.S. Environmental Protection Agency 

ERP Ecological Research Program 

FACA Federal Advisory Committee Act 

FTE Full-Time Equivalent 

FY Fiscal Year 

GIS Geographic Information System 

LTGs Long-Term Goals 

MAIA Mid-Atlantic Integrated Assessment 

MYP Multi-Year Plan 

NGOs Nongovernmental Organizations 

OAQPS Office of Air Quality Planning and Standards 

OMB Office of Management and Budget 

ORD Office of Research and Development 

OW Office of Water 

PART Program Assessment Rating Tool 

ReVA Regional Vulnerability Assessment 

RTP Research Triangle Park 

SEQL Sustainable Environment for Quality of Life 

STAR Science To Achieve Results 

TMDL Total Maximum Daily Load 

USDA U.S. Department of Agriculture