UNITED STATES ENVIRONMENTAL PROTECTION AGENCY 

 WASHINGTON D.C. 20460 

OFFICE OF THE ADMINISTRATOR 

SCIENCE ADVISORY BOARD 

September 4, 2009 

EPA-SAB-09-017 

The Honorable Lisa P. Jackson 

Administrator 

U.S. Environmental Protection Agency 

1200 Pennsylvania Avenue, N.W. 

Washington, D.C. 20460 

Subject: Consultation on Development of EPA’s Report on the Environment (2012) 

Dear Administrator Jackson: 

At the Request of EPA’s Office of Research and Development (ORD), in 2007, the 

Science Advisory Board (SAB) reviewed a draft of the Agency’s Report on the Environment: 

Science Report (ROE). The SAB provided recommendations to improve the ROE before it was 

subsequently published in 2008. In addition, the SAB strongly supported continued development 

of the ROE and provided recommendations to improve future versions of the report. The 2008 

ROE presented a compilation of scientific indicators of the environmental condition of the 

nation’s air, water, and land, and status and trends regarding human health and ecological 

systems. The indicator information in the 2008 ROE was intended to answer status and trend 

questions of critical importance to EPA’s mission. 

ORD requested further advice from the SAB to continue developing the ROE, and a new 

SAB Committee was formed to advise EPA regarding development of the next version of the 

report to be published in 2012. On June 30 - July 1, 2009, the SAB Committee held an initial 

consultation with EPA on a paper, Issues for Initial Consultation with the SAB Advisory 

Committee on EPA’s Report on the Environment. The paper describes EPA’s proposed 

approaches for restructuring the ROE and incorporating conceptual models and supplemental 

information. Committee members discussed the paper and submitted the enclosed written 

comments. This letter highlights key points of the discussion. 

EPA’s ROE is an important document that can be of great value to the Agency as a 

strategic planning and public information tool. EPA has made remarkable progress in 

developing the ROE and has responded to many of the SAB comments on previous drafts of the 

report. However, Committee members found that a clear mission statement for the ROE needs 

to be articulated, and that the clearest statement could originate from EPA leadership. ORD has 

proposed aligning the structure of the ROE with goals and objectives in the Agency’s Strategic 

Plan. This may be a worthwhile goal for the ROE, but the current structure and content of the 
 

 

 

 

 

 

report would require revision to be effective in supporting the strategic planning process. A clear 

articulation of the purpose and objectives of the ROE is needed to guide plans for the next 

iteration of the report. In addition, if the ROE is to be used as a strategic planning tool, some 

information should be included to provide an historical perspective on the condition of the 

environment and benchmarks or goals for the future. Committee members also found that the 

audience of the report is not clearly defined; there seem to be multiple audiences that may need 

somewhat different information. The proposal to align the ROE with goals and objectives in the 

Agency’s Strategic Plan suggests that the main audience of the report would be EPA leadership. 

However, there are multiple potential audiences for the ROE and it may not be possible to write 

the report in a way that will be optimal for all of them. Therefore, EPA must decide who the 

most critical audience is or prepare different versions of the ROE for different audiences. 

As in 2007, SAB Committee members underscore the need for an overarching and 

unifying framework to link and integrate the scientific elements of the ROE. EPA has not yet 

developed a proposed framework or model that provides much insight into how chapters, 

questions, and indicators in the report are connected. Possible approaches to consider in 

developing such an overarching framework are discussed in the enclosed comments from 

Committee members. Finally, Committee members urge EPA to invest more resources in 

continued development of the ROE. Because of the breadth and depth of the ROE, developers of 

the report periodically require additional technical support to update environmental indicators in 

the report. 

Because this was a consultation, there will be no formal report from the SAB. Enclosed 

are my summary of key points discussed by the Committee and the individual comments from 

Committee members. We hope these comments are helpful to EPA and look forward to 

providing additional advice to the Agency as it continues the important work of developing the 

next edition of the Report on the Environment. 

Sincerely, 

/Signed/ 

 Dr. James Sanders, Chair 

 SAB Advisory Committee on EPA’s Report on 

 the Environment 

Enclosures: 

Enclosure 1: Committee Roster 

Enclosure 2: Comments from Committee Members 

Enclosure 3: EPA Charge Questions to the Committee 

cc: Dr. Deborah Swackhamer 

 Chair, EPA Science Advisory Board 

2
 
______________________________________________________________________________ 

Enclosure 1: Committee Roster 

U.S. Environmental Protection Agency 

Science Advisory Board 

Advisory Committee on EPA's Report on the Environment 

CHAIR 

Dr. James Sanders, Director and Professor, Skidaway Institute of Oceanography, Savannah, 

GA 

MEMBERS 

*Dr. David T. Allen, Professor, Department of Chemical Engineering, University of Texas, 

Austin, TX 

Dr. Henry Anderson, Chief Medical Officer, Division of Public Health, Wisconsin Division of 

Public Health, Madison, WI 

**Dr John Bailar, Scholar in Residence, The National Academies, Washington, DC 

Dr. Timothy Buckley, Associate Professor and Chair, Division of Environmental Health 

Sciences, College of Public Health, The Ohio State University, Columbus, OH 

Dr. Ingrid Burke, Director, Haub School and Ruckelshaus Institute of Environment and Natural 

Resources, University of Wyoming , Laramie, WY 

Ms. Lauraine Chestnut, Managing Economist, Stratus Consulting Inc., Boulder , CO 

Dr. Aaron Cohen, Principal Scientist, Health Effects Institute, Boston, MA 

Dr. Loveday Conquest, Professor, School of Aquatic and Fishery Sciences, University of 

Washington, Seattle, WA 

Dr. Jeffrey Griffiths, Associate Professor, Public Health and Community Medicine, School of 

Medicine, Tufts University, Boston, MA 

Dr. Charles Hawkins, Professor and Director, Western Center for Monitoring and Assessment 

of Freshwater Ecosystems, Department of Aquatic, Watershed, and Earth Resources, Utah State 

University, Logan, UT 

Dr. Lynda Knobeloch, Senior Toxicologist, Wisconsin Department of Health Services, 

Madison, WI 

3
 
 

Dr. Allan Legge, President, Biosphere Solutions, Calgary, Alberta, Canada 

Dr. Amanda Rodewald, Associate Professor of Wildlife Ecology, School of Environment and 

Natural Resources, The Ohio State University, Columbus, OH 

*Dr. Peter Scheff, Professor, Environmental and Occupational Health Sciences, School of 

Public Health, University of Illinois at Chicago, Chicago, IL 

Dr. Mark Schwartz, Professor, Department of Environmental Science and Policy, College of 

Agriculture, University of California, Davis, CA 

Dr. Alan Steinman, Director, Annis Water Resources Institute, Grand Valley State University, 

Muskegon, MI 

Dr. Helen Suh, Associate Professor, Environmental Health, School of Public Health, Harvard 

University, Boston, MA 

Dr. Duc Vugia, Chief of Infectious Diseases Branch, Division of Communicable Disease 

Control, California Department of Public Health, Richmond, CA 

*Dr. Kathleen Weathers, Senior Scientist, Cary Institute of Ecosystem Studies, Millbrook, NY 

Dr. Judith S. Weis, Professor, Department of Biological Sciences, Rutgers University, Newark, 

NJ 

Dr. Stephen Weisberg, Executive Director, Southern California Water Research Project 

Authority, Costa Mesa, CA 

SCIENCE ADVISORY BOARD STAFF 

Dr. Thomas Armitage, Designated Federal Officer. U.S. Environmental Protection Agency, 

Washington, DC 

* Unable to attend the June 30 - July 1, 2009 meeting 

** Unable to attend the June 30 - July 1, 2009 meeting but submitted written comments 

4
 
 

 

Enclosure 2: Comments from Committee Members 

Page 
 

SUMMARY OF KEY POINTS OF DISCUSSION BY COMMITTEE CHAIR, 

DR. JAMES SANDERS................................................................................................................ 7
 

COMMENTS FROM DR. HENRY ANDERSON................................................................... 10
 

COMMENTS FROM DR. JOHN BAILAR............................................................................. 12
 

COMMENTS FROM DR. TIMOTHY BUCKLEY................................................................ 14
 

COMMENTS FROM DR. INGRID BURKE .......................................................................... 16
 

COMMENTS FROM MS. LAURAINE CHESTNUT............................................................ 17
 

COMMENTS FROM DR. AARON COHEN .......................................................................... 19
 

COMMENTS FROM DR. LOVEDAY CONQUEST ............................................................. 21
 

COMMENTS FROM DR. JEFFREY GRIFFITHS................................................................ 23
 

COMMENTS FROM DR. CHARLES HAWKINS ................................................................ 24
 

COMMENTS FROM DR. LYNDA KNOBELOCH ............................................................... 26
 

COMMENTS FROM DR. ALLAN LEGGE ........................................................................... 28
 

COMMENTS FROM DR. AMANDA RODEWALD ............................................................. 29
 

COMMENTS FROM DR. MARK SCHWARTZ.................................................................... 35
 

COMMENTS FROM DR. ALAN STEINMAN...................................................................... 38
 

COMMENTS FROM DR. HELEN SUH ................................................................................. 41
 

COMMENTS FROM DR. DUC VUGIA.................................................................................. 43
 

5
 
COMMENTS FROM DR. JUDITH WEIS.............................................................................. 46
 

COMMENTS FROM DR. STEPHEN WEISBERG ............................................................... 47
 

ISSUES FOR CONSULTATION.............................................................................................. 48
 

6
 
Summary of Key Points of Discussion by Committee Chair, Dr. James Sanders 

•	 EPA’s Report on the Environment is an important document that can be of great value 

to the Agency and the public. EPA has made remarkable progress in developing the Report 

on the Environment and has responded to many SAB comments on previous drafts of the 

report. 

•	 Additional improvements are needed to make the ROE a more effective strategic 

planning and public information tool. The developers of the ROE need more resources 

to incorporate these improvements. Because of the breadth of the ROE, developers of the 

report periodically require additional technical support in biostatistics, surveys, 

epidemiology, toxicology, and other relevant disciplines. It is important that EPA commit to 

providing that technical assistance either by hiring additional full-time staff or making 

experts elsewhere in ORD available to work on the report. 

•	 A clear mission statement for the ROE is needed. A clear mission statement is needed not 

only to define the intended audiences of the report, but to clearly define its objectives in a 

more general sense. If properly conceived, the mission statement could provide guidance 

and direction on other key issues such as choice of an overarching and unifying conceptual 

model. The audience of the ROE is not clearly defined. EPA has stated that it intends to 

restructure the ROE to more directly align its chapters with EPA’s Strategic Plan goals, and 

to align policy questions in the ROE with objectives in the Agency’s Strategic Plan. The 

main audience would therefore appear to be EPA leadership. However, there are multiple 

potential audiences. It may not be possible to write the ROE in a way that will be optimal for 

all audiences, so EPA must decide who the most critical audience is or prepare different 

versions of the ROE for different audiences. For example, it may be useful to develop a 

“Citizen’s Guide” to the ROE for the public. This version of the ROE could be prepared 

using simpler language, colorful graphics, and a transparent approach to convey the key 

messages of the document. 

•	 There is a need for an overarching and unifying conceptual model for the ROE. The 

current ROE lacks an overarching conceptual framework that brings together the scientific 

elements of the report and allows the natural placement of each indicator in the framework. 

The conceptual models proposed by EPA for use in the ROE are media specific and provide 

little insight into how chapters, questions, and indicators in the report are interconnected. 

The ROE conceptual framework should provide the foundation for understanding what EPA 

is interested in knowing and why. The conceptual framework should connect the questions 

in the report, embrace important new questions, issues, and indicators, and also provide 

guidance on what should and should not be included in the ROE. EPA may wish to consider 

using the European Environmental Agency’s Pressure-State-Response approach to develop 

an overarching conceptual framework for the ROE. Other possible conceptual modeling 

approaches are provided in the attached comments. 

•	 It may be useful to incorporate additional process-oriented models into the media 

chapters of the ROE. Specific chapter models could address the underlying scientific 

foundation of many of the ROE questions and indicators. Slightly more detailed models may 

7
 
 

also be developed for some of the individual questions. The SAB previously recommended 

that such models be process-oriented. Although EPA has stated that it is not the intent of the 

ROE to identify mechanisms and drivers of the indicators, some recognition of the 

underlying processes is necessary to present the scientific foundation of questions in the 

report and identify strategies to remedy and avoid problems. EPA’s proposed conceptual 

model examples for the drinking water and air questions show linkages from emissions to 

health and environmental effects and this is helpful in putting individual indicators into a 

context. However, not all of the linkages in these models are equally important and not all of 

the linkages are under EPA jurisdiction. Furthermore, the proposed models do not account 

for important “effect modifiers” such as meteorology or age. 

•	 Alignment of the ROE with EPA’s strategic goals and objectives makes sense but 

indicators should be linked to EPA program actions. If the ROE is to be effectively used 

in the strategic planning process it needs to link the indicators to EPA program actions and 

describe benchmarks or goals, where possible, so that progress relative to goals can be 

assessed. The link between reductions in pollutants and improvements in environmental 

quality should be made, with the goal of answering the question, “how much reduction in 

emissions or environmental concentration is needed to produce environmental 

improvements?” The overarching conceptual model for the ROE needs to include the 

feedback loop of EPA regulation and policy as an action/response that affects the 

environment. Consideration of both an ecological process model as well as an operational 

process model that focuses on linking management actions with desired environmental 

outcomes would be appropriate in this regard. In addition, some historical perspective 

should be provided. The reader needs to know where we have come from, where we are 

today, and where we need to be in the future. 

•	 It is important to retain rigorous criteria for the selection of ROE indicators, but 

supplemental information can be useful in filling gaps where national data are 

unavailable and addressing questions of limited geographic scope. This information 

can help identify emerging vulnerabilities or threats and consolidate information that 

may be useful to a broad set of users. EPA’s proposal to include supplemental 

information in the ROE has merit. However, if the supplemental information is quarantined 

in a ROE incubator site or appendix as proposed, the information may be of limited use. 

Therefore, at the very least, the ROE should include information about the indicators that are 

listed in the incubator site. If additional resources are focused on meta-analysis of 

supplemental data, it may be possible to combine results from different supplemental studies 

and develop aggregate results that could be used in the ROE. It may also be possible to 

integrate supplemental information into the ROE and provide additional metadata 

information about the data source, rigor, geographic region of significance, and level of 

confidence in the data. 

•	 It is important to provide a defined place in the ROE for discussions of emerging issues. 

The document should be forward-looking. Emerging issues should be discussed in the 

ROE, perhaps in the supplemental information section but also in an executive summary, to 

show that EPA is aware of and beginning to monitor or consider important new issues. It 

will be these emerging issues that are likely to encompass the EPA’s efforts in coming 

8
 
decades, and in order for the ROE to help drive strategic planning, these issues must be 

identified and prioritized. 

•	 Regional problems require regional treatment. Not all problems are national. This is 

particularly true of ecosystem problems. For example, ecologists have long recognized 

increased tree mortality in the eastern U.S. driven by acid rain, and increased tree mortality in 

some regions of the U.S. driven by invasive species (e.g., gypsy moth) and urban island 

effects (high urban tree mortality). In developing the ROE, EPA should consider using 

regional indicators that represent important status and trend information. EPA may see its 

mandate as national in scope, but many readers of the ROE do not make distinctions between 

national, regional, and state jurisdictions. In considering the use of regional indicators it is 

important to ensure use of appropriate scales for analysis of information. 

•	 The ROE should contain an executive summary that distills and synthesizes the 

indicator information to address the questions in the report. The 2008 ROE is a great 

collection of indicator data, but alone it is not sufficient for planning purposes because it does 

not draw conclusions about the state of the environment. Some synthesis or interpretation, 

rather than just providing summary information and allowing users to interpret or draw 

conclusions, would be useful. If connections cannot be made between indicators and 

outcomes, questions in the report should probably be reworded. The executive summary 

should also be published as a separate document because it is likely to be of interest to the 

general public, law makers, and policy makers who may not read the entire ROE. 

•	 In developing the ROE, EPA should emphasize producing a powerful and interactive 

web-based platform with links to pertinent websites containing additional data and 

information. The 2012 ROE should be available as a printed report, but each chapter should 

be published in an electronic format that is fully searchable. Additional resources will be 

needed to accomplish this. 

•	 Where possible trends in environmental indicators should be linked to indicators for 

ecological condition and human health. For example, EPA has estimated the health 

impacts of air pollution in quantitative terms in other reports. These estimates provide direct 

answers to the ROE question for air and would help fill gaps that exist in the 2008 ROE. 

•	 Several other key points are provided for EPA’s Consideration. 1) It is important to 

maintain the statistical validity and scientific rigor of the ROE. 2) An efficient way to use 

staff, produce fresh products and keep policy-makers focused would be to produce a different 

ROE chapter once per year on a rotating schedule. 3) Integration across agencies that 

conduct environmental monitoring is critical. An interagency monitoring 

effort/clearinghouse/data source would be very useful. 

9
 
Comments from Dr. Henry Anderson 

Charge Question 1: Please comment on whether the EPA’s proposed conceptual modeling 

approaches are logical and useful for: 

a) Aligning the ROE questions with the Agency’s strategic goals and objectives. 

There is a need for an overarching conceptual model in addition to the more focused models 

proposed. This needs to tie everything together. It will be very useful to have the ROE grounded 

in the agency strategic goals and objectives and the conceptual models proposed can easily do 

that. The first ROE was an Administrator directed initiative and it would give the proposed ROE 

greater import if the current Administrator would not only endorse the production of the ROE 

but provide some direction as to how she intends to have it used within the agency. Aligning the 

ROE with the strategic plan gives the ROE a core agency constituency and use. This is not the 

only constituency but is an important one that will give the ROE ongoing utility. For instance 

the Administrator could suggest that all internal research proposals and external RFAs should 

reference what gap or ROE area it will address. This would get the ROE continually referenced 

and read and provide some import to the gaps and needs identified in the ROE. It would become 

a driver for programs to support the ROE development out of self-interest. 

b) Communicating the intent and scope of questions in the ROE. 

The proposed ROE comes at a time of transition in information communication. While a paper 

report needs to be provided, greater emphasis should be placed upon the electronic version of the 

report. The ROE needs to become a multimedia enterprise and not just a single report. By the 

time the ROE is completed in 2012 we can anticipate additional information technology 

advances and the ROE needs to be sufficiently nimble to be “state of the art” when it comes out. 

Right now the ROE needs to be written about in Wikipedia and the other personal 

communication modalities explored. Cross linkages with the CDC Tracking Portal and other 

“indicator” sites need to be negotiated to get the current ROE in greater circulation. An 

electronic format allows for interactive utilization of the data behind static graphs and charts 

allowing individualized reports. The ROE needs to become an integral part of a 

communication/information system. The on-line material can also link to other resources as they 

become available. There are a growing number of state and federal agencies developing 

indicators and the ROE needs to recognize them and not duplicate effort. 

A critical current need is for an evaluation of the previous ROE and an assessment of impact. 

For the next ROE an evaluation plan needs to be developed before it comes out. 

Other comments: 

The ROE indicators should be broadened to include one or more social science indicators. These 

might include the use of the many surveys of public awareness, perceptions and beliefs, some of 

which can even provide trends. If not actual indicators added to each question answer, this type 

of information needs to be provided in the supplemental information sections. Supplemental 

information will add perspective and relevance to the information provided with the indicators. 

10
 
The ROE needs to provide interpretation and strategies to achieve the goals, not just present the 

data. The questions in the current ROE cry out for answers and the reader is disappointed when 

no interpretation is provided. 

It would help to have a “citizen’s guide” to the ROE. 

11
 
Comments from Dr. John Bailar 

Comments on “Issues for Initial Consultation with the SAB Advisory Committee on EPA's 

Report on the Environment” 

First it is a pleasure to commend the quality of this document. It is clear, well organized, easy to 

read, highly informative, and to the point. These features make it much easier to comment 

intelligently, and contribute to making a good product even better (as opposed to making a poor 

product marginally acceptable). I cannot make specific recommendations because I have not 

discussed this with committee members or others, and may miss some of the critical background. 

Still, there are several points that the committee should think about as it considers what 

recommendations would be appropriate. The following are in no particular order. 

The whole process is geared to EPA’s strategic plan, despite words here and there about other 

uses. This internal use is important, but other readers will have other uses for the data, and too 

tight a focus on the strategic plan could undercut broader uses of the ROE that might contribute 

to EPA’s mission in critical but less direct ways. A few of these are: informing the public about 

matters that interest them (and the public support that would generate), greater service to nonEPA governmental organizations, including state and local agencies, and informing various 

public interest groups about emerging issues that EPA cannot deal with for reasons of limitations 

in its mandate or resources. By all means be sure that the ROE is supportive of and consistent 

with the strategic plan, but I urge you to give these other uses of ROE at least as much weight as 

its internal uses. 

It would be helpful, and might significantly advance EPA’s overall mission, to include a short 

section on international comparisons and trends. Such data could support new initiatives at EPA 

by showing what others have already demonstrated to be possible, and would help to put much 

of ROE in a broader context. Candidate items might include age-adjusted mortality rates for 

specific causes (e.g. liver cancer, or selected pneumoconioses), PM2.5 levels in a few cities, or 

some measure of water quality. 

I would adjust the frequency of ROE to match that of the strategic plan, perhaps offset by a year 

to allow for feedback from the plan to ROE and for impact of ROE on the plan as it is developed. 

Among other points, it is not clear to me why ROE should match the structure (as opposed to 

content) of the strategic plan. This is surely a convenience for EPA, but at what cost in 

organization to other users? Again I strongly recommend that you give a lot more attention to 

users outside the EPA community. 

Some policy-relevant questions are beyond EPA’s mission – e.g. some kinds of highly localized 

pollution, de minimus violations that nevertheless cause substantial local concern, or 

exceedances that are very large but do not meet the frequency criterion for some EPA-mandated 

intervention. ROE should address this general issue where it explains the rationale behind its 

choice of metrics. 

12
 
A general comment is that future editions of ROE will become more and more valuable as they 

include longer and longer series of data points. This makes it critical that the data series chosen 

for inclusion be continued. Before you decide to include some series, please give some explicit 

thought to the likelihood that the series will not be continued indefinitely. 

I do not know the internal structure that has led to the production of the present and future ROEs, 

but it seems to me critical that there be a dedicated, full-time, permanent staff for this, including 

competence in biostatistics, surveys, epidemiology, toxicology, and other relevant disciplines. 

I would give some thought to limiting the supplemental data to items that display (or will 

display) trends, not one-off sets of observations. 

The ROE should include a meaty section, written in conjunction with others from all major 

components of EPA, that is focused on future needs for data, both new items and new 

information about present items – new problems that EPA should address, improved data quality, 

more extensive tabulations, etc. Getting this into today’s version of ROE, while the gaps and 

problems are still fresh in memory, and where users can see and comment on them, will be better 

than trying to reconstruct the problems and solve them with old records, new staff, etc. 

Every item in the present (and I am sure future) version of ROE is affected by some uncertainty 

– both sampling uncertainty and bias. The discussion should address matters of bias for each 

table, and sampling uncertainty should be addressed quite consistently with confidence bounds 

or SEs. I feel quite strongly about the need for users to understand the limitations on both 

accuracy and precision in each data point presented. This why I recommend that each section, 

and generally each table, include some discussion of uncertainty, including SEs or confidence 

bounds as well as plausible biases. These matters are particularly important for measures of 

change, where uncertainty may be substantially less than for individual levels because some 

kinds of bias will cancel out, and variances of change should include subtraction of covariances. 

Finally, I see an important role for some single, overall, summary index of “environmental 

quality”. Such an index will require considerable time and effort to develop, including a great 

deal of outside stakeholder involvement, but could be of great value to EPA and others in 

tracking the broadest changes over time. 

13
 
Comments from Dr. Timothy Buckley 

Charge Question: Does the Committee have recommendations concerning other possible 

approaches to conceptual model development that would be useful in identifying or highlighting 

important ROE topics, indicators for consideration, research, or development? 

Response: This important and complex document has continued to evolve and become stronger 

and stronger. It is very well written and is coming closer to meeting its potential. The critical 

comments described here should not in any way be interpreted as a lack of enthusiasm over the 

concept of the ROE and its value to EPA and the public that it serves. Accordingly, the critique 

provided here is given in the context of supporting the document development and making it as 

strong as possible. There is still room for improvement including in the area “other possible 

approaches to conceptual model development” addressed in the charge question above. The 

Agency’s interest in developing appropriate conceptual models to represent both the science and 

application of the ROE are well placed as a strategy to effectively organize, present, and 

communicate the ROE. Previous review panels have highlighted the need for an overarching 

conceptual model and the current ROE still lacks such a model. Figure 1-1 of the new report is 

more oriented toward the application or process of the ROE and therefore does not adequately 

address the need of providing a scientific framework. Such a conceptual model will provide a 

valuable contribution to the ROE in: 

•	 Depicting and introducing the main scientific elements of the ROE and how they relate to 

one another. 

•	 Highlighting the highly integrative nature of the ROE 

An example of a simple integrative conceptual model was provided by a previous ROE review. 

In the context of this charge, I provide an additional rough draft of a alternative model that the 

Agency might consider for further development. 

14
 
In its current form, the ROE is largely limited to emissions (sources), the physical environment, 

and their impact on human health and ecology. This orientation ignores a critical upstream 

dimension that forms an underlying driver of sources and stressors on the environment, that is 

the social and behavioral sciences that underlie behaviors and individual decisions of a growing 

population. These behavioral factors have been shown to have a substantial influence on the 

environment, health, and ecology, and therefore provide strong potential indicators thus far not 

being considered include such issues as: 

•	 energy use and conservation; 

•	 water consumption 

•	 consumer product use (e.g. pesticides) 

•	 urban sprawl 

•	 vehicle mile traveled 

•	 combustion source pollution including cooking, grilling, fireplaces, campfires, 

candle/incense burning 

Such indicators are especially valuable to include because they are amenable to intervention 

although not through traditional EPA regulatory strategies. I think that the ROE could provide 

an incredibly valuable educational tool in showing the link between such behaviors, 

environmental quality, and human health and ecological status. 

15
 
Comments from Dr. Ingrid Burke 

1.	 Charge Questions 1 and 2: proposed conceptual models approaches. 

My interpretation of the SAB review of the ROE is very different than what we heard this 

morning about conceptual frameworks/models. I think what that review states is that there is a 

need for an overarching conceptual framework that provides the glue for the entire ROE. The 

presentation this morning represented an elaboration of the questions within each section of the 

report, that is, conceptual models at a scale that is very much more detailed, rather than 

providing the overarching glue or philosophical basis for the ROE. 

I will dive right in and say that I was disappointed with the approach presented today, in terms of 

depth of thinking. The ROE needs a foundation for answering the question, broadly, “What are 

We Interested in Knowing and Why?”; something that can allow us to understand what falls 

under the umbrella, and what DOESN’T. The Conceptual Framework should represent the 

forest, not the trees. 

The Conceptual Model should connect the questions, should embrace important new 

questions/issues/indicators, and additionally, provide guidance for what should NOT be 

included. 

More comments from Indy: 

1.	 The connection of the ROE to a conceptual framework such as the Millenium 

Assessment that explicitly includes ecosystem services (provisioning of clean air, 

adequate quantity and quality of water, provision of fuel, etc.) will allow the ROE to have 

strong relevance to policy. HOW do we increase quality of air? We make clean air worth 

something, etc. (see recent EPA report on Valuing, etc). This will integrate the 

chapters/indicators/questions really well. 

2.	 The executive summary is critical. 

3.	 An efficient way to utilize the staff, produce fresh products, and keep policy-makers 

focused would be a rotating schedule of chapters, one per year. Each year, on a calendar 

date, a synthesis of one question could be produced, with a high visibility. The staff 

would be focused during that year in depth on one particular issue, and when it is out, it 

will be very fresh. 

4.	 We need to find a way to send the message forward to the Administrator that integration 

across agencies conducting environmental monitoring is Critical. An interagency 

effort/clearinghouse/data source would be a gigantic contribution. 

16
 
Comments from Ms. Lauraine Chestnut 

Overview comments: three main points 

1. Alignment with the strategic plan, and applying the indicators to support the policy and 

planning process, makes a lot of sense and is consistent with the apparent original intention of 

the effort to develop a report on the state of the environment. However, to be effectively used in 

the planning process the report needs to link the indicators to EPA’s programs and needs 

to describe benchmarks or goals, where possible, so that progress relative to goals can be 

assessed. 

2. If the primary purpose of the report is for planning and assessment of EPA’s progress, 

the overarching conceptual model needs to include the feedback loop of EPA regulation 

and policy as an action/response that affects the environment. This loop includes an 

assessment of how are we doing relative to benchmarks or goals, which then influences planning 

in a feedback process. 

3. The title, “report on the environment,” suggests a comprehensive view and raises expectations 

of conclusions and assessment. The current (2008) document is a great collection of 

indicator data, but alone it is not sufficient for planning purposes because it does not draw 

conclusions about the state of the environment. A more descriptive title of this document 

would be, “report on environmental indicators.” 

Specific charge questions 

1a: Do the proposed conceptual modeling approaches help to align the ROE questions with the 

Agency’s strategic plan? 

The modeling approaches, as currently described in the proposal, show the linkages from 

emissions to health and environmental effects, which is helpful in putting individual indicators 

into a context. However, the models are not sufficient to support alignment of the indicators 

with the strategic plan, because the models do not include EPA’s programs and actions that are 

intended to affect change in environmental conditions. For alignment with the planning process, 

a more relevant overarching model would be one that includes the assessment of environmental 

problems and priorities that comes from interpretation of the indicators, and the feedback of this 

into the EPA’s actions. These in turn influence emissions that ripple through to the human 

health and environmental effects, as reflected in the indicator levels. 

The proposal to align the ROE with the strategic plan makes a lot of sense and will help focus 

the ROE on the most important indicators needed to support policy and planning decisions, 

including identifying gaps in the available indicators that need to be addressed. It is a step 

toward greater integration of the indicators work with the EPA’s planning process, which is a 

step in the right direction. 

17
 
An important caution or limitation in terms of alignment of the ROE with the strategic plan is 

that some benefits of EPA’s programs and regulations result from preventing problems. Not all 

benefits come on the form of reducing existing problems. Environmental problems that have 

been prevented cannot be measured with indicator trends (other than showing a flat trend 

reflecting sustained quality of a resource); indicators based on observed data cannot show what 

would have happened in the absence of the program or policy that prevented the problem from 

happening. 

1b: Do the proposed conceptual modeling approaches help to communicate the intent and scope 

of the ROE questions? 

The proposed conceptual modeling approaches are very helpful in showing where individual 

indicators fit into the chain from initial pollution emissions (or land use actions); to pollution 

transport and transformation via air, water, land, and biota; and to ultimate human health and 

environmental effects. Such models can help guide selection of appropriate indicators and show 

how “upstream” indicators may be a proxy for “downstream” effects that may be more difficult 

to measure with observational data. The selection issue is important because the ROE is in 

danger of growing thicker and thicker without necessarily providing better answers to the 

relevant questions. 

However, the proposed conceptual models raise the same questions about linkages that the ROE 

questions themselves raise, and that the 2008 report does not address. For example, the ROE 

question, “What are the trends in outdoor air quality and their effects on human health and the 

environment?” explicitly links air quality and human health, but the 2008 ROE puts the 

indicators on air quality and on human health in separate chapters. It is left to the reader to draw 

any conclusions about whether the trends in health indicators have anything to do with the trends 

in air quality. Admittedly, it may not be possible to draw such conclusion given the multitude of 

factors that influence many of the health indicators, but the ROE questions raise expectations 

that an attempt to link them is going to be made. 

One detailed comment on the draft model for outdoor air is that it seems to overlook human 

welfare effects. Part of the problem is that it separates human and ecological exposure as if these 

are independent tracks. It is hard to see how effects on visibility aesthetics, agricultural yields, 

or manmade and cultural materials, and nonuse values for ecosystem preservation enter into the 

chain. It also leaves unmentioned the link from ecological exposure to changes in ecological 

services that in turn affect human health and welfare. It is important to keep the models 

relatively simple, but at the same time general enough to encompass the detail. Another example 

is with mercury where the effect on human health (fish consumption) and welfare (fishing 

advisories and restrictions) travels through the ecological exposure. 

18
 
Comments from Dr. Aaron Cohen 

Issues for discussion of questions 1 and 2 

1.	 In general, EPA has responded to many of the comments in 2007 review, including the 

request for conceptual models. 

2.	 The proposed conceptual models for water and air are useful, but would suggest: 

o	 Adding policy action as first stage of conceptual model for air (with appropriate 

feedback loops): CAA has huge influence on air quality. 

o	 Adding susceptibility factors at individual - and aggregate-levels: e.g., diet, health 

care, socio-economic factors. These are relevant to both geographic scaling 

issues and environmental justice concerns. 

3.	 The current ROE does not provide answers to the ROE questions re. health 

effects/impacts. Here are some suggestions: 

o	 Authors should consider moving the relevant health information into the water and air 

chapters. This would help draw the links necessary to answer ROE questions re. 

health effects/impacts. 

o	 EPA has estimated the health impacts of air pollution in quantitative terms, including 

economic valuation, and these estimates have been extensively and independently 

reviewed (e.g., by NAS). These methods and results are directly relevant to the ROE 

question re. health effects of air pollution, and I have heard no good reason why they 

should not be used. NB EPA’s PM and ozone Criteria Documents (or ISA) and prior 

(and future) staff papers are based on systematic review of the literature, as are EPA’s 

impact assessments. For this reason, they are not subject to criticisms re. “cherrypicking.” In contrast, citing only specific studies, no matter how compelling (e.g., 

Pope et al. 2009), opens the door for such criticism. 

o	 The ROE should also note current efforts/results re. indicator development for health 

impacts of air pollution. NB 2007 review suggested things such as the following: 

	 CDC Environmental Public Health Tracking 

	 EPA’s BENMAP computer-based impact analysis program 

	 Impact assessments by California Air Resources Board, EC and WHO, 

including the Global Burden of Disease/Comparative Risk Assessments. 

Top 3 recommendations 

1.	 A clear mission statement for the ROE is needed. This should come from the highest 

levels of the Agency, and should provide clear guidance on issues such as the relation to 

EPA’s strategic planning and the intended readership. 

19
 
2.	 A unified model/conceptual framework is needed in addition to specific conceptual 

models, e.g. for water and air. The European Environmental Agency’s Pressure, State, 

Response model is an example of one possible approach (as noted at the June Panel 

meeting). 

3.	 EPA estimates of the health impacts of air pollution provide direct answers to the ROE 

question for air and should be included in the ROE. These estimates use data and 

indicators that have been extensively and independently reviewed, and have been used in 

regulatory decision-making. 

20
 
Comments from Dr. Loveday Conquest 

Three ROE Recommendations and Charge Question 3 and 4 comments. 

Three recommendations for the ROE: 

1. There needs to be a conceptual framework that brings together the different media and allows 

the natural placement of each indicator in the framework. (The Example Conceptual Framework 

Diagram in Appendix C-2, EPA-SAB-09-007 Report, is a very decent start, and it does 

incorporate ecosystem processes and services.) 

2. As more of the ROE ends up on the web, it will be important to devote an appropriate level of 

resources (FTEs) to this effort so that sections are easy to access, the links are appropriate, etc. 

A savvy Wikipedia entry is a good idea! 

3. Continue to maintain the ROE’s statistical validity and scientific rigor (particularly with 

respect to use of Supplemental Information—see comments below). 

Charge Question 3. Comment on the logic and utility of EPA’s proposed use of supplemental 

information to answer questions in the next version of the ROE. 

Supplemental information (SI) can indeed be useful to fill gaps in the data and to identify 

emerging issues, even while not meeting the all the original, stringent criteria that EPA has set 

for its indicators. Presumably EPA has already investigated a variety of potential indicators, 

many of which were not included in the ROE because they did not meet the rigorous criteria. 

EPA should revisit the list and focus on those responses that might otherwise have been included 

(e.g., they addressed an important issue, filled a data gap, addressed an emerging issue) but for 

lack of one or more criteria. (It appears as though there were several indicators that “missed out” 

due to not being collected on a national basis.) All candidate indicators should occupy a natural 

place in EPA’s conceptual framework. 

It might be possible to use meta-analysis to statistically combine results from different studies 

and come up with an aggregate result (realizing that there are still criteria of statistical 

comparability to be met in order to carry this out). Even where meta-analysis is not possible, 

data collected on a regional basis could still provide useful aggregate conclusions, using “weight 

of the evidence” approaches. 

Just as EPA developed rigorous, transparent criteria for national indicators, they should do the 

same for regional indicators and for case studies. 

Investigating all sources of data for potential supplemental information is a daunting task. 

Therefore it is essential that EPA focus up front on key requirements (scientific, policy-driven, 

and transparent) in seeking SI. 

21
 
The “incubator site” seems a natural place to put potential indicators that refer to emerging 

issues. Canvass the scientific literature to find best papers to illustrate trends and to identify 

emerging trends in the environment. 

Charge Question 4. Recommendations for criteria to assure that supplemental information 

included in the ROE is objective, free from bias, scientifically valid, and supports intended 

purpose of the report. 

Such data should in general meet the high standards of the ROE Indicator Definition and 

Criteria. (Presumably datasets won’t meet all of the criteria; if they had they would have been 

included as indicators already.) The highest quality data will have been generated from a 

probability sample. For example, EMAP and REMAP did a good job of getting regions, states, 

and other public agencies started in terms of sampling using probability samples. EPA should 

look at those indicators to ascertain which ones would provide useful SI. 

Appendix F of the “Issues” paper also mentions potential usefulness of long-term data sets not 

based on probability sampling approaches. These may still be incorporated into future ROEs, 

but should be investigated carefully to ensure the rigorousness of their conclusions. Similarly, 

case studies (“a sample size of one”) may prove useful, for example, to illustrate a process. The 

“Issues for Initial Consultation” (2009) paper (Appendix F) mentions use of the National Science 

Foundation’s Long-term Ecological Research (LTER) Program sites, USGS ground water basins, 

state agencies, the Great Lakes, Lake Tahoe, and Lake Mendota. There are probably others. 

It is important not to “cherry-pick” indicators for use as SI. Thus, only studies that have been 

adequately peer-reviewed should be used. The appearance of cherry-picking can also be avoided 

by using meta-analysis on a group of studies. 

22
 
Comments from Dr. Jeffrey Griffiths 

The most recent Report on the Environment is an impressive and substantial work. The team that 

has assembled the pieces of the Report, and the Report itself, deserve high praise. 

Although my oral comments have been incorporated into the summary document, please allow 

me to expand a bit on several points. 

First, a number of the public health consequences of environmental pollutants - e.g., the effects 

on the population - could be expanded. To say that mercury is toxic is accurate, however more 

detail - perhaps even just a few sentences - may help to anchor the information in the Report 

more concretely to protection of the public. In general, the background information on the issues 

help to provide a foundation for judging the progress, or lack thereof, in addressing a public 

health issue. 

Along the same lines on the flip side of the equation, there is little interpretation of what the 

improvements in the environment suggest or imply in terms of human health. For example, to 

state that finished drinking water now has many fewer Cryptosporidium parasites in it due to 

recent rules to enhance the efficiency of filtration is one thing; it would be more helpful to state 

that this means that the population served by that system may have X or Y less cryptosporidiosis 

because of this change. 

Second, the report itself could be a living document on line... with individual areas or specific 

topics frequently or continually updated as more information is acquired. We discussed this at 

length during the meeting as the movement to the internet is socially well advanced and it may 

be prudent to both (1) allow updates to occur as they are relevant, rather than on a pre-set 

schedule; and (2) provide the Report in ways that will reach the most people. 

Third, the Report will benefit from being linked to the EPA strategic plans which are currently 

being developed. This will help to focus the document and provide a mechanism whereby the 

public and the Agency can reflect upon the progress being made in the environment and judge its 

success. 

Lastly, one of the questions posed to us regarded the use of data that reflects regional rather than 

national trends. Often no national data sets exists for an issue, which may be of regional but not 

national importance. My recommendation is to use those regional data sets, when they are of 

high quality, when they are illustrative of regional trends that can inform the public. 

23
 
Comments from Dr. Charles Hawkins 

In general, the 2008 ROE presents information on a large number of environmental indicators 

that will be of interest and use to many readers of the report including individual citizens through 

policy makers. The report could be further strengthened by addressing the following issues: 

1.	 The audience is not clearly defined. Given that the ROE staff tried to align the questions 

included in the ROE with EPA’s strategic goals and objectives as a way of better 

informing Agency priorities, the main audience would appear to be EPA leadership. 

However, from our discussions, it was rather clear that there are multiple potential 

audiences. The ROE cannot be written in a way that will be optimal for all audiences, so 

EPA must decide who the most critical audience is – or prepare different versions of the 

ROE for difference audiences. In general, I expect that a format similar to the 2008 report 

will be of most use to researchers and those in need of data. Such a compilation is 

critically needed for a variety of reasons but it is probably not the best way to hold the 

attention of most readers. The current Highlights of National Trends boils down some of 

this information in a way that is suitable for the public, but another type of summary will 

probably be most useful to EPA leadership – a summary that focuses on critical gaps and 

spans/integrates the different chapters. Finally, a 1-2 page summary is needed for press 

releases and Congressional staff. 

2.	 Consider ways to present the indicator information in some type of common format. 

Although tabular data will be useful to data miners, it is an ineffective way to convey 

either status or trends. Also, consider ways to summarize all of the indicator information 

into a central figure that provides an overall visualization of which parts of the 

environment are in good shape and which ones are in poor shape. STAR diagrams are 

often used effectively for these purposes, but the diversity of types of indicators and their 

scales of measurement may make this approach difficult. Nonetheless, a standard 

summary diagram that could be repeated in each report would go far toward describing 

what the overall condition of the environment is. Assigning ‘grades’ to indicator values 

may be one way to avoid this problem, but be cautious in how such grades are 

implemented and what they imply (e.g., 25% species lost does not mean fair or a C in my 

mind, a score that could be applied given the tyranny of how we often chunk up the 0-

100% scale into school grades). 

3.	 The way the central questions are phrased (“What are the trends in ...... and their effects 

on human health and the environment?”) sets the reader up to expect to see connections 

between indicators and outcomes. But the ROE explicitly states elsewhere that no 

connections will be made. This results in the reader becoming confused and/or frustrated 

with the ROE’s content. If connections cannot or should not be made, these questions 

should probably be reworded. Also, note that you present only status information for 

many indicators and trend data do not yet exist. Explicitly address/recognize this fact and 

wordsmith accordingly. 

4.	 I don’t believe we (the Advisory Committee) provided very clear guidance regarding the 

use of conceptual models and found our discussion of this issue somewhat vague and 

24
 
confusing. To the extent that you do explore incorporating a ‘grand’ conceptual model, 

consider developing it in the sense of a ‘conceptual framework’ that helps the reader 

better understand the general philosophy of indicator measurement and interpretation that 

will be applied to all chapters. I suspect your choice of how to best do this will be 

strongly influenced by who you decide you audience is. Also, consider if the amount of 

detail you present in the specific conceptual models used in the report can affect either 

choice or interpretation of indicators. 

5.	 Retain the rigorous criteria for indicator selection when bringing in supplemental data of 

regional importance. 

25
 
 

Comments from Dr. Lynda Knobeloch 

Assignment: Does the committee have recommendations concerning other possible approaches 

to conceptual model development that would be useful in identifying or highlighting important 

ROE topics, indicators for consideration, research, or development? 

Comments: 

The conceptual model should be updated to reflect the most current publication formats and risk 

assessment methodologies, as well as the rapidly evolving fields of environmental science and 

synthetic chemistry. 

1. Publication formats: The 2012 ROE should be available as a hard copy report. In addition, 

each chapter should be published in an electronic format that is fully searchable. The ROE 

website should include links to other relevant websites and publications and should be easy to 

access using popular search engines. 

2. Health Indicators: Indicators for human health should be updated to consider exposures 

during prenatal development, early childhood, adolescence and old age. This change is 

recommended because the biological response to many chemical exposures varies over the 

course of the human lifespan. Developmental windows including prenatal growth, childhood 

and puberty are particularly vulnerable to neurological toxicants and endocrine disruptors, for 

example. The range of human health outcomes considered in the report should be expanded to 

include endocrine disorders, such as thyroid disorders and diabetes, and chronic neurological 

diseases, such as Parkinson’s Disease and Amyotrophic Lateral Sclerosis. 

3. Environmental Indicators: Several indicators used in the 2008 ROE chapters on air and water 

focus on chemicals that are no longer produced or sold in the United States. While it may be 

important to continue to track these substances in the environment, the report should provide 

insights into the effects of newly-synthesized chemicals including chemicals used in industry, 

nanomaterials, pharmaceuticals, and the new generation of agricultural pesticides. 

4. Interpretation: When appropriate, environmental indicators should be linked to indicators for 

ecological condition and human health. For example, temporal trends for air quality indicators, 

such as ozone and particulate air pollution, should be linked to temporal trends for the incidence 

of asthma, COPD, and cardiovascular disease. Similarly, temporal trends for mercury emissions 

and mercury levels in fish tissue should be compared to trends in human biomonitoring for 

mercury. The authors of the report should interpret the data for each indicator and provide 

conclusions. Readers want to know whether things are getting better, staying the same, or 

getting worse and whether there are regional differences within the U.S. Therefore, temporal 

and regional trends should be described whenever possible. 

Additional comments: 

26
 
 

For future editions of EPA’s Report on the Environment to be useful to policy-makers and 

others, it should provide information for chemicals that are currently sold and used in on 

currently emitted substances. It should also address environmental concerns such as the 

unexplained disappearance of honey bees, the finding of hermaphroditic fish and deformed frogs, 

and the decline of wildlife populations such as grassland birds. Many experts have expressed 

concern that newly designed pesticides, bioengineered crops, estrogenic pharmaceuticals or 

industrial solvents may play a role in these disturbing trends. Previous editions of the ROE have 

focused on a relatively small number of chemicals. Many of the chemicals, such as PCBs, DDT, 

mirex and lindane are legacy pollutants that haven’t been produced or sold in the U.S. for nearly 

half a century. Continued focus on these chemicals to the exclusion of newer chemicals along 

with the lack of linkage between ecologic or human health indicators to chemical pollutants 

found in air and water limits the usefulness of the report. 

Future ROEs should strive to find a more holistic way to assess the condition of our environment 

that doesn’t consider chemicals individually. Is there a way to consider, for example, the full 

range of man-made chemicals that have been introduced in the past century to determine whether 

our water and air are really cleaner today than they were 20 years ago? Or are we simply 

replacing old chemicals that are closely monitored with newer chemicals that are not yet 

recognized as threats to the environment? 

27
 
 

 

 

 

Comments from Dr. Allan Legge 

CHARGE QUESTIONS: 

Charge Question 1. Please comment on whether EPA’s proposed conceptual modeling 

approaches are logical and useful for: 

a) Aligning the ROE questions with the Agency’s goals and objectives. 

b) Communicating the intent and scope of questions in the ROE. 

c) Presenting the underlying scientific foundation of questions in the ROE. 

d) Providing a framework for selecting indicators and identifying associated gaps, 

limitations, and useful supplemental information. 

Charge Question 2. Does the Committee have recommendations concerning other possible 

approaches to conceptual model development that would be useful in identifying or highlighting 

ROE topics, indicators for consideration, research, or development? 

Charge Question 3. Please comment on the logic and utility of EPA’s proposed use of 

supplemental information to answer questions in the next version of the ROE. 

Charge Question 4. Does the Committee have recommendations for criteria to assure the 

supplemental information included in the ROE is objective, free from bias, scientifically valid, 

and supports the intended purpose of the report? 

Comments on Charge Questions: 

There is no doubt that a conceptual model is needed for the ROE. That being said, a simple 

model is better than a complex model. My overriding concern, however, relates to the ROE as a 

communications tool about the ‘state of the environment’. The reader needs to know where we 

have come from, where we are today and where we need to be in the future. The reader of the 

ROE needs historical perspective. To know where you need to go you must know where you 

have been. The linkages between and among environmental components need to be clearly 

emphasized. The ROE must be more than simply a report card as to how we have responded to 

environmental regulations/legislation. Further, it is not sufficient to simply provide data in the 

ROE and leave it to the reader to interpret the data. The data/information in the ROE needs to be 

interpreted for the reader because the writers/contributors to the ROE have the necessary 

knowledge and understanding to provide the interpretations. One has a sense that the ‘Agency’ 

has lost its’ way and forgotten that it is called the Environmental Protection Agency with an 

emphasis on ‘protection’. Put another way, the Agency needs to go back to its’ roots with a solid, 

imaginative, proactive, forwarding-looking ‘Strategic Plan’. First Nations Peoples place great 

‘value on the land’ because they recognized long ago when you take care of the land, the land 

will take care of you. It is not too late for us to learn. The ROE can help with this. It is essential 

that adequate and timely support be provided to ROE staff and contributors to rethink and 

reshape the ROE. 

28
 
Comments from Dr. Amanda Rodewald 

Top three recommendations: 

1) Clarify purpose and users of ROE, including supplemental information (see comments 

below). In particular, clarifying the extent to which ROE will be used to evaluate 

progress towards meeting strategic goals/objectives and/or identify/assess current and 

emerging threats and vulnerabilities. 

2) Construct an overarching, but simple, conceptual model that illustrates how ROE fits 

within the agency’s decision-making framework (e.g., the pressure, state, impact, 

response model). 

3)	 Craft an executive summary that distills and synthesizes (not summaries) the indicators to 

address the questions. This synthesis requires some level of risk-taking to “take a stand”, 

so to say, rather than just summarize information to allow the user to interpret and draw 

conclusions. 

Charge Question 3. Logic and utility of EPA’s proposed use of supplemental information 

to answer questions in the next ROE. 

Supplemental information could seem to be used in three primary ways: 

(1) to fill gaps in indicators / strategic measures in order to better address the questions 

articulated in ROE, with the ultimate aim to evaluate the effectiveness of policies to avoid, 

reduce, or mitigate undesirable environmental and health outcomes. 

(2) to identify emerging vulnerabilities or threats that require attention in the future (i.e., 

pressures/states/impacts that are not currently recognized as having high importance; giving us a 

“heads up”, so to say). 

(3) to consolidate datasets or other information that may be useful to a broader set of users. 

Relative to the first function, additional info is needed to develop strategic measures to track 

progress of goals/objectives. To the greatest extent possible, indicators should parallel strategic 

measures. 

Relative to the second function, if part of the ROE’s purpose is to identify emerging 

problems/threats, then supplemental information is KEY because of the low likelihood that there 

already are widely used/robust monitoring efforts of environmental or health attributes not yet 

perceived to be threatened or vulnerable. 

Relative to the third, it might suffice simply to include a list (with links) of datasets within an 

Appendix. My supposition is that this third possible use has, at best, low priority. 

29
 
These two categories might be consistent with Dr. Anderson’s suggestions of leading vs. trailing 

indicators. 

Currently, the EPA does not include either non-representative data sets (regional, case studies), 

nor indicators that are not of national significance. The tension here is that many emerging 

threats / future threats are not necessarily of national significance or widespread yet. 

Some environmental / health problems or vulnerabilities that EPA should address are regional in 

nature. The focus on U.S.-Mexico border, Pacific Island territories, Great Lakes all illustrate the 

relevance of regional indicators. 

Another benefit of considering regional data is that one could tract or identify issues or threats 

that are obscured at national scales and only strongly affect certain regions or demographic 

groups (e.g., urban vs. rural, economic status [environmental justice], ethnic groups). 

Charge question 4. Recommendations for criteria to assure that supplemental information 

included in ROE is objective, free from bias, scientifically valid and supports the intended 

purpose of the report. 

I like the concept of an indicator “incubator” site, where individuals could comment on 

usefulness or shortcomings of various potential indicators. That said, the reality is likely to be 

that few users/readers of ROE will access the incubator site. Consequently, these 

future/potential indicators would be effectively invisible if not included in the report. At the 

least, the report should include information about which indicators are listed in the incubator site. 

Another possibility is that EPA could use an expert/Delphi process to select and/or evaluate 

supplemental information. 

One also could incorporate supplementary information with a simple explicit “coding” of the 

reliability in the indicator (e.g., BBS uses color codes of blue, yellow, and red to indicate relative 

reliability of tend data). This system provides a very quick, easy to read, approach to identifying 

shortcomings. 

OTHER CHARGE QUESTIONS 

1. Usefulness and logic of proposed conceptual modeling approaches for: 

a. Aligning ROE questions with Agency’s strategic goals and objectives 

I agree that this alignment is essential to make the ROE more useful and relevant for EPA and 

evaluating progress towards achieving Strategic Plan objectives. The ROE team suggest that the 

first three media chapters align directly with the first three goals (Clean Air & Global Climate 

Change; Clean and Safe Water; Land Preservation and Restoration, but a new chapter for 

Healthy Communities and Ecosystems; what about Compliance and Environmental 

Stewardship?). Taking it further, organizing ROE into chapters where each represents a different 

goal and the subsections the objectives (e.g., Healthier Outdoor Air, Protect the Ozone Layer) 

with subheadings representing the Strategic Sub-objectives and/or Strategic Measures would be 

30
 
conceptually attractive. However, given the fact that the Strategic Plan may change 

considerably, a complete reorganization of ROE may not be desirable now. At the least, ROE 

could include a table at the end of each goal chapter that lists the Objectives/Subobjectives and 

the results of the strategic measures. 

Regarding cases where there are important indicators for which there is no corresponding 

Strategic goal or objective, the EPA could (as they suggest) add a strategic goal/objective or 

include a section within each chapter that includes additional indicators of environmental 

condition. 

b.	 Communicating the intent and scope of questions in ROE. 

It seems that the last SAB Committee recommended that the models be process oriented, a 

recommendation with which I agree. However, the EPA explicitly states that the intent of the 

ROE is not to identify mechanisms / drivers of the indicators. I suggest that some recognition of 

underlying processes is necessary in order to identify the best strategies to remedy/avoid 

problems. 

c.	 Presenting underlying scientific foundation of questions in ROE 

Would seem to need some sort of process/mechanistic piece to do this. 

d.	 Providing framework for selecting indicators and identifying associated gaps, 

limitations, and useful supplemental information. 

A closer alignment facilitates the identification of information gaps. For example, there were 

many cases where the strategic measures identified in the paper for the next strategic plan (2009-

2009-2014) were not reflected in any of the current indicators. 

2.	 Recommendations concerning other possible approaches to conceptual model 

development that would be useful in identifying or highlighting important ROE topics, 

indicators for consideration, research or development. 

Perhaps using a Pressure-State-Response approach, as developed by Organization of Economic 

Cooperation and Development (OECD). 

The European Environmental Agency uses: Driving Force, Pressure-State-Impact-Response – 

 DF = underlying forces (e.g., basic sector trends, GDP) 

 P = human activities directly affecting environment 

 S = observable change of environment 

I = Effect of the change in environment 

 R = response of society to solve the problem 

In the case of the overarching conceptual model that integrates the ideas in the ROE and the tie 

to policy, one could tie together the ROE components with the following fused model: 

31
 
States 

Pressures 

Responses 

Impacts 

Pressures = human activities 

(emissions, chemical 

applications, etc.) 

States = air/water/land 

conditions (e.g., amt of 

chemicals or pollutants in the air 

or water) 

Impacts on ecological condition 

and human health 

Responses by Agency that 

influence pressures (e.g., 

policies, regulations). 

Or in a more fleshed out example, 

States of air, water, & land 

Concentrations of pollutants in air or 

water, acidity of lakes and streams, high 

and low flows, land cover, etc. 

Human activities 

(stressors, pressures) 

Emissions, applications of 

pesticides and fertilizers, land use 

changes, etc. 

Response of EPA 

Policies, regulations, 

adjustments to goals and 

priorities. 

Impacts on human health 

and ecological condition 

Rates of illness and disease, causespecific mortality rates, biodiversity, 

carbon sequestration, hypoxic zones, etc. 

This model also provides a nice framework to see where the indicators fit in the iterative process, 

something that helps the reader/user to conceptually “organize” the data and shows points where 

the Agency is lacking information. For example, related to indicators, the detection time lag 

increases through the model, with “impacts” being the most delayed. Therefore, if it became 

32
 
apparent that for some issues, all of the indicators fell on the impact end of things, that there 

would be large time lags between the human actions and the ability of EPA to detect the 

consequences. In such cases, EPA might expend more effect to develop indicators focused on 

Pressures, given that they might provide more time for EPA to address the problem before the 

unwanted environmental and/or health consequences are realized. 

Other miscellaneous issues 

1.	 Purpose: 

The EPA develops policies to regulate particular stressors precisely because they are thought to 

be drivers of, or contributors to, certain environmental and health outcomes. The ROE would 

seem to be examining indicators of environmental and health outcomes in order to evaluate the 

effectiveness of policies. Thus, the usefulness of the ROE rests upon an implicit, if not explicit, 

link between the stressors and outcomes. This also touches on the previous ROE committee’s 

recommendations that the models be more process oriented, better articulating relationships 

among stressors, media, health and environmental outcomes. Ultimately understanding the 

processes that underlie the outcomes is needed to identify the most effective policies. 

2.	 For ecological condition, they might want to more closely align both Strategic Plan and the 

ROE with the EPEC’s 2002 recommended framework for assessing and reporting on 

ecological condition. EPEC uses the following PSR framework: 

Goals/Objectives – Essential Ecological Attributes – Ecological Indicators (Endpoints) – 

Measures (monitoring data) 

The essential ecological attributes are: 

1.	 landscape condition 

2.	 biotic condition at ecosystem/community, species/population, and organism levels 

3.	 chemical and physical characteristics (water, air, soil, sediment) 

4.	 ecological processes 

5.	 hydrology & geomorphology 

6.	 natural disturbance regimes 

The ROE uses only part of this framework to articulate questions in the ecological condition 

chapter. 

3.	 Including “scorecards” of sorts that overview the indicators. A simple table that has the 

question followed by indicators (as in the table of contents) and the status (e.g., improving, 

worsening, no change) would be very helpful. Inclusion of this overview would facilitate 

use and interpretation of ROE. 

4. While I think that the popular and widely-used ecosystem services framework is useful in 

many contexts, I am concerned about structuring ROE around it, which would presumably 

involve recasting the ROE questions to identify the levels/qualities of the supporting, 

provisioning, regulating, and cultural services provided by the environment. Many of the 

33
 
 

 

specific questions and jurisdictional/regulatory mandates do not fit neatly within the services 

framework because the indicators frequently assess stressors/pressures, states, and impacts. 

Extending those metrics to services would involve quantifying the loss in services that follows 

those impacts. I prefer aligning ROE with strategic goals/objectives. 

Fig. 1 Constituents of
 

Ecosystem Services Human Well-being
 

Supporting 

• Carbon pool 

storages 

• Carbon 

sequestration 

Provisioning 

• food, fiber, fuel 

• water 

• Nutrient Regulating 

cycling climate, air & water 

• Habitat & 

refugia 

quality, flood control, 

disease & pest 

control, fire control 

• Primary 

production Cultural 

• Biodiversity Recreational, 

aesthetic, spiritual, 

educational, existence 

value, sense of place 

Security 

Personal safety, secure 

resource access, 

security from disasters 

Freedom of 

choice and 

action 

Opportunity 

to achieve 

Basic material for what an 

good life individual 

Livelihoods, nutrition, 

shelter, goods 

values doing 

and being 

Good social relations 

Social cohesion, 

mutual respect, 

ability to help others 

Modified from Millennium Ecosystem Assessment 

34
 
Comments from Dr. Mark Schwartz 

(Question #2). 

A. Paragraph on question #2. 

a.	 The conceptual model is incomplete as presented to us in the material for this 

meeting. The conceptual model presented to us is also somewhat ungratifying 

because it is very flat--- everything is connected to everything else at equal 

intensity. In thinking about this, I see the need for thinking about this modeling 

framework in three distinct ways. First, an overarching model (Dr. Rodewald’s 

model, the previous SAB panel suggestion, the Millennium Ecosystem 

Assessment sorts of approaches). This overarching model acts as a roadmap that 

can guide the reader throughout the document to understand where they are and 

why what is being covered matters. The second sort of model would be an 

organizational one for querying the report from different angles. I can imagine a 

user wanting to query the eROE on how policies and enabling laws have worked 

(has cap and trade worked); on strategic goals (how many are trending toward 

achievement?; on particular issues (is haze and visibility getting better or worse 

and where); or on outcomes (what are the causes of coastal water degradation and 

what is the status of what we know about that). These all provide opportunities for 

the ROE to allow a query to link to the indicators that address that issue. This 

would alleviate comments such as whether media and outcome chapters should 

remain separate or integrated, and why it is difficult to find some information. 

This requires careful thought to a three dimensional linkage model for this and 

how it gets programmed into a web access tool. The third kind of model is more 

like the stressor to outcome model presented to us. This has to be a component of 

the second model, but I distinguish it because it needs to make some arrows 

darker than others: not all linkages are equally important; not all linkages are 

under EPA jurisdiction. This kind of cause and effect model can help the reader 

understand the structure of the report and why it contains the information it does 

in the same way that the overarching model can. 

B. Other thoughts on the remaining questions 

a.	 (3). Supplemental indicators are critical for making this useful, particularly for 

prospective problems. The proposed use of them is good. It could much further, 

with additional resources. 

b.	 (4). Meta-analysis; contracted peer reviewed reviews of major issues. Take a look 

at the Annual Reviews of Ecology, Evolution and Systematics or Annual Reviews 

of Environment and Resources. I could see the EPA sponsoring such reviews for 

pathways in the overarching model that are not well captured by national 

indicators and then using this as an unbiased report on status and trends on that 

question. 

35
 
c.	 (1a). Aligning the ROE questions with the strategic goals. This is an outstanding 

idea. It is very important. This has to be accompanied with some rationale that the 

reader can get to that explains the strategic goals. Although this is not the ROE 

team purview per se, a goal is not strategic unless it is accompanied by a strategy. 

As it is written, there are no obvious strategies linked to strategic goals. It would 

be helpful to the reader to outline EPA’s strategies for achieving these goals. 

d.	 (1b). Linking the report to strategic goals and a conceptual model that draws the 

linkage from driving legislation through EPA policies all the way to human and 

ecosystem health outcomes is a way to define the scope of this report. This is 

critical because it is not intended to be a report on the entirety of the environment, 

and that can make that point very clear. It needs to be very clear. 

e.	 (1abcd). There is room for interpretation of the data. I strongly urge the ROE team 

to make the leap from simply reporting the data to interpreting the data for the 

reader. 

C. Other thoughts, in general. 

a.	 Terrific work. This is coming much further, much faster than I had imagined 

when I was on the panel in 2004. You have a huge task. You are definitely going 

in the right direction. 

b.	 I support the SAB to make a strong statement to the administrator on how this 

represents a terrific opportunity for the agency to shine, but that it needs more 

resources at its disposal to do the job justice. Probably at least double the staff and 

a dedicated web design person, maybe more. 

D. Three most important things 

a.	 Get the model frameworks right for readability and accessibility. ROE needs an 

overarching conceptual model that links the media (land, air, water) into an 

integrated look at ecosystem and human health and well-being outcomes. But, an 

electronic ROE also needs an operational model that allows one to approach the 

information from a variety of angles and get to the information quickly and 

efficiently. I describe this above in my thoughts on a modeling framework. 

b.	 Simple access tools to increase impact. Use simple color (or other) codes for how 

well the indicator actually captures problems such as: (a) the degree of uncertainty 

in the indicator trend; (b) the indicator trend; or (c) the degree to which an 

aggregate of trends indicate positive or negative change. Executive summaries 

and briefing sheets are good. Staggered rollouts on specific issues would help 

impact as well. 

36
 
c.	 Regional problems require regional treatment. Not all problems are national. This 

is particularly true of ecosystem problems. Recent studies have emerged showing 

an alarming increase in tree mortality in the west. This is likely climate change 

driven. We have long recognized increased tree mortality in the east driven by 

acid rain. We also have increased tree mortality through invasive species (e.g., 

gypsy moth) and urban island effects (high urban tree mortality). These impacts 

have different drivers, and hence different monitoring programs and hence 

different data sets. Together they represent something very important, but lacking 

EMAP, there might not be a good way to capture this with an indicator. Regional 

indicators would help alleviate this problem. The EPA may see its mandate as 

national in scope, but most readers want to know about the state of the 

environment as it relates to EPA and its state equivalents. They will not make the 

fine distinction between national, regional and state jurisdictions. Hence the report 

should make every effort not to distinguish either. 

37
 
 

Comments from Dr. Alan Steinman 

Question 3. Please comment on the logic and utility of EPA’s proposed use of supplemental 

information to answer questions in the next version of the ROE. 

Supplemental information can play an important, complementary role in helping inform EPA 

about the status of the environment in the US. I believe these data should be used in a 

discriminating way, especially in cases where national data are unavailable. 

One of the key questions will be how these data are used in the ROE. If the supplemental data 

are quarantined in some appendix, they will have little value and in which case, it is hard to 

recommend they be included given the limited resources available to produce the ROE. A much 

better approach (in my opinion) would be to either 1) include hyperlinks in the main body of the 

report to the relevant supplementary data set, which would render the physical location of these 

data moot, at least in the electronic version; or 2) integrate them in the main body of the text, 

clearly identified as supplementary, as they pertain to key questions. 

One possible approach would be to attach a metadata file (or table) with each supplementary data 

set that contains information on data source (i.e., agency, etc.), data rigor (i.e., have they been 

peer-reviewed and/or published in the scientific literature?), geographic region of significance, 

and EPA opinion on overall confidence level of data (low, moderate, high). We adopted a 

similar approach when assessing the quality of economic data we used to generate valuations of 

ecosystem services in west Michigan. See 

http://invest.wri.gvsu.edu/croplandfoodproductionvalues.html for an example of our approach. 

Question 4. Does the Committee have recommendations for criteria to assure that supplemental 

information included in the ROE is objective, free from bias, scientifically valid, and supports 

the intended purpose of the report? 

The standard approaches for ensuring data quality should be applied first: have the data been 

peer-reviewed and/or have they been published in the scientific literature? There should be a 

secondary level of screening, as I suspect many potentially useful data sets have not been peerreviewed, which would include assessment of data collection methodology, such as the presence 

or absence of a QAPP. 

General Comments on Conceptual Models: 

1) EPA may want to consider using conceptual models with similar architectures, especially 

when at the same hierarchical level. There is an intuitive appeal in this type of parallel 

construction, although I recognize there may be a need for different architectures for 

human health vs. environmental condition models. 

2) Two different general classes of conceptual models to consider include what I refer to as 

a) ecological process models and b) operational process models. General examples of 

each are given below: 

38
 
 

 

I. Ecological Process Model: 

II. Operational Process Model: 

Three critical issues for ROE: 

1) Greater integration across indicators and chapters. 

39
 
2) Addressing appropriate spatial scales for analysis (i.e. prior concern over use of EPA 

regions as the spatial scale of choice). 

3) Develop a “Citizen’s Guide” to the ROE that uses simplistic language, colorful graphics, 

and a transparent (but rigorous) approach to convey the key messages of the document. 

Final General Comment: 

This issue emerged at the very end of our meeting, but I raised it at the start of the meeting with 

Dr. Teichman (funny how we remember our own questions best) about dedicated resource 

allocation to the ROE effort. This was a concern we have expressed in the prior two ROE 

committees, and Dr. Teichman’s response was pithy (our staff is dedicated in multiple ways...) 

but lacked substance in terms of whether or not the allocation was sufficient. 

I believe it would be very instructive to have an analysis done on how many FTEs and resources 

(since much of the work on the ROE is contracted out) have been allocated to this effort, per 

fiscal year, since its inception. This should be relatively easy to track, and would provide us with 

a useful “indicator”. 

40
 
Comments from Dr. Helen Suh 

MEMORANDUM 

Charge Question: (1) Please comment on whether EPA’s proposed conceptual modeling 

approaches are logical and useful for presenting the underlying scientific foundation of 

questions in the ROE. (2) Provide a framework for selecting indicators and identifying 

associated gaps, limitations, and useful supplemental information. 

The proposed conceptual models are helpful and when finalized, should achieve their stated 

goals to (1) better support for the Agency’s planning, problem formulation, and decision making 

and (2) make the conceptual underpinnings of the questions and indicators clearer to the reader. 

It would be helpful to include an overall conceptual model that span all media and indicators and 

media-specific models in the introduction to supplement media specific models in the relevant 

chapters. The models could be improved further with the following: 

•	 A clear understanding and statement of the Report’s mission, objectives and intended 

audience; 

•	 Inclusion of boxes or arrows or some other indicators of “effect modifiers”, such as 

meteorology, time-activity patterns, or age that will influence movement from one 

column to the next (e.g., emissions to concentrations or concentrations to exposures); 

•	 Explicit linkages between the conceptual model and the relevant indicators, their 

limitations and gaps; 

•	 Illustration of how the indicator trends affect movement into the adjacent columns (e.g., 

contaminant source, concentration, exposure, etc.); 

•	 Inclusion of how climate change fits within the conceptual models; 

•	 An indication of priorities or most important aspects within each column of the 

conceptual model and explanation of these priorities (e.g., drinking water from XXX 

source is less important than that from a different source given the small percentage of 

people who drink water from this source; inhalation not as important as ingestion for 

some pollutants). 

In addition, after the Report mission is decided, it will be important to determine where the 

conceptual model fits into the organization of the report as soon as possible. For example, an 

overall conceptual model that spans all media would conceivably fit into the ROE introduction; 

however, it is not clear how the media-specific conceptual models would be integrated into the 

document. Would the media-specific models go into the beginning of each media-specific 

chapter? If so, would the health and exposure chapter refer back to the conceptual model in that 

Chapter? If not, would the model be placed into a conclusion or summary of the document? 

This quandary speaks to the need to reexamine the organization of the document and whether it 

makes sense to incorporate the Health and Exposure and Ecology chapters within the mediaspecific chapters, the possibility of moving to a one environment approach, and the division 

between the print and web-based platforms. 

Other Issues: 

41
 
In the preparation of the Report, the following issues or changes should be considered: 

•	 Greater linkage between indicators, trends and policy actions. 

•	 A defined place for discussions of emerging indicators and issues, perhaps in 

supplemental information but also in an executive summary, to show that EPA is aware 

and is beginning to monitor (or at least think about) these emerging or possibly important 

issues. 

•	 Substantial emphasis on a powerful and interactive web-based platform, with links to 

pertinent governmental websites; possible to de-emphasize the written document and 

replace with greater presence of web-based platform; may need additional resources to 

address this issue. 

42
 
Comments from Dr. Duc Vugia 

In the EPA’s Office of Research and Development draft of June 2009 “Issues for Initial 

Consultation with the SAB Advisory on EPA’s ROE,” the 2012 ROE is proposed to be 

restructured to align more visibly with the Agency’s strategic plan, to include conceptual models 

for each ROE question to make the relevance of each question clearer and more useful to 

readers, and to include “supplemental information” to provide additional insight into 

environmental issues, particularly those with indicator gaps and limitations. Here are my 

comments on whether EPA’s proposed conceptual modeling approaches are logical and useful 

for: 

1c. Presenting the underlying scientific foundation of questions in the ROE. 

Two examples of possible conceptual models were presented, one on the 2008 ROE outdoor air 

policy question and the other on the 2008 ROE drinking water question. I see these proposed 

conceptual modeling approaches as very logical and very useful to both present and enhance the 

understanding of the underlying scientific foundation of the ROE questions to readers. As 

readers of the ROE will more likely be non-scientific people, the level of details of the models 

should be easy to understand and to grasp, probably not much more detailed than the illustrated 

examples given. And as several of the questions may overlap regarding a conceptual model, it 

may be helpful to consider, for each medium such as air or water, starting with a more inclusive 

conceptual model for that particular medium addressing several or as many questions under that 

medium as possible, and then for some but not necessarily all questions, provide a slightly more 

detailed conceptual model particular to that question. An overarching conceptual model 

integrating all environmental media and the effects of their contamination on human, animal, and 

ecological health would be very helpful at the beginning of the ROE, before breaking out into 

each medium, to set the stage for readers on how components and concerns of the ROE fit into 

this overall understanding of environment and health. 

1d. Providing a framework for selecting indicators and identifying gaps, limitations, and useful 

supplemental information. 

The proposed conceptual models, such as the examples for the drinking water and outdoor air 

questions, would provide a useful common framework for scientists to identify and discuss 

information needs, gaps, and supplemental information. For non-scientists and policy makers, 

the proposed conceptual models provide a common framework to understand why there are such 

information needs and where the indicators and supplemental information fit within the 

framework. 

However, the strict criteria for selecting national indicators limit the number and variety of 

useful measures that can be selected, thereby leaving wider information gaps than really need be 

given the large amount of useful measures available in the current literature or in collected data. 

If the criteria for national indicators can be relaxed slightly to include similarly rigorous and 

scientifically reproducible data collected at some regional level, or at several but not all cities 

and states, then several more useful measures can be considered to give a clearer understanding 

43
 
of the current status and trend of several environmental questions, and their informational gaps 

would be decreased. If the national indicator criteria cannot be relaxed, then more supplemental 

information is needed to provide better understanding of status and trend on the questions (more 

comments on supplemental information below). 

Regarding the three most critical points that EPA should consider for the 2012 ROE, my current 

thoughts are: 

1.	 Consider adding, into the beginning of the ROE before breaking out into the five EPA 

areas of interest, an overarching conceptual model integrating all environmental media 

and questions showing how elements of the environment affect human, animal, and 

ecologic health. 

2.	 Consider adding a moderately substantive and informative Supplemental Information 

section to each of the five areas/media using defined criteria similar to but not as strict as 

the criteria used for national indicators. 

3.	 Consider expanding on Emerging Issues under each area/medium to acknowledge recent 

or emerging environmental concerns and to provide some information of ongoing 

research into these issues. 

All these points will make the 2012 EPA ROE more useful and informative to readers of all 

backgrounds, and it may help policy makers understand environmental issues better and be more 

open to requests for funding for emerging environmental issues or where information gaps exist. 

Substantive additions of Supplemental Information and Emerging Issues in the next ROE may 

also help influence the direction of some research. 

Regarding adding Supplemental Information, I fully support this as it will be an important part of 

the next ROE to address information gaps or lack of suitable national indicators for several 

questions. It shows readers that EPA is aware of and is willing to share or point out good data 

and information that help shed more light on some questions, particularly since there are not 

enough national indicators to show status and trend on all questions. Supplemental Information 

to be included in the next ROE should be selective, and the criteria to consider for selection can 

include many of the same criteria as for national indicators, particularly that the information is 

useful (it answers a question in the ROE), objective (accurate, unbiased manner), transparent and 

reproducible, characterized by sound collection methodologies, and comparable across time and 

space. In contrast to criteria for national indicators, supplemental information do not have to be 

available from all states across the nation, but can be from several states or cities or from 

regional areas. Additionally, data from multiple sites and published in peer-reviewed journals 

should also be considered; if meta-analyses have been done on such studies or multiple data sets, 

then those comprehensive review studies should be even better as they likely standardized 

criteria across studies to address an issue. Finally, potential national indicators may be identified 

from Supplemental Information, which then serves as an incubator for future useful national 

indicators. 

44
 
I look forward to the next conference call or meeting of this EPA SAB Advisory Committee on 

the 2012 ROE. 

45
 
 Comments from Dr. Judith Weis 

Question 1 c, d. Conceptual modeling for presenting underlying scientific foundation of 

questions and providing a framework for selecting indicators and identifying gaps, limitations 

etc. 

The conceptual models are a good framework for presenting the underlying science and thinking 

about the questions. I think they could be placed initially into the larger picture of “How are we 

doing?” Introductory material should discuss how unfettered pollution and environmental 

degradation in the 1960s led to the creation of EPA and passage of regulatory laws. We have 

been reducing emissions for ~40 years and the ROE is a look at how well these reductions in 

emissions has led to reductions in ambient concentrations of pollutants and to improved 

environment and human health. 

The links between reductions in releases/emissions of pollutants and the resulting improvements 

in environmental quality or human health are not always clear or “provable,” but, based on 

decades of scientific study we know they are linked. Whenever there are data that can be used to 

demonstrate this link, these should be emphasized (perhaps in supplemental information). A goal 

within the report (that may not be realized, but should be stated as a major need) should be to be 

able to answer the question “how much reduction in emissions or environmental concentration 

is needed to produce how much environmental improvement?” For example, “how much do we 

have to reduce nitrogen inputs to an estuary in order to reduce phytoplankton blooms and 

hypoxia by x amount?” These questions may point the way toward needed regulatory changes – 

it may be necessary to ratchet down the releases more than we have done thus far in order to 

achieve the desired environmental improvements – and should lead to changes in Total 

Maximum Daily Loads (TMDLs) or other standards (and perhaps, for the first time to regulation 

of non-point source runoff). 

46
 

Comments from Dr. Stephen Weisberg 

Question 1c. Are EPA’s proposed conceptual modeling approaches logical and useful for 

presenting the underlying scientific foundation for questions in the ROE? 

Only to a limited degree. The models properly communicate a basic underlying scientific 

foundation by connecting stressors with state and response. However, the models that were 

presented are media specific, providing little insight into how priorities were assigned across 

media or how the chapters are intertwined. There should be an overarching model that more 

holistically integrates across media. Perhaps the SAB’s recent endorsement of ecosystem 

valuation could provide that framework. 

It is also interesting that the chapter models are used differently in selecting the questions. For 

instance, the air chapter includes numerous questions and indicators about effectiveness of 

source control, but comparable questions/indicators are missing from the water chapter, even 

though the same issues are important and apparent in the conceptual model for that chapter. 

Question 1d. Are EPA’s proposed conceptual modeling approaches presenting a logical 

framework for selecting indicators and identifying associated gaps, limitations and supplemental 

information? 

Again, only partially. EPA’s conceptual model is based on ecologic processes, describing 

drivers, stressors, state, and response. That may be appropriate for selecting the best possible 

indicators from a scientific perspective, but that may not be the best model for meeting the needs 

of this document’s audience, for which scientists are secondary. The document seems targeted 

toward providing feedback to EPA’s management about their performance in protecting the 

environment. This is emphasized in linking the ROE questions with EPA’s strategic plan. In 

that case, an operational process model that focuses on linking management actions with desired 

environmental outcomes would be a more appropriate conceptual model. These two types of 

models are not mutually exclusive, as the operational model is hopefully informed by the 

ecological process model, but there are differences in emphasis which would lead the operational 

model toward more relevant questions and indicator needs. 

Top 3 priority suggestions: 

1)	 Audience: Develop a clear mission statement that identifies your client(s) and the 

decisions you hope to affect with the ROE. 

2) Linkage: Move past presenting a series of independent status indicators, instead 

presenting them in parallel with indicators that assess effectiveness of agency actions 

intended to affect that status. 

3) Communication: Prepare shorter summary documents that better communicate the core 

messages. 

47
 
Enclosure 3: EPA Charge Questions to the Committee 

Issues for Consultation 

Background 

Purpose and History of EPA’s Report on the Environment (ROE) 

In 2001 EPA initiated work to assemble an extensive set of environmental indicators in order to 

provide high quality information on the state of the environment. A goal of this effort was to 

compile and present indicator status and trend information that would enable EPA and the public 

to assess progress toward accomplishing EPA’s goals of cleaner air, purer water and better 

protected land. EPA presented these indicators in its 2003 Draft Report on the Environment 

Technical Document and its summary Draft Report on the Environment Highlights Document. 

Following Science Advisory Board (SAB) reviews of the 2003 draft ROE and a revised 2007 

draft of the report, EPA published its 2008 Report on the Environment. In 2008 EPA also 

published a shorter ROE document that highlighted national trends. EPA’s 2008 ROE provided 

indicator information to answer 23 “policy- relevant” questions that the Agency found to be 

critically important to its mission. Thus, although the 2008 ROE provided information on a 

broad range of indicators, the focus of the report shifted toward providing information that was 

relevant to EPA’s mission. 

In September 2008, EPA also released an online electronic Report on the Environment, or eROE. 

The eROE (www.epa.gov/roe) contains the current indicator data that are updated quarterly. 

SAB Recommendations to Improve the draft 2007 ROE 

In its peer review of the draft 2007 ROE, the SAB emphasized the value and importance of the 

report. The SAB strongly supported continued development of the ROE and provided 

recommendations to improve the report before its publication in 2008. The SAB also provided 

long-term recommendations to improve future versions of the ROE. In general, the SAB found 

that: 

•	 The ROE lacked a framework describing the scientific understanding of relationships 

between indicators and the basis for including them in the report. 

•	 The ROE presented status information to establish baselines for reporting future trends, but 

the lack of long-term trend information in the document precluded trend analysis for many 

indicators. 

48
 
•	 The rigid application of indicator selection criteria resulted in the exclusion of valuable and 

relevant information that could be used to further analyze trends. 

•	 The ROE was limited because it contained little data interpretation and no conclusions 

supported by statistical analysis. 

The current SAB consultation focuses on issues regarding the first three bullets. 

Scope of the ROE 2008 

The 2008 ROE was organized around five main chapters, “Air,” “Water,” “Land,” “Human 

Exposure and Health,” and “Ecological Condition.” Each chapter was organized around a set of 

critical “policy-relevant” questions that EPA wanted to answer with confidence in order to be 

adequately informed about important environmental trends. However, EPA stated that these 

questions could not necessarily be fully answered with indicators that met the Agency’s indicator 

definition and six indicator selection criteria in the report. In the ROE 2008 EPA defined an 

indicator as a numerical value derived from actual measurements of a stressor, state, or ambient 

condition, exposure, or human health or ecological condition over a specified geographic 

domain, whose trends over time represent or draw attention to underlying trends in the condition 

of the environment. The ROE 2008 did not include presentations of statistical confidence in the 

status of and trends in the indicators. When indicator trends were reported they were interpreted 

as the direction of change, and did not imply statistical significance. EPA recognized that 

uncertainty is an important issue and stated that it planned to quantify uncertainty in future 

versions of the ROE and its indicators. 

Future Direction of the ROE 

EPA intends to publish the next full edition of the ROE in 2012. Emphasis will be placed on 

presenting the status of and trends in environmental and human health conditions of interest to 

the EPA in order to inform the Agency’s planning and decision making. EPA intends to 

restructure the ROE to more directly align chapters of the report with EPA’s Strategic Plan goals, 

and to align the policy questions in the ROE with objectives in the Agency’s Strategic Plan. Any 

revision of the ROE in this regard must take into account the new Administration’s long-term 

perspectives on strategic planning. Although a draft of EPA’s new strategic plan for 2009-2014 

has not yet been released, the overall architecture of the Strategic Plan (with goals, objectives, 

and sub-objectives) is not likely to change. As stated above, EPA also plans to enhance indicator 

information in future versions of the ROE to include quantitative uncertainty information. 

Issues for the SAB Consultation 

Overarching Issues 

In its review of the draft 2007 ROE, the SAB recommended including conceptual frameworks in 

the report to illustrate scientific understanding of relationships between indicators and the basis 

for including them in the report. The SAB also recommended that EPA relax restrictive 

indicator selection criteria to enable the use of additional indicators that could inform the stated 

49
 
questions. EPA’s Office of Research and Development (ORD) is seeking early consultation with 

the SAB on conceptual models for restructuring and refining the next version of the ROE in 

order to better support Agency planning, problem formulation, and decision making and make 

the conceptual underpinnings of the questions and indicators clearer to the reader. ORD is also 

seeking consultation with the SAB on the proposed use of regional and sub-regional indicators 

and supplemental information to help answer ROE questions. 

ORD has developed an issue paper for the consultation describing how EPA proposes to 

restructure and refine the next version of ROE. The issue paper contains two examples of 

conceptual models to illustrate the scope of the questions and to select indicators. One example is 

a generalized conceptual model framing the 2008 ROE question, “What are the trends in the 

quality of drinking water and their effects on human health?” The other example is a conceptual 

model framing the 2008 ROE question, “What are the trends in outdoor air quality and their 

effects on human health and the environment?” Section 4 of the issue paper discusses EPA’s 

proposed use of supplemental information in the next version of the ROE. Specifically, ORD 

has requested consultation on the following issues. 

Specific Issues for Consultation 

1.	 Please comment on whether EPA’s proposed conceptual modeling approaches are logical 

and useful for: 

•	 Aligning the ROE questions with the Agency’s strategic goals and objectives. 

•	 Communicating the intent and scope of questions in the ROE. 

•	 Presenting the underlying scientific foundation of questions in the ROE. 

•	 Providing a framework for selecting indicators and identifying associated gaps, limitations, 

and useful supplemental information. 

2.	 Does the Committee have recommendations concerning other possible approaches to 

conceptual model development that would be useful in identifying or highlighting important 

ROE topics, indicators for consideration, research, or development? 

3.	 Please comment on the logic and utility of EPA’s proposed use of supplemental information 

to answer questions in the next version of the ROE. 

4.	 Does the Committee have recommendations for criteria to assure that supplemental 

information included in the ROE is objective, free from bias, scientifically valid, and 

supports intended purpose of the report? 

50