Board of Scientific Counselors

Management Review of the 

Particulate Matter Research

Program

Prepared for the Assistant Administrator

Office of Research and Development

U.S. Environmental Protection Agency

December 2000
Table o f Conten ts i

BOSC Management Review of the PM Research Program

Table of Contents

LIST OF ACRONYMS ...................................................................................................................... iii

EXECUTIVE SUMMARY ................................................................................................................... v

CHAPTER 1: REPORT OF THE INTEGRATION SUBCOMMITTEE AND

 RESPONSE TO THE CHARGE ........................................................................................ 1

Introduction .............................................................................................................................. 1

Overall Integration ................................................................................................................... 3

Risk Paradigm .......................................................................................................................... 5

Research Priorities ................................................................................................................... 6

Communication of Activities and Results ............................................................................... 7

Communication to the Broader Community ............................................................................ 7

Leadership ................................................................................................................................ 8

Management ............................................................................................................................. 8

CHAPTER 2: REPORT OF THE EXPOSURE SUBCOMMITTEE ............................................................ 11

Introduction ............................................................................................................................ 11

Evaluation of Response to Self-Study Questions .................................................................. 12

Conclusions and Recommendations ...................................................................................... 15

CHAPTER 3: REPORT OF THE ATMOSPHERIC SCIENCES SUBCOMMITTEE ..................................... 17

Introduction ............................................................................................................................ 17

Evaluation of Response to Self-Study Questions .................................................................. 18

Conclusions and Recommendations ...................................................................................... 21

CHAPTER 4: REPORT OF THE EPIDEMIOLOGY SUBCOMMITTEE .................................................... 25

Introduction ............................................................................................................................ 25

Evaluation of Response to Self-Study Questions .................................................................. 25

Conclusions and Recommendations ...................................................................................... 27

CHAPTER 5: REPORT OF THE TOXICOLOGY SUBCOMMITTEE ....................................................... 29

Introduction ............................................................................................................................ 29

Evaluation of Response to Self-Study Questions .................................................................. 29

Conclusions and Recommendations ...................................................................................... 31

CHAPTER 6: REPORT OF THE ASSESSMENT SUBCOMMITTEE ........................................................ 33

Introduction ............................................................................................................................ 33

Evaluation of Response to Self-Study Questions .................................................................. 34

Conclusions and Recommendations ...................................................................................... 43

References .............................................................................................................................. 45
ii Table o f Conten ts

BOSC Management Review of the PM Research Program

Table of Contents (Continued)

CHAPTER 7: REPORT OF THE RISK MANAGEMENT SUBCOMMITTEE ............................................ 47

Introduction ............................................................................................................................ 47

Findings and Recommendations ............................................................................................ 47

Evaluation of Response to Self-Study Questions .................................................................. 50

Conclusions and Recommendations ...................................................................................... 52

APPEND IX: RESPONSES TO SELF-STUDY QUESTIONS
List of Acronyms iii

BOSC Management Review of the PM Research Program

List of Acronyms

ACD

AD

ALD

APG

APM

AQC

AQCD

AQRS

ARIES

ATSDR

AWMA

BOSC

CAP

CASAC

CD

CDC

CENR

COPD

EBB

EPA

EPRI

FY

GPRA

HEI

HSD

IAG

LD

NAAQS

NAC

NARSTO

NCEA

NCER

NCERQA

NERL

NHEERL

NIAID

NIEHS

NIH

NPM

NRMRL

NRC

OAQPS

ORD

OAR

OMS

ORIA

Assistant Center Director

Associate Director

Assistant Laboratory Director

Annual Performance Goal

Annu al Performan ce Measu re

Air Quality Criteria 

Air Quality Criteria Document

Air Quality Research Subcommittee

Aerosol Research Inhalation Epidemiology Study

Agency for Toxic Sub stances and D isease Registry

Air and Waste Management A ssociation

Board of S cientific Coun selors

Conc entrated Amb ient Partic ulate

Clean Air Scientific Advisory Committee

Center Director

Centers for Disease Control and Prevention

Committee on Environment and Natural Resources

Chronic obstructive pulmonary disease

Epidemiology and Biomarker Branch

Environmental Protection Agency

Electric Power Research In stitute

Fiscal Year

Government Performance and Results Act

Health Effects Institute

Humans Studies Division

Interagency Agreement

Laboratory Director

National Ambient Air Quality Standards

National Academy of Sciences

North American Research Strategy for Tropospheric Ozone

National Center for Environmental Assessment

National Center for Environmental Research

National Center for Environmental Research and Quality Assurance

National Exposure Research Lab oratory

National Health and Environme ntal Effects Res earch Labo ratory

National Institute of Allergy and Infectious Disease

National Institute of Environmental Health Sciences

Nation al Institutes o f Health

National Program Manager

National Risk Mana gement Research Lab oratory

Nation al Resea rch Co uncil

Office of Air Quality Planning and Standards

Office of Research and Development

Office of Air and Radiation

Office of Mobile Sources

Office of Radiatio n and Indoor A ir
List of Acronyms (Continued)

iv List of Acronyms

BOSC Management Review of the PM Research Program

ORMA

OSP

PI

PM

RCT

RFA

RIVM

RTP

SAB

SBIR

STAR

Office of Resource Management and Administration

Office of Science Policy

Principal Investigator

Airborne Particulate Matter

Research Coordination Team

Request for Applications

National Institute of Public Health (The N etherlands)

Research T riangle Park

Science A dvisory Board

Small Business Innovation Research

Science to Achieve Results 
Executive Sum mary v

BOSC Management Review of the PM Research Program

Executive Summary

The Board of S cientific Counse lors (BO SC) met on October 2 8-29, 1 999, in Durh am, North Ca rolina, to

review the management of the Environmental Protection Agency’s (EPA) Office of Research and

Development (ORD) Particulate Matter Research Program. The review was based on a charge from the

ORD, a self-study document developed by EPA-ORD in response to the BOSC’s questions, and

interviews conducted by BOSC Ad H oc Subcommittees in Durham during the meeting. Ad Hoc

Subc ommittees on Inte gration, Exposu re, Atmosphe ric Scien ces, Ep idemio logy, Tox icology, Assessme nt,

and Risk Man agement consisted of thre e to four members, includ ing mem bers of th e BOSC an d its

consultants. In this review, the BOSC focused on the management, communication, and integration of

ORD ’s Particu late Matte r (PM) Researc h Prog ram. Sp ecific qu estions in cluded : 

1. Is ORD sufficiently coordinating research across categories of the risk assessment paradigm (source,

exposure , dose-response , assessment, an d manag ement)?

2. Is the structure in place for the Program to address the highest priority research needs?

3. Are re search activitie s and results being well commu nicated, both inte rnally an d exte rnally.

4. Is EPA -ORD providin g sufficien t scientific leadership within a nd ou tside the A gency on particulate

matter research? 

The final ch arge to the BO SC wa s to assess wheth er any change s are needed in the mana gement struc ture

of the Program.

The BOSC fo und that the ORD Pa rticula te Ma tter Re search Progr am is well organized and e fficient. It

should be considered a model for other res earch p rograms in the Agency. The recom mend ations in th is

report are largely “fine-tun ing,” aimed to improve the program ev en further.

ORD coordinates PM research in a matrix structure that cuts across its Laboratories/Centers as well as

components of the risk paradigm. There is a National Research Program Manager (NPM) and an

Executive Lead. The management structure is working well; however, the BOSC recommends that ORD

bolster the position o f the Na tional Pro gram M anager so that he/sh e can act e fficiently and decisively

when changes are needed in the Program. A dedicated budget for the Program Manager should be

available to provide resource sufficiency and flexibility for contingencies. For example, if there is a

particula te matter air p ollution e vent tha t provide s a rare opp ortunity for d ata gathering, the N PM s hould

have a budget to immediately respond to that opportunity. If it becomes apparent that a key study or

person is missing in the PM program, the NPM should be able to respond directly in consultation with the

Laboratory/Ce nter Directors. 

Coordination between ORD and the EPA Program Offices on particulate matter research is good. The

BOSC enco urages co ntinue d efforts b y EPA, especially th e ORD wo rking tog ether with the Offic e of Air

and Radiation (OAR), to implement the Presidential directive to expand the science of particulate matter

health effects and to improve air monitoring me thods and cost-effective mitigation strategies. 

ORD PM research priorities and, to some extent, the portfolio of projects are guided by the National

Research Council (NRC) in an arrangement between EPA and the NRC. Two reports have been

published including one on research progress and the portfolio (NRC, 1998), and one on priorities and

long-range researc h (NRC , 1999). The arrangement works well and bene fits the Agenc y. 
vi Executive Sum mary

BOSC Management Review of the PM Research Program

The BOSC recommends that ORD continually update its research priorities. Some assessment tools that

could aid in this endeavor are a gap analysis or sensitivity analysis to determine the efficacy of proposed

risk management efforts and public health benefits of the PM Program. The BOSC is aware that a revised

draft Air Quality Criteria Document (AQCD ) is to be published in early November, and that a revised

nationa l risk assessment will follow. Th e BOSC b elieves tha t such a p relimina ry national ris k analysis is

important, and encourages its speedy completion. Such an analysis can engage the public, give a clear

indication of the status of the PM Program, an d identify target research areas to reduce u ncertainties. 

Finally, the BOSC recommends that more long-range planning be performed to assess and compare the

public health benefits of the PM Program relative to other pollutants, mixtures of pollutants, or other

major public health priorities.

The BOSC concludes that ORD is doing a good job of coordinating its research activities and

communicating its findings both within ORD and to the broader community. However, greater use of the

World W ide Web might allow oth er scientists and the p ublic-at-large to track the pro gress of its research. 

ORD has contracted with the Health Effects Institute (HEI) to initiate this effort, and it should be linked

to the ma in EPA and ORD W eb sites. Close commun ication with Con gress, the p ublic, an d indu stry will

ensure continued success and support for the PM Research Program. Therefore, this remains a very high

priority for both ORD an d the A gency.

The sc ientific lead ership th at EPA -ORD has shown on PM re search is ex emplar y. Real stren gth is

apparent in its exposure modeling, atmospheric sciences, and toxicology research. The BOSC found that

expertise in ep idemiology is less we ll developed, alth ough it has improved with recent new hires. 

Continued collaboration, contracting, and leveraging with other agencies on epidemiology will be

necessary for the ne ar future. 

The BO SC recommends th at ORD strive to continuou sly examine an d improve its research produ ctivity. 

Better metrics are n eeded to de termine if EPA-ORD is makin g an impac t on the research c ommu nity. 

Gover nmen t Performance an d Resu lts Act (GPRA ) indices s hould be exam ined as w ell. Acco untab ility

of the overall program, sub-programs, and personnel is required.

The BOSC does not recommend any major changes in management structure. However, the success of

the PM Research Program depends on continued fund ing to complete program goals and objectives over

the next several years. Congress and the Executive Branch must be continuously updated on research

results as they are obtained and provided estimates of public health benefits and control costs for

implementing proposed PM ambient air quality standards. Hiring freezes and cyclical budgets are an

anathema to a q uality researc h progr am. Th e BOSC rec ommends th at EPA do everyth ing in its p ower to

ensure the funding that has been planned and requested for the PM research program, so that ORD can

continue to perform quality research that meets the national need in this high priority area.
Chapter 1: Report of the Integration Subcommittee and Response to the Charge 1

BOSC Management Review of the PM Research Program

Chapter 1

Report of the Integration Subcommittee 

and Response to the Charge

Subcom mittee Mem bers: Costel Denson, Ph.D. (Ch air)

Raym ond Loehr, Ph .D. (Vice-Chair)

Carol Henry, Ph .D. (Member)

Jerald Schnoor, Ph.D . (Member)

INTRODUCTION

The U .S. En vironmental Pro tection Agency (EPA) h as been directed to develo p and implement a grea tly

expanded, interagency research program on airborne particulate matter (PM). Program expansion

benefited from inclusion by Congress of additional funds for PM research above those requested by the

President for FY 1998 and FY 19 99. The funding by Congress has been contingent upon, and provided at

levels consistent with, recommendations prepared by the National Research Council (NRC) Committee

on Research Priorities for Airborne Particulate Matter. The scope and magnitude of the research program

activities reflect the importance of PM as a public health threat and the need to reduce uncertainties

through research. The priority uncertainties identified by NRC include those associated with estimating

actual human exposures to PM, identifying components of PM presenting hazard, elucidating

mechanisms of PM toxicity alone and in combination with other air pollutants, and characterizing

susceptibility, among others. The scope and magnitude of the PM Research Program also present

challenges to the EPA to develop and manage a large, complex and effective multi-laboratory/center and

multi-disciplinary research program. An Agency team, including Office of Research and Development

(ORD) and Office of Air and Radiation (OAR) staff, has managed the intramural program and

coordinated extensively with external research organizations. There is a need, given the importance of

this Program to the EPA, for the BOSC to provide review and advise the Agency on the management of

the PM Research Program.

This rep ort is a respo nse to ch arges from EPA’s ORD to con duct a rev iew of the current management of

the PM Research Program. The general charges include:

Conduct an evaluation of the ORD’s management approach to PM research including management

structure, vertical and horizontal communication and coordination, emphasizing how the ORD

Laboratories and Centers work together to plan and integrate their work; and

Prepar e recommend ations for improving PM Research P rogram m anagement.

The specific charges to the BOSC were:

1. Risk Paradigm. Are the ORD La boratories (i.e., NH EERL, NERL, NRM RL) and Centers (i.e.,

NCEA , NCER) sufficiently coordinating research across the risk paradigm (e.g., source-exposuredose-respon se and ma nagemen t)?

2. Research Priorities. Are the Laboratory/Center research programs integrated to add ress the highest

priority research needs?
2 Chapter 1: Report of the Integration Subcommittee and Response to the Charge

BOSC Management Review of the PM Research Program

3. Communication of Activities. Are the research activities and results sufficiently communicated

among the Laboratories and Centers, ensuring maximum leveraging and coordination of research

efforts?

4. Commun ication to th e Broad er Commun ity. Is there sufficient communication of results to the

broader scientific and regulatory communities?

5. Scientific Leadership. Is EPA-ORD providing sufficient scientific leadership within and outside the

Agen cy?

6. Management. Are changes to the management structure and/or processes needed? 

Two previous reports by the National Research Council (NRC 1998, NRC 1999) have evaluated EPA

research priorities, progress, and portfolio of projects. Tracking EPA’s implementation of its multi-year

research portfolio developed by the NRC and its PM mon itoring programs resides with the NRC and/or

with the Clea n Air Scien tific Advisory Committee (CA SAC) of the Agen cy’s Science A dvisory Board

(SAB). The B OSC effort focu ses on in ternal ORD management of the PM P rogram a nd is

complementary to these NRC and CASA C efforts. The BOSC examined man agement and integration of

the Program as structured by the risk paradigm, which cuts across Laboratory/Center cooperation,

prioritization, and implementation.

Charg es were co nveyed from Dr. Norin e Noon an, Assistant Administrator for ORD, to the BO SC in

January 1999. In preparation for the BOSC review of the management of the PM Research Program, the

BOSC provided a series of self-study questions to the PM National Research Program Manager (NPM)

and to the Directors of the National Risk Management Research Laboratory (NRMRL), National

Exposu re Research Laboratory (NERL), Na tional Health an d Environmental Effec ts Research Laboratory

(NHEERL), and the National Center for Environmental Assessment (NCEA). The BOSC requested

development of a self-study document addressing overall programmatic questions posed to Drs. John

Vand enberg , Nation al Program Manager, a nd William Farland, O RD E xecutiv e Lead; a nd spe cific

questions to the various Laboratory/Center Directors, organized into categories of the risk assessment

paradigm. EPA-ORD compiled the responses to those questions in a unified self-study document, as

provided in the Appendix to this report. Following receipt of the self-study report, the BOSC conducted

an onsite Programmatic Review of the OR D Partic ulate Matter Rese arch Pro gram in Durh am, North

Carolina, on October 28-29, 1999. This report is the summary of those findings and discussions of the

results at BOSC meetings on February 28-29 and May 30-31, 2000, in Washington, DC.

The report is organized into chapters that reflect the BOSC Ad Hoc Subcommittee assignments for the

management review of EPA-ORD’s Particulate Matter Research Program:

Chapter 1: Report of the Integration Subcommittee and Response to the Charge

Chapter 2: Report of the Exposure Subcommittee

Chapter 3: Report of the Atmospheric Sciences Subcommittee

Chapter 4: Report of the Epidemiology Subcommittee

Chapter 5: Report of the Toxicology Subcommittee

Chapter 6: Report of the Assessment Subcommittee

Chapter 7: Report of the Risk Management Subcommittee.

Major con clusions and recomme ndations are p rinted in bold -face type through out the report an d are

summarized in the Executive Summary and in the Conclusions and Recommend ations at the end of each

chapter.
Chapter 1: Report of the Integration Subcommittee and Response to the Charge 3

BOSC Management Review of the PM Research Program

OVERALL INTEGRATION

The fin dings o f the Integra tion Su bcommittee tha t cut across the six sp ecific char ges to the B OSC are in

this review are presented below. These are important to the successful integration of the overall PM

Research Program.

Management

Based on its programmatic review, the BOSC concludes that the EPA-ORD PM Research Program

is well organized and running efficiently. It is truly a model that EPA-ORD may wish to follow for

other programs. Examples of a good management structure include the appointment of a National

Program M anager and the matrix manageme nt of teams that cu t across Laboratory/Center boun daries. 

Research funds flow to elements of the matrix with high priority, and the structure is well understood by

manag ers in OR D. Fu rthermo re, the distrib ution of “ program-driven re search” v ersus “c ore researc h” is

appropriat e to me et the goals of th e Age ncy.

Still, the BOSC recognizes that matrix management has its virtues and its challenges. The PM Program

Manager has no lines of authority and has limited staff. The Program depends on strong working groups

and continuous coordination among managers. Contingencies and unexpected developments sometimes

require immediate action by the N ational Program Manag er to meet urgen t needs. The BOSC

recomm ends that ORD consider approving increased bud get authority to its Particulate Matter

National Pr ogram M anag er to ensure that he/sh e has th e flexibility and autho rity to m eet the g oals

and deliverables o f the PM Progra m. 

Integrated Agency Leadership

In a memo randum to EPA Admin istrator Carol Brow ner on July 1 6, 1997 , President Clinton said, “The

EPA, in partnership with other federal agencies, will develop a greatly expanded coordinated interagency

PM R esearch Prog ram. T he Pro gram will con tribute to expan ding th e science associa ted with

particulate matter health effects, as well as developing improved monitoring methods and cost-effective

mitigation strategies.” To meet this d irective, the Board of Scientific Cou nselors encou rages the entire

Agency to take responsibility for imp lementing th e Presidential direc tive. 

Coordination across programs requires integrating structures outside of the matrix. For example,

integration of results from the glo bal chan ge progr am and other multimed ia program s is necessa ry to

leverage resource s and optimize results for co-pollutan ts. Looking ou tside ORD to Agency regu latory

staff with expertise in PM issu es also is critical for the matrix manageme nt formulation to be successfu l. 

For example, OAR has con siderable exp ertise and controls th e air monitoring p erformed b y states. The

BOSC comm ends the cooperation between O AR and ORD o n the PM issue as a m odel for other

programs, and encourages the continuation and fostering of such cooperation in the PM Research

Program.

Considerable effort has been placed on developing the Air Quality Criteria Document (AQC D). Now that

it is nearly completed, it is important to look beyond the AQCD and to ensure the long-term success of

the PM Program. For overall risk reduction to occur in the United States, EPA needs greater emphasis on

mixtures of p ollutants and o ther compo nents of air qua lity, such as air toxics and ozone. In terms of

man agem ent, the BO SC reco mm ends m ore long -range p lanning and rese arch inv estmen ts to

ensure the integration of PM with other rese arch ne eds. For example, o zone remains a major pu blic

health priority; interactions between particulate matter and hazardous air pollutants (HAPs) also may

prove to be important. 
4 Chapter 1: Report of the Integration Subcommittee and Response to the Charge

BOSC Management Review of the PM Research Program

Metrics and Accountability

The National Research Council has played an important role in shaping the PM Research Program, but

EPA is responsible for assuring that the research is useful. Having a multi-year plan that was generated

from outside the Agency (NRC, 1999) by an organization with great stature, the National Academy of

Sciences, is useful and may serve as a model for other programs with major public health significance.

The BOSC questioned whether the Government Performance and Results Act (GPRA) should have been

discussed to a greater extent in the self-study document (see Appendix) and whether GPRA can provide

useful metrics or in dices for man aging the PM Program in the future. The BOSC recommends that the

GPRA format continue to be used in multiyear planning and other planning efforts, allowing gap

analyses to be p erformed , and for p urposes of m onitoring an d tracking pro gress. Accou ntability

measures n eed to be dev eloped for the p erformance o f sub-programs and person nel (Program Manag er,

Princip al Investiga tors, Bran ch Ch iefs, teams) involved in the PM Program. GPR A me asures sh ould

continually be in tegrated into the manageme nt plan for the PM Prog ram whe re possible. 

Other measures of performance should be tracked to determine if EPA-ORD PM research is making an

impact. D o other co untries u se ORD stud y results? A re EPA -ORD models, monito ring data , and resu lts

considered state-of-the-art by the scientific community? Are EPA-ORD and cooperator publications

being cited in Science Citations Index? What is the impact rating of the journals in which publications

are submitted? Metric s are ne eded to determ ine if EPA re search is making an impact; EPA-O RD is

in the best position to de velop these m etrics in consultation w ith other stakeholders. Produ ctivity

indices u sed in ac ademia such as p ublicatio ns per fu ll-time equ ivalent (FTE) an d impa ct factors may help

to determine progress as the Agency develops research expertise in new areas; the trend line as well as the

absolute magnitude are important. We realize that many measures are already being tracked by EPA

including major external award s and prestigious journal publications (Scienc e, Natu re, Cell, Atmo spheric

Environment, Journal of Geophysical Research, Environmental Science and Technology, Environmental

Health Perspectives, and oth ers). It wou ld be go od to make the re sults availab le to Princ iple

Investigators, so that they fully understand that good science is greatly appreciated and paramount to the

Agency. 

Now that ORD has its PM Research Program in place, the BOSC recom mend s that a “Ga p Analy sis”

or similar effort a t assessing the value o f inform ation be perform ed. This w ould rep resent a

structured ass essment of prioritie s based on a sensitivity analy sis of how in form ation g aps w ould

affect the efficacy of risk man agemen t and pub lic health benefits. The analysis should be updated

over time to refine and modify research priorities. Progress by an Agency can be charted as to how

problems are solved (risk and uncertainties reduced), and how new priorities emerge. A continuous

critical analysis of current hyp otheses shou ld incorporate th e broadest po ssible feedback to ensure

adequate exploration of alternative and competing hypotheses. In this regard, the BOSC recognizes that

Air Quality Criteria Documents are developed and national risk assessments are performed at regular

intervals. Such analyses are critical to possible programmatic changes in the direction of the PM

Program. Assessment efforts can engage the public, give a clear indication of the status of the PM

Program, and identify target research areas to reduce uncertainties.

Communications and Congressional Funding

ORD should make information dissemination a high priority, and Congress should be made fully aware at

frequent intervals of the progress of its PM Program.

 

We rec ognize th at, in lieu of a Cong ressional re allocation of Scien ce and T echno logy (ST) monie s, it is

difficult for EPA-ORD to alter components of its research funding. Improved flexibility in funding

allocation s is neede d. The end of th e hiring fre eze in 20 00 wa s a welcom e develo pmen t, so that po st-
Chapter 1: Report of the Integration Subcommittee and Response to the Charge 5

BOSC Management Review of the PM Research Program

doctoral positions could be allocated within ORD to fill knowledge gaps. However, the BOSC

encourages EPA to continue to pursue development of a long-range research plan and assist the Agency

in maintaining support for the PM Program. Increasing Congressional line items represent a threat to the

long-term stability of the Program and its integration. The BOSC recommends that EPA and ORD do

everything possible to ensure flexibility of fu nding allocation s, so that the Agen cy can be m ost

responsive to na tional needs.

If the Congress, the public, an d leaders of ind ustry are to make info rmed dec isions, EPA needs to

transm it good scientific and te chnica l information routinely , and in a ma nner th at is easily

understandable . This is not just an ORD responsibility, but a resp onsibility of the entire Agency. 

Innovative ways to disseminate information need to be considered. World Wide Web sites, such as the

PM Research Inventory that is being developed by the Health Effects Institute (HEI) for ORD, is a good

metho d for trackin g of inform ation an d dissemination to the pub lic. It would make se nse to ha ve this site

hyperlinked w ith EPA’s Web Site. 

Following are responses to the six specific charge questions on the Risk Paradigm, Research Priorities,

Communication of Activities, Communication to the Broader Community, Scientific Leadership, and

Man agement.

RISK PARADIGM

A brief description of the coordination along the risk paradigm follows. ORD develops information

across the entire risk paradigm. In turn, NCEA uses extramural research developed largely through the

STAR p rogram in the National Center for Environmental Research (NCER), and research information

developed through EPA intramural activities, primarily from N RMRL (for source ch aracterization data),

NERL (for exp osure an d atmos pheric sc iences d ata), and NHE ERL (for health effects info rmation ) to

produce the AQCD. The document then is turned over to its client, the Office of Air Quality Planning

and Standards (OAQP S) in OAR. NCEA evaluates the world’s literature in a particular subject area, such

as PM, in the AQCD and prepares a research needs report. In the preparation of these documents, NCEA

plays an integrative role in drawing together research findings from NRM RL, NERL, and NHEERL, as

well as results from the STAR program and other extramural research (e.g., research conducted by Dutch

scientists), and in the future, the five STAR PM research centers. Subsequent to the development of the

AQCD, NCEA also plays a pivotal role in supporting OAR in the development the OAR staff paper, and

in executive level and external briefings on risk in support of regulatory decision making. Given the key

position of NCEA in the risk assessment process, coordination and integration across EPA Laboratories

and Offices is critical to success. NCEA’s coordination activities and mechanisms appear to be strong.

We are very positive about the overall effort, in particular the long-term planning process for

research and research integration. This has resulted in funding the key areas, both (1) fundamental

new science (heavily weighted to the extramural program), and (2) integrating, synthesizing, and building

consensu s on existing scien ce and state-of-the-art scienc e (primarily intramu ral).

Risk ma nagem ent alternatives currently a re receiving too little focus. Although costs need not

become part of the risk management process, risk-risk tradeoffs are very relevant and should be given a

larger role in both setting research priorities and in thinking about risk management decisions to attain the

PM N ational Ambient Air Quality Standard s (NAA QS). 

The PM research effort is well coordinated across the risk paradigm. From an organizational

perspective, it is clear that a stron g effort has been made to en sure the Lab oratories are communicating. 

Over dependence upon the paradigm in shaping the research should be avoided, because the information

needs of the public health and regulatory communities are extensive. The broad public health importance
6 Chapter 1: Report of the Integration Subcommittee and Response to the Charge

BOSC Management Review of the PM Research Program

of air pollution should not be obscured by division of the science into the compartments of the risk

paradigm.

Within the indiv idual rea lms of ha zard ide ntification and ex posure assessme nt, the pr ograms are well

integrated . In the futu re, cross-inte gration b etween these two realms sh ould b e given a higher p riority

(e.g., does-respon se assessment in both animals and humans).

Animal toxicology and human exposure projects are well coordinated, and clearly are facilitated by

the geograp hical proxim ity of the anim al and hu man testing Labora tories; both whole animal and in

vitro studies are bein g used effectively to gu ide design an d interpretation of in vivo animal and human

studies. The toxicology program is using relevant PM materials, obtained directly from demonstrated

sources, a nd the P M materials are b eing use d to exp lore and define re levant h ypothese s that relate to

possible human health effects. ORD needs to further integrate its epidemiology and toxicology staff as

intend ed. OR D cou ld proce ed in a small way dow n this pa th now with ex ploratory stu dies (e.g., tryin g to

develop exposu re mod els for hypo thesized toxic con stituents). S uch join t explora tory studies would

foster commun ication an d integra tion in the future. Current p lannin g has ex posure modelin g efforts

lagging d ata collection by a few yea rs. Cou ld they be more syn chrono us, and could th e mode ling efforts

and documentation be published and made available earlier? We encourage ORD to explore the use of

pilot data for model development and testing.

It is very important that ORD is starting to work with OAQPS on the Supersites program. There is good

coordination with regard to health effects; however, with regard to atmospheric characterization,

mode ling, and monito ring, this co ordinatio n has n ot been very succe ssful in th e past. It is enc ouragin g to

see this improving for PM.

There is very good integration along the risk paradigm, especially at the management level and somewhat

at the grassroots level, but primarily at the nearest neighbor level. Geographic proximity of the

Laboratories in Research Triangle Park (RTP), NC, helps ORD integrate across the risk paradigm, which

is very helpful. For those ORD PM researchers who are not proximate (e.g., those located in Reston,

VA), this may be a problem. It is unclear if integration along the risk paradigm happens at the

investigator level. Postd octoral rotations migh t help with this. ORD sh ould initiate an introductory

progra m fo r postdo ctoral scientists to gain exp osure at o ther key lo cations w ithin OR D. Through

visits, seminars, or short rotations, they could gain an appreciation and understanding of all parts of the

risk parad igm. Th is would be in ad dition to th e curren t good p ractice of h aving p ostdocto ral scientists

spend time in co operative agreement labo ratories tha t correspo nd to their own s pecializatio ns. Th is

would give postdoctoral scientists both personal and p rofessional links and increase integration across

ORD.

RESEARCH PRIORITIES

The Labo ratories have heeded the direction of the NRC Reports and research directions are

consistent with the guidance of the Academy. Research priorities have not been generated by any

formal integrative process that directly identifies value of information. The priorities currently under way

in risk management are intuitive. Although they appear sound, adjustments to priorities as initial research

results emerge will b e more d ifficult to ju stify withou t a more fo rmal valu e-of-information asse ssmen t.

In the past, major data collection efforts have not been followed with appropriate level efforts to use and

analyze the data. The example given of setting aside $1.6 m illion to analyze the $1 5-20 million e ffort

from the Su persites program shows a con tinuing prob lem (a 1:10 ratio is too small). Mor e emp hasis is

needed on compilation and analysis of Supersites and monitoring data. 
Chapter 1: Report of the Integration Subcommittee and Response to the Charge 7

BOSC Management Review of the PM Research Program

Incentives for PIs to engage in valuab le research are needed, as well as formal metrics to determine when

a particu lar line of rese arch ha s produ ced info rmation that indic ates “mo re research is not nec essary” to

improve Agenc y and state risk management decisions. In addition, the BO SC notes that there is a

potential tension between promotion based primarily on peer-reviewed publications and having a research

program that is flexible and responsive to changes in research p riorities.

COMMUNICATION OF ACTIVITIES AND RESULTS

Coordination at high levels of the management hierarchy appears to be strong and well structured.

Excellent communication networks exist among the Laboratories and Centers for keeping abreast of

research results; however, optimum leverage is compromised by fiscal constraints. There is good

coordination with OAR/OAQ PS to address regulatory needs. This appears to be a lesson learned from

the ozone experience.

Commun ication within ORD among the Lab oratories an d Cen ters occurs throug h an ov erall matrix

structure and team buildin g. The matrix organization a nd teams rep resent an imp ressive attempt to en sure

communication.

There should be mor e interactio n with th e other p rograms; for example, the G lobal Change program will

have similar issues (e.g., measurement techniques, impacts, trends, etc.), and more interaction is needed

with the Risk Assessmen t Forum on me thods for risk and uncertainty analysis.

The extent of staff level coordination is unclear. We can not determin e if bench-level scien tists are

knowledgeable of the coordination and integration that are occurring at higher levels. Concerns about

this were validated in the previous NERL management review by the BOSC in 1998. In an organization

the size of ORD, it must be recognized that investigator communication will be a continual challenge,

particularly given the large amount of extramural work.

COMMUNICATION TO THE BROADER COMMUNITY

Impressive efforts are under way to communicate with the scientific community. Through workshop

development an d spon sorship, the broa der scien tific community is regu larly engag ed in sh aring resu lts

and b uilding consen sus for fu ture resear ch direc tions and hypothe sis genera tion. Commu nication in

workshop s and researc h mee tings is an importan t comp onen t of coordina ting with the outsid e commun ity,

and ORD is to be commend ed for the worksh ops it has organize d and for encou raging its sc ientists to

participate in these meetings. These workshops have been successful in reaching the research

community. Peer review in these forums is an important component. Coordination (e.g., cosponsoring)

of conferences with professional societies might increase the visibility, effectiveness, and regularity of

this form of communication , while decreasin g costs. Emphasis should be placed o n sha ring basic

science work, rath er than prog ram ov erviews.

ORD believes th at it mainta ins a good balance of interest ac ross num erous stak eholde rs involve d with its

work. The ener gy industry and the automo tive sectors a re represe ntative of th e industrial sectors with

which ORD interacts. O RD n otes that it also interacts with the American Lung Associa tion, pu blic health

groups, the N ational Gove rnors Associa tion, and ind ividual states. The e xtent o f the interaction is

unclear, in particular, with the public health research community. The importance of ensuring that

input is balanced across the various stakeholder groups cannot be overstated.

Web Sites. With regard to Web sites, the key will be their maintenance—keeping them updated and ever

green. There should be some ongoing evaluation of the usefulness of the Web site, in addition to the

statistics that are collected on an ongoing b asis.
8 Chapter 1: Report of the Integration Subcommittee and Response to the Charge

BOSC Management Review of the PM Research Program

Commun ication of S cience to th e Regu latory Commun ity. EPA should consider emulating NIH’s

practice o f regularly commu nicating scientific p rogress. O ther fede ral agenc ies, such as NIH, actively

communicate their scientific progress in addressing regulatory issues to the general public, to the broader

regulatory community, and to Congress. OR D also shou ld consider inc reasing its efforts in this regard

vis-a-vis EP A He adqu arters. Little h as been said to demonstr ate the inc lusion of the regu lated commun ity

and state and local agencies in the research planning process. This is where the translation of research

results is most critical. 

ORD needs external feedback on the quality of its endeavors, in general and on specific programs such as

the PM Program. One form of external feedback is the receptiveness of peer-reviewed journals and

research ers to OR D resea rch as sh own b y the form o f public ation of th at research and citatio n of it.

LEADERSHIP

EPA clearly has played a leadership role in the area of particle research, particularly through

capacity building throughout the academic community. ORD has don e a good job integ rating with

and serving in a leadership ro le for other research org anizations. ORD leadership on the CENR,

especially the Air Quality Research Subcommittee (AQRS), is laudable, and should be continued.

ORD toxicology investigators are well recognized in the scientific community for their leadership on PM

toxicology issues. In toxic ology , ORD appears to be providing scien tific lead ership both n ationa lly

and internationally.

The fact that NCEA is helping to expedite peer reviewed publications in leading scientific journals and

that it is holding a major PM conference in conjunction with the Air and Waste Management Association

(AWMA)—on e of the larg est professional socie ties in the air quality field—attests to ORD’s lead ership

role in th e profe ssiona l comm unity.

There is a concern that ORD is not able to recruit sufficient numbers of young scientists, thereby

threatening its ability to sustain scientific leadership. ORD should explore ways of acquiring the

resources and means of attracting young, talented staff to augment ORD’s current staff. Insufficient

information was provided to the BOSC regarding the scientific leadership provided by the professional

staff of the Laborato ries and Cen ters to fully assess this issue. However, it is clear that man y of them are

active and hold leadership roles in the professional communities. In the exposure area, ORD has made

some exce llent recent hires, thou gh it is short senior staff in som e critical areas, such as ex posure

modeling. To ensure that ORD’s productivity in leading the science in this area, and given the recent

infusion of postdoctoral scientists, it is important that ORD scientists have appropriate staff support. The

NCE A autho rs of chapters in the AQC D appe ar to be leading a uthorities with go od pub lication records. 

In other a reas, such as epidem iology, ORD is not leading . For this area in particular, it is unclear

whether ORD should develop its own expertise or cooperate with and rely more on other federal agencies

that have this expertise, such as the Centers for Disease Control and Prevention (CDC). Some basic level

of expertise in ep idemiology is need ed within EPA ORD, so th at scientists in toxicology, ex posure

characterization, and risk assessment have close collaborators with whom to interact across the risk

paradigm.

MANAGEMENT

The PM m anag ement structure is exem plary. ORD should stay the course and continua lly evalua te

its results. The overall framework for management that is emerging includes a National Program

Manager, matrix management, team approach, committee coordination horizontally, and extensive use of

peer review (NRC, CASAC PM Subcommittee, extensive peer reviewed extramural program, and

encouragement of pee r-reviewed pu blications), but it is un clear how it is work ing at the lower leve ls. 
Chapter 1: Report of the Integration Subcommittee and Response to the Charge 9

BOSC Management Review of the PM Research Program

Matrix management does facilitate adaptation to changing needs. For instance, the PM Program

management recognized the need to link the ambient air quality monitoring programs with the research

needs. 

Contin uity in the cu rrent high ly compe tent and suppo rtive prog ram ma nagement shou ld be a h igh priority

planning and development activity. FTE limitations and hiring freezes have impeded the development of

capacity in key areas. This is most apparent in epidemiology, although there appear to be staff limitations

in other a reas as we ll. There clearly is a critical need for additional resources to meet the scope of

ORD ’s activities. Mechanisms to fill these resource gaps are needed.

ORD manag ement s hould be enco uraged to reque st ST monies to en able effec tive and timely respo nses to

desirable alterations in components of the ongoing research period, i.e., improved flexibility in funding

allocations. Encourage establishment of a contingency resource to rapidly respond to events that have

direct and immediate implications for the impact of particle pollution on human health (e.g., forest fires

and conc erns of relationship s of wood smoke to asthma ).

There is a need for better public outreach. For example, the advocacy community, local officials, and the

general public need to be creatively engaged in shap ing priorities and enhancing the translation of results.
10 Chapter 1: Report of the Integration Subcommittee and Response to the Charge

BOSC Management Review of the PM Research Program
Chapter 2: Report of the Exposure Subcommittee 11

BOSC Management Review of the PM Research Program

Chapter 2

Report of the Exposure Subcommittee 

Subcom mittee Mem bers: Jerald L. Schnoor, Ph .D. (Chair)

Ann Bostrom , Ph.D. (Vice-Chair)

Jana Beth M ilford, Ph.D. (Memb er)

Nancy K. K im, Ph.D. (Mem ber)

INTRODUCTION

The goal of the BOSC PM Research Program review was to evaluate the management of the PM Program

rather than the sc ientific directions of the p rogram. Th e following fou r questions were presented to Dr.

Gary Fo ley, the Dire ctor of the N ERL, by the BOSC, with the request th at he add ress them with rega rd to

exposure activities across ORD:

Describe how indoor exposures are addressed by the PM exposure research.

Discuss how the exposure efforts relate to (are integrated with) toxicological and epidemiological

efforts.

Discuss how O RD is developing the fu ndamental tools, and staff expertise, needed to assess

exposure.

Discuss, with examp les, how ORD is addressing the exposure qu estions included in the NRC reports.

The self-study report addressed these four questions. The BOSC Subcommittee met with members of the

EPA-ORD team in Durham, NC, on October 29, 1999. The Subcommittee was briefed by Dr. John

Vandenb erg, National Research Program Manager for Particulate Matter. The B OSC S ubcommittee also

met with members of the Exposure team including: Dr. Gary Foley, NERL Director; Dr. Judith Graham,

NERL Associate D irector for Health; D r. Ross High smith, NERL Bran ch Chief; D r. Roy Zweid inger,

NERL Branch Chief; Dr. H aluk Ozka ynak, NERL Scien ce Team Leader for Ex posure M odeling; and Dr.

Linda Sheldon , NERL S cience Team Lead er for Exposure Measurements.

The ORD human exposure program (conducted predominantly within NERL and the National Center for

Environmental Research [NCER], but with ties and inputs from the National Health and Environmental

Effects Resea rch Laborato ry [NHEERL], N ational Risk M anageme nt Research Laboratory [NR MRL],

and National Center for Environmental Assessment [NCEA]) has two major goals: addressing the

exposure questions of the NRC as described in Research Topics 1 and 2, and providing exposure-related

expertise to NHEERL in its conduct of health studies, especially as related to NRC Research Topics 5 and

8. The goal of Research T opics 1 a nd 2 co llectively is to un derstand the exp osure of su sceptible

subpo pulation s to particu late matter (P M) an d co-occu rring gase s of amb ient origin that are likely to

cause adverse health effects. The primary distinction between the two topics is that Topic 2 focuses on

toxic PM species. Achieving this thus requires iterative research within “exposure” and between

“expo sure” an d “hea lth.” Methods, measu remen ts, and models are n eeded , as are inp uts from health

scientists to identify those PM components that may be most toxic. The goals of Research Topics 5 and 8

are to characterize hazardous PM com ponents and to characterize susceptible subp opulations,

respectively. NERL exposure analysis scientists collaborate in these health studies by providing the

exposure portion of NHEERL epidemiology studies, characterization of PM before and after the

concentration used in NHEERL human clinical and animal toxicological studies, characterization of
12 Chapter 2: Report of the Exposure Subcommittee

BOSC Management Review of the PM Research Program

particles used in NHEERL mechanism studies, etc. The STAR p rogram also forges such

interrelationships, esp ecially through the P M Cen ter grants. 

Integration of ORD’s PM ex posure research a cross the ris k parad igm is esse ntial for ORD to be able to

provide national leadership on particulate matter research. Coordination among other offices within EPA,

extramural resear chers, an d other g overnment researc h agen cies is crucia l for obtain ing the b est science in

the most cost-effective manner. Integration across the risk paradigm is illustrated by the match between

projects in NERL (e.g., as specified in sections 2, 4, 5, and 7 of the NERL research plan—see self-study

page 37) an d the comp onents spec ified by NRC in its 199 8 and 19 99 reports. 

ORD’s human exposure sciences research is fully coordinated with like research being conducted by

outside public and private organizations, both national and international. Formally, this coordination

occurs through organizations such as the North American Research Strategy for Tropospheric Ozone

(NARST O) and the Air Quality Subcommittee (AQSC) of the Committee on Environment and N atural

Resources (CENR); ORD scientists and managers play lead roles in both of these organizations. A new

standing work group of AQSC on PM Research Coordination is being formed with ORD managers as

Executive Co-Chair and lead Executive Staff. Informally, exposure research is coordinated through the

Coordinating Committee for PM Human Exposure Research and through the NERL PM Research

Coordination Team. The Coordinating Committee for PM Research has members from NERL,

NHEERL, and academic institutions performing panel studies funded by NERL, ORD’s STAR program,

the Health Effects Institute (HEI), the Electric Power Research Institute (EPRI), the American Petroleum

Institute, and the Mickey Leland National Urban Air Toxics Research Center. The ORD PM Research

Coordin ating Committee has formal connection s to NRM RL, NH EERL, and NCEA . 

ORD leadership on the CENR, especially AQRS, is laudable, and should be continued. ORD

development of the Federal Reference Method for ambient air quality monitoring of PM2.5 also is

important. Additionally, we encourage ORD to continue its efforts to bring together the

atmospheric science and health effects community for particulate matter exposure research.

It has been suggested that NARSTO be expanded to provide a public/private partnership that integrates

the science on the exposure side. This appears to be in the works, but coordinated by HEI, not NARSTO.

ORD should evaluate the choice of coordinating body carefully, with attention to whether HEI has

the prop er institutiona l links to make this ha ppen effectively. 

This review focused on the written and verbal responses to the self-study questions submitted by the

BOSC to EP A; the Subcommittee’s review of the responses is described below. Specific Subcommittee

recommendations are identified by bold type in the text of the review.

EVALUATION OF RESPONSE TO SELF-STUDY QUESTIONS

1. Describe how indoor exposures are addressed by the PM exposure research.

In terms of managing its research portfolio and answering regulatory questions associated with the

proposed PM2.5 rule, OR D’s cu rrent focu s on ambient air q uality is good . However, the S ubcom mittee is

left with some concerns regarding several aspects of indoor exposure research, including: (a) the lack of

emphasis on indoor air source characterization, which will be necessary to support research on statistical

and mass balance mixing models to understand the contribution of ambient air versus indoor air; (b)

stronger focus on chronic obstructive pulmonary disease (COPD) than other participants in the panel

studies, which could lead to critical missed opportunities on other sensitive populations (these studies do

include participants with COPD, chronic heart disease, and asthma as well as healthy elderly people and

minoritie s with low income s); and (c) the lack of expertise in assessin g human activities. This is
Chapter 2: Report of the Exposure Subcommittee 13

BOSC Management Review of the PM Research Program

especially true considering the deletion of a budget for indoor air quality last year. The BOSC

unde rstands th e need to move towards m ore exp osure-relate d que stions on indoor a ir, and we appre ciate

that there has been an expansion of activity in this regard, but we are concerned that previous areas of

indoor air research may be forgotten or under appreciated.

Ther e appears to b e minimal s ocial science exp ertise in O RD. Activity studies o f susceptible

subpopulations such as children and elderly people require such expertise; the survey and research

instruments being used have been borrowed from previous studies, and it is not clear to what extent

they are receiving adequate peer review. OR D should consider strengthening its social science

expertise. 

In the three Coop studies described in the response to the self-study questions, the majority of unhealthy

participants selected are specified as COPD . There is a need to link with researchers and projects that

cover non-COPD endpo ints, to avoid missing opportunities. There is a link to asthma in the InnerCity Asthma stud y, and there are signs that ORD is moving in this direction, but a more concerted

effort is needed. 

2. Discuss how the exposure efforts relate to (are integrated with) toxicological and

epidemiological efforts.

The Atlanta pilot study shows the promise of linking monitoring programs with exposure studies. The

Aerosol Research Inhalation Epidemiology Study (ARIES) in A tlanta, which in volved Harvard, Emory

Univ ersity, an d othe rs, rece ived fu ndin g from multip le sour ces (including the U.S. Dep artment of Energy,

Electric Power Research Institute, American Automobile Manufacturers Association, American

Petroleum Institute, and various utility companies) and is credited with contributing to Atlanta being

designa ted one of the first EPA Supersites. The ob jective of A RIES was to inv estigate the relationsh ip

betwee n air qu ality and h uman health with epidemiologic al and ex posure studies, an d to prod uce resu lts

in 2000 , in time for the revision s to the Nationa l Ambien t Air Quality Stan dards (NA AQS ) for PM. ORD

should continue and expand linkages of exposure studies with existing monitoring programs

(Supersites, chemical speciation , and the Science to A chieve Results [STA R] prog rams). Co-location

of various kinds of research at large site studies is a cost-effective strategy to get the best results for the

dollars invested. 

ORD is trying to integrate its epidemiological and tox icological research, b ut most of the d eliverables are

slated for comp letion and pu blication 3-5 years in th e future. Perhaps ORD c ould p roceed in a sm all

way down this path now with exploratory studies (e.g., trying to develop exposure models for

hypo thesized tox ic constituen ts). These studies would foster communicatio n and speed in tegration in

the future.

ORD coordination with the Office of Air Quality Planning and Standards (OAQPS) already has proved

successful in health effects studies. It is essential that the Supersites program be similarly coordinated. On

atmospheric pollutant characterization, modeling, and monitoring for other issues such coordination has

not bee n very suc cessful in the past. It is en couragin g to see this im proving in the case of particu late

matter.

3. Discuss how ORD is developing the fundamental tools and staff expertise, needed to

assess exposure.

The BOSC senses that th ere is greater sta ff satisfa ction as a r esult of be ing able to follow projects

throug h from conception to com pletion, an d recom mend that this co ntinue. 
14 Chapter 2: Report of the Exposure Subcommittee

BOSC Management Review of the PM Research Program

It is unclear if integration across the risk paradigm happens at the level of the bench scientist. We cannot

say much ab out staffing at the prin cipal investigator (PI) level b ecause, althou gh we requ ested it, we were

not able to meet any scientists at this level. We cannot discern if PIs are aware of the coordination and

integration that is occuring at higher levels. Concerns about this were raised in the previous NERL

managem ent review by B OSC in 1998 . Post-d octora te rotatio ns to ke y OR D loca tions m ight he lp

them to see how their research fits into the “big pictur e.” This would be in addition to the current

good p ractice of h aving p ost-docs sp end time in labor atories of co operators (coope rative agreements w ith

EPA ’s ORD) that c orrespon d to their own area s of specia lization. This wou ld give th e post-do cs both

personal and professional ex perience and contacts, and it would increase in tegration across ORD. ORD

urgently need s to fill their existing post-doctorate po sitions. Hiring freezes are a nathema to staffing. 

Such freezes could give the EPA post-doctorate program a poor reputation if they result in broken

promises. 

Geographic prox imity of the several PM laboratories at RTP, NC, helps ORD integrate across the risk

paradigm. For those ORD PM researchers who are not proximate (e.g., Reston, VA), this may pose a

problem. 

Regarding staffing expertise, there was concern by the BOSC regarding whether the PM Program has

adequate so cial science exp ertise (as noted earlier). 

Workshops and Research Meetings

Communication in workshops and research meetings is an important component of coordinating with the

outside community. OR D is to be commended for encouragin g its scientists to participate in thes e events. 

Peer review in these forums is critical to a scien tifically credible program. Coord ination (e.g .,

cosponsorin g) of confer ences with pro fessional societies migh t increase the visibility, effectiveness,

and regularity of this form of comm unication, while decreasing costs. Emphasis should be p laced

on sha ring ba sic science m ilestones, rath er than p rogram overview s. 

While the Coo rdinatin g Committee for PM Human Expo sure Re search cre ates an ex cellent forum for said

coordination, it was unclear to the Subcommittee if recent findings are communicated regularly in that

forum. Although we did not have any formal discussions with EPA grantees, it appears that ORD may be

missing an o pportunity to learn from its grantees an d increase communication among th em. ORD sho uld

consider making an extra effort to see that STAR grantees present their results to EPA and to one

another regular ly, given th at STA R wo rkshops do no t appea r to be held annually for a ll grantees. 

Those not directly supported by ORD, such as ATSDR grantees, may be unaware of recent

progress in extramural ORD research on PM. The HEI Web Site could help with this, and might

be expanded to increase cro ss-Labo ratory and cross-A gency co mm unication . 

In the exposu re area, ORD has made some excellen t recent hires such as the science lead s whom we met. 

It would appe ar that ORD is short som e senior staff in critical area s, such as exposure m odeling. 

To ensure the productivity of senior staff in leading the science, and given the recent infusion of

post-docs who rely on senior scientists for mentoring, it is important that senior scientists have

appropriate staff support. OR D needs mo re staff and adm inistrative support to free-up science

leads and Branch Chiefs. If there is an overload of administrators relative to scientists, perhaps

new h ires should provide clerical and other staff support. 

A good overall framework is emerging for ORD’s PM Program, including:

 National Program Manager

 Matrix manag ement, team ap proach , committee coord ination h orizontally
Chapter 2: Report of the Exposure Subcommittee 15

BOSC Management Review of the PM Research Program

 Extensive use of peer review 

NRC, CASAC PM Subcommittee, BOSC

Extensive peer reviewed extramural program

Encouragement of peer-reviewed publication

This BOSC Subcom mittee was impressed with the overall manag ement framework. Th e only weakness

may be a t the first-level sc ientific po sitions—the que stion of whether p ost-doctor ates and bench scientists

understand the role that they play in the overall PM Research Program.

4. Discuss, with examples, how ORD is addressing the exposure questions included in the

NRC reports.

NRC Recommendations

ORD is followin g throug h on th e first recommend ation con cerning exposu re, but O RD n eeds to b egin

now on the second recommendation as well. Is ORD putting the mechanisms in place now, so that when

it does get d ata, it will be a ble to use them an d truly integ rate across th e risk parad igm? O RD is

conducting studies on hypothesized toxic constituents and exposure panel studies include metals, ultrafine

particles, co-occurring gases, etc. NERL and NR MRL are collecting samples to study in some biological

test systems. Thus, while a sma ll elemen t of the seco nd NRC re commendatio n is und erway, most of this

NRC recommendation is a followup to studies in progress.

Timelines

Current planning has exposure modeling efforts lagging data collection by a few years. Could they be

more syn chrono us, and could th e mode ling efforts a nd do cumen tation be p ublishe d and made a vailable

earlier? We encourag e ORD to explore th e use of p ilot data fo r mod el develop ment and testing. 

CONCLUSIONS AND RECOMMENDATIONS

Overall, the Subcommittee was impressed with ORD’s success in developing, coordinating, and

integrating exposure research on PM. Notable are ORD’s leadership in national and international

coordination of PM exposure research, and the success of staffing changes, especially the post-doctoral

program. Many of the recommendations that follow are simply to continue or strengthen current good

practices.

Continue the laudable ORD leadership on the CENR. Continue efforts to bring together the

atmospheric science and health effects community for particulate matter exposure research.

Evaluate carefu lly who should be the coo rdinating body for integ rating exp osure science, with

attention to whether HEI has the proper institutional links to make this happen effectively.

Evaluate the impact of the lack of emphasis on indoor air source characterization.

Make more concerted efforts to link with researchers and projects that cover non-COPD

endpoints, to av oid missing o pportunities.

Continue and expa nd linkages of exposure studies w ith existing mon itoring progra ms.
16 Chapter 2: Report of the Exposure Subcommittee

BOSC Management Review of the PM Research Program

Continue efforts to bring together the atmospheric science and health effects community for

particulate matter exposure research.

Explore the use of pilot data for exposure model development and testing.

Contin ue the go od pra ctice of allow ing staff to follow projects thr ough from conception to

completion.

Strengthen social science expertise in ORD.

Prov ide m ore staff support f or senio r scientists, to ensu re the productivity o f senio r staff in

leading the science, recognizing that mentoring post-docs is time-consuming and necessary.

Start an introductory program for the post-docs involving rotations through other key locations

within ORD so that the post-docs gain an appreciation and understanding of all parts of the

risk paradigm (in addition to current rotation s).

Consid er making an extra effo rt to see that STAR grantees p resent their resu lts to EPA and to

one another regu larly, g iven that STAR w orkshops do not a ppea r to be h eld annually for all

grantees. Con sider expandin g the HEI Web Site to increa se cross-Labora tory and cro ssAgency com mun ication. 

More frequent coordination (e.g., cosponsoring) of conferences with professional societies might

increa se the visibility, effectiveness, an d regu larity o f this fo rm o f com munication, wh ile

decreasing costs. Emphasis should be placed on sharing basic science milestones, rather than

program overviews.
1 Both of the se topics we re revised an d update d in the seco nd (199 9) report of the Nation al Researc h Counc il

Committee on Rese arch Prio rities for Airbo rne Particu late Matter. II: Evaluating Research Progress and

Upda ting the Po rtfolio.

Chapter 3: Report of the Atmospheric Sciences Subcommittee 17

BOSC Management Review of the PM Research Program

Chapter 3

Report of the Atmospheric Sciences Subcommittee

Subcom mittee Mem bers: Mitchell J. Small, Ph.D. (Chair)

Murray V . Johnston, III, Ph.D. (Memb er)

Armistead G. R ussell, Ph.D. (Member)

INTRODUCTION

The goal of the BOSC PM Research Program review was to evaluate the management of the Program

rather than its scien tific dire ctions. The following three qu estions were presen ted to D r. Gary Foley,

Director of EPA ’s NERL, b y the BO SC: 

1. Describe, with examples, how research in this area is coordinated with other efforts such as

NARSTO.

2. Discuss the linkage between research on PM sampling and characterization, and research on

modeling the formation, fate, and transport of PM in the atmosphere.

3. Discuss the link between the ORD research program on particle formation and characterization and

exposure and health effects research.

The self-study report addressed these three questions. In addition to the meeting with the full BOSC PM

review committee and the overall brie fing by Dr. John Vand enberg on Oc tober 29 , 1999 , the Atmosphe ric

Scienc es Sub committee met with EPA scientists an d man agers wo rking in th e area of atmosphe ric

sciences, including: Dr. Gary Foley, NERL Director; Dr. Judith Graham, NERL A ssociate Director for

Health; Dr. Jason Ching; Dr. Larry Cupitt; Dr. Kenneth Schere, Chief, Atmospheric Model Development

Bran ch; Dr. Fran k Sch ierme ier, Director, A tmosp heric S cienc es Mo deling Division; D r. James Vic kery;

and Dr. Russell Wien er.

PM atmospheric sciences research at EPA ORD addresses topics and disciplines affecting the formation

and measurement of ambient PM, including: atmospheric chemistry, air quality modeling, ambient

measurement method s development and eva luation, and so urce appo rtionment an d receptor modeling. 

This research is necessary to support EPA’s efforts to implement NAAQS for PM, including the

development and application of models to show the relationship between emissions of primary PM and

secondary PM precursors, and ambient PM conc entrations, and monitoring m ethods to both support

model development and ensure regulatory compliance. Modeling research includes the study of

fundamental processes affecting atmospheric transport and chemistry, the efficient implementation of

models in ve rifiable codes and computin g systems, and th e testing of mod els against observe d field data. 

Mon itoring methods r esearch a ddresse s both Federal Reference Meth ods for u se in fede ral and sta te

monitoring networks and new research-oriented methods for better characterization the composition and

spatial and temporal distribution of ambient PM. 

The ORD atmospheric sciences research program specifically addresses issues raised under NRC Topics

3 and 41
18 Chapter 3: Report of the Atmospheric Sciences Subcommittee

BOSC Management Review of the PM Research Program

3. Characterization of Emissions Sources: What are the size-distribution, chemical composition, and

mass-emission rates of particulate matter emitted from the collection of primary-particle sources

in the United States, and what are the emissions of rea ctive gases that lead to second ary particle

formation through atmosph eric chemical reactions?

4. Air Quality-Model Development and Testing: What are the linkages between emission sources

and amb ient concentration s of the biologically imp ortant comp onents of particu late matter?

In addition, the research addresses the area referred to by the NRC as “Tech nical Assistance,” for

standard development and implementation.

Although NERL maintains the principal responsibility for atmospheric sciences research within ORD,

important contributions also are made by NCER, through the STA R program and the Supersites Research

Cente rs, and N RMRL, through its emission s characte rization an d sourc e profile res earch for in put to

atmospheric models. Interactions with NHEERL and NCEA also are important to ensure that the ambient

modeling and monitoring research efforts are focused on the “biologically important” components of PM

(see NRC Topic 4 ab ove).

This review focused on the written and verbal responses to the self-study questions submitted by the

BOSC to EP A; the Subcommittee’s review of the responses is described below. Specific Subcommittee

recommendations are identified by bold type in the text of the review.

EVALUATION OF RESPONSE TO SELF-STUDY QUESTIONS

1. Describe, with examples, how research in this (atmospheric sciences) area is

coordinated with other efforts such as NARSTO.

ORD has provided effective scientific leadership in the area of atmospheric sciences, both in the

development of fundamental new science and scientific methods (weighted toward the extramural

program) and by integrating, synthesizing, and building consensus on the existing science and defining

the state-of-the-art for regulatory assessment (primarily through the intramural program). One

mechan ism for ensurin g leadership for, an d coordination with, the broad er scientific and regu latory

community is through direct research collaboration. ORD has been effective in encouraging

collaboration among its scientists and those in academia and other research organizations. These

collaborations have, in the past, often been primarily with universities located near the particular EPA

research Laboratory or Center. The STAR program provides opportunities for EPA scientists to interact

with a broad er, more geogra phically dispersed set of collaborators. In addition to the productive

collaborations with loca l unive rsities, EP A scien tists shou ld be en coura ged to expan d their

interactions to include a broader set of outside scientists, through collaboration with STAR grant

recipients an d other v enues. 

EPA’s scientific leadership on the PM issue is evident in its role in a number of outside activities and

forums, including NARSTO, which is “a public/private partnership,” whose membership spans

government, th e utilities ind ustry, and academia throug hout M exico, th e Unite d States, a nd Canada . Its

primary mission is to coordinate and enhance policy-relevant scientific research and assessment of

tropospheric pollution behavior.” (For a further description of NARSTO, see http://www.cgenv.com/

Narsto/whatisnarsto.html.) While originally formed to address issues of ozone in North America, the

organiza tion has since exp anded its research coordin ation age nda to in clude PM an d other re gional air

pollutants.
Chapter 3: Report of the Atmospheric Sciences Subcommittee 19

BOSC Management Review of the PM Research Program

EPA plays a ma jor and leading role in NARSTO at b oth the sc ientific and manag erial levels, an d this

appears to be having a very positive impact on how the Agency is managing the PM Program. For

example, Ken Schere is a major contributor to the NARSTO assessment, and has been allowed to spend

considerable time in that role. This has provided increased interaction to better understand the directions

in which the community outside of EPA is moving. ORD also is playing a major role in the CENR,

coordinating at the upper level management of the research. EPA coordination at the international level

(beyond North America) is hampered somewhat by limitations on foreign travel—it is much easier for

ORD to bring scientists from Europe and Asia to the United States for meetings than it is for EPA

scientists to attend meetings overseas (e.g ., in the Federa l Repub lic of German y, FRG). Given the

growing need for international collaboration in efforts to understand and manage atmospheric PM

and related air pollution issues, EPA’s leadership role in science planning would be strengthened

via greater participation of EPA scientists in international meetings. In particular, the U.S.-FRG

meetings are a bit sporadic and have not been directed at PM as much as ozone, and further focus

on PM issues should be explored in this forum.

2. Discuss the linkage between research on PM sampling and characterization, and

research on modeling the formation, fate, and transport of PM in the atmosphere.

Sampling and measurement methods development and evaluation research is conducted in three ways: by

NERL scientists, by STA R gran t recipients, and b y manu facturers. A t the curre nt time, N ERL primarily

evaluates methods for regulatory purposes, while external group s develop and evaluate research me thods,

as well as p otential methods f or future regulatory u se. This a rrangement of the research p ortfolio is

appropriate because it builds upon the strengths of each sector. In the past, there has appeared to be

relatively little interaction at the scientific level; this should change, and appears to be doing so, with the

advent and expansion of the Sup ersites program. It is important for EPA scientists to continue to take

a key participatory role in the Supersites Centers, with the objective of having continued, detailed,

in-house knowledge and access to expertise on the state-of-the-science in sampling and

measu remen t metho ds research and how those techniques can be used to suppo rt EPA ’s mission. 

The issu e of good interaction among EPA scientists an d STAR g rant recip ients in th e area of atmosphe ric

sciences is of general importance. Although research activities and results are currently well coordinated

across ORD Laboratories and Centers, some gaps appear in the level of interaction between STAR grant

recipients and intramural researchers. These interactions appear to have occurred on more of an ad hoc

basis, due to previous knowledge or experiences between the researchers, rather than as a result of

consistent and well-coordinated efforts to ensure interaction . More concerted effort is needed to ensure

interaction between the full set of STAR grant recipients involved in PM research and EPA

intramural researchers.

Similarly, there appear to be opportunities for greater interaction between the ORD PM Program and

other research programs, including the global change program (where PM measurement, trends, and

impacts also play an important role) and broader research program s on chemical risks and risk

assessment, where again ambient monitoring programs and air pollution modeling are a key component of

assessment. Additional effort is needed to ensure synergy and exchange of information between the

ORD PM R esearch Program and rela ted prog rams o n globa l chang e and ch emical risk assessment.

At present, it is a bit early to fully assess the overall linkage between the sampling/characterization

research and the modelin g research in EPA ’s atmos pheric sc iences res earch p rogram b ecause b oth are in

their early stages of implementation. EPA has shown that it is laying the foundation for that linkage and

there are activities that show that the two communities at EPA are interacting. (Examples include

emissions groups and receptor modelers discussing how to take measurements, participation on the

Supersites steering committees, and urging the STAR grantees to cooperate directly with Supersites
20 Chapter 3: Report of the Atmospheric Sciences Subcommittee

BOSC Management Review of the PM Research Program

researchers to target their research for modeling needs.) Because most of this represents what will be

future eff orts, it is important to ass ess at regu lar intervals h ow we ll these link ages are working. EPA is

putting a lot of stock into the PM Supersites and Centers to provide an integrated effort between the PM

characterization and the exposure/health effects research. Because most of this research will be

conducte d in the future , EPA m ust monitor an d assess the effectiven ess of the plann ed linkages. 

One a rea whe re further improv ement m ay be nee ded inv olves the re lative allocatio n of fun ds for da ta

collection and an alysis. The examp le of setting a side $1 .6 million to analyze about $ 15-20 million of data

collection effort in the Supersites program (a 1:10 ratio, which is too small) suggests a continuing

problem. Experience by members of the Subcommittee with data collection programs for particles and

other air pollutants suggest that as much time and effort is needed to provide for effective compilation and

interpretation of data as is required to collect the data in the first place. Although most data collection

programs do involve a significant amount of interpretation by the investigators involved, and the

Supersites program should be no exception in this regard, the potential for broader and more extensive

use of the data is often unrealized. To provide the opportunity for such use, ratios (of subsequent

application and interpretation budget to collection budget) closer to 1:4, or even 1:3, are considered to be

preferred . Failure to provide adequ ate resour ces for da ta applica tion and interpreta tion cou ld specific ally

inhibit th e ability of EPA to a ttribute ob served PM con centration s (and su spected associated health

effects) to specific sources of PM, and limit the ability to provide the tools needed by the states and the

regulated community to best manage PM air quality. More effort is ne eded to en sure that a dequa te

resources are availa ble fo r, and allocat ed to, th e application and in terpreta tion of data c ollected in

major field studies such as Supersites. A higher proportion of resources should be allocated to the

data analysis efforts that follow data collection.

The ab ility to link resear ch on am bient monitoring method s and atmosphe ric mod eling is esp ecially

important for the practical planning and implementation of NAAQS compliance strategies. Much of the

contribution of the EPA will come about through integration, consensus building, and application of

existing science in these areas, and not necessarily through new science development. The efforts of EPA

in this rega rd have been ve ry good, with espec ially effective in teractions in dicated w ith the air

office—OAR/OA QPS. In this regard, the research and program arms of EPA appear to have learned

lessons from some of the previous, less-than-optimal experience at integrating their efforts in addressing

the ozone issue.

3. Discuss the link between the ORD research program on particle monitoring and

characterization and exposure and health effects research.

ORD has, through its Supersites program, encouraged particle characterization and monitoring

researchers to interact w ith health effects research ers, e.g., to sample key co nstituents in the atm osphere

of interest to health endpoints, has sponsored/cosponsored meetings to bring the two communities

together, and has organized its internal resources to help integrate its research. In general, the

Subc ommittee is please d with th e degree of interactio n and linkage th at is eviden t. Althou gh there is

already significant grass-roots interaction at the “nearest neighbor” level (and this is apparent throughout

ORD’s program, e.g., between source and ambient concentration research, between ambient concentration

and exposure, between exposure and dose and dose-response groups), greater communication among

groups and laboratories at the scientist level across the entire risk paradigm could be further encouraged

and facilitated. One example would be for the atmospheric modelers and emissions researchers to interact

with tho se involv ed in do se studies. Efforts to in itiate and e ncoura ge inter-lab oratory pro jects also would

help in this regard. For example selected scientists could be appointed with joint affiliation with two

Labora tories, so tha t they can fa cilitate scientific communicatio n between them . To a de gree, ind ividuals

changing Laboratories can accomplish this, though this appears to be limited at this time. Another

approa ch wou ld be to more fully utilize integrated assessme nts, from emission s throug h atmos pheric
Chapter 3: Report of the Atmospheric Sciences Subcommittee 21

BOSC Management Review of the PM Research Program

sciences, exposure, and health effects, as a basis for organizing selected PM meetings and colloquia,

explicitly identifying how the outputs from particular research efforts serve as inputs to others, as well as

their specific inpu t and impo rtance to the overall asse ssment exe rcise. More should be don e to

encourage interaction of EPA scientists at the grass- roots level across the full risk paradigm, using

mech anism s such a s joint appoin tmen ts in EP A La bora tories or Cente rs, and mor e explicit

emphasis on integrated assessment as an organizing framework for research input and

coordination.

The lack of awareness and appreciation for the broader opportunities for interaction across the full risk

paradigm appears to be most serious among STAR grant recipients. These grantees have not always been

aware of the op portunity for participa tion in major PM colloqu ium meetin gs. A more explicit effort

should be ma de by STAR Project Officers to en courag e grant rec ipients working o n PM issues to

participate in the major PM colloquia.

CONCLUSIONS AND RECOMMENDATIONS

The following major findings and recommendations are drawn from this evaluation of the ORD

atmospheric sciences research program. These findings and recommendations are organized around the

five questions posed by ORD to the BOSC for consideration in our evaluation of this program:

1. Are the ORD Laboratories (i.e., NHEERL, NERL, NRMRL) and Centers (i.e., NCEA,

NCER) sufficiently coordinating research across the risk paradigm (e.g., sourceexposure-dose-response and management)?

ORD’s PM Research Program is very well coordinated along the risk paradigm, especially at the

management level. Reasonable coordination also is apparent at the grass-roots level among EPA

scientists and grant recipients; however, this appears to be limited mostly to nearest neighbor interactions

(e.g., between source and ambient concentration research, between ambient concentration and exposure,

between exposure and dose and dose-response), with less awareness, appreciation, and broader

opportunities for interaction across the full risk paradigm. This is particularly true among STAR grant

recipients, who have not always been aware of the opportunity for participation in major PM colloquium

meetings.

More sho uld be don e to encourag e interaction of EPA scientists at the grass-roo ts level across

the full risk paradigm, using mechanisms such as joint appointments in EPA Laboratories or

Centers, and more explicit emphasis on integrated assessment as an organizing framework for

research input and coordination.

A more explicit eff ort shou ld be m ade by STAR Pro ject Officers to encourage g rant recipients

working on PM issues to participate in the major PM colloquia.

2. Are the Laboratory/Center research programs integrated to address the highest

priority research needs?

The ORD resea rch programs are very well integrated to a ddress the h ighest priority research n eeds. 

Coord ination w ith regula tory needs is especially g ood as a re sult of effec tive interac tions with

OAR/OA QPS. One area where further improvement may be needed involves the relative allocation of

funds for da ta collection and an alysis. 
22 Chapter 3: Report of the Atmospheric Sciences Subcommittee

BOSC Management Review of the PM Research Program

It is important for EPA scientists to continue to take a key participatory role in the Supersites

Centers, w ith the objective of having co ntinued , detailed, in-h ouse knowledge and access to

expertise on the state-of-the-science in sampling and measurement methods research and how

those techn iques can be used to suppo rt EPA ’s mission. 

More effort is needed to ensure that adequate resources are available for, and allocated to, the

applicatio n and interpretatio n of da ta collected in majo r field studies such as S upersites. A

higher p roportion of reso urces should be a llocated to th e data analysis eff orts that fo llow da ta

collection.

3. Are research activities and results sufficiently communicated among the Laboratories

and Centers, ensuring maximum leveraging and coordination of research efforts?

Although research activities and results currently are well coordinated, some gaps are apparent in the

level of interaction between STAR grant recipients and intramural researchers. These interactions appear

to have occurred on more of an ad hoc basis, due to previous knowledge or experiences between the

researchers, rather th an consistent an d well-coordin ated efforts to ensure interaction. Similarly, there

appear to be opportunities for greater interaction between the ORD PM program and other research

programs, including the global change program (where PM measurement, trends, and impacts play an

importa nt role) an d broad er research programs on chemical risks and risk a ssessme nt.

More concerted effort is needed to ensure interaction between the full set of STAR grant

recipients involved in PM research a nd EPA intram ural researchers.

Additional effort is needed to ensure synergy and exchange of information between the ORD

PM R esearch Program and rela ted prog rams o n globa l chang e and ch emical risk assessment.

4. Is there sufficient communication of results to the broader scientific and regulatory

community?

EPA has established and main tained very good interactions with th e broader scien tific and regulatory

community involved with the PM issue. This is especially true vis-a-vis scientists at u niversities lo cal to

the ORD research laboratories.

In addition to the productive collaborations with local universities, EPA scientists should be

encouraged to expand their interactions to include a broader set of outside scientists, through

collabor ation w ith STA R gran t recipients an d other v enues. 

5. Is EPA-ORD providing sufficient scientific leadership within and outside the Agency?

ORD has provided effective scientific leadership, both in the development of fundamental new science

(weighted toward the extramural program) and integrating, synthesizing, and building consensus on

existing science and defining the state-of-the-art for regulatory assessment (primarily through the

intramural pro gram). This lea dership is evid ent in their role in ou tside organization s such as NARSTO. 

EPA leadership and coordination at the international level could be strengthened through greater

participation of EPA scien tists in international me etings. 

Given the growing need for international collaboration in efforts to understand and manage

atmospheric PM and related air pollution issues, EPA’s leadership role in science planning

would b e strengthened v ia greater participation of EPA scientists in international meetings. In
Chapter 3: Report of the Atmospheric Sciences Subcommittee 23

BOSC Management Review of the PM Research Program

particular, the U.S.-FRG m eetings are a bit sporadic and have not been d irected at PM as much

as ozone, and further focus on PM issues should be explored in this forum.
24 Chapter 3: Report of the Atmospheric Sciences Subcommittee

BOSC Management Review of the PM Research Program
Chapter 4: Report of the Epidemiology Subcommittee 25

BOSC Management Review of the PM Research Program

Chapter 4

Report of the Epidemiology Subcommittee 

Subcom mittee Mem bers: Thom as A. Burke, Ph.D. (Ch air)

William E. Coo per, Ph.D. (Vice-Chair)

Jerald A. Fagliano, M.P.H., Ph.D. (Mem ber)

Richard W. C lapp, Ph.D. (Mem ber)

INTRODUCTION

The goal of the BOSC PM Research Program review was to evaluate the management of the PM Program

rather than its scientific directions. Epidemiologic studies of air quality and human health have provided

the pub lic health b asis for the Clean Air Act an d the reg ulation o f PM. Strong s cientific cap acity in

epidemiology is essential to the success of the ORD PM Research Program. The Epidemiology

Subcommittee for the BOSC review was Chaired by Dr. Thomas A . Burke, Vice Chaired by Dr. William

Cooper, and included Dr. Richard Clapp of Boston University and Dr. Jerald Fagliano of the New Jersey

Department of Health a nd Sen ior Services. Th e review focuse d on the three questions p resented below . 

Written responses to these questions were submitted to the BOSC by ORD prior to the review. The

findings also are based upon overview presentations from the PM management leaders, as well as detailed

discussions with researchers from the Epidemiology and Biomarkers Branch and the Clinical Research

Branch within the Human Studies Division, and the Pulmonary Toxicology Branch and the

Immunotoxicology Branch within the Experimental Toxicology Division.

EVALUATION OF RESPONSE TO SELF-STUDY QUESTIONS

1. Describe how the EPA’s epidemiology efforts are integrated with those of NIH, NIEHS,

and CDC.

The ep idemio logists of the Human Stu dies Bra nch h ave mad e a conce rted effort to integrate th eir efforts

with other involved agencies. However, it is important to keep in mind that the Environmental Protection

Agen cy (EPA ), the Na tional Institu tes of He alth (NIH ), the Na tional Institu te of Env ironme ntal He alth

Science (NIEHS), and the Centers for Disease Control and Prevention (CD C) have different priorities,

mandates, and research agendas. These divergent missions inevitably limit integration of epidemiology

research efforts on any particular topic, including PM research. Furthermore, the Committee on

Environm ent and N atural Resou rces (CEN R) is just now formalizing its interagen cy Standing W ork

Group on PM Research Coordination. More effective integration of epidemiology efforts are now being

planned and ex ecuted with CEN R oversight, however, it is too soon to evaluate the success of those

efforts at this time.

There currently are interagency agreements with NIH, specifically the National Institute of Allergy and

Infectious Diseases (NIAID), on the Inner City Asthma Study, where EPA has successfully integrated an

environmental monitoring component onto an on-going project. In addition, there is an interagency

agreement with CDC on a U.S.-Mexico border air pollution epidemiology study. There also are some

informal individual collaborations between EPA epidemiologists and NIEHS or CDC researchers which

seem to be well-integrated on specific clinical epidemiology issues. However, EPA has limited

epidemiologic capacity to be involved in more extensive collaboration and integration with the other

federal ag encies at th is time. It is recog nized th at most of O RD’s research in epidemiology is o f necessity

extramural, but increased intramural expertise should be developed to ensure proper communication and

understanding among PM toxicologists, epidemiologists, and other researchers on the problem.
26 Chapter 4: Report of the Epidemiology Subcommittee

BOSC Management Review of the PM Research Program

Some attempts at collaboration between E PA and C DC have b egun but have n ot progressed because

programmatic priorities at CDC precluded further work. Similarly, a potential collaboration between

EPA and NIEHS on gene-environment interactions has not progressed because of limited capacity at EPA

in the gen etic epidemiology ar ea. For similar reason s of limited capacity, EPA h as not be en able to

integrate with PM epidemiology research efforts by other stakeholders, such as State agencies, or to take

advantage o f surveillance data th at might inform or enrich EP A research. 

In spite of limited staffing and in-house epidemiologic capacity, EPA ORD has made impressive gains

over the past year to integ rate its PM ep idemiology research efforts with those of oth er federal agencies. 

With the development of the CENR Work Group, these gains can be expected to continue over the next

several years.

2. Describe ORD’s capacity to address epidemiological issues.

The ORD capacity for epidemiological research resides primarily in the National Health and

Environmental Effects Research Laboratory’s (NHEERL) Epidemiology and Biomarkers Branch (EBB)

in the Human Studies Division (HSD). There is a growing appreciation of epidemiology in the PM

Research Program. Within the HSD, there is a small cadre of epidemiologists with diverse and

complementary backgrou nds. The group wa s described b y the Laboratory Director as “lean and mean,”

and it is apparent that they have made excellent progress in establishing an excellent foundation for PM

activities.

The greatest need in the area of epidemiological research is for additional personnel. Enhancing exp ertise

and credibility will be difficult in the face of the growing demands for PM research. At the present time,

there are only four fu ll-time equivalents (FTEs) dev oted to the PM work, and e ight total investigators

including th ose with other rese arch respon sibilities. Adequ ate administrative an d technical su pport staff

also are essential to assure the timeliness and quality of the work. Given the importance of epidemiology

to evaluating human health risks of PM, and evaluating the efficacy of risk management efforts, the

current number of FTEs is not adequate.

Accord ing to the NHE ERL Labora tory Directo r, given th e realities of fisc al and p ersonn el constrain ts, it

is not likely that the ORD epidemiology capacity will grow in size to rival similar groups at the CDC or

the National Cancer Institute (NCI). However, it must be recognized that the ultimate credibility of

science at EPA requires that the Agency achieve a level of excellence and leadership in environmental

epidemiology equ al to those p rograms. Increasin g EPA respon sibilities to ad dress no ncanc er health

effects, evaluate cumulative population risks, and develop human health indicators, will only increase the

demand s upon the HSD. 

Forma l coopera tive efforts with Pub lic Health Service a gencies a nd un iversity-based epidemiologists

provide a mechanism to enhance the ORD capacity. The HSD has established some strong working

relationsh ips with th e PM Centers, and is re aching out to estab lish others. Outrea ch to the S tate

Epidemiologists has been initiated, and represents a potentially important step forward not only for ORD,

but also for states, where there also is a critical need for developing capacity. To assure active

collaboration, cooperative agreements should be considered. A number of ORD researchers have stated

that cooperative ag reements pro vide greater enco uragemen t for collaboration than grants. 

At the present time, there is a dearth of well-trained environmental epidemiologists. This is a major

impediment to the Agency’s efforts build capacity in epidemiology. Improved support for environmental

epidemiolo gical training is essential to the long term suc cess and cred ibility of epidemiology at O RD. 
Chapter 4: Report of the Epidemiology Subcommittee 27

BOSC Management Review of the PM Research Program

3. Describe the extramural epidemiological research that EPA is funding.

Extramural epidemiological research is funded through interagency agreemen ts, cooperative agreements,

and grants. Agreements are generally developed and implemented through NHEERL and other EPA

Laboratories, while grants are administered through the National Center for Environmental Research

(NCER). 

At present, there is a limited amount of extramural epidemiological research funded through interagency

or coope rative agreements. T hrough an inte ragency ag reemen t with NIAID, O RD (N HEERL) is

providing supplemental funds and actively collaborating in a PM component to the Inner City Asthma

Study sp onsored by NIAID and NIEH S. OR D also h as an inte ragency ag reemen t with the CDC to

support ep idemiological stu dies of air pollution, in cluding PM and other copollutan ts, along the U.S.-

Mexico border. With the National Exposure Research Laboratory (NERL), NHEERL is contributing

funds to support a health effects component to a study of PM exposure in patients with cardiovascular and

pulmon ary disease. 

ORD also sponsors extramural research through the STAR program coordinated by NCER. The role of

NHEERL in this granting process is to assist in the preparation of Requests for Applications (RFAs) and

to provide “relevancy review” of proposals that have passed scientific merit review by ad hoc peer review

panels. G rant awa rds und er STA R hav e been m ade to esta blish five a cademic Airbo rne Partic ulate

Matter Res earch Cen ters, each of wh ich have cap acity and plans for c onductin g epidemiological research. 

In addition, ORD has made grants to other institutions for epidemiological research on PM.

The overall scope of the extramural research program appears to be largely determined through the grant

process. W hile this p rocess en courage s creative ap proach es to stud y, it also limits the ability of OR D to

ensure the deve lopme nt and im plemen tation of a tig htly integrate d PM Researc h Prog ram. It also limits

the ability of ORD/NH EERL scientist to actively collaborate, monitor, and affect the direction of research

conducte d by grantees. 

Extramural epid emiolog ical research activity is likely to inc rease as fu nds ex pand in comin g years. Th is

will increase demands on th e limited number of NH EERL scientists. Add itional intramural reso urces are

neede d to ensu re that the o verall resear ch agen da for ep idemio logy remain s focused and ad aptable to

changing priorities.

CONCLUSIONS AND RECOMMENDATIONS

Building leadership, excellence, and capacity in epidemiology is essential to the success of the PM

Research P rogram. 

ORD has succe eded in b uilding a n outstanding core staff for PM epidem iology. 

Although the expertise of current researchers and managers in epidemiology is excellent, the

current sta ffing lev els for epid emiolo gy with in the Hu man Studies Branch a re not ad equate to

meet the increasing intramural and extramural research needs of the PM program.

Specific management goals for the appropriate balance between intramural capacity and

extramural research need to be developed.

The sm all size of the cu rrent epidemiolo gy staff imped es ORD ’s capacity to integrate

throughout the agency, oversee and participate in extramural research, and actively pursue

oppo rtunities fo r collabor ative effo rts with o ther agen cies. 
28 Chapter 4: Report of the Epidemiology Subcommittee

BOSC Management Review of the PM Research Program
Chapter 5: Report of the Toxicology Subcommittee 29

BOSC Management Review of the PM Research Program

Chapter 5

Report of the Toxicology Subcommittee 

Subcom mittee Mem bers: Dr. James S. Bu s, Ph.D. (Chair)

Frederick J. Miller, Ph.D. (Member)

Hanspeter Witschi, Ph.D. (Mem ber)

INTRODUCTION

Toxicology is an in tegral resear ch componen t in the ove rall evalua tion of the potential p ublic he alth

effects associated with PM exposu re. The strength of toxicology studies is the ability to explore and test

hypotheses establishing the plausibility of suspected human effects induced by PM. This review focused

on the written and verbal responses to three self-study questions submitted by the BOSC to EPA; the

Subcommittee’s review of the response is described below. Specific Subcommittee recommendations

are identified by bold type within the text of the review.

EVALUATION OF RESPONSE TO SELF-STUDY QUESTIONS

1. How, using examples, is the National Health and Environmental Effects Research

Laboratory (NHEERL) research program addressing the toxicology questions included

in the NRC report?

The N HEERL rese arch pro gram is eff ectively coor dinated and inte grated at a m anagement level within

the NRC health-related Research Topics 5-9. In particular, the toxicology program is effectively using

relevant PM materials obtained from realistic field situations.

Animal toxicology and human exposure projects are well coordinated, and are clearly facilitated by the

geographical proximity of the animal and human testing laboratories. Both whole animal and in vitro

studies are bein g used effectively to gu ide design an d interpretation of in vivo animal and human studies. 

Attention needs to be directed to ensuring that dose-response information is available for the

inhalation toxicolo gy studies (both huma ns and animals). The coordination and integration within the

Utah Valley Study and Baltimore and F resno Field stu dies are excellen t examples o f intra- and interLaboratory/Ce nter activities. In the future, managemen t integra tion of exposure studies on susceptible

subpopulations (NR C research topic 2) to effects (NRC research topic 8) is desirable.

2. How is NHEERL research addressing both the strengths and weaknesses of in vitro and

in vivo experimental models (or proposed to be used) in supporting assessments of

potential human health effects of PM?

Efforts to integrate in vitro and in vivo studies in animals and humans have been successful. The abovementioned Utah Valley Study could well serve as a mo del for similar end eavors in the futu re. 

Manag ement needs to ensure that investigators use in vitro systems in a way that capitalizes on the

strengths o f such sy stems an d min imizes their weakn esses. The primary strengths of such systems are

the ability to ex plore an d refine u nderstan ding of m echan ism(s) of ac tion of ch emical an d particu late

toxicity at the cellular level. The value of information from such in vitro studies for use in risk

evaluatio ns, how ever, mu st always be conside red with in the con text that organism-level toxicity

respon ses are sign ificantly imp acted by th e added comple xity of cell-cell an d cell-organ -physiolog icaltoxicokinetic influ ences. 
30 Chapter 5: Report of the Toxicology Subcommittee

BOSC Management Review of the PM Research Program

Deve lopm ent of long-r ange research manag emen t plans that deliver ap propriate rese arch r esults in

needed timefram es should be continued, and will be essential for maintaining continued Congressional

support for the technically complex PM program.

Communication of the strengths and implications of the research to the scientific community-at-large has

been excellent, and has contributed to the recognition of NHEERL toxicologists as leaders in the PM

research efforts. These external outreach efforts need to be continued to assure that current hypotheses

are constantly scrutinized and challenged. The conduct of biomedical research is increasingly complex,

requiring inv estigator interactions an d exploitation o f research information across mu lti-disciplinary

research teams to fully understand the human health implications identified from animal and cellular

toxicity studies. The data and information being generated by ORD’s STAR program offers an

opportunity to potentially improve the ability of NHEERL toxicology investigators to rapidly and

effectively incorporate relevant toxicity information into the design and interpretation of NHEERL-based

research studies. Man agem ent shou ld develop effectiv e mech anism s to enable appro priate

collabor ations w ith investiga tors fun ded thro ugh the STAR prog ram. For example, such

mechanisms might include creation of opportunities for regular face-to-face meetings of EPA and STAR

investigators addressing related research hypotheses, ultimately leading to improved study designs and

mutual sh aring of research stu dy materials and information. 

3. How is the toxicology research program being coordinated with human exposure

characterization efforts?

The animal toxicology research is w ell coordinated w ith the Agen cy’s human clinical studies. 

Comparable endpoints are being addressed in many of the projects. A strength of animal toxicology

studies is th e ability to exp lore and test hypoth eses estab lishing th e plausib ility of suspec ted hum an health

effects ind uced b y PM. H ighly invasive meth ods can be used in animal toxicolo gy studies. PM m aterials

are being use d to explore an d define relevan t hypotheses that relate to p ossible human health effects. 

Both in vitro and in vivo animal models are being used and developed to investigate postulated

mecha nisms o f PM-in duced tissue dam age and disease. Characteriz ation do se-respon se assessments in

both animals and humans will be essential for defining plausible causation of potential PM-induced

human health effects. Thus, manag ement needs to continue to supp ort the assessment of a broader

array of endpoints in the animal studies so that an assessment of the full range of potential PM

effects can be elucida ted. Such efforts also will lead to mechanistic understandings of PM effects.

Exten sive chara cterization of particles in CAP S studie s is important for gain ing insigh ts on pote ntially

putative constitue nts. Alth ough a ccess to well-characte rized commun ity PM so urces is b eing effec tively

coordinated , management needs to foster effective cross-laboratory analytical efforts to support the

timely inte gration o f particulate analy tical data into toxicolog y studies. Access to analytical d ata is

critical for selection of appro priate dose and test material. 

Excellent communication networks exist between the Laboratories and Centers for keeping abreast of

research results. Optimum leverage is compromised, however, by fiscal constraints, including

adequ ate funding fo r assuring regular f ace-to-fa ce investiga tor interactio ns. These include not only

interactions within the EPA research laboratories but also interactions with external investigators engaged

in related research activities.

The tox icology pro gram is u sing relev ant PM materials, a s is seen from the Utah Valley Stu dies. Th is

PM material was obtained by creatively exploiting an historical exposu re episode. Adequate laboratory

and fiscal resources should be available to rapidly respond to unanticipated events that have

potential value in assessing the impact of particle pollution on human health (e.g., forest fires and

concerns of relationship of wood smoke to asthma). 
Chapter 5: Report of the Toxicology Subcommittee 31

BOSC Management Review of the PM Research Program

Management should be supportive of flexible allocation EPA S&T monies (e.g., re-allocations

between R&D and Supply and Equipment budgets) to address not only general research needs but

also una nticipated research o r external en vironm ental even ts.

CONCLUSIONS AND RECOMMENDATIONS

The following conclusio ns and recommen dations were e xtracted from th e text in this chap ter:

The NHEERL research program is effectively coordinated and integrated at a management level

within the NRC health-related Research Topics 5-9.

Attention needs to be directed to ensuring that dose-response information is available for the

inhalation toxicolo gy studies (both huma ns and animals).

Manag ement integration of exposure studies on susceptible subpop ulations (NRC research

topic 2) to effects (NRC research topic 8) is desirable.

Manag ement needs to ensure that investigators use in vitro systems in a way that capitalizes on

the strengths of su ch systems an d minim izes their weaknesses.

Developmen t of long-range research man agement plans that deliver appro priate research

results in needed timeframes should be continued.

Man agem ent shou ld develop effectiv e mech anism s to enable appro priate collab orations with

investigators funded through the STAR program.

The animal toxicology research is well coordinated with the A gency’s human clinical studies.

Management needs to continue to support the assessment of a broader array of endpoints in the

animal studies so that an assessment of the full range of potential PM effects can be elucidated.

Man agem ent needs to fo ster eff ective cro ss-labo ratory analy tical eff orts to su ppor t the tim ely

integration of p articulate analytical da ta into toxicology stud ies.

Optimum intra- and inter-laboratory leverage of researcher communications is compromised by fiscal

constraints, including adequate fund ing for assuring regular face-to-face investigator interactions.

Adequate laboratory and fisc al resource s should be ava ilable to rap idly respond to

unanticipated events that have potential value in assessing the impact of particle pollution on

hum an hea lth (e.g., forest fires and co ncerns o f relationsh ip of w ood sm oke to asth ma). 

Management should be supportive of flexible allocation EPA S&T monies (e.g., re-allocations

between R&D and Supply and Equipment budgets) to address not only general research needs

but also una nticipated research o r external environm ental events.
32 Chapter 5: Report of the Toxicology Subcommittee

BOSC Management Review of the PM Research Program
Chapter 6: Report of the Assessment Subcommittee 33

BOSC Management Review of the PM Research Program

Chapter 6

Report of the Assessment Subcommittee 

Subcom mittee Mem bers: Rae Zimm erman, Ph.D . (Chair)

Marilyn Brow n, Ph.D. (Vice-Chair)

Lauren Zeise, Ph.D. (Mem ber)

INTRODUCTION

The goal of the BOSC PM research review was to evaluate the management of the PM Research Program

rather than the sc ientific directions of the p rogram. NCEA is on e of the two cen ters located within O RD. 

NCEA’s headquarters is in Washington, DC, and it has offices in RTP, NC, and Cincinnati, OH. The

main mission of NCE A with resp ect to PM is to p repare the AQCD mandated under the Clean Air Act. 

NCEA conducts research as well as research needs assessments for PM that emerge during the

development of the AQCD. Research emanating from this needs assessment is used as a basis for

subsequent AQCD s. NCEA also has a close connection with the development and review of the Staff

Paper on related regulatory issues produced by OAR.

This is a review of EPA’s management of the PM risk assessments. Specifically, the Subcommittee

addressed the following issues:

1. The methodological approach being used in the risk assessment for PM.

2. The in tegration o f results from the intramural and extramu ral research programs into the a ssessme nt.

3. The incorporation of advice/recommendations contained in the BOSC Program M anagement Review

of NCEA into the PM program.

The Assessment Subcommittee met on October 28-29, 1999, and continued the discourse thereafter via email and telephone. This review is based on:

Presentations by Dr. John Vandenberg, National Research Program Manager for Particulate Matter

(NPM ).

A Subcommittee meeting with NCEA Director, Dr. William Farland, and NCEA RTP D ivision

Director, Dr. Lester Grant. 

A subsequent meeting with selected members of the NCEA staff, RTP Director Grant (in charge of

the overa ll prepara tion of the AQC D), Ch on Sh oaf (respo nsible for NCE A plan ning), N CEA criteria

development staff, Allan M arcus, David Mage, La rry Folinsbee, and William W ilson. 

Documents provided by NCEA or prepared previously by the BOSC listed in the reference section of

this chapter.

The Subcommittee’s review of the self-study response is described below and the specific BOSC

recommendations are identified by bold type in the text of the review.
34 Chapter 6: Report of the Assessment Subcommittee

BOSC Management Review of the PM Research Program

EVALUATION OF RESPONSE TO SELF-STUDY QUESTIONS

1. What is the methodological approach being used in the risk assessment for PM?

The PM stand ard reflects an impo rtant p ublic health conce rn and is a high priority for the A gency.

Multiple major risk assessment documents are related to the revision of the PM standard: (1) the PM

AQCD (e.g., U.S. EPA, 1996a), which provides the health effects assessment including exposure-effect

relationships; (2) the PM Staff Paper (e.g., U.S. EPA, 19 96b), which sum marizes PM sources,

atmospheric fate and transport, exposure, health effects, and dosimetry information and provides

determinations regarding the adequacy of the current PM standard; and (3) the Research Needs document

(e.g., U.S. EPA, 1998) delineating data and knowledge gaps identified during the development of the

AQCD and research strategies to fill them. The PM AQCD and Research Needs docume nts are

developed by NCEA, wh ile the Staff Paper is developed by OAQPS, which is within OAR. Standardsetting an d some aspects of the AQ CD d evelopm ent proc ess, such as the formal Clean Air Scie ntific

Advis ory Committee (CASA C) pee r review o f the AQ CD, a re Con gressionally mand ated. Due to its

public health importance and the economic impact of regulatory action, the processes and products of the

AQCD development are heavily scrutinized by the regulated and public health communities, states,

Congressional staff, and others. Overall, NCEA appears to be doing a very good job managing the

development of the PM AQC D and the Research Needs document, and the process for developing the

Staff Paper appears sound.

Planning the Content of the PM AQCD

As explained by the Agency, the process begins with internal discussion and identification of critical

issues to b e addre ssed in th e AQCD, in cludin g those id entified b y the NRC (NRC, 19 98). This is

followed by scientific workshops targeted at exploring critical issues. Research needs also are considered

and intr amura l and ex tramura l research m ay be initiated to develo p critical missing da ta. A pla n to

develop the criteria d ocument then is drafted a nd circu lated to the CAS AC fo r review an d comm ent; it

also is made available to the public for comment. The plan is revised as needed with the planned approach

widely disseminated within EPA. This approach to planning the AQCD appears to be a good one. The

appare nt close working re lationship with CASA C and the inpu t from the NRC appear to be qu ite

beneficial. One improvement to the planning process identified by the Subcommittee entails the active

participation of O AR d uring the pro blem formulation phas e of the risk assessme nt process. In particu lar,

the BOSC encourages in depth discussions between NCEA and OAR staff at the early stages of the

assessment process, regarding the possible structures for the PM standard a nd the type of research

and risk characterizations required to support the rulemaking fo r the different structures. This

now occurs at the end of the AQCD cycle, during the development of the Research Needs docume nt (e.g.,

U.S. EPA, 1998). Thus, should an improved structure be identified that is not sufficiently supported by

research and the AQCD, it will no t be implemented un til the subsequ ent AQCD cycle. NCEA also

should review its approach to consider other related issues during the problem f ormulation phase

of the risk assessment process. In this regard, the Subcommittee refers the Agency to a recent NRC

publication o n this topic (Stern and Fine berg, 199 6).

A second area where improvement is needed involves consideration of health effects from PM exposures

in the contex t of other co-occurrin g pollutants suc h as ozone. To the extent feasible, the BOSC

recommends establishment of a process whereby the PM reviews are conducted with reviews of

other pollutants. This process will ensure that analysts consider exposures to PM in the presence of

other important h azardou s and crite ria air pollu tants. Th e research and da ta analysis p lans reflect th is

broader focus.
Chapter 6: Report of the Assessment Subcommittee 35

BOSC Management Review of the PM Research Program

AQCD Development

Curren tly, the Age ncy is taking special car e to coord inate the tim elines for re lease of resu lts from its

extramural and intramural research programs and AQCD revisions. The timeline for development of the

AQC D, alon g with th e large number o f entities inv olved, impose limitations on efficient u se of gran t,

contract, and E PA research resources. The BOSC recommends further strengthening of the

connection between NCEA, other pa rts of the ORD research prog ram, and other research

resources to further improve the AQCD development program.

Each chapter of the current PM AQCD is authored by senior scientists from NCEA, academia, and EPA

Laboratories, many of whom have specialized and exten sive expertise in the chapter topic. The BOSC

strongly endorse s the use of such talen ted and experienced scientists in the drafting of the A QCD . 

As detailed below, the BOSC believes the maintenance of critical scientific expertise within the

Agency is essential and that steps are needed to ensure continuing scientific excellence in AQCD

develop ment. EPA no ted that rather than providing the encyclopedic literature reviews of some past

AQC Ds, it develop s an analytic review foc used on critical scien tific issues for the health an d dosimetry

characte rizations. The BO SC ag rees with this app roach an d notes th at this un derscore s the nee d to

maintain a h ighly talented and experienced staff. 

The analysis and distillation of data that must occur in characterizing PM dose-effect relationships and

hazard s involve s consid erable scie ntific exp ertise and judgment. To m ake this p rocess more transp arent,

NCE A shou ld clearly articulate and commun icate the process u sed in cond ucting such analyses. 

Peer Review

Peer review of Agency work products and plans occurs at many levels and points in the AQCD

development cycle. EPA engages both formal and informal peer review, depending on the nature of the

document or issue. As is Congressionally mandated, the AQCD undergoes formal peer review through the

CASAC, a standing committee of the EPA Science Advisory Board. Depending on the topic, the

committee is augmented by consultants, with some of the same consultants across reviews to achieve

continuity. In addition to the AQCD, at the Agency’s discretion, other documents relevant to the

establishment of air quality criteria are reviewed by CASAC, including the AQCD development plan,

Research Needs, and the Staff Paper. The public also has the opp ortunity to comment on these

docum ents; scien tist peers in th e regulate d and public in terest commun ities freque ntly exercis e this

opportunity. In addition to formal CASAC reviews, the Agency conducts workshops on the individual

AQC D cha pters and provide s further o pportu nity for pu blic comment. F inally, the A gency activ ely

sponso rs other form s of scientific peer review—it conven es scientific w orkshop s and symposia, su pports

staff scientists to attend a nd pre sent their P M wo rk at other sc ientific me etings, inf ormally req uests

review of work in progress by pe ers, and cond ucts intramura l lecture series. 

The BO SC commends E PA for its app roach to peer rev iew of PM research and risk assessment p roducts. 

It also notes the impo rtance of ensu ring high-qu ality reviews. The BOSC encour ages the A gency to

rema in vigilant in its selection of v enues an d review ers to ensure that fa ir, objective, and in-depth

reviews occur both in fact and a ppearance. 

Record of AQCD

Impressively, NCEA maintains an extensive, readily accessible record of AQCD development and

proceedings. This includes indexed copies of all publications relied upon in the AQCD, all comments and

letters submitted to the Agency on PM AQCDs, and responses to these comments and letters. The
36 Chapter 6: Report of the Assessment Subcommittee

BOSC Management Review of the PM Research Program

synthesis of finding s and the referen ces (not the pu blications) relied up on in AQ CD d evelopmen t are

made availab le on CD ROM . 

Resources

The Agency has con siderable exp erience in assessin g PM ex posures and health effects. Staff members

are quite capable, some with extensive expertise in PM assessment and with excellent publication

histories an d stand ing in the scientific commu nity. It is essentia l to mainta in and b uild up on this sc ientific

capability to maintain the quality of the AQCD . 

More effective use of post-doctorate appointments is desirable. With 3-year appointments and the

possibility of subseq uent hiring, they p rovide an excellen t mecha nism for bringin g in youn ger capa ble

and highly trained staff to provide the scientific leadership in the AQCD program of the future.

The BOSC recommends that the Agency continue to support scientific work and professional

development of staff. Coop erative resea rch and development ag reemen ts are a good mecha nism to

enhan ce expe rtise of existin g staff. Cu rrently, staff ap pears to b e functio ning u nder tigh t time con straints

and a large workload, and additional staff is clearly desirab le. Admin istrative and man agement su pport

are needed to recruit able young investigators into the gro up. EPA also shou ld attempt othe r remedies. 

Greater rotation of Laboratory staff into the criteria de velopmen t group is need ed, but ma y require

induc ements to convin ce Labo ratory staff to jo in in the e ffort. An other su ggestion to increase the criteria

group’s scien tist capabilities is through joint appoin tments between the Agency and u niversities. 

2. How are the results from the intramural and extramural research programs integrated

into the assessment?

Like ORD in general, NCEA depends upon workshops, organizational meetings, conferences (a lot of

discussion occurs at poster sessions, which have a laboratory-level focus), reports (briefings and premeeting notes report), e-media, and other means of exchange for generating or obtaining research

information. ORD is depending heavily upon the Web-based inventory of research that HEI has

developed in collaboration with EPA. This inventory is now available online at http://www.pmra.org.

ORD also depends on many of these same methods to integrate research into the assessment. For

example, the process of developing the AQCD heavily involves outside expertise through professional

societies. Papers from the PM 2000 conference held by the Air and Waste Management Association

(AWM A) with sponsorship b y EPA and others, will undergo expedited peer review by several journals,

becau se EPA primarily relie s on pee r-reviewed articles in th e develo pmen t of the AQCD . The jo urnals

include, for example, the Journal of Air and Waste Management (JAWMA), Inhalation Toxicology, and

the Journal of Exposure Analysis and Environmental Epidemiology.

When the staff members were asked how they actually distill all of this information into the AQCD, they

respon ded tha t expert ju dgme nt is used in part, an d some times, for ep idemio logical stud ies, meta-an alysis

is used. NCEA und erscored the fact that they are acquiring new statistical methods to evaluate studies

better. 

To ens ure that se conda ry data inco rporated into the A QCD is of highe st quality, NCEA consults with

individual authors and tries to acquire the und erlying data to verify the published findings. Getting access

to the data, however, often has proved difficult. They cited the example of the attempt to obtain data from

the Harvard six citie s study. Ultimately, the data h ad to b e given to an in depe nden t third p arty, HE I,

which then eva luated th e data ind epend ently for EP A. En gaging a third pa rty for indep enden t review is
Chapter 6: Report of the Assessment Subcommittee 37

BOSC Management Review of the PM Research Program

clearly preferable to hav ing no review, but in genera l, the BOSC recommends the establishment of a

clearer data exchange protocol to enable NCEA to verify the results of outside studies as needed.

A major NCEA product is the Research Needs for risk assessment that takes into account intramural and

extramural research and external inputs from meetings, workshops, and conferences. The latest NCEA

report is entitled, “Particulate Matter Research Ne eds for Human Health Risk Assessment to Support

Future Reviews of the N ational Ambient Air Quality Standard s for Particulate Matter” (U.S. EP A, 199 8). 

The report serves as an intermediary between the research to be conducted and its integration into the

criteria document and assessment. The NCEA Director indicated that they plan the Research Needs

report 2 years prior to the preparation of the AQCD so that it can explicitly provide inputs to the

document. In the case of PM, the 1998 Research Needs document will be an input into the Year 2000 PM

AQCD . The Director indicated that the research needs assessment is vetted by the CASAC. He noted that

the research ne eds identified affec t funding d ecisions of other gro ups, such as HEI and NARSTO. 

EPA’s intramural and extramural PM research seems to work hand in hand. That is, much of the

intramural research appears to link to an extramural research project. Intramural and extramural research

are guided by the priorities outlined by the NRC (1998, 1999). The NPM, Dr. Vandenberg, pointed out

that after the PM standards were established, Congress doubled, at the expen se of other EPA program s,

the PM budget in 1 998 an d 1999 , but required EPA to contract with N RC to iden tify priorities for nearand long-term research. The 1998 and 1999 reports are two of what will be five NRC publications. So

there is an expectation that EPA’s work on PM will be aligned with what the NRC recommends and

discussions with EPA staff suggest this will be the case. It is within this overall context that intramural

and ex tramura l research is condu cted, and as such , is the con text with in whic h NCEA syn thesizes th is

research in the AQCD.

That th e NRC repo rts shape ORD research in general a nd the r esearch u pon which NCEA relies in

particular, was illustrated by an example provided by Dr. Vandenberg. The first NRC report, he pointed

out, was very c ritical of the fac t that particul ate mo nitorin g was n ot sup portin g the re search community.

As a result of this observation, the OAR and ORD became more formally integrated. They adopted an

adaptive management style—a team matrix approach that already existed within ORD. This group was

able to make monitoring design recommendations for the Supersites Program (characterization of ambient

environment in five to seven locations throug hout the cou ntry). 

Intramural Research

ORD’s internal grants program is the major vehicle for generating research by staff within NCEA. The

NCEA Director, Dr. Farland, indicated that overall 40 percent of NCEA’s time is spent upon internal

grants. The Director indicated he reserves 10 percent of his research funds to the internal grants program.

This is his discretionary decision. Of the $15 million for grants, $1.5 million is reserved for internal

grants (for all program areas). Of the $1.5 million, $500,000 is for PM research. The Director believes

that internal grants keep the staff sharp, and develop their resources. Discussions with a recent recipient

of one of these grants indicate that this is a worthwhile practice.

Extramural Research

NCEA has a variety of different sources for generating extramural research. The STAR program and the

Laboratory-based research center grants are two examples. Interagency agreements are another means of

generating extramural research. The ORD staff indicated that EPA worked with the National Institute of

Allergy and Infectious Diseases (NIAID), for example, on the inner city asthma study. ORD also can use

cooperative agreements with universities.
38 Chapter 6: Report of the Assessment Subcommittee

BOSC Management Review of the PM Research Program

These programs can provide the basis for integrating research into the assessments, because they

potentially can ena ble a large amou nt of interaction be tween intern al research staff and researchers

externa l to the agen cy. Wh ether this in teraction is a ctually occu rring wa s difficult to a scertain ex plicitly

for NCE A. 

NCE A inco rporates e xtramu ral research in gener al into the A QCD in a series of stages inv olving a p ublic

airing of th e Cente r’s work . First, wor kshop s are held at which researche rs have th e oppo rtunity to

provide inputs. Second, the criteria document development plan is produced, which identifies what

NCEA will inc lude in the AQCD. This is provide d to the pub lic and the CASA C for c omment. In

addition, NCEA takes draft chapters through peer-review workshops and public comment. No problems

with this process were identified. 

The STAR Program

The ST AR p rogram is the largest reso urce commitment to EPA’s PM program oth er than the ORD staff,

and it is targeted to complement ORD’s capabilities. For the PM Program, the Requests for Applications

(RFAs) also are designed to address key hypotheses. In addition to contributing to the specification of the

RFAs, NCEA also is involved in the relevancy review of STAR grant proposals. These interactions

should b e helpful in p romoting STAR research that add s value to NCEA’s w ork. Wh ether or not they are

sufficient is still an open question.

The STAR research s uppo rts many as pects of th e PM program. While it was stated that the STAR grants

generally do not appear to provide a great deal of support to meeting risk assessment research needs, two

examples of valuable ST AR gran ts were described . One ST AR gran t produced instruments to m easure

the semi-volatile and non-volatile components of particulate matter. These instruments were used in a

follow-on critical study of h eart-rate variability in dogs. A se cond ST AR gran t was awarde d to Harvard

University to contin ue development of a pa rticle concentrator.

Five research ce nters have ju st received fund ing from ORD STAR grants: Harvard , New Y ork

University, the University of Rochester, University of Washington, and a consortium led by the

University of Southern California. It is too early to tell the value of the Centers’ contributions to the ORD

PM a ssessme nt. How ever, the E PA Laboratories and Centers, inc luding NCE A, hav e had co nsiderab le

input into the preparation of the RFAs. For this reason NCEA believes there is a high probability that the

grant research w ill be useful for the PM AQ CD. As the STAR funded research proceeds at the five

university centers, ORD should develop a process to actively interact with them and to ensure that

the research prod uced is useful to the a ssessment pro cess.

Staff Input

In discussions with the staff, some areas of ongoing research that could not be fully integrated into the

current assessments were evident: (1) the role of co-pollutants and their modulation of PM levels and

health effects, (2) the role of co-pollutants in model selection (NCEA currently has a cooperative

agreement with the University of Washington National Research Center for Statistics and the

Environment on this problem), (3) exposure measurement errors, and (4) methods to relate pollutant

concentrations with sources and health effects. Some of these were valuable examples of how research

needs from the 1997 AQCD were identified and carried forward for the Year 2000 AQCD. However, due

to the time between the initiation of research and the publication of results, the research may not impact

this cycle of criteria document development, bu t would be available for a future A QCD . 
Chapter 6: Report of the Assessment Subcommittee 39

BOSC Management Review of the PM Research Program

3. Have the advice/recommendations contained in the BOSC Program Management

Review of NCEA been incorporated into the PM Program?

The NCE A self-study report identified and addressed five recommendations in the BOSC Program

Management Review of NCEA. The Sub committee compared these to the list in the Management Review

and found that they matched the general areas covered in the recommendations with the exception of

benchmarking, which was not addressed. How each of these five recommendations have been

incorporated is described below.

Improved integration and alignment of NCEA’s strategic plan and activities with the direction

and priorities for the ORD Plan and the better communication of priorities to NCEA staff.

A draft NCEA Strategic Plan was prepared in early October 1999. It is now being reviewed internally and

will be revised. NCEA staff indicated that NCEA strategic directions in the plan are linked closely to the

ORD Strategic Plan 2 000 an d to program-specific strategic plans (e.g., ex posure). It identifies futu re

human resource need s. 

NCE A has c learly mad e PM research o ne of its hig hest, if not its highest p riority. This is r eflected in its

allocation of h uman resources. A do zen fu ll-time st aff primarily dev ote the ir time to the PM AQCD. It

also is reflected in the numerous activities it has underway as part of the PM process.

An N CEA Web Site also h as been develop ed. Th is will facilitate c ommunicatio n of strateg ic priorities to

NCE A staff and ex ternally. 

Increased un derstanding of client needs.

NCEA appears to maintain a close relationship to what it regards as its key client, the OAR. See

recommendations u nder Question 1 for enh anced interactions.

Improved project planning and tracking of progress toward accomplishing goals meeting client

needs.

NCEA has developed a three-tiered system for tracking the progress of individual documents, where the

tiers reflect the level of imp ortance o f the doc umen t.

The system of the Government Performance and Results Act (GPRA) goals and objectives also is adding

rigor to the NCEA system of tracking progress. According to the PM Program Manager, monitoring of

progress is expanding rapidly. A process is being developed by which EPA commits to an annual list of

performance goals and performance measures. EPA is going beyond the 2-year GPRA period to 8

years—the p eriod neede d to achieve b oth standard s setting and stan dards imp lementation. The BOSC

supports the attem pt to exp and th e GPRA p lannin g perio d to a m uch longer p eriod, b ecaus e it

encourag es a more effective use of resou rces.

GPRA ’s major goal is clean air (GPRA goals are outcome oriented). GPRA annual reports for the Agency

are required. The first one is due in March 2000.

The timeline for the PM AQCD is well specified. At least two of the following PM milestones are GPRA

subgoals; therefore, these milestones are likely to be closely monitored:

External Review Draft of AQCD: October 1999

CASAC Review Public Meeting: December 1999
40 Chapter 6: Report of the Assessment Subcommittee

BOSC Management Review of the PM Research Program

Second External Review Draft: June 2000

CASAC Review: September 2000

Revised PM AQCD: December 2000

The timelin e for the PM regulatory activity also is clear:

2000: Revised PM AQCD

2002: NAA QS decision

2005: Revised PM AQCD

2007: NAA QS Decision

2010: PM2.5 Standards Implemented

Given the relatively frequent cycle for criteria dev elopmen t and the mu ltiple criteria, the current staff

resources and the depth of the criteria document, the extent of deliberation on scientific issues may

become increasingly jeopardized.

Integration an d comm unication w ithin and acro ss NCEA , ORD, the Agency, f ederal agencies,

and the broader general scientific community.

Within NCE A Un its

NCE A has th ree divisio ns, each at differen t geograp hic locatio ns: Washingto n, DC, RTP , and Cincinn ati.

Each location has a director. Re source need s are generally shared among th e locations, however,

resources that augment staff are concentrated in Washington, DC. For examp le, all of the American

Associa tion for the Adva nceme nt of Scie nce (AAAS ) Fellows are located in DC. The Subcom mittee did

not uncover any particular problems associated with communication among the various divisions. The

Subcommittee was unable to fully explore the issue of the allocation of scarce resources among the

different locations.

Across ORD Units

The PM P rogram Manager emphasized that ORD works through teams, and illustrated the large number

of teams within the Office. NCEA, however, was not listed as a participant on any of the teams

mentioned . The BOSC recommends that the PM Program Manager reconsider the team

comp osition to determine if greater N CEA involvem ent wo uld be v aluable. However, the PM

Program M anager conv enes weekly telep hone communication s, and several N CEA staff membe rs are

present at those meetings.

An important e xamp le of interac tion and communicatio n amon g the EP A Lab oratories oc curs with

respect to the temp orary transfer of person nel amon g the Labora tories as needs arise. 

Within the Agency

NCE A inco rporates g eneric ex posure informa tion into th e AQCD. O AQP S, wh ich is officially

responsible for the PM Staff Paper, develops a separate exposure assessment document. NCEA an d that

regulatory office appear to work closely. Indeed, given the close working relationship between NCEA

and OAQ PS in the preparation of the Staff Paper, opportunities for staff exchanges between these offices

should be explored. Early consideration of emerging regulatory issues in the development of the Research

Needs papers and the AQ CD could b e advantageous and w ould be facilitated by such exchanges.
Chapter 6: Report of the Assessment Subcommittee 41

BOSC Management Review of the PM Research Program

Across Federal Agencies

NCEA is a major participant, through its Director, in CENR, which consists of representatives from

numerous federal organizations, many of whom apparently have no history of interacting. The U.S.

Department of Energy (DOE), National Oceanic and Atmospheric Administration (NOAA), and EPA

representatives report to the ORD Assistant Administrator (AA) through the Air Quality Research

Subcommittee, and the NCEA Director is the Executive Co-Chair of the PM Research Coordination

Working Group of the Subcomm ittee along with a representative from the National Institute of

Environm ental Health S ciences (NIEHS). 

The Scientific Community

NCEA interacts with the broader scientific community through many of the mechanisms identified earlier

unde r Question 2 ab ove for ide ntifying an d integra ting resear ch into assessmen t.

Acquiring app ropriate h uma n resour ces and e xpertise to en sure app ropriate h igh qua lity

scientific inputs to NC EA assessm ents.

NCEA obtains human resou rces and expertise for its work from a numb er of different sources.

The Director indicated that NCEA has a total staff of 180—100 are located in Washington, DC (of which

25 are within the Director’s office); 35 are in the RTP office under the RTP Division Director’s direction,

roughly split between assignments to the Criteria D ocumen t program an d Air Tox ics; and abou t 30 are

located in Cincin nati.

A team of appro ximately a dozen full-time staff memb ers works on the A QCD along with others o n detail

from oth er EPA Labora tories and unde r Interagen cy Person nel Ag reemen ts (IPA). H oweve r, this grou p is

not working entirely on the PM AQ CD. They also are devoting some time to working on the carbon

monox ide AQCD. W hen the list of auth ors and con tributors to the AQCD are counted , the numb er more

than doubles. The production of the AQCD is concentrated in RTP under the RTP Division Director. He

can draw u pon staff in NCEA ’s Wash ington, DC, and Cin cinnati offices if necessa ry. More incentives

are needed to ensure that staff members from the different NCEA divisions are drawn upon by

individual divisions when needed.

During interviews with staff in the Environmental Criteria and Assessment Office and NCEA Director

Farland, a shortage in human resources became evident. Staff work on multiple assessments. In some key

areas, ther e is insuffic ient in-ho use exp ertise, so the expertise must b e obtain ed exter nally. Alth ough in

general staff typically is working on a number of different thin gs at one time, the P M effort involv es staff

predominantly dedicated to the PM AQCD at this time. However, given the timeline and the extent of

work required, the effort appeared to be understaffed.

In addition to deficiencies in the numbers, particular expertise deficiencies are apparent, requiring the u se

of external experts as main authors. For example, there are only a few epidemiologists within NCEA, and

there is very limited expertise in immunological effects. Extramurally, ORD tries to target the STAR

investigators to complement intramural capabilities. As mentioned previously, the STAR program is the

largest research commitment ab ove and b eyond the resou rces supportin g NCE A’s staff. 

When asked about what areas of expertise require reliance on external resources, the Director responded

that it varies for different documents. For PM, much of the epidemiology is conducted externally. The

internal expertise in epidemiology is insufficient to cover key areas and important pollutants. As a general

management issu e, the D irector b elieves that the main discip lines ar e cove red, su ch as to xicolo gy,
42 Chapter 6: Report of the Assessment Subcommittee

BOSC Management Review of the PM Research Program

exposure assessment, an d epidemiology, but the cove rage is not deep enough . The scientific staff,

therefore, is in part doing the work and in part managing professional services agreements for the outside

contractors to supplement NCEA expe rtise. In certain key areas, such as epidemiology, greater in-house

expertise would be ben eficial. However, it must be ac knowledged that epid emiolog ists need to , in

general, conduct epidemiological investigations to retain their expertise. Thus, if they are only relegated

to workin g on criteria documents, ove r time their e xpertise w ill diminis h. An alogous situations will app ly

to some of the oth er scientific disciplines. The A gency is encou raged to explo re solutions to this

dilem ma, such a s joint appoin tmen ts betw een ac adem ic institutio ns and the EPA cr iteria

developm ent group o r between this group and EPA research progr ams.

Contracting

For the preparation of AQCDs in general, 40 percent of the staff are from outside NCEA; for the PM

AQCD, 30 percent are outside NCEA. Clearly, there is a greater emphasis on in-house expertise for the

PM AQCD . The Director stated that critical issues are better defined by in-house staff, and the

Subcommittee agrees, especially given the short time period for this cycle of the PM AQCD

development. Most of th e budget, h owever, is un derstandab ly directed toward P M research—the d ollars

primarily go outside o f NCE A’s Environm ental Criteria and A ssessme nt Office (ECA O). A r elatively

small au gmen tation of th e criteria dev elopment bud get to incre ase staff size b y 5-10 ind ividuals c ould

potentia lly render sig nificant improvements to the criteria prog ram in general, an d the PM pro gram in

particula r. A relative ly small redir ection of e xternal monies to th e criteria pro gram would red ress this

need.

The staff clearly indicated a need for direct, external contracting of well-defined and circumscribed tasks.

They believe that contracting is the only way to get a specific answer on a specific date. Over the past 10

years, a dramatic shift has occurred to move away from contracting and toward grants and cooperative

agreements. Thus, the Agency has apparently less control for particular defined tasks. Although the

Subc ommittee was u nable to fully explo re this issue , it notes the value of h aving in place an easily

applied contracting mechanism.

Obstacles to contracting were noted. Becau se scientists’ expertise is not fully utilized when they must

primarily ov ersee con tract work , contract sp ecialists are n eeded to serve as Contractin g Officers, with

senior scie ntists servin g as advis ors and providin g genera l oversigh t for techn ical conten t.

Staff Transfers Between the Lab oratories and NCEA (Temporary Assignments)

It would be helpful to rotate, on specialized assignments, more people in from the EPA research

Laboratories to the NCEA AQCD development effort to take advantage of the specialized, scientific 

expertise within the Laboratories. 

Post-Doctorates

NCE A doe s not hav e many p ost-doctor ates. In gen eral, NCEA an d the ECAO at RTP have difficulty

attracting q ualified p ost-doctor al candid ates with th e necessa ry expertise . In a recen t attempt to recruit

post-doctorates, NCEA received 40 applications of which only 2 were legally qualified, that is, they had

finished their Ph.D. and were an American citizen or legal immigrant. It is difficult to bring in foreign

applicants because it must be established that the candidate is uniquely qualified to fill the specific area of

expertise needed.

The sta ff indicate d that NCEA has a total o f only four to five post-d ocs. Th ree are in th e DC office, one is

in Cincinnati, and one is in RTP, primarily involved in the PM issue. Post-doctorates are attracted to the
Chapter 6: Report of the Assessment Subcommittee 43

BOSC Management Review of the PM Research Program

research Labora tories mor e than N CEA where th ey can con tinue the ir academic research . This ag ain is

an area to consider creative partnerships with laboratories and universities and working with the

administrative staff to ov ercome institution al barriers to implement promisin g solutions. The

Subco mm ittee recom mends the eff ort and use of crea tive mechanism s to obtain more postdoctorates be continued and expanded.

CONCLUSIONS AND RECOMMENDATIONS

Strengths

The Subcommittee was impressed by the magnitude and diversity of NCEA’s work and the importance

and very central position of its function given its limited resources. The BOSC also was impressed by

NCEA’s track record in producing high-quality criteria documents. The overall planning process for the

PM AQ CD, including the mechanisms for incorporating research in the document, were considered good

approaches.

The Sub comm ittee regarded as a stren gth the fact th at the D irector p lays suc h a cen tral role in Agen cywide and inter-Agency work in the PM area through the CENR. Within ORD, the NPM and the NCEA

Director seem to have a close working relationship on several committees and with respect to other

aspects of the matrix management system.

The Subcommittee found that NCEA has made considerable progress in incorporating the advice and

recommendations contained in the BOSC Program Management Review of NCEA into the PM program.

NCEA has developed a strategic plan, made PM research a very high priority, and developed

communicatio n mech anisms in the form of the Web site. It ma intains a c lose relation ship to its k ey client,

OAR, and has developed a m echanism to track its progress on criteria documents.

Recommendations

AQCD Development and Review Mechanisms

In depth discu ssions between NCEA and O AR staff at the early stages of th e assessment pr ocess

are encouraged, regarding the po ssible structures for the PM standard and the type of research

and risk chara cterizations required to support the rulem aking for the different structures.

NCEA should review its approach to consider other related issues during the problem

formu lation phase of the risk assessment p rocess.

To the extent feasible, the BOSC recommends establishment of a process whereby the PM

reviews are con ducted w ith reviews of o ther pollutants.

The BOSC recommends further strengthening of the connection between NCEA, other parts of

the ORD research program, and other research resources to further improve the AQCD

development program.

The BOSC strongly endorses the use of talented and experienced scientists in the drafting of the

AQCD. The BOSC believe s the m ainten ance o f critical s cientific expertis e within the A gency is

essential and that steps are needed to ensure continuing scientific excellence in AQCD

develop ment.
44 Chapter 6: Report of the Assessment Subcommittee

BOSC Management Review of the PM Research Program

The BOSC encour ages the A gency to rema in vigilant in its selection of v enues an d review ers to

ensure that high-quality reviews occur both in fact and appearance.

The BOSC recommends the establishment of a clearer data exchange protocol to enable NCEA

to verify the results of outside studies as needed.

The techniques that are being developed to exp edite the synthesis of scientific information, such

as the advanced statistical techniques that NCEA described, should be watched carefully and

implemented as a m eans to expedite reviews where inform ation is scarce.

Resources

The Su bcom mittee reco mm ends the e ffort an d use of creative m echanism s to obtain more postdoctorates be continued and expanded.

The BOSC recommends that the Agency continue to support scientific work and professional

development of staff through cooperative research and d evelopment agreem ents and greater

rotation of Laboratory staff.

The BOSC supports the attempt to expand the GPRA planning period to a much longer period,

because it encou rages a m ore effective use of resources.

As the STAR funded research proceeds at the five university centers, NCEA should develop a

process to actively interact with them and to ensure that the research produced is useful to the

assessment pr ocess.

More incentives are needed to ensure that staff from the different NCEA divisions are drawn

upon by individual divisions when needed.

The BOSC emph asizes that p roactive m eans be e xplored to expan d the resou rces targeted to

PM within NCEA. This is critical given the scope and centrality of its responsibilities for

producing the AQCD and related materials. NCEA synthesizes an enormous amount of

technical information from the other EPA Centers and Laboratories in the process of

producing the AQ CD and inputs to the Staff Paper. It plays a unique role in seeing that the risk

assessment para digm that links the various stages of environmental fate an d transp ort,

exposure, and health effects, is carried forward into the process of estab lishing PM standards.

This can occur through a closer relationship with the newly established PM Cen ters, interLaboratory/Center transfers, greater access to post-doctorates, and through contract work.

This can be accomplished better via improved communication between NCEA and the rest of

ORD and the A gency . The ch anne ls that a lready have been e stablish ed within the ma trix

manag ement system could b e used to enhance such com munication. The S ubcomm ittee

recommends that such improved communications be undertaken.

The AQCD s are regarded w ithin the scien tific commun ity as high -quality scien tific docum ents. 

To some extent this status is threatened by the increasing workload, short timeframe, and

limited ability to recruit and retain new scientific talent. Some augmentation of budget for

modest increase in staffing as well as creative administrative solutions to bring academic and

research scientists to the criteria group on a temporary, periodic, or shared basis should be

given careful consideration.
Chapter 6: Report of the Assessment Subcommittee 45

BOSC Management Review of the PM Research Program

REFERENCES

National Research Co uncil, Nation al Academy of Sciences. 19 98. Research Priorities for Airborne

Particu late Ma tter. I: Imm ediate Priorities an d a Lo ng-Range Resea rch Po rtfolio. Washington, DC:

National Academy Press.

National Research Co uncil, Nation al Academy of Sciences. 19 99. Research Priorities for Airborne

Particu late Ma tter. II: Eva luating Resea rch Pro gress an d Up dating the Po rtfolio. Washington, DC:

National Academy Press.

Stern, PC and HV Fineberg, ed s. 1996. Und erstand ing Ris k. Inform ing Decisions in a Democr atic

Society . Washington, DC: National Academ y Press.

U.S. EP A, 199 6a. Air Quality Criteria for Particulate Matter. Research Triangle Park, NC: National

Center for Environmental Ass essme nt – RTP O ffice; Report N o. EPA/60 0/P-9 5/00 1aF-cF. 3v. NTIS

PB96-168224.

U.S. EP A, 199 6b. Review of the National Ambient Air Quality Standards for Particulate Matter: Policy

Assessment of Scientific and Technical Information. OAQPS Staff Paper. Research Triangle Park, NC:

Office of Air Quality Planning and Standards; Report No. EPA/452/R-96-013. NTIS PB97-115406REB.

U.S. EP A, 199 8. Particulate Matter Resea rch Needs for Hu man Health Risk Assessment to Su pport

Future Reviews of the National Ambient Air Quality Standards for Particulate Matter. Researc h Trian gle

Park, NC: National Center for Environmental Assessment; Report No. EPA/600/R-97/132F.

U.S. EP A, Office of Research and Developm ent. May 19 , 1999. Airborne Particulate Matter Research

Strategy. Research Triangle Park, NC: ORD.

U.S. EP A, Office of Research and Developm ent. Octobe r 1999. Responses to Self-Stud y Questions.

Board of Scientific Counselors’ Review of ORD’s Management of the PM2.5 Research Program. Research

Triangle Park, NC: ORD.

U.S. EPA, Office of Research and Development. October 28-29, 1999. “Particulate Matter Research

Program – Manag ement Overview,” prese ntation slides to the B OSC by John V andenb erg, Ph.D.,

National Research Program Manager. Research Triangle Park, NC: ORD.

U.S. EP A, Office of Research and Developm ent, Board o f Scientific Cou nselors. April 1 998. Program

Review of the National Center for Environmental Assessment (NCEA). Final Report of the Ad Hoc

Subcommittee on the Review of NCEA. Washington, DC: ORD.

U.S. EPA, O ffice of Re search an d Dev elopment, Na tional Ce nter for Environm ental Assessme nt.

“Development of Air Quality Criteria Documents: A Different Approach to Environmental Health Risk

Assessment,” by William H. Farland, Ph.D., Director, NCEA and Executive Lead EPA PM Research

Program. October 28-29, 1999, Research Triangle Park, NC.
46 Chapter 6: Report of the Assessment Subcommittee

BOSC Management Review of the PM Research Program
Chapter 7: Report of the Risk Management Subcomm ittee 47

BOSC Management Review of the PM Research Program

Chapter 7

Report of the Risk Management Subcommittee

Subcom mittee Mem bers: Michael C. Kav anaugh, Ph .D., P.E. (Chair)

Bonnie McC ay, Ph.D. (Vice-Chair)

Anne Sm ith, Ph.D. (Memb er)

Cliff Davidson, Ph .D. (Member)

INTRODUCTION

The Risk Management Su bcommittee was charged with conducting a management review of the National

Risk Management Research Laboratory’s (NRMRL) participation in the PM2.5 research program. The

following report summarizes the findings and recommendations of this Subcommittee.

NRMRL is one of three EPA Laboratories charged with conducting the PM2.5 research program. The PM 

researc hers, fo und in all br anch es of the Air P ollution Preventio n and Con trol Division , are gu ided b y a

PM te am hea ded b y Gene T ucker (rep resented during the man agement review by Dou g McK inney). It is

our understanding that the ultimate goal of the PM2.5 research program is not to generate data or fill gaps

but to advance the understanding of what is necessary to reduce risks to human health and the

environment.

NRMRL’s PM research activities are focused primarily on two research areas: (1) emissions

characterization, and (2) emissions control technology. These two areas provide, in part, information

essential for risk management analyses of regulatory options for control of PM2.5. Our Subcommittee

found it difficult to understand the role of risk management in the PM2.5 research p rogram, and in

particular, the precise role of NRMRL, whose primary mission is conducting risk management related

research programs. The organizational role of NRMRL within the PM2.5 program was well defined in a

management diagram provided to the BOSC, at least in terms of the line structure and the position of PM

research ers within it. Howe ver, risk management has n ot been identified as one of th e 10 priority

research areas by the NRC, which would suggest a modest role for NRMRL in the overall PM2.5 research

effort. Furthermore, previous NRC recommendations on the use of risk assessment in the federal

government, (NRC, 1983) are very explicit on the need to maintain clear distinctions between risk

assessment and risk management activities of federal agencies. Thus, our Subcommittee recognized the

inheren t limitations on insertin g risk man agement conc erns into a research p rogram w hose mission is

driven b y the risk asse ssmen t paradigm. No netheless, as we ela borate in the following discu ssion, it is

our opinion that risk management issues can play a larger role in the PM2.5 Research Program than the

role described to u s during this manageme nt review. 

FINDINGS AND RECOMMENDATIONS 

1. Breadth of Coverage. Our Su bcommittee sees the need fo r a more global co nsideration of lon g-term

research strategies for NRMRL. For example, the research should go beyond the immed iate expertise

of NRM RL perso nnel if such w ork is needed to advance ou r understan ding of risk redu ction. 

Current risk management research generates information on emission characterization and control

technology, wh ich can be u sed by OA QPS a nd the states. We recommend that NRMRL focus on

increasing the scope of information provided in this area, in particular how different PM

constituents may increase or decrease with different control m easures.
48 Chapter 7: Report of the Risk Management Subcomm ittee

BOSC Management Review of the PM Research Program

In response to a q uestion abou t the incorporation of human behavior into th e “holistic” framew ork of 

EPA, we heard various accounts, including that EPA always has been reluctant to venture into an area

that migh t suggest re gulation of private p roperty an d beh avior (suc h as with respect to in door air

pollution ) and th at EPA ’s incorp oration of huma n dime nsions is lim ited to stud ying hum an activity

patterns (with resp ect to exposu re questions). We recom mend that the PM program leaders enlist

team and PI memb ers in rethinking the boundaries of holistic framewo rks, with a view toward

recognizing the human element, even for areas traditionally focused on engineering. To our

minds, risk management also requires improved scientific knowledge of human behavior and social

and econ omic institutions b ecause of their stron g effects on source s, prevention, an d complian ce. 

2. Basis for Setting Research Priorities. The priority-s etting p rocess occurs at two levels: “Agen cy”

and “Laboratory.” Neither has a formal risk-based approach to priority setting. The NRC has

provided guidance on priorities, but the NRC cannot be expected to refine these priorities given the

scope of the NRC review. The staff pointed to the Agency’s “needs document” that is produced

when the criteria document is finalized as a source of information on overall Agency research

priorities. Howev er, this document apparen tly does not formally use risk analysis to set priorities. 

The staff reasoned that at the “Laboratory” level, they have good intuition about important research

needs that the y then communicate to the A gency plann ing process. 

We recommend that “what if” scenarios or sensitivity analyses would be helpful in identifying

benefits and tradeoffs from risk control/managemen t programs. Sensitivity analysis and valueof-information an alysis using a pro forma risk assessment, for exam ple, could create a clearer

linkage between risk man agemen t and research n eeds. This cannot be expected of NRC, but EPA

and its Labo ratories/Centers can engage in more formal analysis of valu e of information . 

3. Stopp ing Poin ts. The stopp ing points for ind ividual research p rojects need to b e better defined . A

danger is that projects, once decided upon, will continue indefinitely. Criteria for evaluating research

success in pro ducing n ew know ledge for better managing risks, an d wheth er and wh en to shift

resources to other areas of research are not evident. We reco mm end tha t the Agency take concrete

steps to continually evaluate the value of continuing individual areas of research once they have

been initiated. A formal value-of-information process will help determine when research has

been sufficient. 

4. Interaction with External Research Groups. Although NRM RL personnel appear to be acquainted

with relevant PM research groups outside of EPA, new opportunities are developing that may

enhance the value of NRMRL research. These include the Research Centers funded by EPA,

especially the Supersites. A key goal of the Supersites Program is to establish research monitoring

sites to characterize chemical composition and sizes of PM, to develop improved PM instrumentation,

and to link PM information with source emissions and health effects data. The new instrumentation

proposed for use at the Supersites will provide chemical composition and size information at high

resolution, greatly enhancing the ability to conduct source-receptor modeling. As suggested by

NRMRL personnel, teaming with the Supersites may provide information on whether the improved

resolution permits significantly better control de cisions. 

5. Accou ntability. Accountability and authority are unclear. One apparent example that we noted is that

the PM team lead ers are said to be held accoun table bu t lack either lin e or bud get auth ority. This

seems to be an awkward management arrangement, although we were told that it works. The team

approach appears to be working despite ambiguous and awkward management structures; numerous

examples of effective integration were offered.
Chapter 7: Report of the Risk Management Subcomm ittee 49

BOSC Management Review of the PM Research Program

6. Communication. Reorga nization of ORD in 1 995 h as mad e a significa nt differen ce to

communication among the various branches, but there is still room for improvement. Staff report that

cooperation is at a historic peak; the slope is in the right direction. The general impression is that

staff seems to be intera cting positively. 

Research coordination with other groups occurs mainly between the “source” and “exposure”

components of the paradigm, such as in a new collaboration between NRM RL and NERL on source

apportionment; and between NRMRL and NHEERL on PM producing toxicity (where NRMRL does

sample extraction, NHEERL does toxicity testing). Although this particular connection came about

in a serendipitous way, rather than as a result of planning and needs assessment, it is apparent that

renewe d efforts h ave been und er way to ide ntify areas of c ollaboratio n between the Laboratories in

this research area. These efforts should at a minimum be maintained.

7. Incentives. We were informed that promotion decisions generally follow the academic model, relying

heavily on publication in peer-reviewed journals and individual development of national and

international scientific reputations. Willingness to collaborate and to modify programs to address

changing priorities may be hampered by narrow application of the academic model for promotion and

advancement. A researcher gains more material for publication through long-term commitment to a

single line of research . If one is rewarded fo r one’s p ublicatio n record , one will n aturally be a verse to

shifting to new lines of research that may have heigh tened priority for the b roader policy proc ess. 

Similarly, researchers may naturally gravitate towards research areas that appear to be fertile ground

for new and publishable findings, and these areas will not necessarily coincide with priorities for

better risk manag ement dec isions. We reco mm end tha t the Agency care fully con sider how to

align incentives created b y prom otional criteria explicitly with goo d program matic priorities,

even if this may m ean lesser adherence to an academ ic view of excellence.

8. Integration and Relevance of NRMRL. More focused involvement of NRMRL in risk management

may be on the horizon, via the planned development of Risk Management Evaluation (RME)

frameworks. PM is one of four pilot RME subjects. RMEs can be thought of as hypothetical

scenario s for differen t control stra tegies, gen erating estim ates of risks. R MEs might b e supp osed to

help clarify the tradeoffs and consequences of choices of control technology and techniques.

Insufficient information was provided to evaluate exactly what NRMRL intends RM Es to be and how

they will function. H owever, RMEs co uld be imp lemented in a way that serves as the beginning of a

concep t to identify th e decision value of re search res ults. We recogn ize that the RME conc ept is still

undergoing development and that applications of this methodology to PM2.5 research will be

exploring uncertain terrain. However, the potential benefits of this approach support aggressive

development. Ther efore, we rec ommen d that the RM E con cept be develo ped a s soon as possible

even in a pro-fo rma o r prelimin ary m anner, to show how the individ ual NR MRL proje cts

wou ld pro vide input fo r an ev aluatio n of th e effica cy of this con cept. It is im portant to do this

even be fore it is possib le to know wha t actual resea rch results w ill be and ju st how results

would be used.

EVALUATION OF RESPONSE TO SELF-STUDY QUESTIONS

1. Describe the role of risk management [research] in directing the priorities of the PM2.5

research program.

At the time the BOSC posed this question, the BOSC was using the term “risk management” in the

broadest sense of decisions that manage risks, rather than in the sense in which ORD has used this term.

ORD is using the term to describe activities aimed at identifying and improving the options available as

tools for managing risks. This semantic distinction has created some confusion. The NRMRL has done
50 Chapter 7: Report of the Risk Management Subcomm ittee

BOSC Management Review of the PM Research Program

its best to resp ond to th e questio n in terms of how its particula r capab ilities and re search res ults help

direct priorities of the PM2.5 research program.

As defined by ORD, research on risk management is quite specific, focused on improving emissions

estimates and the effectiveness of control technology. Most of this is conducted by NRMRL; according

to the ORD response, some is accomplished via extramural research. This extramural research was not

emphasized in our meetings, although it appears that there are Small Business Innovation Research

(SBIR) projects in these areas. We were left with the impression that extramural research was a minor

compon ent, possibly beca use there is little hypothesis-testing or “what-if” ana lysis going on to iden tify

research gaps that they have not filled sufficiently using in-house research capabilities.

NRM RL’s w ritten self-stud y respons e provid es a clear statement of how N RMRL fits in to the ove rall

Agency program and good examples illustrating ways its research products have helped modify the

elemen ts of the ove rall program. Ou r convers ations with NRMRL leaders hip and staff corrob orated th is

statement in their response to Question B.1, making it apparent that the group shares a common vision of

the its roles, responsibilities, and contributions as described in Response B.1. This is a positive indication

that organ izational issu es have b een we ll manag ed and that the gro up can focus on meeting its goals

effectiv ely.

The self-study document and people who participated in our meetings emphasized the PM planning

process a s a med ium for in fluencin g priorities. A team led by Gen e Tuck er, a senior scientist in th e Air

Pollution Prevention and Control Division of NRMRL has responsibility for interfacing with line

management to implement resource allocations decided by the team; the team also “serves as the focal

point for interactions with other researchers and policymakers,” during ORD coordination m eetings,

scientific and tech nical meetings, m eetings with oth er federal agencies, a nd so on. A ccording to the selfstudy document, these interactions are the major way that the risk management research group influences

priorities of the PM Research Program.

Coordination of risk management research depends on the planning and priority-setting that takes place

through the work of the Air Research Coordination Team (RCT). The PM research strategy of May 19,

1999, is under revision in response to review by CASAC. Therefore, we cannot determine to what extent

priorities are actually reflected in risk management research.

Althou gh there appears to be a clea r organiza tional vision , the Risk Man agement Sub committee detects

that the role NRMRL currently is playing in the overall PM2.5 research program is more reactive than

proactive. Evidence of this is the fact that NRMRL’s activities were not addressed in the first list of NRC

priorities and were only partially accommodated in the second iteration, under “Source Characterization.” 

Even there, NRMRL had to extend its range of capabilities to effectively serve this priority need. The

fact that NRMRL adapted to be able to serve this priority need is to be commended. Another major

component of its research strategies, RME (discussed in item 8 in the previous section), is on an apparent

side-line that is not addressed by NRC and not core to the PM Research Plan (see p. 5, February 1999

Progress Report on Fine Particulate Matter, NRMRL). The reason for this lies in the NAAQS-setting

regulatory paradigm, which does not formally consider cost or cost-effectiveness in establishing

standards.

The Su bcom mittee be lieves that N RMRL should strive to build f or itself a m ore centra l role in its

contributions to PM risk research, despite the apparen t constraints of the N AAQS decision process.

For example, NRMRL has the data and core competencies to become a stronger advocate for

conside ration of th e risk-risk trad eoffs assoc iated with PM m ass contro l actions. A s long as th ere rema in

multiple hypotheses about which constituents of PM are potent, and whether co-pollutants may have an

important role in potency, control actions that reduce PM mass in equal amounts may have very different
Chapter 7: Report of the Risk Management Subcomm ittee 51

BOSC Management Review of the PM Research Program

implications for management of health risks. They may alter the mix of PM constituents and copollutan ts in very diffe rent ways a nd also m ay alter exp osure p atterns. NRMRL is in a uniqu e position to

develop the data n ecessary to u nderstan d these tra deoffs, an d to use k nowled ge abou t these trad eoffs to

substantively inform priority setting for research on hazardous components, mechanisms of injury, and

risk assessment for setting the NAAQS themselves. In addition to targeting its RME activities towards

this more central role , NRM RL also should focus on develop ing inn ovative effo rts to comm unicate th is

information effectively so that it can leverage its “risk management research” into an actual role in PM

decision-making, and thus improve the quality of the risk management implicit in those decisions.

The challenge will be to communicate risk management research results in terms that highlight the

relative potential that alternative control actions may actually increase risks even while they reduce total

fine PM mass emitted. Th is will requ ire emph asizing h ow PM control measures a ffect exp osures to

multiple individual constituents within the spectrum of particles, and exposures to co-pollutants that also

are potential culprits. At a minimum, this will demand developing a wider range of metrics than the

typical uni-dimensional estimate of cost-effectiveness at reducing total PM mass, without regard to how

the composition and size distribution of that mass is affected. To be truly effective, however, NRMRL

should perform integrated assessments of the likelihood that health risks would be reduced by various

control methods. Such integrated assessments would combine information about how control measures

alter the composition as well as mass of particles emitted with information on alternative hypotheses

about what is (are) the culprit constituent(s) driving health risks. The emerging concept of an RME

represents an opportunity to move in this direction.

2. Describe how the results of ORD’s PM2.5 research will be used to determine potential

risk management strategies.

A key objective of NRM RL is to provide the information necessary to develop the rang e of potential risk

reduction strategies th at can be imp lemented b y the states to meet the req uirements of th e Clean A ir Act. 

The BOSC asked NRMRL to provide a summary of how research results from the ORD research

portfolio on PM2.5 will be used to guide risk management/risk reduction decisions.

The major link o f NRM RL’s w ork to risk management lies in efforts to imp rove data and too ls used in

decision-making about alternative risk management strategies. The emphasis of NRMRL is more on

providing improved info rmation for modeling and decision-mak ing than on hypothesis-driven research. 

NRMRL has two functions within the current PM2.5 management structure. The first is to lead efforts on

sourc e char acteriza tion. The sec ond is to inform imp lemen tation a genci es abo ut the t echn ical feasibility,

potential cost, and secondary consequen ces of alternative control strategies.

Currently, the primary form of PM2.5 research activities in NRMRL is source characterization. We see

positive signs that this work is proceeding satisfactorily and that PIs are working in an integrated fashion,

across La boratory, Branch, a nd Division bo unda ries. The reasons fo r this integra tion app ear to be p artly

the new ORD organizational structure, partly the leadership of the PM2.5 program and the Branch Chiefs,

and partly the inherent integration bias of the engineering branch o f ORD. W hatever the causes, these

signs are promising.

Examples of these projects explained to us or identified in documents reviewed by the Subcommittee

include:

Emissions characterization (wood stoves, heavy oil combustion, on-the-highway emissions, animal

feeding operations, extramural and intramural projects to fingerprint PM2.5 sources by size and

composition ).
52 Chapter 7: Report of the Risk Management Subcomm ittee

BOSC Management Review of the PM Research Program

Source control (capabilities of various technologies to reduce fine particulate emissions from

stationary and mobile sources).

Projects in support of the second key NRMRL function, providing data for assessment of alternative

control strategies, are less well advanced. The PM team for NRMRL proposes to do several things as

noted in the self-study document and as presented during the breakout sessions; for example:

Sponsor worksho ps summarizing research results.

Publish reports and pap ers.

Produce a series of “integrated o utputs.”

Develop and implement an RM E framework to integrate research results.

Althou gh we a pplau d these e fforts, the statu s of these p rojects is still rath er vague , and lack of specific

details make assessment of the effe ctiveness and utility of this approach d ifficult. How ever, w e strongly

encourage NRMRL, and the PM2.5 senior management to invest the resources necessary to make

this effort a success. NRM RL is ideally qu alified to develop th e necessary sub-models in an RME. W e

see this tool as an essential component in making decisions on future research priorities within the NRC

framework of 10 priority research areas. RMEs can be used for hypothesis testing to determine which

sources m ust be ad dressed for max imum risk redu ction pe r dollar inv ested. Ide ntifying the se source s will

then determine which control technologies need to be improved, or which new technologies should be

developed to meet presumed emission standards. The R ME app roach also allows feasibility and cost

effectiveness analyses of alternative state implementation plans for non -attainment areas.

Although we see many benefits to this approach, we are concerned that insufficient resources are being

directed to the RME initiative. For example, there are no STAR grants addressing the topic. Extramural

fundin g for NR MRL proje cts appe ars limited to control te chnolo gy develo pmen t.

In the Recommendations and Findings Section of our chapter, we have described a very broad role that

we believe the RME concept sho uld take (see item 8 ). In particular, we would like to see RMEs 

focused on informing others about the risk trade-off associated with alternative PM2.5 reduction

techniqu es. By actively informing EPA and states about the many dimensions of risk reduction,

NRM RL can become an adv ocate for more effective risk man agement and s erve a mo re effective ro le in

the PM2.5 research program.

Finally, NRMRL should provide leadership within the PM2.5 research program regarding the

importan ce of hypo thesis testing of alternative com pliance and co ntrol strategies to ensure the m ost

efficient use of research funds. The rec ent letter from EPA’s SAB to Ca rol Brow ner (July 1 999) c learly

captures this theme, and we fully support this approach.

CONCLUSIONS AND RECOMMENDATIONS

We recommend that the PM program leaders enlist team and PI members in rethinking the

boundaries of holistic frameworks, with a view toward recognizing the human element, even for

areas traditionally focused on engineering.

We recommend that “what if” scenarios or sensitivity analyses would be helpful in identifying

benefits and tradeoffs from risk control/managemen t programs. Sensitivity analysis and value-
Chapter 7: Report of the Risk Management Subcomm ittee 53

BOSC Management Review of the PM Research Program

of-information an alysis using a pro-forma risk assessment, for exam ple, could create a clearer

linkage between risk man agemen t and research n eeds.

We recommend that the Agency take concrete steps to continually evaluate the value of

continuing individual areas of research once they have been initiated. A formal value-ofinform ation pr ocess will help determ ine wh en research has been sufficient. 

We recommend that the Agency carefully consider how to align incentives created by

promotional criteria explicitly with good program matic priorities, even if this may mean lesser

adherence to an academ ic view of excellence.

We recommend that the RME concept be developed as soon as possible even in a pro-forma or

prelim inary man ner, to show how the individua l NRM RL p rojects w ould p rovide input to it

and help fill it out. It is important to do this even before it is possible to know what actual

research results will be and just how results would be used.
54 Chapter 7: Report of the Risk Management Subcomm ittee

BOSC Management Review of the PM Research Program
Appendix:

Responses to Self-Study Questions