### UNITED STATES ENVIRONMENTAL PROTECTION AGENCY RESEARCH TRIANGLE PARK, NC 27711

OFFICE OF AIR QUALITY PLANNING AND STANDARDS



## This expert elicitation report was prepared in suppor t of the characterization of uncertainty in EPA's benefits analyses associated with reductions in exposure to particulate matter pollution. As recommended by the National Academy of Sciences, EPA is using expert judgment as part of an effort to better describe the uncertainties inherent in any benefits analysis. This report and its findings are intended solely for this purpose

.

### 1 

## Expa nde d E x pert Judgme nt

Asse ssm e nt of t h e
Co nce n t r ati o n-R e spo n se

R e lationship Between PM 2.5 

## Exposur e a n d Mor t al it y

Final R e port | September 21, 2006

prepared fo r:

Office of Air Qua l ity Planning and Standards

U

. S. E n vir o nm en tal Pro t ectio n A g ency

Research Triangle Park, NC 27711

prepared by:

Industrial Econ omics, Incorp orat ed

2067 Massachusetts A v enue

Cambridge, MA 0

2140i 

### EXECUTIVE SUMMARY



### In 2002, the National Research Council (NRC) report, Estimating the Public Health Benefits of Proposed Air Pollution Regulations , presented the results of an NRC review of U.S. Environmental Protection Agency’s (EPA’s) benefits assessment methods for evaluating proposed regulations of air pollutants. The NRC committee approved of EPA's benefits analysis framework but provided a number of recommendations for improving EPA’s characterization of uncertainty in benefit analysis. The NRC report recommended that probability distributions fo r key sources of uncertainty be developed using available empirical data or through fo rmal elicitation of expert judgments in situations where scientific data are limited or conflicting.

The effect of changes in ambient fine particulate matter (PM 2.5 

### ) levels on mortality constitutes a key component of the EPA's appr oach for assessing potential health benefits associated with air quality regulations targeting emissions of PM 2.5 

### and its precursors. Avoided premature deaths constitute, on a dolla rs basis, between 85 and 95 percent of the monetized benefits reported in EPA’s re trospective and prospective Section 812A benefit-cost analyses of the Clean Air Act (U.S. EPA, 1997 and 1999) and in Regulatory Impact Analyses (RIAs) for rules such as th e Heavy Duty Diesel Engine/Fuel Rule (U.S. EPA, 2000) and the Non-road Diesel Engine Rule (EPA, 2004). Because it is such a large component of benefits, obtaining a good characterization of uncertainties regarding the mortality effects of changes in PM 2.5 

### exposure could capture the largest portion of uncertainty characterization of an entire be nefit analysis (aside from unquantified or unmeasurable benefit endpoints).

In response to the 2002 NRC report, EPA h as taken steps to incorporate the formal elicitation of expert judgments into uncertainty analyses for the benefits of air pollution rules affecting PM 2.5 

### . Since 2003, Industrial Economics (IEc) has worked with a team of EPA and OMB analysts (collectively, the Project Team) 1 

### to conduct two expert judgment studies of the mortality impacts of PM 2.5 

### for EPA’s Office of Air and Radiation (OAR). The first was a pilot study of five experts conducted in 2003 and 2004 aimed at exploring and refining the application of expert elicita tion methods in the context of air pollution policy, and the second, which began in late 2004, was a full-scale study of 12 experts that built on the experience gained from the pilot and incorporated numerous refinements. This report documents the full-scale study of expert judgments concerning the impact of a

1 The term “Project Team” refers to IEc staff and subcontractors, including Dr. Katy Walker, a specialist in expert judgment

elicitations, and Patrick Kinney, an expert in the health effects of PM 2.5 , plus a group of analysts from several EPA offices (Office of Air and Radiation, Office of Research and Development, and Office of Policy, Economics, and Innovation) and from the Office of Management and Budget.ii 

### one μ g/m 3 

### change in ambient, annual average PM 2.5 

### on annual, adult, all-cause mortality
in the U.S.

Expert elicitation uses carefully structured interviews to elicit from each expert his best
estimate of the true value for an outcome or variable of interest as well as his uncertainty about the true value. This uncertainty, e xpressed as a subjective probabilistic distribution of values, reflects each expert’s interpre tation of theory and empirical evidence from relevant disciplines and ultimately his belie fs about what is known and not known about the subject of the study.

Exhibit ES-1 provides an overview of the st eps of the expert elicitation process followed for this study, which includes the followi ng elements: development of an elicitation protocol, selection of experts, development of a briefing book, conducting elicitation interviews, the use of expert workshops pr ior to and following individual elicitation of judgments, as well as the expert judgments themselves.

The expanded PM 2.5 

### -mortality elicitation involved pers onal interviews with 12 health experts who have conducted research on the relationship between PM 2.5 

### exposures and mortality. The elicitation interview consisted of a protocol of carefully structured questions about the nature and magnitude of the relationship between changes in annual average PM 2.5 

### and annual, adult, all-cause mortality in the U.S. The Project Team developed the interview protocol between Oc tober 2004 and January 2006. Development of the protocol was informed by an April 2005 symposium held by the project team where numerous health scientists and analysts provided feedback, by detailed pre-testing with independent EPA scientists in November 2005, and by discussion with the participating experts at a Pre-elic itation Workshop in January 2006.

The 12 experts participating in the study were selected through a two-part peernomination process and included eight experts in epidemiology, three in toxicology, and one in medicine. The peer nomination process was designed to obtain a balanced set of views and serves to minimize the influence of the analysts and sponsors on expert selection. The experts selected for par ticipation are shown in Exhibit ES-2.E X HIBIT ES -1: O V ER VIEW O F THE EXPER T JUDGME NT PROC ESS



# EPA Sym posium

Protocol Pre-testi ng

Post-elicitation Workshop Elicitation Interview s Pre-elicitation

Workshop Briefing Book Development Protocol Development

Expert Selecti on

Expert Judgments

iiiiv EXHIBIT ES-2: FINAL EXPERT LIST

NAME AFFILIATION

Dockery, Doug W. Harvard School of Public Health Ito, Kazuhiko New York Univ ersity School of Medicine Krewski, Daniel University of Ottawa

Künzli, Nino University of Southern California Ke ck School of Medicine (currently at Institut Municipal d'Investigac ió Mèdica (IMIM) - Center for Research in Environmental Epid emiology, Barcelona, SPAIN) Lippmann, Morton New York Univ ersity School of Medicine Mauderly, Joe Lovelace Resp iratory Research Institute
Ostro, Bart D. California Office of Environmental Health Hazard Assessment Pope, C. Arden III Brig ham Young University Schlesinger, Richard Pace University Schwartz, Joel Harvard Sc hool of Public Health Thurston, George D. New York Un iversity School of Medicine Utell, Mark University of Rocheste r School of Medicine and Dentistry



### In January 2006, the Project Team held a Pre-elicitation Workshop with the selected experts. The workshop educated the expert s as to the objectives of the study and the expert judgment process. It also fostered discussion among the participants about the key evidence available to answer questions in th e protocol and provided an opportunity for experts to provide feedback on the protocol prior to beginning the interviews.

The elicitation interviews were conducted between January and April 2006. Each expert was provided a briefing book of reference materi als and a copy of the elicitation protocol prior to the interviews. Each interview l asted approximately 8 hours and covered both qualitative and quantitative questions. The qua litative questions probed experts' beliefs concerning key evidence and critical sources of uncertainty and were intended to make the conceptual basis for their quantitative j udgments explicit. These questions covered topics such as potential biological mechanisms linking PM 2.5 

### exposures with mortality; key scientific evidence on the magnitude of the PM-mortality relationship; sources of potential error or bias in epidemiological results; the likelihood of a causal relationship
between PM 2.5 

### and mortality, and the shape of the concentration-response (C-R) function. The main quantitative question asked each expe rt to provide a probabilistic distribution for the average expected decr ease in U.S. annual, adult, all-cause mortality associated
with a 1 μ g/m 3 

### decrease in annual average PM 2.5 

### levels. In addressing this question, the experts first specified a functional form for the PM 2.5 

### mortality C-R function and then developed an uncertainty distribution for the sl ope of that function (the mortality impact per unit change in annual average PM 2.5 

### ), taking into account the evidence and judgments discussed during the qualitative questions.

When answering the main quantitative question, each expert was instructed to consider that the total mortality effect of a 1 μ g/m 3 

### decrease in ambient annual average PM 2.5 

### may reflect reductions in both short-term p eak and long-term average exposures to PM 2.5


### . Each expert was asked to aggregate the effects of both types of changes in his answers.v 

### Each expert was given the option to integrate their judgments about the likelihood of a causal relationship and/or threshold in the C-R function into his distribution or to provide
a distribution "conditional on" one or both of th ese factors. The interviewers asked each expert to characterize his distribution by assigning values to fixed percentiles (5 th 

### , 25 th



### , 50 th 

### , 75 th 

### , 95 th 

### ). To assist experts in the elicitation process, the interviewers provided realtime feedback during the interviews in the form of graphs and example calculations, using spreadsheet tools and Internet telec onferencing. During the interviews, experts were able to view their responses plotted ont o a distribution using a software interface. They then adjusted their estimates until the distribution represented the views they expressed during the day-long interview.

Following the interviews, the Project Team reconvened the experts for a Post-elicitation Workshop in June 2006. At this workshop, the Project Team anonymously shared the results of all experts with the group, and gave experts the opportunity to raise issues that may have emerged during the interviews for further discussion with the group. The workshop was not intended to promote consensu s, and the Project Team made no effort to encourage experts to change their res ponses. Experts were, however, allowed to modify their responses privately if they chose to do so based on insights gained during workshop discussions, provided they included the detailed rationale for their changes. Four experts made adjustments to their j udgments following the workshop; one of them made changes to the percentiles of his distri bution. (A sensitivity analysis conducted on the results showed minimal impact of these cha nges when compared to the results prior to this workshop.)

In Exhibits ES-3 and ES-4 we display the responses of the experts to the main quantitative elicitation question. The distribu tions provided by each expert, identified by the letters A through L, are depicted as box and whisker plots with the solid circle symbol showing the median (50th percentile); the open circle showing the mean; 2 

### the box defining the interquartile range (bounded by the 25th and 75th percentiles); and the ends of the "whiskers" defining each expert's 5th and 95th percentiles. Each expert's stated best estimate of the likelihood of a causal relationship between PM 2.5 

### and mortality is shown on the x-axis and the experts are arra yed in order of decreasing certainty of causality. Exhibit ES-3 displays the distributions for the experts who chose to provide a distribution conditional on the existence of a causal relationship between PM 2.5 

### and mortality; Exhibit ES-4 shows the distributions for the group who chose to integrate their judgments about the likelihood of causality directly into their distribution. Each figure displays the expert distributions for two different PM 2.5 

### levels, 18 μ g/m 3 

### and 7 μ g/m 3 

### , to observe the implications of four experts' (B, F, K, and L) assumptions about nonlinearities in the C-R function and about differing degrees of uncertainty in the slope of the function across specific ranges of PM. Also, as a point of reference for the results, we include box plots of two epidemiology studies used in EPA benefit analyses – Pope et al., 2002 and Dockery et al., 1993.

2 The mean of the expert’s distributi on was not directly elicited. IEc generated this value using Crystal Ball TM statistical

modeling software.vi Note: Box plots represent distributions as provided by the experts to the elicitation team. When asked, experts in this group preferred to give conditional distributions and k eep their probabilistic judgment about the likelihood of a

causal or non-causal relationship separate.

*Experts' C-R coefficient distribution changes between 7 and 18 μg/m 3

. Expert K specified a threshold (not shown).

Expert L provided two different likelihoods of causality for his C-R coefficient dist ributions at 7 and 18 μg/m 3

, although his distribution appears in the same location in both graphs. 0 0.5

1 1.5

2 2.5

3

E
99% L*
99% B*
98% D
95% I
95% G
70% K*
35% Pope et al., 2002 Dockery et al.,
1993% Decrease in Mortality per 1 ug/m3 PM2.5 Expert C-R Coefficient Distributions at a Baseline Annual Average PM 2.5 of 18 ug/m 3

Key: Closed circle = median; Open circle = mean; Box = interquartile range; Solid line = 90% credible interval

Expert Causality Likelihood

0 0.5

1 1.5

2 2.5

3

E
99% L*
75% B*
98% D
95% I
95% G
70% K*
35% Pope et al., 2002 Dockery et al.,
1993% Decrease in Mortality per 1 ug/m3 PM2.5 Key: Closed circle = median; Open circle = mean; Box = interquartile range; Solid line = 90% credible interval

Expert Causality Likelihood Expert C-R Coefficient Distributions at a Baseline Annual Average PM 2.5 of 7 ug/m 3 GROUP 1: UNCERTAINTY DISTRIBUTIONS FOR PM 2.5 - MORTALITY C-R

COEFFICIENTS; CONDITIONAL ON THE EXISTENCE OF A CAUSAL RELATIONSHIP EXHIBIT ES-3:EX HI B I T E S

- 4 : GRO U P 2: EXPER T UN CER T AINT Y DIST RIBUTIONS F O R TH E PM 2 . 5 -MOR T A L I TY

CO E F F I C I EN T I N C O R P O R A T I N G T H E EXP E R T ' S LI K E L I H O OD O F A C A U S A L

RE L A TIONS H I P

Note: B o x pl ots r e p r esent distr i buti ons a s prov id ed by the exp e rts t o the e licit a t io n t e am. W h e n aske d

, e x pe rts i n th i s

grou p pref erre d t o giv e d i stribut i ons th a t incor p ora t e th eir lik elih o o d tha t the P M 2.5

mo rtali t y ass o ci ati o n may be no n

causal

*E x p erts' C-R co effi cient distributi on c h ang e s between 7 and 18 μg /m 3

. 0 0. 5

1 1. 5

2 2. 5

3

F*
100% C
99% J
99% A
95% H
90% P ope et al.

, 2002 D o c k e ry et al

., 1993% Decrease inMortality per1 ug/m3 PM2.5 E x p e r t C

- R C o e f f i ci e n t D i st ri b u t i ons at a Ba se l i n e A n n u a l A v er a g e P M 2. 5 of 18 ug /m 3

K e y : C l os ed c i r c l e = medi a n ; O p e n c i rc l e = mean; Bo x = i n t e rqu a rt i l e r ange ; S o l i d l i n e = 9 0 % c r edi b l e i n t e r v al

Ex p e r t Ca us a l ity Lik e lihood

0 0. 5

1 1. 5

2 2. 5

3

F*
100% C
99% J
99% A
95% H
90% P ope et al.

, 2002 D o c k e ry et al

., 1993% Decrease inMortality per1 ug/m3 PM2.5 E x pe rt C

- R C o eff i ci en t D i s t rib u t io ns at a Ba se lin e A n n u a l A v er ag e P M 2. 5 of 7 ug / m 3

K e y : C l os ed c i r c l e = medi a n ; O p e n c i rc l e = mean; Bo x = i n t e rqu a rt i l e r ange ; S o l i d l i n e = 9 0 % c r edi b l e i n t e r v al

Ex p e r t Ca us a l ity Lik e lihood

vi iviii 

### Among the experts who provided distributions th at were conditional on the existence of a causal relationship (Exhibit ES-3), median estimates ranged from a 0.7 to 1.6 percent decrease in annual, adult, all-cause mortality per 1 μ g/m 3 

### decrease in annual average

PM 2.5 

### . For five of the seven experts in this group the effect of ultimately integrating their judgments about causality in a benefits assessment will be small, given the high likelihoods of causality they expressed. Median benefits estimated using the distributions of the two experts in this group who expressed greater doubt about a causal relationship will show more significant declines. Among th e experts who directly incorporated their views on the likelihood of a causal relationship into their distributions (Exhibit ES-4), the median estimates ranged from a 0.7 to 1.6 percent decrease in annual, adult, all-cause mortality per 1 μ g/m 3 

### decrease in annual average PM 2.5


### . Because of the different approaches taken by the experts to characterizing their views about uncertainty in the PM 2.5 

### mortality relationship, it is difficult to compare all the distributions. Direct comparison can only be done when the distributions are applied to the same scenario in a benefits analysis. However, certain observations and conclusions can be drawn from these plots and from the experts' responses to the qualitative questions:

• 

### Experts in this study tended to be confident that PM 2.5 

### exposure can cause

premature death. Ten of twelve expert s believed that the likelihood of a causal relationship was 90 percent or higher. The remaining two experts gave causal probabilities of 35 and 70 percent. Recent research in both epidemiology (e.g., Jerrett et al., 2005, Laden et al., 2006) and toxicology (e.g., Sun et al., 2005) significantly contributed to experts' confidence.

• 

### Only one of 12 experts explicitly incorp orated a threshold into his C-R function. 3



### The rest believed there was a lack of empirical and/or theoretical support for a population threshold. However, three othe r experts gave differing effect estimate distributions above and below some cut-off concentration. The adjustments these experts made to median estimates and/or uncertainty at lower PM 2.5 

### concentrations
were modest.

• 

### Experts relied upon a core set of cohort epidemiology studies to derive their

quantitative estimates, mainly those associated with the ACS and Six Cities
cohorts. The Six Cities results tended to be weighted more highly by experts in this study than in the pilot study. Th e greater emphasis on Six Cities appeared to result from corroborating evidence in the r ecent Six Cities follow-up (Laden et al., 2006) and from concerns about potential e xposure misclassification issues and/or effect modification in the ACS cohort (see below).

3 Expert K indicated that he wa s 50 percent sure that a threshold existed. If there were a threshold, he thought that there

was an 80 percent chance that it wo uld be less than or equal to 5 μ g/m 3 , and a 20 percent chance that it would fall

between 5 and 10 μ g/m 3

.ix • 

### Although the quantitative question asked expe rts to consider mortality changes due

both to short-term and long-term PM 2.5 

### exposures, all experts based their median effect estimates on effects due to long- term exposures. Short-term exposure
effects were sometimes used to derive lower-bound effect estimates.

• 

### Confounding of epidemiological results tended to be a minor concern for most

experts. Only one of 12 experts expressed substantial concern about confounding as a source of error in the key literature on PM 2.5 

### and premature mortality.

• 

### Experts’ concerns regarding potential negative bias in the ACS main study results

due to effect modification (see Pope and Dockery, 2006) and/or exposure misclassification (Jerrett et al., 2005; Willis et al., 2003; and Mallick et al., 2002) led many experts to adjust the publishe d results upwards when considering the percentiles of their distribution.

• 

### A sensitivity analysis conducted using a si mplified benefits analysis demonstrated

that no individual expert’s distribution of e ffect estimates had more than a plus or minus 8 percent impact on an overall, pooled distribution of effects. The influence of individual experts appeared symmetrically distributed.

Compared to the pilot study, experts in this study were in general more confident in a causal relationship, less likely to incorporate thresholds, and reported higher mortality
effect estimates. The differences in results co mpared with the pilot appear to reflect the influence of new research on the interpretation of the key epidemiological studies that were the focus of both elicitation studies, more than the influence of changes to the structure of the protocol. They may also reflect differences in the composition of the expert panel. The experts’ distributions in the current study display a similar diversity in the degree of uncertainty expressed as in the pi lot study. The variation in the experts’ responses reflects a number of factors, including differences in their views about the degree of uncertainty inherent in key epidemiological results from long-term cohort studies, the evidential support for a causal rela tionship, and, for one expert, the shape of the C-R function. In almost all cases, however , the spread of the uncertainty distributions elicited from the experts exceeded the statistical uncertainty bounds reported by the most influential epidemiologic studies, suggesti ng that the expert elicitation process was successful in developing more comprehe nsive estimates of uncertainty for the PM 2.5 

### mortality relationship.

### T A BLE OF CONTENTS

E X ECUTIVE SUMMA R Y i



### CH APT E R 1 INTROD UC TION 1-1


### 1.1 Pur pose and Scope 1-2


### 1.2 Relations hip to the Pil o t Stud y 1-3



### 1.2.1 Overview of the Pilot Stu d y 1-3



### 1.2.2 Im prov e m ents in the Full-Scale Study 1-4 

### 1.3 Organization of t h is Do cu m e nt 1-6



### AN AL Y T IC A L AP PRO AC H 2-1


### 2.1 Prot ocol Develop m ent 2-3 

### CH APT E R 2

2.1.1 Structurin g the Proble m 2-3



### 2.1.2 EPA S y m posiu m 2-4



### 2.1.3 Protocol P r e-Testing 2-6



### 2.1.4 Protocol Desig n 2-6


### 2.2 Ex pert Selection 2-10 

### 2.3 Briefing Book 2-15


### 2.4 Pre-elicitation Worksh op 2-15


### 2.5 Elicitation Interviews 2-16


### 2.6 Post-elicit ation Worksh op 2-20


### 2.7 Evaluatin g Experts' Judgm ents 2-21 

### 2.8 Approaches to Presenti ng Results 2-22



### CH APT E R 3 RES U L T S 3- 1 

### 3.1 Responses to Conditi oning Questions 3-1



### 3.1.1 Maj o r Cause s o f and Mechani s m for PM 2.5 

### Rel a ted Mortality 3-2



### 3.1.2 Relativ e I m port a nce of Mortalit y I m pact s fro m Short-Ter m and Long-Term Ex posure Chang e s 3-5



### 3.1.3 Key Epide m iol ogical Evidence 3-8



### 3.1.4 Confoundin g 3-13



### 3.1.5 Effect Modifica tion 3-15



### 3.1.6 Exposure Issue s 3-17



### 3.1.7 Likelihood of a Cau s al Rel a tion s hip Bet w een L ong- and Short-Ter m PM 2.5 

### Ex posure s and
Mortality 3-20



### 3.1.8 Threshold s 3-25 

### 3.2 Responses to the Quantitative Questi on 3-26



### 3.2.1 Shape of the Co ncentration-Response Fun c tion 3-26

x

### T A BLE OF CONTENTS (CONTIN U ED)

3.2.2 Expert Charact erization of Uncertainty 3-28


### 3.3 Post Elici t ation Works hop 3-40



### 3.3.1 Exposure Mi scl a ssification 3-41



### 3.3.2 Effect Modifica tion 3-41



### 3.3.3 Cau s ality 3-42



### CH APT E R 4 DISCUSSIO N 4-1


### 4.1 Sensitivit y Analy s is 4- 2


### 4.1.1 Sensitivity to Individual Experts 4-2


### 4.1.2 Sensitivity to Choice of Distribution a l for m to Characterize Uncertaint y 4-3


### 4.1.3 Sensitivity to Pre-Elicitat ion W o rkshop Participation an d to Change s Made after the PostElicitation W o rkshop 4-4


### 4.2 Com p arison to t h e Pilo t Stud y Findi ngs 4-4 

### 4.3 Evaluation of the Expert Elicitation Process 4-8



### 4.3.1 Strengths 4-8



### 4.3.2 Lim i tatio n s 4-10



### CH APT E R 5 CONCL U SI ONS 5-1



### REF E R E NC ES R- 1

xixii 

### APPENDICES

Appendix A: Elicitation Protocol Appendix B: Briefing Book Materials Appendix C: Sensitivity Analysis

EXHIBITS

Exhibit ES-1: Overview of the Expert Judgment Process iii 

### Exhibit ES-2: Final Expert List iv 

### Exhibit ES-3: Group 1: Uncertainty Distributions for PM 2.5 

### - Mortality C-R Coefficients,

Conditional on the Existence of a Causal Relationship vi 

### Exhibit ES-4: Group 2: Expert Uncertainty Distributions For The PM 2.5 

### -Mortality Coefficient Incorporating The Expert's Likelihood Of a Causal Relationship vii 

### Exhibit 2-1: Overview of the Expert Judgment Process 2-2


### Exhibit 2-2: Structure Of the Elicitation Protocol 2-8


### Exhibit 2-3: Nominators Derived from Publication Count 2-13 

### Exhibit 2-4: Final Expert List 2-14 

### Exhibit 3-1: Conceptual Framework for Short-Term and Long-Term Mortality

Effects 3-6 

### Exhibit 3-2: Relative Importance of Short-Term and Long-Term Impacts 3-7 

### Exhibit 3-3: Epidemiologic Studies Discussed by Experts while Answering Conditioning Questions 3-11 

### Exhibit 3-4: Summary of Experts’ Views on Effect Modifiers 3-17 

### Exhibit 3-5: Summary of Experts’ Views on Exposure Issues 3-20 

### Exhibit 3-6: Likelihood of a Causal Relationship Between PM 2.5 

### and Total All-Cause Mortality 3-24 

### Exhibit 3-7: Expert Judgments Concerning Form of the C-R Function 3-27 

### Exhibit 3-8: Break-Points Delineating Segments of the PM 2.5 

### -Mortality C-R

Function 3-27 

### Exhibit 3-9: Summary of Expert Subjective Uncertainty Distributions for C-R

Coefficients 3-30 

### Exhibit 3-10: Group 1: Uncertainty Distributions for PM 2.5 

### - Mortality C-R Coefficients;

Conditional on the Existence of a Causal Relationship 3-32 

### Exhibit 3-11: Group 2: Expert Uncertainty Distributions for the PM 2.5 

### -Mortality Coefficient Incorporating the Expert's Likelihood of a Causal Relationship 3-33xiii 

### EXHIBITS (CONTINUED)

Exhibit 3-12: C-R Coefficient Distributions for Experts who Specified Non-Linear

Functions 3-36 

### Exhibit 3-13: The Effect of Incorporating a Threshold into the C-R Coefficient Uncertainty Distribution of Expert K 3-37 

### Exhibit 3-14: Epidemiologic Studies Relied Upon by Experts in Creating their C-R Coefficient Uncertainty Distributions 3-39 

### Exhibit 4-1: Percent Change in the Mean and Standard Deviation of an Example Pooled Mortality Benefit Estimate Based on Results of the Expert Elicitations, After Removing Each Expert 4-3 

### Exhibit 4-2: Pilot Study Results - Comparison Of Experts’ Judgments About the Percent Increase in Annual Non-Accidental Mortality Associated with a 1 μg/m 3


### Increase In Annual Average Exposures To PM 2.5 

### (U.S. Baseline 8 to 20 μg/m 3 

### ) 4-5 

### Exhibit 4-3: Pilot Study Results on the Li kelihood of a Causal Relationship Between

PM 2.5 

### and Non-Accidental Premature Mortality 4-71-1 

### CHAPTER 1 | INTRODUCTION

The effect of changes in ambient fine particulate matter (PM 2.5 

### ) levels on mortality constitutes a key component of the U.S. E nvironmental Protection Agency's (EPA's) approach for assessing potential health benefits associated with air quality regulations targeting emissions of PM 2.5 

### and its precursors. Avoided premature deaths constitute, on a dollars basis, between 85 and 95 percent of the benefits reported in EPA’s retrospective and prospective Section 812A benefit-cost analyses of the Clean Air Act (U.S. EPA, 1997 and 1999) and in Regulatory Impact Analyses (RIAs) for rules such as the Heavy Duty Diesel Engine/Fuel Rule (U.S. EPA, 2000) a nd the Non-road Diesel Engine Rule (EPA, 2004). Because it is such a large component of benefits, uncertainties regarding the mortality effects of changes in PM 2.5 

### exposure could have a significant impact on the range of plausible benefit values reported fo r air pollution rules and on the interpretation of the results of benefit analyses.

In 2002, the National Research Council (NRC) report, Estimating the Public Health Benefits of Proposed Air Pollution Regulations , presented the results of an NRC review
of EPA's benefits assessment methods fo r evaluating proposed regulations of air pollutants. The NRC committee approved of EPA's benefits analysis framework but provided a number of recommendations for improving the implementation of that framework, including recommendations for improving uncertainty analysis. The committee recommended that EPA take step s to improve its characterization of uncertainties associated with key components of its health benefits analyses, such as PMrelated mortality effects. The NRC report recommended that probability distributions for key sources of uncertainty be developed using available data or, where scientific data are limited or conflicting through formal elicitation of expert judgments.

In response to the 2002 NRC report, EPA h as taken steps to incorporate the formal elicitation of expert judgments into uncertainty analyses for the benefits of air pollution rules affecting PM 2.5 

### . Since 2003, Industrial Economics (IEc) has conducted for EPA’s Office of Air and Radiation (OAR) two expert judgment studies of the mortality impacts
of PM 2.5 

### ; the first was a pilot study of five e xperts conducted in 2003 and 2004 aimed at exploring and refining the application of expe rt elicitation methods in the context of air pollution policy, and the second, which began in late 2004, was a full-scale study of 12 experts that built on the experience gained from the pilot and incorporated numerous refinements. This report documents the full-scale study of the formal elicitation of expert judgments concerning the mortality impacts of a 1 μ g/m 3 

### change in ambient, annual average PM 2.5


### .1-2 

### In the remainder of this introductory chap ter, we discuss the purpose and scope of the full-scale study, discuss the relationship of this study to the pilot, and provide a road map to the rest of the report.

1.1 PURPOSE AND SCOPE 

### The purpose of this project is to provide a more complete characterization, both qualitative and quantitative, of the uncertainti es associated with the relationship between
reductions in ambient PM 2.5 

### (measured as total gravimetric mass) and mortality. The study was designed to produce results that EPA can apply in its benefit models when preparing future regulatory analyses.

The full-scale study involved personal interv iews with 12 health experts who have conducted research on the relationship between PM 2.5 

### exposures and mortality. These experts were selected through a peer-nom ination process and included experts in epidemiology, toxicology, and medicine. The elicitation interview cons isted of a protocol of carefully structured questions about the nature and magnitude of the PM 2.5 

### -mortality relationship, developed by IEc in consultation with a Project Team of EPA and Office of Management and Budget (OMB) sta ff scientists (hereafter "Project Team"). The protocol included both qualitative and quantitative ques tions. The questions requiring qualitative responses probed experts' beliefs concerning key evidence and critical sources of uncertainty and were intended to make th e conceptual basis for their quantitative judgments explicit. 4 

### The main quantitative questi on sought experts’ probabilistic judgments about the average expected decrease in U.S. adult, annual all-cause mortality
associated with a 1 μ g/m 3 

### decrease in annual average PM 2.5 

### levels. Experts were instructed to consider that the total mortality effect of a 1 μ g/m 3 

### decrease in ambient annual average PM 2.5 

### may reflect reductions in both short-term peak and long-term average exposures to PM 2.5 

### . Experts were asked to characterize the distribution of possible values for the mortality effect (hereaf ter, “uncertainty distribution”) by assigning values to fixed percentiles (5 th 

### , 25 th 

### , 50 th 

### , 75 th 

### , 95 th 

### ). They were also asked for the
minimum and maximum values of the distribution.

The elicitation focused on the concentra tion-response (C-R) function relating PM 2.5 

### mass changes with mortality, and was not intende d to characterize the uncertainty surrounding the role of specific PM components (e.g., diesel particulates) or sources (e.g., power plants) in the PM 2.5 

### -mortality relationship. Also this study did not ask experts to characterize the time sequence of an y mortality reductions following a 1 μ g/m 3 

### decrease in ambient annual average PM 2.5


### .

4 These questions covered topics such as potential biological mechanisms linking PM 2.5 exposures with mortality; key

scientific evidence on the magnitude of th e PM/mortality relationship; sources of po tential error or bias in epidemiological results; the likelihood of a causal relationship between PM 2.5 and mortality, and the sh ape of the C-R function.1-3 1.2 RELATIONSHIP TO THE PILOT STUDY 

### The current study is not an extension of the pilot. Rather, it is a separate study that features a number of refinements intended to improve the elicitation process. It also
reflects more recent research on PM 2.5 

### -related mortality that has become public since the pilot study. To provide some perspective on the current study, we provide below an overview of the pilot study and its results and then discuss the key refinements incorporated into the full-scale study.

1.2.1 OVERVIEW OF THE PILOT STUDY 

### In 2003 and 2004, the Project Team conducte d a pilot-scale elicitation study with five experts to explore the effectiveness of expe rt judgment techniques for characterizing uncertainty and to explore the use of the expert judgment results in the context of economic benefits analysis (IEc, 2004) (hereafter, “pilot study”). In particular, the pilot study was designed to provide feedback on the efficacy of the protocol developed and the analytic challenges, as well as to provide in sight regarding potential implications of the results on the degree of uncertainty surrounding the C-R function for PM 2.5 

### mortality.

Like the full-scale study, the pilot study cons isted of individual interviews featuring a series of qualitative and quantitative ques tions about the nature of the PM 2.5 

### -mortality relationship. One key difference was that the quantitative questions in the pilot protocol asked experts to provide separate uncertainty di stributions for changes in mortality due to long-term exposure (a 1 μ g/m 3 

### decrease in annual average PM 2.5 

### levels) and changes in mortality due to short-term exposure (a 10 μ g/m 3 

### reduction in daily average PM 2.5 

### levels). A detailed report on the pilot study (IEc, 2004); the results of a peer review of the pilot conducted in the summer of 2004; and descriptions of EPA’s use of the pilot results as part of its uncertainty analysis for assessing be nefits of the Non-road Diesel Engine Rule (EPA, 2004), the Clean Air Interstate Rule (EPA, 2005a), and the Guidelines for Best Available Retrofit Technology (BART) (EPA , 2005b) can be found at the following
website: http://www.epa.gov/ttn/ecas/benefits.html

. For changes in annual average PM 2.5 

### levels (which are the focus of the present study), experts in the pilot study estimated medi an values ranging from zero to a 0.7 percent reduction in mortality associated with a 1 μ g/m 3 

### decrease in annual average PM 2.5 

### levels in the U.S. Four of the five experts provided median values of 0.5 percent or less, and the combined mean effect across all experts was 0. 33 percent. Most experts were influenced most heavily by results of the American Ca ncer Society (ACS) cohort, including the HEI reanalysis (Krewski et al., 2000) and the Pope et al. follow-up (2002). Experts placed less weight on the results of the Six Cities cohort (Dockery et al., 1993), despite citing numerous strengths of that study during the in terviews. Experts varied in their level of confidence that the relationship between mortality and long-term PM 2.5 

### changes was causal, with three experts providing probabilities of a causal relationship in the 40 to 50 percent range and two providing probabilities in the 80 to 90 percent range. Two of the five experts incorporated thresholds into their distributions (IEc, 2004).

The peer review of the pilot study was genera lly positive but raised three main issues. First, reviewers recommended that we pr ovide opportunities for communication between1-4 

### experts before and/or after the elicitation in terviews, which was not possible in the pilot due to time constraints. Second, they enc ouraged us to take steps to minimize the potential that experts anchor too closely to individual studies (i.e., anchoring and adjustment bias), which can lead to overconfidence in judgments. 5 

### Third, several reviewers expressed reservations about the reporting of combined (or pooled) expert judgments in the pilot report. 6

1.2.2 IMPROVEMENTS IN THE FULL-SCALE STUDY 

### The full-scale study features numerous improvements intended to address both comments from the peer review of the pilot and i ssues identified by the Project Team while conducting the pilot. Changes from the pilot study include the following:

• 

### Shorter protocol. This study features a shorter protocol focused on total changes

in adult, all-cause mortality associated with a 1 μ g/m3 decrease in annual average

PM 2.5 

### in the U.S. The protocol for the pilot study was too long and difficult to complete in an 8-hour day, because it required elicitation of two separate distributions, one for short-term a nd one for long-term impacts of PM 2.5 

### exposures. As a result, the interviews often felt rush ed, which may have had an impact on the quality of the judgments we received. Based on our experience in the pilot study and advice received from health experts who reviewed early drafts of the protocol, the full-scale study takes a more integrated approach that covers issues related to short- and long-term exposure, but elicits a single C-R function that integrates experts' judgments about the impacts of changes in peak and long term exposure
associated with a 1 μ g/m3 decrease in annual average PM 2.5 

### in the U.S.

• 

### Additional expert review and feedback on protocol. Protocol development

benefited from additional review and fee dback from the scientific community. The Project Team conducted an EPA Symposiu m in April 2005 with more than 30 internationally-recognized experts knowledg eable about PM health effects. The purpose of the EPA Symposium was to present EPA’s plans for conducting the full-scale elicitation and to get feedback on the structure and content of the protocol.

• 

### Larger expert panel. This study interviewed a larg er panel of 12 experts selected

from an unrestricted pool of potential experts via a peer nomination process. The panel includes expertise in epidemiology, toxicology, and medicine. Expert selection in the pilot used 5 experts selected from a restricted pool derived from the

5 “Anchoring and adjustment” refe rs to when an expert begins his estimates with, or “anchors” on, a particular value then

develops confidence inte rvals by adjusting that value to account for va rious factors that influe nce his judgment. Some expert judgment research has shown that use of this approach may lead to over confidence, and thus poor accuracy, because respondents often fail to adjust confidence intervals adequa tely for what they do not know.

6 One reviewer did not feel that it was appropriate to combin e the individual distributions in to a single estimate. Another

preferred that expert results be applied in benefits analysis in dividually and then pooled, ra ther than being combined as part of the expert judgment study. Some reviewers commen ted that the combination of ju dgments using averaging across experts might generate results with which none of the experts would agree. While not endorsing combination of responses, some of the experts indicated th at if results must be combined, they would be most comfortable with a process that uses equal weighting.1-5 

### membership of two NAS committees. The sm all sample size of the pilot study was a concern noted by the EPA Science Ad visory Board (SAB) Health Effects Subcommittee, which reviewed early plans for the pilot study (EPA SAB, 2004).

• 

### Improved briefing book materials. The study featured improved briefing book

materials for experts, including a CD containing over a hundred relevant studies, plus background information pages with data on air quality in the U.S., population demographics, health status, summaries of published effect estimates, and data on other factors (air conditioning use, housing stock, PM composition) that may affect experts' judgments. While the pilot study relied more heavily on compendia such as EPA's criteria document for PM for briefing book materials, we supplemented these materials in the full-scale study to provide clear and easy access to relevant studies and key data useful for developing quantitative estimates specific to the current U.S. population.

• 

### Pre-elicitation Workshop. In response to peer-review comments, the full-scale

study included a Pre-elicitation Workshop held in January 2006 to better prepare experts in advance of the interviews . The workshop provided training on the history of subjective judgment, and opportunity for review and comment on the elicitation protocol, and encouraged experts to share and critique data and analyses they believed were relevant to the questions in the protocol.

• 

### Better accounting of experts’ views on relevant scientific issues. The new

protocol employed a more systematic approach to cataloguing and assessing expert views related to confounding, effect modification, exposure issues, and other potential sources of error or bias in publis hed mortality effect estimates. During the pilot study, the Elicitation Team at times found it challenging to make sure that each expert systematically addressed in his quantitative estimates each of the major issues he may have raised in the conditioning phase of the protocol. Following a method used in a protocol designed for a study of climate change (Morgan et al., 2001), we modified the protocol for the full-scale study to give each expert a set of cards on which we asked him to write down key factors he wished to discuss. The cards were then used to organize and rank th e factors, and to ensure that all factors
are discussed and evaluated.

• 

### Greater flexibility. The new protocol was designed to allow experts greater

flexibility in specifying the shape of the C-R function and in eliciting the values for their uncertainty distributions. Our expe rience from the pilot elicitation indicated that the pilot protocol was not sufficientl y explicit about the options available for characterizing the C-R function. We revised the protocol to facilitate the elicitation of judgments for experts who wished to specify a non-linear C-R function and/or incorporate uncertainty about thresholds. Also, at the EPA Symposium, several experts indicated that they would prefer to be able to explore specifying parametric distributions to assist them in estimating their intermediate percentiles, such as the 25 th 

### and 75 th 

### . The real-time feedback system, described below, provided alternative approach es to characterizing the uncertainty
distributions.1-6 • 

### More extensive real-time feedback. The elicitation of judgments is a complex

and iterative process. The pilot study lack ed mechanisms to provide experts with real-time data and graphs that would allo w them to better visualize, evaluate, and refine their initial judgments. The interviews for the expanded study included a new real-time feedback system that used spreadsheet models and Crystal Ball TM probabilistic modeling software to provide experts with graphs and data during the elicitation. The system, provided via Inte rnet teleconferencing allowed for experts to visualize their distributions, to assess the effect of judgments about causality and threshold, to compare their results against published mortality effect estimates, and to estimate and compare the change in deaths associated with specific PM 2.5 

### reductions against U.S. mortality data for major causes of death. 7

• 

### Post-elicitation Workshop. The current study also included a workshop held

following completion of the interviews (h ereafter, “Post-elicitation Workshop”) that allowed for a final discussion of themes that emerge in expert responses, differences in interpretation of key studies used to support responses, and any areas of confusion that arose during the interviews. The goal of this workshop was not to promote consensus, but again to ensure that all experts had access to the same information, could have ou tstanding questions addressed, a nd were able to provide feedback to EPA on the study. Experts had the opportunity to adjust their estimates following the workshop. As in the case of the Pre-elicitation Workshop, the Post-elicitation Workshop addressed a frequent concern expressed by peer reviewers of the pilot study that there should be more opportunity for communication between the experts.

• 

### Reporting of individual expert distributions only. In response to concerns

raised by peer reviewers related to th e potential pitfalls of generating and presenting a single combined estimate as well as other feedback received during the planning for the pilot and full-scale st udies, we have chosen to present only the individual expert distributi ons in this report, preservi ng the diversity of opinion across experts on this topic. However, we have conducted a sensitivity analysis using data pooled across experts from a si mplified mortality benefits analysis (see Chapter 4 for further details).

1.3 ORGANIZATION OF THIS DOCUMENT 

### The remainder of this document is organized into four chapters. Chapter 2 describes IEc’s analytical approach to conducting the e xpert judgment study, including the design of an elicitation protocol, testing of the protocol, selection of experts, expert workshops, and the interview process. Chapter 3 presents the results of the assessment, summarizing expert responses to both quantitative and qualitative questions. Chapter 4 provides discussion of both the quantitative findings of the study and the elicitation process itself. Chapter 5 includes the overall conclusions of the study.

7 The elicitation team used either WebE xTM or Go To MeetingTM Internet confer encing software duri ng the interviews.2-1 

### CHAPTER 2 | ANALYTICAL APPROACH

While researchers have been studying the pr ocess of eliciting and interpreting expert judgments for several decades, no single accepted, standardized method has emerged for this type of assessment. However, numerous studies conducted over the last two decades (e.g., Morgan et al., 1984 & 2001; Evans et al., 1994; Walker et al., 2001) have often shared common elements such as use of criteria for the selection of experts, the use of a detailed written protocol, the preparation of briefing materials, the elicitation of judgments in individual interviews, and where feasible, the convening of experts prior to and following the elicitation.

The motivation behind the development of the elements of formal elicitation studies is ultimately to help experts provide judgme nts that are informative, unbiased and wellcalibrated. Specifically, they are intended to help avoid some of the well-documented heuristics and biases (Kahneman et al., 1982) that lead to poorly calibrated judgments. These heuristics include the tendency to rely on data or studies most cognitively available or recent (the availability heuristic), and the tendency to adhere too closely to published or initial estimates and failing to make ad equate adjustments for key uncertainties (anchoring and adjustment heuristic). These and other heuristics can contribute to expert over-confidence (i.e., the tendency to express mo re certainty about a quantitative estimate than is warranted).

IEc has relied on the experience of earlier st udies to inform the design of the overall process for this study. Exhibit 2-1 outlines the expert judgment process of this study. The main elements of the process included development of an elicitation protocol, selection of experts, development of a briefi ng book, the elicitation interviews, and expert workshops prior to and following individual elicitation of judgments. The diagram also
indicates the influences of certain elements in the process on the development of others. For example, developing the protocol was an iterative process that was influenced not only by Project Team discussion, but also by the EPA Symposium and pre-testing of the protocol with two independent EPA scientis ts with expertise in the study of PM 2.5 

### - mortality. In addition, discussions durin g the Pre-elicitation Workshop influenced elements of the protocol as well as the briefi ng book. Finally, the expert judgments were elicited during the personal interviews with members of the project team, but experts were allowed to adjust their responses following the Post-elicitation Workshop.

The following sections discuss ea ch of the main elements of the expert judgment process described above. We also provide discussion of the decisions reached by the project team regarding the evaluation and presen tation of experts’ judgments.EXH I BIT 2-1: OVERVIEW OF TH E EXPERT JUDGMENT PROCESS



# EPA Sym posium

Protocol Pre-testi ng

Post-Elicitation Workshop Elicitation Interview s Pre-Elicitation Workshop Briefing Book Development Protocol Development

Expert Selecti on

Expert Judgments

2-22-3 2.1 PROTOCOL DEVELOPMENT 

### In a formal elicitation process, the interview protocol serves several purposes. It ensures that experts are answering the same, clearly specified question; 8 

### makes explicit the critical assumptions and rationale underlying th e expert's judgments; encourages experts to think critically about potential uncertainti es; and helps the expert avoid some of the common pitfalls and biases when providing subjective judgments about uncertainty (Otway and von Winterfeldt, 1992). The Proj ect Team developed a protocol over a period of 16 months that included a clear statement of the questions to be answered, including the critical assumptions underlying the main quantitative question, and provided a process for identifying and raising key issues, eviden ce, and interpretations upon which the expert could base his quantitative judgments.

The protocol evolved through a series of steps beginning with collaborative discussions with the Project Team aimed at scoping and st ructuring the problem that would be posed to the experts. A draft protocol was next reviewed and discussed by health experts at an expert elicitation sympos ium sponsored by EPA. 9 

### Following revisions to the protocol based on recommendations from the EPA symposium, the protocol underwent pre-testing with two independent EPA experts on PM 2.5 

### mortality. The Project Team finalized the protocol in January 2006, following the Pre-e licitation Workshop with the selected expert panel (discussed below).

The following sections discuss the development of the protocol, in particular the processof structuring the problem, the influence of the EPA Symposium, and the protocol pretesting. It also describes the final structure of the protocol. A copy of the protocol used in the interviews is included as Appendix A.

2.1.1 STRUCTURING THE PROBLEM 

### The expert judgment literature discusses tw o broad approaches to elicitation of judgments; an aggregated and a disaggregated approach. As the term implies, an aggregated approach asks the expert to estim ate the quantity of interest directly; for example, the numbers of newspapers sold in the U.S. in a particular year. In a disaggregated approach, the expert (or group of experts) would be asked to construct a model for estimating the quantity of interest and would be asked directly about the inputs to that model (e.g., population in each state, percentage of the population that reads newspapers). The theory behind the disaggregat ed approach is that it is easier for experts to answer questions about the intermediate quantities than about the total quantity. Relatively few studies have been done to co mpare the two approaches, however. Morgan and Henrion (1990) note that studies that have attempted to document the superiority of the disaggregated approach have shown mixed results.

8 The "clairvoyance" test is often used to de termine if a question is well-specified. The clairvoyance test asks whether an all

knowing individual would be able to answer the question without asking for additional information (Hora, 1992).

9 IEc also received valuable early input from Dr. Andrew Wils on of the Harvard Center for Risk Analysis, a researcher in

expert judgment elicitation.2-4 

### The Project Team carefully considered the relative advantages and disadvantages of the two approaches. A major advantage of the disa ggregated approach is a more structured and transparent characterization of the key i nputs and sources of uncertainty in the final quantity of interest. However, this method does require additional time and resources to develop not only a set of input variables, but also a model structure (or in some cases, multiple models) specifying the mathematical relationship between the variables on which the experts can agree prior to the indi vidual elicitations. An aggregated approach can be easier to implement, both because it does not require extensive consultation with the expert panel to reach consensus on a model structure and because can reduce the length of the elicitation protocol. However, these benefits must be weighed against concerns about transparency of expert res ponses and the difficulties faced by experts who will be expected to mentally integrate the various problem components to provide their judgments.

The Project Team opted to pursue a larg ely aggregated approach, with some disaggregated elements. The protocol elicits a single distribution of the mortality impact of a unit change in annual average PM 2.5. 

### This effect distribution integrates the mortality impacts of the changes in both short-term (e.g., peak 24-hour) and longer-term PM 2.5 

### exposures that may contribute to a 1 μg/m 3 

### reduction in annual average ambient PM 2.5


### . The Project Team selected this approach for several reasons. First, our experience with the pilot study showed that eliciting separate distributions for the mortality impacts of short-term and long-term changes in exposure was very time consuming and difficult to complete in the 8-hour timeframe of the interview. Second, the team believed the large panel of experts recruited for this study ma de it unlikely that we could reach agreement among the group on a quantitative model for disaggregating the PM mortality effect estimate. Attendees of the EPA Symposium (s ee below) also supported this approach.

In an effort to ameliorate concerns about the aggregated approach we developed a comprehensive and detailed set of conditioning questions and also incorporated some quantitative disaggregated elements. The te am designed the conditioning questions to promote transparency and provide a thor ough understanding of the foundation of beliefs underlying each expert's quantitative distribution, including the ranking of key elements of uncertainty and careful documentation of key evidence in support of judgments. In addition, we gave experts the option to disa ggregate two potentially challenging elements of an aggregated model: the likelihood that a causal relationship does or does not exist linking PM 2.5 

### exposures with mortality, and the like lihood and location of a threshold for PM mortality effects.

2.1.2 EPA SYMPOSIUM 

### The protocol development was also influe nced by an EPA Symposium on the proposed expert elicitation project in April of 2005 . The purpose of the EPA Symposium was to discuss the role of expert elicitation in EPA's efforts to improve the characterization of uncertainty in regulatory benefits analysis, particularly in the relationship between longterm PM 2.5 

### exposures and mortality. Attendees of th e first day of the symposium included representatives of EPA and OMB, a group of invited scientists with expertise in the
health effects of PM 2.5 

### , and members of the public.2-5 

### During the first half of the EPA Symposium, which was open to the public, members of the Project Team provided an overview of th e use of expert elicitation in uncertainty analysis, including a presentation of the results of the pilot study and their use in an RIA as well as a discussion of plans for the full-scale expert elicitation study on PM 2.5 

### and mortality. This portion of the EPA Sym posium concluded with an opportunity for audience members to comment on the design and content of the elicitation.

The second half of the EPA Symposium consiste d of a closed meeting with Project Team
members and the invited scientis ts to carefully review the draft elicitation protocol for the full-scale study. A series of moderated discussions were held to help assess whether the draft protocol was focused on the right qu estions, that it was comprehensive (i.e., addressing key elements of uncertainty) and that the questions and associated assumptions were clearly specified.

Following the symposium the Project Team and additional representatives from EPA and OMB met for half a day to discuss possible modifications to the draft elicitation protocol in response to suggestions made during th e EPA Symposium. Comments by members of the public, the invited experts, OMB, and EPA were considered when revising the elicitation protocol and other study design aspects. Examples of specific decisions informed by the EPA Symposium included the following:

• 

### The questions should focus on PM 2.5 

### (mass concentration); data on PM 2.5



### components are too limited to support meaningful judgments;

• 

### The questions should explicitly encourage experts to think broadly about evidence

and theories from all relevant disciplines, and should be clear to both epidemiologists and non-epidemiologists;

• 

### The protocol should provide experts with the option to either include judgments

about causality either directly into their effect distribution or separately;

• 

### The main quantitative question should fo cus on all-cause mortality as the outcome,

rather than eliciting separate C-R func tions for specific causes of death;

• 

### The protocol should employ an aggregated approach to eliciting total mortality

effects from long-term and short-term PM 2.5 

### exposures, rather than a disaggregated approach in which judgments about the effects of short-term and long-term exposures are elicited separately;

• 

### The Project Team should include PM values in the study ranging from background

up to the upper end of concentrations found in epidemiologic studies (4-30 μ g/m 3 

### );

• 

### The protocol should specify that housing stock and levels of susceptibility in the

U.S. population will remain constant fo r purposes of the study and co-pollutant
concentration is unknown and re mains a source of uncertainty;

• 

### The briefing book should include data on the current U.S. population, such as

health status and educational attainment;

• 

### The interview should feature real-time feedb ack tools to allow experts to visualize

their distributions as well as the impact of their views on causality and threshold.2-6 

### In addition, the tools would provide experts with a “back of the envelope” calculation to show how their judgments translate into number of deaths as a "reality check."

2.1.3 PROTOCOL PRE-TESTING 

### Pre-testing of the protocol using experts not on the expert panel is critical to developing a well-functioning protocol. It enables the elicitation team to test the clarity of the questions and enables the team to practice and refine the process for administering the protocol. IEc conducted two full-length pre-test s of the protocol with two EPA experts in PM-related mortality: Dr. Tony Huang of EPA’s National Health and Environmental Effects Research Laboratory (NHEERL), an expe rt in PM toxicology in humans, and Dr. Lucas Neas of EPA’s NHEERL, an expert in PM epidemiology. The pre-tests were conducted in November of 2005. The pre- test subjects were not paid for their participation.

The pre-tests provided valuable input to the protocol development. Changes made to the protocol in response to the pre-tests included clarifications to the question about population thresholds and to the main quan titative question, the addition of a section where experts could discuss any important topics not previously covered (e.g., publication bias), and the inclusion of ques tions asking the experts to describe the strengths and limitations of the studies used to support their answers on mechanisms, causality, and threshold.

The pre-tests were also useful for identifying topics that would be helpful to discuss with experts at the workshop held before the Pr e-elicitation Workshop. For example, it was
evident that some confusion remained ove r the distinction between the causality and threshold question. Therefore, this was included as a topic discussion.

2.1.4 PROTOCOL DESIGN 

### The structure of the protocol is outlined in Exhibit 2-2. The protocol was divided into three parts: the preview and assumptions, conditioning questions regarding issues and evidence, and the quantification of a C-R func tion. (For experts who did not attend the Pre-elicitation Workshop, we also spent some time at the beginning of the interview discussing the objectives of the elicitation proj ect and reviewing the expert elicitation process.)

The ultimate goal of the protocol was to get the experts to answer the following question:

What is your estimate of the true percent change in annual, all-cause mortality in the adult U.S. population resulting from a permanent 1 μg/m 3 

### reduction in annual average ambient PM 2.5 

### across the U.S.? In formulating your answer, please consider mortality effects of both reductions in long-term and short-term exposures.2-7 

### The protocol is divided into the following three parts:

1) Preview and Assumptions. In this section the elicitation team previewed the key quantitative question concerning the mortality impact of changes in annual average PM 2.5 

### concentrations in the U.S. In addition to emphasizing the ultimate goal of the elicitation, this section gave each expert the opportunity to examine carefully the underlying assumptions about factors such as historical and baseline exposures, regulatory implementation that might result in the hypothetical change in

PM 2.5 

### concentrations, and characteristics of the U.S. population over time. After the preview, the expert had the opportunity to ask clarifying questions and raise any concerns rega rding the specification of the main question and the assumptions.

2) Conditioning Questions. This section consisted of largely qualitative questions about factors to consider when characterizing the relationship
between PM 2.5 

### exposure and premature mortality. These questions covered the following categories:

• Scientific evidence supporting physiological mechanisms linking PM 2.5 

### exposures to mortality;

• Key causes of death associated with PM 2.5 

### exposures;

• Conceptual framework for mortality effects of short-term and long-term PM 2.5 

### exposures;

• Role of study design in capturing effects of changes in annual average PM 2.5 

### exposures;

• Epidemiologic evidence for the impact of exposures to PM 2.5 

### on mortality;

• Evidence and impacts of confounding;

• Evidence and impacts of effect modification;

• Evidence and impacts of exposure misclassification;

• Likelihood of a causal relationship between PM 2.5 

### exposures and mortality;

• Potential thresholds in the C-R function; and

• Other influential factors (e.g., selection bias, statistical methodology, publication bias)EXHIBIT2-2: STRUCTUREOFTHE ELICITATIONPROTOCOLPreview &AssumptionsQuantify C-RFunctionExaminequantitativequestionMechanismsConceptualframework forshort- and longterm mortalityConfoundingEpidemiologyevidenceReviewAssumptionsEffectModificationExposureAssessmentCausality*ThresholdOther issuesShapeRe-visitcausalityRe-visitthresholdSpecifypercentiles ofC-R functionuncertaintydistributionRe-visitConditioning Questions:Issues and Evidence*Red items involve quantitative responses2-82-9 

### These questions encouraged th e experts to thoroughly and comprehensively consider key concepts, theories, and scientific evidence concerning the nature and potential magnitude of the

PM 2.5 

### /mortality C-R relationship. The expert's responses provided a conceptual foundation to support hi s quantitative judgments in the final section of the protocol. In answering these questions, experts were encouraged to rely on evidence from multiple disciplines and to consider impacts of both short-term and long-term PM 2.5 

### exposure, to the extent they believe both contribute to the mortality impacts of changes in annual average PM 2.5


### . Unlike the pilot study, this protocol did not include questions about the relative toxicity of different PM components and sources. While the issue of differential toxicity remains a topic of interest to EPA, we removed these questions because of concerns expressed by the experts in the pilot study and the experts attending the EPA Symposium that available research is insufficient to support meaningful judgments on this issue.

3) Elicitation of Quantitative Judgments. The final section of the protocol presented the key quantitative question to be elicited:

What is your estimate of the true percent change in annual, all-cause mortality in the adult U.S. population resulting from a permanent 1 μg/m 3 

### reduction in annual average ambient PM 2.5 

### across the U.S.? In formulating your answer, please consider mortality effects of both reductions in long-term and short-term exposures. To characterize your uncertainty in the C-R relationship, please provide the 5 th 

### , 25 th 

### , 50 th 

### , 75 th 

### , and 95 th 

### percentiles of your estimate.

In this section, the expert specified a functional form for the PM 2.5 

### mortality C-R function and then developed an uncertainty distribution for the slope of that function (the mortality impact per unit change in annual average PM 2.5 

### ). Experts could specify a functional form with a single slope that applies across the entire range of PM 2.5 

### annual average values specified in this study (4-30

μ g/m 3 

### ) or could specify a more complex function by segmenting the study range of PM 2.5 

### and specifying different slopes for different
segments. 10 

### The expert was also asked in this section to consider whether and how to factor his quan titative responses to the questions

10 The PM 2.5 study range was developed through revi ew of EPA monitoring data for the U.S., consultation with the EPA Project Team, and discussions during the EPA Symposium.2-10 

### about the likelihood of a causal relationship and the potential for a population threshold into his distribution.

As shown in Exhibit 2-2, the elicitation was an iterative process. During the elicitation of the probabilistic distribution of PM 2.5 

### mortality effect estimates in the last part of the protocol, the elicitation team revisited res ponses to conditioning questions to ensure the expert's distribution was consistent with the judgments provided in the second section of the protocol regarding the various sources of uncertainty and their potential impacts.

2.2 EXPERT SELECTION 

### A well-designed expert selection process for an expert elicitation study should ensure that the final panel of experts has the appropriat e expertise to address the questions posed to them and represents, as a group, a reasonably balanced range of respected scientific opinion on the issues being addressed. Previous studies (e.g., Hawkins and Graham, 1990) have also identified several additional criteria: the process should be explicit and reproducible, be reasonably cost-effective a nd straightforward to execute, and should minimize the level of control of the researcher conducting the elicitation.

We conducted expert selection in two parts, both of which relied on a peer nomination process and chose 12 experts for participation in this study. The first phase of the expert selection process was designed to select nine e xperts. The initial decision to include nine experts was based on several factors, including 1) a literature search that found most of the elicitation studies conducted to date (60 per cent) use panels of six to eight experts, and 90 percent use panels of 11 or fewer experts (Walker, 2004); 2) it was deemed that nine experts would provide a balanced set of views on this topic; 3) the pilot study was criticized for the small panel size of five experts (EPA SAB, 2004); 4) government agencies are required to undergo an Inform ation Collection Request process for the Paperwork Reduction Act if information is co llected from more than nine individuals; and 5) resource and time requirements increas e with each additional expert. The first phase of the selection process included the following steps:

1. Identifying Experts to Provide Peer Nominations. IEc conducted a literature search and publication count to identify a group of experts in the field who have published articles in peer-reviewed journa ls on the mortality and/or morbidity effects of PM exposure. We performed a search of pertinent literature published in the past 60 years (1945-2005) using the search engine Web of Science. 11 

### We
used the results of the literature search to rank the authors based on the total number of first, second and last authorships combined. We selected the 32 highest-ranking authors to serve as nominators. The 32 nominators obtained from the publication count represented a ra nge of expertise and perspective on the elicitation topic.

11 The search included the following sear ch terms: ("air pollution" or "particulate matter" or "fine particles" or "PM 2.5 " or

"PM 10 ") and ("death" or "mortality" or "survival" or "morbidity") and ("epidemiology " or "human" or "time-series" or "cohort" or "clinical"). IEc acknowledges support from Andrew Wilson of the Harvard Center for Risk Analysis in conducting the publication count.

### 2. Obtaining Peer No minations. To enco urage nom ination of a broad array of experts on thi s topic, we divided the p o o l of potent i al nom inators into four grou p s of eight. P o t e ntial nom inators were assigned random ly to a gr oup. We asked

nom inators to provi de 10 n a m e s of indiv i duals that met general criteria co mm on to all groups. We also aske d that the in dividuals nom inated meet a set of criteria that was specific to each group (See text boxes for the criteria.) For instance,

nom inators in Group 1 received a list of criteria that i n cluded all of those listed in
the “General Selection Criteria” as well as the two criteria listed in the “Group Specific Criteria” under “Group 1.” Exhibit 2- 3 shows the group assign m e nts.
We received 25 sets of nominations overa ll (seven fr om Groups 1 and 2; five
from Group 3 ; and six from Group 4; no mina tors who provi ded list s are shown in
bolded italics in Exhi bit 2- 3). 12

GENE RA L SELE CTION CRIT ERIA

1. Ideal expert s should posses s the educatio nal backgro und and/o r exper i ence t o bo t h
display a tho r ough underst anding o f result s fr om the epidemiological li terature addressi ng

the relationshi p between chr o nic exposure to PM 2. 5 and mortali t y, and to evaluate these

results in t h e co nt ext o f othe r evidence pertinent to the P M 2. 5 /mo r t a li t y issue, such as

relevant toxic o logical and ph ysiological li ter a ture.

2. Experts may include prima r y scientific re s earchers as w e ll as prominent individuals from
scientific panels, inst itutions, journal editor ial boards, and other such gro ups who, throu g h
their educational background and experi en ce, are in a p o s i tion to caref u lly interp ret t h e k e y evidence regarding PM 2. 5 exposures and mortality.

3. The overall set of experts nominated should be a balan c ed gro up that reflec ts the fu ll
range of respected scientific opinions conce r ning the stren g th o f the e v id ence link ing
premature mortality with ambient PM 2. 5 con c entrations.

4. The nomine es should all be base d in either the U.S. or Canada.

12 While this study did not allo w nomina tors to sel f -nomina t e, eig h t ex p e rts in E x hibi t 2-3 were nominated by others in the

l ist and we re ul timate l y se le cte d as part o f o u r e x pe rt pane l

. Given that the pro c ess o f ident i fying nominators produced a

l ist of frequentl y p u bl ished researchers in the a r ea of PM mor t a l ity, we be lieve it is r e a s ona b le tha t ther e would be significant overlap between th e nomi n a tor s a n d the selec t ed exper t s.

2-11GROUP SPECIFIC CRITE R IA

We would like you to nomina te experts that you feel:

Group 1:

• Are the most knowledgeable about th e rela t i onship between long- t erm PM 2. 5 exposures and mortality; and / or

• Have studied i n -depth the uncertainties an d m e thod ologi c al lim i tations o f exis ting cohort studies on PM 2. 5 and mortali t y.
Group 2:

• Have m a d e the m o s t s i gnificant con t rib u ti on s to our understanding of the potential underlying biological mechanisms of the PM 2. 5 / m ortality rel a tions h ip ; and/ or

• Have m a d e the m o s t s i gnificant con t rib u tion s to our und ers t and i ng of the lik e lihood of a causal relationship betw een PM 2. 5 and mortality.

Group 3:

• Display signific ant experience analyzing the relationship be tween PM 2. 5 and

m o rtality thro ugh particip ation in expert committees and wo rksho p s, and/o r publication of review articles; and/or

• Display signific ant experience analyzing an d ap p l ying the PM/ m ortality lit e rature within a risk as sess ment and/ or policy con t ext.
Group 4:

• Are conducting innovative, cu tting-edge res earch investigating the rela tio n ship between PM 2. 5 and mortality; and/or

• Have m a d e the m o s t s i gnificant con t rib u tion to our und ers t and i ng of the relationship be tween health effects and PM 2. 5 expo sures.



### 3. Ranking and Selecting Experts. We ran k ed experts based on the num b er of pee r

nom inations they received within each speci fic group. We also com p iled the gr oups into a com b ined ranked list. We then selected the top two nom inees from e ach of the four gr ou ps, plus the m o st highl y n o m inated expert from the remaining

co m b ined list, for a total of nine experts. 13



## 4. Replacing Experts. If an invited expert was unwilling or unable to participate in the

asse ss ment, IEc replaced t h e expert with th e next m o st highly nom inated candidate in that expert’ s group, provided the y w e re no m i nate d by at least half of the respondents i n that group. Otherwise, IE c replaced the expert with t h e next m o st highl y nom inated expert overall.

To aid the experts in the nom i nation process, w e des c ribed the overall objectives of the expert elicitation pr oject and prov ided th em with an al phabetical list of first and l ast
authors with at least two authorships as id entified thr ough the literature sear ch, who are
based either in the U.S. or Canada. To increase response rates, we followed-up t h e mailings with phone calls to the nom inators.

13 When faced with a group of two or m o re ex p e rts with th e same number of nominations, we randomly selected fr om the

tied set o f ex perts.

2-122-13 EXHIBIT 2-3: NOMINATORS DERIVED FROM PUBLICATION COUNT

NOMINATOR AFFILIATION

GROUP 1
Burnett, Richard T. Health Canada
Ito, Kazuhiko New York University School of Medicine Katsouyanni, Klea University of Athens Medical School
Saldiva, Paulo Hilario Nascimento University of Sao Paulo
Zanobetti, Antonella Harvard School of Public Health
Devlin, Robert B. U.S. EPA Dockery, Douglas W. Harvard School of Public Health Pope, C. Arden Brigham Young University

GROUP 2 Brunekreef, Bert Utrecht University
Diaz, Julio Central University of Public Health, Madrid
Ghio, Andew J. U.S. EPA
Samet, Jonathan M. Johns Hopkins University Sunyer, Jordi Institut Municipal d'Investigacio Medica (IMIM)
Costa, Daniel L. U.S. EPA

Ostro, Bart D. California Office of Environmental Health Hazard Assessment
Peters, Annette GSF-National Research Center for Environment and Health

GROUP 3 Braga, Alfesio Luis Ferreira University of Santo Amaro Hoek, Gerard Utrecht University
Jerrett, Michael University of Southern California
Künzli, Nino University of Southern California Lee, Jong-Tae Yonsei University Spengler, John D. Harvard School of Public Health
Anderson, H. Ross St. George's Hospital Medical School
Schwartz, Joel Harvard School of Public Health

GROUP 4 Ballester, Ferran (Diez) Valencian School of Studies for Health
Brauer, Michael University of British Columbia
Dominici, Francesca Johns Hopkins University Goldberg, Mark S. McGill University
Krewski, Daniel University of Ottawa Levy, Jonathan I. Harvard School of Public Health Lipfert, Frederick W. Independent Consultant Thurston, George D. New York University School of Medicine

Note: Names in bolded italics indicate those who responded with a list of nominations.2-14 

### While this process featured a good acceptance ra te and yielded nine experts, the panel exhibited less diversity in expertise than we anticipated in design, with most experts being epidemiologists. In an effort to in crease representation of the biological, medical, and toxicological disciplines, we conducted a second phase of selections. EPA sought additional nominations of experts in these fi elds based on nominations provided by the Health Effects Institute (HEI). The general criteria for nominations were the same as for the first part of the selection process (Holmstead, 2005). HEI provided EPA with an unranked list of 10 nominees, plus two alternates (O’Keefe, 2005). 14 IEc used a random ordering process to determine the order in wh ich these experts would be contacted, with a goal of inviting three additional experts from this list, for a total of 12 experts overall. If an expert declined, either the specified altern ate for that expert or the next expert on the list was contacted. The acceptance rate for this second step was not as high as the first; IEc recruited three additional experts to participate. 15 Exhibit 2-4 presents the final list of the 12 participating experts.

EXHIBIT 2-4: FINAL EXPERT LIST

NAME AFFILIATION

Dockery, Doug W. Harvard School of Public Health Ito, Kazuhiko New York Univ ersity School of Medicine Krewski, Daniel University of Ottawa

Künzli, Nino* University of Southern California Ke ck School of Medicine (currently at Institut Municipal d'Investigac ió Mèdica (IMIM) - Center for Research in Environmental Epid emiology, Barcelona, SPAIN) Lippmann, Morton New York Univ ersity School of Medicine Mauderly, Joe Lovelace Resp iratory Research Institute
Ostro, Bart D. California Office of Environmental Health Hazard Assessment Pope, C. Arden III Brig ham Young University Schlesinger, Richard Pace University Schwartz, Joel Harvard Sc hool of Public Health Thurston, George D. New York Un iversity School of Medicine Utell, Mark University of Rocheste r School of Medicine and Dentistry

*Dr. Künzli was based in the U.S. at the time of expert selection, and subsequently be gan a sabbatical in Barcelona

midway through the project.

The following experts were contacted, bu t declined to participate: Dr. Richard Burnett; Dr. Jonathan Samet; Dr. Michael Brauer; Dr. Carol Henry; Dr. Judith Graham; Dr. John Balm es; Dr. Gunter Oberdorster; and Dr. James Crapo. The most frequently cited reason for declining was scheduling concerns given the level of commitment asked of the experts.

14 The alternates were provided fo r specific nominees, based on simil arity of background and expertise.

15 Three experts from the first group declined to particip ate, whereas five experts from the HEI list declined.2-15 2.3 BRIEFING BOOK 

### All experts were sent a “briefing book” binde r at least two weeks in advance of his interview (IEc, 2006). The purpose of the briefing book was to provide experts with a
baseline set of materials to assist them in preparing for their elicitation interview; however, experts were free to consider other ma terials not included in the briefing book. The briefing book contained the following materials:

• 

### The elicitation interview protocol;

• 

### A CD containing over 150 relevant pa pers and compendia, searchable both alphabetically and by topic area; 16

• 

### A set of background information pages with recent U.S. data on air quality, health

status, population demographics and other topics that may factor into the experts’ probabilistic judgments; and

• 

### Background materials, including a document describing factors to consider when

providing probability judgments in order to avoid potential sources of bias, and an excerpt from 2002 National Research Counc il report on estimating public health benefits of proposed air rules.

Experts were given the opportunity to comment on the protocol and other materials either at the Pre-elicitation Workshop (discussed belo w) or privately, and where necessary, materials were updated and re-sent to all experts in response to the comments received prior to commencement of the first interview. Examples of the briefing book materials are included in Appendix B.

2.4 PRE-ELICITATION WORKSHOP 

### The expert panel was invited to participate in a day-long workshop in January of 2006. The workshop was designed to introduce the project, provide background information to the experts on expert judgment and the elicita tion process, and to foster discussion about the key evidence available to answer the questions posed by the study.

The first half of the workshop consisted of presentations by the Project Team focused on educating the experts about the history of expert judgment elicitation, the objectives of this study, the design of the protocol, and the overall structure of the project. The experts also participated in a calibration exercise d esigned to familiarize them with the process of providing subjective probabilistic judgments. Another important goal of the morning session was to obtain feedback on the protocol.

The second half featured structured discussion sessions on three topics related to the

PM 2.5 

### -mortality issue (causality, shape of the C-R function, and evidence for quantitative estimates of the mortality effect of PM 2.5 

### exposure). In each discussions session, one

16 For some of the topic areas, publications on the CD were restricted to those publ ished since 2000 due to a large volume of

studies in that area. The binder also included a list of addi tional PM health studies that ha d been compiled separately by

EPA’s National Center for Environmental A ssessment for another purpose. Experts were able to request copies of these or any other papers they felt were relevant to the topic.2-16 

### expert presented evidence related to the t opic being discussed, and two other experts served as discussants. The purpose of these sessions was to both introduce important evidence for making judgments about each topic and to provide an opportunity for a critical discussion among all the expert particip ants about the strengths and limitations of that evidence. The Project Team was explic it that the goal of these discussions was not to encourage a consensus within the group, but to allow each expert to develop his own informed judgments.

Although every effort was made to find a date for the workshop that was amenable to all of the experts, eight of the 12 experts were able to participate in person for the full workshop; one participated by telephone for part of the day; and three were unable to participate. A workshop summary and copies of presentation slides from the workshop were sent to all 12 experts. Experts who did not attend the workshop were encouraged to review these materials carefully. In addition, papers cited at the workshop that were not already included in the briefing book were sent to all 12 experts in advance of their interviews, as was the finalized protocol, reflecting minor revisions requested by the experts at the workshop. 17

2.5 ELICITATION INTERVIEWS 

### As in the pilot study, the Project Team c hose to elicit the judgments of each expert individually during a personal interview. Th is choice reflects a preference based on a review of the expert judgment literature, and the goals of the project. Numerous approaches for obtaining subjective judgments from groups of experts with differing opinions have been proposed and demonstrated in the expert judgment literature (Morgan and Henrion, 1990; Cooke, 1991). Approaches va ry widely in the degree of interaction between experts during or after the process and in whether the process is intended to achieve group consensus. Unfortunately, little research has been done that rigorously examines the relative ability of the different methods to achieve well-calibrated results (Morgan and Henrion, 1990). Most of the research comparing methodologies has involved almanac-type questions (e.g., "What is the height of Mount Everest?") and has shown little difference in the quality of the r esults obtained by the various approaches. The research does suggest that interactions between experts can increase rather than decrease the problem of overconfidence (a nd thus, poor calibration) (Morgan and Henrion, 1990). Whether these findings can be extrapolated to studies like this one that feature complex questions posed to relevant experts requires further research.

Prior to the interviews, IEc provided each expe rt a copy of the protocol and the briefing book materials, including the pre-elicita tion workshop summary. The material was provided to allow each expert to familiarize hi mself with the questions to be asked and the resources available in the briefing book.

17 IEc clarified that experts should assu me for the purpose of this exercise th at the regulatory action will achieve

proportional reductions in all PM 2.5 components, and added differential impacts of PM 2.5 sources and/or PM 2.5 components to the examples of factors experts may a ddress in the Other Issues question.2-17 

### The elicitations were conducte d between January 31 and April 28, 2006. The Elicitation
Team consisted of two inte rviewers, one experienced in the elicitation of expert judgments, Dr. Katherine Walker, and one with expertise in PM exposure assessment, Dr. Patrick Kinney of Columbia University. Dr. Kinney was selected for his expertise in the subject matter of the elicitation and for his ab ility to objectively evaluate the responses of the various experts. Mr. Henry Roman and/or Ms. Tyra Gettleman of IEc also participated in the latter half of each in terview, using internet-based conferencing software to provide experts with real-time feedback on their quantitative responses, as
discussed below. Eleven of the 12 interviews were conducted at the expert’s offices; one was conducted at Dr. Kinney’s office. 18



### Most of the elicitations were conducted over the course of a single 8-hour day. Due to scheduling constraints, one of the elicitations was conducted over two days, the first half being completed the afternoon of the first da y and the second half the next morning. Typically, covering the introductory material took about an hour, while the remaining time was split approximately evenly between answering the conditioning questions, and answering the quantitative questions.

For each question in the protoc ol, experts were asked to think systematically about the relevant evidence and to consider any sources of uncertainty, error, or bias that might influence their interpretation of that evidence. The elicitation team requested that experts cite specific studies or other evidence supporting their judgment on a particular issue, and also prompted experts to consider specific evidence that would support an opposing or alternative position. This approach gave e xperts the opportunity to more fully evaluate the robustness of their conclusions, allowi ng them to make adjustments to their judgments in light of the full range of evidence or explain why they found the alternative evidence unpersuasive.

For the conditioning questions related to the influence of confounding, effect modification, and exposure misclassification on published epidemiological studies, the elicitation team asked the expert to write each factor he wished to discuss on a card. The team then asked the expert to physically group and rank the cards in response to questions about the direction and size of the impact of these factors on the results from each of the epidemiological studies the expert cited as most relevant.

For the quantitative questions, each expert w as first asked to specify his assumptions about the overall shape of the PM 2.5 

### -mortality C-R function for the range of PM 2.5 

### concentration changes specified. This step included questions about the functional form for the C-R relationship (e.g., linear, log lin ear, piece-wise linear, curvilinear and whether he planned to incorporate a threshold).

Another key decision each expert faced at this juncture was whether he wished to incorporate his quantitative judgments about the likelihood of a causal relationship

18 One expert's schedule could not accommoda te an interview at his office during the timefram e of this study. The expert was able to travel to New York during this time period, howe ver, and he agreed to conduct the interview at Dr. Kinney’s office at Columbia University instead.2-18 

### directly into his characterization of uncertain ty or whether he wished to have it be incorporated at a later stage. Experts who chose the latter approach were asked to acknowledge that they planned to develop their uncertainty distributions about the PM 2.5 

### - mortality relationship conditional on the assump tion that a causal relationship exists.

As part of the process of eliciting quantitative va lues, we asked each expert to identify the most important factors that might contribute to bias in published, peer-reviewed estimates of the percent increase in mortality associat ed with PM exposures and to quantify the effect of such biases on the overall uncertainty in those estimates. Each expert was then instructed to consider whether his quantitativ e responses needed to be adjusted to account for those potential biases.

Each expert was then asked to estimate the percent reduction in all-cause mortality
associated with a 1 μ g/m 3 

### decline in PM 2.5 

### associated with several fixed percentiles; minimum, 5 th 

### , 25 th 

### , 50 th 

### , 75 th 

### , 95 th 

### , and maximum of the distribution intended to describe his uncertainty in the magnitude of the “true” but unknown relationship (hereafter, his “uncertainty distribution) In an effort to minimize use of the “anchoring and adjustment” heuristic, the protocol was designed to begin with the theoretical basis for bounding the estimates (min, max, 5 th 

### and 95 th 

### percentiles), although some experts found this approach counter-intuitive preferred to begin with the median.

Each expert was also given the opportunity to use spreadsheet tools developed by the Elicitation Team that provide real-time feedback regarding the expert's responses. The Elicitation Team used two spreadsheets that were shared via an internet conferencing software. The first spreadsheet ("Spreadsheet Tool 1") was used to record the expert's estimated values for each of the fixed percentiles, his likelihood of a causal relationship, and his quantitative views on the existence of a threshold. The expert could either specify all of the percentile values of the distribution directly, or could specify a parametric distribution to assist with estimating some percentiles and/or to visualize the final distribution. Experts who chose the parame tric approach could specify two or more parameters (e.g., 5 th 

### and 95 th 

### percentile or mean and standard deviation) of a parametric distribution (e.g., normal, Weibull) that described their judgments. The Elicitation Team used Spreadsheet Tool 1, along with Crystal BallTM probabilistic software (CB), to build the distribution and then showed the expert a probability density function (PDF) and/or a cumulative density function (CDF) re presentation of his distribution.

If the expert specified that his mortality effect distribution was conditional on the existence of a causal relationship between PM 2.5 

### exposures and mortality, the protocol originally called for the Elicitation Team to in corporate the expert’s estimate of the causal likelihood into the distribution probabilistically. 19 

### The Elicitation Team then displayed a

19 The elicitation team multiplied the expe rt’s uncertainty distribution by a Yes/No distribution representing the expert’s

likelihood of a causal relationship to il lustrate how his view of causality would im pact his distribution when applied in a benefits analysis. For example, if an expert specified a causal likelihood of 95 pe rcent, his distribution would be multiplied by one for 95 percent of the time and by zero for five percent of the time. This approach assumes that the expert’s causality distribution and conditional mortalit y effect distribution are independent.2-19 

### PDF and CDF of his distribution incorporating the causal likelihood for the expert to
review. 20



### As in the pilot study, when experts used approaches to characterize the concentration response function as non-linear, or incorpor ated thresholds, and/or developed
distributions conditional on a causal relati onship, the elicitation team thought it was important for experts to be able to visua lize the integrated impact of these various assumptions on an overall uncertainty distribu tion for the percent reduction in mortality. The elicitation team accomplished this step using Spreadsheet Tool 1 and CB. Crystal Ball was used to sample from the expert's un certainty distribution and his distribution of possible threshold levels, in relationship to the 2002 distribution of population-weighted

PM 2.5 

### annual average concentrations in the U. S., obtained from EPA’s benefits analysis model, BenMAP. 21 

### A similar approach was also used to illustrate the effect of specifying different slopes of the C-R function for different ranges of baseline annual average PM 2.5


### . The second spreadsheet ("Spreadsheet Tool 2") was intended to show the expert alternate displays of his distribution as well as some of the implications of his distribution for estimates of mortality in the U.S. The firs t display in Spreadsheet Tool 2 showed each expert a graph plotting his uncertainty distribution as well as the analogous distributions from selected epidemiologic studies. Alternativ ely an expert could view his uncertainty distribution in the form of juxtaposed box plots of mortality estimates from the epidemiologic studies on which the expert relied for his quantitative estimates. 22 

### Finally, an expert could view two “back of the e nvelope” calculations based on his distribution. 23



### The first was an estimate of the number of a nnual deaths that might be avoided if the annual average PM 2.5 

### concentration in the U.S. were reduced from 12 to 11 μg/m 3 

### . The
second was an estimate of the number of annua l deaths that might be avoided if the annual average PM 2.5 

### concentration in the U.S. were reduced from 12 to 4 μg/m 3



### . 24 

### The expert was then shown a bar graph plotting the estimated annual deaths attributable to this drop in air pollution compared with annual deaths attributed to other major risk factors

20 As discussed later under the discussion of the post-elicitation wo rkshop, the Project Team agreed that this approach was

not suitable approach for combining these two elements of th e experts’ judgments, and subs equently has presented them separately. The result of this change is that the uncertainty distributions of those experts who incorporated causal likelihood estimates are not directly comparable to those who did not and th ey are therefore presented separately.

21 The Example Applied distribution was crea ted as follows: On each iteration of CB, a value for baseline PM2.5 was selected

from the BenMAP distribution, a value for the effect threshold was selected from the expert's threshold distribution, and a mortality effect estimate was selected fr om the expert's distribution incorporatin g causality. For each iteration, if the PM2.5 value equaled or exceeded the threshold, then the valu e from the distribution incorp orating causality was selected for the example applied distribution; othe rwise zero was selected. The process wa s repeated 10,000 time s to generate the Example Applied distribution.

22 All box plots consisted of the median, interq uartile range, and 90% confidence interval.

23 In order to perform the “back of the en velope” calculations of avoi ded deaths, we used the following damage model D = P

x M x (exp( β x ∆ PM) – 1), where: D = Number of Annua l Deaths Avoided; P = U.S. Popula tion (data taken from the Census Bureau website (www.census.gov)); M = Ba ckground mortality rate in the U.S. (deaths/100,000 population) (data taken

from the Centers for Disease Cont rol and Prevention (www.cdc.gov)); β = expert's C-R coefficient (percent change in mortality per 1 μg/m 3 change in PM 2.5 divided by 100); and ∆ PM = change in annual average PM 2.5 (μg/m 3 ).

24 In this example, 4 μg/m 3 is assumed to be the background level.2-20 

### (e.g., smoking) and to the major causes of d eath in the U.S. The calculations were intended to provide a “reality check” for the expert. The two calculations were performed for the expert’s 5 th 

### , 50 th 

### , and 95 th 

### percentiles of his distribution, and were also completed for other values at the request of the expert.

The Elicitation Team took extensive notes during the interviews. In some cases, experts may have written or sketched responses to cer tain questions. Following each interview, IEc provided the expert with a summary of his qualitative and quantitative judgments for review, adjustment and/or confirmation of his responses.

To maintain confidentiality, each expert w as assigned a randomized letter between A and J with which his judgments would be asso ciated in this report. We provided confidentiality to allow experts the freedom to express candid, independent opinions even if they should differ from those he has expressed publicly or from those of his employer.

2.6 POST-ELICITATION WORKSHOP 

### The project team held a final workshop with the experts following the completion of the elicitation interviews. The objectives of this workshop were to:

• 

### Anonymously share the results of all experts with the group;

• 

### Highlight areas where expert opinion varied for possible additional discussion;

• 

### Clarify points of confusion that may have emerged during the interviews;

• 

### Discuss data not available to all experts at outset of interviews;

• 

### Give experts the chance experts to rai se issues with which they had struggled

during the interview for a broader discussion; and

• 

### Encourage critical review by the experts of their judgments.

The Post-elicitation Workshop was not intended to force or otherwise promote a consensus. If points of agreement on issu es occurred naturally, we documented them. No efforts were made to encourage expert s to change their responses. All expert responses were presented using randomly assigne d letters. Individual expert summaries prepared following the elicitations were not provided to the rest of the expert group.

The Post-elicitation Workshop was held in J une of 2006. Eleven of the 12 experts participated in at least part of the workshop, and ten were present for three quarters of the discussions. The 4-hour workshop consisted of an overview of the qualitative and quantitative results from the 12 interviews, hi ghlighting both areas of general agreement and areas where expert opinion varied. The overview was followed by a series of discussion sessions focused on topics identified by the elicitation team that exhibited significant variation in opinion across experts and that were viewed as having a significant impact on the results. The purpose of these discussions was not to promote consensus, but rather to clarify points of confusion or differences in interpretation of specific questions. The workshop included an open discussion session where experts could raise topics for discussion or clarification, as well as a session where experts could2-21 

### provide feedback on the elicitation process. A discussion of the results of this workshop can be found in Chapter 3.

Following the Post-elicitation Workshop, a meeting summary and other follow-up materials requested at the Workshop were sen t to all 12 experts. All experts were provided an opportunity to revise their judgments privately following the workshop, on the basis of insights gained from the Po st-elicitation Workshop discussions, and were given a form to complete if they opted to ma ke changes. Experts were allowed to modify both their quantitative probabilistic distri butions as well as other quantitative and qualitative responses given during the interview. Experts were asked to clearly describe the changes requested and provide a detailed rati onale suitable for inclusion in this report. Where necessary, the Elicitation team followed up with the experts by phone to clarify
information submitted on the modification form.

2.7 EVALUATING EXPERTS’ JUDGMENTS 

### To understand how experts’ pe rformance is assessed, both in the broader expert judgment field and for this project, it is important to understand that each expert’s probabilistic judgment is a reflection of his or her own st ate of knowledge. (How well or with what degree of certainty does he think he can predic t the quantity of interest?) One measure of
his success is therefore related to how well he knows the limits of his knowledge. In the expert judgment field, this measure is known as calibration. An expert is well-calibrated if, for example, when asked to give his 90 percent confidence intervals for 100 predictions, his intervals contain the true value 90 percent of the time. A second important measure of the quality of an expert’s judgment is “informativeness,” an attribute reflecting the breadth of his confiden ce intervals. Two experts, one giving very broad intervals and the other very narrow, can both be well calibrated, but the latter is more informative. Finally, good judgment s should be unbiased. That is, even if an expert is well-calibrated using the definition above, his median should fall close to the expected “true” median for the predicted value.

Of particular concern for analysis and decision-making is that research has suggested that experts tend to be overconfident, and ther efore poorly calibrated (see Morgan and Henrion, 1990 for a review). In essence, th ey think they know more about the subject than they do, and therefore they express great er certainty in their predictions than is
warranted. Their confidence intervals tend to be overly narrow, causing them to “miss’ the true value entirely, or they may provid e biased estimates. In the calibration example discussed above, an overconfident expert’s 90 percent confidence intervals would contain the true value less than 90 percent of the tim e. Such inaccuracy in a key input to an analysis could have significant effects on the outcome.

At the same time, we do know that experts can perform well making predictions in their own areas of expertise. Studies have also shown that experts who receive regular feedback on their judgments (e.g., weather forecasters, see Murphy and Winkler, (1992); and physicians, see Winkler and Poses, (1993)). Walker et al. (2003) demonstrated that2-22 

### exposure experts asked to predict benzene concentrations in EPA Region V were relatively well-calibrated.

Techniques do exist to objectively evaluate experts’ performance. The “gold standard” for judging their performance requires that the “truth,” for example the true PM 2.5 

### C-R function, become known. For studies like this one, the gold standard is clearly beyond reach due to data gaps. Other investigator s have used additional sets of questions, for which the truth can subsequently be know n, to assess the calibration of experts (Cooke, 1991). This approach requires careful selection of a set of questions that are likely to be valid predictors of calibration on the questi ons that are the focus of the assessment.
Neither of these calibration alternatives was av ailable for this study. Concern about feasibility of devising calibration questions that would be equally applicable to experts with the variability in technical background represented by the expert panel led the Project Team to decide not to incorporate a calibration component into this study.

In the absence of these calibration measures, we developed the protocol, followed elicitation procedures, held workshops, and prov ided extensive briefing materials with the objective of helping the experts avoid some of the common biases and errors of judgment (also referred to as heuristics) that can le ad to poor calibration. Both during and subsequent to the interviews, we have eval uated the results of this study considering whether: 1) the judgments were statistically coherent; 2) the judgments were reasonably consistent with the rationales given by the experts; and 3) whether the process did help experts to avoid some of the common pitf alls associated with giving subjective judgments. The results of this evalua tion are described in Chapter 4.

2.8 APPROACHES TO PRESENTING RESULTS 

### When faced with differing judgments across e xperts, analysts must give careful thought to whether and how to combine these judgments into a single value or distribution. Many investigators (e.g., Hawkins and Graham, 1990; Winkler and Wallsten, 1995; and Morgan et al., 1984) have preferred to keep expert judgments separate in order to preserve the diversity of opinion on the issues of interest . In such situations, the range of values expressed by the experts can help decision-makers by serving as inputs to sensitivity analyses of analytical models and thereby bounding possible outcomes. Individual judgments can also illustrate dichotomies of opinion arising from different disciplinary perspectives or from the rational selection of a lternative theoretical models or data sets (Morgan and Henrion, 1990).

For this analysis, IEc has presented only the individual quantitative distributions of the mortality effect estimate elicited from the 12 e xperts interviewed. This approach differs from the Pilot Study, where we presented bot h individual expert distributions and a single, combined distribution using equal we ights for each of the five experts. The Project Team discussed this issue extensivel y during the planning phase of this project
and considered feedback on this issue from se veral sources. The decision not to combine expert responses was ultimately based on consideration of peer review comments on the pilot study, advice from EPA's Science Advisory Board (SAB), discussions from EPA's2-23 

### April 2005 symposium on the full-scale PM Expe rt Elicitation study, and the composition and nature of the expert panel.

The external peer review of the pilot study yielded extensive comments on the issue of combining of expert responses. The peer reviewers disagreed about whether experts' judgments should be combined as part of the expert elicitation study and none identified a generally agreed-upon method of combining expe rt responses. One reviewer felt strongly that expert judgments should not be combined, cautioning that the combined distribution could produce a result that none of the expe rts would endorse, and another recommended that combination should only occur outside of the expert elicitation study (e.g., through the pooling of benefit estimates generate d by running each expert's distribution individually through a benefits model.) No consensus approach to this issue emerged from the external peer review.

The Health Effects Subcommittee (HES) of EPA's SAB Advisory Council on Clean Air Compliance Analysis provided EPA with a limited review of the pilot study methodology as part of its review of plans for EPA’s second prospective Analysis of the Costs and Benefits of the Clean Air Act (EPA-S AB-COUNCIL-ADV-04-002, March 2004). In its recommendations, the committee strongly emphasized the presentation of individual judgments:

"...the HES advises EPA to present the entire collection of individual judgments; to carefully examine the collection of individual judgments noting the extent of agreement or disagreement; to thoughtfully assess the reasons for any disagreement; and to consider formal combinations of judgments only after such deliberation and with full awareness of the

context ...”

At the April 2005 EPA Symposium, the issu e of combining results was discussed among the invited health experts, some of who cite d concerns over the equal weighting of results in the pilot study. Several attendees sugg ested the use of weights derived from the content and quality of responses. 25 

### Ideally, these weighting systems would address problems of uneven calibration (i.e., accuracy) and informativeness (i.e., precision) across experts, as well as potential motivational biases (Cooke, 1991). 26 

### However, implementation can be problematic. Weights determined based on the analyst's judgment or the judgment of the other experts are likely to have a significant subjective component and may themselves be subject to motivational biases; more objective methods that rely on an expert's answers to calibration questions with known answers are appealing, but not universally endorsed, in part because of the difficulties of identifying appropriate

25 The literature includes several approache s that have been used to assign weights to individual expert s based on content

and expertise. Weights can be assigned ba sed on the analyst's opinion of the re lative expertise of each expert; on a quantitative assessment of the calibratio n and informativeness (i.e., precision) of ea ch expert based on their responses to a set of calibration questions (as described in Cooke, 1991); or on weights assigned by each expert, either to him or herself or to the other experts on the panel (see Evans et al., 1994 for an example of this approach).

26 "Motivational bias" refers to the willful distortion of an expert's true judgments. The origins of this bias can vary, but co uld

include, for example, a reluctance to co ntradict views expressed by one's employer or a deliberate attempt to skew the outcome of the study for political gain.2-24 

### questions for measuring expertise in the subject matter. The Project Team ultimately did not identify a preferable weighting scheme for combining responses.

Finally, the Project Team considered the selection process for the expert panel and its implications for combining judgments. A key objective of the selection process was to ensure representation of the full range of r espected scientific opinion. However, the resulting panel is not a statistical sample; it does not reflect the relative weights of the various opinions expressed by th e experts. Thus, a combin ation approach using equal weighting, for example, might overweight some opinions and underweight others. Lacking data on the prevalence of opinion in the expert universe, the team felt it would be inappropriate to present a mathematically comb ined distribution based on this sample.

The Project Team also considered behavioral approaches to combining expert responses, which require experts to interact in an effort to reach a consensus opini on. As in the pilot study, we chose not to consider a behavioral approach for several reasons. First, because there is disagreement regarding certain aspect s of the likely nature of the PM-mortality relationship, we viewed the potential for achie ving consensus to be very limited. Second, because the experts were selected to reflect the range of respected scientific opinions on this issue, individual elicitations enabled us to examine and preserve the variability in the experts’ responses. Finally, the use of indi vidual elicitations also avoids the dominance of the group opinion by any one individual and “attempts to preserve the unique perspective of each expert” (Wolff et al., 1990).

### CHAPTER 3 | RESUL T S

This chapter of the report presents the expert s' responses to the questions posed i n the elicitation pro t ocol. We di vide the results into two sections. We b e gin b y sum m arizing expert discussions in respo n se to the con d itioni ng q u estions, which covered topics addressing the key evidence for or against a PM 2. 5 

### -m ortality relationship as well as specific characteristics of such a relationshi p. The conditioning se ction generally follows
the order in w h ich the topics were raised in the elicitation interview. We then summarize the experts' re sponses to Part 4 of the protoc ol, including, their judgments concer ning the shape of the concentration-response (C- R ) f unction a nd their quantitative esti m a t es of the percent change in annual, a ll-cause m o rtality i n the adult U.S. population resulting from a

per m anent for a 1 μ g/m 3 

### re duction in annual average PM 2. 5 

### concentration across the U.S.

In the sections that follow, we identify t h e ke y ratio nales, key sourc es of data, and m a jor
uncertainties behind t h e experts’ quant it ative estimates of uncertainty, hig h lig hti ng the
i m portant commonalities and differences am ong th e experts’ opi ni ons. Where feasible, we also pres e n t tables summarizing the responses of each of the experts on a particular issue. (To pr eserve confidentiality of responses, we r e fer to experts using the let t ers A through L, w h ich were ran dom ly assigned to th em .) Detailed su mmarie s of eac h expert’ s responses to the protocol questions can be found in t h e Technical S upport Document for
this stud y (IE c, under deve lopm ent).

3.1 RES P ONSES T O CO NDITIO NIN G QUESTI ONS 

### The first half of each interview was devot ed to addressing a broad se t of "conditioning questions." The goal of these questions w as to help each expert bring to m i nd and critique the scientific evidence they thought t o be rel e vant to answering the qua ntitative questions about the PM 2. 5 

### -m ort a lity relationship. Experts were encouraged to c onsider
both relevant theor y as well as em pirical evidence from a wide ran g e of scientific disciplines (e.g., epidem iology , clinical medi cine, toxicology , exposure ass ess me nt).

Becaus e expe rts (as w e ll as lay people) tend to bring to m i nd the evidence with which they are m o st fam iliar or to which the y have been m o st recently exposed, the role of the
elicitors in thi s discussion is to bring for w ard evidence that an expert may not ha ve considered. While the pro t ocol intro duc ed the conditi oning questio ns in a particular

sequence, ex perts were offered the oppo rtunit y t o respond in the or der that made the m o st
sense to the m

.

3-13-2 3.1.1 MAJOR CAUSES OF AND MECHANISMS FOR PM 2.5 -RELATED MORTALITY 

### The quantitative characterization of PM 2.5 

### -related mortality focused on total mortality. Consequently, the first set of questions in the protocol was important for understanding what the experts' thought the principal “drivers” of total mortality might be, such as the causes of death, the biological mechanisms, and the relative importance of long-term exposures versus short-term exposures in contributing to total mortality. These discussions also provided an important foundation for the experts' later evaluations of the strength of the causal relationship and their articulation of the potential existence of
thresholds in the C-R function.

Experts were first asked to discuss whether th ey thought the mechanisms for PM related mortality differed between short-term exposur es and long-term exposures. Experts A, B, C, F, G, H, K, and L chose to discuss the effects separately and the remaining four experts discussed the effects together. Despite their distinguishing between the effects of long and short-term exposures, it became clear that these eight experts saw some overlap across the two temporal domains. 27



### Regardless of experts' decisions to discu ss the mechanisms and causes of death from short-term and long-term exposures separately or together, the types of effects often fell into similar categories. The main causes of death resulting from short-term exposures to

PM 2.5 

### discussed by the experts were related to acute cardiac or respiratory events. Some experts also mentioned ischemic stroke as another plausible outcome. The three main contributors to mortality from long-term exposure discussed were cardiovascular mortality, chronic respiratory disease, and lung cancer.

Although the summaries below discuss some of the general patterns observed and the literature cited, it is important to recognize that there was often substantial variation in the depth and detail provided by i ndividual experts. Some provi ded more general statements about kinds of evidence; others provided either detailed diagrams of mechanistic pathways or cited individual papers in support of particular viewpoints.

Cardiovascular disease 

### The mechanism that most experts thought was most plausible for explaining acute cardiac effects resulting from short-term particle exposures involved an oxidative stress response to particles deposited in the lung or translo cated systemically, with release of reactive oxygen species leading to acute inflammation, cytokine release, compromise of the cardiac endothelium, and a fatal cardiac event. Acute responses could involve changes in blood coagulability/viscosity (Seaton et al., 1999; Peters et al., 2001; Riediker et al.,

27 In particular, most of the experts cited the potential role of the "oxidative stress" pathways in which exposures to particles

lead to inflammation and the release of cy tokines, fibrinogen and othe r factors that may either di rectly impact the lung or be transported to and affect the heart or brain. Experts thought that this path way represented a link between mechanisms for effects from short-term and long-term exposures. Pope et al. (2004a) and Ghio et al. (2000 & 2004) were often cited for the basic oxidative stress hypo thesis. Additional scientific research supporting this hypoth esis most frequently included work by Godleski (2000); Wellenius et al. (2003); Costa and Ko davanti (2003), and Sun et al. (2005). Expert E also cited work by Gurgueira (2002), Evelson and Gonz alez-Flecha (2000), and Rhoden (2004) in su pport of the role of reactive oxygen species and their impact on inflamma tion in the endothelium (cells lining the inner walls of blood vessels).3-3 

### 2004), and decreased plaque stability leading to the potential for increased risk of myocardial infarction (Zeka et al., 2005; Wellenius et al., 2003; Dominici et al., 2006). Experts A, E, G, and L also discussed ischem ic stroke as potential causes of death, since it too could be triggered by this mechanis m. Several experts mentioned "endothelial dysfunction" as another contributor to ca rdiovascular mortality from short-term exposures. Expert E cited a study by O'Neill et al. (2005) on brachial artery reactivity; Expert K mentioned Brook et al. (2002); Expert L mentioned Yamawaki et al.'s (2006) work on endothelial dysfunction and carbon black, though more in connection with a discussion of long-term exposures.

A second short-term cardiac mechanism discu ssed by several experts was related to particle-induced changes in the autonomic nervous system. Most experts discussed evidence showing that particle exposures ma y be associated with disturbances in the autonomic nervous system leading to cha nges in heart rate variability and other alterations. Here experts cited research on changes in heart rate variability (Pope et al., 1999; Gold et al., 2000; Creason et al., 2001; Devlin et al., 2003; Schwartz et al., 2005) and the defibrillator studies (e.g., Peters et al., 2000; Dockery et al., 2005). Two experts (E, L) thought that it would ultimately be f ound that this mechanism is related to the oxidative stress pathway, citing Schwartz et al. (2005).

Discussions of the cardiovascular effects of lo ng-term exposures to fine particles were dominated by the hypothesis that chronic oxidative stress and inflammation in the endothelium contributing to accelerated formation of atherosclerotic plaque. The Sun et al. (2005) work was particularly influentia l in these discussions. This study reported progression of atherosclerosis in a strain of mice genetically pre-disposed to atherosclerosis (ApoE-/-) chronically exposed to fine particles. Many of the experts noted that this study provided a critical empirical link between studies showing the release of reactive oxygen species, thickening of the lining of carotid arteries (e.g., Kunzli et al., 2005), endothelial dysfunction, and epidemiologic studies showing increased cardiovascular mortality in populations exposed to fine particles. Expert L noted that works by Suwa et al. (2002) and Goto et al. (2004) were also supportive in this regard.

Respiratory Mortality 

### The second cause of death most experts discussed in connection with exposure to fine particles was respiratory-related diseases. Several experts expressed the view that the evidence was weaker for respiratory than for cardiac effects, due in part to diminished statistical power to detect respiratory related deaths (e.g., deaths from respiratory disease are less prevalent and are more likely to be miscoded).

Several experts expressed a role for short-term exposures to particles in contributing to respiratory mortality. These experts generally indicated that short-term PM exposures contribute to respiratory mortality largely through exacerbation of existing conditions (e.g., Chronic Obstructive Pulmonary Disease (COPD), asthma, allergies, other underlying pulmonary disease, or heart disease). Experts D and F specifically discussed alteration in immune responses; F cited studies by Gilmour et al. (2002) and Zelikoff et al. (2003) showing increased mortality rates in animals exposed to both streptococcus and3-4 

### particles compared with animals exposed to streptococcus alone and Plopper and Fanucchi (2000) showing altered immu ne defenses in exposed animals.

Experts E, F, G, H, I, J, K, and L chose to discuss a potential role for long-term exposures to particles in contributing to increased resp iratory mortality. They generally focused on mechanistic and epidemiological evidence for changes in lung function and the implications for mortality. Expert E discu ssed particle related development of COPD,positing a role for reactive oxygen species in promoting inflammation, mucous hypersecretion, and structural damage to the lung. He cited Saldiva et al.’s (2002) concentrated air particles (CAPs) study showing increased capillary wall thickness in the lungs of animals exposed to particles compared to unexposed animals and epidemiological evidence from the Adventists Health and Smog (AHSMOG) studies (Abbey et al., 1991 & 1999; McDonnell et al., 2000; Chen et al., 2005), Schwartz (1993), and the Children's Health Study (Avol et al., 2001; Gauderman et al., 2004). In support of the mechanism for lung damage, Experts F and H cited other CAPs studies in animals by Ghio et al. (2000) showing increased inflammation as indicated by blood fibrinogen. Experts E, G, K, and L all brought up the Children's Health Study in California (Avol et al., 2001; Gauderman et al., 2004) showing diminished lung function growth in high-pollution communities, and decreases in lung function in children who moved from lower to higher pollution areas as evidence of the possible contribution of chronic particle exposure to changes in lung health and ultimately to respiratory mortality. Expert K was not convinced that these changes were particle specific, noting that NO 2 

### appeared to play a role. Other experts suggested that NO 2 

### might have been a surrogate for traffic-related particulate matter.

Views on the possible contribution of partic le exposure to the development of asthma were limited. Two experts cited growing evid ence that early exposures to particles might contribute to the development of asthma; E xpert K cited Diaz-Sanchez et al. (1999) as some evidence for a role of particles in the development of asthma. Expert L did not think the evidence on particles and asthma development provided a clear story.

Cancer 

### Several experts discussed lung cancer in c onnection with long-term exposures to PM 2.5


### . Lung cancer arose in discussions of short-term exposures only to the extent that cancer patients might be a susceptible subgroup for short-term exposure effects.

The most common view expressed was that a connection between long-term PM exposure and lung cancer was scientifically plausible. Many expressed the basic argument that particulates contain mutagenic and/or carcinogenic chemicals that can act as initiators and/or promoters and that lung inflammation and the resulting cell turnover can act as a promoter. Several experts raised smoking as an analogy.

However, most noted that the epidemiologi c data remained limited for lung cancer. Several experts cited Pope et al. (2002) as the strongest epidemiologic evidence of a link between particulate exposure and lung cancer and thought that findings from several other studies, though not statistically significant, were generally supportive. The strength of opinion about cancer ranged from Expert B, who thought there was a greater weight of3-5 

### evidence supporting a link between lung cancer and particle exposure than for cardiovascular mortality, to Expert K who argued that the link was not strong and likely a function of poor control for confounding by smoking (notin g the unreliability of personal smoking histories).

3.1.2 RELATIVE IMPORTANCE OF MORT ALITY IMPACTS FROM SHORT-TERM AND

LONG-TERM EXPOSURE CHANGES 

### The goal of this elicitation was ultimately to obtain a quantitative estimate for the total annual mortality effect that may resu lt from a reduction in annual average PM 2.5 

### including both changes in short-term (e.g., 24-hour) and long-term exposures to PM 2.5 

### . Developing an aggregated mortality estimate that refl ects reductions in both short-term peak and long-term average exposures to PM 2.5 

### requires both assessing the relative contribution of short-term and long-term impacts to the overall change in mortality and thinking about the potential for overlap in the signals repor ted in different types of epidemiological studies (e.g., how much of the short-term impacts are captured by long-term cohort studies). 28 

### To assist experts in approaching this problem, we discussed with each of them the conceptual framework first presented in Kunzli et al. (2001). This framework uses Venn diagrams to describe the relationship betw een the deaths attributable to long-term exposures and those attributable to short-term exposures to fine particles (see Exhibit 3- 1). Experts were encouraged to review and critically evaluate this diagram and discuss their own views about the relative contributions of different types of mortality impacts, developing alternative frameworks if necessary.

Nearly all experts found the structure described by Kunzli et al. (2001) a useful starting point for discussions though most discussed possible modifications to this basic conceptual framework. While the experts were not asked to specifically quantify the relative contributions of the different types of mortality, many discussed their views of the relative sizes of the different sets of mortality impacts. Exhibit 3-2 presents a summary of the experts’ views on this issue based on responses to this and other sections of the protocol.

28 For example, cohort studies focus primarily on analyzing the impact of long-term exposures to PM 2.5 but may also capture

some of the impact of short-term variat ions in exposure during the cohort follow-up period. Time-series studies analyze the impacts of daily or short-term variations in PM concentrations and can characterize th e cumulative impact of exposure over a few days, but not over a longer peri od of time. Those who rely exclusively on cohort studies may not account for all of the mortality impacts of short-term exposures, and therefore, may underestima te total mortality impacts. Those who rely on a sum of effects estimated in both cohort and time-s eries studies may overestimate mortality impacts.EX HI B I T 3-1 : CO N C EPT U A L FR A M EW O R K F O R S H O R T

- T E R M AN D L O NG- T E R M M O R T AL I T Y E F F E C T S

Im pact o f A i r Po llutio n Catego ry o f Cases Underlying frailt y due to air po ll utio n Occurrence o f death (event) trig gered by air pol l ution

A Y e s Y e s B Y e s N o C N o Y e s D N o N o

A: A i r pollution incr eases both the risk of und erlying diseases leading to frailty and th e sh or t-te r m r i sk of de ath a m on g th e fr ai l. For

exam pl e, patients with chro nic bro n chitis that has been enhanced by l o ng-term air po ll utio n expo s u re m a y be ho spital ized with a n
acute air po l l u tio n -rel ated exacerbatio n o f thei r illness leading to death shortly afterward.

B: Air pollution incr eases the risk of chronic diseases leading to frailty but is unrelated to timing of death. For example

, a pe rson’s

suffering fro m c h ro nic bro n chitis m a y be enhanced by lo ng-term a m bient air po llut i on exposure but the person may die due to acu te

pneumonia acqui r ed during a clean air period.

C: A i r pollutio n is unrelated to risk o f chro nic disease but sho r t-term expo sure increases m o rtality a m o n g perso n s who are frail. For

exam pl e, a pers o n with diabetes m e l l i tus m a y be susceptibl e to h e art attacks due to long-standing coronary disease; in such a c ase, an
air po ll utio n epis o d e m a y trigger the fatal infarction leading to death.

D: Neither underl y i n g chro nic disease no r the event o f death is rel a t e d to expo sure to air po l l u tio n

. 

# A

C B All Ai r Poll ution
Rel a t ed D e ath s

Mixed Effects All D e aths



# s h

or t

- t

erm
e

ff

ec t s l ong- t erm

e

ff

ec t s



# D -

Circle sizes do not reflect relative effe cts. (Adapted from Kunzli et al., 2001).

3-63-7 EXHIBIT 3-2: RELATIVE IMPORTANCE OF SHORT-TERM AND LONG-TERM IMPACTS

RESPONSE EXPERTS COMMENTS

Emphasized longer-term impacts A, B, C, E, F, G, H, I, J, L Emphasized categories A, B, and/or A+B. Category C viewed as small contributor to overall mortality.

Uncertain D Thought he had insufficient information to discuss relative sizes of sets and their overlap, but relied on cohort studies for quantification Emphasized short-term impacts K Emphasized the importance of peak exposure episodes in exacerbation of underlying disease process.



### All but two experts placed much more weight on long-term exposure changes as drivers of changes in mortality. This conclusion was based on epidemiological evidence, supported by toxicological and clinical ev idence for plausible mechanisms. These experts emphasized the potential for cumulative chronic damage over time from PM exposure leading to increased frailty and corr esponding risk of death (Category B), with
or without the involvement of short-term e xposures as acute triggers of mortality. All experts in this group thought that the short-te rm mortality impacts not included in relative risk estimates from the cohort studies represen ted a very small percentage of the total mortality impact. 29 

### Though not explicitly addressed by the protocol, some experts did discuss their views on latency for mortality related to long-term PM 2.5 

### exposures. Most experts believed that the cardiovascular and respiratory mortality effects observed in the cohort studies are primarily due to exposures over the previous five to 10 years. Experts mentioned intervention studies, as well as the Si x Cities follow-up analysis of Laden et al. (2006), as empirical support for latency periods measured in years rather than decades. Experts noted that we still lack empiri cal data on this important question.

One expert (K) expressed greater confidence that the mortality impacts reflect changes in short-term peak exposures. He found the bod y of evidence for short-term impacts more complete, compelling, and coherent than that for long-term impacts. Expert K expressed the belief that the mortality impact largely represents acute exacerbation of underlying disease processes. He also noted that his interpretation of this framework would

29 In support of this position, Experts H and J cited studies that cast doubt on the hypothesis that time-series results

represented "harvesting" (i.e., the displa cement of mortality among the frail by a very short time period). H cited an analysis by Zeger et al. (1999) that simu lated the potential effect of harvesting on a dataset and could not demonstrate the anticipated harvesting effects upon analy sis of the data. In addition, he indicate d that papers by Joel Schwartz (2000) found increasing coefficient size with longer time window, which “is the opposite of ... harvesting.” Expert J cited the work by Zeger and Schwartz as well as a study of the harvesting issu e by Dominici et al. (2003). He thought that the evidence suggests, “that daily time-series studies util izing only short-term time, day to day va riability, are observing more than just the phenomena of short-term harvesting or mortality displa cement. These results suggest the daily time-series studies capture only a small amount of the over all health effect of long-term-related exposure to particulate air pollution."3-8 

### significantly shrink the size of the sets of air pollution-related deaths (circles B and C) relative to All Deaths (circle D).

One expert (Expert D) thought the categories of death postulated in the Kunzli diagram made sense conceptually, but thought he l acked sufficient data to inform a decision regarding the relative sizes and overlap of the different categories. He did indicate that he was comfortable that the estimates from cohort studies of long-term exposures are capturing the percent change in mortality associ ated with changes on in annual average

PM 2.5 

### levels.

3.1.3 KEY EPIDEMIOLOGICAL EVIDENCE 

### Discussions of key epidemiological evidence supporting or refuting a relationship
between PM 2.5 

### and mortality permeated the interv iews. As part of the conditioning questions, experts first discussed the role of epidemiologic study design in characterizing the total impacts of PM 2.5 

### on mortality and then were ask ed to describe the characteristics of an “ideal” epidemiologic study for addressing this issue. The discu ssion then turned to the strengths and limitations of the existing epidemiological evidence, and how it compares with their ideal study. The expert s revisited this topic during the quantitative questions while deciding the relative emphasis they would place on each study in developing their quantitative estimates. Expe rts were not required to provide quantitative weights, but were asked to indicate when or how they used specific studies in developing quantitative estimates. This section focuses on the theoretical discussion and the initial survey and evaluation of studies the experts found relevant.

The Role of Epidemiological Study Design in Characterizing the Total Impacts of

PM 2.5 Exposures on Mortality 

### The experts were first asked to describe the types of epidemiologi c study designs that they thought were most useful for estimating the change in total annual mortality related to a permanent reduction in ambient PM 2.5 

### concentration. The protocol then asked the experts to explain the extent to which the st udy designs that they chose capture effects from short-term and long-term exposures.

All of the experts thought that cohort studies would capture a large portion of the annual deaths from a drop in annual average ambient PM 2.5 

### exposures. Experts thought that these studies captured long-term effects as well as some short-term effects. Most of the experts thought that cohort studies missed d eaths caused by very short-term exposureresponses. However, most thought that this small proportion of deaths missed by cohort studies was very small in relation to those captured by cohort studies. For example, Expert F estimated that roughly 97% of the mortality effects reported by cohort studies were due to long-term exposures with the rema ining 3% attributable to cumulative shortterm exposure effects, whereas Expert L thought that the short-term effects missed by the cohort studies might be around 0.5% per 10 μg/m 3 

### (i.e., similar to the effect estimates found in the NMMAPS study (Samet et al., 2000a & b)), which is equivalent to about five one-hundredths of a percent for a change of 1 μ g/m 3 

### . Expert E thought that cohort studies looking at changes in air pollution over time were the most directly relevant for measuring the mortality effects of changes in annual average PM 2.5 

### concentrations. He3-9 

### thought that the Six Cities follow-up (Laden et al., 2006) is the only such study that has assessed the effect of long-term annual av erage PM changes over time on mortality.

All of the experts also thought time-series studies with various lag lengths could be useful in capturing true short-term effects (e.g., lags up to one week) and potentially some intermediate length effects (e.g., lags up to 1-2 months), depending on the length of the lag. Expert H thought that since the majority of short-term effects were already captured by the cohort studies, that time-series st udies were more informative in establishing plausibility than in quantifying mortality effects. Ultimately, as discussed below, most experts did not rely on time-series studies wh en creating their quantitative distributions, except in some cases to support a lower-bound effect estimate.

All of the experts discussed intervention st udies to some extent when answering this question. Some thought of them as supporting evidence, and not directly useful for quantifying the C-R function. Others thoughtthese studies quantified some intermediatelength effects (e.g., between time-series and cohor t) and some short-term effects. Expert G thought they were useful for determini ng which PM components might be causing the mortality. Expert A indicated that intervention studies would be most informative for the quantitative question because he thought most of the mortality effects occur in weeks to months. However he noted that the re-analy sis of the Six Cities data (Laden et al., 2006), which looks over the last 10 years, might capture some long-term effects as well.

Experts H and J thought that cross-sectional or ecologic studies would capture the same effects as cohort studies (long-term and some short-term), but Expert H expressed
concern about the lack of control for indi vidual level confounders in these studies. Expert F thought that case-crossover studies w ould capture the same effects as time-series studies (short-term). Expert L indicated that case-control studies were useful for examining long-term effects for specific outcomes, such as lung cancer or COPD.

Ideal Epidemiologic Study 

### Experts were asked to describe an ideal epidemiologic study for answering the specific quantitative question posed by the protocol. The intent of this question was to provide experts with a “gold standard” against which to evaluate existing epidemiologic studies. The following are characteristics of an ideal epidemiologic study that were mentioned by several experts:

• 

### Geographically representative of the entire U.S. (e.g., monitoring sites across the

country);

• 

### Collection of information on individual ri sk factors and residential information

both at the beginning and throughout the follow-up period;

• 

### Large sample size that is representative of the general U.S. population;

• 

### Collection of genetic information from cohort members to identify and assess

potential effect modifiers;

• 

### Monitoring of individual exposures (e.g., with a personal monitor);3-10 • 

### Collection of data on levels of several co-pollutants (not only those that are

monitored for compliance purposes);

• 

### Accurate characterization of outcome (i.e., cause of death);

• 

### Follow-up for a long period of time, up to a lifetime; and

• 

### Prospective study design.

Experts E and J thought it would be useful to include cities that have differing changes in air pollution over time. Expert G said he w ould conduct a cohort and time-series study in the same population. He would then conduct st udies in three cities, each of which has a different level of PM (although of a similar mix). Expert K expressed a preference for having an intervention in exposure. Expert L thought that having spatially resolved exposure data at the neighbor hood level would be useful.

Epidemiologic Evidence for the Impact of Exposures to PM 2.5 on Mortality 

### Experts were asked to discuss which specifi c existing epidemiologic studies they think are informative for addressing the quantitative question and to describe their strengths
and limitations.

Exhibit 3-3 indicates the epidemiologic studies mentioned by each expert in response to the conditioning question. All of the experts cited the following long-term, cohort-based studies as major evidence in support of a positive relationship between ambient annual average PM 2.5 

### concentrations and mortality:

• 

### The Six Cities Cohort Studies (including the original study (Dockery et al., 1993),

the reanalysis (Krewski et al., 2000a & b) and the follow-up (Laden et al., 2006));
and

• 

### The ACS Studies (including the original st udy (Pope et al., 1995), the reanalysis

(Krewski et al., 2000a & b), the follow-up (Pope et al., 2002) and analysis of follow-up data with a focus on cardiovascular outcomes (Pope et al., 2004))

The experts were uniform in their statements that the original Six Cities and ACS studies were well-conducted and that their results pr oved robust upon extensive reanalysis by Krewski et al. (2000a & b). One key strength of the Six Cities study often noted was that it was designed specifically for the purpose of evaluating the relationships between air pollution and health. The Six Cities study was prospective in nature, included recruitment of representative samples of subjects in each community, had reasonable control for possible confounders and eff ect modifiers, and set up purposeful air monitoring at sites chosen to characterize cohort exposures. In addition, some experts thought the choice of six cities with a wide range of PM levels was a strength. Frequently cited limitations of the Six Cities study incl uded the small sample size, limited number of cities, and concerns about the representativen ess of the six cities for the U.S. as a whole (since important regions of the U.S., such as the Southwest, Midwest and California, were not represented).3-11EXHIBIT 3-3: EPIDEMIOLOGIC STUDIES DISCUSSED BYEXPERTS WHILE ANSWERING CONDITIONING QUESTIONS

Six Cities (Dockery et al., 1993; Krewski et al., 2000a & b; Laden et al., 2006) ACS (Pope et al., 1995, 2002 & 2004; Krewski et al., 2000a & b) ACS LA Reanalysis (Jerrett et al., 2003 & 2005 Utah Valley (Pope et al., 1989, 1991, 1996; Ghio et al., 2004) Netherlands Cohort Study (Hoek et al., 2002) AHSMOG (Abbey et al., 1991 & 1999; McDonnell et

al., 2000; Chen et al., 2005) Veteran's (Lipfert et al.,

2000, 2003 & 2006) Dublin Study (Clancy et al., 2002) Elderly Californians Study

(Enstrom et al., 2005) Hong Kong Study (Hedley et al., 2002)

APHEA 1 NMMAPS (Samet et. al., 2000a & b) Willis et al., 2002 Canadian Time-Series Studies (Burnett et al., 2000 & 2003) Finkelstein et al., 2004 MESA Cohort 2 Mallick et al., 2002 Filluel et al., 2005 Woodruff et al., 1997 Women's Health Initiative 2Expert A999999999Expert B9999999999Expert C999999Expert D999999Expert E99999Expert F999Expert G9999Expert H9999Expert I999999999999Expert J9999999999Expert K9999Expert L999999Total:121210555544332211111111The Air Pollution and Health - A European Approach (APHEA) includes a large group of studies.For full list of papers, pleaseconsulthttp://airnet.iras.uu.nl/products/reports_and_annexes/APHEA/APHEA_publications.pdf.2Study not yet published at the time of the interview.3-12 

### Many of the experts praised the ACS study for its large sample size, the large number of cities, broad geographic scope, and collection of individual risk factors, including those pertaining to cancer and cardiovascular disease. The limitation of the ACS study mentioned by most experts was the method of recruitment for the study, which resulted in a group with higher income, more education, and a greater proportion of whites than is representative of the general U.S. population. Also, several experts thought that the ACS exposure assessment was more problematic than the Six Cities study; the ACS study had to rely on whatever monitors were available to the study. This led to a single monitor representing exposure for an entire metropolitan area (containing several counties), whereas the Six Cities study often had exposures assigned at the county level.

Ten out of the twelve experts discussed the ACS Los Angeles (ACS LA) reanalysis by Jerrett et al. (2005). Experts explained that this study examined a subset of ACS cohort members in LA, assigning them to exposures based on a model estimating PM 2.5 

### in the same zip code as their residence. The effect estimates were two- to three-fold higher than the original ACS study. Experts thought that the better spatial resolution in this study was its main strength, and many believed th at the higher effect estimates reflected reduced exposure errors. However, several experts were concerned that it only included one city with a specific mix of PM that may di ffer from the rest of the U.S. Experts F and H thought that only including one city may lead to spatial autocorrelation and residual confounding and Expert F a dded that there could be within-MSA mobility impacts on exposure. Two experts discussed other analyses examining the effect of improving exposure characterization. Expert B discussed a paper by Mallick et al. (2002) that performed a hypothetical analysis that attemp ted to correct for exposure misclassification in the Six Cities data. This study also found two- to three-fold higher effect estimates than the original study. Experts B and E mentioned a paper by Willis et al. (2003) that only included those individuals in the ACS cohort who lived in the same county as the exposure monitor. There was a doubling of th e sulfate coefficient for all-cause mortality in the restricted cohort compared to includi ng all individuals in the metropolitan-area. Although this coefficient was for sulfates, he thought that it provided evidence that better spatial resolution reduces exposure misclassifi cation and leads to increased effect
estimates.

Three other cohorts that were either disc ussed by experts or brought up during the interviews by the elicitors included the Veterans’ Association (VA) Cohort Studies (Lipfert et al., 2000, 2003 & 2006), AHSMOG study of Seventh Day Adventists in southern California (Abbey et al., 1991 & 1999; McDonnell et al., 2000; Chen et al., 2005), and a study of elderly Californian’s (Ens trom et al., 2005). Generally, experts did not weight these studies as highly as the cohort studies described above and only Expert J
used these studies to inform his uncertainty in the C-R coefficient. Many of the experts were not familiar with the methodological details of these studies. Of the experts that did discuss the studies, many thought that they we re generally supportive of the PM-mortality relationship and could be used qualitatively, but not quantitatively. Some of the main limitations of the AHSMOG study mentioned by experts included small sample size, too many subgroup analyses, and poor exposure characterization. Some experts thought the analytical approach used in the VA study w as unclear. Experts thought all three studies3-13 

### included populations that were not representativ e of the general U.S. population and also were concerned that they were not published in , and thus did not receive peer review by, epidemiologic journals. 30



### Eight experts discussed intervention or exposur e change studies during this section of the protocol. The studies most often cited by experts included the “Dublin Study” (Clancy et al., 2002), the “Hong Kong Study” (Hedley et al., 2002), the “Utah Valley Studies” (Pope et al., 1989, 1991 & 1996; Ghio et al., 2004) and the Six Cities “Cha nge” analysis (Laden et al., 2006). Six of these experts thought that these studies provided supporting evidence for plausibility, rather than informing th e quantitative question. The exception is the Laden et al., 2006 “change” estimate, which was relied on by a two experts (see Exhibit 3-14). The most commonly cited strengths of these studies include large changes in exposure and avoidance of potential confounde rs associated with differences between populations. Expert E also thought that the Laden et al., 2006 was the only existing study that directly addressed the question posed by the protocol. Commonly discussed limitations included potential confounding by time-varying covariates, the fact that not all
of these studies used PM 2.5 

### as an exposure measure, and the fact that they do not capture all of the long-term effects, as discussed above.

3.1.4 CONFOUNDING 

### Confounding is an issue that often is rai sed when interpreting quantitative results of epidemiologic studies. In the present context, a potential confounder is a variable that is both an independent risk factor for prematur e mortality and also is correlated with PM 2.5 

### concentrations. Only if both conditions are met can a variable act as a confounder. If not adequately controlled in the design and/or analysis of an epidemiology study, confounders could lead to biased results (e ither upwards or downwards) in the effect
estimate for PM 2.5 

### . Because of the potential importance of this issue in the present context, discussion of potential confounders was a key component of the elicitation
interview.

Questions on confounding were directed at understanding each expert’s views on the influence, if any, of confounding in the sp ecific studies he cited as most relevant to his quantitative judgments concerning the form, magnitude, and uncertainty in the C-R function for mortality related to ambient PM 2.5 

### exposure. Experts began by listing what they believed to be potential confounders of the relationship between exposure to PM 2.5 

### and mortality in relevant studies of long-t erm and/or short-term exposures. Examples from the epidemiologic literature were provided as a starting point for discussion. Experts then discussed the theoretical rati onale (e.g., biological or toxicological mechanism) or empirical evidence (e.g., clinical, epidemiological, animal, or exposure studies) for the impact of each potential confounder on the PM 2.5 

### -mortality effect.

30 The VA study assessed male Veterans wi th high blood pressure, originally recr uited to assess the efficiency of antihypertension drugs, and the AHSMOG study involves only Seventh Day Adventists, a relatively small pool likely to be representative of a healthier popula tion. The Enstrom et al. paper on ly included elderly Californians.3-14 

### Experts discussed their views on the extent to which each potential confounder had been adequately controlled for in the relevant ep idemiologic studies. For confounders that were considered to have not been adequately controlled, experts discussed the direction and magnitude of the possible resulting bias in the PM 2.5 

### effect estimate. We asked experts to try to classify the magnitude of bi as on a three-level scale: 1 for minimal bias; 2 for medium bias; and 3 for major bias. Th e quantitative interpretation of this scale was left to the expert to define.

Experts mentioned a number of potential confounde rs in the interviews, but most of them believed that most or all of the key potentia l confounders had been adequately controlled for in the key studies they cited (i.e., the ACS and Six Cities studies, and their re-analyses and extended analyses: Pope et al., 1995 & 2002; Krewski et al., 2000a & b; Jerrett et al., 2005; Dockery et al., 1993; Laden et al., 2006). 31 

### For this reason, few made adjustments to their quantitative judgments to take account of bias due to confounding (see below). Three of 12 (F, G, D) made some mention of confounding (by co-pollutants and/or occupational exposures) in deriving their medi an estimate for the mortality effect of

PM 2.5


### . Contextual socio-economic status (SES) va riables and co-pollutants were the two categories of potential confounders that were discussed most extensively and for which moderate concern regarding bias was sometimes noted.

• 

### Contextual SES. The ACS and Six Cities cohort studies both assessed SES based

on educational attainment, which is onl y one of many possible measures of economic status. The Jerrett et al. (2005) re-analysis of the Los Angeles ACS cohort explored the influence of an ex tended set of Census-based SES variables on the mortality- PM 2.5 

### relationship. In general, a dding more of these “contextual”
variables tended to diminish but not eliminate the PM 2.5 

### effect estimate. Some Experts viewed this as evidence that c ontextual SES variables are confounders that, if not taken into account, lead to positive bias in the PM 2.5 

### effect estimate. Others, such as Expert L, pointed out that some of the contextual SES variables might be surrogates for exposures to PM 2.5 

### , in which case including them would lead to a negative bias in the PM 2.5 

### effect estimate. Ultimately however, no experts adjusted their median effect estimates for this factor.

• 

### Co-pollutants. Co-pollutants were also frequently discussed. Most experts

considered the SO 2 

### effect observed in the ACS study as implausible since SO 2 

### levels were quite low and most SO 2 

### deposits in the upper respiratory system. Several experts were of the opinion that SO 2 

### can be viewed as a surrogate for PM in the ACS study, since sulfate particles are derived from SO 2 

### . Expert D made a case for confounding by volatile and semi-volatile organic compounds freshly
emitted from motor vehicles. He noted th at these compounds have been shown in laboratory studies to have relevant health effects, and also correlate with patterns

31 Potential confounders cited included smok ing, age, socio-economic position (at both the individual and contextual levels),

ethnicity, occupation, co-polluta nts, diet, exercise/obesity, we ather, pre-existing health stat us, time-varying trends in health-care, indoor exposures, and differential migration.3-15 

### of PM 2.5 

### concentrations in the ambient air, making them plausible confounders. Expert D adjusted his median effect estimate downwards slightly to account for the positive bias that may exist in the cohort studies. Expert K also noted concern about un-regulated co-pollutants, but made no adjustment for this factor. Expert G believed that the Six Cities study was bi ased upwards due to confounding by copollutants (NO2, ozone, SO 2 

### , semi-volatile organics), and took account of this in deriving his median effect estimate. In th e context of later discussions of exposure issues, Expert J mentioned the difficult y of directly addressing the quantitative question that was the focus of the elicitation (i.e., what is the mortality effect of a change in PM 2.5 

### concentration, holding all co-pollutants constant). Since PM 2.5

### contrasts (spatial as well as temporal) occur in concert with contrasts in copollutants (due to overlapping sources), it is difficult to quantify the marginal impact of PM 2.5 

### . If one wants the marginal effect of PM 2.5 

### alone, which is the stated focus of the elicitation, the PM eff ect estimates in key epidemiology studies
could be moderate over-estim ates of the true PM effect for the U.S. population since they incorporate some unknown quantity of co-pollutant effects. This issue was also discussed by other experts, as well as the related issue of differential toxicity for different particle components.

Among the other potential confounders, o ccupational exposures and smoking were frequently mentioned. Only Expert F adjusted his median effect estimate downwards slightly to account for possible bias due to occupational exposures. Smoking was mentioned by most experts, while noting that both the ACS and Six Cities studies controlled well for this factor using individua l-level smoking histories. Only Expert K thought that smoking had not been adequately controlled for in the cohort studies. Expert K rated smoking as a potential level 1-2 positive bias, but ultimately did not incorporate this into his quantitative judgments.

3.1.5 EFFECT MODIFICATION 

### Effect modification is a term used by epidemiologists to refer to the phenomenon in which the magnitude of an estimated health e ffect (e.g., percent change in mortality for a

1 μ g/m 3 

### change in exposure to PM 2.5 

### ) differs for different groups of people defined on the basis of some variable such as age, gender, r ace, economic status etc. The variable used to stratify the population is referred to as the effect modifier. Whereas uncontrolled confounding may result in biased and invalid effect estimates, effect modification does not threaten the validity of a study. It can , however, influence the generalizability of an effect estimate reported by an individual study to the general population. This can happen if the distributions of the effect modifier differ in the general population as compared with the study population. Dependi ng on the direction and magnitude of this difference, the effect reported by the study may underestimate or overestimate the effect in the general population. Because of the pot ential importance of this issue in the present context, discussion of effect modification was a key component of the elicitation
interview.

Discussions were directed at understanding experts’ views on the influence, if any, of effect modification in the specific studies upon which they planned to base their3-16 

### quantitative judgments concerning the form, magnitude, and uncertainty in the C-R function for mortality related to ambient PM 2.5 

### exposure. Experts began by listing what they believed to be potential effect modifier s in the relevant studies of long-term and/or short-term exposures. Examples from the epidemiologic literature were provided as a starting point for discussion. Experts then discussed the theoretical rationale (e.g., biological or toxicological mechanism) or empirical evidence (e.g., clinical, epidemiological, animal, or exposure studies) for the impact of each effect modifier on
the PM 2.5 

### -mortality effect. Experts discussed their views on the influence each effect modifier may have had on the relative risks reported in the relevant epidemiologic studies, and whether these reported risks were likely to be underestimates or overestimates of the average effect for the full U.S. adult population. Experts discussed the direction and magnitude of the possible under- or overestimation. We again asked Experts to try to classify the magnitude of the under- and overestimation on a three-level scale: 1 for minimal; 2 for medium; 3 for major. The quantitative interpretation of this scale was left to the Expert to define.

Experts mentioned a broad array of factors that could serve as effect modifiers of longand/or short-term mortality effects of PM 2.5 

### , including educational attainment and related SES factors, co-pollutants such as SO 2 

### , housing characteristics like air conditioner use, race, smoking, and pre-existing conditions including diabetes, obesity, systemic inflammation, and genetic predisposition. Of these, educational attainment, co-pollutants, and race were each discussed by three or more experts.

Several experts noted that educational attain ment was reported by Krewski et al. (2000a & b) to be an effect modifier in both the ACS and Six Cities studies, with mortality
effects of PM 2.5 

### greatest in the sub-populations with less than a high school education and lowest or non-existent in the sub-population with more than a high school education.Because of the recruitment strategy employed in the ACS study, that population underrepresented persons with less than high school education (11 percent) compared to statistics for persons 25 years or older in the nation as a whole (30 percent in 1980). In the Six Cities study population, 28 percent ha d less than a high school education, similar to the national average in 1980. These statistics suggested to many experts that the effect estimates reported by the ACS study were likely to underestimate effects for the adult U.S. population. Several experts noted that education per-se is unlikely to be responsible for the observed effect modification, but rather that education may represent a
constellation of factors includi ng housing, income, access to medical care, exercise, diet, underlying health status. Effect modification by educational attainment was discussed at some length during the pre-elicitation work shop, where Dr. Pope presented numerical results from a then-unpublished analysis th at re-calculated ACS RR’s based on reweighting on educational attainment preval ence reported in the Six Cities study. These calculations resulted in a 30-50 percent increas e in the RR compared to that originally reported in the ACS study. Several experts refe rred to this analysis during the elicitation. Six experts (A, C, E, G, I, J) chose to take account of this in deriving their median effect estimate in the quantification, either by inflating the reported ACS effects by 30-50 percent, or by placing weight on the Six Cities study, which, unlike the ACS study, did not under-represent persons with less than a high school education as compared to the3-17 

### U.S. adult population. Dr. Pope’s analys is has been published recently as part of a critical review of PM health effects (Pope and Dockery, 2006).

Among the other potential effect modifiers discussed, three experts (C, E, J) discussedrace, mainly on a theoretical basis. Both the ACS and Six Cities studies underrepresented non-whites in their populations, but experts noted that there are few empirical data demonstrating effect modification by race to -date. Some noted that it’s difficult to separate race from various measures of SES. In the end, none made any quantitative adjustments for race.

Co-pollutants were also raised as theoreti cal effect modifiers, with little empirical evidence, by three experts (F, H, K). Higher SO 2 

### levels in the northeastern U.S. was noted as a possible explanation for high reported PM 2.5 

### effects in the Six Cities study. Expert F referred to this issue in deriving his quantitative judgments. Housing factors like air conditioner use were mentioned by two experts (E, K), mainly from the perspective of modifying indoor exposure to PM of outdoor origin.

EXHIBIT 3-4: SUMMARY OF EXPERTS’ VIEWS ON EFFECT MODIFIERS

EXPERT RESPONSES

EFFECT MODIFIER

A B C D E F G H I J K L

Education ⊗ ⊗ ⊗ x ⊗ x ⊗ ⊗ x x Co-pollutant ⊗ 1 x x Housing x x

Race x x x Smoking x Underlying Susceptibilities (e.g., diabetes, obesity, systemic inflammation, COPD) x

Genetic Polymorphisms x Air Conditioning x

Key: X = Discussed

⊗ = Discussed and adjusted me dian to account for factor

1 Specifically for SO 2

3.1.6 EXPOSURE ISSUES 

### Exposure assessment is both an essential and pa rticularly challenging component of air pollution epidemiology. The challenge for e xposure assessment is to measure healthrelevant contrasts in exposures with as much accuracy and precision as possible. Exposure contrasts in currently-available epidemiology studies of particulate matter are
based on either concentration variations o ccurring over time within a single location or variations in average concentrations acro ss locations. In either case, population-based3-18 

### studies typically rely on one or more central-site ambient monitors in a city to estimate air pollution exposure contrasts. This raises questions about the relationship between temporal/spatial concentration contrasts measured at the central site and temporal/spatial

exposure contrasts experienced by the population at risk. Uncertainties in this relationship may arise due to variations in ambient concentrations over space within a city, incomplete penetration of ambien t pollution into homes and workplaces, and patterns of population activity . Indoor sources may also contribute significantly to
individual PM 2.5 

### exposures. In addition to uncertain ties related to central site vs. individual exposures, there may be uncerta inties regarding the timing of relevant exposures. In a cohort study of mortality, what is the relevant period of exposure that leads to increased mortality risk? Furtherm ore, what uncertainties are introduced if cohort members migrate to or from a location with different long-term PM 2.5 

### concentrations? Finally, there are uncertain ties in both whether there are differences in
toxicities of different PM 2.5 

### component species (e.g., sulfates, nitrates, elemental and organic carbon, metals) and the extent to whic h such potential differential toxicity may be responsible for differences in the relations hip between health and PM exposure observed
in different studies. Such differential toxi city would have implications for comparing
studies conducted in different cities as well as interpreting analyses that use city-to-city
differences in fine PM mass to ch aracterize variations in exposure.

The elicitation interview probed experts' views on the influence, if any, of these exposure issues on their judgments concerning the form, magnitude and uncertainty in the C-R functions for mortality related to ambient PM 2.5 

### exposure, with particular reference to the key studies upon which they planned to base th eir quantitative judgments. Experts began by listing and defining what they believed to be the most influential exposure issues, and then discussed the theoretical rationale (e.g., biological or toxicological mechanism) or empirical (e.g., clinical, epidemiological, animal, or exposure studies) evidence for the impact of each potential exposure issue on the PM 2.5 

### -mortality effect. Experts were then asked to characterize the direction and magnitude of any under- or overestimate in reported mortality relative risks (RRs) from specific studies that may have resulted from the exposure issue. The magnitude of unde r- or overestimation of RRs was reported on a three level scale, with one indicating mi nimal, two indicating medium, and three indicating major effect. The quantitative interpre tation of this scale was left to the expert
to define.

Uncertainties in population exposures assessed us ing central-site monitoring was raised by all experts as an important issue, and in many cases as a major issue (level 3), and nine experts took this issue into account when deriving their median effect estimate of the mortality effects of a 1 μ g/m 3 

### change in PM 2.5 

### . Most considered this to be a more important uncertainty for the ACS main study, where central sites were used to represent exposure over entire metropolitan areas, and many thought that this issue caused
underestimation of the effects of PM 2.5 

### on mortality. The reason citied for this underestimation was the well-known effect of exposure measurement error (“misclassification”) in biasing epidemiological effect estimates towards the null. Experts noted that the higher RRs reported in Six Cities might relate in part to finer spatial scale of monitoring there vs. ACS. Also the Jerrett LA ACS analysis, which3-19 

### featured a finer spatial resolution of exposure assessment than the original ACS study and reported results more comparable to Six Cities, was often cited in support of this idea. However, some experts thought that the larger effects reported by Jerrett might also be due to a different mix of particle components in LA vs. the U.S. as a whole, perhaps related to motor vehicle pollution, or might reflect some bias due to spatial confounding by SES. Two experts also referred to the Willis et al. (2003) reanalysis of sulfate data from the ACS cohort, using only monitors located in the county of residence of cohort members. The RR based on the county-level s ubset was approximately double that in the full cohort where the unit of observation had been the metropolitan area. Also noted was the theoretical analysis of Mallick et al. (2002). One expert pointed out that there is likely to be an optimal spatial resolution that is fine enough to avoid spatial error but large enough to capture the spatial activity range of community residents. Several experts noted that a recent unpublished analysis by Jerrett and colleagues for NYC gave contrasting results to those observed in the LA analysis. These data were also discussed at some length at the Post-elicitation Workshop. However, given the lack of detailed information on this unpublished work, the NYC results were not relied upon by experts for the purposes of the elicitation.

Nine of the experts discussed whether the relevant time course of historical exposures was well-captured in the published cohort studi es. A mismatch could introduce additional random exposure misclassification or could theoretically result in upward bias in effect estimates. Theoretically, as noted by Expert J, “if the time scale [for long-term effects] is
on the order of decades and we use more recent PM 2.5 

### measures where the rank ordering is the same, but the [concentrations] are lower, then we are overestimating the effects.” In other words, a more recent, more-nar row range of measured exposure differences across cities would serve as a surrogate for a larger, health-relevant, but unmeasured range of exposure differences that existed in the past. After considering the evidence and theory, however, experts decided not to take this into account in their quantification, noting that the bias would be significant only if latency periods were very long (i.e., multiple decades), for which there is little epidemiologic evidence.

A few experts thought that migration of cohort members out of the community of initial residence could lead to additional uncertainties, leading to underestimate of RR. This effect was thought to be greater for the ACS study, which lacked follow-up data on residence after initial enrollment. Differential migration by SES might lead to different degrees of misclassification of exposure, and might explain some of the effect modification noted in ACS study. No expert s took this issue into account in quantifying
their median effect estimate.

Expert J mentioned the difficulty of direc tly addressing the quantitative question that was the focus of the elicitation (i.e., what is the mortality effect of a change in PM 2.5 

### concentration, holding all co-pollutants constant). Since PM 2.5 

### contrasts (spatial as well as temporal) occur in concert with contrasts in co-pollutants (due to overlapping sources), it is difficult to quantify the marginal impact of PM 2.5 

### . If one wants the marginal effect of

PM 2.5 

### alone, which is the stated focus of the e licitation, the PM effect estimates in key epidemiology studies could be moderate over-estim ates of the true PM effect for the U.S.3-20 

### population since they incorporate some unknown quantity of co-pollutant effects. This issue was also discussed by other experts, as well as the related issue of differential toxicity for different particle components, but in the end, experts concluded that there was insufficient empirical data to guide any quan tification of these influences. It should be noted that this issue is really one of confounding by co-pollutants. Other issues raised but left un-quantified by experts included the use of estimated rather than measured PM 2.5 

### data in the Six Cities follow-up analysis (Laden et al., 2006), and the influence of air conditioning use on penetration of ambient PM 2.5 

### indoors. Experts thought that these issues might introduce some biases, but we re not able to quantify them.

EXHIBIT 3-5 SUMMARY OF EXPERTS’ VIEWS ON EXPOSURE ISSUES

EXPERT RESPONSES

EFFECT MODIFIER

A B C D E F G H I J K L

Spatial resolution ⊗ ⊗ ⊗ x ⊗ ⊗ x ⊗ ⊗ ⊗ x ⊗ Migration x ⊗ Co-pollutants ⊗ Temporal changes in
exposure x x x x x x x x x

Estimated PM 2.5 x x x

PM 2.5 composition x x x x Air conditioning use x

Key: X = Discussed

⊗ = Discussed and adjusted me dian to account for factor

3.1.7 LIKELIHOOD OF A CAUSAL RELATI ONSHIP BETWEEN LONG- AND SHORT-TERM

PM 2.5 EXPOSURES AND MORTALITY 

### One of the more influential discussions held with each expert involved his assessment of the likelihood of a causal relationship between annual average exposures to PM 2.5 

### and total annual all-cause mortality (from both s hort-term and long-term exposures). Experts were asked to estimate this likelihood after completing most of the other conditioning questions, which we re designed to evaluate th e scientific evidence on causes
of and mechanisms for PM 2.5 

### -related mortality, epidemiological evidence, and factors relating to the quality of the key epidemiological studies (e.g., confounding, effect modification, exposure issues). The question began by having each expert talk generally about the types and/or strength of scientific evidence he would like to have in order to believe that an exposure/response relations hip is actually causal. Experts were not required to follow a particular rubric or set of causal criteria. In addition to providing a
framework from which to view individua l experts’ assessments of causality, this component of the question also provided some insight into some of the differences between experts’ approaches to weighing evidence.3-21 

### Each expert was then asked to estimat e quantitatively the probability that the relationships between short- term exposures and mortality, between long-term exposures and mortality, or both were causal. They were asked for a "best estimate" as well as for a range of probabilities to give a sense of the level of uncertainty in their estimate.

Although this question had been discussed bot h at the EPA expert elicitation symposium and at the pre-elicitation workshop, the estimation of the causal likelihood and its use in the characterization of uncertainty was difficu lt and controversial in practice. Questions about the clarity of the question on causal likelihood and its use in the development of thefinal quantitative uncertainty distributions were a major focus of discussions at the postelicitation workshop.

Some experts questioned whether the causal likelihood should be linked to different concentrations, that the question should be "causal at what concentration?" A related question was whether they might think differe ntly about the likelihood of a change in mortality for a small incremental ch ange in concentration (e.g., 1 μg/m 3 

### ) versus a large incremental change (e.g., 25 μg/m 3 

### ). In our view, these types of questions could be addressed in discussions of thresholds or in the mathematical expression of the concentration response relationship. We anticipated an answer to, "What is the likelihood that there is a causal relationship between reductions in annual average exposures to

PM 2.5 

### (including reductions in both short-and/ or long-term exposures) and changes in mortality at PM 2.5 

### levels currently experienced in the U.S. (e.g., annual averages of 4-30 μg/m 3 

### )?"

Asking the experts to develop probabilities refl ecting some joint judgment about the role of short-term and long-term exposures also a dded to difficulties in answering the question and interpreting responses. As indicated in the summaries in the exhibit, though many experts thought long-term exposures were dom inant determinants of mortality, several thought repeated short-term exposures were important. Separating their roles was
difficult.

Three experts B, D, and K discussed an additi onal interpretation of the question. Expert
D made a distinction between the likelihood th at there is a causal relationship between

PM 2.5 

### (on a mass concentration basis) and the lik elihood that particles are "causing all the effects ascribed to them." He thought the former was highly likely but that the latter was very low, but appropriately based his response on the first question. Expert B echoed this
view: " 

### It's not to say that PM 2.5 

### is the only air pollutant that may have a causative role in mortality, but certainly if you ask does PM 2.5 

### contribute to mortality, my answer is almost certainly yes." Expert K appeared to base his likelihood of a causal relationship at least in part on his assessment of whether particles per se were responsible for the mortality
effects observed in studies.

Next, experts sometimes found it difficult to make the conceptual distinction between a
statistical confidence interval and the uncertain ty assessment for this project. As noted by Expert D, "on statistical grounds, 95% confid ence is sufficient grounds for 'conviction" (i.e., for rejecting the null hypothesis of no re lationship). However, the development of a subjective probability distribution allows fo r judgments to be made over the whole3-22 

### interval from 0 to 1 on the basis of an aggreg ate assessment of the strength of evidence. Some experts found it to be almost a conceptual conflict to say that they were highly certain of a causal relationship (often 95%) but then to have to reflect that they were implicitly putting a 5% probability of no causal re lationship. In fact, we are talking about two entities -- statistical confidence intervals on a particular finding and a subjective
confidence interval or "credible" interval as it is sometime referred to in Bayesian
statistics.

Basis for Causal Likelihood 

### Although not specifically mentioned by name, many of the experts drew on one or more of the basic Bradford Hill causal criteria (Hill et al., 1965). The following are conditions that the experts frequently mentioned as he lpful for assessing whether an association is
causal:

• 

### Consistency (across epidemiological study designs);

• 

### Biological Plausibility;

• 

### Coherence (see associations across range of health outcomes);

• 

### Temporal Relationship (between exposure and outcome);

• 

### Strength (of the association);

• 

### Analogy (to other exposures);

• 

### Specificity (of response); and

• 

### Statistically significant findings (robust to concerns about confounding, effect

modification).

Not all experts focused on the same attributes, nor relied on the same studies when citing support for particular attributes.

Short term vs. Long term Exposures 

### Prior to developing their quantitative assessment of the likelihood of a causal relationship
between total annual all-cause morta lity and annual average ambient PM 2.5 

### exposures, experts were asked whether they would like to make a distinction between the likelihood of a causal relationship for short-term and long-term exposures in preparation for their overall assessment. Five experts (B, F, G, I, J, K) initially indicated they would, though only one gave separate causal likelihood for short-term exposures (Expert F placed 90% confidence in the causal relationship between short term exposures and mortality versus 100% for long term). Ultimately, most of the experts relied on assessment of the strength of the literature relating to the effects of long-term exposures. In their discussions of the conceptual framework for mortality effects, mo st expert indicated that long-term effects predominate (See Section 3.1.2)

Likelihood of a Causal Relationship 

### Exhibit 3-6 summarizes the probabilities given by each of the experts to describe their
assessment of the likelihood of a causal relationship between exposures to PM 2.5 

### and total3-23 

### mortality. The first column shows each expert's "best estimate", the estimate on which they would place their greatest probability weight, while the second column gives the range of values they thought were plausi ble. The final column provides qualitative summary statements, if given, to describe the expert's qualitative sense of the strength of the relationship. As part of this question, however, we had them consider the range of possible values as they evaluated alternative wa ys of weighing or interpreting the existing data. For example, they were asked to think about what factors or constellation of factors made it least plausible that there was a causal relationship and to contrast that with what factors made the relationship more plausible. This approach was sometimes helpful as an intermediate step.

As indicated in Exhibit 3-6, 10 of the 12 of the experts thought there was strong scientific support for a causal relationship with best estimates ranging from 90 to 100%. These experts most often cited the following kinds of arguments: the consistency in the pattern of epidemiological results across multiple st udy designs; the robustness of the Six Cities and ACS epidemiological results under the scru tiny of Health Effects Institute (HEI) reanalyses (Krewski et al., 2000a & b); the importance of intervention studies as "experiments" establishing the temporal relationship between decline of exposures and in mortality; the support from toxicological studies in animals (in particular, but not restricted to, the recent Sun et al. studies (2005)) and from in vitro studies for a plausible biological explanation. Only one expr essed absolute certainty that there was no alternative explanation. Most expressed an unwillingness to eliminate all doubt, some indicating that "nothing is for certain."

The two experts who expressed stronger reservations about the plausibility of the causal relationships focused on similar concerns with the available scientific data although they ultimately selected different likelihoods. In our discussions, both essentially argued that that statistical associations from epidemiological studies, in particular the cohort studies, were not sufficient to establish a causal link. Expert K in addition raised at least two concerns that undermined his confidence in epidemiological studies; he thought that there was still a potential for confounding by smoking in cohort studies relying on questionnaire data and he thought that the absence of personal exposure data in both
cohort and more recent defibrillator studies ma de it difficult to link mortality definitively to particular exposures. Both thought the Sun et al. (2005) study was an important new contribution but questioned whether the mouse model or exposures were truly relevant to humans and chronic ambient exposures to PM 2.5


### .E X H I B I T 3

- 6: LI K E LI H O O D OF A CA US AL R E LA TI ONS H I P B E T W E E N P M 2 . 5 AND

ALL-CAU S E MOR T AL I T Y

EXPERT "BEST

EST I M A TE" RANG E QUALIT AT IVE

STA T EM EN TS

F 1 0 0 1 0 0 -- J 99 80-99 "[P]ret t y likely ... t h e literature is p r etty compelling ... that there is a causal relationship b o th f o r s h o r t- term exposure and long-term."

C 9 9 9 9 "[V]ery, very li kely"

L 99 90-10 0 “The ev idence in t h e sho r t

- t e rm ... is so st ro ng that ... [i t] is actually enough to say that it is a cause of death ... I think it's extremely hard to argue that this is only an acute effect story and nothing else."

E 99 80-99 "I'm pret t y co nv inced, ... but nothing is for certain"

B 9 8 9 0

- 9 9 "[E]xtremely li kely"

A 9 5 70-95. 5 -- I 9 5 80-10 0 -- D 95 90-10 0 "[E]xt remely high likelihood for both"

H 9 0 8 0

- 9 5 -- G 70 60-80 The state of th e science "strongly suggests a causal relation ship."

K 3 5 5

- 5 0 "[I] don't have at this
point great confidence

t h at ... bringing do wn t h e
levels curren t l y experienced in the U.S.

are going to change

mo r t alit y."

-- Expert did not gi ve a summary statement ab out likelihood of a causal r e lationship.

3-243-25 3.1.8 THRESHOLDS 

### The protocol asked experts for their judgments regarding whether a threshold exists in the

PM 2.5 

### mortality C-R function. The protocol focused on assessing expert judgments regarding theory and evidential support for a population threshold (i.e., the concentration below which no member of the study popula tion would experience an increased risk of death). 32 

### If an expert wished to incorporate a threshold in his characterization of the concentration-response relationship, the team then asked the expert to specify the
threshold PM 2.5 

### concentration probabilistically, incorporating his uncertainty about the
true threshold level.

From a theoretical and conceptual standpoi nt, all experts generally believed that
individuals exhibit thresholds for PM-related mortality. However, 11 of them discounted the idea of a population threshold in the C-R function on a theoretical and/or empirical basis. Seven of these experts noted that theo retically one would be unlikely to observe a population threshold due to the variation in susceptibility at any given time in the study population resulting from combinations of ge netic, environmental, and socioeconomic
factors. 33 

### All 11 thought that there was insufficient empirical support for a population threshold in the C-R function. In addition, two experts (E and L) cited analyses of the ACS cohort data in Pope et al. (2002) and a nother (J) cited Krewski et al. (2000a & b) as supportive of a linear relationship in the study range.

Seven of the experts favored epidemiologi cal studies as ideally the best means of addressing the population threshold issue, because they are best able to evaluate the full range of susceptible individuals at environmen tally relevant exposure levels. However, those who favored epidemiologic studies ge nerally acknowledged that definitive studies addressing thresholds would be difficult or impossible to conduct, because they would need to include a very large and diverse population with wide variation in exposure and a long follow-up period. Furthermore, two experts (B and I) cited studies documenting difficulties in detecting a threshold using epidemiological studies (Cakmak et al. 1999, and Brauer et al., 2002, respectively). The experts generally thought that clinical and toxicological studies are best suited for researching mechanisms and for addressing thresholds in very narrowly defined groups . One expert, B, thought that a better understanding of the detailed biological mech anism is critical to addressing the question
of a threshold.

One expert, K, believed it was possible to make a conceptual argument for a population threshold. He drew an analogy with smoking, indicating that among heavy smokers,

32 As part of this section, the elicitatio n team reviewed the key assumptions about th e study population for this elicitation.

We assume the following about the U.S. po pulation: 1) the population is 25 years of age or older, 2) the distribution of susceptible individuals across po pulation reflects current patte rns of susceptibility, and 3) this pattern will remain the same. The impact of projected changes in the age distribution on mortality, however, is incorporated directly into EPA’s

benefits model.

33 Ten of the experts addressed this issue in an integrated fashion wi th respect to impacts of short-term and long-term

exposures, or focused on long-term impact s, which they thought were the dominant contributor to changes in mortality. Two experts (B and G) suggested that a threshold may be more plausible conceptua lly for short-term mortality impacts of

PM 2.5 than for long-term impacts, base d on the postulated mechanisms.3-26 

### only a proportion of them gets lung cancer or demonstrates an accelerated decline in lung function. He thought that the idea that there is no level that is biologically safe is fundamentally at odds with toxicological theory. He did not think that a population threshold was detectable in the currently av ailable epidemiologic studies. He indicated that some of the cohort studies showed gr eater uncertainty in the shape of the C-R function at lower levels, which coul d be indicative of a threshold.

Expert K chose to incorporate a threshold into his C-R function. He indicated that he was 50 percent sure that a threshold existed. If th ere were a threshold, he thought that there was an 80 percent chance that it would be less than or equal to 5 μ g/m 3 

### , and a 20 percent
chance that it would fall between 5 and 10 μ g/m 3



### .

3.2 RESPONSES TO THE QUANTITATIVE QUESTION 

### In the final part of the interview, experts were asked to address the following question:

What is your estimate of the true percent change in annual, allcause mortality in the adult U.S. population resulting from a permanent 1 μg/m 3 

### reduction in annual average ambient PM 2.5 

### across the U.S.? In formulating your answer, please consider mortality effects of both reductions in long-term and short-term exposures. To characterize your uncertainty in the C-R relationship, please provide the 5 th 

### , 25 th 

### , 50 th 

### , 75 th 

### , and 95 th 

### percentiles of your estimate.

In addressing this question, the experts fi rst specified a functional form for the PM 2.5 

### mortality C-R function and then developed an uncertainty distribution for the slope of that function (the mortality impact per unit change in annual average PM 2.5 

### ), taking into account the evidence and judgments discussed during the conditioning questions. We present the quantitative results provided by each expert below.

3.2.1 SHAPE OF THE CONCENTR ATION-RESPONSE FUNCTION 

### The experts were asked to provide their judgments about the true shape of the C-R function relating mortality with changes in annual average PM 2.5 

### concentrations across the specified study range of 4 to 30 μ g/m 3 

### . Experts specified the functional form and indicated if the function included a threshold (i.e., a PM 2.5 

### level below which changes in annual average concentration would have no impact on mortality). The experts’ responses are summarized in Exhibit 3-7. Ei ght of the 12 experts specified a log-linear C-R function with no threshold; three specified a non-linear function comprised of two log-linear segments with no threshold. One expert specified a non-linear function comprised of two log-linear segments that included a threshold. 34

34 As noted in the section on thresholds (3.1.8), the experts all discussed a theoretical model in which the C-R relationship

should exhibit a threshold at the individual level. That is, an individual mi ght have a threshold for a particular cause of
death. However, individual th resholds may vary both across in dividuals and by cause of death for a given individual. Most experts agreed that variation in thresholds among individuals or population subgroups limited the ability of epidemiological studies to detect a population-l evel threshold. Though one expert’s quantita tive estimates employed a threshold, none of the experts could cite a study that provid ed strong evidence of a threshold for PM 2.5 mortality effects.3-27 EXHIBIT 3-7 EXPERT JUDGEMENTS CONCERNING FORM OF THE C-R FUNCTION

EXPERT RESPONSES

FORM

A B C D E F G H I J K L

Log-linear,
No threshold x x x x x x x x

Piecewise Log-linear,
No threshold x x x

Piecewise Log-linear,
Threshold x



### All of the eight experts who specified a single log-linear function for the entire range
believed that the available cohort data were c onsistent with a log-linear interpretation and that existing data did not support the iden tification of any population thresholds.

The remaining experts specified non-linear functions using two log-linear segments across the study range of PM 2.5 

### . These experts chose a nonlinear function to characterize what they believed to be the increased un certainty in mortality effects at lower concentrations seen in the major epidemiological studies. The break point defining “lower concentrations” differed a bit among th ese experts, as shown in Exhibit 3-8. One expert (K) thought he had insufficient data to support anything other than splitting the study range in half. The other three experts cited the lower ranges of observed data in the cohort studies such as ACS in determining th eir break point, but acknowledged that data are limited to specify precisely the locati on where the slope changes and/or where uncertainty in the mortality impact increases.

EXHIBIT 3-8: BREAK-POINTS DELINEATING SEGMENTS OF THE PM 2.5 -MORTALITY CR FUNCTION

EXPERTS BREAK-POINT CONCENTRATION

F 7 μ g/m 3
B, L 10 μ g/m 3

K 16 μ g/m 3



### The detailed results for these experts will be discussed below in the section on the experts’ distributions. In general, the cha nges expressed at lower PM concentrations by these experts were modest and indicate a sh allower slope for the lower range and/or slightly expanded uncertainty.3-28 3.2.2 EXPERT CHARACTERIZATION OF UNCERTAINTY 

### Following the discussions on the shape of th e concentration-response (C-R) function, experts were asked to begin the process of de veloping the probability distributions that would be used to characterize uncertainty in the PM-mortality relationship. Because all of the experts described the overall C-R f unction as linear or log-linear over all or portions of the study’s concentration range, this step essentially characterized the distribution of possible values for the PM 2.5 

### mortality coefficient (i.e., the percent change in mortality per unit change in PM 2.5 

### concentration). The experts were specifically asked
to estimate the minimum and maximum and the 5 th 

### , 25 th 

### , 50 th 

### , 75 th 

### , and 95 th 

### quantiles of
their distributions.

The elicitation protocol and accompanying e licitation tools were designed to allow experts some flexibility in developing their di stributions. They were given options in 1) the incorporation of their judgments about the likelihood of a causal relationship, and 2) the elicitation of specific quantiles. Expert s first decided whether they wanted to integrate their uncertainty about the likelihood of a causal relationship (as elicited in the conditioning questions, described in Chapter 2) directly in their uncertainty distributions, or whether they preferred to characterize uncer tainty in the mortality effect conditional on the assumption that the relationship was causal. 35 

### The uncertainty distributions for the latter group of experts would then theoretically reflect sources of uncertainty other than those affecting the experts’ judgments about wh ether or not the causal relationship exists. Five experts (A, C, F, H, and J) chose to incorporate their causal likelihood estimates directly into their distributions. The remain ing experts (B, D, E, G, I and K) preferred that these two judgments be presented separate ly and combined only if deemed by EPA for the purposes of a benefits analysis.

The Elicitation Team then worked with each expert to elicit indi vidual quantiles of his distribution. The basic approach outlined in the protocol was direct elicitation of individual quantiles, beginning with the minimum and maximum, the 5 th 

### and 95 th 

### percentiles, and followed by the elicitation of the 25 th 

### and 75 th 

### percentiles, and finally the median. However, in response to feedback received by the Project Team during the EPA Symposium and the Pre-elicitation Workshop, experts who preferred to work with parametric distributions were given the oppor tunity to elicit a more limited number of quantiles and to fit a parametric distribution using the Crystal BallTM software. Experts B, F, H and L developed distributions using the first approach. Seven of the remaining experts decided it was intuitively more strai ghtforward to develop rationales for two or more quantiles, or to estimate a mean and standard error, adjusted for additional uncertainty (Expert E), and to fit a distributio n with Crystal Ball. Expert I preferred to characterize uncertainty by choosing three studies whose effect estimates represented
different conditions about which he was un certain, assigning subjective weights to each study, and combining their results via Monte Carlo simulation. Whatever their approach, individual experts were asked to explain the rationale for every quantile elicited and to

35 In the latter case, the experts were in formed that their conditional C-R coeffi cient uncertainty distribution and their

probabilistic distribution for causality would ultimately be in tegrated when applied by EPA in future benefits analyses.3-29 

### conduct “reality checks” of the values they sel ected. These reality checks included using the spreadsheet tools described in Chapter 2 to: 1) allow each expert to visualize his distribution and compare and contrast it with distributions from the key published studies he cited as relevant; and 2) generate exam ple national-scale back-of-the-envelope benefit calculations based on quantiles of the expert s C-R coefficient distribution and compare them with data on other major causes of death in the U.S.

Exhibit 3-9 displays the individual values for each of the percentiles requested of the
experts. 36 

### The first column indicates the distributional form of the distributions developed by each expert. The term “Custom” indicates that all of the percentiles were individually elicited. Of the eight who select ed parametric distributions, six fit normal (i.e., Gaussian) distributions, one chose a triangular distribution, and one a Weibull. Individually elicited values for each expert ar e indicated in boldface and italic type; all other values were generated by the elicita tion team using Crystal BallTM statistical software to replicate the expert-specified distributions. Note that the experts who chose normal distributions to characterize their uncertainty distributions chose different percentiles with which to fit their distributi ons. For the four experts who used piecewise log-linear functions to characterize the C-R relationship over the 4-30 μ g/m 3 

### range of

PM 2.5 

### , the table displays both of their distributions.

36 Note that all results presente d in this report in Exhibits 3-9 through 3-13 reflect the expert s’ final judgments after making

changes after the Post-elicitation Workshop, if applicable.3-30EXHIBIT 3-9: SUMMARY OF EXPERT SUBJECTIVE UNCERTAINTY DISTRIBUTIONS FOR C-R COEFFICIENTSDISTRIBUTIONTYPEMINIMUM5THPERCENTILE25THPERCENTILE50THPERCENTILE(MEDIAN)75THPERCENTILE95THPERCENTILEMAXIMUMINCLUDESLIKELIHOOD OFCAUSALITY?3CAUSALITYLIKELIHOODTHRESHOLDSPECIFIED?ANormal00.291.11.62.12.94.0Y0.95NB (4-10μg/m3)Custom0.010.100.201.22.12.62.8N0.98NB (>10-30μg/m3)Custom0.100.200.501.22.12.62.8N0.98NCNormal00.400.901.21.52.0N/AY0.99ND1Triangular0.100.350.660.901.11.41.6N0.95NENormal01.01.62.02.43.0N/AN0.99NF (4-7μg/m3)Custom0.370.580.730.931.11.41.7Y1.0NF (>7-30μg/m3)Custom0.290.770.961.11.41.61.8Y1.0NGNormalN/A0.700.881.01.11.31.5N0.70NHCustom000.400.701.32.03.0Y0.90NI2Normal0.200.380.901.31.62.12.3N0.95NJWeibull00.150.530.901.32.03.0Y0.99NK (4-16μg/m3)NormalN/A0.100.280.400.520.700.80N0.35YK (>16-30μg/m3)NormalN/A0.100.450.70.951.31.5N0.35YL (4-10μg/m3)Custom00.200.571.01.41.62.7N0.75NL (>10-30μg/m3)Custom0.020.200.571.01.41.62.7N0.99NNote: Numbers in bold indicate a percentile value directly provided by the expert. All other numbers were generated by the elicitation team using Crystal Ball for the expert’sspecified parametric distribution based on percentiles and other information provided by the expert.Expert K chose to incorporate a threshold into his C-R function. He indicated that he was 50% sure that a threshold existed.If there were a threshold, he thought that therewas an 80% chance that it falls between 0 and 5 μg/m3, and a 20% chance that it falls between >5 and 10 μg/m3. The elicitation team tookthis information and created aprobabilistic distribution in Crystal Ball with 50% of the weight at zero, 40% of the weight between >0 and 5 μg/m3, and 10% of the weight between >5 and 10 μg/m3.1Expert D also provided a most likely value of 0.95 that was used to generate atriangular distribution in Crystal Ball.2Expert I provided three beta coefficients and standard deviations from three epidemiologic studies and had the elicitation team combine these into a single distribution usingCrystal Ball (placing equal weights on each study). The resultingdata was fit to a normal distribution, as specified by the expert.3Values in this column represent theexpert's view on the likelihood thata causal relationship exists between PM2.5and mortality, as described in Section 3.1.7.3-31 

### Because of the varied approaches taken by the experts to characterizing their views about uncertainty in the PM 2.5 

### mortality relationship, it is diffic ult to convey and to compare all their results directly in one exhibit. Direct comparison can only be done when their distributions are applied to the same scenario in a benefits analysis. We consequently portray the results in a series of graphs, grouping experts on the basis of whether they incorporated causal likelihoods into their distributions and whether they characterized the C-R function as linear or non-linear over the study range of PM 2.5


### . Exhibits 3-10 and 3-11 provide comparisons of the experts’ distributions in the form of boxplots. The “whiskers” delimit the lower 5 th 

### and upper 95 th 

### percentiles. The box represents the interquartile range. The median is represented by a closed circle and the mean, generated by Crystal BallTM, is repr esented by the open circle. The two boxplots on the far right give the analogous percentiles for the mortality coefficients from the two epidemiological studies often used in be nefits analysis to characterize the PM 2.5 

### -mortality relationship, Pope et al. (2002), and Dockery et al. (1993). The vertical axis is in units of percent decrease in mortality per unit decrease in PM 2.5 

### and the horizontal axis displays the individual experts.

Exhibit 3-10 compares the distributions for those experts who preferred to give conditional uncertainty distributions and k eep their probabilistic judgment about the likelihood of a causal or non-causal relationship separate (Group 1). Exhibit 3-11 displays the distributions for those experts who chose to incorporate the likelihood of a causal relationship directly into their distribu tions (Group 2). Both Exhibits 3-10 and 3- 11 array the experts in order of decreasing elicited likelihood of a causal relationship, with the experts on the left being most convinced the relationship is causal. The likelihood of a causal relationship elicited from each expert is shown below his letter on the horizontal axis. The two sets of graphs for each group provide comparisons of the experts at two annual average PM 2.5 

### concentrations, 18 μ g/m 3 

### and 7 μ g/m 3 

### , in order to observe the implications of particular e xperts’ assumptions about non-linearities in the CR function and about differing degrees of uncertainty in the slope of the function across specific ranges of PM.

Exhibit 3-11 illustrates how experts in Group 2 incorporated assumptions about the likelihood of a causal relationship into thei r uncertainty distributions. For example, Expert H, who was 90 percent sure that the relationship between PM 2.5 

### and mortality was causal, has a 10 percent chance that there is no causal relationship (and thus that the C-R coefficient is zero). The 5 th 

### percentile of Expert H’s boxplot therefore extends to zero.3-32 Note: Box plots represent distributions as provided by the experts to the elicitation team. When asked, experts in this group preferred to give conditional distributions and k eep their probabilistic judgment about the likelihood of a

causal or non-causal relationship separate.

Expert K specified a threshold (not shown).

Expert L provided two different likelihoods of causality for his C-R coefficient dist ributions at 7 and 18 μg/m 3

, although his distribution appears in the same location in both graphs.

*Experts' C-R coefficient distribution changes between 7 and 18 μg/m 3

. 0 0.5

1 1.5

2 2.5

3

E
99% L*
99% B*
98% D
95% I
95% G
70% K*
35% Pope et al., 2002 Dockery et al.,
1993% Decrease in Mortality per 1 ug/m3 PM2.5 Expert C-R Coefficient Distributions at a Baseline Annual Average PM 2.5 of 18 ug/m 3

Key: Closed circle = median; Open circle = mean; Box = interquartile range; Solid line = 90% credible interval

Expert Causality Likelihood

0 0.5

1 1.5

2 2.5

3

E
99% L*
75% B*
98% D
95% I
95% G
70% K*
35% Pope et al., 2002 Dockery et al.,
1993% Decrease in Mortality per 1 ug/m3 PM2.5 Key: Closed circle = median; Open circle = mean; Box = interquartile range; Solid line = 90% credible interval

Expert Causality Likelihood Expert C-R Coefficient Distributions at a Baseline Annual Average PM 2.5 of 7 ug/m 3 GROUP 1: EXPERT UNCERTAINTY DISTRIBUTIONS FOR PM 2.5 - MORTALITY C-R

COEFFICIENTS; CONDITIONAL ON THE EXISTENCE OF A CAUSAL RELATIONSHIP EXHIBIT 3-10:EX HI B I T 3-1 1 : GRO U P 2: EXPER T UN CER T AINT Y DIST RIBUTIONS F O R TH E PM 2 . 5 -MOR T A L I TY

CO E F F I C I EN T I N C O R P O R A T I N G T H E EXP E R T ' S LI K E L I H O OD O F A C A U S A L

RE L A TIONS H I P

0 0. 5

1 1. 5

2 2. 5

3

F*
100% C
99% J
99% A
95% H
90% P ope et al

., 2002 Doc k ery et al

., 1993% Decrease inMortality per1 ug/m3 PM2.5 E x p e r t C

- R C o e ffi cie n t D i st rib u t i on s at a Ba se lin e A n n u a l A v er ag e P M 2. 5 o f 1 8 ug /m 3

Ke y : C l o s e d c i rc l e = medi an; O pen c i rc l e = mea n ; B o x = i n t e r quar ti l e ran ge; S o l i d l i n e = 90% c r ed i b l e i n t e r v a l

Ex p e r t Ca us a l ity Lik e l i hood

0 0. 5

1 1. 5

2 2. 5

3

F*
100% C
99% J
99% A
95% H
90% P ope et al

., 2002 Doc k ery et al

., 1993% Decrease inMortality per1 ug/m3 PM2.5 Ex pe rt C

- R Co ef fic i e n t Di str i b u ti on s at a B a s e l i n e A n nu a l A v era g e P M 2.5 of 7 ug / m 3

Ke y : C l o s e d c i rc l e = medi an; O pen c i rc l e = mea n ; B o x = i n t e r quar ti l e ran ge; S o l i d l i n e = 90% c r ed i b l e i n t e r v a l

Ex p e r t Ca us a l ity Lik e l i hood

Note: Bo x plots represent distribut i ons as provided by t h e ex p e rts to the El icitation Te am. Whe n aske d, ex perts in this g r oup preferred
to g i ve distributio n s that inco rpo r ate t h eir l i kel i ho o d that the PM 2. 5 mortal ity associatio n may b e non- causal. * E x perts’ C-R coef ficient distr i bu tion changes betw een 7 and 18 μg/m 3

.

3-333-34 

### Generally, the results for the experts in each group show substantial variation in the amount of uncertainty expressed, even among individuals expressing similar views on the strength of the causal relationship. For example, in Group 2, Expert F (100% likelihood) predicts a much narrower range of values for the PM 2.5 

### mortality coefficient than Experts A or H, whose likelihoods of a causal relati onship were 95% and 90% respectively. We see similarity in the variation in Group 1, for example Expert B has a much wider range of values than Expert L, although both have similar causality likelihoods (at 18 μ g/m 3 

### ).
We do not see dramatic differences in uncer tainty between the subjective probability distributions at high versus low concentra tions (i.e., the upper and lower panels of Exhibits 3-10 and 3-11).

Exhibit 3-12 displays the box and whiskers plots of the distributions given for each PM range by each of the four experts who favored a non-linear function. The exhibit shows that with the possible exception of Expert K, the changes in the C-R slope and uncertainty expressed by the experts between high and low PM 2.5 

### concentrations were very modest:

• 

### Only two experts, F and K, specified a lower median slope estimate for the lower

range of their functions. Expert F reduced his median estimate by 18 percent, K by 43 percent.

• 

### Expert B expanded his uncertainty by reducing his minimum, 5 th 

### and 25 th



### percentile estimates for the lower range, leaving the rest of his distribution unchanged. The mean of his distribution, which was not directly elicited, moved slightly closer to his median (1.2 percent) as the result of the changes in the lower
distribution.

• 

### Expert F specified a distribution for the lower range with slightly lower values for

all percentiles elicited. The lower distribution was slightly more skewed towards lower values, but the spread of his two distributions was similar. He thought equally confident in his distributions for both segments, but that the distribution for the lower range would be shifted downwards. 37

• 

### Expert L’s lower and upper box plots appear identical; however, there are two

differences in his distributions. First, his minimum value for the lower distribution is zero, as compared to 0.02 for the upper distribution. Secondly, Expert L expressed less confidence in a causal relationship at lower PM concentrations, specifying a 75 percent best estimate likelihood that the relationship is causal in the lower range and a 99 percent best estimate likelihood of a causal relationship in the upper range. The difference in causality could have an impact when Expert L’s

37 He thought that there might be a mech anism operating that would adversely affect health, but believed that at lower

levels of exposure that mechan ism would operate more slowly, so that some exposed people would be likely to die from

another cause first.3-35 

### judgments are applied in a benefits an alysis context where judgments about causality are factored in along with the mortality impact distributions. 38

• 

### Expert K’s function displays the most significant differences between his upper

and lower distributions in values for the C-R slope. Also unlike the other three experts in this exhibit, the spread of his lower distribution is much narrower (“more informative” in expert elicitati on terms) than his upper distribution, indicating greater confidence in his lowe r range estimate than his upper range.

As discussed in Section 3.1.8, Expert K thought that a population threshold could exist at lower PM concentrations, although he was uncertain about where it might fall. 39 

### Exhibit 3-13 shows the difference in the subjective pr obability distribution for Expert K with and without his assumptions about a threshold for an example application to a sample of

PM 2.5 

### annual average concentrations. As one might expect, the impact of incorporating a
threshold is to shift the distribution dow nward to some degree. Ultimately, the implications for estimated mortality benefits of Expert K's views on thresholds as well as of the other experts’ judgments about the non-linearities in the C-R function at lower concentrations are best explored as part of a sensitivity analysis. We report on such an analysis in the next chapter. Only with a pplication to a common scenario can all of the experts' quantitative judgments be directly compared.

38 Assuming Expert L’s causality distribution and mortality impa ct distribution are independen t, incorporating the causality

judgment as part of the evaluation of a regulatory change in PM would result in lesser benefits being estimated, on average, for individuals whose annual mean PM 2.5 concentration before the change is 10 ug/m 3 or less. The overall extent of this impact would depend on what proportion of the po pulation in the benefits an alysis experiences baseline PM 2.5 annual average concentrations of 10 ug/m 3 or less.

39 He indicated that he was 50 pe rcent sure that a threshold exist ed. If there were a threshold, he thought that there was an

80 percent chance that it would be less than or equal to 5 μ g/m 3 , and a 20 percent chance that it would fall between 5 and

10 μ g/m 3

.3-36EXHIBIT3-12:C-RCOEFFICIENTDISTRIBUTIONS FOREXPERTSWHO SPECIFIEDNON-LINEAR FUNCTIONS00.511.522.54-10 ug/m398%>10-30ug/m398%

% D ecrease i n Mo rt al i t y p e r 1 u g / m 3 PM2. 5Range ofPM2.5ConcentrationsPercentLikelihoodofCausalityExpert BDistributionsKey:Closedcircle =median; Open circle =mean;Box=interquartile range; Solid line =90% credible interval00.511.522.54-7ug/m3100%>7-30 ug/m3100%

% D ecrease i n Mo rt al i t y p e r 1 u g / m 3 PM2. 5RangeofPM2.5 ConcentrationsPercentLikelihoodofCausalityExpertFDistributionsKey:Closed circle =median; Opencircle =mean; Box =interquartile range;Solid line =90% credible intervalNote: Box plots represent distributions as provided by the experts to the elicitation team. When asked, Experts B, K, and L preferredto give conditional distributions andkeep theirprobabilistic judgmentaboutthe likelihoodof acausal or non-causal relationshipseparate, whereas ExpertFpreferred togive a distributionthat incorporatedhislikelihoodthatthe PM2.5mortalityassociation may be non-causal.Expert K specified a threshold (not shown).00.511.522.54-16 ug/m335%>16-30ug/m335%

% D ecrease i n Mo rt al i t y p e r 1 u g / m 3 PM2. 5Range ofPM2.5ConcentrationsPercentLikelihoodofCausalityExpert KDistributionsKey:Closedcircle =median; Open circle =mean;Box =interquartile range; Solid line =90% credible interval00.511.522.54-10ug/m375%>10-30 ug/m399%

% D ecrease i n Mo rt al i t y p e r 1 u g / m 3 PM2. 5RangeofPM2.5 ConcentrationsPercentLikelihoodofCausalityExpertLDistributionsKey:Closed circle =median; Opencircle =mean; Box=interquartile range;Solid line =90% credible intervalT H E EFF E C T OF I N COR P ORA T IN G A TH RES H O L D INT O THE C

- R COE FFIC I EN T

UNCE R T AI NTY DIS T RIBUTIO N O F E X PER T K EX HI B I T 3-1 3 :

N o te: E x p e rt K indi cated that he wa s 50% sure that a thr e shol d ex isted. If t h er e wer e a thr e sh old, h e
tho u g h t that there was an 80% chance that it f a l l s between 0 and 5 μg /m 3 , and a 20% chance that it f a lls between >5 and 10 μg /m 3 . The e l icitatio n te am cre a te d a pro b abil istic d ist ribution in Cry s tal Bal l TM with 50% of the weig ht at z e ro, 40% of the weig ht between >0 and 5 μg /m 3 , a n d 10% of the weight between >5 and 10 μg /m 3 . The elicitation team then used Mo nte Ca rlo sampl i ng to integrate E x pert K’s C-R

coefficient and threshol d d ist r i but i ons for a sa mple of p o pula t i on-w eight e d PM 2. 5 concentrations in the

U.S. 0 0. 1 0. 2 0. 3 0. 4 0. 5 0. 6 0. 7 0. 8

Original I n c o rpo r at ing T h re s hold% Decrease in Mortality per 1 ug/m3 PM2.5 Ex pe rt K' s C- R Co ef f i ci en t D i s t ri bu t i o n a t a B a s e l i n e A n nua l A v er ag e P M 2. 5 of 7 ug /m 3

K e y : C l os ed c i r c le = m edi an; O p en c i r c l e = mean; Box = inter q uar t ile r ange; Soli d line = 90% c r edible inter v a l



### A visual co mparison of the experts' distributio ns to th ose fro m the two prim ary c ohort studies, Pope et al. (2002) a nd Docker y et al. (199 3) su ggests that most of the sub j ective
distributions enco m p ass th e 90% confidence intervals from both those studies. The
medians and/or estim ated means fall at or between the m e dian for the ACS study (0.6%

per μ g/m 3 

### ) and the m e dian for the original Six Cities study (1.3% per μ g/m 3 

### ) for m o st of the experts. Experts A and E centered their distributi ons above the original Si x Cities results although both their distributions place so me probabilit y wei ght on the A C S results
as well.

3-373-38 

### Exhibit 3-14 helps explain some of the rationales behind both the similarities and the differences. It summarizes the studies that the experts specifically cited in connection with the development of either their medi an or with the percentiles characterizing uncertainty (minimum, 5th, 25th, 75th, and 95th, and maximum). The closed circle [ ~ 

### ] indicates those studies from which the experts drew particular effect estimates or which they used to support adjustments to the prim ary effect estimate on which they relied for the median. The open circle [ { 

### ] indicates studies mentioned in association with one or more of the percentiles used to characterize uncertainty. More detail on how each expert chose to use data from particular studies may be found in their individual summaries in the Technical Support Document for this study (IEc, under development).

Exhibit 3-14 shows a strong reliance by most experts on the Pope et al. (2002) ACS study, the Jerrett et al. (2005) re-analysis of the ACS cohort in the LA basin, the Dockery et al. (1993) original Six Cities study and finally, the Laden et al. (2006) cross-sectional analysis based on the Six Cities cohort for their median estimates. However, the particular use of individual studi es varied from expert to expert.

For many experts, the original ACS study by Pope et al. (2002) and Six Cities study by Dockery et al. (1993) formed an importa nt initial foundation for their judgments. However, questions about the implications of the non-representativeness of the original ACS cohort with respect to educational atta inment level (see Pope and Dockery, 2006) and questions about potential exposure miscl assification (due to spatial resolution of exposure) raised by the Jerrett et al. (2005) st udy played critical roles in how the experts weighed the findings from both studies. Seve ral experts thought that the original ACS study results were likely to have been un derestimates of U.S. effects due to underrepresentation of the poorly educated and/or to exposure misclassification and adjusted their median estimates to reflect the impact of educational attainment, exposure misclassification, or both (see al so Exhibits 3-4 and 3-5). Others saw the Jerrett et al. (2005) findings as rationale for placing more confidence in the original Six Cities study and its extended analyses than they previously had or to rely more extensively on the Jerrett et al. (2005) study itself. The e ffect on the central estimates from these two approaches was essentially to converge on a si milar range of values. Median effect estimates were lower when experts argued agains t adjusting the original estimates for one of these factors (e.g., Expert H did not think it appropriate to adjust for educational attainment; Expert K adjusted for neither).EXHIBIT3-14: EPIDEMIOLOGIC STUDIES RELIEDUPONBYEXPERTSIN CREATINGTHEIRC-R COEFFICIENTUNCERTAINTYDISTRIBUTIONSACS (Popeetal., 2002)ACS LAReanalysis(Jerrett etal.,2005)Six Cities(Dockery etal.,1993)Six Cities(Ladenetal., 2006(CrossSectional))ACS (Popeet al., 1995)NetherlandsCohortStudy (Hoeket al., 2002)Six Cities(Ladenetal., 2006(Changeestimate))Mallick etal., 2002Willis etal., 2002NMMAPS(Sametetal., 2000)Women'sHealthInitiative2AHSMOG (Abbeyet al.,1991 & 1999;McDonnelletal., 2000; Chenetal., 2005)Expert A{~{~{{Expert B~~~{~Expert C~~~Expert D~~Expert E~~~~~~Expert F{{~~Expert G~{~{Expert H~~{{Expert I~~~Expert J1~~{~{O{{{Expert K~~{Expert L{~{{{Total~:986411111000Total{:445111100311~=Expert usedthe study toinformthe median ofhis C-R coefficient distribution(s).{= Expert used the study toinform the uncertaintyof his C-R coefficient distribution(s).1ExpertJalsocited the following short-term studies as supportfor his uncertainty: Levyetal., 2000; Steib etal., 2002; Andersonet al., 2005; Ostroetal., 2005;Schwartzetal., 1996, Klemm etal., 2000; Burnettetal., 2003).2Study not yetpublishedat the timeof the interview.3-393-40 

### Another influential factor was the particular statistical model on which experts chose to rely on from the Jerrett et al. (2005) study. The authors explored several different models that varied with respect to the number and na ture of variables included in the model. Experts who found the base model with 44 individual covariates more compelling for their median estimates (e.g., Experts A and E) discussed them in the context of the Laden et al. (2006) study and tended to have hi gher median estimates. Other experts (e.g., Experts D and L) made arguments for use of the Jerrett et al. (2005) model that included parsimonious contextual covariates in addition to the 44 individual covariates.

The Hoek et al. (2002), Mallick et al. (2002), an d Willis et al. (2003) studies were cited occasionally in experts’ rationales for median estimates but usually in support of the Jerrett et al. (2005) findings or in support of upward adjustments to the original ACS study to account for possible exposure misclassification.

Short term studies, (e.g., NMMAPS (Samet et al ., 2000a & b) were cite d relatively rarely in the development of experts distributions, even among those experts who generally found the data base for the effects of short term exposures more compelling. They were most often raised to help define the lowe r limits of the distributions, minimum to 25th percentiles. This use of the studies appears consistent with the views of those experts who argued that mortality from short-term e xposures (i.e., those deaths caused primarily by an acute exposure) were likely to make up a small proportion of total deaths from

PM 2.5 

### exposure.

3.3. POST-ELICITATION WORKSHOP 

### The purpose of this workshop was to reconvene the panel of experts to present preliminary results of the study and hold discussions on the elicitation process. The workshop consisted of a briefing describing preliminary results followed by several discussion sessions. The discussion sessions were intended to explore topics where expert opinions varied significantly; discuss new scientific evidence that emerged during the course of conducting the 12 interviews; and encourage each expert to critically review his judgments. The workshop was not intended to promote consensus with in the group, but rather to share ideas among experts and discuss evidence for or against such ideas. The workshop also served as an opportunity fo r experts to provide EPA with feedback on the expert elicitation process.

Topics discussed during the workshop included quantification of effects by PM level, the role of intervention studies in quantifying impacts, exposure error due to the use of central site monitors, effect modificatio n due to educational attainment, and the integration of each expert's judgments on the likelihood of a causal relationship into his overall distribution. There was also an open session at the end where experts could discuss other topics and provide feedback to EPA and the Project Team.

At the end of the workshop, the experts we re given instructions on the process by which experts could update their judgments. It required completing a modification form in which the expert was asked to specify the changes he wished to make and provide detailed rationale explaining and supporting the changes.3-41 

### Although the workshop gave the opportunity to discuss a number of topics where cogent, but conflicting arguments had been made, most of the discussion focused on three topics: exposure misclassification, effect modification, and causality.

3.3.1 EXPOSURE MISCLASSIFICATION 

### The purpose of this session was to discuss the issue of adjusting effect estimates for exposure misclassification due to the use of centr al site monitor exposure estimates. This topic was raised primarily because of the variance in opinions expressed by the experts as to the importance of this issue, and the LA ACS results, for developing C-R coefficient distributions. Another reason for discussing it was that during the time the elicitation interviews were being conducte d, preliminary results from an analysis by Jerrett et al. of ACS data in New York (NY ACS) were presen ted at the annual Health Effects Institute (HEI) meeting in April. Like the LA ACS study, the NY ACS study used modeling in an attempt to improve exposure characterization for the cohort. Some of the experts who were interviewed subsequent to the HE I meeting discussed the preliminary NY ACS results, which did not generally corroborate the findings of the earlier analysis of the LA ACS data, when reflecting on the overall ev idence for the role of improved spatial resolution. Thus, the Project Team also raised the issue in order to give all experts an opportunity to discuss and critique the newer findings.

The topic sparked a lively discussion that focused less on the Jerrett findings in NY and more about when it is appropriate to cut o ff the introduction of new information to an elicitation process in general, and what criteri a should be used to decide what studies are introduced. The concern expressed by some e xperts was that new information is always becoming available and that allowing individual studies to be considered on an ad hoc basis could reflect bias. The project team generally agreed that while it made sense to allow experts to use whatever knowledge they have available to them, the project should
have some formal cut-off for the introduction of new information.

Experts did discuss some possible reasons fo r the different results in the NY study, including the smaller range of PM levels over a smaller geographic area, and about uncertainty in the exposure surface model a pplied to NYC with its street canyons. Ultimately, however, most experts simply thought that the results are too new and too preliminary to give them much weight.

None of the experts chose to alter their quan titative judgments with respect to this topic.

3.3.2 EFFECT MODIFICATION 

### The purpose of this session was to discuss the issue of adjusting ACS cohort effect estimates for effect modification due to educational attainment, as demonstrated in the HEI reanalysis (Krewski et al., 2000). A number of experts adjusted their effect estimates upward to account for this effect, or relied on studies (e.g., Six Cities) that the experts thought were less affected by it. The discussion centered around whether or not the adjustment of the ACS estimate was r easonable, given the observation during the
interviews that the focus of the current e licitation was on the U.S. population today and the fact that the percentage of high-school educated people in the U.S. today is more
similar to that in the ACS cohort than in the Six Cities cohort.3-42 

### One expert argued that re-weighting for education is helpful to see if it explains differences between the Six Cities and ACS study results. Because he believes the education factor really is a proxy for other factors related to socioeconomic status, he was not convinced that changes in education over time would necessarily affect the true factors underlying the difference attributed to education. Others agreed with this interpretation, one calling it an issue of norm alizing the results of the ACS and Six Cities studies. One expert added that the HEI reanalysis found effect modification by education in both the original ACS and Six Cities studies (Krewski et al., 2000).

On the subject of the relevance of current educational attainment rates, one expert suggested that a high school degree implied more about education and lifestyle/SES in the past than it does today. Another expert noted that unpublished results from the Women’s Health Initiative, a study involving a modern, highly educated cohort, showed some of the largest effect estimates observed to date and no effect modification by
education.

One expert, Expert I, made changes to his uncer tainty distribution, partially in response to this discussion. He modified his approach (described previously) to include a specific upward adjustment of the 2002 ACS results fo r educational attainment and revised the subjective weights he had assigned to the three studies he had chosen to combine (adjusted 2002 ACS, LA ACS, and the Six Cities Follow-up) to weight each equally.

3.3.3 CAUSALITY 

### The most extensive discussion during the workshop focused on the question of how judgments about causal likelihoods were incorporated into distributions characterizing uncertainty in the PM 2.5 

### -mortality coefficient. As discussed in section 3.2.2, the protocol gave each expert the choice of 1) incorpora ting his views directly into his uncertainty distribution; or 2) first developing his distribution conditional on the assumption that the relationship was causal, and then having th e elicitation team combine that distribution probabilistically with his previously elicited likelihood of a causal relationship. During each elicitation interview, experts who c hose the second option (the "disaggregated approach") were shown box plots of both thei r distribution "conditional on causality" and a joint distribution that incorporated both th eir judgments about the C-R coefficient and about causality. 40 

### The joint distribution was presented to help these experts assess the potential impact of their beliefs on causality on their distribution of C-R coefficients. During the Post-elicitation Workshop, these joint distributions were presented along side distributions from experts who chose the firs t option above (the "aggregated approach"), in order to facilitate comparison across experts.

As discussed in Section 3.1.7, the approach to causality, although a small component of the protocol, engendered an extensive discu ssion during the workshop. Comments raised during the discussion of causality included the following:

40 The joint distribution was ge nerated using the spreadsheet t ools described in Chapter 2 and Crystal Ball, assuming the two

distributions were independent.3-43 • 

### One expert suggested that the quantitativ e estimates of causality are overly precise

and recommended that the experts instead be assigned to qualitative categories based on their responses.

• 

### One expert said the causality question was confusing and poorly defined.

• 

### Others thought that it was unclear during the interview how the causality results

were to be combined with their distribut ions if they were first developed first conditional on a causal relationship. Althoug h this approach had been laid out in the Pre-elicitation Workshop and in the briefing book, not all experts had fully considered the implications of the approach.

• 

### One expert said it was difficult to comp letely eliminate the concept of causal

likelihood when characterizing an uncer tainty distribution conditional on the existence of a causal relationship, and ther efore, that combining their distribution with causality was potentially double coun ting. Another expert suggested the probabilities might not be independent and if so, cannot be multiplied.

• 

### One expert argued that his uncertainty distribution was derived from his

interpretation of the literature and that it was conceptually distinct from his evaluation of the likelihood of a causal relationship. For example, he noted that his views on the findings of a particular well-designed epidemiological study might not necessarily change over time; however, independent toxicology, mechanisms, clinical studies might change his views of the likelihood that the relationships observed in the epidemiological study are real. He therefore preferred that these concepts be kept separate in the final report.

Overall, the discussion highlighted the difficulties of eliciting conditional distributions from the experts as well as a general unease am ong the experts with applying the concept of expected value in this context to their mortality effect estimates. Because several expressed discomfort with presenting the joint distributions, we have not included them in this report.

Experts also asked to see a revised causality question. The Project Team agreed to review the causality question following the workshop, but after further review concluded the question was sufficiently clear.

Of the eight experts who originally specified a C-R coefficient distribution conditional on causality, four specified changes to their quantitative judgments following the workshop
discussion: 41

• 

### Expert A changed the interpretation of his distribution from conditional on a causal

relationship to NOT conditional, becau se of the concern over double-counting described above;

• 

### Expert E changed his best estimate of the likelihood of a causal relationship from

95 percent to 99 percent. In explaining the rationale for his change, Expert E

41 Note that Exhibits 3-9 through 3-14 i ncorporate the changes made by expert fo llowing the post-elicitation workshop.3-44 

### referred to a statement made by another expert during the workshop that the assigning of some probability of no effect was really an assigning of a finite probability that there was not even a very small effect. After considering his previously cited evidence in light of this statement, he changed his previous assumption that there was a five percent probability of zero effect to there being a one percent probability of there being zero effect;

• 

### Expert L expressed different causal likelihoods for the two segments of his nonlinear C-R function. He lowered his best estimate of the likelihood of a causal relationship for the lower PM range (4-10 μ g/m 3 

### ) from 99 percent to 75 percent, keeping his estimate for the higher range unchanged at 99 percent. Expert L cited the inherent difficulty of assessing effects at annual average PM 2.5 

### levels below 10 given current data. He changed his li kelihood of a causal relationship responses for the lower segment of his function to a range of zero to 95 percent with a best estimate of 75 percent;

• 

### Expert K expanded his range of causality estimates from 20 to 50 percent to 5 to 50

percent, but left his best estimate of causality unchanged at 35 percent. Expert K cited concerns about exposure error and unaccounted for potential confounders in extending his lower bound on the likelihood of a causal relationship.4-1 

### CHAPTER 4 | DISCUSSION

In response to recommendations made in th e 2002 National Academy of Sciences (NAS) report, “Estimating the Public Health Benefits of Proposed Air Pollution Regulations,” EPA has been exploring ways to improve the characterization of uncertainty in its analyses of the health benefits of proposed or existing regulations affecting air quality. EPA has now conducted two projects, a pilot study (IEc, 2004) and a larger-scale study that has been the focus of this report. Both studies sought to provide a more complete characterization, in qualitative and quantitative terms, of the uncertainties associated with the relationship between reductions in ambient PM 2.5 

### and mortality, using judgments elicited from scientific experts.

The previous chapter summarized the resu lts of the expanded study, which were generated though detailed intervie ws with 12 experts. The experts were selected via a peer nomination process intended to yield a set of highly qualified experts from key scientific disciplines who have extensiv ely studied the health effects of PM 2.5 

### exposures. The peer nomination process led to selection of eight epidemiologists, one clinician/scientist, and three toxicologists. Experts were invited to attend a Pre-elicitation Workshop to discuss evidence relevant to the project and review the protocol in detail; to participate in a one-day elicitation session; a nd to attend a final Post-elicitation Workshop to review results and discuss remaining questions.

As discussed in detail in the preceding chap ter, the experts relied on a broad array of evidence from varied scientific disciplines to develop their judgments about the likelihood and magnitude of a relationship between PM 2.5 

### and mortality. The epidemiological studies based on the Six Cities and ACS cohorts were central to many discussions, but their results were critiqued in the context of newer evidence from more recent epidemiological studies in the U.S. and abroad, as well as toxicological and clinical studies. The various “intervention” studies, the Jerrett et al. (2005) re-analyses of the ACS cohort in Los Angeles, and the S un et al. (2005) studies were an important focus of many discussions. Overall, the imp act of these more recent studies led to larger mortality effect estimates and greater certainty about a causal relationship than was observed in the pilot.

The remainder of this chapter provides add itional perspective on these results. We first present findings from analyses conducted to evaluate the sensitivity of the quantitative judgments to various factors. We next compar e the results of this study to those of the pilot study. We conclude with an evaluation of the strengths and limitations of this expert judgment study.4-2 4.1 SENSITIVITY ANALYSIS 

### IEc conducted a simplified benefits analysis to assess the sensitivity of the results generated to the responses of i ndividual experts and to three factors in the study design: the use of parametric or non-parametric approaches by experts to characterize their uncertainty in the PM 2.5 

### -mortality coefficient, partic ipation in the pre-elicitation workshop, and allowing experts to change their judgments after the post-elicitation workshop.

The details of the simplified example benefits analysis can be found in Appendix C. In brief, the individual quantitative expert judgments were used to estimate a distribution of benefits, in the form of number of deaths a voided, associated with a reduction in ambient, annual average PM 2.5 

### concentrations from 12 to 11 μg/m 3 

### . The 12 individual
distributions of estimated avoided deaths we re then pooled using equal weights to create a single overall distribution reflecting input from each expert. This distribution served as the baseline for the sensitivity analysis, which compared the means and standard
deviations of the baseline distribution with several variants.

4.1.1 SENSITIVITY TO INDIVIDUAL EXPERTS 

### The first analysis examined sensitivity of th e mean and standard deviation of the overall mortality distribution to the removal of individual expert’s distributions. Individual expert distributions were sequentially remove d from the combined distribution, the mean
and standard deviation were re-calculated, and the percent change from the full set of experts calculated.

The results of the analysis are shown in Exhibit 4-1 in the form of a histogram of the percent changes in mean and standard deviation resulting from the removal of individual experts. The histogram shows that the mean of the combined distribution shifted less than +/- 4 percent for 10 of the 12 experts and by +/- 2 percent for seven of the 12 experts. Two experts were more influential, with removal of Expert E causing the distribution mean to shift downward by eight percent and the removal of Expert K causing the distribution to shift upward by eight percent. In general, the results suggest a fairly equal split between those experts whose removal shifts the distribution mean up and those who shift it down, and relatively modest impacts of individual experts.E X H I BIT 4

- 1: PERC EN T CH AN GE IN TH E M E AN AND ST A N DAR D DE VIA T I O N O F AN

E X A M PLE POO L E D MOR T ALIT Y BENE FIT ES TIMA T E BAS E D ON RES U L T S OF THE

EXP E R T EL I C I T A T IO N S

, A F T E R R E M O V I NG E A CH EX PER T

Note: M o rtal ity benefits were pooled of a 1 μg/m 3 decrease in PM 2. 5 , beginn ing at an annual average

conc entration of 12 μg/m 3 . The poo l ed estima te weighted ea ch e x per t equ a lly

. 0 1 2 3 4 5 6 7 8 9 10

- 8

- 7

- 6

- 5

- 4

- 3

- 2

- 1 012345 678

Pe r c e n t C h a n g eNumb Me a n
S t and a r d D e v i at io ner of Experts



### The standard deviation of t h e com b ined distributio n was also not strongl y affected b y

rem oval of individual ex perts; Experts E and K again had the greatest influence on the spread of res u lts both causing the overall standa rd deviation to decline b y five p e rcent (K) and eight percent (E) whe n re m oved. Overall, these results indicate that no one expert
dom inate s the results.

4. 1.2 SENSITI V I T Y T O CH OICE OF DIST RIBUTIONA L FORM T O CH ARAC TERIZE

UNCER T AI NTY 

### The protocol gave experts so m e latitude in how t o estim ate the value of the PM 2.5



### m o rt ality coe fficient for di fferent percen tiles. They c ould estim at e indivi dual pe rcentiles directly or they could fit parametric dist ri butions eith er by estim ating the parameters of
the distributi on (e.g., m e an and standar d devi ation) or fitting distributions t o individual elicited percentiles (e.g., 50 th 

### and 95 th 

### percentiles). The elicitation team had some concern that experts fitting param e tri c distributions m i ght ultim ately be m o re overconfident

. While we have no way o f know ing ult i m a tely which jud g m e nts are closer to the “truth,” we can obser ve how the di stributions de rived using t h e two approaches
differ.

The sensitivity anal y s is suggests that the sta ndard deviation of the pooled distribution of experts who specified parametric distrib u tions in fact is slightl y lar g er than that of the pooled distribution of expe rts fro m whom all the indi vidual percentiles were elicited (i.e.,

4-34-4 

### those specifying “custom” or non-parametric distributions). The standard deviation of thepooled estimate for the parametric group was 23 percent greater than that for the nonparametric group. (The means of the two di stributions remained close to that of the overall mean for all experts.) It is worth no ting, however, that the two most influential experts in the sensitivity analysis above, E and K, are both in the parametric group. When Experts E and K are removed from the parametric group, the difference in the spread of the distributions for the parametric and non-parametric groups falls to less than 3 percent. Thus, this analysis suggests that the use of parametric distributions led to distributions with similar or slightly increased uncertainty compared to experts who provided percentiles of a non-parametric distribution.

4.1.3 SENSITIVITY TO PRE-ELICITATION WORKSHOP PARTICIPATION AND TO

CHANGES MADE AFTER THE POST-ELICITATION WORKSHOP 

### Reviewers of the pilot study suggested th at more opportunities for communication and sharing of information between experts, incl uding after the individual interviews, would improve the elicitation process. The goal of such communication is to facilitate sharing of new information and to resolve questions that might have arisen during the elicitation process. However, residual concerns a bout such workshops remain based on early approaches to expert elicitation, most notably Delphi methods, where sharing of
individual results was used to he lp bring experts to consensus.

Participation in the Pre-elicitation Workshop di d not appear to have a significant effect on experts’ judgments. The mean and standard deviation for a pooled estimate of avoided deaths for those experts who did not attend the Pre-elicitation Workshop were within 10 percent of the mean and standard deviati on of the pooled estimate for Pre-elicitation Workshop attendees.

As discussed in earlier sections, we found th e Post-elicitation Workshop to be important for discussing some critical points about the design of the study and presentation of results, most notably on the question of causal likelihood. Only four experts chose to make changes to their distributions following our discussions at the Post-elicitation Workshop. Using the same methodology as above, we compared the combined distributions using results prior to and following the Post-elicitation Workshop. We found a minimal impact due to the changes made after the Post-elicitation Workshop. The overall mean number of estimated deaths increased by approximately one percent compared to the estimate prior to the Workshop, and the standard deviation did not change.

4.2 COMPARISON TO THE PILOT STUDY FINDINGS 

### It is reasonable to ask whether the results of the current study are comparable to those observed in the pilot study (IEc, 2004). The pilot study was conducted in preparation for the current study and thus has many similarities; however a number of changes, both in the basic formulation of the fundamental qu estions asked, and in the body of available scientific data, make it necessary to be caref ul about direct quantitative comparisons of individual responses. Some of the differe nces between the two study designs include:• 

### The pilot study asked experts to develop separate quan titative distributions for t h e
PM 2. 5 

### -m ortality effect for long-term (an nua l average) and for short

- term exposures (24 hour average). The current study asked experts to esti m ate a total m o rtality effect integrating effects of long-term an d short-term exposures. However, m o st experts in thi s study ultim ately relied on PM studies that focused on the effects of long-term or interm ediate exposures for t h eir quantitati ve estim ates of the total

m o rt ality effect, and relied on short-term studies onl y when esti m ati ng possible
lower bounds . As a result, the distributi ons in th is stud y m a y be most com p arab le to those elicited for the effects of long-t e rm exposure in the pil o t st ud y (See Exhibit 4-2).

EX HI B I T 4-2 : P I L O T S T U D Y R E SUL T S - C O MP A R I S O N O F EX PE R T S’ JUD G M E NT S

AB OUT T H E PER C EN T I N C R EAS E IN ANN U A L NON

- A CCI DENT AL MOR T A L IT Y

AS SOCIA T ED WIT H A 1 μ g/ m 3 I N C R EA S E I N A N NU AL A V ER AG E E X PO SU R E S T O PM 2 . 5 ( U .S . B A SE LINE 8 T O 2 0 μ g/ m 3 )

0 0.5

1 1.5

2

A B* C- -8 ug /m 3 C--1 0 ug /m 3 C--15 ug /m 3 C--20 ug /m 3 D E

Exp e rt% Increase in Mortalityper1ug/m3PM2.5 

### Med i an

5 % ile

In terqu a rtile

R a nge 

### ◊ 9 5 % ile

*Exp ert B specified this distribution fo r the PM-mortali ty coefficient above a n unce rtain threshold, whic h he

cha r a cter ized a s r a nging betw een 4 a n d 15 with a moda l va lue of 12 μg/m 3

• 

### The pilot study asked experts to esti m ate a PM m o rtality effect assuming an 8-20

μg/m 3 

### concen tration range for annual av erage exposures in the U.S. The current study asked experts to assume a 4-30 μg/m 3 

### concentration range for annual average

PM 2. 5 

### . For those experts w ho assumed linear or log-linear functions, we m i ght not expect the quantitative esti m ates to change dram atically as a result of expanding the range. Ex panding the r a nge on the l o wer end may have influe nced some experts’ decision to prov id e a seg mented, no n-linear function.

4-54-6 • 

### In the pilot study, the protocol design le d experts to incorporate their estimates of

the likelihood of a causal relationship direc tly into their overall distributions. In the current study, the protocol gave experts the option of incorporating their causal likelihoods directly into their distributi ons or to have them probabilistically combined with their mortality effect dist ribution conditional on the existence of a causal relationship. Following the post-elicitation workshop, nearly half the experts expressed a preference for displaying their distributions and likelihoods of causality separately. The distributions for these two approaches are therefore presented in separate exhibits, making it difficult to compare across all 12 experts directly and to compare the conditional distributions directly to the distributions in the pilot study.

Despite these differences, some interesting observations can be made about the results of the interviews in the current study (See Exhib its 3-10 and 3-11) relative to the pilot study (Exhibit 4-2).

• 

### While the overall range of median estimates given by the experts in this study is

similar to that given by experts in the pilot study, most experts in the current study had higher effect estimates than in the pilo t study. In the pilot study, most experts provided median values of 0.5 percent or less for the reduction in mortality per 1 μg/m 3 

### reduction in PM 2.5 

### , and the bulk of most distributions was below 1 percent (only one included a 75 th 

### percentile above 1). In this study, most of the experts estimated the median to be near or above 1 percent.

• 

### The experts’ distributions in the current study display a similar diversity in the

degree of uncertainty expressed as in the p ilot study. The distributions range from Expert F whose uncertainty was similar in sp read to that of the Pope et al. (2002) study, to Expert A whose uncertainty was 3-4 times broader. However, all of the experts in this study gave distributions that were roughly similar or wider than those published in the ACS and Six City cohort studies.

• 

### In the pilot study, two of five expert s incorporated a threshold into their

distribution; in the current study only one of 12 experts incorporated a threshold.

• 

### The experts’ views on the likelihood of a causal relationship between PM 2.5



### exposure and mortality also appeared to show some marked changes from the pilot study. (Note that the experts in the current study were asked about the likelihood of a relationship between annual average exposures, including the influence shortterm or peak exposures, and all-cause mort ality, whereas the pilot study asked the experts about long-term and short-term e xposures separately.) In general, both the
best estimate likelihoods and the likelihood ranges were higher in the current study than in the pilot study. In the current study, two of 12 experts expressed some doubt about the strength of the causal relationship, giving overall likelihoods of 35% and 70%. The remainder expressed probabilities of 90% or above. As indicated in Exhibit 4-3, the experts participating in the pilot study expressed much more skepticism, and were less convinced by the evidence for long-term exposures than for short-term exposures.4-7 EXHIBIT 4-3: PILOT STUDY RESULTS ON THE LIKELIHOOD OF A CAUSAL

RELATIONSHIP BETWEEN PM 2.5 AND NON-ACCIDENTAL PREMATURE MORTALITY

EFFECTS OF LONG-TERM PM 2.5

EXPOSURE EFFECTS OF SHORT-TERM PM 2.5

EXPOSURE

EXPERT LIKELIHOOD (CATEGORICAL) LIKELIHOOD RANGE A (BEST ESTIMATE) % LIKELIHOOD (CATEGORICAL) LIKELIHOOD RANGE A (BEST ESTIMATE) %

A Highly likely 85-90 (88) Highly likely 90-95 (93)

B Somewhat unlikely 40-50 Somewhat likely 65-80

C Somewhat unlikely 50 b Somewhat unlikely to somewhat likely 50 b

D Somewhat likely 50 Somewhat to highly likely 80-90

E Likely 80-98 (95) Likely 80-98 (95)

a. Represents minimum to maximum for ca tegorical likelihood specified. Each expert specified his own quantitative

probability estimates to match his qu alitative categorical description. b. Expert C wanted to answer this qu estion separately for different parts of the range. The 50 percent value

represents his “average” for the range, with little or no probability of a causal relationship at the low end of the range and a high probability at the upper end of the range.



### The differences in results compared with the pilot appear to reflect the influence of new research on the interpretation of the key epid emiological studies that were the focus of both elicitation studies, more than the influe nce of changes to the structure of the protocol. They may also reflect differen ces in the composition of the expert panel.

The higher central tendency appears to have been influenced in particular by research that has been published since the pilot study was conducted. More of the experts in the current study made explicit adjustments to their estimates to account for effect modification and exposure misclassification in the ACS cohort studies. While effect modification by educational attainment was raised by several experts in the pilot study as a result of the Krewski et al. (2000) reanalysis, it was viewed there more as a source of uncertainty rather than a rationale for adju stment of study findings. The Pope andDockery (2006) evaluation of the impact of educational attainment, discussed at the preelicitation workshop and by many of the expe rts during their interviews, appeared to convince several experts that the Pope et al. (2002) mortality effect estimates were likely to be biased low to some degree relative to the effect estimate that might have been observed in a population more representative of the U.S. population. The Jerrett et al. (2005) study, which many experts believed to demonstrate the effect of improved spatial resolution of exposure in reducing exposure misclassification, influenced several experts to think that the Pope et al. (2002) estimates were likely to be biased low and that the higher effect estimates in the Dockery et al. (1993) study might reflect in part better exposure measurement. Results from the various intervention and exposures change studies (Clancy et al., 2002; Laden et al., 20 06), which showed higher mortality effects than Pope et al. (2002), were also cited as support for experts' subjective judgments about the magnitude of potential mortality reducti ons, although experts varied in their views about what types of exposures (long- or short-term) may have contributed to these changes in mortality. The impact of rece nt literature illustrates how expert views can4-8 

### evolve in the face of new information and suggests that there may be benefits to following up with experts in the future as th e scientific knowledge of this issue continues
to advance.

It also appears that experts’ increased ov erall confidence in a causal effect of PM 2.5 

### on mortality was driven by the recent contributions to the body of scientific evidence. These were discussed in greater detail in Chapter 3 of the report, but some examples include the Clancy et al. (2002) and Laden et al. (2006) studies which several experts cited as specifically relevant to this elicitation because they showed declines in mortality following declines in particle air pollution. The long-term animal study of Sun et al. (2005) was considered by several experts to provide important support for a causal link between long-term PM 2.5 

### exposures and the development of atherosclerosis and increased risk for cardiovascular related mortality. As discussed above, several experts also believed the Jerrett et al. (2005) study lent support not just to the ACS findings but also to the Six Cities study, which had received less wei ght in the pilot study. The impact of the differences between the current and pilot stud y in the strength of the causal relationship are not so evident in the displays of the distributions themselves. It will become more evident in benefits analyses where the causal likelihoods will factor directly into the
calculations of numbers of deaths.

4.3 EVALUATION OF THE EXPERT ELICITATION PROCESS 

### We begin our evaluation of the expert elicitation process with a summary discussion of the strengths and potential concerns arising fro m the particular design of this study. The
discussion touches on some of the differences between the design of the pilot study and the current study.

4.3.1 STRENGTHS 

### Below we cite strengths of the process in the areas of the expert panel size, expert communication, and elicitation pr otocol and interview process.

Expert Panel Size 

### This study involved a larger number of experts than the pilot study. The purpose of the larger panel was primarily to achieve br oad representation of prevailing scientific opinions about PM 2.5 

### and mortality. The availability of a larger panel also permitted some observations about the marginal contri bution of additional experts, particularly when they share highly similar views or rely on the same data.

Our study tends to corroborate the work of othe rs (e.g., Clemen et al., 1985) that show when judgments are highly correlated, the marginal influence any individual on the predictive value of the group of judgments is small. As discussed in previous sections, all of our experts relied on many of the same core studies, though their evaluations of those studies were colored by differing experience, perspectives, and knowledge of additional research. We therefore expect that many of these judgments are not independent. We do not have means of calibrating these judgment s, individually or collectively, but our sensitivity analysis suggests that none of our individual experts, even those with more4-9 

### independent views, had a strong influence on the outcome of the simple mortality estimates. This result suggests that adding more experts would add relatively little information. Recall that a broad representation of viewpoints in this context does not mean a statistically representative sample but rather a range of well-reasoned opinions.

Expert Communication 

### The current study allowed for a substantial degree of inter-expert communication. The EPA Symposium in April 2005 gave a wide variety of interested scientists and policy makers the opportunity to hear about the project and to comment on the proposed protocol. Several changes were made as a result, including the decision to drop separate questions about the role of PM 2.5 

### components, and to offer a disaggregated approach to incorporating causal likelihood into the uncertainty distributions.

The workshops held with the experts prior to and following the individual interviews were an important element of the process. The Pre-elicitation Workshop was used to improve the level of preparedness for th e interviews by increasing understanding of project objectives, clarifying questions a nd assumptions in the protocol, sharing and critiquing new data or analysis. The Post-e licitation Workshop was very valuable for revisiting points of confusion or concern about the protocol, in particular regarding presentation of the causal likelihood informati on, as well as for providing feedback to the project team. Unlike some elicitation pro cesses where these sessions may be used to bring scientists with divergent views “into th e fold,” these workshops were explicit that consensus was not the goal. We did see that information presented and discussed at the pre-elicitation workshop was influential in inte rviews and that clarifications in the postelicitation workshop led four experts to change their distributions slightly. The sensitivity analysis suggests that these modifica tions did not lead to substantial changes
or movement towards a consensus view. We found that individual expert’s views were often deeply held and resistant to challenge.

Elicitation Protocol and Interview Process 

### Newly-developed, computerized elicitati on tools facilitated the development of quantitative distributions in the current study. In response to the varied approaches to characterizing the PM 2.5 

### -mortality relationship in the p ilot study, in particular the specification of thresholds and other non-lin ear functions, we developed spreadsheets to enable the experts see in “real-time” the implica tions of some of their stated assumptions or opinions. They allowed the experts the fr eedom to characterize their uncertainties in
the PM 2.5 

### -mortality effect using a variety of distributional forms, parametric and nonparametric. They allowed for reality checks on the mortality implications of any individual estimate. While we had some concern that parametric approaches might lead to greater reliance on the “anchoring and adjustment” heuristic, the sensitivity analysis suggest that is not the case. The distri butions given by the experts who assumed parametric distributions were collectively broa der than the non-parametric distributions.

Compared to the pilot study, the protocol fo r the current study used a more systematic approach to discussing some of key sources of uncertainty in each epidemiologic study the expert relied upon for his quantitative estimates, such as confounding, effect4-10 

### modification, and exposure meas urement. Though some experts were more comfortable with identifying and discussing individual fact ors than others, the process did give the
elicitation team a clearer record with which to revisit specific assumptions during the quantitative step of the protocol.

The protocol for the current study was designed with a focus on a question of particular relevance to regulatory analysis, the impact of annual average exposures (including the long and short-term exposures) on all cause mortality. The experts were faced with developing one uncertainty distribution rather than two (effects of long-term and effects of short-term) as in the pilot study. This a pproach did add some complexity to the task since experts essentially had to integrate thei r views on the influence of long-term and short-term exposures into their final responses.

4.3.2 LIMITATIONS 

### Below we cite potential limitations of the process in the areas of expert selection, protocol development, eliciting probabilities, and generalizability.

Expert Selection 

### Some experts expressed concerns about the breadth of expertise (as indicated by professional training) and institutional affiliati ons represented by the current panel. We are aware that the initial expert selection process followed in this study was not as successful at identifying individuals with as broad a range of expertise and institutions as anticipated. The expert selection process did specify a number of criteria for peer
nominators to consider that were intended to foster nomination of individuals with different backgrounds. However, it appear ed that in some cases nominators overlooked these criteria in favor of choosing names that were most familiar. The additional steps taken with the assistance of HEI to augment the numbers of experts with backgrounds in fields outside epidemiology were essential. In the end, the expert selection process did succeed at recruiting 12 individuals represen ting key disciplines whose work has been widely-cited and respected in the PM-mortality field.

We do not believe that havi ng multiple experts from the same institution is necessarily detrimental to the study. Merely being co lleagues does not necessarily imply that experts share the same views. We do recognize that many of the experts have collaborated on projects together; however, restricting expert selection on this basis would severely limit the pool of qualified experts and restrict our ability to interview top-notch researchers.

The process is labor intensive and requires a s ubstantial commitment by the experts. The level of commitment can and did in some cases impact the willingness of experts to participate in the project, and this can also affect the final composition of the group.

Finally, as we observed in the pilot study, defining “expert” for complex multidisciplinary questions like those posed in this study is a difficult task. Making judgments
about PM 2.5 

### mortality requires extensive scientific knowledge spanning epidemiological studies of various designs, in vitro and in vivo toxicological studies, clinical medicine, and exposure analysis, among other fields . Few, if any experts are equally knowledgeable about all these fields. This limitation poses challenges for expert4-11 

### selection and the design of elicitation protocols, and for interpreting the experts’ uncertainty distributions, which refl ect these differing degrees of knowledge.

Protocol Development 

### Assessing the appropriate degree of disaggreg ation for quantitative questions remains a challenge. As noted above, the current study tried to simplify the process by reaggregating the quantitative question relative to the pilot but may have made it more challenging to be certain about the elements included in the final response from a few experts. While a disaggregated approach has intuitive appeal, it can sometimes introduce complications by requiring that the exact math ematical relationship between the elements be specified. When experts agree on the basi c relationship, it is not a problem. In the case of the discussions on causal likelihoods and how to incorporate them into distributions, we encountered disagreements during the elicitation exercises that were resolved only in the post-elicitation workshop.

Eliciting Probabilities 

### As in the pilot study, we observed some va riability in the ease with which experts could translate their opinions into quantitative probabiliti es. Expertise in a particular scientific discipline does not always require expertise in probability and, even when it does, subjective probability elicitations can be ch allenging. Yet, one of the key challenges facing the acceptance of expert subjective judgm ents is whether experts have successfully represented their knowledge and expertise in the form of distributions. This is an issue that is common to all subjec tive probability elicitations and one which has been addressed in some studies by incorporating a calibration step to evaluate the expert’s judgments relative to known values or by combini ng the judgments for use in analysis.

Generalizability 

### It is important to recognize the constraints on the generalizability of the results from this study. As with any study, this one was c onducted with a particular objective in mind – to estimate the effect on all-cause mortality of a 1 μg/m 3 

### decline in PM 2.5 

### . It required a number of simplifying assumptions (e.g., regard ing baseline conditions in the U.S., the impact of regulation on co-pollutants and components, and the distribution of susceptible individuals in the U.S. population, among many others), as described in the protocol in Appendix A, which should be reviewed when considering the applicability of these
results.5-1 

### CHAPTER 5 | CONCLUSIONS

Several conclusions can be drawn from the r esults of this expert elicitation project.

• 

### The experts’ distributions in the current study display a similar diversity in the

degree of uncertainty expressed as in the pilot study. Compared to the pilot study, experts in this study were in general more confident in a causal relationship, less likely to incorporate thresholds, and reported higher mortality effect estimates. The differences in results compared with the pilot study appear to reflect the influence of new research on the interpre tation of the key epidemiological studies
that were the focus of both elicitation studi es, more than the influence of changes to the structure of the protocol. Th ey may also reflect differences in the composition of the expert panel.

• 

### Experts in this study tended to be confident that PM 2.5 

### exposure can cause

premature death. Ten of twelve expert s believed that the likelihood of a causal relationship was 90 percent or higher. The remaining two experts gave causal probabilities of 35 and 70 percent. Recent research in both epidemiology (e.g., Jerrett et al., 2005, Laden et al., 2006) and toxicology (e.g., Sun et al., 2005) significantly contributed to experts' confidence.

• 

### Among the experts who provided distribu tions that were conditional on the

existence of a causal relationship, medians ranged from 0.4 to 2.0 percent (Exhibit 3-10). For five of the seven experts in the group, the effect of integrating causality in a benefits assessment would be small, given the high likelihoods of causality they expressed. Median benefits estimated using the distributions of two of the experts in this group would show more significant declines.

• 

### Among the experts who directly incorporated their views on the likelihood of a

causal relationship into their distributions (Exhibit 3-11), the central (median) estimates of the percent change in all-cause mortality in the adult U.S. population that would result from a permanent 1 μg/m 3 

### drop in annual average PM 2.5 

### concentrations ranged from 0.7 to 1.6 percent

• 

### Distributions of uncertainty around these me dian estimates varied across experts,

but in almost all cases exceeded the statis tical uncertainty bounds reported by any one epidemiologic study, suggesting that the expert elicitation process was successful in developing more comprehe nsive estimates of uncertainty for the

PM 2.5 

### mortality relationship.

• 

### Only one of 12 experts explicitly incorpor ated a threshold into his C-R function.

The rest beloved there was a lack of empirical and/or theoretical support for a population threshold. However, three other experts gave differing effect estimate5-2 

### distributions above and below some cut-off concentration. The adjustments these experts made to median estimates and/or uncertainty at lower PM 2.5 

### concentrations
were modest.

• 

### Although the quantitative question asked expe rts to consider mortality changes due

both to short-term and long-term PM 2.5 

### exposures, all experts based their median effect estimates on effects due to long- term exposures. Short-term exposure
effects were sometimes used to derive lower-bound effect estimates.

• 

### A sensitivity analyses conducted using a simp lified benefits analysis demonstrated

that no individual expert’s distribution of e ffect estimates had more than a plus or minus eight percent impact on an overall, pooled distribution of effects. The influence of individual experts appeared symmetrically distributed (see Exhibit 4- 1).

• 

### Confounding of epidemiological results tended to be a minor concern for most

experts. Only one of 12 experts expressed substantial concern about confounding as a source of error in the key literature on PM 2.5 

### and premature mortality.

• 

### Experts relied upon a core set of cohort epidemiology studies to derive their

quantitative estimates, mainly those associated with the ACS and Six Cities
cohorts. The Six Cities results tended to be weighted more highly by experts in this study than in the pilot study. Th e greater emphasis on Six Cities appeared to result from corroborating evidence in the r ecent Six Cities follow-up (Laden et al., 2006) and from concerns about potential e xposure misclassification issues and/or
effect modification in the ACS cohort.

• 

### Experts’ concerns regarding potential negative bias in the ACS main study results

due to effect modification (see Pope a nd Dockery et al., 2006) and/or exposure misclassification (Jerrett et al., 2005; Willis et al., 2003; Mallick et al., 2002) led many experts to adjust the published results upwards when considering the percentiles of their distribution.R- 1 

### REFERENCES

Abbey, D. E., P.K. Mills, et al. (1991). Long-term ambient concentrations of total suspended particulates and oxidants as related to incidence of chronic disease in California 7th-Day-Adventists. Environmental Health Perspectives 94: 43-50.

Abbey, D.E., N. Nishino, et al. (1999). Long-term inhalable particles and other air pollutants related to mortality in nonsmokers. American Journal of Respiratory and
Critical Care Medicine 159(2): 373-382.

Avol, E.L., W.J. Gauderman, et al. (2001). Respiratory effects of relocating to areas of differing air pollution levels. American Journal of Respiratory and Critical Care
Medicine 164:2067-2072.

Brauer, M., J. Brumm, et al. (2002). Exposure misclassification and threshold concentrations in time series analyses of air pollution health effects. Risk Analysis 22(6): 1183-1193.

Brook, R.D., J.R. Brook, et al. (2002). Inha lation of fine particulate air pollution and ozone causes acute arterial vasoconstriction in healthy adults. Circulation 105(13):

1534-1536.

Cakmak, S., R.T. Burnett, et al. (1999). Me thods for detecting and estimating population threshold concentrations for air pollu tion-related mortality with exposure

measurement error. Risk Analysis 19(3): 487-496.

Chen, L.H., S.F. Knutsen, et al. (2005). The association between fatal coronary heart disease and ambient particulate air pollution: Are females at greater risk? Environmental Health Perspectives 113: 1723-1729.

Clancy, L., P. Goodman, et al. (2002). Effect of air-pollution control on death rates in Dublin, Ireland: an intervention study. Lancet 360(9341): 1210-4.

Clemen, R.T, R.L. Winkler (1985). Limits for the Precision and Value of Information from Dependent Sources. Operations Research 33(2): 427-442.

Cooke, R.M. (1991). Experts in uncertainty: opinion and subjective probability in science. (New York: Ox ford University Press).

Costa, D.L., U.P. Kodavanti (2003). Toxic responses of the lung to inhaled pollutants: benefits and limitations of lung-disease models. Toxicology Letters 140-141: 195-

203.

Creason, J., L.M. Neas, et al. (2001). Particulate matter and heart rate variability among elderly retirees: the Baltimore 1998 PM study. Journal of Exposure Analysis and Environmental Epidemiology 11(2): 116-122.

Devlin, R. B., A. J. Ghio, et al. (2003). Elderly humans exposed to concentrated air pollution particles have decreased heart rate variability. European Respiratory
Journal 21: 76S-80S.R-2 

### Diaz-Sanchez, D., A. Tsien, et al. (1999). Effect of topical fluticasone propionate on the mucosal allergic response induced by ragweed allergen and diesel exhaust particle challenge. Clinical Immunology 90(3): 313-22.

Dockery, D.W., C.A. Pope, et al. (1993). An association between air pollution and mortality in six U.S. cities. New England Journal of Medicine 329(24): 1753-1759.

Dockery, D.W., H. Luttmann-Gibson, et al. (2005). Association of air pollution with
increased incidence of ventricular tach yarrhythmias recorded by implanted
cardioverter defibrillators. Environmental Health Perspectives 113(6): 670-674.

Dominici, F., A. McDermott, et al. (2003). Airborne particulate matter and mortality:
timescale effects in four U.S. cities. American Journal of Epidemiology 157(12):

1055-1065.

Dominici, F., R.D. Peng, et al. (2006). Fine particulate air pollution and hospital admission for cardiovascular and respiratory diseases. Journal of the American
Medical Association 295(10): 1127-1134.

Enstrom, J.E. (2005) Fine particulate ai r pollution and total mortality among elderly Californians, 1973-2002. Inhalation Toxicology 17: 803-816.

Evelson, P., B. Gonzalez-Flecha (2000). Time course and quantitative analysis of the adaptive responses to 85% oxygen in the rat lung and heart. Biochimica et Biophysica Acta 1523(2-3): 209-216.

Gauderman, W.J., E. Avol, et al. (2004). The effect of air pollution on lung development from 10 to 18 years of age. New England Journal of Medicine 351(11): 1057-1067.

Ghio, A.J., C. Kim, et al. (2000). Concen trated ambient air particles induce mild pulmonary inflammation in healthy human volunteers. American Journal of Respiratory and Critical Care Medicine 162(3 Pt 1): 981-988.

Ghio, A.J. (2004). Biological effects of Utah Valley ambient air particles in humans: a
review. Journal of Aerosol Medicine 17(2): 157-164.

Gilmour, P.S., I. Rahman, et al. (2002). Histone acetylation regulates epithelial IL-8 release mediated by oxidative stress from environmental particles. American Journal of Physiology. Lung Cellular and Molecular Physiology 284(3): L533-40.

Godleski, J. J., R. L. Verrier, et al. (2000). Mechanisms of morbidity and mortality from exposure to ambient air particles. Research Report 91, Health Effects Institute, Boston, MA.

Gold, D.R., A. Litonjua, et al. (2000). Am bient pollution and heart rate variability.
Circulation 101(11): 1267-1273.

Goto, Y., J.C. Hogg, et al. (2004). Exposure to ambient particles accelerates monocyte
release from bone marrow in atherosclerotic rabbits. American journal of physiology. Lung cellular and molecular physiology 287(1): L79-85.R-3 

### Guigueira, S.A., J. Lawrence, et al. (2002). Rapid increases in the steady-state concentration of reactive oxygen species in the lungs and heart after particulate air pollution inhalation. Environmental Health Perspectives 110(8): 749-755.

Hawkins, N.C., J.D. Graham (1990). E xpert scientific judgment and cancer risk assessment: a pilot study of pharmacokinetic data. Risk Analysis 8: 615-625.

Hedley, A.J., C.M. Wong, et al. (2002). Cardiorespiratory and all-cause mortality after restrictions on sulphur content of fuel in Hong Kong: an intervention study. Lancet 360(9359): 787-788.

Hill, A.B. (1965). The Environment a nd Disease: Association or Causation? Proceedings of the Royal Society of Medicine 58: 295-300.

Holmstead, J. (2005). EPA, Office of Air and Radiation. Letter to Robert O’Keefe, Health Effects Institute. July 11.

Hora, S.C. (1992). Acquisition of expert judgment: Examples from risk assessment. Journal of Energy Engineering 118(2): 136-148.

Industrial Economics, Inc. (2004). An expertjudgment assessment of the concentrationresponse relationship between PM 2.5 

### exposure and mortality. Prepared for the Office of Air Quality Planning and Standa rds, Research Triangle Park, NC.

Industrial Economics, Inc. (2006). Briefing Book Binder. January.

Industrial Economics, Inc. (Under Developm ent). Technical Support Document for Expanded Expert Judgment Assessment Of The Concentration-Response Relationship Between PM 2.5 

### Exposure And Mortality: Expert Interview Summaries. Prepared for the U.S. EPA Office of Air Quality Planning and Standards.

Jerrett, M., R.T. Burnett, et al. (2005). Spatial analysis of air pollution and mortality in Los Angeles. Epidemiology 16(6): 1-10.

Kahneman, D., P. Slovic, et al. (1982) Judgment Under Uncertainty: Heuristics and Biases. Cambridge University Press, Cambridge, UK.

Krewski, D., R.T. Burnett, et al. (2000). Reanalysis of the Harvard Six Cities Study and the American Cancer Society Study of Particulate Air Pollution and Mortality. Special Report to the Health Effects Institute, Boston MA, July.

Kunzli, N., S. Medina, et al. (2001). Assessment of deaths attributable to air pollution:
should we use risk estimates based on time series or cohort studies? American Journal of Epidemiology 153(11): 1050-1055.

Kunzli, N., M. Jerrett, et al. (2005). Ambient air pollution and atherosclerosis in Los Angeles. Environmental Health Perspectives 113(2): 201-206.

Laden, F., J. Schwartz, et al. (2006). Reduc tion in Fine Particulate Air Pollution and Mortality: Extended Follow-up of the Harvard Six Cities Study. American Journal of Respiratory and Critical Care Medicine 173: 667-672.R-4 

### Lipfert, F. W., H. M. Perry, et al. (2000). The Washington University-EPRI veterans' cohort mortality study: Preliminary results. Inhalation Toxicology 12: 41-73.

Lipfert, F.W., H.M. Perry Jr., et al. (2003). Air pollution, blood pressure, and their longterm associations with mortality. Inhalation Toxicology 15(5): 493-512.

Lipfert, F.W., R.E. Wyzga, et al. (2006). Traffic density as a surrogate measure of environmental exposures in studies of air pollution and health effects: Long-term mortality in a cohort of U.S. veterans. Atmospheric Environment 40: 154-169.

Mallick, R., K. Fung, et al. (2002). Adjusting for measurement error in the Cox Proportional Hazards Regression Model. Journal of Cancer Epidemiology and

Prevention 7(4): 155-164.

McDonnell, W. F., N. Nishino-Ishikawa, et al. (2000). Relationships of mortality with the fine and coarse fractions of long-term ambient PM10 concentrations in nonsmokers. Journal of Exposure Analysis and Environmental Epidemiology 10(5): 427-436.

Morgan, M.G., S.C. Morris, et al. (1984). Technical uncertainty in quantitative policy analysis - a sulfur air pollution example. Risk Analysis 4: 201-216.

Morgan, M.G., M. Henrion, (1990). Uncertainty : a guide to dealing with uncertainty in quantitative risk and policy analysis. (N ew York: Cambridge University Press).

M. Granger Morgan, L.F. Pitelka, et al. (2001). Elicitation of Expert Judgments of Climate Change Impacts on Forest Ecosystems. Climate Change 49: 279-307.

Murphy, A.H, R.L. Winkler, (1992). Diagnostic verification of probability forecasts. International Journal of Forecasting 7(4): 435-455.

National Research Council (NRC). (2002). Estim ating the public health benefits of proposed air pollution regulations. The Nati onal Academies Press: Washington, D.C.

O’Keefe, R. (2005). Health Eff ects Institute. Letter to Jeff Holmstead, EPA Office of Air and Radiation. August 15.

O'Neill, M.S., A. Veves, et al. (2005). Diabetes enhances vulnerability to particulate air pollution-associated impairment in vascular reactivity and endothelial function.
Circulation 111:2913-2920.

Otway, H., D. von Winterfeldt (1992). Expert judgment in risk analysis and management: process, context, and pitfalls. Risk Analysis 12(1): 83-93.

Peters, A., E. Liu, et al. (2000). Air pollution and incidence of cardiac arrhythmia. Epidemiology 11(1): 11-7.

Peters, A., M. Frohlich, et al . (2001). Particulate air polluti on is associated with an acute phase response in men; results from the MONICA-Augsburg Study. European Heart
Journal 22(14): 1198-1204



## . Plopper, C.G., M.V. Fanucchi (2000). Do urban environmental pollutants exacerbate childhood lung diseases? Environmental Health Perspectives 108(6): A252-253.R-5 

### Pope, C.A. (1989). Respiratory disease associ ated with community air pollution and a steel mill, Utah Valley. American Journal of Public Health 79(5): 623-628.

Pope, C.A. (1991). Respiratory hospital admissions associated with PM10 pollution in Utah, Salt Lake, and Cache Valleys. Archives of Environmental Health 46(2): 90-97.

Pope, C. A., M. J. Thun, et al. (1995). Particulate air-pollution as a predictor of mortality in a prospective-study of U.S. adults. American Journal of Respiratory and Critical
Care Medicine 151(3): 669-674.

Pope, C.A. (1996). Particulate pollution and health: a review of the Utah valley experience. Journal of Exposure Analysis and Environmental Epidemiology 6(1):

23-34.

Pope, C.A., D.W. Dockery, et al. (1999). Oxygen saturation, pulse rate, and particulate air pollution: a daily time-series panel. American Journal of Respiratory and Critical
Care Medicine 159(2): 365-372.

Pope, C.A. (2000). Epidemiology of fine particulate air pollution and human health: biologic mechanisms and who's at risk? Environmental Health Perspectives 108(Suppl 4): 713-723.

Pope, C. A., R. T. Burnett, et al. (2002). Lung cancer, cardiopulmonary mortality, and long-term exposure to fine particulate air pollution. Journal of the American Medical

Association 287(9): 1132-1141.

Pope, C.A., R.T. Burnett, et al. (2004). Cardiovascular mortality and long-term exposure to particulate air pollution: epidemiological evidence of general pathophysiological pathways of disease. Circulation 109(1): 71-77.

Rhoden, C.R., J. Lawrence, et al. (2004). N-acetylcysteine prevents lung inflammation after short-term inhalation exposure to concentrated ambient particles. Toxicological

Sciences 79(2): 296-303.

Riediker, M., R.B. Devlin, et al. (2004). Ca rdiovascular effects in patrol officers are associated with fine particulate matter from brake wear and engine emissions. Particle and Fibre Technology 1(1): 2.

Saldiva, P.H., R.W. Clarke, et al. (2002). Lung inflammation induced by concentrated ambient air particles is rela ted to particle composition. American Journal of Respiratory and Critical Care Medicine 165(12): 1610-1617.

Samet, J. M., F. Dominici, et al. (2000a). The National Morbidity, Mortality, and Air Pollution Study Part I: Methods and Methodologic Issues. Research Report 94, Health Effects Institute, Boston, MA.

Samet, J. M., S. L. Zeger, et al. (2000b). The National Morbidity, Mortality, and Air Pollution Study Part II: Morbidity and Mortality from Air Pollution in the United States. Research Report 94, Health Effects Institute, Boston, MA.

Schwartz J. (1993) Particulate air pollu tion and chronic respiratory disease. Environmental
Research 62: 7-13.R-6 

### Schwartz, J., D.W. Dockery, et al. (2000). Harvesting and long term exposure effects in the relation between air pollution and mortality. American Journal of Epidemiology

151:440-448.

Schwartz J., S.K. Park, et al. (2005). GSTM1, Obesity, Statins, and autonomic effects of particles: gene by drug by environment interaction. American Journal of Respiratory
and Critical Care Medicine 172:1529-1533.

Seaton, A., A. Soutar, et al. (1999). Particulate air pollution and the blood. Thorax 54:

1027-1032.

Sun, Q., A. Wang, et al. (2005). Long-term air pollution exposure and acceleration of
atherosclerosis and vascular inflammation in an animal model. Journal of the
American Medical Association 294(23): 3003-3010.

Suwa, T., J.C. Hogg, et al. (2002). Particulate air pollution induces progression of
atherosclerosis. Journal of the American College of Cardiology 39: 935-942.

U.S. Environmental Protection Agency. (1997). The benefits and costs of the Clean Air Act, 1970 to 1990. Prepared for U.S. Congress by U.S. EPA, Office of Air and Radiation/Office of Policy Analysis and Review, Washington, DC

U.S. Environmental Protection Agency. (1999). Benefits and costs of the Clean Air Act: 1990-2010, EPA Report to Congress. U.S. EPA, Office of Air and Radiation and Office of Policy. Washington, DC. Document No. EPA-410-R-99-001. November.

U.S. Environmental Protection Agency. (2000). Regulatory impact analysis for the heavy-duty standards/diesel fuel rulemaking. Prepared by: Innovative Strategies and Economics Group, Office of Air Quality Planning and Standards, Research Triangle Park, NC December.

U.S. Environmental Protection Agency, Science Advisory Board. (2004). Advisory on plans for health effects analysis in the analytical plan for EPA’s second prospective analysis – benefits and costs of the Clean Air Act, 1990-2020. Advisory by the
Health Effects Subcommittee of the A dvisory Council on Clean Air Compliance Analysis. EPA-SAB-COUNCIL-ADV-04-002.

U.S. Environmental Protection Agency. (2004). Final regulatory impact analysis: control of emissions from non-road diesel engines

. Office of Transportation and Air Quality.

EPA-420-R-04-007.

U.S. Environmental Protection Agency. (2005a) Regulatory impact analysis for the final
Clean Air Interstate Rule. Office of Air and Radiation. EPA-452/R-05-002.

U.S. Environmental Protection Agency. (2005b). Regulatory impact analysis for the final Clean Air Visibility Rule or the Guidelin es for Best Available Retrofit Technology (BART) Determinations under the Regional Haze Regulations. Office of Air and
Radiation. EPA-452/R-05-004.

Walker, K.D., P. Catalano, et al. (2003). U se of expert judgment in exposure assessment: part 2. Calibration of expert judgments about personal exposures to benzene. J Expo Anal Environ Epidemiol. 13(1): 1-16.R-7 

### Walker, K.D. (2004). Industrial Economics, Inc. Memorandum to Jim Neumann, Henry Roman, and Tyra Gettleman, Industrial Ec onomics, Inc. Appropriate Number of Experts for the Particulate Matter Expert Judgment Project. November 11.

Wellenius, G.A., B.A. Coull, et al. (2003) I nhalation of concentrated ambient air particles exacerbates myocardial ischemia in conscious dogs. Environmental Health Perspectives 111(4): 402-408.

Willis, A., M. Jerrett, et al. (2003). The asso ciation between sulfate air pollution and mortality at the county scale: an exploration of the impact of scale on a long-term exposure study. Journal of Toxicology and Environmental Health, Part A 66(16-19):

1605-1624.

Winkler, R.L., R.M. Poses (1993). Evaluating and combining physicians' probabilities of
survival in an intensive care unit. Management Science 39(12): 1526-1543.

Winkler, R. L., T.S. Wallsten (1995). An assessment of the risk of chronic lung injury attributable to long-term ozone exposure. Operations Research 43(1): 19-28.

Wolff, S.K., N.C. Hawkins, et al. (1990). Selecting experimental data for use in quantitative risk assessment: an expert judgment approach. Toxicology and
Industrial Health 6:275-291.

Yamawaki H., N. Iwai (2006). Mechanisms underlying nano-sized air-pollution-mediated progression of atherosclerosis: carbon black causes cytotoxic injury/inflammation and inhibits cell growth in vascular endothelial cells. Circulation Journal 70(1):

129-140.

Zeger, S.L., F. Dominici, et al. (1999). Harvesting-resistant estimates of air pollution effects on mortality. Epidemiology 10(2): 171-175.

Zeka, A., A. Zanobetti, et al. (2005). Short te rm effects of particulate matter on cause specific mortality: effects of lags and modification by city characteristics. Occupational and Environmental Medicine 62(10): 718-725.

Zelikoff J.T., L.C. Chen, et al. (2003) Effects of inhaled ambient particulate matter on pulmonary antimicrobial immune defense. Inhalation Toxicology 15(2): 131-150.