UNITED ST ATES E N VI RON M E N T A L PR OTE C TION AGE NCY WASHING T O N D. C. 20460

OFFI CE OF THE A D M I N ISTR A T OR SCI E NCE A D VI SORY BO A RD

July 11, 2008

EPA-COUNCIL-08-002

The Honorable Stephen L. Johnson Adm i nistra tor U.S. Environm ental Protection Agency 1200 Pennsylvania Avenue, N.W

. Washington, D.C. 20460

Subject: Characterizing Uncertain ty in Particulate Matter Be nef its Using Expert Elicitation

Dear Adm i nistrator Johnson:

Prior to issuing the 2006 National A m bi ent Air Quality Standard (NAAQS) for particula t e m a tter (PM 2. 5 ), EPA’s Office of Air Quality Planning and Standards (OAQPS) within th e Of f i ce of Air and Radiatio n comp leted a m u lti-year ef f

o rt to ch arac ter i ze th e estim ated benefits of reduced prem ature m o rtalities associated with exposures to PM 2. 5

. EPA used expert elicitation to quantitativel y assess the relationship between exposures to
PM 2. 5 and th e incid e nce of m o rtality, thus comp lem e nting and expanding the epidem iological liter a tur e on this sub j ect and incorporating probabilistic uncertainty analysis. Comp leted in 2006, the PM 2. 5 -Morta lity Expert E l icita tion add r essed the concentration-respons e function between PM 2. 5 and m o rtality and provided probabilistic characterizations of uncertainty fr om 12 independent experts. The PM 2. 5 - M orta lity Expert Elicitation received very favorable peer revi ews (RTI I n terna tiona l, Peer Review of Expert E l icitation , Sep t em ber 2006) and becam e the basis for assessing monetized benefits of the 2006 PM 2. 5 NAAQS. EPA then asked the Science Advisory Board (SAB) Staff Office to convene an expert panel to review the application of the PM 2. 5 -Morta lity Expert Elicitation results to the ben e f its asse ssm ent f o r PM 2. 5 . In particular, EPA asked for guidance on the interpretation of expert elicitation results and presentation in the Executive S u mm ary, Press Releas e and Benefits Analysis chapter of EPA’s Regulatory
Impact Analysis associated with the 2006 PM 2. 5 standard.

To conduct this review, the Advisory Council on Clean Air C o mp liance Analysis (Council) was augm ented with noted experts in the health effects of air pollution and expert elicitation (see enclosed roster). The Council and i nvited experts met on May 8, 2008, to discuss charge ques tions from EPA. The Counc il’s detailed advice and

recomm endations are p r ovided in th e enclos ed Advisory with highlights below.The Council endorses E P A’s application of the expert elicita tion results. The Council finds that EPA accurate ly characterized each expe rt’s concentration-response function and expressed the uncertainty surround ing these functions in a technically sound

m a nner. The probability distributions propaga ted for the expe rts’ c oncentration-response functions were appropriately constructed and applied to estim ate benefits. EPA thoroughly captured and expressed the bread th and diversity of opinion among experts and clearly differentiated between estim ates based on em pirical data (i.e., individual epidem iological studies) and those based on expert judgm ents (that are inform ed by epidem iological studies). The Council was asked whether EPA’s benefits assessm ent responded to the National Research Council (N RC) recomm endation to “m ove the

assessm ent of uncertain ties from its ancill ary an alyses into the prim ary analysis by conducting probabilis tic, m u ltiple-so u r ce uncertainty analyses.” (NRC, Estimating th e Health-Risk-Reduction B e nefits of Proposed Air Pollution Regulations , 2002). Our answer is yes

.

The Council was asked whether the Agen cy should m ove toward presenting a central estim a te with uncertain ty bounds or continue to provide separate estim a tes for each expert. The Council believes th e answer to this question depends on the contex t o f the expert elicitation and its results. On issues where experts have a wide range of opinions, it is im portant to provide separate estimates for ea ch expert (or c l uste r of experts sharing sim ilar views), thus emphasi zing the unce r ta inty assoc i ated with the issue. But where experts largely ag ree, it would be appropriate to collap se the variou s estim ates in to a single distribu tion (o r point estimate with un c e rta i nty bou nds) while s t ill providing the individual estim ates elsewhere, perhaps in an A ppendix or website. In future analy s es, the decision about aggregat ion must be m a de in the context of each analysis and its purpose.

On the critical side, the Council believe s there is room for improvem e nt in conveying the differences in assump tions (i ncluding the influence of key em pirical studies) that drive the dif f

erences am ong expe rts’ concentration-response functions. It would be useful to know why the experts agr ee on som e things and disagree on others. The benefits chapter could be improved by devoting less space to the experts’ quantitative judgm ents in exchange for m o re discussion to characteri ze their rationales. The text cou l d better elu c idate the relative importance of various sources of uncertainty: both those that were quantified and those that were not quantified. Th ese issues cou l d be addressed in the chapter and brought forward into the Executive Summ ary.

The Council has concerns about the Execu tive S u mm ary and Press Release. The
PM 2. 5 -Mortality Expert E licitation sh owed a stron g cons ensus am ong scientists, however the Executiv e Summ ary and Press R e leas e failed to show this central m a ss of expert opinion. Instead, the Press Release presente d the tails of the distribution, showing a range of $8 to $76 billion dollars in net benefi ts. Presented w ith th is range, the casual reade rcould easily inf e r substantia l d i f f

e rences in scien tif ic o p inion when , in f act, the r e was a pronounced central cluster of views on PM 2.5 m o rtality. To communicate with a wider audience, the Executive Sum m a ry and Pr e ss Release should have clearly stated that

iiscien tif ic dif f

erences ex isted only with respe c t to the m a gnitude of the effect of PM 2. 5 on mortality, n o t whether s u ch an effect ex isted. T h e Executive Summ ary would have benefited from a short description of the PM 2. 5 -Mortality Expert Elicitation and the rationale for its use in the context of the PM 2. 5 regulatory process. Additional efforts are needed to identify the m o st effective m ean s of communicating both m e thods and results to different kinds of readers. This is of particu l a rim portance f o r the Exe c utive Sum m a ry and Press R e lease, which are m u ch more likely to be read in their entirety. The Council suggests that alternative and less complex graphics would provide m u ch more useful inf o rm ation than the tables tha t are in cluded in th e Executiv e Summ ary.

Detailed recomm endations are in clud ed in the enclosed Advisory. On behalf of the entire C ouncil, we appreciate this opportun ity to provide tim ely advice to the Agency. We hope these comm ent s are helpful to EPA as it proceeds with this important work.

S i n c e r e l y ,

/Signed/

Jam e s K. Hammitt, Chair Advisory Council on Clean Air Compliance A n a l y s i s

Enclosures

iii

### U.S. Environmental Protection Agency Advisory Council on Clean Air Co mpliance Analysis Augmented for Ben efits of Reduced PM-Mort a lit y using Expert Elicitation

CHAIR

Dr. James K. Hammitt , Professor, Center for Risk Analysis, Harvard University, Boston, MA.

COUNCIL MEMBERS

Dr. David T. Allen , Gertz Regents Professor in Che m ical Engineering, Departm e nt of Che m ical Engineering, University of Texas, Austin, TX.

Dr. Dallas Burtraw , Senior Fellow

, Resources for the Future, W a shington, DC.

Dr. Shelby Ger k ing , Professor of Econom ics, Departm e nt of Econom ics, College of Business Adm i nistration, University of Central Florida, Orlando, FL.

Dr. Wayne Gray , Professor, Departm e nt of Econom i cs, Clark University, W o rcester, MA.

Dr. F. Reed Johnson , Senior Fellow and Principal Econom ist, RTI Health Solutions, Research Triangle Institute, NC.

Dr. Katherine Kie l , Associate Professor, Departm e nt of Econom i cs, College of the Holy Cross, One College Street , W o rcester, MA.

Dr. Virginia McConnell , Senior Fellow and Professor of Econom i cs, Resources for the Future, W a s h ington, DC.

Dr. David Popp , Associate Professor of Public Adm i nistration, Center for Policy Research, T h e Maxwell School, Syracuse University, Syracuse, NY.

Dr. Chris Walcek , Senior Research Scientist, At m o spheric Sciences Research Center, State University of New York, Albany, NY.

HEALTH EFFECTS SUBCOM MITTEE MEMBERS

Mr. J. Fintan Hurley , Scientif ic Directo r , Ins t itute of Occupational Medicine (IOM),

ivEdinburgh, Scotland, U n ited Kingdom

.

Dr. Michael T. Kleinman , Professor, Departm e nt of Community & Environm ental Medicine, University of California, Irvine, CA.

Dr. Rebecca Parkin , Professor of Environm ental and Occup a tional Health and Asso ciate Dean for Research and Public Health Pract ice, School of Public Health and Health Services, George W a shington Univers ity Medical Center, W a shington, D C

.

INVITE D EXPE RTS

Dr. Aaron Cohen , Principal Sc ientist, Health Ef f ects Institute, Charlestown Navy Ya rd, Boston, MA.

Dr. John Evans , Senior Lecturer on Environm e n tal Sc ience, Harvard Univers ity, Ku wait Public Health Project, Portsm outh, NH.

Dr. H. Christopher Frey , Professor, Civil, Cons truction and Environm ental Engineering, College of Engineering, No rth Carolina State U n iversity, Raleigh, NC.

Dr. Ronald Wyz g a , Technical Executive, Air Quality Health and Risk, Electric Po wer Research In stitu te, Pa lo Alto, CA.

SCIENCE ADVISORY BOARD STAFF

Dr. Holly S t allw orth, Designated Federal Officer, Scien ce Advisory Board Staff Office, Environm ental Protection Agency, Washington, DC.

vNOTICE

This report has been written as part of the activ ities of the U.S. Environmental P r otection Agency’s Advisory Council on Clean Air Comp liance Analysis (Council), a federal advisory co mmittee ad m i nistrativ ely located un der the EPA Science Ad visory Board (SAB) Staff Office. The Council is chartered to provide extramural scientific inform ation and advice to the Adm i nistrator and other offici als of the EPA. The Council is structured to provide balanced, expert assessm ent of scie ntif ic m a tters r e lated to is su e and proble m s facing the A g ency. This report h a s n o t been reviewed for approval by the Agency and, hence, the contents of th is repo rt do not n ecess a rily repres ent the views and policies o f the EPA, nor of other agencies in the Execu tive Branch of the Federal governm e nt, nor does m e ntion of trade nam e s or comm ercial products constitute a recommendation for use. Council reports are pos ted on the SAB W e b site at: http://www.epa.gov/sab

.

vi

### Advisory Council on Clean Air Compli ance Anal ysis Advisory on Characteri z i ng Uncertai nty in Particulate Matter Benefits Using Expert Elicitation

1. In the PM N AAQS benefits chap ter, has EPA accurately characterized ea ch expert’s

concentra tio n-response function as expresse d in the PM-Mor t ality Expert Elicitation report and conveyed th e differences in assu mptio ns (includin g the in fluen ce of key empirical studies) that dr ive the differences among the concen tration

- resp onse functions?

In the benefits chapter (Chapt er 5) of the particulate m a tter (P M) National Am bient Air Quality Standard (NAAQS) regulatory impact analysis (RIA), EPA has accurately described the experts’ concentration-respons e (C-R) functions in general term s and has clearly sum m a rized the im plications of each expert’s C-R function for th e expected reduction in fatalities (and their m one tary va luation) in Figs. 5-10 – 5-13. The benefits chapter does not report each expert’s C-R f unction nor describe the factors (such as differences in assump tions and reliance on par ticular studies) that drive the differences

am ong the C-R functions in the chapter. Extens ive description of the individual experts’ C-R functions as well as their perspectives, ra tio nales, and re lianc e on empirical studies in form ulating their judgm e nts are describe d in th e orig inal re ports of the expert elicitation study, including The Expanded Expert Judgment Assessment of the Concentration-Response Relationship between P M 2. 5 Exposure and Mortality (Industrial Econom ics, 2006) and its technical support docum ents. The Council believes that it would be useful if Chapter 5 provided som e discussion of the prim ary studies on which the experts relied and of the factors that drive dif f

erences am ong their responses, though we are sensitive to concerns that the regul atory analysis should not be overly long and

com p lex.

2. In applying the PM-Mor t ality Expert Elicitati on r e sults in EP A’s benefit a nalysis, is our mathematical treatment of concepts such as the probability of causality, thresholds, and shape of the function t echnically sound, as well as transparent?

The m a thematical treatment of concepts such as the probability of causality, thresholds, and shape of the function is technically sound and transparen t. For each ex pert, EPA

com b ined the expert’s quantitative assessm ent of his beliefs about annual average PM 2. 5 and m o rtality hazard into an uncondition a l dist ribution of that expert’s v i ews of how mortality hazards change per uni t ch ange in annu al average P M 2. 5 , at different baseline levels of that annual average in the range 4 – 30 μg/m 3 . For e xperts who expressed a nonzero prob ability of a th re shold, EPA m a de appropria te assum p tions abou t how that probability was distribu ted with in ranges of annual average P M 2. 5 concentrations.

Som e Council m e m b ers expressed c oncern abou t how the de rived uncon dition a l distribution for each exp e rt is used to produ ce es tim a tes of morta lity impacts. The PM NAAQS be nefits chapter uses th e C-R functions to estim a te annual “attributable deaths,”

1using an apparently simp le static m e thodology. It is increasingly recogn ized (i) that there are difficulties underlying this concept, incl uding that the estim a ted annual deaths do not reproduce year-on-year; and (ii) that these difficulties can be overcom e by use of life tables, which also allow benefits to be expr essed as gains in life expectancy (see, e.g., Rabl 2003, 2006). The Council encourages EP A to identify this conceptual issue in the use of both the expert elicitation and the c ohort study results for benefits analysis. W e believe both types of C-R functions could be us ed in life-table calcu lation s. (Indeed, s u ch calculations were apparently reported in an Appendix to the RIA, though not in Chapter 5). (W e note that EPA suggests that m easures of the gain in life expectancy m a y provide a “theo r etically p r eferred” m e thod to valu e chan ges in m o rtality risk (p. 5-56), but it does not discuss the assump tions used to es tim ate attribu t able deaths o ra ttemp t to quantif y the uncertainties in im pacts a nd valuation estim ates th at resu lt. )

3. Do the tables, text, conclusions, and E xecutive S u mmary adequately d i stinguish the benefit estim ates based on data-derived components of the un certainty assessment from those based on expert judgment? How should the mortality estimates based on the elicitation be compared to those deri ved from the empir i ca l studies of the PMmortality association?

The tables, text, and conclusions of Chapter 5 c l early d i sting u ish the b e n e f it estim a tes based on direct application of C-R functions from epidem io logical studies (so-called “data-derived com ponents”) from those based on the C-R functions elicited from the experts (that are, of course, inform ed by epid em iological studies and other data). Overall, the Council agrees that separately identifyi ng the estim ates based on their sources, and reporting es tim a tes base d on m u ltiple relevant ep idem iologic a l stud ies an d on each expert’s C-R function, is a useful an d appropriate m e thod for accurately p o rtray i ng th e uncertainty about the effects of PM 2. 5 on m o rtality (whether reported as num b er of

prem ature deaths averted or as a m o netary value).

It should be noted that the estim a tes derive d from prim ary epidem iological studies and f r om expert elic ita tion ar e not f u lly c o m p arable. The epidem iologica l stu d ies c ited a r e cohort studies used to estim ate the longer-ter m influences of mort ality; the expert elic ita tion a ddressed total m o rtality changes th at could be as socia t ed “with a redu ctio n in annual average PM2.5 including both changes in short–term (e.g., 24 hour) and long-term exposures to PM2.5.”

In addition the two epidem iological studies cited are am ong several that were considered by the experts; it would be useful to presen t the range of estim ates from several of the other epidem iological studies (see E xhibit 3-3 in the Report) considered to see wheth e r a

more comp r e hensive co nsiderati on of these studies yields as much variation in results as the expert opinions. This could provide gr eater insight into the variation in expert elic ita tion re sults.

The Council believes that graphical representa tions, such as the box-and-whiskers plots in Figs. 5-12 and 5-13 in Chapter 5, provide a clear and concise m e t hod to represent this

2inform ation. These figures convey infor m ation about the likelihood of different ranges of values as p r edicted from each C-R function (not sim p ly a single rang e) an d about the degree of clustering and overlap among the diffe rent C-R functions (i.e., f r om individual epidem iological studies and expert’s judgm ents ). The distribution f unctions presented in Figs. 5-14 and 5-15 provide slightly m o re inform ation, but most Council m e m b ers fi nd them less inform ative, perhaps becau s e the f unctions tend to s t ack on top of each other. The Council was enthusiastic about the gr aphic shown below that shows cum u lative distributions of benefits calculated using two epidem iological studies and selec t ed fractiles of each expert’s C-R function, all clearly disti nguished by using distinctive

sym bols. This graphic was presented to th e Council but not included in the RIA benef its chapter itself. In contrast, the tables included in the chapter and Executiv e Summ ary perm it only an im poverished representation of the degrees of certainty and uncertainty.

Graphic take n fr om “Char a cteriz i ng the Uncer ta in ty in Estim a te d B e nefits o f Red u ced P M

Mor t ality Usi n g E x per t Elicitati o n, ” pres entati on b y Li sa Conner, Br yan H ubbell and H a r v ey Richm o nd, Office of Air Qualit y Plan ning an d S t and a rd s, Advis o r y Council on Clean Air Com p liance Analysis meetin g, May 8, 2008.

Recognizing that standard box-and-w h iskers plots such as Figs. 5-12 and 5-13 are probably m o re com p lex than appropriate for the E x ecutive Summary, the Council suggests that alternative and less complex graphics, such as the figure above, would still provide m u ch m o re useful inform ation than th e tables that are includ ed in the Executive Summ ary. Another option is a sim p lified box-an d-whiskers plot including only the m ean

or m e dian and the 5th and 95th percentile va lu es for each epidem iological study and each expert.

3In attemp ting to summ ar ize the rich infor m ation about uncertainty it has developed, EPA evidently had difficulty choosing term inol ogy. Table 5-1 and Table ES-3 present the

sam e m a terial, but the concepts that ar e labeled “Lower Bound Expert Result” and “Upper Bound Expert R e sult” in Table 5-1 ar e labeled “Low Mean” and “High Mean” in Table ES-3. These concepts are not defined or ex plained in either location

. Moreover, the ranges for th ese concep ts are not properly described. In Ta ble ES-3, they are not even labeled. In T a ble 5-1, they are im properly labe led as “confiden ce intervals” rather than “cred ibility intervals.” T h e concepts of conf idence and cred ib ility in tervals are distinct and have dif f

erent interpretations. A 90 percen t c onf idence in terva l is a statis tic (i.e., a random variable) constructed from data using a procedure such that the probability that the interval includes the true value is 90 percent (conditional on the m ode l assum p tions). A 90 percen t cred ibility inte rval is a n inte rval ch osen by an e xpert who b e liev e s the r e is a 90 percent chance that the true value is in the interval (c onditional on whatever

assum p tions he m a y spe c ify).

4. Does the EPA’s present effort to incorpor ate uncertainty analyses and discussions into th e primary analysis, as exemp lified in the PM NAAQS RIA chapter, adequately address the NRC’s request to move the assessm ent of uncertainties into its primary analyses? If not, what more could the EPA do to satisfy this request?

The short answer to the f i rst question is “yes.” To understand why this is so, it is important to reflect on the way uncertainty wa s being addressed by the EPA at the time of 2002 NRC r e port. At that tim e benefits esti m a tes for particulate air pollution would typically p r esent a bas e -case analysis which relied on the Pope et al. Am erican Cancer Society (ACS) study, a sensitivity analysis which provided upper estim a tes of effect drawing on the Dockery et al. S i x Cities st udy, and lower estim ates from t h e tim e-series literature, an d these would be accom p anied by a q u alitative discussion of the issues rela ted to d r awing causa l inf e renc es f r om this literatu re, esp e cially the co hort stud ies.

The NRC was unsatisfied with this form of presentation because it left unresolved the important question of determ ining how m u ch we i ght to assign to thes e various alternative estim ates. If the cohort studies did not reflec t causation, the effect estim ates would need to be based on the results of tim e series stud ies w h ich are roughly a factor of 10 s m aller than the AC S Study. And if they did reflect causal associations, then the relative plausibility of the coefficients from t h e Si x Cities Study (that were about 3 tim es larger than thos e o f the Am erican Cancer S o ciety S t udy ) was left un specified.

The NRC committee was of the view that m o st users of EPA regulat ory analyses (i.e., regulators, Congressm e n, the general public) w e re not in as good a position to evaluate these questions as scientific experts in ep idem iology and toxicology would be and so it

recomm ended that EPA explore the possibili ty of eliciting scientific opinion, using for m al m e thods for probabilistic expert judgm ent, as a m eans of addressing this concern.

EPA’s current effort reflects a carefu l atte m p t to do just that. EPA’s analysis of uncertainty (in dose-resp onse coefficients for PM 2. 5 ) is integ r a tive, quan t itative, and

4central. The analys is is integra tive in the sense th at it deals with all sources of uncertainty – both aleatory and epistem i c – affe c ting PM dose-response functions. Aleatory uncertainty is the inherent variation asso ciated with the physical system or the environm ent, som e ti m e s referred to as stochastic uncertainty or irreduc ible uncertainty. Epistem i c uncertainty stem s from a lack of knowledge of quantities or processes of the system or the environm ent, som e times referr ed to as m odel uncertainty or reducible uncertainty. In this way it differs from m e ta -analysis, which would be valuable if the only questions were about the m a gnitude of th ese relationships and not about the strength of

evidence f o r causal in terpr e ta tion of the epide m iologic stu d ies. The an alysis is quantitative in the sense that it provides probabilistic statem ents about the relative plausibility of alternative interpretations of the eviden ce – both about the relative strengths of various studies and also about the likelihood that these study results are artifacts of confounding or have little biologi cal s upport. In addition, b e cause the analysis presents sep a rately th e q u antitative interpr e ta tio ns of 12 experts, it provides the user with a sense of the extent of scientific consensus am ong these exp e rts. The an alysis is cen t ral to the EPA’s docum ent in that these probabi lis tic charac ter i za tions of unc erta inty a r e presented in the body of the RIA and not rele gated to technical appendices or supporting docum ents.

While the C ouncil com m e nds EPA for this work that clearly responds to the NRC

recomm endations, we b e liev e there are wa ys in which future efforts could be strengthened. The Council unde rstands that there are lim itations to any approach, including form al elicitation of expert judgm en t, to quantitatively characterize the nature and strength of scie ntific understanding of quantities (such as concentration-response slopes) relevant for environm enta l decision m a king. Am ong t h ese are:

• Selection of experts – The first questi on in any effort to interpret am biguous or conf lic tin g scien tif ic inf o rm ation is to determ ine which scientis ts to consult. The EPA’s analysis relies on the views of 12 experts in epidem i o logy and toxicology. The fact that the EPA is open and transparent about who these experts are and how they were selected invites qu estions about whether the group was representative, whether the samp le was a probability sam p le, whether th e group was balanced (with regard to d i scip line, ins titu tional affiliation, o rother facto r s), and so o n. These are certainly importan t questions. B u t it is necessary to recognize that any effort to resolve questions about the extent of epistem i c uncertainty (which often is th e dom i nant source of

uncertain ty) m u st rely on the in terp reta tions of scien tis ts an d theref ore involve these sam e issues of whic h scientists, how chosen, whether representative, how balan ced, and so on. Thus, the question could have been asked of previous EPA regulatory analys es that a l so re lie d on professional judgm ent, but without the transparency of a for m al expert elicitation.

• Aggregation of expert opinion – Th e EPA has chosen to first present separately th e views of each of the 1 2 scien tis ts who particip ated in their expert elicitation. This is en tirely consistent with “b est practices” in the field. But because of concerns about the scien tific legitim acy of any approach fo r

5aggregating expert opinion, the EPA has sa id that it declined to aggregate and it does no t p r esent any aggregate estim ate of the central tendency of expert opinions. H o wever, EPA does present a range bounded by the m ean estim ates of the experts with the lowest and hi ghest m ean estim ates. The Council notes that this is, in fact, a form of aggreg ation that assigns positive weight to the

most extrem e judgm ents and zero weight to all the others (or perhaps suggests a uniform di stribution between these extrem e values). The Council feels this is not the best aggregation and recommends that the EPA consider other form s of weighting

. e.g., assign ing equal wei ght to each expert' s distribution o r assigning w e ights based on other approach es, such as peer weighting, self weighting, or perform a nce on calibra tion questions. See the Council’s discussion in Question 6b.

• Lim its of rationality – All judgm ents are subject to well-know n cognitive

anom alies such as sensitivity to fram ing, anchoring, probability weighting, etc. Even trained experts such as scien tists and physicians are not imm une to such ef f

ects. To protec t a g ainst these dif f i cultie s, this well-des igned exper t elic ita tion in cludes ch ec ks on consistency and logic a nd provide experts with an opportunity to reconsider and revise their evaluations.

• Costs – The EPA did not provide the C ouncil with estim ates of the costs of conducting this expert elicitation, but th ere is a sense that the costs were “high” (without giving a m a gnitude or any real compar ison). This is a clear case, however, where the be nefits of the expert e licitation for understanding and resolv in g the la rge d i f f

e rences in estim ated PM regulatory benefits were even larger. There seem s to be a consensus that the study provided benefits well beyond its costs for analysis of the effects of the PM regulation. However, it m a y be that sim ilar expe rt elicitation efforts would not be appropriate for all RIAs. Estim ates of the cost of this study, and any lessons learned about ways to reduce costs of fu ture expert e lic it ation studies, would be useful for future regulatory analysis.

The Council recomm ends that EPA develop cr iteria for determ ining whe n system atic polling of scientific judgm e nt s woul d enhance the regulatory analysis, usefully inform decision m a king, and justify the associated anal ytical costs. The Council understands that the Agency is developing guidance on the us e of expert judgment and encourages EPA to consider this topic if it is not already doing so.

The prim ary area in which the EPA’s effort ha s not been responsive to the NRC report is in its explanation of the rationale behi nd the experts’ judgm ents. Understanding why experts disagree about the im plications of th e availab l e ev idence m a y be as important as their specific judgm ents of the probability distribution for the PM 2. 5 C-R function itself. Much inform ation of this type was develope d and reported in the final report (IEc, 2006) and other docum e nts describing the expert j udgment study, but it is not evident how, if at all, this inform ation was reflected in the PM NAAQS benefits analysis.

6Expert judg m e nts involve intu itiv e weighti ng and interpretation of exis ting evidence. Thus in some ways expert e lic ita tion is sim ilar to m e ta-ana lysis, which also serves to integrate and synthesize a body of evidence. Identifying the specific data, decision weights, ad justm e nts, and interpretations each ex pert em ployed could help inform discussions am ong a wider group of experts and stakeholders , foster explicit evaluation of the m e rits of experts’ s ubjective criteria, and suggest opportunities for aggregating or clustering judgm e nts across experts.

Som e Council m e m b ers believe it w ould be usef ul to analyze how, if at all, experts’ judgm ents are correlated with factors such as: field of expertise (epidem i ologists vs. toxico logis t s / clinic ians) ; aut horship of prim ary epidem iol ogical studies (do authors put

more empha sis on their own work? ) ; and in stitution where the expert resides (the elic ita tion in cludes three e xperts from one institution and two from another).

The Council recomm ends that EPA evaluate the qualitative inf o rm ation collected during the elicitation, the post- elicitation workshop, and ot her interactions with the experts. T h is analysis could help decision m a kers understa nd the divergence of e xpert opinion, identify particular gaps or deficien cies in the evidence that experts be lieve con t rib u te to uncertainty, and identify fruitful aven ues for future research.

In summ ary, EPA’s use of expert elicitation s a tis f ies the NRC’s reque s t and represen ts a state-of-the-art example of e xpert elicitation m e thods. The be nefits chapter serves as an excellent proof of concept for quantifying uncer tainty in regu lato ry analy s is. In this particula r in stance, the r e sults serve to increase d ecis i on m a kers’ and the public’s confidence that the health benefits of PM 2. 5 controls exceed costs by a comfortable

m a rgin. This is largely b ecause, as a group, th e experts hav e great confidence that the epidem iological studies upon which the EPA has relied reflect causal relationships between exp o sure and morta lity (i.e., th e expe rts place little weight on n on-causa l interpretations) and because they emphasi ze the relevance and va lidity of the cohort studies for answering the questions of interes t to the EPA.

Despite our strong support of this analysis, the C ouncil urges EPA to anticipate challenges to expert elicitation when it is used in more contro versia l app l ication s . It is reasonab l e to expect that EPA will be requi red to def e nd the process us ed f o r expert selection. B u t as noted above, th is challenge should apply to any effort to use expert opinion, whether through for m al elicitation or inform al consultation, in support of regulation.

5. Has the EPA adequately communicated the unce rtainty information associated with the PM premature mortality estimate to the audiences that the RIA addresses, including: scientists, policy analysts, decision makers, and the public?

Not yet. The Council appreciates that the Execu tive Summ ary, and especially the benefits chapte r, pres ent the qu an titativ e resu lts in detail using diverse ta bular and especially graphical approaches. H o wever, we raise ge neral concerns related to: 1) m e thods of

7presentation m o st appro p riate for th e RI A’s diverse readership; and 2) the proper

m e tric(s) f o r

charac ter i z i ng the resu lts of the el icitation with regard to the distribution of the experts’ subjective probabi lities and their effect on the health impact and economic valuation estim a tes.

The Council stres s es th at address i ng these c oncerns will hav e important benefits for all for m s of communication about the expert elicitation includ ing the RIA, the Executive Summ ary, and the Press Release.

a. Considering the examples provided by the EPA, are there other methods the EPA should use, instead of or in addition to those employed, to summarize and communicate th e resu lts of the P M-Mor t ality Expert Elicitation in the benefits chapter and the Executive Summary for communication to technical and non-technical audiences?

Yes. The Council notes that the intended readership of the RIA is diverse, and we appreciate that EPA explored a ra nge of approaches to presenting the results of th e elicitation. Nonetheless, we believe that additional efforts are needed to identify the m o st effectiv e m eans of communicating both m e thods and results to different kinds of readers. This is of particular im portance for the Executiv e Summ ary and Press R e leas e, whic h are m u ch more likely to be read in their entirety by most r eaders

. Specific su ggestions in clude:

• Provide in the Executiv e Summ ary a description of the elicitation with rega rd to the ra tiona le f o r its use, what it comprised, and how it was conducted. Figure 5-1 coul d be useful in this regard.

• Make m o re extensive use of gra phical displays in the Executive Summ ary, rather than (o r in add ition to) tables.

• Add som e indication of the “botto m line” in the P r ess Rele ase

. For example, language could be added to state that disagreem e nts am ong experts are limited to the magnitude of health b e nef its as sociate d with PM 2. 5 reductions, not whether those benefits exist. See also the Council’s comm ents on reflecting central tendency in Questions 5b, 5c and 6b.

b. To what extent do the typ e s of sta t em ents made in the Execu tive Summary of the PM N AAQS RIA s u ccessfully comm unicate the extent of uncertainty (and/or the certainty) in the estimate of PM prem ature mortality to those who are not familiar w i th the PM- M o rtality Exp e rt Elicitatio n?

As discussed above (in response to Ques tion 3), the Council questions the use of ranges to characterize the uncertainty in impact and valuation estim ates in the Executive Summ ary and its tables. Pa nelists note that the range appears to imply that any value within it enjoys an equal degree of support from the

8experts while m o re detailed descriptions of the results show that the experts’ judgm ents are m o re clustered. Som e indication of the clustering in the elic ita tion re sults shou ld be conside r e d (see also Question 5c, point 1 below).

The Council notes two sources of uncertain ty not explicitly a ddressed in either the chap ter or the Execu tive Summ ary:

• The m e thods and criteria used to select th e experts are n e ither presented in detail nor critically ev alu a ted. Pane lis ts note that this stage of the elic ita tion p r ocess is crit ical to ensuring that the p a nel of experts adequately represents expert opinion about the effect of PM on mortality. (As noted in Question 4, expe rt s e le ction is a l so c r itical to alternative approaches such as consensus panels, but m a y be perceived

as m o re salient for expert elicit ation, perhaps because individual expert’s distributions are reported.)

• As noted above (Question 2), esti mation from cohort study data of annual numbers of attributable d eaths, as opposed to m easures of longevity (e.g., years of life lost), is problem atic.

c. Are there additional sum m ary statem ents that are important to deduce from the res u lts of the P M NAAQS benefits chap ter to th e Exe c utiv e Summary?

Yes. Panelis ts noted that the chap ter lacks :

• A com p rehensive statem ent of the “b ottom line” with rega rd to the expert elicitation results, e.g., that it supports the conclusion that the benefits of PM 2. 5 control are very likely to be substantial; and

• An integrated discussion of the re lative im portance of various sources of uncertainty: those that were not qu antif ied (e.g. , rela tive tox i city of PM sources /constituents) versus tho s e that were, as well as th e relativ e importance of the various uncertain ties that were quantified (e.g., uncertainties in the C-R function vs. the valuation).

Table 5-5 identifies seven pr im ary sources of uncertain ty that are in cluded in the RIA, however this recognition o f m u ltiple u n certainties does not perm eate the rest of the chapter. It would be usef ul to acknowledge uncertainties at each stage of the analytic pro cess and thu s where the range of possible values incre a ses. It m i ght be helpf u l to have a char t tha t outlin es eac h step, ass e s s es the degree o f uncertain ty at that ste p, and reports how it is handled. This would allow the reader to understand how the final range of num b ers reported depends on the various steps in the an alysis, and to m o re easily see which uncertainties contribute most to the overall uncertainty. This inform ation should also be summ ari zed the Executive Summary.

96. Has the EPA adequately summarized the resu lts of the PM-Mortality Expert Elicitation a c ross the experts in the PM NAAQS RIA benefits chapter and executive summary?

The results are pres ented in summ ary for m in terms of m ean values and 90 percent confidence intervals. (As noted in response to Question 3, the latter should be referred to as 90 percent credibility intervals since they represent a judgm e nt about uncertainty and not an inference from the samp l i ng distribution of a statistic.)

The Executive Summ ary is adequate in te rm s of c onveying the central tendency and range of estim ates based on the experts’ C- R functions. However, it could be m a de clearer that the results sh ow n are not the actual ju dgm e nt s of the experts – i.e., the experts did not m a ke judgm e nts regarding the avoided pr em ature m o rtality. Rather, the avoid e d prem ature morta lity was estim at ed us ing the C-R f unction elic ited f r om each expert. Sim ilarly, th e benefits assessm ent should not be attributed to the experts, but it should be m a de clear that the expe rt judgment was simp ly one of many inputs to the benefits assessm ent. This could also be m a de clearer in m a ny of the tables (e.g., Table 5-32).

What is not apparent is w hy the expert judgm ents differ. For exam ple, Figure 5-10 illus t r a tes su bstantial inter-exp ert v a riab ility in r e sults. Ye t, the signif i c a n ce of this variability to the conclusions of the benefits assessm ent seems not to be addressed. Moreover, it appears that th e experts could be grouped in to clusters, such as a low cluster (Experts G, K), central cluster, a nd high cluster (Experts A, E, and perhaps B and C). It w ould be useful to know more a bout why the experts agree within clusters, and why they disagree between clusters. For example, do experts within a cluster tend to rely m o re heavily on a part icular study than do experts in other clusters? If so, why do m e m b ers of different clusters put more weight on different studies? Are there comments f r om the post- elic ita tion in terv ie ws that shed light on why the experts continue to disagree even after seein g each other’s judgm ents? Fo r example, do the experts d i f f

e r with re spe c t to which data se ts th e y deem to be m o st repre s enta tive o r useful or regarding inference m e thods (e.g., biological plausibili ty, statistical power of e m pirically-based m odels)?

In the PM N AAQS benefits chap ter, the EPA presents th e mortality results based on each of the twelve individual expert’s responses along with results based on concentration-response functions derived fr om empirical studies. The EP A has also considered employing methods to aggregat e results based on the elicitation into a single combined estimate. In particular, the EPA considered calculating a simple average of estimates across experts after the concentration-response functions of each expert had been applied in the benefits model (i.e., th e average of the resulting estimation of the change in mortality inci dence) . Other options for summarizing the results include: a weighted average of th e resulting change in incidence, a trimmed

10means approach, and a fitted distribution to th e overall set of concentration-response functions.

a. Should the E PA continue to present the re sults of the individual experts in future benefits analyses as was done in the PM N AAQS RIA? Should the EPA develo p metrics th at aggregate ac ross the individua l experts? If aggregate m e asures are considered appropriate, should the EPA present these in addition to or instead of the individual estimates?

The Council recomm ends that EPA con tinue to present the results of the individual experts in future benefits analyses, whethe r or not an aggregate or

com b ined distribution is presented.

Results f r o man expert elic ita tion c a n be used for several purposes, and whether agg r egation or com b ination acr oss experts is useful depends on the intended purpose. If the goal is to characterize the uncer tainty about an

outcom e (e.g., the benef its of controlling PM 2. 5 ), a presen tatio n that shows the distr i bution of estim ated benef its con d ition a l on e ach exper t ’s C-R f unction (such as Figs. 5-10 – 5-11) is extrem ely useful, as it shows the im plications of each expert’s uncertain ty and the diversity of ju dgm e nts among experts. However, such a rich presentation of uncertainty is excessive for m a ny purposes, an d will in evitably be co lla psed by some sort of agg r egation

m e thod, whether by EP A or by others (e.g., new s m e dia).

The Council considered a num ber of a pproaches to aggregation but judges that none are ideal inasmuch as the appropriate for m of aggr egation m a y depend on the purposes of the elicita tion and its r e sults. Fo r a ccura tely portray ing the range of expert op inion, reportin g results using each exp e rt’s judgm ents m a y be most useful. For eval uating alternative re gulatory options, a decision-analytic perspective suggests it m a y be most useful to com b ine the experts’ distributions into a single distribution, though the best m e thod for doing so is unclear. W h e n the result s suggest that m a ny of the experts’ judgm ents fall into one or a few cluste rs, it m a y be importa nt to identify and describe those clusters and to separa tely describe any significant outliers. Alternatively, if the experts’ judgm ents are approxim a tely uniform ly distributed across a rang e, a statem ent of this result m a y be most useful.

b. If a combination (aggregation) of resu lts is considered appropriate, what technique for aggregation would you recommend?

The basic goal is to explain both th e range and distribution of judgm e nts within it. There does not seem to be a pe rfect or single best technique, as this will depend on the purpo se of the agg r egation and the data. A virtu e of f

o rm al decision analysis is that it provides a rigorous an d theore tic ally justif ied

m e thod for m a the m atically com b ining inform ation about uncertainty and

11preferences. Alternative approaches that rely on a decision m a ker to holistica lly weigh m u ltiple f actor s in his or he r head are sus c eptib le to cognitive lim itations such as the heuris tics and biases identified by Tversky and Kahneman (1974) and the tendency to overw eight those factors that appear especially salient while neglecting others.

EPA could consider developing an oper ational approach to describing the distribution of experts’ judgm e nts, while acknowledgi ng that there is no single best approach and that the choice of approach is a m a tter of judgm e nt and context. The Council considered severa l examples of possible approaches:

• Approach 1 : Present a range of experts’ m e dian or m e an values. The range could be defined by fr actiles (e.g., the interquartile range) o ras a trimm e d estim ate by excluding judgm e nts of the p highest and p lowest values. This has the advantage of being easy to explain and does not str ongly imply a probabilistic interpretation, which is appropria te s i nce the exp e rts are not a probability s a mp le. A lim itation is that it does n o t take into account the range of uncertainty elicited from each expert o r inf o rm ation f r om the experts with the highest and lowest central values.

• Approach 2 : Sim ilar to Approach 1 but instead of using a range of central values, use a range enclosed by ranking the 5th

percen tiles o f the experts with the lo west such values and the 95th percentiles of the experts wi th the highest such values, and reporting som e summar y of thes e ranked extreme values (e.g., the m ean or m e dian 5th percentile; the sm allest 5th percentile). This approach incorporates som e inform ation about the variation in both the locations and widths of the experts’ distributions. It

em phasizes the range of opini on but provides no inform ation about any clustering within it.

• Approach 3 : If ther e are m u ltiple clu s ters, EPA c ould descr i b e each clu s ter, perhaps us ing Approach 1 or 2. The identification of clusters m a y take into account qualitative inform ation regard ing sim ilarities in the bas i s of the judgm ents of m u ltiple experts in a cluster; it requires judgm e nt.

• Approach 4 : A com b ine d distribution can be produced by aggregating the probab i lity associ ated with each value across experts. This is equivalent to estim ating a distribution by Monte Carlo sim u lation in which experts are sam p led (with equal or unequal probability) and values draw n from the selected expert’s distribution. Unlike the previously described approaches, this

m e thod has the advantage of using all of the inform ation provided by all of the experts, is reasonably easy to explain, and has been shown to perform reasonably well in characterizing uncertainty (Clem e n and W i nkler, 1999). However, this m e thod

12has the disadvantage of suggesting that the experts constitute a probability s a mp le of relevant opin i o n. If expert weights ar e used, these can be based on se veral sources, e.g., ranking of experts by each other or expe rts’ perform a nce in providing probability distributions in re spons e to calibratio n questions. (Calibration questions ask about variables whose value is unknown to the experts at the tim e of elicitation but known to the analys t at th e tim e of combini ng the experts’ distributions; Cooke, 1991). Unequal weighting of experts’ judgm e nts can lead to superior perform a nce if the weights (or calibration questions on which th ey are based ) are well selected, but is often viewed skeptically b ecause it ap pears to g i ve inadequate respect to experts whose judgm ents are give n little weight and/or m a y discourage experts from partic ipating in a tim e -consum ing elicitation process if their judgm en ts m a y ulti m a tely get little weight. Unequal weighting m e t hods are m o re complicated to conduct and explain than som e of the other approaches.

As is eviden t from these exam ple appr oaches, none is perfect. Thus, EPA m a y wish to consider selectin g a relatively simp le approach that conveys the range and central tendency, while acknowledging that a variety of approaches are possible, explaining the advantages and disadvantages of candidate approaches, and explaining why the sele cted approach was chosen. It is possible to u s e m u ltiple approaches in com b ination (e.g., Approaches 1 an d 4), as the situation warrants.

c. If a combined estimate is consider ed appropriate, what interpretation should be applied to the percentiles of the uncerta inty distribution derived from the elicitation (e.g., the mean es timate of a combined elicitation function, or the 5 th -95 th percentiles)?

The appropriate interpretation depend s on the combination method, and is perhaps best characterized by explicit description of the m e thod (e.g., for Approach 1 above the inte rquartile range of the e xperts’ m ean estim a tes). Because it is typically no t appropria te to characterize th e experts as a probability s a mp le of som e relevant population o f expert opin i on, it do es not

seem appropriate to ch aracterize a co mbined distribution as a probabilistic distribution of expert opinion.

d. If a combined distribution is not appropriate, how should the EPA characterize the estimates of the P M premature mortality effect? One option employed in th e Executive S u mma ry of the PM NAAQS RIA is to present the estimates as a range from the average value associated with the steepest concentra tion-response function to the average value associated with the flattest concentr ation-response function. Is this the best approach? What other op tions would you recommend?

13As described above, characterizing effect s as a simple range provides a very lim ited summ ary of the rich inf o rm a tion about uncertainty provided by the expert elicitation. The C ouncil finds that graphical disp lays (such as Figs 5-10 – 5-11 and a graphic like that illu str a te d in respon se to Question 3) tha t portray the u n certainty about prem atur e m o rtality conditional on each ex pert’s judgm ent, and the variability in estim a tes a m ong experts, provide a

com p rehensive summ ary. In attem p ting to com p ress this inf o r m ation into a sim p ler f o rm at, the Council sugg es ts tha t the ap propria te su mmary will depend on the data and encourages EPA to atte mpt to chara c ter i ze the e x tent to which experts’ judgm ents are co ngruent and overlapping or broadly distributed across an overall range.

14References

Clem en, R.T., and W i nkler, R.L., “Com bining Pr obability Distributions from Expe rts in Risk Analysis,” Risk Analysis , 19: 187-203, 1999.

Cooke, R.M., Experts in Uncertainty: Opinion and Subjective Probability in Science

, New York: Oxford Uni v ersity Press, 1991.

Industrial Econom ics, Inc., The Expanded Expert Judgment Assessment of the Concentration-Response Relationship between P M 2. 5 Exposure and Mortality, Final Report prep ared for EPA Office of Air Quality Planning and Standards. Available at http://www.epa.gov/ttn/ecas/regda ta/Uncertainty/pm

_ ee_report.pdf , 2006.

National Research Council, Estimating The Public Health Benefits Of Proposed Air Pollution Regulations , Washington, D.C.: National Academ ies Press, 2002.
Rabl, A., “Analysis of Air Pollution Mortalit y in Term s of Life Expectancy Changes: Relation between Tim e Series, In tervention, and Cohort Studies,” Env i ro nmental Health: A Global Access Science Source 5: 1-11, 2006.
Rabl, A., “Interpretation of Air Pollution Morta lity: Num b er of Deaths or Years of Life Lost? ” Journal of the Air and Wast e Management Association 53: 41-50, 2003.
Tversky, A., and Kahnem a n, D., “Judgm e nt Unde r Uncertainty: Heuristics and Biases,” Science 185(4157): 1124-1131, 1974.

15