Document ID: EPA-HQ-OAR-2002-0076-0371
Agency: epa
Document Type: Supporting & Related Material
Title: 
Posted Date: 2005-06-13T04:00Z

1.2.3
Uncertainty
in
the
Benefits
Estimates
Characterization
of
health­
related
benefits
associated
with
PM
reductions
is
a
complex
process
which
is
subject
to
a
variety
of
potential
sources
of
uncertainty.
Key
assumptions
underlying
the
estimate
of
avoided
premature
mortality
include
the
following:

 
Inhalation
of
fine
particles
is
causally
associated
with
premature
death
at
concentrations
near
those
experienced
by
most
Americans
on
a
daily
basis.
Although
biological
mechanisms
for
this
effect
have
not
yet
been
established,
the
weight
of
the
available
epidemiological
and
experimental
evidence
supports
an
assumption
of
causality.

°
All
fine
particles,
regardless
of
their
chemical
composition,
are
equally
potent
in
causing
premature
mortality.
This
is
an
important
assumption,
because
PM
produced
via
transported
precursors
emitted
from
EGUs
may
differ
significantly
from
direct
PM
released
from
automotive
engines
and
other
industrial
sources.
However,
no
clear
scientific
grounds
exist
for
supporting
differential
effects
estimates
by
particle
type.

°
The
C­
R
function
for
fine
particles
is
approximately
linear
within
the
range
of
ambient
concentrations
under
consideration.
Thus,
the
estimates
include
health
benefits
from
reducing
fine
particles
in
areas
with
varied
concentrations
of
PM
including
both
regions
that
are
in
attainment
with
the
fine
particle
standards
and
those
that
do
not
meet
the
standard.

°
The
forecasts
for
future
emissions
and
associated
air
quality
modeling
are
valid.
Although
recognizing
the
difficulties,
assumptions,
and
inherent
uncertainties
in
the
overall
enterprise,
these
analyses
are
based
on
peer­
reviewed
scientific
literature
and
up­
to­
date
assessment
tools,
and
we
believe
the
results
are
highly
useful
in
assessing
this
rule.

Probabilistic
Approaches
As
part
of
an
overall
program
to
improve
the
Agency's
characterization
of
uncertainties
in
health
benefits
analyses,
we
have
developed
probabilistic
approaches
to
characterize
uncertainty.
These
methods
are
fully
described
in
the
benefits
analysis
associated
with
CAIR
RIA
(
EPA,
2005).
Specifically,
one
approach
generates
a
distribution
of
benefits
based
on
the
classical
statistical
error
expressed
in
the
underlying
health
effects
and
economic
valuation
studies
used
in
the
benefits
modeling
framework.
This
approach
captures
only
a
limited
portion
of
the
uncertainty
about
the
parameters
(
i.
e.,
that
based
on
statistical
error
and
cross­
study
variability).
As
such
we
generate
a
second
estimate
that
relies
on
the
results
of
our
recent
pilot
expert
elicitation
project
to
characterize
other
key
aspects
of
uncertainty
in
the
ambient
PM2.5/
mortality
relationship.
By
substituting
the
probability
distributions
specified
by
our
experts
for
that
associated
with
the
statistical
error
associated
with
the
Pope
et.
al.
(
2002)
study
usually
used
to
quantify
the
impact
of
PM2.5
on
premature
morality,
we
attempt
to
capture
other
sources
of
uncertainty,
particularly
model
specification
and
aspects
of
the
health
science
not
captured
in
the
studies.
Both
approaches
provide
insights
into
the
likelihood
of
different
outcomes
and
about
the
state
of
knowledge
regarding
the
benefits
estimates.
EPA
is
continuing
its
research
of
methods
to
characterize
uncertainty
in
total
benefits
estimates,
and
is
conducting
a
full­
scale
expert
elicitation.
The
full­
scale
expert
elicitation
is
scheduled
to
be
completed
by
the
end
of
2005.

Although
schedule
and
resource
limitations
prevented
us
from
completing
a
full
uncertainty
analysis
using
either
method,
we
present
a
partial
analysis
that
focuses
on
characterizing
the
uncertainty
in
the
premature
mortality
estimate.
Since
premature
morality
accounts
for
between
85­
95
percent
of
the
benefits
estimated
in
most
of
EPA's
recent
air
pollution
rulemakings,
these
partial
analyses
provide
an
indication
of
the
uncertainty
inherent
in
the
overall
estimates
in
the
current
rulemaking.

Our
primary
estimate
based
on
the
Pope
et
al.
(
2002)
study
for
CAVR
shows
that
the
average
number
of
premature
deaths
avoided
in
2015
is
XXX.
This
is
higher
than
the
estimate
based
on
four
of
the
experts
and
lower
than
one
expert
who
participated
in
the
pilot
expert
elicitation,
and
falls
within
the
uncertainty
bounds
of
all
but
one
expert.
The
average
annual
number
of
premature
deaths
avoided
in
2015
ranges
from
approximately
(
based
on
the
judgments
of
Expert
C)
to
(
based
on
the
judgments
of
Expert
E).
The
medians
span
zero
to
with
the
zero
value
due
to
the
high
threshold
associated
with
one
of
the
expert's
distributions.
The
statistical
uncertainty
bounds
of
all
of
the
estimates,
including
the
Pope
et
al.­
based
distribution,
overlap.
Although
the
uncertainty
bounds
for
each
expert
include
zero,
and
some
distributions
have
significant
percentiles
at
zero,
all
of
the
distributions
have
a
positive
mean
estimate.
[
fix
this
paragraph
based
on
the
numbers
Bryan
said
he
could
run
from
his
spread
sheet]

To
put
these
morality
estimates
in
a
monetary
context,
consider
that
the
confidence
intervals
from
the
pilot
elicitation
applied
to
the
CAIR
benefit
analysis
ranged
in
value
from
zero
at
the
5th
percentile
to
a
value
at
the
95th
percentile
that
is
seven
times
higher
than
the
Pope
et
al.,
2002­
based
estimate.
Note
however,
that
the
results
are
highly
dependent
on
the
air
quality
scenarios
applied
to
the
concentration­
response.
Thus,
the
characterization
of
uncertainty
discussed
in
the
CAIR
RIA
could
differ
greatly
from
what
would
be
observed
for
CAVR
due
to
differences
in
population­
weighted
changes
in
concentrations
of
PM2.5
(
i.
e.,
the
location
of
populations
exposure
relative
to
the
changes
in
air
quality),
and
may
be
especially
sensitive
to
the
differences
in
baseline
PM2.5
air
quality
experienced
by
populations
prior
to
implementation
of
the
CAVR.

Supplemental
Analyses
In
addition,
as
part
of
the
current
analysis
we
conducted
a
variety
of
supplemental
analyses
designed
to
provide
the
reader
with
an
understanding
of
the
degree
of
uncertainty
that
may
be
associated
with
the
benefits
resulting
from
implementation
of
this
regulation.
Because
estimates
of
premature
mortality
contribute
the
most
to
the
monetized
benefits,
our
efforts
focused
on
the
sensitivity
of
the
final
benefits
estimate
to
analytic
judgments
regarding
this
relationship.
Specifically,
we
conducted
analyses
designed
to
characterize
the
degree
of
uncertainty
in
the
slope
(
magnitude)
of
the
PM2.5
concentration­
response
function,
the
form
of
the
PM2.5
concentration­
response
function
(
i.
e.,
the
potential
for
a
threshold),
and
the
cessation
lag
(
i.
e.,
temporal
relationship
between
cessation
of
exposure
and
reduction
in
adverse
health
effects).
Both
discrete
and
probabilistic
approaches
were
used
to
characterize
the
uncertainty
associated
with
the
concentration
response
function.

These
supplemental
analyses
yield
the
following
insights:

 
Substitution
of
the
steeper
concentration
response
function
for
PM2.5
 
premature
mortality
from
the
Six
Cities
study
increases
the
value
of
the
total
benefits
from
$
billion
to
$
billion
in
2015.

 
Substitution
of
plausible
alternative
lag
structures
has
little
overall
impact
on
the
estimate
of
total
benefits
(
reductions
are
on
the
order
of
5
to
15
percent).

 
The
assessment
of
alternative
assumptions
regarding
the
existence
(
and
level)
of
a
threshold
in
the
PM2.5
premature
mortality
concentration
response
function
highlights
the
sensitivity
of
the
analysis
to
this
assumption.
Only
percent
of
the
estimated
premature
mortality
is
due
to
changes
in
exposure
above
15mg/
m3,
while
over
percent
of
the
premature
morality
related
benefits
are
due
to
changes
in
PM2.5
concentrations
occurring
above
10ug/
m3.

4.3
Probabilistic
Analysis
of
Uncertainty
in
the
Benefits
Estimates
Methods
The
recent
NRC
report
on
estimating
public
health
benefits
of
air
pollution
regulations
recommended
that
EPA
begin
to
move
the
assessment
of
uncertainties
from
its
ancillary
analyses
into
its
primary
analyses
by
conducting
probabilistic,
multiplesource
uncertainty
analyses
(
NRC,
2002).
The
probability
distributions
required
for
these
analyses
should
be
based
on
available
data
and
expert
judgment.
The
NRC
also
recommended
that
EPA
use
both
internal
and
external
experts
as
needed,
in
each
case
identifying
those
experts
whose
judgments
are
used
and
the
rationales
and
empirical
bases
for
their
judgments.

As
part
of
an
overall
program
to
improve
the
Agency's
characterization
of
uncertainties
in
health
benefits
analyses,
this
section
describes
EPA's
current
approaches
to
probabilistically
characterize
uncertainties
associated
with
the
PM
mortality
C­
R
relationship
and
valuation.
Specifically,
we
conducted
two
different
Monte
Carlo
analyses,
one
based
on
the
distribution
of
reductions
in
premature
mortality
characterized
by
the
mean
effect
estimate
and
standard
error
from
the
Pope
et
al.
(
2002)
study
(
our
primary
estimate),
and
one
based
on
the
results
from
a
pilot
expert
elicitation
project
(
IEc,
2004).
The
Pope
et
al.
study
is
described
earlier
in
this
chapter.
The
pilot
expert
elicitation
project
is
summarized
below.

The
first
approach
generates
a
distribution
of
benefits
based
on
the
classical
statistical
error
expressed
in
the
underlying
health
effects
and
economic
valuation
studies
used
in
the
benefits
modeling
framework.
This
approach
captures
only
a
limited
portion
of
the
uncertainty
about
the
parameters
(
i.
e.,
that
based
on
statistical
error
and
cross­
study
variability).
As
such
we
explored
using
the
results
our
recent
pilot
expert
elicitation
project
to
characterize
other
key
aspects
of
uncertainty
in
the
ambient
PM2.5/
mortality
relationship.
By
substituting
the
probability
distributions
specified
by
our
experts
for
that
associated
with
the
statistical
error
associated
with
the
Pope
et.
al.
(
2002)
study
usually
used
to
quantify
the
impact
of
PM2.5
on
premature
morality,
we
attempt
to
capture
other
sources
of
uncertainty,
particularly
model
specification
and
aspects
of
the
health
science
not
captured
in
the
studies.
The
details
associated
with
both
Monte
Carlo
analyses,
are
described
in
Appendix
B
of
Chapter
4
of
the
CAIR
RIA
(
EPA
2005).

For
both
approaches,
the
distributions
of
all
other,
nonmortality
health
endpoints
are
characterized
by
the
reported
mean
and
standard
deviations
from
the
epidemiology
literature.
We
are
unable
at
this
time
to
characterize
the
uncertainty
in
the
estimate
of
benefits
of
improvements
in
visibility
at
Class
I
areas.
As
such,
the
visibility
benefits
are
treated
as
fixed
and
add
them
to
all
percentiles
of
the
health
benefits
distribution.

These
analyses
provide
likelihood
distributions
both
for
the
total
dollar
benefits
estimate
and
for
the
incidence
of
premature
mortality
to
show
the
uncertainty
described
by
each
expert's
judgment
relative
to
the
range
of
uncertainty
associated
with
the
standard
error
in
the
Pope
et
al.
(
2002)
study.
The
uncertainty
about
the
total
dollar
benefit
associated
with
any
single
endpoint
combines
the
uncertainties
from
two
sources
 
the
C­
R
relationship
and
the
valuation
 
and
is
estimated
with
a
Monte
Carlo
method.
1
Our
estimates
of
the
likelihood
distributions
for
total
benefits
should
be
viewed
within
the
context
of
the
wide
range
of
sources
of
uncertainty
that
we
have
not
incorporated,
including
uncertainty
in
emissions,
air
quality,
and
baseline
health
effect
incidence
rates.

Both
approaches
provide
insights
into
the
likelihood
of
different
outcomes
and
about
the
state
of
knowledge
regarding
the
benefits
estimates.
The
uncertainty
estimates
based
on
statistical
error
have
the
strength
of
presenting
a
statistical
measure
of
the
uncertainty
in
the
underlying
studies
serving
as
the
basis
for
the
estimates
used
in
the
analysis.
However,
this
approach
captures
only
a
limited
portion
of
the
uncertainty
about
1
In
each
iteration
of
the
Monte
Carlo
procedure,
a
value
is
randomly
drawn
from
the
incidence
distribution,
and
a
value
is
randomly
drawn
from
the
unit
dollar
value
distribution.
The
total
dollar
benefit
for
that
iteration
is
the
product
of
the
two.
If
this
is
repeated
for
many
(
e.
g.,
thousands
of)
iterations,
the
distribution
of
total
dollar
benefits
associated
with
the
endpoint
is
generated.
For
details
on
the
specific
Monte
Carlo
approach
we
used,
see
Appendix
B.
the
parameters.
The
5th
and
95th
percentile
points
of
the
distributions
are
based
on
statistical
error
and
cross­
study
variability
and
provide
some
insight
into
how
uncertain
our
estimate
is
with
regard
to
those
sources
of
uncertainty.
However,
it
does
not
capture
other
sources
of
uncertainty
regarding
the
model
specification
and
other
inputs
to
the
model,
including
emissions,
air
quality,
and
aspects
of
the
health
science
not
captured
in
the
studies,
such
as
the
likelihood
that
PM
is
causally
related
to
premature
mortality
and
other
serious
health
effects.

The
approach
that
uses
the
pilot
expert
elicitation
results
is
included
primarily
as
an
illustration
of
the
impacts
of
using
probabilistic
(
expert
elicitation
and
statistical
errorbased
distributions
for
premature
mortality
associated
with
PM2.5
compared
with
EPA's
traditional
approach.
EPA
is
continuing
its
research
of
methods
to
characterize
uncertainty
in
total
benefits
estimates,
and
is
conducting
a
full­
scale
expert
elicitation.
The
full­
scale
expert
elicitation
is
scheduled
to
be
completed
by
the
end
of
2005.

Although
schedule
and
resource
limitations
prevented
us
from
completing
a
full
uncertainty
analysis
using
either
method,
we
present
a
partial
analysis
that
focuses
on
characterizing
the
uncertainty
in
the
premature
mortality
estimate.
Since
premature
morality
accounts
for
between
85
and
95
percent
of
the
benefits
estimated
in
most
of
EPA's
recent
air
pollution
rulemakings,
these
partial
analyses
provide
an
indication
of
the
uncertainty
inherent
in
the
overall
estimates
in
the
current
rulemaking.

Results
Our
primary
estimate
based
on
the
Pope
et
al.
(
2002)
study
for
CAVR
shows
that
the
average
number
of
premature
deaths
avoided
in
2015
is
XXX.
This
is
higher
than
the
estimate
based
on
four
of
the
experts
and
lower
than
one
expert
who
participated
in
the
pilot
expert
elicitation,
and
falls
within
the
uncertainty
bounds
of
all
but
one
expert.
The
average
annual
number
of
premature
deaths
avoided
in
2015
ranges
from
approximately
(
based
on
the
judgments
of
Expert
C)
to
(
based
on
the
judgments
of
Expert
E).
The
medians
span
zero
to
with
the
zero
value
due
to
the
high
threshold
associated
with
one
of
the
expert's
distributions.
The
statistical
uncertainty
bounds
of
all
of
the
estimates,
including
the
Pope
et
al.­
based
distribution,
overlap.
Although
the
uncertainty
bounds
for
each
expert
include
zero,
and
some
distributions
have
significant
percentiles
at
zero,
all
of
the
distributions
have
a
positive
mean
estimate.
[
fix
this
paragraph
based
on
the
numbers
Bryan
said
he
could
run
from
his
spread
sheet]
(
and
as
much
more
as
you
can
do )

To
put
these
morality
estimates
in
a
monetary
context,
consider
that
the
confidence
intervals
from
the
pilot
elicitation
applied
to
the
CAIR
benefit
analysis
ranged
in
value
from
zero
at
the
5th
percentile
to
a
value
at
the
95th
percentile
that
is
seven
times
higher
than
the
Pope
et
al.,
2002­
based
estimate.
Note
however,
that
the
results
are
highly
dependent
on
the
air
quality
scenarios
applied
to
the
concentration­
response.
Thus,
the
characterization
of
uncertainty
discussed
in
the
CAIR
RIA
could
differ
greatly
from
what
would
be
observed
for
CAVR
due
to
differences
in
population­
weighted
changes
in
concentrations
of
PM2.5
(
i.
e.,
the
location
of
populations
exposure
relative
to
the
changes
in
air
quality),
and
may
be
especially
sensitive
to
the
differences
in
baseline
PM2.5
air
quality
experienced
by
populations
prior
to
implementation
of
the
CAVR.

Summary
of
Pilot
Expert
Elicitation
Project
As
a
first
step
in
addressing
the
NRC
recommendations
regarding
expert
elicitation,
EPA,
in
collaboration
with
OMB,
conducted
a
pilot
expert
elicitation
to
characterize
uncertainties
in
the
relationship
between
ambient
PM2.5
and
mortality.
This
pilot
was
designed
to
provide
EPA
with
an
opportunity
to
improve
its
understanding
of
the
design
and
application
of
expert
elicitation
methods
to
economic
benefits
analysis,
to
lay
the
groundwork
for
a
more
comprehensive
elicitation,
and
to
provide
more
information
about
the
uncertainty
in
the
PM2.5­
mortality
relationship
in
the
context
of
the
Nonroad
Diesel
RIA
and
similar
analyses
conducted
in
the
near
term.

The
pilot
project
elicited
the
judgments
of
five
experts
in
the
PM
health
sciences,
all
members
of
at
least
one
of
two
recent
National
Academy
of
Sciences
scientific
committees
focused
on
particulate
matter.
The
specific
process
used
to
select
experts
is
summarized
in
Appendix
B
and
detailed
in
the
technical
report
describing
the
elicitation
(
IEc,
2004)
along
with
additional
information
about
the
experts'
affiliations
and
fields
of
expertise.
The
responses
of
each
expert
to
questions
enumerated
in
the
elicitation
protocol
provide
the
inputs
for
developing
distributions
of
mortality
benefits.

The
elicitation
approach
tested
in
the
pilot
was
peer
reviewed
by
four
experts
(
Mansfield,
2004)
and
generally
received
favorable
comments,
including
 
the
pilot
project
followed
"
best
practices"
for
expert
elicitation
and
was
well
documented,
 
the
elicitation
was
well
conducted
and
is
an
appropriate
technique
for
characterizing
uncertainty,
 
experts
chosen
are
well
known
and
respected
and
represent
a
range
of
views,
and
 
the
expert
selection
process/
protocol
was
very
good.

The
peer­
review
report
also
provides
recommendations
on
how
to
improve
the
process
for
a
more
comprehensive
expert
elicitation
addressing
the
PM
mortality
C­
R
relationship.
Specifically,
the
peer­
review
recommendations
included
 
holding
a
pre­
elicitation
workshop
to
ensure
that
all
experts
are
properly
motivated
and
conditioned
(
i.
e.,
on
the
same
footing)
before
the
interviews
and
to
allow
for
information
sharing
prior
to
the
elicitation;
 
allowing
for
some
form
of
communication
following
the
individual
interviews
to
allow
review
of
the
experts'
responses
and
allow
them
to
adjust
their
estimates
if
necessary
 
one
way
to
accomplish
this
is
through
a
post­
elicitation
workshop;
and
 
changing
the
encoding
process
to
ensure
that
extreme
values
(
upper
and
lower
ranges)
are
collected
prior
to
judgments
on
central
tendency.
This
sequencing
would
avoid
anchoring
or
adjustment
heuristics
associated
with
biased
estimates
of
uncertainty.
In
addition,
although
not
listed
explicitly
as
a
criticism,
some
of
the
peer
reviewers
noted
the
small
number
of
experts
participating
in
the
pilot
and
suggested
a
larger
number
of
experts
should
be
used
in
EPA's
next
elicitation.

The
peer
reviewers
also
offered
varying
comments
on
the
methods
for
combining
the
results
of
the
pilot
elicitation.
Several
of
the
reviewers
preferred
that
the
expert
opinions
not
be
combined
or
stated
that
they
knew
of
no
agreed­
upon
method
for
combining
results
from
expert
elicitations.
They
stated
that
presenting
each
expert's
response
independently
allows
for
differences
in
the
individual
distributions
to
be
recognized.
Two
of
the
reviewers
indicated
that
they
were
reasonably
comfortable
with
the
method
used
in
this
study
to
combine
the
results,
while
the
other
two
reviewers
offered
comments
on
the
combined
result
of
the
elicitation.
One
reviewer
stated
that
the
combined
distributions
do
not
adequately
capture
the
opinions
of
individual
experts
but
rather
average
them
out.
He
states
that
it
is
possible
in
such
cases
that
the
combined
judgments
may
generate
results
that
none
of
the
experts
would
agree
on.
Another
reviewer
stated
that
expert
elicitation
studies
typically
do
not
combine
judgments,
but
if
one
were
to
combine
them,
he
recommended
that
the
response
of
each
be
maintained
independently
from
the
other
experts
and
run
through
the
benefits
model
completely
prior
to
combining
the
results.
For
more
details
regarding
the
peer­
review
comments,
see
Appendix
B
of
Chapter
4
of
the
CAIR
RIA
(
EPA,
2005).
The
full
peerreview
report
is
also
available
at
www.
epa.
gov/
ttn/
ecas/
benefits.
html.