Uncertainty Analysis in Ecological Risk Assessments of Pesticides

Dwayne R.J. Moore – Intrinsik Environmental Sciences Inc., New Gloucester, ME

April 22nd, 2014
Why Uncertainty Analysis?

• Explicitly acknowledges 

uncertainty that we know exists

• Comprehensive characterization 

of distribution risk

• Not conservatively biased

• Better understanding of system, 

available data, and importance 

of data gaps

• Has been recommended for 

refined risk assessments by 

numerous expert panels 

including NAS Panel
Types of Uncertainty: Variability

• Arises from natural stochasticity

• Parameter variability due to

– Spatial distribution of pesticide concentration 

– Temporal fluctuations in rainfall

– Genetic differences among individuals

• Not reducible by empirical effort, but can be 

better understood
Types of Uncertainty: Incertitude

• Arises from incomplete knowledge

• Parameter uncertainty due to 

– Limited sample size, presence of non-detects

– Measurement error, systematic error

– Use of surrogate data, poor system understanding

• Reducible with empirical effort
QUA Methods

• Variety of available methods

• Several well suited to data-poor assessments

• Many more methods than presented here

– Interval analysis

– Fault trees

– Bayesian methods

– Fuzzy arithmetic

– Empirical distributions

• Easy-to-use software available for most methods 

(though education and training required for intelligent 

use!)
Some Existing Probabilistic Tools 

for Pesticides

• Joint probability curves (ECOFRAM, 1999; Solomon et 

al. 2000. Crop Protection 19:649)

• Probabilistic PRZM model (Warren-Hicks et al. 2002. 

ET&C 21:1570)

• EPA’s Terrestrial Investigation Model 

(http://www.epa.gov/scipoly/sap/meetings/2004/033004

_mtg.htm)

• LiquidPARAM (Moore et al. 2014. Rev EC&T 231:163), 

GranPARAM (Moore et al. 2010. IEAM 6:260) 

• See Application of Uncertainty Analysis to Ecological 

Risks of Pesticides, Warren-Hicks, W.J. and Hart, A.V. 

(eds.). SETAC Press, Pensacola, FL.

• Many population models (Railsback, S.F. and V. 

Grimm. 2011. Agent-Based and Individual-Based 

Modeling: A Practical Introduction. Princeton Univ

Press, Princeton, NJ)
A Simple Example: Risk to Aquatic 

Invertebrate Prey

• Aquatic exposure

– PRZM/EXAMS: 30-year distribution of 

annual peak (acute), 21-day average 

or 60-day average (chronic) 

concentrations of pesticides in 

standard-sized ponds

– Modified PRZM/EXAMS

– SWAT, AGRO

• Effects: Species sensitivity 

distribution for aquatic 

invertebrates 

• Risk = Integration of exposure and 

effects curves
Exposure Distribution

0

20

40

60

80

100

0 5 10 15 20 25 30

P

r

o

b

a

bility

Concentration (mg/L)

0 0.005 0.01 0.015 0.02 0.025 0.3
SSD for Aquatic Invertebrate Prey

0.0

0.2

0.4

0.6

0.8

1.0

0.0001 0.001 0.01 0.1 1 10 100 1000 10000

Pro

p

ortio

n

of S

p

e

cie

s

Concentration (mg/L)

Log Gompertz Model

95% Confidence Limit

Data
Risk Curve

0

20

40

60

80

100

0 20 40 60 80 100

E

x

c

e

e

d

e

n

c

e

Pro

b

a

bility (%)

Percent of Species Affected (%)
Monte Carlo Simulation

How?

– replace each point estimate with a distribution

– repeatedly sample from each, tally answers in a 

histogram

Why?

– simple to implement and explain

– summarizes entire distribution of exposure or risk

Why not?

– requires a great deal of empirical information

– may need to invoke expert judgment in places

– difficult to quantify dependencies
Monte Carlo Analysis

Generate

Output

Risk

Fre

q

u

e

n

c

y

Summarize

Results

Dependencies

Risk = f(x,y,z)

Risk

Equation

Analysis

Fine Tune

Input Variable x Input Variable y

Fre

q

u

e

n

c

y

Input

PDFs

Input Variable z
Monte Carlo Output

0

20

40

60

80

100

0 50 100 150 200 250 300 350

Total Body Burden (mg ai/kg bw)

C

u

m

ulativ

e

P

r

o

b

a

bility (%).

American widgeon gorge 

feeding on alfalfa fields on the 

day of application of pesticide X 

at 1 lb a.i./A
Propagating Variability and 

Incertitude

• Most PRAs combine variability and incertitude

• NAS, EPA, DOE, etc. emphasize importance of 

tracking variability and incertitude separately

• Major advantages:

– Allows assessor to be forthright about what is known 

and what is not

– Relative importance of variability and incertitude can 

be quantified
Second-order Monte Carlo

How?

– let parameters of input distributions be distributions

– nest the Monte Carlo analyses

Why?

– partitions variability and incertitude

– plots get narrower with better empirical information

Why not?

– difficult to include some sources of uncertainty

– results are cumbersome to interpret and explain

– often difficult to separate incertitude and variability
Constants Random Variables

Well

Characterized

Having

Incertitude

Inputs

D

1

2

3

4

5

P(D)

B

1-5

P(B)

C

P

2 5 1 3 4

(C)

A

1-5

P(A)

Output

Variability

ƒ(A,B,C1,D1)

ƒ(A,B,C2,D2)

ƒ(A,B,C3,D3)

ƒ(A,B,C4,D4)

ƒ(A,B,C5,D5)

Second-order Monte Carlo
Second-order Monte Carlo Output
Probability Bounds Analysis

How?

– specify what you are sure about

– establish bounds on probability distributions

– specify dependencies (no assumption, independence, perfect, etc.)

Why?

– accounts for incertitude about parameter values, distribution 

shape, and dependencies

– can bound on 1

st order Monte Carlo results, bounds get narrower 

with better empirical information

Why not?

– cannot handle second-order probabilities

– optimum bounds difficult to compute when variables repeated
Probability Bounds Analysis 

Output

0

20

40

60

80

100

0 20 40 60 80 100

% Decline in Fecundity

E

xceedence Probability (%)

Monte Carlo

LPB - Lower Probability Bound

UPB - Upper Probability Bound

Low - Intermediate Criterion

Intermediate - High Criterion
Decision Making With Uncertainty 

Analysis

• “Bright lines” lacking for the 

outputs of PRA

• Probabilistic bright lines can 

assist communication and 

decision making

• Bright lines will likely differ 

between receptor groups

‒ Indirect vs direct effects

‒ Low and high fecundity 

listed species

‒ Threatened vs endangered

0

20

40

60

80

100

0 20 40 60 80 100

E

xceedence Probability (%)

% Species or Individuals Affected

Low risk?

Intermediate risk?

High risk?

Mitigation required?
IUCN Red List of Threatened 

Species

• Widely recognized for evaluating 

conservation status of plant and 

animal species

• When uncertainty leads to a wide 

range in results of species 

assessments, range of possible 

outcomes should be specified

• Single category (e.g., critically 

endangered, vulnerable, least 

concern) must be chosen and basis 

for decision documented

• Uncertainty may be represented as a 

plausible range around a best 

estimate

0 20 40 60 80 100

1

0.8

0.6

0.4

0.2

0

Time (years)

Pro

b

a

bility

critical

endangered

vulnerable Pro

b

a

bility

of E

xtin

ctio

n
Assessing Risks of Pesticides to 

Endangered Species

• Use simple techniques (e.g., risk quotients compared to levels 

of concern) to identify risk scenarios of potential concern

• Conduct sensitivity analysis to identify critical variables – guides 

data collection, indicates degree of confidence in QUA output

• Use Monte Carlo analysis in higher tier assessments – but 

consider combining with a 2D method to determine degree of 

incertitude

• Evaluate model performance and/or relevance and veracity of 

empirical data

• Collect other lines of evidence (e.g., empirical data, field 

studies), conduct weight-of-evidence evaluation

• List sources of uncertainty not included in analyses – discuss 

possible influence on risk estimates

• Engage risk manager, trustees, industry, public, etc. throughout 

process
Selected References

• Akçakaya, H.R. and Ferson, S. 2001. RAMAS® Red List: Threatened Species 

Classifications under Uncertainty. Version 2.0. Applied Biomathematics, New York. 

• Akçakaya, H.R., Ferson, S., Burgman, M.A., Keith, D.A., Mace, G.M. and Todd, C.A. 2000. 

Making consistent IUCN classifications under uncertainty. Conservation Biology 14: 1001-

1013. 

• Ferson, S., Moore, D.R.J., Van den Brink, P.J., Estes, T.L., Gallagher, K., O’Connor, R. and 

Verdonck, F. 2010. Bounding uncertainty analyses. In Application of Uncertainty Analysis to 

Ecological Risks of Pesticides, Warren-Hicks, W.J. and Hart, A.V. (eds.). SETAC Press, 

Pensacola, FL. pp. 89-122.

• Moore, D.R.J., R.S. Teed, S.I. Rodney, R.P. Thompson and D.L. Fischer. 2010. Refined 

avian risk assessment for aldicarb in the United States. IEAM 6(1): 83-101.

• Moore, D.R.J., Warren-Hicks, W.J., Qian, S., Fairbrother, A., Aldenberg, T., Barry, T., Luttik, 

R. and Ratte, H.-T. 2010. Uncertainty analysis using classical and Bayesian hierarchical 

models. In Application of Uncertainty Analysis to Ecological Risks of Pesticides, WarrenHicks, W.J. and Hart, A.V. (eds.). SETAC Press, Pensacola, FL. pp. 123-141.

• Morgan, M.G. and M. Henrion. 1990. Uncertainty: A Guide to Dealing With Uncertainty in 

Quantitative Risk and Policy Analysis. Cambridge University Press, Cambridge, U.K.

• Tucker, W.T., S. Ferson, A.M. Finkel and D. Slavin. 2008. Strategies for risk communication: 

Evolution, evidence and experience. Annals of the New York Academy of Sciences, Volume 

1128.
ESA Workshop 

April 22, 2014

Opportunities for Improvements

Summary
Next Steps

An on-going public dialogue on endangered 

species assessment process and methods 

Identify which recommendations: 

• can be adopted immediately

• merit further discussion

• schedule future meetings to move these forward

25
26

• Build on EPA FIFRA Risk Assessment 

• Analyses should be conducted separately for direct and indirect 

effects

• Include taxon specific thresholds 

• Include biological data and habitat requirements 

• Develop focal species for crops and use biology to estimate risk 

to species

• Refine exposure estimates/upgrade models to include greater 

realism in the risk assessment

Suggestions for May Affect / No Effect Decision
Examples of Further Steps in Developing Approach

• Adopt NAS recommendations regarding uncertainty 

analyses 

 Implement qualitative and quantitative approaches in 

future assessments

 Communicate uncertainty 

• Adopt a systematic W. of Evidence approach

 Include a scoring system for strength of evidence 

• Develop Additional Guidance

 Probabilistic risk assessment 

Interpretation of results of assessment

 Population Models

 Higher Tier RA tools / identified needs 

27