Analysis of Sensitivity Distributions 

for Estimation timation of Acut of Acute Hazard 

Concentrations to Aquatic Animals 

Matthew Etterson

1

Matthew Etterson

Office of Research and Development

Mid-Continent Ecology Division, Duluth, MN

U.S.EPA Scientific Advisory Panel Review 

January 31, 2012 

Arlington, VA
Talk Outline

1. What is an SSD?

2. Definitions

3. Methodological background

4. Seven questions about SSD performance

5. Future directions

2
0.7

0.8

0.9

1

0.7

0.8

0.9

1

1. What is an SSD?

0.7

0.8

0.9

1

0.7

0.8

0.9

1

0.7

0.8

0.9

1

0.7

0.8

0.9

1

0.7

0.8

0.9

1

triangular normal logistic gumbel pareto weibull burr

100

1. Do we have enough data?

2. Is the distribution of samples adequate?

3. How did we fit the statistical distribution?

4. Is the distribution a good representation of the 

-1 0 1 2 3 4 5 6 7 8

0

0.1

0.2

0.3

0.4

0.5

0.6

Log10 LC50

Cumulativ

e

P

ercentile

-1 0 1 2 3 4 5 6 7 8

0

0.1

0.2

0.3

0.4

0.5

0.6

Log10 LC50

Cumulativ

e

P

ercentile

3

-1 0 1 2 3 4 5 6 7 8

0

0.1

0.2

0.3

0.4

0.5

0.6

Log10 LC50

Cumulativ

e

P

ercentile

-1 0 1 2 3 4 5 6 7 8

0

0.1

0.2

0.3

0.4

0.5

0.6

Log10 LC50

Cumulativ

e

P

ercentile

p = 0.05

-1 0 1 2 3 4 5 6 7 8

0

0.1

0.2

0.3

0.4

0.5

0.6

Log10 LC50

Cumulativ

e

P

ercentile

-1 0 1 2 3 4 5 6 7 8

0

0.1

0.2

0.3

0.4

0.5

0.6

Log10 LC50

Cumulativ

e

P

ercentile

-1 0 HC5 1 2 3 4 5 6 7 8

0

0.1

0.2

0.3

0.4

0.5

0.6

Log10 LC50

Cumulativ

e

P

ercentile

4. Is the distribution a good representation of the 

data?

5. Can we predict the location of the true HC5 with a 

specified level of confidence?

6. Did we choose the right distribution?

7. Why not fit multiple distributions?

Malathion

← More sensitive Less sensitive →
2. Definitions

• Previous definitions: SMAV, GMAV, HC5, MDRs, 

typical 3 OPP species

• θ = set of parameters that define the specific 

form of a distribution (for example for the normal 

distribution: θ = [μ, σ])

• Reference value = empirical HC5 calculated using 

all available GMAVs

• AChEI = acetylcholinesterase inhibition

4
3. Methodological Background

• Data - Acute LC50 or EC50 from 96 h exposures (48 

h for Cladocerans)

• Statistical distributions: 

• normal, logistic, triangular, Gumbel

• Pareto, Weibull, Burr

• Fitting methods (parameter estimation)

• maximum likelihood

• moment estimators

• graphical methods

5
Parameter estimation (Maximum Likelihood)

 n

| 

n

  pdf x θ

ln   n| 

n

L p   df x θ

(normal: θ = [μ , σ] )

3. Methodological Background

6

n

     

2

2

1

1

ln 2 ln

2 2

n

i

i

n

L n   x 

 

          

2

2

1

1

ln 2 ln

2 2

n

i

i

n

L n   x







     

(normal: θ = [μ , σ] )

For these analyses, maximum likelihood was available for all 

seven distributions.
sample mean: = theoretical mean: f(θ)

sample variance: s

2 = theoretical variance: g(θ)

x

Distribution parameters mean variance estimator 1 estimator 2

Parameter estimation (Moment Estimators)

3. Methodological Background

7

Distribution parameters mean variance estimator 1 estimator 2

normal

logistic

triangular

2

2

3





ˆ  x ˆ  s

ˆ  x

ˆ 3

s





 



2 

θ   , 

θ   , 

For these analyses, moment estimators were available for 

the normal, logistic, triangular, Gumbel, and Pareto 

distributions

θ  a b, 

2

a b   

2

24

b a 

ˆ a x ˆ   2 b

ˆb x   s 6
1. Calculate the empirical percentile:  

1

i

i

rank x

p

n





Parameter estimation (Graphical Methods)

Example below uses normal distribution with μ = 0 , σ = 1

3. Methodological Background

8

2. Calculate ‘standard’ quantile:

3. Set up a linear model in z and x:

4. ‘Solve’ the linear model using least-squares

 

1

| 0,1 i i z F p





i i x z    

For these analyses, graphical methods were available for only 

the normal, logistic, and triangular distributions.
25

ˆ ˆ ln p

HCp  

 

   

Sampling variance and the HC5

3. Methodological Background

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

0

5

10

15

20

p

9

ˆ ˆ ln

1

p

HCp

p

 

 

      

     

2

ˆ var var ˆ var ln

1

p

HCp

p

 

   

          

2

ln

1

p

p

            
4.1. How does sample size influence accuracy 

(bias and variance) of the estimated HC5?

Rationale: 

• sampling variance decreases with 

increasing sample size

• Previous researchers have reported 

substantial bias with some fitting methods 

(especially maximum likelihood) at sample 

sizes in the range of interest (3 ≤ n ≤ 8)

10
Methods: 

• Simulated 10,000 replicate sets of data at 5 

sample sizes (n = [3, 8, 20, 50, 100]) under seven 

generating distributions (simulation data). (based 

on malathion)

4.1. How does sample size influence accuracy 

(bias and variance) of the estimated HC5?

on malathion)

• Fit all seven distributions (normal, triangular, etc.) 

to each replicate using three different fitting 

techniques (where possible)

• Calculated bias and sampling variance for all 

combinations of generating distribution and fitted 

distribution 11
R

a

tio

o

f

t

h

e

a

v

e

r

a

g

e

e

s

tim

a

t

e

d

H

C

5

t

o

t

h

e

t

r

u

e

H

C

5

4.1. How does sample size influence accuracy 

(bias and variance) of the estimated HC5?

10

100

Generating distribution = triangular

Estimation Method = maximum likelihood

triangular normal logistic gumbel pareto weibull burr

100

12

R

Sample Size

a

tio

o

f

t

h

e

a

v

e

r

a

g

e

e

s

tim

a

t

e

d

H

C

5

t

o

t

h

e

t

r

u

e

H

C

5

3 8 20 50 100 0.01

0.1

1 5
4.1. How does sample size influence accuracy 

(bias and variance) of the estimated HC5?

3 8 20 50 100 0.01

0.1

1

10

100

Generating distribution = triangular

Estimation Method = maximum likelihood

triangular normal logistic gumbel pareto weibull burr

3 8 20 50 100 0.01

0.1

1

10

100

Generating distribution = burr

Estimation Method = maximum likelihood

10

100

R

atio

of th

e

a

v

era

g

e

e

stim

ate

d

H

C

5

to th

e tru

e

H

C

5

Generating distribution = triangular

Estimation Method = moment estimators 10

100

Generating distribution = burr

Estimation Method = moment estimators

13

3 8 20 50 100 0.01

0.1

1

10

R

atio

of th

e

a

v

era

g

e

e

stim

ate

d

H

C

Estimation Method = moment estimators

3 8 20 50 100 0.01

0.1

1

10 Estimation Method = moment estimators

3 8 20 50 100 0.01

0.1

1

10

100

Sample Size

Generating distribution = triangular

Estimation Method = graphical estimators

3 8 20 50 100 0.01

0.1

1

10

100

Sample Size

Generating distribution = burr

Estimation Method = graphical estimators
1,000

10,000

Generating distribution = triangular

Estimation Method = maximum likelihood

triangular normal logistic gumbel pareto weibull burr

10,000

Wid

t

h

o

f

9

5

%

c

o

n

fid

e

n

c

e in

t

e

r

v

al a

r

o

u

n

d

t

h

e

H

C

5

4.1. How does sample size influence accuracy 

(bias and variance) of the estimated HC5?

14

3 8 20 50 100 1

10

100

Sample Size

Wid

t

h

o

f

9

5

%

c

o

n

fid

e

n

c

e in

t

e

r

v

al a

r

o

u

n

d

t

h

e

H

C

5
3 8 20 50 100 1

10

100

1,000

10,000

Generating distribution = triangular

Estimation Method = maximum likelihood

triangular normal logistic gumbel pareto weibull burr

3 8 20 50 100 1

10

100

1,000

10,000

Generating distribution = burr

Estimation Method = maximum likelihood

10,000

Width

of 9

5

%

CI aro

u

n

d th

e

H

C5

Generating distribution = triangular

Estimation Method = moment estimators

10,000

Generating distribution = burr

Estimation Method = moment estimators

4.1. How does sample size influence accuracy 

(bias and variance) of the estimated HC5?

15

3 8 20 50 100 1

10

100

1,000

Width

of 9

5

%

CI aro

u

n

d th

e

H

C

Generating distribution = triangular

Estimation Method = moment estimators

3 8 20 50 100 1

10

100

1,000

Generating distribution = burr

Estimation Method = moment estimators

3 8 20 50 100 1

10

100

1,000

10,000

Sample Size

Generating distribution = triangular

Estimation Method = graphical estimators

3 8 20 50 100 1

10

100

1,000

10,000

Sample Size

Generating distribution = burr

Estimation Method = graphical estimators
Conclusions: 

• When (3 ≤ n ≤ 8), bias is likely to be present 

(but direction depends on fitting method)

4.1. How does sample size influence accuracy 

(bias and variance) of the estimated HC5?

• When (3 ≤ n ≤ 8), sampling variance is likely 

to be large

• To control bias and variance, additional 

constraints must be imposed

16
4.2. How do taxonomic requirements 

influence accuracy of the estimated HC5?

4.2. How do taxonomic requirements 

influence accuracy of the estimated HC5?

Rationale: 

• Previous result showing large bias and 

variance at sample sizes of interest (3 ≤ n ≤ 8)

• Imposition of data requirements is intended 

to ensure the inclusion of results from both 

sensitive and insensitive taxa

17
Methods: 

• Used data on 12 AChEI pesticides (8 

organophosphates and 4 carbamates)

• Drew random samples of size n = 8 that either 

4.2. How do taxonomic requirements 

influence accuracy of the estimated HC5?

4.2. How do taxonomic requirements 

influence accuracy of the estimated HC5?

• Drew random samples of size n = 8 that either 

met or did not necessarily meet the 8 minimum 

data requirements of the 1985 guidelines 

(empirical data)

• True HC5 unknown so I substituted the reference 

HC5 (estimated using the same distribution & 

fitting method) 18
normal logistic triangular Gumbel Pareto Weibull Burr

0.0001

0.001

0.01

0.1

1

10

100 a.

8 GMAVs chosen at random 8 GMAVs meeting ALC MDRs

10

100

e

stim

ate

d

u

sin

g

5

7

G

M

A

V

s

b.

4.2. How do taxonomic requirements 

influence accuracy of the estimated HC5?

4.2. How do taxonomic requirements 

influence accuracy of the estimated HC5?

Maximum Likelihood

19

normal logistic triangular Gumbel Pareto 0.0001

0.001

0.01

0.1

1

10

R

atio

of H

C

5

at n

=

8 to

H

C

5

e

stim

ate

d

u

sin

g

5

7

G

M

A

V

s

normal logistic triangular

0.0001

0.001

0.01

0.1

1

10

100

Distribution

c. malathion

Moment Estimators

Graphical Methods
normal logistic triangular Gumbel Pareto Weibull Burr

0.01

0.1

1

10

100

1,000 a.

8 GMAVs chosen at random 8 GMAVs meeting ALC MDRs

100

1,000

Width

of 9

5

%

CI aro

u

n

d th

e

H

C

5

b.

4.2. How do taxonomic requirements 

influence accuracy of the estimated HC5?

4.2. How do taxonomic requirements 

influence accuracy of the estimated HC5?

Maximum Likelihood

20

normal logistic triangular Gumbel Pareto 0.01

0.1

1

10

100

Width

of 9

5

%

CI aro

u

n

d th

e

H

C

normal logistic triangular

0.01

0.1

1

10

100

1,000

Distribution

c.

malathion

Moment Estimators

Graphical Methods
Conclusions: 

• Impositions of taxonomic requirements 

introduces a negative (conservative) bias 

relative to reference values (estimated 

4.2. How do taxonomic requirements 

influence accuracy of the estimated HC5?

4.2. How do taxonomic requirements 

influence accuracy of the estimated HC5?

relative to reference values (estimated 

using all data)

• Taxonomic requirements reduce sampling 

variance

21
Rationale: 

• 1985 Guidelines describe the use of 

graphical methods

4.3. How do different estimation methods 

influence accuracy of the estimated HC5? 

4.3. How do different estimation methods 

influence accuracy of the estimated HC5? 

• Previous researchers have reported 

substantial bias with maximum likelihood

• Likely avenues for future work would 

employ maximum likelihood

22
Methods: 

• Used the simulation data described under 

question 1

• Used empirical data described under 

4.3. How do different estimation methods 

influence accuracy of the estimated HC5? 

4.3. How do different estimation methods 

influence accuracy of the estimated HC5? 

• Used empirical data described under 

question 2

• Fit seven distributions to all sets of data 

(both simulated and empirical) using three 

methods: maximum likelihood, moment 

estimators, graphic methods 23
Generating Distribution = normal, n = 3

0.01

0.1

1

10

100

Generating distribution = normal

Estimation Method = maximum likelihood

100

R

atio

of th

e

a

v

era

g

e

e

stim

ate

d

H

C

5

to th

e tru

e

H

C

5

Generating distribution = normal

Estimation Method = moment estimators

3. How do different estimation methods 

influence accuracy of the estimated HC5? 

3. How do different estimation methods 

influence accuracy of the estimated HC5? 

24

0.01

0.1

1

10

R

atio

of th

e

a

v

era

g

e

e

stim

ate

d

H

C

Generating distribution = normal

Estimation Method = moment estimators

triangular normal logistic Gumbel Pareto Weibull Burr

0.01

0.1

1

10

100

Distribution

Generating distribution = normal

Estimation Method = graphical methods
Generating Distribution = normal, n = 3

0

10

100

1,000

10,000

Estimation Method = maximum likelihood

10,000

Width

of th

e

9

5

%

c

o

nfid

e

n

c

e ra

n

g

e

of H

C

5

e

stim

ate

s

Estimation Method = moment estimators

4.3. How do different estimation methods 

influence accuracy of the estimated HC5? 

4.3. How do different estimation methods 

influence accuracy of the estimated HC5? 

25

0

10

100

1,000

Width

of th

e

9

5

%

c

o

nfid

e

n

c

e ra

n

g

e

of H

C

Estimation Method = moment estimators

triangular normal logistic Gumbel Pareto Weibull Burr

0

10

100

1,000

10,000

Distribution

Estimation Method = graphical methods
Conclusions: 

• At (3 ≤ n ≤ 8) maximum likelihood was usually

positively biased (liberal) 

• Moment estimators were usually least biased

4.3. How do different estimation methods 

influence accuracy of the estimated HC5? 

4.3. How do different estimation methods 

influence accuracy of the estimated HC5? 

• Moment estimators were usually least biased

• Graphical methods were usually negatively 

biased (conservative )

• Sampling variance did not differ much between 

methods, but was least with graphical methods

26
4.4. Are measures of fit (goodness-of-fit, 

Akaike’s Information Criterion) reliable 

indicators of performance?

4.4. Are measures of fit (goodness-of-fit, 

Akaike’s Information Criterion) reliable 

indicators of performance?

Rationale: 

• When we fit a distribution to empirical data we will always 

get an HC5 estimate. Can we diagnose how good it is?

• Model selection criteria (AIC) may help to choose among 

competing distributions

27
Methods: 

• Used the simulation data

• Assessed goodness of fit for each replicate using 

4.4. Are measures of fit (goodness-of-fit, 

Akaike’s Information Criterion) reliable 

indicators of performance?

4.4. Are measures of fit (goodness-of-fit, 

Akaike’s Information Criterion) reliable 

indicators of performance?

• Assessed goodness of fit for each replicate using 

parametric bootstrap sampling and tallied the 

proportion of fits with significant (at α = 0.05) lack 

of fit

• Tallied AICc weight (among 7 distributions) 

assigned to each sample

28
n = 8 normal logistic triangular Gumbel Pareto Weibull Burr

normal 0.11 0.11 0.08 0.07 0.03 0.07 0.09

logistic 0.03 0.04 0.03 0.06 0.07 0.04 0.03

G

e

n

e

r

a

tin

g

Dis

t

rib

u

tio

n

Fitted Distribution

4.4. Are measures of fit (goodness-of-fit, 

Akaike’s Information Criterion) reliable 

indicators of performance?

4.4. Are measures of fit (goodness-of-fit, 

Akaike’s Information Criterion) reliable 

indicators of performance?

29

triangular 0.05 0.10 0.07 0.07 0.05 0.09 0.05

Gumbel 0.07 0.06 0.10 0.02 0.03 0.09 0.05

Pareto 0.30 0.19 0.31 0.02 0.01 0.12 0.15

Weibull 0.13 0.08 0.13 0.09 0.20 0.05 0.13

Burr 0.07 0.11 0.12 0.05 0.06 0.09 0.06 G

e

n

e

r

a

tin

g

Dis

t

rib

u

tio

n

Frequency with which lack of fit is concluded at α = 0.05 

simulation data (n = 8)
n = 100 normal logistic triangular Gumbel Pareto Weibull Burr

normal 0.07 0.15 0.45 0.57 1.00 0.75 0.13

logistic 0.17 0.07 0.86 0.55 1.00 0.82 0.12

G

e

n

e

r

a

tin

g

Dis

t

rib

u

tio

n

Fitted Distribution

4.4. Are measures of fit (goodness-of-fit, 

Akaike’s Information Criterion) reliable 

indicators of performance?

4.4. Are measures of fit (goodness-of-fit, 

Akaike’s Information Criterion) reliable 

indicators of performance?

30

triangular 0.10 0.23 0.09 0.63 1.00 0.73 0.17

Gumbel 0.81 0.63 1.00 0.03 1.00 0.61 0.07

Pareto 1.00 1.00 1.00 0.11 0.05 0.26 0.10

Weibull 0.78 0.60 1.00 0.22 1.00 0.04 0.85

Burr 0.20 0.05 0.88 0.49 1.00 0.87 0.05 G

e

n

e

r

a

tin

g

Dis

t

rib

u

tio

n

Frequency with which lack of fit is concluded at α = 0.05 

simulation data (n = 100)
n normal logistic triangular Gumbel Weibull Pareto Burr

AICc weight assigned to 10,000 samples of log-logistic data

4.4. Are measures of fit (goodness-of-fit, 

Akaike’s Information Criterion) reliable 

indicators of performance?

4.4. Are measures of fit (goodness-of-fit, 

Akaike’s Information Criterion) reliable 

indicators of performance?

31

8 0.14 0.13 0.20 0.12 0.17 0.23 0.01

20 0.18 0.19 0.24 0.12 0.16 0.06 0.06

50 0.22 0.32 0.17 0.07 0.09 0.00 0.13

100 0.24 0.46 0.06 0.02 0.03 0.00 0.20
Conclusions: 

• Neither AICc nor goodness-of-fit tests are 

likely to help diagnose fit or choose among 

4.4. Are measures of fit (goodness-of-fit, 

Akaike’s Information Criterion) reliable 

indicators of performance?

4.4. Are measures of fit (goodness-of-fit, 

Akaike’s Information Criterion) reliable 

indicators of performance?

c

likely to help diagnose fit or choose among 

distributions at (3 ≤ n ≤ 8)

• Both would be quite useful at larger sample 

sizes

32
4.5. What are actual coverage rates of 95% 

confidence limits?

Rationale: 

• We wish to be able to estimate: 

• But we can actually estimate:

“the probability that the true HC5 is higher than a 

given concentration.”

• But we can actually estimate:

33

“the probability that a calculated lower boundary 

concentration is less than the true HC5.”

Pr 5 0  .95 nx k  s HC 

Pr(lower boundary conc. ≤ HC5) ≈ 0.95

Pr x s   kn HC5 0   .95 Aldenberg & Slob 1993
Methods: 

• Used both simulation data and empirical data

• Used published extrapolation constants (kn, Aldenberg & 

4.5. What are actual coverage rates of 95% 

confidence limits?

• Used published extrapolation constants (kn, Aldenberg & 

Slob 1993, and others) to calculate lower 95% confidence 

bounds on the HC5

• Applied the methodology to distributions in the locationscale family only (normal, logistic, triangular)

• Tallied the frequency with which the actual (or reference) 

HC5 was greater than the estimated 95% boundary

34
Fitting Method

Distribution

normal logistic triangular

4.5. What are actual coverage rates of 95% 

confidence limits?

35

normal logistic triangular

Maximum Likelihood 0.95 0.96 0.94

Moment Estimators 0.94 0.95 0.93

Graphical Methods 0.95 0.97 0.95

Frequency with which the calculated lower boundary is lower 

than the true HC5 for simulation data
Fitting Method

Distribution

normal logistic triangular

4.5. What are actual coverage rates of 95% 

confidence limits?

36

normal logistic triangular

Maximum Likelihood 0.70 0.73 0.66

Moment Estimators 0.82 0.85 0.78

Graphical Methods 0.70 0.65 0.64

Frequency with which the calculated lower boundary is lower 

than the reference HC5 for empirical data and MDRs not 

met (n = 8)
Fitting Method

Distribution

normal logistic triangular

4.5. What are actual coverage rates of 95% 

confidence limits?

37

Frequency with which the calculated lower boundary is lower 

than the reference HC5 for empirical data and MDRs met 

n = 8).

normal logistic triangular

Maximum Likelihood 1.000 0.999 0.998

Moment Estimators 1.000 1.000 1.000

Graphical Methods 0.999 0.997 0.989
Conclusions: 

• For data simulated from a known 

distribution 95% confidence limits provide 

coverage at the nominal rate

4.5. What are actual coverage rates of 95% 

confidence limits?

coverage at the nominal rate

• For empirical data 95% confidence limits 

provide less than nominal coverage 

without data requirements and greater 

than nominal coverage with data 

requirements 38
4.6. How do different candidate distributions 

perform relative to each other?

Rationale: 

• Choice of distribution has been described 

as both important and difficult in existing 

literature

• There are many distributions to choose 

from!

39
Conclusions: 

• In general, distributions on transformed data 

performed better than distributions on 

untransformed data

• Of the above, triangular performed best across all 

4.6. How do different candidate distributions 

perform relative to each other?

• Of the above, triangular performed best across all 

AChE inhibitors

• Of distributions on untransformed data, Burr 

performed best

• Regardless of the above points, there remains 

considerable uncertainty about which distribution 

performs best 40
4.7. Does model-averaging across distributions 

improve estimates of the HC5?

4.7. Does model-averaging across distributions 

improve estimates of the HC5?

Rationale: 

• If no one distribution is best, perhaps 

model-averaging methods can be used to 

generate a weighted average HC5 estimate 

across all distributions?

41
Conclusions: 

• Calculating a model-averaged HC5 did not 

help with simulated data

• In contrast, with empirical data, weighted 

4.7. Does model-averaging across distributions 

improve estimates of the HC5?

4.7. Does model-averaging across distributions 

improve estimates of the HC5?

42

• In contrast, with empirical data, weighted 

HC5 estimates were far less biased than 

estimates from a single distribution

• The reason for the different performance 

for simulated versus empirical data is not 

clear and should be investigated further
5. What future research is suggested 

by these results? 

• We need more data (information), but 

problem is defined for (3 ≤ n ≤ 8)

• “Borrow” information from other species & 

chemicals

• Use hierarchical modeling techniques

43
EC/LC50 ~ n logistic ormal   ,, 

5. What future research is suggested 

by these results? 

44

EC/LC50     0, MOA  de Zwart (2002)

EC/LC50     0, 

  f t axonomy, chemical

Web ICE
45