/•

SDMS Document ID

2037052

FINAL

METHODS AND STANDARDS FOR EVALUATING

THE PERFORMANCE OF THE

LAKE COUNTY COMMUNITY HEALTH PROGRAM (LCCHP)

July, 2002

Prepared by:

LCCHP Performance Standards Working Group
I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

EXECUTIVE SUMMARY

This document identifies four Performance Standards that will be used to evaluate the

performance of the Lake County Community Health Program (LCCHP) as the remedy selected

for Operable Unit 9 at the California Gulch Superfund Site. These performance standards are

intended to help guide the activities performed under the LCCHP so that the Remedial Action

Objectives identified for OU9 are achieved. Once a demonstration has been made that the

Performance Standards have all been achieved, the OU9 remedy will be considered complete.

Remedial Action Objectives

The Remedial Action Objectives (RAOs) established by EPA for OU9 are as follows:

RAO-1. No more than 5% of children age 0-72 months residing within OU9, either now

or in the future, should have blood lead values exceeding 10 ug/dL

RAO-2. No more than 1% of children age 0-72 months residing within OU9, either now

or in the future, should have blood lead values exceeding 15 ug/dL

For convenience, the percent of children with blood lead values above 10 ug/dL is referred to as

"P10", and the percent above 15 ug/dL is referred to as "PI5".

Statistical Units

For the purposes of evaluating performance and compliance with RAOs, Operable Unit 9 is

divided into two "Statistical Units". Each Statistical Unit will be evaluated independently.

Performance Standards for OU9 Remedy

The remedy for a Statistical Unit will be considered complete when the following performance

standards are achieved.

Standard 1

The total number of blood lead program participants from the Statistical Unit over the preceding

3-year period (i.e., the sum of the number of participants for each of three running years) shall

be at least 180 children (age 6-72 months), and the 3-year average recruitment of newly eligible

individuals shall be 50% or greater.

The purpose of this standard is to help ensure that tests for Standards 2, 3, and 4 (below) are all

based on a large and representative fraction of the eligible population. This is necessary to have

confidence the data generated by the program are generally applicable, and are not based on a

small (and un-representative) part of the community.

ES-l
Standard 2

Based on the consolidated blood lead data set for the three year period, the calculated P10 value

shall be 5% or less, and the calculatedPJ5 value shall be 1% or less, with no individual PJO

value in any of the three years exceeding a value of 8%.

The purpose of this standard is to ensure that the incidence of children with elevated blood lead

values (greater than 10-15 ug/dL) is not greater than specified in the RAOs for the site.

Standard 3

Based on the consolidated blood lead data set for the three year period, the probability shall be

less than 5% that the true P10 value exceeds 7%, and the probability shall be less than 5% that

the true PI5 value exceeds 2%.

The purpose of this standard is to ensure that if a Type I decision error occurs for Standard 2 as a

result of statistical variability in the data, the magnitude of the error will be small.

Standard 4

Based on the set of soil lead data for blood lead program participants during the preceding three

year period, the proportion of soil lead values above the 70' percentile of the Statistical Unit as

a whole will not be significantly lower than 30%, unless the reason for the difference is

unavoidable.

The purpose of this requirement is to demonstrate that the distribution of soil lead values for

participants in the blood lead program does not under-represent the high end (70th percentile and

above) of the soil lead distribution from the Statistical Unit. In the case that this demonstration

is not achieved, the LCCHP will investigate to determine if it is reasonable and practical to

achieve the standard, either by increased recruiting of children who reside in areas where soil

leads tend to be high, and/or by increasing the total number of observations (and hence the

power). If so, the LCCHP will take all reasonable and practical steps to seek compliance with

Performance Standard 4. If the reason for the failure is found by the LCCHP to be unavoidable,

Performance Standard 4 will be waived.

Decision Rule

If a Statistical Unit achieves compliance with the performance standards detailed above, ongoing

demonstration of compliance with the RAOs will not be required for that Statistical Unit. The

LCCHP will be considered complete for the Operable Unit when both Statistical Units have

achieved the standards.

ES-2
METHODS AND STANDARDS FOR EVALUATING

THE PERFORMANCE OF THE LCCHP

1.0 Purpose of this Document

This document describes the approach that the Lake County Community Health Plan (LCCHP)

Work Group shall utilize to evaluate the on-going performance of the program, and identifies

performance standards that must be satisfied to demonstrate that the remedy for Operable Unit 9

(OU9) is complete.

2.0 Remedial Action Objectives (RAO-1 and RAO-2)

The Remedial Action Objectives established by EPA for OU9 are as follows:

RAO-1. No more than 5% of children age 0-72 months residing within OU9, either now

or in the future, should have blood lead values exceeding 10 ug/dL

RAO-2. No more than 1% of children age 0-72 months residing within OU9, either now

or in the future, should have blood lead values exceeding 15 ug/dL

For convenience, the percent of children with blood lead values above 10 ug/dL is referred to as

"P10", and the percent above 15 ug/dL is referred to as "PI5". Because children age 0-6 months

usually have low exposure to lead from soil or dust, and because collection of blood from infants

in this age range is generally not desirable, this program focuses on children age 6-72 months of

age rather than children age 0-72 months of age.

3.0 Statistical Units

For the purposes of evaluating compliance with RAO-1 and RAO-2, the site (Operable Unit 9)

will be divided into two "Statistical Units". Statistical Unit 1 will contain Sub-areas A, B, D,

and E, while Statistical Unit 2 will contain Sub-areas C, F and G. This grouping is based on the

known similarities in the characteristics of housing and sources of environmental lead exposure

within the combined Sub-areas. A more detailed rationale for this grouping, including the data

evaluated to identify and combine similar Sub-areas, is provided as Attachment 1.

Each Statistical Unit will be evaluated independently. If a Statistical Unit achieves compliance

with the RAOs, as defined by the assessment methods and performance standards detailed

below, ongoing demonstration of compliance with the RAOs will not be required for that

Statistical Unit. However, the LCCHP will continue to make blood lead testing and

environmental testing and remediation services available to all eligible residents in the Statistical

Unit until completion of the LCCHP program when the other Statistical Unit also achieves

compliance with the RAOs.
4.0 Performance Standards for OU9 Remedy

The remedy for a Statistical Unit will be considered complete when the following performance

standards are achieved:

1) The total number of blood lead program participants from the Statistical Unit over the

preceding 3-year period (i.e., the sum of the number of participants for each of three

running years) shall be at least 180 children (age 6-72 months), and the 3-year average

recruitment of newly eligible individuals shall be 50% or greater.

2) Based on the consolidated blood lead data set for the three year period, the calculated

P10 value shall be 5% or less, and the calculated PI5 value shall be 1% or less1, with no

individual P10 value in any of the three years exceeding a value of 8%.

3) Based on the consolidated blood lead data set for the three year period, the probability

shall be less than 5% that the true P10 value exceeds 7%, and the probability shall be less

than 5% that the true PI5 value exceeds 2%.

4) Based on the set of soil lead data for blood lead program participants during the

preceding three year period, the proportion of soil lead values above the 70th percentile of

the Statistical Unit as a whole will not be significantly lower than 30%, unless the reason

for the difference is unavoidable. The purpose of this requirement is to determine if blood

lead program participants are from residences that under-represent the high end (70th

percentile and above) of the soil lead distribution from the Statistical Unit.

The LCCHP will be considered complete for the Operable Unit when both Statistical Units have

achieved the standards above, as well as all other responsibilities of the program as specified in

the LCCHP Workplan.

Because the existing site conditions were used as the basis for the design of certain elements of

these performance standards, the standards will be subject to review and modification if future

site conditions are substantially different than those that exist at present (2002). For example, a

future drop or rise in the child population may warrant modification of performance standard

number 1 to reflect the new population numbers. Any modifications recommended by the

LCCHP Workgroup will be consistent with the rationale presented in this document and subject

to final review and approval by the U.S. EPA.

5.0 Statistical Procedures

5.1 Data Reduction

The individual blood lead data values for each year will be evaluated in accord with previously

1 All calculated P10 and PI5 values will be rounded to the nearest 1%.
I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

established procedures (Attachment 8). Individuals who have elevated blood lead values that are

attributable to sources beyond the control of the program will not be included in the data set used

to evaluate program performance. This determination will be made by the LCCHP workgroup

on a case-by-case basis.

5.2 Calculation of PI 0 and PI 5 Values

The values of P10 and P15 can be estimated directly from a data set (e.g., 5 children out of 100

with blood lead values above 10 ug/dL corresponds to an observed P10 value of 5%). However,

the values of P10 and PI5 calculated in this way tend to be statistically unstable when the

number of children is not large, and small differences in the number of children above the RAO

values can have a large effect on the P10 and P15 values. Therefore, for the purposes of tracking

the performance of the LCCHP, the value of P10 and PI 5 will be calculated based on a statistical

curve-fitting approach that utilizes the entire blood lead data set for a Statistical Unit.

In brief, the blood lead data for children (6 to 72 months of age) residing in each Statistical Unit

will be fit to an appropriate continuous parametric distribution, and based on the best fit model to

the data, the percent of the samples exceeding 10 ug/dL and 15 ug/dL will be calculated from the

model parameters.

Attachment 2 provides two alternative Excel spreadsheets that automatically performs the

calculation of the P10 and PI 5 value based on the assumption that the blood lead values are

likely to be distributed approximately lognormally. The first sheet should be used if the data set

contains no non-detect values, while the second spreadsheet should be used if the data set

contains one or more non-detects. In either case, the fit based on the assumption of lognormality

should be accepted if the goodness of fit statistic is less than the critical value. If the lognormal

fit is rejected, then alternative models (e.g., gamma, Weibull, beta, Johnson Sb, etc.) should be

tested to see if a significantly improved fit can be achieved.

5.3 Uncertainty in the PI 0 and PI 5 Values

The P10 and PI 5 values calculated as described above represent the best estimate of the true P10

and PI 5 value for the population of children residing in the Statistical Subunit, based on the

measured blood lead values in the participating population. However, the calculated values are

uncertain because the calculations are based on a random sample of children rather than the

entire population. Attachment 6 provides an Excel-Crystal Ball spreadsheet that estimates the

uncertainty in the calculated P10 and PI5 values. The required inputs are the observed

geometric mean and geometric standard deviation of the blood lead data set and the number of

data points in the data set for the preceding three year period. The output is the uncertainty

distribution around the best estimate of PI 0 or PI 5. The probability that the true P10 value

exceeds 7%, or that the true P15 value exceed 2%, may be determined directly from the outputs.
5.4 Representativeness of Soil Lead Values for Program Participants

The representativeness of the soil lead levels for the blood lead program participants in a

Statistical Unit will be evaluated by comparing the distribution of soil lead values from the

residences of blood lead program participants to the distribution of soil lead values for the

statistical unit as a whole at a given point in time.

The statistical method for this comparison will be a test of proportions (Canavos 1984). In brief,

this test categorizes each soil lead value in a data set into one of two bins: above some specified

concentration level, or below that level. Because the purpose of the test is to determine if the

blood lead program participants are from residences that under-represent the high end of the soil

lead distribution, the value selected for the categorization will be the 70th percentile of the

"reference" data set (all soil values from the statistical unit). Thus, by definition, the proportion

of all samples in the reference data set above this value will be 30%. The percent of all samples

above that same concentration value is then computed for the "test" data set (the soil leads for

properties where blood lead study participants reside). The probability that the true proportion

of samples in the test data set is lower than the proportion in the reference data set is evaluated

using the test of proportions (see Attachment 7). If the p value of the test is lower than 0.05,

there is a 95% probability that the blood lead program participants are from properties that

under-represent the high end of the soil lead distribution in the Statistical Unit. If the p value is

greater than 0.05, the test data set will not be declared different than the reference data set,

unless the power of the test to detect a difference of 10% is too low. This is accomplished by

articulating the decision rule as follows:

Performance Standard 4 is satisfied if p > a, where a = 0.05 / Power

In the case that this decision rule is not satisfied, the LCCHP will investigate to determine if it is

reasonable and practical to improve the representativeness of the blood lead study population by

increased recruiting of children residing in areas that tend to have high soil lead levels, and/or to

increase the power of the test by increasing the total number of data points in the data set. If so,

the LCCHP will take all reasonable and practical steps to seek compliance with Performance

Standard 4. The program shall not be declared complete until Performance Standard 4 has been

achieved, unless the reason for the non-compliance is impractical to correct due to an absence of

eligible children, especially those residing at properties with elevated soil lead levels. If the

reason for the failure is found by the LCCHP to be unavoidable, Performance Standard 4 will be

waived.

5.5 Estimation of Recruitment Rate for Newly Eligible Individuals

Estimation of the annual recruitment rate of previously untested children is difficult because

detailed statistics on the number of untested children are not available, and because the precise

identity of the untested population changes each year as children are bom in the area or as they

move into or out of the area. However, a screening-level population model (see Attachment 4)

reveals that the annual steady-state recruitment rate may be estimated on a site-wide basis as the
ratio of the number of first-time tested children (age 6-72 months) divided by the estimated

number of newly eligible children in the site. Newly eligible children are defined as children

who are born in the site area or children less than 6 years of age who move into the site area.

The estimate of newly eligible children each year will be derived from available information a)

the percentage of Lake County children who live in OU9, as indicated by the year 2000 census,

b) an estimate of the number of children born in Lake County each year, and c) an estimate of

the number of new children eligible to enroll in public school each year. The number of births

and number of new school-age children within the site (OU9) will be computed from the Lake

County estimates based on the percentage of Lake County children living within the site's (OU9)

boundaries at the time of the 2000 census. The 2000 census data, examples of the supporting

birth count and school enrollment data to be used, and example calculations are presented in

Attachment 5.

6.0 Discussion: Basis of the Performance Standards

6.1 Use of a Three- Year Averaging Period

The use of a three-year averaging period is selected to ensure that achievement of the

performance standard is stable over time, and to minimize the risk of Type I and Type II errors

due to short-term random fluctuations in the calculated values of P10 and PI5. Based on data

obtained to date, typical annual samples sizes are around 160-200 children total, distributed

approximately evenly between the two Statistical Units. Thus, the expected number of

participants for each Statistical Unit is about 80-100. Most GSD values range from 1.4 to 1.9,

with a value of about 1.7 being typical. Assuming a GSD of 1.7 and assuming a minimum

sample size of 60/year, the Type I and Type II error rates based on a three-year running average

P10 value (calculated using Monte Carlo simulation, as detailed in Attachment 3) are as follows:

True P10 Value

3%

4%

5%

6%

7%

8%

Error Type

Type II

Type I

Error Rate

2%

19%

51%

20%

6%

2%

Type I error = declare P10 to be 5% or less when it is actually higher than 5%

Type II error = declare P10 to be higher than 5% when it is actually 5% or lower

As seen, if the true P10 value is only slightly higher than RAO-1 (e.g., 6%), there is about a 20%

chance that the measured three-year running average P10 value would be 5% or less, leading to a

Type I error (deciding the site was in compliance when it really was not). However, if the

difference between the true P10 value and RAO-1 is larger (e.g., 7% or higher), then the

probability of a Type I error becomes quite small (about 6%). Similarly, if the true P10 value is
only slightly below RAO-1 (e.g. 4%), then the probability of a Type II error (declaring the data

to be higher than 5% when it really is not) is about 19%, and this value decreases rapidly as the

true P10 value becomes smaller. These error rates are not sensitive to the assumed GSD value,

but do tend to decrease if the number of blood lead program participants each year is higher than

the assumed participation rate of 60/year (see below).

A similar pattern is obtained for errors in the PI5 value, as shown below:

TruePlSValue

0.50%

0.75%

1.00%

1.25%

1.50%

2.00%

Error Type

Type II

Type I

Error Rate

7%

29%

56%

24%

14%

4%

Type I error = declare P15 to be 1% or less when it is actually higher than 1%

Type II error = declare PI5 to be higher than 1% when it is actually 1% or lower

As above, the chances of making a decision error are highest when the true PI 5 value is very

close to the decision value (1%), and the error rates decrease rapidly as the true value becomes

either smaller or larger than the decision value.

6.2 Requirements for Participation Rate

Confidence in the calculated P10 and P15 values is directly related to the number of individuals

who participate in the program, with low participation corresponding to low confidence. To help

ensure a minimum level of confidence, two types of requirements are established regarding

annual and cumulative participation:

a) Total Number of Participants Each Year

The first requirement for participation rate is based on the total number of participants each year, including both new

and repeat participants. This requirement is needed to minimize the statistical chances of a Type I or a Type II error

in the calculation of the P10 and PI 5 values. As discussed in Appendix 3, both Type I and Type II error rates are

expected to be within acceptable bounds if the annual number of participants in each Statistical Unit is 80 or more,

but are likely to become marginal or unacceptable if the number of participants falls to about 40 per year. On this

basis, the minimum three-year running average number of participants (i.e., individual children tested) per Statistical

Unit is 60 per Statistical Unit per year (180 total for a three-year period).

b) Fraction of Untested Children Recruited Each Year

The second element of the participation rate requirement is the number of new (i.e., previously

untested) participants recruited into the program each year. This requirement is needed to ensure
that there is good representativeness across age strata (year of birth) in the population, and to

ensure there is a good probability that each eligible child will be tested at least once. Assuming

that the odds of recruiting a previously untested child into the program is about constant over

time, then the odds of a child participating at least once during the 6 year eligibility period (age

6-72 months) is

P=l-(l-p) 6

where:

p = probability of participating at least once in 6 years

P = probability of a previously untested child being recruited in any one year

If the value of p is 50%, then the value of P is 98%. On this basis, a 50% recruitment of

previously untested children is identified as the minimum target participation rate from across

the site each year.

6.3 Representativeness of the Program Participants

An important goal of any program of this type is to ensure that the participants in the program

are representative of the community as a whole, and that specific sub-populations are not

substantially over- or under-represented. There are many different indices that may be used to

evaluate representativeness, including age, gender, race, socioeconomic status, education, and

environmental exposure levels to lead in soil, paint, and water.

a) Representativeness of Soil Lead Levels

Of the numerous alternative metrics of representativeness, the one that is most important for the

purposes of this program is the level of lead in soil. That is, it is considered of special

importance to ensure that the children who participate in the program are drawn from residences

with soil lead values that do not significantly under-represent the high end of the soil lead

distribution in the Statistical Unit. It is for this reason that Performance Standard 4 requires a

statistical test to demonstrate that the soil lead values from the yards of blood lead program

participants contain a proportion of high soil lead values that is similar to the Statistical Unit as a

whole. Note that, because the on-going operation of the LCCHP is expected to cause a

downward shift in the distribution of soil lead values in each Statistical Unit, this comparison

must be based on the most current estimate of the soil lead distribution in the Statistical Unit.

b) Other Indices of Representativeness

There are no quantitative performance standards linked to any other metric of representativeness,

but it is considered important that the LCCHP work hard to ensure that children from all

segments of the community are effectively recruited into the program. Therefore, the LCCHP

workgroup will monitor data on the demographic attributes of the individuals participating in the
blood lead monitoring program, and seek to ensure that the participating population is

representative of the community as a whole. This will be done by monitoring the following

indices of representativeness:

a) Spatial Representativeness. Spatial representativeness will be tracked by mapping

the sub-area of the residence of each child participating in the blood lead program,

and calculating the estimated participation rate for each of the two Statistical Units

and for each of the Sub-areas (A-G). These data will be monitored annually by the

LCCHP Work Group, as described in the annual report outline, and indications of low

participation rate from any subarea will be responded to by increased recruiting

efforts in that subarea. Factors to consider in planning the increased recruiting effort

for an area may include available information on lead levels in potential sources in

the area (soil, leaded paint, etc), the number and status of children in the area, the

historic frequency of elevated blood lead values from the area, etc.

b) Age Group Representativeness. The preferred time of first blood testing for an

eligible child is between age 1 and 2. This is because most blood lead surveys

indicate that blood lead values tend to be highest in this age group, presumably

because children in this age group have the highest tendency to ingest lead from nondietary sources. However, data collected from a child at any age between 0 and 6

years will be considered acceptable. The workgroup will monitor yearly statistics on

the relative number of participants stratified by age group, and will make all

reasonable efforts to maximize participation by children age 1-2.

c) Other Demographic Factors. Ideally, the ethnic and socioeconomic status (SES) of

children participating in the program should be representative of the population

residing in each statistical unit. This is because most blood lead surveys reveal that

there are statistically significant differences in geometric mean blood lead value in

groups of children stratified by race or SES. In this program, data are available on

ethnicity of program participants, and the LCCHP workgroup shall analyze these data

annually to assess whether the program participants are representative of the

community makeup as a whole. If not, the LCCHP shall increase efforts to recruit

participation of any sub-groups that may tend to be under-represented.

7.0 Evaluation of "Residual Risk"

It is recognized that a certain degree of risk from potential lead exposures will remain in the

community once a remedy is completed. However, a goal of the program is to ensure that

whatever residual risks that remain at unremediated and remediated properties, at completion of

the LCCHP, will be sufficiently low as to not exceed the RAOs. That is, given compliance with

the performance standards, it is expected that no more than 5% of a set of children selected at

random from the community (OU9) will have a blood lead value above 10 ug/dL, and no more

than 1% will have a blood lead value above 15 ug/dL.
I

I

I

i

i

i

i

i

i

i

i

i

i

i

i

i

i

i

i

Using the performance standards as defined above, coupled with other requirements of the

LCCHP program (including an active outreach program to ensure that all area residents are

aware of the program and have been given reasonable opportunity to participate before the

program ceases operation), it is expected that this goal will be achieved by the time the LCCHP

is completed. This conclusion is based on the fundamental assumption that the participants in

the LCCHP are generally representative of the community as a whole, and that the statistical

attributes of the program participants also apply to the properties and individuals who do not

participate.

8.0 References

D'Agostino RB, Stephens MA (eds). 1986. Goodness-of-Fit Techniques. Marcell Decker, Inc.,

New York.

Canavos GC. 1984. Applied Probability and Statistical Methods. Little, Brown and Co.

Boston.
I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

ATTACHMENT 1

Rationale for Statistical Unit Definitions

The two Statistical Units represent combinations of the existing Subareas into groups

having similar environmental conditions with respect to potential sources of lead

exposure. The environmental data and property information collected by the University

of Cincinnati in 1991 and by the Kids First Program from 1994 through May of 2001

were combined and then used to make comparisons of environmental conditions in the

various Subareas. The data used for these inter-area comparisons are the propertyaverage lead concentrations in yard soil, interior paint, exterior paint, water and dust.

These data are presented in a series of "box and whisker" plots that display the

distribution of lead concentrations in each media type and within each of the Subareas.

Other factors such as percentage of homes that are owner (or renter) occupied and age of

housing were also evaluated and considered in comparison of the Subarea characteristics.

Areas A, B, D and E are similar in their yard soil, interior paint, exterior paint, water and

dust lead levels. In addition, Areas A, B and D have approximately the same proportion

of owner-occupied and rental properties and similar age housing. These Subareas make

up the older portions of the City of Leadville. Area E, Stringtown, has more owner

occupied properties and slightly newer housing than Areas A, B and D, but despite the

newer housing age, paint and dust lead concentrations are not distinct from those in the

older portions of Leadville. These four Subareas have been combined to establish

Statistical Unit 1.

Areas C, F and G all have newer housing than the other areas. Areas C and G are similar

in their distributions of lead in soil, interior paint, exterior paint, water and dust, and

except for the drinking water data, these distributions are distinct from those observed in

Areas A, B, D and E. Area F soil lead concentrations have typically been higher than in

either C or G and more like those observed in Areas A and D. However, soil remediation

for all properties within Area F was initiated in 2001 and is expected to be complete in

2002. For this reason, we expect that the soil lead concentrations in Area F will be more

like those in Areas C and G after 2002 than those that remain in Areas A, B, D and E.

Lead concentrations in water initially appeared higher in Area F than in any of the other

areas, but this difference was mainly due to relatively high lead concentrations in the tap

water samples collected by the University of Cincinnati in 1991. Tap water samples

collected by the Kids First Program from over the last 5 years show lower lead

concentrations (refer to following box and whisker plots for Kids First Program water

and Bornschein study water samples). This difference in the two sets of tap water results

was not apparent in any other Subarea so it did not appear to be an artifact of slight

differences in the water sampling procedures used by those two programs. All of the

residences within Area F, the Lake Fork Trailer Park, obtain their tap water from a single

private well adjacent to the trailer park. The current manager of the trailer park reported

that two water supply wells were used in the past and both were likely in use during the
1991 sampling event; at present, only one of the wells is used for water supply to the

trailer park. Given this information, it appears that the well no longer in use may have

provided water with higher lead concentrations than the remaining water supply well.

The lead concentrations in water samples collected from Area F in the last 5 years do not

appear distinct from those in any other Subarea of the site. Given these data, Areas C, F

and G have been combined to establish Statistical Unit 2.

Combining the existing Subareas into two Statistical Units provides relatively large

numbers of young children eligible to participate in blood lead monitoring from each of

the units. Based on annual participation numbers from the last seven years, we expect

participation from approximately 90 young children from Areas A, B, D and E combined

and approximately 100 young children from Areas C, F and G combined.

Explanation of Box and Whisker Plots:

The lead distributions in various environmental-lead sources are presented in the

following "box and whisker" plots. These plots were prepared from the lead-testing data

collected by the Kids First Program (1995 through May 2001) and the University of

Cincinnati, or R. Bornschien, study (1991), and they can be used to compare the lead

concentrations in environmental media from the various Subareas. For each of the

environmental media, the lead concentrations measured at a single property were

averaged, and the property-average values were then used to prepare the box plots.

The box and whisker plots show the median value (solid line), 25th and 75th percentile

values (box limits), 10th and 90th percentile values (whisker end points), and 5th and 95th

percentile values (outside circles) for the lead concentration distribution in each Subarea

data set.
IIIIIIIIIIIIIIIIIII

Average Soil Lead Concentration

Participating Properties

•^ 14000 -

,§. 12000 -i

Oro 1000

0 -

§c 800

0 -

0g 600

0 -

CO 4000 -

0)D) 5oj 200

0 -

>

0 -

•T+A

•TVB

,C

•

~|~

4rD

Area

•I 4-E

0

•_.

-4- I ^ I

F G
IIIIIIIIIIIIIIIIIII

Average Dust Lead Concentration

Participating Properties

5000

4000 -

3000 -

S 2000 -

1000 -

T

-r

D

Area
IIIIIIIIIIIIIIIIIII

30

20 -

£

 1JH

o0Jj 1

0

0DJ

Average Interior Paint Lead Concentration

Participating Properties

I

B C D E F G

Area
30

20 -

"

0)

_ CD

Q.

x 10

111

d>

D)

* c

I

 5H

Average Exterior Paint Lead

Participating Properties

I

T 4,

C D

Area
Average Lead in Drinking Water

Kids First Program Properties

30

25 -

20 -.

O)

TO

0)

15 -

10 -

5 - T

D

Area
35

Lead in Drinking Water

Bornschein Study (1991) Data

30 ---

D> 25 --

0)

o

O

15 --•

10

0 -r

C D E

Area
ATTACHMENT 2

Spreadsheets for P10 and P15 Calculation

Based on a Lognormal Blood Lead Distribution

Two alternative Excel spreadsheets are attached for use in calculation of blood lead statistics,

assuming the blood lead distribution is lognormally distributed.

The spreadsheet entitled "PbB LN Fit-No Censoring.xls" should be used if none of the data

points in the data set are below the analytical detection limit. This sheet derives the distribution

parameters using maximum likelihood (MLE) methods, and the goodness of fit (GOF) test is

based on the Anderson-Darling statistic. This GOF method is selected because it assigns higher

weights to observations in the tails of the distributions than other goodness of fit methods.

The spreadsheet entitled "PbB LN Fit-With Censoring.xls" should be used if one or more of the

data points in the data set are below the analytical detection limit. This sheet derives the

distribution parameters using linear regression, and the goodness of fit test is based on the chi

squared statistic. This approach is somewhat less robust that use of the MLE and Anderson

Darling methods, but techniques for application of these methods to censored data are not

established.
ATTACHMENT 3

I

I

I

I

I

I

•

Estimation of Type I and Type II Error Rates

Using Monte Carlo Simulation

I

I

I

I

I

I

I

I

I

I

I

I
1.0 INTRODUCTION

The basic process to be used for evaluation of the performance of the LCCHP is summarized

below:

1. Fit a blood lead data set from a Statistical Unit to a continuous parametric distribution

(e.g., lognormal)

2. Based on the best fit parameters, calculate the P10 and PI5 values

3. Compare the P10 value to RAO-1 (5%) and the P15 value to RAO-2 (1%)

4. Declare the remedy successful for that Statistical Unit if the running 3-year average

value for P10 is 5% or less and the running 3-year average value for PI5 is 1% or

less, given compliance with other requirements for participation rate and

representativeness.

Because the values of P10 and P15 are based on blood lead data sets that are subject to

variability over time and space, random variations in the blood lead values of the program

participants could lead to the chance of making an incorrect decision. These errors are of two

types:

Type I Error:

Deciding that the RAOs have been satisfied for a Statistical Unit when actual blood lead

values do not satisfy the RAOs

Type II Error:

Deciding that the RAOs have not been satisfied for a Statistical Unit when actual blood

lead values do satisfy the RAOs

In order to evaluate the chances of making a Type I or a Type II error for a Statistical Unit under

the proposed process, Monte Carlo simulation was employed, as described below.

2.0 MONTE CARLO SIMULATION

All Monte Carlo simulations were performed using a commercially available software package

(Crystal Ball, available from Decisioneering. Inc) as an add-on to Microsoft Excel. The basic

procedure used to assess the chance of Type I and Type II errors was as follows:

1. A "true" distribution of blood lead values was defined by specifying the true P10 value

and the true geometric standard deviation (GSD).

2. From this specified distribution, a random sample of n blood lead values was drawn.

3. From this random set of blood lead values, the best fit parameters of a lognormal

distribution were calculated, and the sample-specific P10 value was calculated

4. Steps 2 and 3 were repeated many times (10,000)

5. If the true P10 value was above 5%, the fraction of all trails that yielded a calculated P10

value of 5% or less was calculated. This is the Type I error rate.
I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

6. If the true P10 value was above 5% or below, the fraction of all trails that yielded a

calculated P10 value above 5% was calculated. This is the Type II error rate.

This process was repeated for a series of alternative value of P10 and GSD, testing in each case

how the Type I and Type II error rates depend on a) the number of years across which a running

average is calculated, and b) the number of participants each year.

3.0 RESULTS

3.1 Averaging Time

The first issue investigated was how error rates depended on the number of years used to

calculate the running average. Based on blood lead data collected to date, a typical value for

GSD is about 1.7, and the typical number of participants per Statistical Unit is 80 or more per

year. To be conservative, a value of 60/year was assumed. The value of P10 was set to a series

of values ranging from 3% to 8%, and five separate random data sets (corresponding to 5

consecutive years of data) were drawn. Then the running average P10 value was calculated for

averaging times of 1, 2, 3, 4, and 5 years.

The results are shown below, and are presented graphically in Figure A3-1.

True P10

3%

4%

5%

6%

7%

8%

Error Rate (%)

1 Yr

10.6

28.7

48.5

34.6

21.3

11.6

2Yr

4.5

23.5

50.2

26.3

11.0

4.0

3Yr

2.5

19.6

51.2

21.2

6.4

1.7

4Yr

1.3

16.4

52.9

17.2

3.8

0.7

5Yr

1.2

14.2

53.9

14.1

2.3

0.2

True P15

0.50%

0.75%

1.00%

1.25%

1.50%

2.00%

Error Rate {%)

1 Yr

15.6

32.7

48.6

38.0

28.4

14.5

2Yr

9.6

31.4

53.9

29.0

16.5

5.4

3Yr

6.2

29.4

56.1

22.4

11.1

2.2

4Yr

4.3

27.2

58.9

18.7

7.0

1.0

5Yr

3.1

25.1

60.6

16.0

4.8

0.4

As seen in the upper panel, if the true P10 value is only slightly higher than RAO-1 (e.g., 6%),

there is about a 21% chance that the measured three-year running average P10 value would be

5% or less, leading to a Type I error (deciding the site was in compliance when it really was not).

However, if the difference between the true P10 value and RAO-1 is larger (e.g., 7% or higher),

then the probability of a Type I error becomes quite small (about 6% based on a 3-year

averaging period). Similarly, if the true P10 value is only slightly below RAO-1 (e.g. 4%), then

the probability of a Type II error (declaring the data to be higher than 5% when it really is not) is
about 20% based on a 3-year averaging period, and this value decreases at the true P10 value

becomes smaller. A similar pattern exists for errors in the PI5 value (lower panel). These error

rates are not sensitive to the assumed GSD value, but do tend to decrease if the number of blood

lead program participants each year is higher than the minimum value of 60/year that was

assumed (see below).

3.2 Number of Participants per Statistical Unit

The second issue investigated was how error rates depended on the total number of participants

in each year and for the three-year averaging time. These calculations were all based on an

assumed running 3-year averaging time for participation numbers and a GSD of 1.7 for each of

the three annual data sets. The results are shown below and are presented graphically in Figure

A3-2:

True P10

3%

4%

5%

6%

7%

8%

Error Rate (%)

20/yr

13.8

32.6

53.3

30.1

17.7

10.5

40/yr

5.5

23.9

52.7

24.6

10.4

4.1

60/yr

2.4

18.6

51.2

20.4

6.1

1.7

80/yr

1.1

14.8

51.6

16.6

3.9

0.7

100/yr

0.7

11.5

51.9

14.2

2.5

0.3

TrueP15

0.50%

0.75%

1.00%

1.25%

1.50%

2.00%

Error Rate (%)

20/yr

25.1

44.3

59.1

28.4

19.7

9.6

40/yr

13.3

35.6

57.4

25.8

14.4

4.5

60/yr

7.0

29.3

56.2

23.5

10.6

2.0

80/yr

3.8

24.9

55.4

20.6

8.3

0.9

100/yr

2.1

21.4

55.1

19.2

6.5

0.5

As shown in the upper panel, estimated Type I error rates for P10 values decrease as the number

of participants increases. If the number of participants is about 80 per year (about what is

expected), the Type I error rate is about 17% if the true P10 value is 6%, and about 4% if the true

P10 value is 7%. If the number of participants decreases to about 40 per year, Type I error rates

are higher, reaching 25% for a true PI 0 of 6% and 10% for a true P10 of 7%. Likewise, if the

true P10 is 4%, Type II error rates are generally acceptable (15%) when there are 80 participants

per year, but become marginal (24%) when there are only 40 participants per year. A similar

pattern exists for P10 error rates (lower panel).
FIGURE A3-1. DEPENDENCE OF ERROR RATE ON AVERAGING TIME

P10 ERROR RATE AS A FUNCTION OF AVERAGING TIME (N=60/yr)

60

50

40

S

& 30

i

UJ

20

10 -

Type II Error

(Declare P10 unacceptable

when It Is acceptable)

3% 4%

Type I Error

(Declare P10 acceptable

when it Is not acceptable)

5% 6%

TrueP10(ug/dL)

7%

Yean

Year 2

70 -

60

so

sfl]

o:

40

30

20

10

0

P15 ERROR RATE AS A FUNCTION OF AVERAGING TIME (N = 60/yr)

Type II Error

(Declare P15

unacceptable when it is

acceptable)

Type I Error

(Declare P15 acceptable

when it is not

acceptable)

0.50% 0.75% 1.00% 1.25%

TrueP15(ug/dL)

1.50% 2.00%
FIGURE A3-2. DEPENDENCE OF ERROR RATE ON NUMBER OF PARTICIPANTS

60

50

40

30

20

10

P10 ERROR RATE AS A FUNCTION OF PARTICIPATION RATE (3-YR PROGRAM)

Type II Error

(Declare P10 unacceptable

when it is acceptable)

3% 4%

Type I Error

(Declare P10 acceptable

when it is not acceptable)

5% 6%

True PI 0 (ug/dL)

7% 8%

P15 ERROR RATE AS A FUNCTION OF PARTICIPATION RATE (3-YR PROGRAM)

70 -i60

50

o

I

40

30

20

10

Type II Error

(Declare P15 unacceptable

when it Is acceptable)

0.50%

Type I Error

(Declare P15

acceptable when it Is

not acceptable)

80/yr

0.75% 1.00% 1.25%

True P1 5 (ug/dL)

1.50% 2.00%
ATTACHMENT 4

Estimation of Recruitment Rate

As discussed above, achieving an adequate participation rate is an important performance goal of

the LCCHP, and one element of that goal is achieving an adequate recruitment of new

(previously untested) participants in to the program each year. However, measurement of the

recruitment rate is not simple, since a child is counted as having participated if he/she is tested

anytime during a 6-year period, rather than during some specified narrow time window. In

addition, the population of children changes each year, as new children are born or move into the

site, and as children move away or "graduate" (i.e., reach an age of 73 months or more).

In order to evaluate the relative merits of several alternative metrics that might be used to

quantify recruitment rate, a screening-level population model was constructed. For the purposes

of this model, the following definitions and terms are employed:

"recruitment" is defined as the first-time donation of a blood sample by an eligible child

"eligible child" is defined as a child living in the site between 6 months and 72 months of

age who has not previously participated in the LCCHP blood lead program

"newly eligible" means a child who is bom to a family residing in the site or a child age 6

to 72 months who moves into the site

For simplicity, the model assumed a "stationary" population. That is, the total number of

children age 6-72 months who enter the population each year is equal to the number who move

away or "graduate". Based on this assumption, the model was constructed as follows:

Let N equal the total number of children age 6-72 months of age who live in the site.

Because of the assumption that the population is stationary, N is constant and does not

change over time

Let Ne(t) equal the number of eligible children in year t

Let p(t) equal the number of eligible children who are tested in year t

Let q equal the number of children who enter the population each year. Because of the

assumption that the population is stationary, this is also the number of children who leave

the population each year.

Based on these definitions, the recruitment rate (RR) in year t is given by:
RR(t) = p(t)/Ne(t)

The number of eligible children in year t is equal to the number of eligible children in year t-1,

plus the number of newly eligible children who join the population, minus the number of eligible

children who were tested and the number who leave the population without having been tested:

Ne(t) = Ne(t-l) + q - p(t) - q*(l - p(t)/Ne(t-1))

Thus:

PR(t) = p(t) / [Ne(t-l) + q - p(t) - q*(l-p(t)/Ne(t-l)]

In order to implement the model, the value of N was assumed to be 500, and the value of q was

assumed to be 80. At t = 0, the value of Ne(t) = N = 500. For simplicity it was assumed that p(t)

is approximately constant over time, and has a value of 50. These values are arbitrary, but are

fairly realistic based on the best information available for the community.

Based on these assumptions, the value of RR as a function of time is as shown in Figure A4-1.

As seen, the initial value of RR is 50/500 = 10%, but this value increase over time as eligible

children are tested in subsequent years. Ultimately, the recruitment rate plateaus at a value of

62.5%, which is equal to the number of newly tested children each year divided by the number of

newly eligible children entering the population each year:

Plateau value of RR = p/q

The plateau value is always given by the ratio p/q, regardless of the assumed numeric values of

N, q and p. The time to reach the plateau decreases as the value of p/q increases.

Even though this model is highly simplified, it reveals that the ratio of p to q is a useful indicator

of the recruitment rate, and this ratio was selected as the metric for use by the LCCHP.
FIGURE A4-1. POPULATION MODEL RESULTS

70%

60% -

g

2 50% -

| 40%

1

o> 30% -

O.

Q.

20%

10%

0%

- Yearly Rate

-Tested/New

10 15 20

Year of Program

25 30 35
ATTACHMENT 5

Data and Example Calculation of Annual Recruitment Rate
Lake County Year 2000 Census Data Compilation for Young Children (0 to 72 months)

Percentage of Lake County's Young Children Who Reside within OU9

Census Area

Tract 961 6

Tract 961 7

Tract 961 8

Lake County

Block 201 2, Tract 961 8

Tract 9617 and Block 2012

Description

County area north of Leadville

Leadville and outlying areas to south

County area approximately southwest of Cal. Gulch

Entire county area

Includes all of Lake Fork Trailer Park

All areas within OU9

Within OU9?

no

yes

partially

yes

yes

Number of Children 0 to

72 Months Old

210

428

90

728

58

486

Percentage of Young

Children County Wide

0.29

0.59

0.12

1.00

-

0.67

Recruitment Rate in LCCHP Blood Lead Monitoring Program (within OU9 boundaries)

Year

1993

1994

1995

1996

1997

1998

1999

2000

2001

Live-births Count

92

96

112

114

100

129

133

139

NA

Kindergarten Count

113

124

117

102

98

115

106

New Children

Moved In

10

2

3

-8

New Eligible Children

Lake County

110

131

136

131

OU9

74

88

91

88

First-time Tested

OU9

152(1994+1995)

149

145

88

77

87

61

Recruitment Rate Index

OU9

119%

88%

95%

69%

Notes:

Live-births Count from Lake County Health Department.

Kindergarten Count from The Center based on number of children 5 years and older on October 1 of calendar year.

New Children Moved In is the kindergarten count this year minus the live births 5 years previous.

New Eligible Children/Lake County is the sum of # of live births from previous year and # of children moved into site.

New Eligible Chi/dren/OU9 represents 67 percent of Lake County value.

Participation Rate is the first-time tested count divided by the estimate of new eligible children in OU9.

Italics indicates estimated values. NA = Not Available.
IIIIIIIIIIIIIIIIIII

ATTACHMENT 6

CRYSTAL BALL SPREADSHEET FOR ESTIMATING

UNCERTAINTY IN P10 ESTIMATES
IIIIIIIIIIIIIIIIIII

ATTACHMENT 7

EXCEL SPREADSHEET FOR EVALUATION

OF SOIL LEAD DISTRIBUTIONS
I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

TEST OF SOIL LEAD REPRESENTATIVENESS

A statistical test will be performed to determine if the population of children who participate in

the blood lead program are from residences that do not adequately represent the upper end of the

soil lead distribution in the statistical unit. Let Fl and F2 represent the true (but unknown)

fraction of each population that is above some specified soil lead level. The input parameters of

the test are the following:

Data Set

Soil lead values from blood

lead program participants

All soil lead values from the

Statistical Unit

Number

nl

n2

Observed proportion greater

than the selected value

PI

P2

For large nl and n2, the probability that Fl is lower than F2 is estimated by:

p = 1 - NORMSDIST(z) where z = (P1-P2) / sqrt (P1 *( 1 -P1 )/n 1 + P2( 1 -P2)/n2)

The power of this test to detect a difference of size delta between Fl and F2 at the 5%

confidence limit is estimated by:

Power = NORMSDIST(Zb) where Zb = delta/sqrt((Pl*(l-Pl)/nl + P2(l-P2)/n2) - 1.645

The usual decision rule for declaring that Fl is lower than F2 is that p < 0.05. However, if

power of the test is low (this is mainly a function of nl and n2), then there is decreased

confidence in decisions where p > 0.05. In order to address this, the decision rule used in this

program is:

Declare Fl to be lower than F2 if p < a, where a = 0.05 / power.

In this case, Performance Standard 4 will not be satisfied unless it is not possible to address the

problem in a practical way.
IIIIIIIIIIIIIIIIIII

ATTACHMENT 8

DATA REDUCTION APPROACH
1

1

1

1•

1

•

1

•

•

1

1

MEMO

To: Rebecca Thomas

From: Bill Brattin

Date: 12/17/97

Subject: Data Reduction Approach

Rebecca:

In accord with the agreements reached during the conference call today, the following is the

for calculation of summary statistics for blood lead data reported by the Lake County Health

Department.

Question 1:

What age group should be evaluated?

Plan:

plan

Children age 6-72 months (inclusive) will be included in calculation of summary statistics. The

age at time of blood sampling will be calculated by subtracting Date of Birth from Sample Date,

1

1

1

1

1

1

1

and the result will be expressed in months (rounded to the nearest whole month).

Question 2:

What restrictions should be placed on length of time at current residence and/or length of time in

Leadville?

Plan:

Blood lead data for children who have resided in Leadville for less than 3 months should be

excluded in the calculation of summary statistics.

Question 3:

What areas should be included in statistical summaries?

Plan:

Overall summary statistics presented in the Proposed Plan will include Areas A-G only.

Statistics for Area H may be helpful later.
I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

Question 4:

What are the approximate quantitation limits for capillary blood and venous blood? How should

samples with values at or below the quantitation limits be handled?

Plan:

Based on available information, it appears that the detection limit for both venous and capillary

samples was about 2 ug/dL in 1996 and earlier, and is about 1 ug/dL for data reported in 1997.

(This is due to a change in laboratories). Unless data become available to the contrary, all values

of 2 ug/dL reported in 1996 and earlier and all values of 1 ug/dL reported in 1997 and later will

be assumed to be non-detects and will be evaluated by assigning a value equal to 1/2 the detection

limit.

Question 5:

How should multiple blood lead data from a specific child be evaluated?

Plan:

a) Data for a child should be averaged within a year, but not across years.

b) If all of the samples within a year are the same type (either capillary or venous), simply

average all values for the year. If both types of values are present within a year, average

only the venous data.

c) The age that is assigned to the average blood lead should be the average for the actual

ages at the time of each sample used.

d) If a child moves within a year so that some blood lead data within the year apply to one

area and other data apply to a different area, assign the mean blood lead value to the area

where the child resided when the first blood sample was taken.

e) Because of the potential bias introduced by the tendency for children with elevated blood

leads to undergo repeat testing across multiple years, two types of analysis are

appropriate:

« All children (whether or not they have been tested in previous years)

Only children who have not been tested in previous years

These two types of analysis should be clearly labeled so they are not confused with

each other.
I

I

I Question 6:

What is the meaning of zero entered under blood lead in the database? Does this mean that a

• reliable analysis was not achieved?

Plan:

• A zero means that there was insufficient blood volume to perform an analysis. All "missing"

values will be removed from the data set.

I Question 7:

• Should the reported statistic of central tendency be the arithmetic mean or the geometric mean?

Proposal:

| The geometric mean will be reported.

I

I

I

I

I

I

I

I

I

I

I
TARGET SHEET

EPA REGION VIII

SUPERFUND DOCUMENT MANAGEMENT SYSTEM

DOCUMENT NUMBER: 2037052

SITE NAME: CALIFORNIA GULCH

DOCUMENT DATE: 07/01/2002

DOCUMENT NOT SCANNED

Due to one of the following reasons:

D PHOTOGRAPHS

D 3-DIMENSIONAL

D OVERSIZED

0 AUDIO/VISUAL

D PERMANENTLY BOUND DOCUMENTS

D POOR LEGIBILITY

D OTHER

D NOT AVAILABLE

D TYPES OF DOCUMENTS NOT TO BE SCANNED

(Data Packages, Data Validation, Sampling Data, CBI, Chain of Custody)

DOCUMENT DESCRIPTION:

CD - LCCHP ATTACHMENTS 2. 6. &7 (MAY 2002)

Contact the Superfund Records Center to view available document.

(303)312-6473