p.1 

OAR Box 1112 

Prepped by Ryan Dugan 

Document Number: 

33) ll-C-3a 

Docket Number: 

A-83-22 

Printed 4/16/2008 9:12:07 AM Sheet 33 of 97 
P.2 

PROTOCOL FOR ENCODING PROBABILITY JUDGMENTS ABOUT 

DOSE-RESPONSE CURVES FOR LEAD-INDUCED HEMOGLOBIN 

DECREMENTSAND FOR DISCUSSING POSSIBLE ADVERSE 

HEALTH EFFECTS OF ELEVATED EP 

1. INTRODUCTION 

2. LEAD-INDUCED HEMOGLOBIN DECREMENTS 

2.1 Definition of Hemoglobin Decrement as an Adverse Health Effect 

3. POPULATION AT RISK 

4. EXPOSURE CONDITIONS 

5. PHYSIOLOGICAL CONDITIONS 

6. FACTORS TO CONSIDER 

7. FACTORS TO KEEP IN MIND WHEN MAKING PROBABILITY JUDGMENTS 

7.1 Sequential Consideration of Information 

7.2 Effects of Memory on Judgment 

7.3 Estimating Reliability of Information 

7.4 Relation Between Event Importance and Probability 

7.5 Estimation of Probabilities 

7.6 Recommendations 

8. POSSIBLE ADVERSE HEALTH EFFECTS OF ELEVATED ERYTHROCYTE 

PROTOPORPHYRIN 

9. FINAL PREPARATION FOR ELICITATION OF PROBABILITY JUDGMENTS 

(To be provided during initial visit) 

ersion 84.9.21 
P.3 

PROTOCOL FOR ENCODING PROBABILITY JUDGMENTS ABOUT 

DOSE-RESPONSE CURVES FOR LEAD-INDUCED HEMOGLOBIN 

DECREMENTS AND FOR DISCUSSING POSSIBLE ADVERSE 

HEALTH EFFECTS OF ELEVATED EP 

1. INTRODUCTION 

The Environmental Protection Agency (EPA) is charged by the Clean Air Act 

with setting and revising National Ambient Air Quality Standards (NAAQS) for selected 

pollutants, at levels sufficient to protect the public health with an adequate margin of 

safety. As you know, the scientific bases for NAAQS are presented and reviewed in 

criteria documents. In support of the forthcoming revision of the lead NAAQS, EPA has 

just prepared a new .\ir Quality Criteria for Lead. It presents scientific evidence from 

which the most susceptible populations can be determined and from which various 

adverse health effects can be identified. The criteria document summarizes and 

evaluates the available clinical, epidemiological, and animal or toxicological laboratory 

evidence with regard to the physiological and adverse health effects of lead, and 

therefore represents our most up to date knowledge on lead effects. 

As one aspect of the review process, EPA assesses health risks by identifying the 

most sensitive populations for each pollutant, and estimating probabilistically the 

numbers of people in the populations who may suffer each of various well-defined 

adverse health effects attributable to the pollutant. It is believed that information about 

the health risks associated with various potential standards will aid the EPA 

administrator in selecting that standard which, in his judgment, protects the public 

health with an adequate margin of safety. 

Because the risk estimates that EPA seeks are often based in part on doseresponse relationships and lead exposure estimates which are uncertain, it is necessary to 

make probability judgments about relevant dose-response functions based on the 

available evidence, and to probabilistically estimate lead exposure under alternative 
P.4 

NAAQS. Obtaining the health risk estimates then involves combining dose-response and 

exposure probability estimates. 

The problem of estimating dose-response relationships is similar to that which 

exists in clinical medicine when there do not exist data that bear precisely on the 

patient's problem. In that case it is necessary to use scientific judgment to extrapolate 

from the data to make a best decision for the patient. Here, too, it is necessary to use 

scientific judgment to extrapolate from the available data. The extrapolation is not 

certain and, therefore, we will aid you to represent your opinion probabilistically. 

Furthermore, since the extrapolation depends on one's interpretation of the literature, 

different people will have different judgments. For each health effect we intend to 

obtain the probabilistic judgments of about five experts to sample the range of respected 

opinions. The model for estimating risks will not merge these judgments into a single 

average judgment, but rather will estimate the range of risks based on the range of 

judgments. If we as risk analysts do our job properly, then not only will we be able to 

show the EPA administrator the range of estimates based on the range of judgments, but 

we will also be able to show some of the sources of the disagreements. Indeed, a side 

benefit of this exercise in which we probe your knowledge in a structured manner may be 

to help identify sources of greatest disagreement. 

Based on the evidence in the lead criteria document, two populations have been 

identified as being most susceptible to the effects of lead intoxication. One is children 

from birth through the seventh birthday, and the other is pregnant women, or more 

precisely, the fetuses being carried by the pregnant women. A number of adverse health 

effects have been identified for which we would like to estimate dose-response functions 

including the one discussed below. 

2. LEAD-INDUCED HEMOGLOBIN DECREMENTS 

We would like to represent in probabilistic form your opinion about the location 

and shape of dose-response functions for certain well defined levels of lead-induced 
P.5 

hemoglobin reduction in the population of U.S. children from conception through the 

seventh birthday. We realize that available data do not fully define these functions; if 

they did, then it would be unnecessary to obtain your and other experts' opinions about 

them. Nevertheless, if the population, the exposure conditions, and the health effects 

are all precisely defined, then such functions in fact exist and we would like your best 

judgment about what they would look like if the data could be collected. 

2.1 Definition of Hemoglobin Decrement as an Adverse Health Effect 

There are differences of opinion, of course, as to what degree of reduction in 

hemoglobin level constitutes a health risk. The Clean Air Act makes it clear that EPA 

should set its standards to protect against adverse health effects. However, the 

reduction in hemoglobin level considered to be adverse may be different for regulatory 

purposes than for clinical action. Since it is generally agreed for children that a 

hemoglobin level of about 12 grams per deciliter is normal and about 9 grams per 

deciliter is anemic, we will specify two hemoglobin levels in that interval, namely 9.5 

and 11 gm/dl, and treat each as the physiological effect for purposes of specifying a 

dose-response function. We will elicit from you probability judgments about the shape 

and location of the dose-response curve for each of the two levels of effects in the 

sensitive population. 

3. POPULATION AT RISK 

We are defining the most susceptible group as all United States children from 

conception through their seventh birthday. However, you may believe on various grounds 

that younger and older children in this group have different dose-response functions for 

lead-induced hemoglobin reduction. If so, then we will elicit your judgments separately 

for the age groups 0-3 and 4-6. 
P.6 

4. EXPOSURE CONDITIONS 

We will be asking your judgment about population response rates at various 

blood-lead levels. Assume that the blood-lead level under consideration for a given 

judgment is in equilibrium as a result of a sufficiently long-term constant lead exposure 

without gross excursions above or below the stated level. 

5. PHYSIOLOGICAL CONDITIONS 

Because the effect of lead on hemoglobin level depends on many parameters of a 

person's system, it is necessary to specify assumptions about those parameters in the 

population. The effect of lead in the system depends on the person's nutritional and 

metabolic status. The incidence of iron deficiency is greater in children than in adults, 

and greater yet in children age 0-3 than age 4-6, with a wide range of iron levels in 

children up to their seventh birthday. Assume that the distribution of iron levels is that 

which you believe it actually to be in the age groups 0-3 and 4-6. The effect of lead also 

depends on the levels of zinc, copper, vitamins A, C, D, and E, calcium, phosphorous and 

magnesium. Assume in all cases that these nutrients are distributed in the two age 

groups as you believe they in fact are, taking into account the wide range of diets and 

nutritional levels of U.S. children. 

It is well established, of course, that erythrocyte protoporphyrin (EP) level is 

positively correlated with blood-lead level. However, the correlation is not perfect, and 

there is a range of EP values at any given blood lead. You may believe that the effect of 

lead varies with EP level. If so, assume that EP is distributed at each blood-lead level as 

you believe it in fact is. 

6. FACTORS TO CONSIDER 

In order to help you bring to mind the relevant evidence so that you may consider 

it systematically, and also in order to help us to interpret your judgments, we would like 
P.7 

to ask you to discuss briefly your interpretations of various aspects of the literature. 

First of all, could you tell us something about how, in your judgment, lead and iron 

interact to reduce hemoglobin levels and something about the relation between iron and 

lead levels in the body? Similarly, we would be interested in your opinions about the 

same questions with regard to zinc and lead and to calcium and lead. It appears that lead 

affects hemoglobin levels through three somewhat separate routes: interference with 

heme synthesis, interference with globin synthesis, and decreased life expectancy of 

erythrocytes. Briefly, what is your interpretation of the literature on these various 

effects of lead? EP level increases with iron deficiency, but it also increases with blood 

lead and is considered to be a proxy measure for the amount of lead recently cumulated 

in body tissue. Considering only that portion of EP elevation due to lead, how, in your 

judgment, is the blood-lead hemoglobin dose-response relationship affected by EP level? 

Finally, we know that some groups of children are at greater risk of exposure to 

lead due to their living in deteriorating pre-1950 housing or in urban areas with large 

amounts of vehicular traffic. These children would tend to have high lead levels, but do 

you believe that there are also independent reasons to think that their dose-effect curves 

will be different from children who live in other circumstances? For example, these may 

be the same children who tend to have poorer diets and less access to medical care. 

What is your opinion regarding the possibility of a correlation between increased 

exposure and increased susceptibility? Are there other factors to consider in thinking 

about the dose-response functions for lead-induced hemoglobin decrements that we 

should discuss now? 

7. FACTORS TO KEEP IN MIND WHEN MAKING PROBABILITY JUDGMENTS 

There is usually uncertainty associated with conclusions that we draw from 

research, and more generally in our everyday thinking. However, not everyone is aware 

of all the sources that should contribute to their uncertainty, nor are most people 
P.8 

familiar with the process of actually expressing their uncertainty in probabilistic terms. 

When an expert is asked to make probability judgments on socially important matters, it 

is particularly important that he or she consider the relevant evidence in a systematic 

and effective manner and provide judgments that represent his or her opinions well. 

Experimental psychologists and decision analysts have amassed a considerable 

amount of data concerning the way people form and express probabilistic judgments. The 

evidence suggests that when considering large amounts of complex information, most 

people employ simplifying heuristics and demonstrate certain systematic distortions of 

thought, i.e., cognitive biases, which adversely affect their judgments. The purpose of 

this section is to make you aware of these biases and heuristics so that, as much as 

possible, you can avoid them in making probability judgments. We will first review the 

most widespread biases and heuristics, and then offer some suggestions to help you 

mitigate their effects. 

7.1 Sequential Consideration of Information 

Generally, the order in which evidence is considered influences the final 

judgment, although logically that should not be the case. Of necessity, pieces of 

information are considered one by one in a sequential fashion. However, those 

considered first and last tend to dominate judgment. In part, initial information has its 

undue influence because it provides the framework which subsequent information is then 

tailored to fit. For example, people usually search for evidence to confirm their initial 

hypotheses; they rarely look for evidence that weighs against them. The later evidence 

has its undue effect simply because it is fresher in memory. 

Related to these sequential effects is the phenomenon of anchoring and 

adjustment. Based on early partial information, one forms an initial probability estimate 

regarding the event in question. This anchor judgment is then adjusted as subsequent 

information is considered. Unfortunately, such adjustments tend to be too 
P.9 

8 

conservative. In other words, too little weight is attached to information considered 

subsequent to the formation of the initial judgment. 

7.2 Effects of Memory on Judgment 

It is difficult for most people to conceptualize and make judgments about large, 

abstract universes or populations. A natural tendency is to recall specific members and 

then to consider them representative of the population as a whole. However, the specific 

instances often are recalled precisely because they stand out in some way, such as being 

familiar, unusual, especially concrete, or of personal significance. Unfortunately, the 

specific characteristics of these singular examples are then attributed, often incorrectly, 

to all the members of the population of interest. Moreover, these memory effects are 

often combined with the sequential phenomena discussed earlier. For example, in 

considering the evidence regarding the dose-response curve of a particular pollutant, you 

might naturally first think of a study you or a personal friend recently completed. Or 

you might think of a study you recently read, or one that was unusual and therefore 

stands out. The tendency might then be to treat the recalled studies as typical of the 

population of relevant research, ignoring important differences among studies. 

Subsequent attempts to recall information could result in thinking primarily of evidence 

consistent with the initial items you thought of. 

7.3 Estimating Reliability of Information 

People tend to overestimate the reliability of information, ignoring factors such 

as sampling error and imprecision of measurement. Rather they summarize evidence in 

terms of simple and definite conclusions, causing them to be overconfident in their 

judgments. This tendency is stronger when one has a considerable amount of intellectual 

and/or personal involvement in a particular field. In such cases, information is often 

interpreted in a way which is consistent with one's beliefs and expectations, results are 

overgeneralized, and contradictory evidence is ignored or underestimated. 
P.10 

7.4 Relation Between Event Importance and Probability 

Sometimes the importance of events, or their possible costs or benefits, 

influence judgments about the certainty of the events when, rationally, importance 

should not affect probability. In other words, one's attitudes towards risk tend to affect 

one's ability to make accurate probability judgments. For example, many physicians tend 

to overestimate the probability of very severe diseases, because they feel it is important 

to detect and treat them; and similarly, many smokers underestimate the probability of 

adverse consequences of smoking, because they feel that the odds do not apply to 

themselves personally. 

7.5 Estimation of Probabilities 

Another limitation is related to one's ability to discriminate between levels of 

uncertainty, and to use appropriate criteria of discrimination for different ranges of 

probability. One result of this fact is that people tend to estimate both extreme and 

mid-range probabilities in the same fashion, usually doing a poor job in the extremes. It 

helps here to think in terms of odds as well as probabilities. Thus, for example, changing 

a probability estimate from 0.510 to 0.501 is equivalent to a change in odds from 1.041:1 

to 1.004:1, but a change from an estimate of 0.999 to 0.990 changes the odds by a factor 

of about 10? from 999:1 to 99:1 The closer to the extremes (either 0 or 1) that one is 

estimating probabilities, the greater the impact of small changes. 

7.6 Recommendations 

Although extensive and careful training would be necessary to eliminate all the 

problems mentioned above, some relatively simple suggestions can help minimize them. 

Most important is to be aware of one's natural cognitive biases, and to try consciously to 

avoid them. 
P.11 

10 

To avoid sequential effects keep in mind that the order in which you think of 

information should not influence your final judgment. It may be helpful to actually note 

on paper the important facts you are considering and then to reconsider them in two or 

more sequences, checking the consistency of your judgments. Try to keep an open mind 

until you have gone through all the evidence, and don't let the early information you 

consider sway you more than is appropriate. 

To avoid adverse memory effects, define various classes of information that you 

deem relevant, and then search your memory for examples of each. Don't restrict your 

thinking only to items that stand out for specific reasons. Make a special attempt to 

consider conflicting evidence, and to think of data that may be inconsistent with a 

particular theory. Also, be careful to concentrate on the given probability judgment and 

do not let your own values (how you would make the decision yourself) affect those 

judgments. 

To accurately estimate the reliability of information, pay attention to such 

matters as sample size and power of the statistical tests. Keep in mind that data are 

probabilistic in nature, subject to elements of random error, imprecise measurements, 

and subjective evaluation and interpretation. In addition, the further one must 

extrapolate, or generalize, from a particular study to a situation of interest, the less 

reliable is the conclusion and the less certainty should be attributed to it. Rely more 

heavily on information which you consider more reliable, but do not treat it as "absolute 

truth." 

Keep in mind that the importance of an event or an outcome should not influence 

its judged probability. It is rational to let the costliness or severity of an outcome 

influence the point at which action is taken with respect to it, but not the judgment that 

is made about the outcome's likelihood. 

Finally, in making probability judgments, think primarily in terms of the measure 

(probability or odds) with which you feel more comfortable, but sometimes translate to 
P.12 

11 

the alternative scale, or even to measures of other events (e.g,, the probability of the 

event not happening). When estimating very small or very large likelihoods, it is usually 

best to think in terms of odds, which are unbounded, instead of probabilities, which are 

bounded. For example, one can more easily conceptualize odds of 1:200 than a 

probability of .005. 

8. POSSIBLE ADVERSE HEALTH EFFECTS OF ELEVATED ERYTHROCYTE 

PROTOPORPHYRIN (EP) 

It seems well established that there is a positive dose-response relationship 

between blood lead and EP. Although this elevated EP can be traced to heme synthesis 

interference, there is disagreement as to what, if any adverse health effects are 

associated with lead induced elevated EP. We would like to discuss your views on this 

issue in our second visit after we have completed encoding your probabilistic judgments 

concerning the hemoglobin blood-lead dose-response function. 

In your judgment, is lead induced elevated EP associated with alterations in 

neural tissue or with central nervous system dysfunction either directly or indirectly? If 

so, at what EP level or range of levels would you consider such effects to become 

adverse? 

Is there a relation between lead induced elevated EP and anemia of any sort 

beyond that which can be indexed by blood-lead level alone? 

Does elevated EP interfere with liver or renal function? If so, at what range of 

EP levels would you consider such effects to become adverse? Are there other adverse 

health effects that you believe may be associated with elevated EP? 
P.13 

12 

9. FINAL PREPARATION FOR ELICITATION OF PROBABILITY JUDGMENTS 

The shape and location of the dose-response curve as we defined it above is 

uncertain, because the existing data do not determine the dose-response relationship 

exactly. Yet, if we have defined the dose-response curve precisely, in a mathematical 

sense such a relationship does exist. Our goal is to have you represent probabilistically 

your own uncertainty about the location and shape of this mathematically existent 

function, based on your expertise and the available knowledge. In responding to the 

questions we will ask you, please think carefully about the relevant information reviewed 

in the criteria document, and consult it, the literature, or your files as you deem 

appropriate. 

The previous section suggests ways to think about the relevant data. The purpose 

of that section is to help minimize the biasing effects that frequently accompany the 

information overload naturally resulting from rapid consideration of large amounts of 

complex evidence. You may find it helpful to review the section or raise questions about 

the points made in it before we begin. 

Uncertainty about a dose response relationship can be represented 

probabilistically in two different ways. 

1. Uncertainty about the percentage of the defined population that 

would be affected by given blood-lead concentrations can be 

represented probabilistically. 

2. Uncertainty about the blood-lead concentrations that would be 

required to affect a given percentage of the defined population can 

be represented probabilistically. 

We will concentrate on one way at a time, focusing primarily on the first one. 

For these purposes it is helpful to imagine that everyone in the population has a 

specified blood-lead level that has become stabilized in the manner described above 
P.M 

13 

under Exposure Conditions. Then, as a result, some percentage of the population will 

suffer'the hemoglobin response. 

Now, in order for us to determine your uncertainty about the percentages of the 

population that would be affected by given blood lead levels we must introduce a 

definition. Let C be the given blood-lead concentration in question. 

Definition: R(C) is the percentage of the population for which a blood-lead level 

of C would cause the defined hemoglobin health effect. 

Thus, R(C) is precisely the percent of the population that would show a response if the 

entire population had blood-lead levels of C under the conditions defined above. R(C) is 

usually called the population response rate. 

The value of R(C) for a given C is uncertain, and we would like to obtain 

probability judgments from you about its possible values. We will elicit your judgments 

about the possible values of R(C) by specifying a particular percentage and having you 

consider how likely it is that R(C) is less than that value. To help you make your 

probability judgments, we will make use of a device called a probability wheel, which has 

adjustable sectors of blue and orange. We can read on the back of the wheel the 

percentage of the wheel that is each color. In making your judgments you are to imagine 

that the wheel is a perfectly fair random device, and that therefore the probability of its 

stopping with the pointer on blue is exactly represented by the relative area that is blue. 

We will then proceed as follows. For each blood lead concentration C, we will 

specify a particular percent r, and also set the probability wheel to have a specific 

relative area of blue. Then you are to consider carefully the question: 

Do you consider it more probable that the true population response rate R(C) is 

less than r or that the wheel would stop with the pointer on blue (on a random 

spin)? 

You can give one of three responses: 

(1) You judge it to be more probable that R(C) is less than r; 
P.15 

14 

(2) You judge it to be more probable that the wheel would stop with 

the pointer in blue; 

(3) You cannot judge either event as more probable than the other. 

For a particular concentration C and percent r, some wheel settings will have a 

small enough relative area of blue that you will feel confident making response (1). 

Other wheel settings will have a large enough relative area of blue that you will feel 

confident making response (2). The intermediate settings will be more difficult to 

judge. However, we will manipulate the wheel settings to find the one for which you feel 

most comfortable making response (3). 

Once we have determined the point at which you are most comfortable with 

response (3), and still focusing on the given blood lead C, we will specify a new percent 

r', and repeat the procedure. This will continue for the given C until we have specified 

various percents. Then we will have elicited one of the probabilistic representations of 

your uncertainty about the dose-response relationship we need. We will obtain the next 

probabilistic representation by specifying a new C and continuing as before. We will do 

this for a number of values of C. 

It frequently happens that an expert's judgments alter somewhat over the course 

of a session such as this, as he or she considers the evidence from various perspectives 

and thinks about the various responses called for. Hence, we will graph your responses 

and, at appropriate times, show them to you for your consideration and comparison. At 

these times you may wish to change some of the judgments you gave earlier. 

We should emphasize that the judgments we are asking you to make are not 

simple ones, nor of course are there known correct answers. Rather, we want your best 

and most considered judgment in light of the available relevant scientific data, 

Therefore, please reflect on the available data carefully, feeling free to consult the lead 

criteria document or other sources as you wish as you formulate your judgments. 
P.16 

15 

(Encode judgments for at least two concentrations of one dose-response function, 

then continue with instructions.) 

Recall that uncertainty about the dose-response function can also be expressed in 

terms of the blood-lead concentration necessary to produce the effect in a given 

percentage of the population r, if the entire population had the same blood-lead 

concentration, stabilized as the result of exposure in the manner discussed earlier. More 

specifically, proceeding as before, let us introduce a definition: 

Definition: C(R) is the blood-lead level that would cause the defined hemoglobin 

health effect in R percent of the population. 

The value of C(R) is uncertain for a given R, and we want to obtain your 

judgments about its possible values. Analogously to what we have already done, for a 

given R we will specify concentrations c and ask you to consider how likely it is that the 

true C(R) is less than the specified c. More specifically, utilizing the wheel as before, 

we will ask you the question: 

Do you consider it more likely that the true concentration C(R) is less than c, or 

that the wheel would stop with the pointer on blue (on a random spin)? 

You can give one of three responses: 

(1) You judge it more probable that C(R) is less than c; 

(2) You judge it more probable that the wheel would stop with the 

pointer in blue; 

(3) You cannot judge either event as more probable than the other. 

After determining the wheel setting at which you feel most comfortable with 

response (3), we will specify a new c and repeat the process. As before, for each R we 

will elicit your comparative judgments for various values of c.