OAR Box 1112 

Prepped by Ryan Dugan 

Document Number: 

98) II-B-2J 

Docket Number: 

A-83-22 

Printed 4/16/2008 10:34:18 AM Sheet 98 of 98 

p.1 
P.2 

A -<f-7- 2 1 

OUTLINE 

PROTOCOL FOR ENCODING PROBABILITY JUDGMENTS 

ABOUT OUTCOMES OF EXPERIMENTS TO ASSESS 

LEAD-INDUCED IQ DECREMENTS 

1. INTRODUCTION 1 

2. LEAD-INDUCED IQ DECREMENTS 3 

2.1 Central Nervous System and Behavioral Effects of Lead 3 

2.2 Definition of IQ Decrement as an Adverse Effect 4 

2.3 Hypothetical Ideal Experiments 4 

3. POPULATION AT RISK 6 

4. EXPOSURE CONDITIONS 7 

5. PHYSIOLOGICAL AND ENVIRONMENTAL CONDITIONS 7 

5.1 Physiological Conditions 7 

5.2 Environmental Conditions 7 

6. FACTORS TO CONSIDER 8 

Y FACTORS TO KEEP IN MIND WHEN MAKING PROBABILITY JUDGMENTS 9 

7.1 Sequential Consideration of Information 9 

7.2 Effects of Memory on Judgment 10 

7.3 Estimating Reliability of Information 10 

7.4 Relation Between Event Importance and Probability 11 

7.5 Estimation of Probabilities 11 

7.6 Recommendations 12 

8. FINAL PREPARATION FOR ELICITATION OF PROBABILITY JUDGMENTS 14 

(To be provided during initial visit.) 

Version 85.3.29 
P.3 

PROTOCOL FOR ENCODING PROBABILITY JUDGMENTS ABOUT OUTCOMES 

OF EXPERIMENTS TO ASSESS LEAD-INDUCED IQ DECREMENTS 

1. INTRODUCTION 

The Environmental Protection Agency (EPA) is charged by the Clean Air Act 

with setting and revising National Ambient Air Quality Standards (NAAQS) for selected 

pollutants, at levels sufficient to protect the public health with an adequate margin of 

safety. As you know, the scientific bases for NAAQS are presented and reviewed in* 

criteria documents. In support of the forthcoming review of the lead NAAQS, EPA has 

just prepared a new Air Quality Criteria for Lead. It presents scientific evidence from 

which the most susceptible populations can be determined and from which various 

adverse health effects can be identified. The criteria document summarizes and 

evaluates the available clinical, epidemiological, and animal or toxicological laboratory 

evidence with regard to the physiological and adverse health effects of lead, and 

therefore represents our most up to date knowledge on lead effects. 

As one aspect of the review process, EPA assesses health risks by identifying the 

most sensitive populations for each pollutant, and estimating probabilistically the 

numbers of people in the populations who may suffer each of various well-defined 

adverse health effects attributable to the pollutant. It is believed that information about 

the health risks associated with various potential standards will aid the EPA 

administrator in selecting that standard which, in his judgment, protects the public 

health with an adequate margin of safety. 

Because the risk estimates that EPA seeks are often based in part on doseresponse relationships and lead exposure estimates which are uncertain, it is necessary to 

make probability judgments about relevant dose-response functions based on the 

available evidence, and to probabilistically estimate lead exposure under alternative 

NAAQS. Obtaining the health risk estimates then involves combining dose-response and 

exposure probability estimates. 
P.4 

The problem of estimating dose-response relationships is similar to that which 

exists in clinical medicine when there do not exist data that bear precisely on the 

patient's problem. In that case it is necessary to use scientific judgment to extrapolate 

from the data to make a best decision for the patient. Here, too, it is necessary to use 

scientific judgment to extrapolate from the available data. The extrapolation is not 

certain and, therefore, we will aid you to represent your opinion probabilistically. 

Furthermore, since the extrapolation depends on one's interpretation of the literature, 

different people will have different judgments. For each health effect we intend to 

obtain the probabilistic judgments of about five experts to sample the range of respected 

opinions. The model for estimating risks will not merge these judgments into a single 

average judgment, but rather will estimate the range of risks based on the range of 

judgments. If we as risk analysts do our job properly, then not only will we be able to 

show the EPA administrator the range of estimates based on the range of judgments, but 

we will also be able to show some of the sources of the disagreements. Indeed, a side 

benefit of this exercise in which we probe your knowledge in a structured manner may be 

to help identify sources of greatest disagreement. 

Based on the evidence in the lead criteria document, two populations have been 

identified as being most susceptible to the effects of lead intoxication. One is children 

from birth through the seventh birthday, and the other is pregnant women, or more 

precisely, the fetuses being carried by the pregnant women. A number of adverse health 

effects have been identified for which we would like to estimate dose-response functions 

including the one discussed below. 

2. LEAD-INDUCED IQ DECREMENTS 

2.1 Central Nervous System and Behavioral Effects of Lead 

A large number of studies reviewed in the Criteria Document suggest that there 

are numerous central nervous system (CNS) and behavioral effects of lead exposure. 
P.5 

Considerable uncertainty surrounds all of these effects, however, because of the 

enormous difficulty in defining and measuring them, and in isolating them from the 

effects of covariates. Since our goal is to obtain probabilistic estimates about the shape 

of a dose-response curve for a particular well-defined effect if a sufficient amount of 

pertinent data could be collected, our first task is to select one CNS or behavioral effect 

that is of acknowledged importance and that can be well specified. 

The search for lead-induced effects has included studies of: EEG effects;, 

sensory motor, perceptual, and attentional deficits; cognitive decrements of various 

sorts; hyperactivity; negative classroom behaviors; and other effects that so far have 

been studied only in animal models. From this large assortment of effects, we have 

selected IQ decrement as the adverse effect upon which to focus. We are not considering 

IQ to be the only nor necessarily the best measure of cognitive abilities. Nor is it being 

considered as a surrogate for the other systems in which effects have been explored. 

(Because of its multifaceted nature it probably involves many of them.) Rather, we have 

selected IQ decrement because there are more data on this effect than on any other, and 

because its functional or "clinical" significance is clear. It is quite conceivable that as 

research continues in future years, other more "pure" CNS or behavioral effects will 

emerge that are obviously adverse and for which a substantial body of data has 

developed. Until such an event occurs, it appears that lead-induced IQ decrement is the 

most appropriate effect to consider. 

2.2 Definition of IQ Decrement 

Because it is generally thought that CNS effects of lead may be cumulative, it is 

necessary to specify the children's ages at which the IQ decrements are estimated. In all 

cases, assume that IQ tests are given on the children's seventh birthdays. Assume further 

that the WISC-R is the test employed. 
P.6 

A lead-induced IQ change cannot be measured directly for a given child, and 

therefore the change would have to be estimated statistically from suitable data. Thus, 

rather than ask you directly about dose-response functions, we will encode your 

judgments about the outcomes of hypothetical ideal experiments. If you agree with the 

assumptions on which the hypothetical experiments are based, then your judgments about 

the potential outcomes will lead naturally to probabilistic estimates about dose-response 

functions for IQ decrements. 

There are differences of opinion, of course, as to what IQ decrements should be 

considered adverse. The Clean Air Act makes it clear that EPA should set standards to 

protect against adverse health effects, but the level defined as adverse may be different 

for regulatory than for clinical or remedial purposes. We will not focus on particular 

magnitudes of IQ decrements, but rather will encode your probabilistic judgments about 

IQ distributions given various blood-lead levels. Then we will use your judgments to 

derive probabilistic statements about the percent of children at each lead level whose IQ 

scores are below any value of interest, such as, for example, below IQs of 70, 80, 90, 

or 100. 

2.3 Hypothetical Ideal Experiments 

Assume that at conception, subjects are randomly assigned to groups 

differentiated by blood lead level targeted for the third birthday. Members of each 

treatment group are exposed to lead from conception until their seventh birthdays, while 

members of a control group are sheltered from lead from conception until their seventh 

birthdays. The level of environmental lead assigned to a child is roughly constant for the 

seven years, however, each child's lead uptake is not constant, due to the changes with 

age of his or her physiology and behavior. Nevertheless, the experimental conditions are 

such that at their third birthdays, all the children in each group have essentially the same 

measured blood lead. Thus, groups differ in terms of mean blood lead on the third 
P.7 

birthday. Environmental lead levels necessary to yield a given blood-lead level at age 

three in a particular child remain constant through the seventh birthday. 

The experimental manipulation affects only lead exposure, and no other aspect of 

the children's lives. Then the WISC-R IQ test is administered to all children on their 

seventh birthdays. The children in each group have a distribution of IQ scores with some 

mean and standard deviation. We are interested in your probabilistic judgments about 

the IQ distributions for groups of children on their seventh birthdays, all of whom had 

specific blood-lead levels on their third birthdays. 

Note three features about this hypothetical experiment. First, it is similar to a 

longitudinal study; children are in a group from conception until their seventh birthdays. 

Second, it involves random assignment of children to groups, making it unnecessary to 

worry about covariates. Third, blood lead is measured at age three, and IQ at age seven. 

If you believe IQ effects of lead to be different for lower SES than for middle 

and upper SES subpopulations of children under these experimental conditions, then we 

will consider separate hypothetical experiments for the two subpopulations. Otherwise, 

we will consider only a single such experiment. 

SES is a complex variable. For simplicity, if we divide the population into low 

SES and middle or upper SES categories, let us define low SES as those children coming 

from households with incomes in the lowest 15 percent. 

Unless you disagree, we will also assume that the distribution of IQ scores is 

normal within each group. (The WISC-R was developed to yield normally distributed IQ 

scores for the general population, with mean IQ equal to 100 and a standard deviation of 

15.) 

Furthermore, if we consider only a single hypothetical experiment, sampling 

from the full population of children, then, unless you disagree, we will assume that the IQ 

standard deviation is 15 at each blood-lead level and we will only have to encode your 

probabilistic judgment about the mean IQ at age seven for each of several blood-lead 

levels at age three. 
P-8 , 

If we consider two hypothetical experiments, each sampling from a different SES 

subpopulation, then the IQ standard deviation will possibly be less than 15. Assuming 

that IQ standard deviation is the same at all blood-lead levels within SES subpopulations, 

we will have to encode your probabilistic judgment about the standard deviation for each 

SES group. 

Then, separately for each SES group or for the population as a whole, as 

appropriate, we will encode your probabilistic judgment about the mean IQ at age seven, 

of the control group. Finally, we will encode your probabilistic judgments about the 

differences in mean IQ at age seven between children with negligible blood lead (the 

control group) and children at each of several elevated blood leads at age three (the 

treatment groups). 

The following sections specify further the conditions to be assumed in this 

hypothetical experiment. 

3. POPULATION AT RISK 

Based on the Criteria Document, we can specify the most susceptible population 

as all United States children from conception through their seventh birthdays. We have 

already defined the effect as being measured on the seventh birthday, but on the assumption that younger children will reach their seventh birthdays, the effect is well-defined 

for the whole population. 

However, as already discussed, you may consider IQ effects of lead to be different for lower SES than for middle and upper SES subpopulations of children, resulting 

in different experimental outcomes for the two groups. If so, then we will elicit your 

judgments separately for lower SES children and for middle and upper SES children. 
P.9 

4. EXPOSURE CONDITIONS 

We will be asking your judgment about mean IQ values on the seventh birthdays 

at various blood-lead levels. Assume that the blood-lead levels under consideration for a 

given judgment have been measured on the children's third birthdays. Assume further 

that external environmental conditions supporting those levels have been more or less 

constant since birth, and that in interacting with the environment, the children exhibited 

the usual range of behaviors at each age. Thus, blood-lead levels were not necessarily, 

constant from birth to age seven, but exposure and behavioral factors were such that at 

age three, blood lead was at a specific level. Finally, assume that the changes in blood 

lead levels from birth until the seventh birthday are distributed as you believe they 

actually are. 

5. PHYSIOLOGICAL AND ENVIRONMENTAL CONDITIONS 

Because the effects of lead depend on many parameters, it is necessary to 

specify assumptions about those parameters in the population(s). 

5.1 Physiological Conditions 

The effect of lead in the system depends on the person's nutritional and metabolic status. Assume that the levels of iron, zinc, copper, vitamins A, C, D, and E, 

calcium, phosphorus, and magnesium are distributed within the population or within each 

of the two SES groups as you believe they in fact are, taking into account the wide range 

of diets and nutritional levels of children within the population or within each of the two 

SES groups. 

5.2 Environmental Conditions 

The effect of lead on IQ also may depend on numerous environmental and caregiving factors that vary within SES level, such as those assessed in the HOME scale 
P.10 

(Home Observation for Measurement of the Environment), parental IQ, and so forth. 

Assume that these factors are distributed within the population or within each of the two 

SES groups as you believe they in fact are. 

6. FACTORS TO CONSIDER 

In order to help you bring to mind the relevant evidence so that you may consider 

it systematically, and also in order to help us to interpret your judgments, we would like, 

to ask you to discuss briefly your interpretations of various aspects of the literature. 

How do you evaluate the research concerning the effects of low level lead exposure on 

cognitive and behavioral development? Is it your feeling that lead has a deleterious 

effect on this development over and above that which can be explained by other factors 

frequently associated with lead exposure? If so, do you believe that the effects of lead 

simply add to the effects of other factors, or that the effects of lead depend on the 

levels of other factors? In the latter case, what are the variables that lead exposure 

interacts with? A related question concerns your opinion about the relationship between 

exposure and susceptibility. That is, do you think that those children who are. at greatest 

risk of exposure due to their living in deteriorating pre-1950 housing or in urban areas 

with large amounts of vehicular traffic or due to other reasons, are also most susceptible 

to the effects of lead? For example, these may be the same children who also have 

poorer diets, less access to medical care, poorer care-giving environments, and less 

intellectual resources to fall back on. 

What is your opinion about the time course of lead exposure and of lead effects 

on CNS, cognitive and behavioral development? What are the relative effects of cumulative versus current exposure. Are the effects reversible? Is there a threshold for the 

effects? What are the implications of the animal model research, both the behavioral 

and the morphological, for human CNS, cognitive, and behavioral effects of lead? Are 

there other factors to consider in thinking about the dose-response functions for leadinduced IQ effects that we should discuss now? 
P.11 

7. FACTORS TO KEEP IN MIND WHEN MAKING PROBABILITY JUDGMENTS 

There is usually uncertainty associated with conclusions that we draw from 

research, and more generally in our everyday thinking. However, not everyone is aware 

of all the sources that should contribute to their uncertainty, nor are most people 

familiar with the process of actually expressing their uncertainty in probabilistic terms. 

When an expert is asked to make probability judgments on socially important matters, it 

is particularly important that he or she consider the relevant evidence in a systematicand effective manner and provide judgments that represent his or her opinions well. 

Experimental psychologists and decision analysts have amassed a considerable 

amount of data concerning the way people form and express probabilistic judgments. The 

evidence suggests that when considering large amounts of complex information, most 

people employ simplifying heuristics and demonstrate certain systematic distortions of 

thought, i.e., cognitive biases, which adversely affect their judgments. The purpose of 

this section is to make you aware of these biases and heuristics so that, as much as 

possible, you can avoid them in making probability judgments. We will first review the 

most widespread biases and heuristics, and then offer some suggestions to help you 

mitigate their effects. 

7.1 Sequential Consideration of Information 

Generally, the order in which evidence is considered influences the final 

judgment, although logically that should not be the case. Of necessity, pieces of 

information are considered one by one in a sequential fashion. However, those 

considered first and last tend to dominate judgment. In part, initial information has its 

undue influence because it provides the framework that subsequent information is then 

tailored to fit. For example, people usually search for evidence to confirm their initial 

hypotheses; they rarely look for evidence that weighs against them. The later evidence 

has its undue effect simply because it is fresher in memory. 
P.12 

10 

Related to these sequential effects is the phenomenon of anchoring and 

adjustment. Based on early partial information, one forms an initial probability estimate 

regarding the event in question. This anchor judgment is then adjusted as subsequent 

information is considered. Unfortunately, such adjustments tend to be too 

conservative. In other words, too little weight is attached to information considered 

subsequent to the formation of the initial judgment. 

7.2 Effects of Memory on Judgment 

It is difficult for most people to conceptualize and make judgments about large, 

abstract universes or populations. A natural tendency is to recall specific members and 

then to consider them representative of the population as a whole. However, the specific 

instances often are recalled precisely because they stand out in some way, such as being 

familiar, unusual, especially concrete, or of personal significance. Unfortunately, the 

specific characteristics of these singular examples are then attributed, often incorrectly, 

to all the members of the population of interest. Moreover, these memory effects are 

often combined with the sequential phenomena discussed earlier. For example, in 

considering the evidence regarding the dose-response curve of a particular pollutant, you 

might naturally first think of a study you or a personal friend recently completed. Or 

you might think of a study you recently read, or one that was unusual and therefore 

stands out. The tendency might then be to treat the recalled studies as typical of the 

population of relevant research, ignoring important differences among studies. 

Subsequent attempts to recall information could result in thinking primarily of evidence 

consistent with the initial items you thought of. 

7.3 Estimating Reliability of Information 

People tend to overestimate the reliability of information, ignoring factors such 

as sampling error and imprecision of measurement. Rather they summarize evidence in 
P.13 

11 

terms of simple and definite conclusions, causing them to be overconfident in their 

judgments. This tendency is stronger when one has a considerable amount of intellectual 

and/or personal involvement in a particular field. In such cases, information is often 

interpreted in a way which is consistent with one's beliefs and expectations, results are 

over-generalized, and contradictory evidence is ignored or underestimated. 

7.4 Relation Between Event Importance and Probability 

Sometimes the importance of events, or their possible costs or benefits, 

influence judgments about the certainty of the events when, rationally, importance 

should not affect probability. In other words, one's attitudes towards risk tend to affect 

one's ability to make accurate probability judgments. For example, many physicians tend 

to overestimate the probability of very severe diseases, because they feel it is important 

to detect and treat them; and similarly, many smokers underestimate the probability of 

adverse consequences of smoking, because they feel that the odds do not apply to 

themselves personally. 

7.5 Estimation of Probabilities 

Another limitation is related to one's ability to discriminate between levels of 

uncertainty, and to use appropriate criteria of discrimination for different ranges of 

probability. One result of this fact is that people tend to estimate both extreme and 

mid-range probabilities in the same fashion, usually doing a poor job in the extremes. It 

helps here to think in terms of odds as well as probabilities. Thus, for example, changing 

a probability estimate from 0.510 to 0.501 is equivalent to a change in odds from 1.041:1 

to 1.004:1, but a change from an estimate of 0.999 to 0.990 changes the odds by a factor 

of about 10 from 999:1 to 99:1 The closer to the extremes (either 0 or 1) that one is 

estimating probabilities, the greater the impact of small changes. 
P.14 

12 

7.6 Recommendations 

Although extensive and careful training would be necessary to eliminate all the 

problems mentioned above, some relatively simple suggestions can help minimize them. 

Most important is to be aware of one's natural cognitive biases, and to try consciously to 

avoid them. 

To avoid sequential effects keep in mind that the order in which you think of 

information should not influence your final judgment. It may be helpful to actually note 

on paper the important facts you are considering and then to reconsider them in two or 

more sequences, checking the consistency of your judgments. Try to keep an open mind 

until you have gone through all the evidence, and don't let the early information you 

consider sway you more than is appropriate. 

To avoid adverse memory effects, define various classes of information that you 

deem relevant, and then search your memory for examples of each. Don't restrict your 

thinking only to items that stand out for specific reasons. Make a special attempt to 

consider conflicting evidence, and to think of data that may be inconsistent with a 

particular theory. Also, be careful to concentrate on the given probability judgment and 

do not let your own values (how you would make the decision yourself) affect those 

judgments. 

To accurately estimate the reliability of information, pay attention to such 

matters as sample size and power of the statistical tests. Keep in mind that data are 

probabilistic in nature, subject to elements of random error, imprecise measurements, 

and subjective evaluation and interpretation. In addition, the further one must 

extrapolate, or generalize, from a particular study to a situation of interest, the less 

reliable is the conclusion and the less certainty should be attributed to it. Rely more 

heavily on information which you consider more reliable, but do not treat it as "absolute 

truth." 
P.15 

13 

Keep in mind that the importance of an event or an outcome should not influence 

its judged probability. It is rational to let the costliness or severity of an outcome 

influence the point at which action is taken with respect to it, but not the judgment that 

is made about the outcome's likelihood. 

Finally, in making probability judgments, think primarily in terms of the measure 

(probability or odds) with which you feel more comfortable, but sometimes translate to 

the alternative scale, or even to measures of other events (e.g., the probability of the 

event not happening). When estimating very small or very large likelihoods, it is usually 

best to think in terms of odds, which are unbounded, instead of probabilities, which are 

bounded. For example, one can more easily conceptualize odds of 1:200 than a 

probability of .005. 
P.16 

14 

8. FINAL PREPARATION FOR ELICITATION OF PROBABILITY JUDGMENTS 

The outcomes of the ideal experiments described above are uncertain. Existing 

data are relevant, but do not allow exact predictions. Our goal is to have you represent 

probabilistically your own uncertainty about the experimental outcomes based on your 

expertise and the available knowledge. In responding to the questions we will ask you, 

please think carefully about the relevant information reviewed in the criteria document, 

and consult it as well as the literature or your files as you deem appropriate. 

The previous section suggests ways to think about the relevant data. The purpose 

of that section is to help minimize the biasing effects that frequently accompany the 

information overload naturally resulting from rapid consideration of large amounts of 

complex evidence. You may find it helpful to review the section or raise questions about 

the points made in it before we begin. 

Uncertainty about the effects of lead exposure on mean decrement in IQ 

(relative to mean IQ at zero blood lead) can be represented probabilistically in two 

different ways. 

1. Uncertainty about the mean IQ decrement that would result when 

the exposed group has a given blood-lead concentration. 

2. Uncertainty about the blood-lead concentration that would be 

required to cause a given mean IQ decrement. 

We will concentrate on one way at a time, focusing primarily on the first one. 

For these purposes recall that everyone in the exposed population has a specified 

blood-lead level while everyone in the zero blood-lead population has negligible blood 

lead, as described above under Exposure Conditions. Then, in order for us to determine 

your uncertainty about the mean IQ decrement of the exposed group with a given bloodlead level relative to the negligible blood-lead group, we must introduce a definition. 

Let L be the given blood-lead concentration in question. 
P.17 

15 

Definition: D(L) is the mean IQ decrement of the exposed group with 

blood-lead level L relative to the negligible group. 

The value of D(L) for a given L is uncertain, and we would like to obtain 

probability judgments from you about its possible values. We will elicit your judgments 

about the possible values of D(L) by specifying a particular mean IQ decrement d and 

having you consider how likely it is that D(L) is less than that value. To help you make 

your probability judgments, we will make use of a device called a probability wheel, 

which has adjustable sectors of blue and orange. We can read on the back of the wheel 

the percentage of the wheel that is each color. In making your judgments you are to 

imagine that the wheel is a perfectly fair random device, and that therefore the 

probability of its stopping with the pointer on blue is exactly represented by the relative 

area that is blue. 

We will then proceed as follows. For each blood lead concentration L, we will 

specify a particular mean IQ decrement d, and also set the probability wheel to have a 

specific relative area of blue. Then you are to consider carefully the question: 

Do you consider it more probable that D(L) is less than d or that the 

wheel would stop with the pointer on blue (on a random spin)? 

You can give one of three responses: 

(1) You judge it to be more probable that D(L) is less than d; 

(2) You judge it to be more probable that the wheel would stop with 

the pointer on blue; 

(3) You cannot judge either event as more probable than the other. 

For a particular concentration L and mean IQ decrement d, some wheel settings 

will have a small enough relative area of blue that you will feel confident making 

response (1). Other wheel settings will have a large enough relative area of blue that you 

will feel confident making response (2). The intermediate settings will be more difficult 

to judge. However, for a fixed blood lead L, and mean IQ decrement d, we will 
P.18 

16 

manipulate the wheel settings to find the one for which you feel most comfortable 

making response (3). 

Once we have determined the point at which you are most comfortable with 

response (3), and still focusing on the given blood lead L, we will specify a new mean IQ 

decrement d', reset the wheel to a new relative area of blue, and proceed to obtain your 

judgments regarding the probabilistic relation between d' and D(L). This will continue for 

the given L until we have obtained comparisons of various mean values to D(L). Then we* 

will have elicited the probabilistic representation of your uncertainty about the mean IQ 

decrement for one blood-lead level. We will obtain the probabilistic representation of 

your uncertainty for the outcome of another experiment by specifying a new L and 

continuing as before. We will do this for a number of values of L. 

It frequently happens that an expert's judgments alter somewhat over the course 

of a session such as this, as he or she considers the evidence from various perspectives 

and thinks about the various responses called for. Hence, we will graph your responses 

and, at appropriate times, show them to you for your consideration and comparison. At 

these times you may wish to change some of the judgments you give earlier. 

Also, although the probability judgments are entirely your own, we must 

introduce one logical constraint: namely, that your final judgments are coherent in a 

sense that we can explain as we go along. It is quite common for initial judgments to 

exhibit some incoherencies. That is another reason we will consider together the graphs 

of your judgments. Our objective is to obtain a coherent set of judgments that represent 

your opinions well by the end of this elicitation process. You are not expected to give us 

such a set immediately. All of this will become clearer as we go along. 

We should emphasize that the judgments we are asking you to make are not 

simple ones, nor of course are there known correct answers. Rather, we want your best 

and most considered judgment in light of the available relevant scientific data. 
P.19 

17 

Therefore, please reflect on the available data carefully, feeling free to consult the lead 

criteria document or other sources as you wish as you formulate your judgments. 

(Encode judgments for two blood-lead concentrations, then continue with 

instructions.) 

Recall that uncertainty about the experimental outcomes can also be expressed 

in terms of the blood-lead concentration necessary to produce a given mean IQ 

decrement D, if the entire exposed population had the same blood-lead concentration, 

established in the manner discussed earlier. More specifically, proceeding as before, let 

us introduce a definition: 

Definition: L(D) is the blood-lead level that would cause a mean IQ 

decrement D in an exposed group relative to the negligible group. 

The value of L(D) is uncertain for a given D, and we want to obtain your 

judgments about its possible values. Analogously to what we have already done in 

obtaining the D(L) judgments, for a given D we will specify concentrations 8, and ask you 

I 

to consider how likely it is that the L(D) is less than the specified £.. More specifically, 

utilizing the wheel as before, we will ask you the question: 

Do you consider it more likely that L(D) is less than i, or that the wheel 

would stop with the pointer on blue (on a random spin)? 

You can give one of three responses: 

(1) You judge it more probable that L(D) is less than 8,; 

(2) You judge it more probable that the wheel would stop with the 

pointer on blue; 

(3) You cannot judge either event as more probable than the other. 

After determining the wheel setting at which you feel most comfortable with 

response (3), we will specify a new 9. and repeat the process. As before, for each D we 

will elicit your comparative judgments for various values of a. 
P.20 

18 

m 

We must encode your probabilistic judgment about the mean.IQ of children in the 

population, or in the two SES groups, who have negligible blood-level under the exposure 

conditions described previously. We start with a definition: 

Definition: Let p be the true population mean IQ for children with negligible 

blood lead. (Or, let y • be the true population mean IQ for children with negligible blood 

lead in SES group i.) 

The value of y or u • is uncertain, and we will obtain your judgment about its 

possible values. Analogously to what we have already done, we will specify a mean y and 

* **k 

ask you to consider how likely it is that u or y . is less than y. We will elicit your 

judgments about various values of u with the aid of the probability wheel, in the same 

manner as was done previously. 

If you felt it necessary to consider separate experiments for two SES groups, 

then you may believe that the within-group IQ standard deviation is different from 15. 

If so, we must encode your probabilistic judgment about that value. If you agree, we will 

assume that the standard deviation is unaffected by lead-level. However, you may feel 

that the standard deviation is different for the two SES groups. If so, we must encode 

your opinion separately about each case. 

As before, we start with a definition: 

Definition: Let oj* be the true IQ standard deviation for SES group i. 

The value of a-* is uncertain, and we will obtain your judgment about its possible 

values. Analogously to what we have already done, we will specify a standard deviation o 

and ask you to consider how likely it is that oj* is less than o. We will elicit your 

judgments about various values of o with the aid of the probability wheel, in the same 

manner as was done previously.