OAR Box 1112 

Prepped by RyanDugan 

Document Number: 

36) ll-C-3d 

Docket Number: 

A-83-22 

Printed 4/16/2008 9:12:07 AM Sheet 36 of 97 

p.1 
P.2 

ARGONNE NATIONAL LABORATORY J £ - 6 " 3 ^ 

9700 South Cass Avenue 

Argonne, Illinois 60439 

ANL/EES-TM-XXX 

ESTIMATING THE RISKS OF LEAD-INDUCED HEMOGLOBIN DECREMENTS 

UNDER CONDITIONS OF UNCERTAINTY: A METHODOLOGY, 

PILOT JUDGMENTS, AND EXAMPLE CALCULATIONS* 

by 

Ronald G. Whitfield 

and 

Thomas S. Wallsten** 

Energy and Environmental Systems Division 

Integrated Assessments and Policy Evaluation Group 

September 24, 1984 

work sponsored by 

U.S. ENVIRONMENTAL PROTECTION AGENCY 

Office of Air Quality Planning and Standards 

"Authorship is shared equally. 

*LL. Thurstone Psychometric Laboratory, University of North Carolina at Chapel Hill. 
P.3 

CONTENTS 

ABSTRACT 1 

1 INTRODUCTION 2 

1.1 Background 2 

1.2 Motivation 3 

1.3 Outline 3 

2 PROBABILITY ENCODING 5 

2.1 Goals of the Encoding Sessions 5 

2.2 Method 6 

2.2.1 Preparation 6 

2.2.2 Probability Encoding 7 

2.3 Subjects 10 

2.4 Results 10 

2.4.1 Pilot Subject 1 10 

2.4.2 Pilot Subject 2 13 

2.5 Discussion 16 

3 MATHEMATICAL FORMULATION 17 

3.1 Criteria 17 

3.2 The Best-Fitting and Equal-Variance Normal-On-Log-Odds 

Distributions 17 

3.3 Combining Blood-Lead Distributions with Probability Judgments 

about Dose-Response Functions 21 

4 ILLUSTRATIVE CALCULATIONS 25 

4.1 Fitting Normal-On-Log-Odds Distributions to Judgments 25 

4.1.1 Pilot Subject 1 25 

4.1.2 Pilot Subject 2 27 

4.1.3 Judgments of Other Experts 30 

4.2 Illustrative Risk Calculations 33 

5 COMMUNICATING THE RISK ASSESSMENT RESULTS TO ADECISION MAKER 36 

6 OBSERVATIONS AND RECOMMENDATIONS 38 

7 REFERENCES 40 

APPENDIX A: Draft Protocol for Encoding Probability Judgments 

about Dose-Response Curves for Lead-Induced 

Hemoglobin Decrements 41 

APPENDIX B: Detailed Mathematical Formulation 56 

APPENDIX C: Detailed Illustrative Calculations 66 
P.4 

FIGURES 

2.1 Subject l's Probability Judgments about Fraction of Population of U.S. 

Children Aged 0-3 Showing Hemoglobin Levels Less Than or Equal 

to 11 g/dl at Each Indicated Blood-Lead Level, L 12 

2.2 Subject 2's Probability Judgments about Fraction of Population of U.S. 

Children Aged 0-3 Showing Hemoglobin Levels Less Than or Equal 

to 11 g/dl at Each Indicated Blood-Lead Level, L 15 

3.1 A Hypothetical Family of Probabilistic Judgments about the Fraction of a 

Well-Defined Population That Will Suffer a Well-Defined Adverse Health 

Effect at Each Indicated Blood-Lead Level, L 18 

3.2 A Hypothetical Derived Probability Distribution about the Fraction of a 

Well-Defined Population That Will Suffer a Well-Defined Adverse Health 

Effect, Obtained by Combining Probability Judgments such as in 

Fig. 3.1 with a Blood-Lead Distribution such as the NHANES-II 23 

4.1 Equal-Variance Normal-On-Log-Odds CDFs for the Probability Judgments of 

Subject 1 That Were Previously Shown in Fig. 2.1 28 

4.2 The 0.05-, 0.50-, and 0.95-Cumulative Probability Estimates of Response 

Rate R at Various Blood-Lead Levels, Based on the Equal-Variance 

Normal-On-Log-Odds Distributions Fitted to the Judgments of 

Subject 1 29 

4.3 Regressed Equal-Variance Normal-On-Log-Odds CDFs for the Probability 

Judgments of Subject 2 That Were Previously Shown in Fig. 2.2, as 

Well as for Intermediate Lead Levels 31 

4.4 The 0.05-, 0.50-, and 0.95-Cumulative Probability Estimates of Response 

Rate R at Various Blood-Lead Levels L Based on the Regressed EqualVariance Nor rnal-On-Log-Odds Distributions Fitted to the Judgments 

of Subject 2 32 

4.5 Derived Risk Distribution Based on Equal-Variance Normal-On-Log-Odds 

Distributions Fit to Subject l's and Subject 2's Judgments, Showing 

in Each Case a Cumulative Probability F that Fraction R of the 

Population of U.S. Children Aged 0-3 Have Hemoglobin Levels 

Less Than or Equal to 11 g/dl 35 

C.l Best-Fitting Nor mal-On-Log-Odds CDFs and the Probability Judgments of 

Subject 1 That Were Previously Shown in Fig. 2.1 69 

C.2 Best-Fitting Normal-On-Log-Odds CDFs for the Probability Judgments of 

Subject 2 That Were Previously Shown in Fig. 2.2 72 

C.3 A Complete Set of Six Modeled Normal-On-Log-Odds CDFs for the Response 

Rate Variable Based on the Parameter Estimates in Table C.5 75 
P.5 

FIGURES (Cont'd) 

C.4 Derived Risk Distribution Based on Best-Fitting Normal-On-Log-Odds 

Distributions for Subject l's Judgments, Showing the Cumulative 

Probability F That Fraction R of the Population of U.S. Children 

Aged 0-3 Have Hemoglobin Levels Less Than or Equal to 11 g/dl 79 

C.5 Derived Risk Distribution Based on the Best-Fitting and Equal-Variance 

Normal-On-Log-Odds Distributions for Subject l's Judgments, Showing the 

Cumulative Probability F That Fraction R of the U.S. Children Aged 0-3 

Have Hemoglobin Levels Less Than or Equal to 11 g/dl 82 

C.6 Derived Risk Distribution Based on the Equal-Variance and Regressed 

Best-Fitting Normal-On-Log-Odds Distributions for Subject 2's Judgments, 

Showing the Cumulative Probability F That Fraction R of the U.S. Children 

Aged 0-3 Have Hemoglobin Levels Less Than or Equal to 11 g/dl 83 

C.7 Probability Density Functions for Derived Risk Distributions Based on 

the Equal-Variance Normal-On-Log-Odds Distribution Applied to the 

Judgments of Subjects 1 and 2 84 

TABLES 

2.1 Subject l's Probability Judgments, F, about Fraction, R, of Population 

of U.S. Children Aged 0-3 Showing Hemoglobin Levels Less Than 

or Equal to 11 g/dl at Each Indicated Lead Level, L 11 

2.2 Subject 2's Probability Judgments, F, about Fraction R, of Population 

of U.S. Children Aged 0-3 Showing Hemoglobin Levels Less Than 

or Equal to 11 g/dl at Each Indicated Lead Level, L 14 

4.1 R-Square Values for the Best-Fitting and Equal-Variance Normal-On-Log-Odds 

Distributions Fitted to the Probability Judgments of Subjects 1 

and 2 Conditional on the Indicated Blood-Lead Levels 26 

4.2 Regression r6 Values for the Fit of Best-Fitting and Equal-Variance 

Normal-On-Log-Odds Distributions to Judgments of Experts Encoded by 

Wallsten, Forsyth, and Budescu (1983) 33 

B.l Summary of Variables Pertaining to the Normal-On-Log-Odds Function 60 

C.l Normal-On-Log-Odds Transformation of Probability Judgments in Table 2.1 

for Subject 1 67 

C.2 Parameter Estimates, Goodness of Fit Statistics, and Mean Response Rates 

for Best-Fitting Normal-On-Log-Odds Distributions Fitted to the 

Judgments of Subject 1 68 
P.6 

TABLES (Cont'd) 

C.3 Parameter Estimates, Goodness of Fit Statistics, and Mean Response Rates 

for the Equal-Variance Normal-On-Log-Odds Distributions Fitted to the 

Judgments of Subject 1 70 

C.4 Parameter Estimates, Goodness of Fit Statistics, and Mean Response Rates 

for the Normal-On-Log-Odds Distributions Fitted to the Judgments of 

Subject 2 73 

C.5 Parameter Estimates, Goodness of Fit Statistics, and Mean Response Rates 

for the Linearly Regressed Normal-On-Log-Odds Distributions, Applied 

to the Judgments of Subject 2 74 

C.6 Parameter Estimates, Goodness of Fit Statistics, and Mean Response Rates 

for Normal-On-Log-Odds Functions with Equal Variance Fitted to the 

Encoded Judgments of Subject 2 Through the Use of Linear Regression 

Techniques as Described in the Text 76 

C.7 Calculations for a Median Estimate of the Fraction of the Population 

Affected, Based on Best-Fitting Normal-On-Log-Odds Distributions 

Fitted to the Judgments of Subject 1 78 

V I 
P.7 

ESTIMATING THE RISKS OF LEAD-INDUCED HEMOGLOBIN DECREMENTS 

UNDER CONDITIONS OF UNCERTAINTY: A METHODOLOGY, 

PILOT JUDGMENTS, AND EXAMPLE CALCULATIONS 

by 

Ronald G. Whitfield and Thomas S. Wallsten 

ABSTRACT 

This report focuses on probabilistic judgments about doseresponse relationships of lead exposure and subsequent hemoglobin 

decrements. It includes methods for obtaining the judgments, representing them in suitable mathematical form, and combining them with 

a measure of lead exposure to produce health risk estimates. The 

methodology is easily extended to other health effects, as well. 

Judgmental dose-response functions for lead-induced hemoglobin 

decrements were obtained for two pilot subjects as part of an 

exercise to test the viability of a protocol to obtain such judgments 

from health experts. The judgments were represented in mathematical form (the normal-on-log-odds function) to facilitate the process of 

combining them with NHANES-H blood-lead data (a measure of 

exposure to lead) to calculate illustrative health risk estimates for 

U.S. children aged 0-3 years. Attention is also given in the report to 

ways of summarizing and presenting large amounts of complex risk 

assessment data and calculations in a form understandable by and 

useful to decision makers. 

The pilot encodings and the ensuing calculations have 

provided valuable experience and preparation for planned encodings of 

the judgments of experts on the health effects of lead, and the 

development of risk estimates partially based on those judgments. 
P.8 

1 INTRODUCTION 

1.1 BACKGROUND 

The Environmental Protection Agency (EPA) is charged by the Clean Air Act 

with setting and revising National Ambient Air Quality Standards (NAAQS) for selected 

pollutants, at levels sufficient to protect the public health with an adequate margin of 

safety. The scientific bases for NAAQS are presented and reviewed in criteria documents. In support of the ongoing review of the lead NAAQS, EPA is preparing a new Air 

Quality Criteria for Lead. It presents scientific evidence from which the most 

susceptible populations can be determined and from which various adverse health effects 

can be identified. The criteria document summarizes and evaluates the available 

clinical, epidemiological, and animal or toxicological laboratory evidence with regard to 

the physiological and adverse health effects of lead, and therefore represents the most 

up to date knowledge on lead effects. 

As one aspect of the review process, EPA assesses health risks by identifying the 

most sensitive populations for each pollutant, and estimating probabilistically the 

numbers of people in the populations who may suffer each of various well-defined 

adverse health effects attributable to the pollutant. It is believed that information about 

the health risks associated with various potential standards will aid the EPA 

administrator in selecting that standard which, in his judgment, protects the public 

health with an adequate margin of safety. 

Because the risk estimates that EPA seeks are often based in part on doseresponse relationships and lead exposure estimates which are uncertain, it is necessary to 

make probability judgments about the relevant dose-response functions based on the 

available evidence and to probabilistically estimate lead exposure under alternative lead 

NAAQS. Obtaining the health risk estimates then involves combining the dose-response 

and exposure probability estimates. 
P.9 

1.2 MOTIVATION 

5 This report is primarily concerned with two of the three components of health 

risk estimates: representing probabilistic judgments about dose-response relationships, 

and combining these with a measure of exposure to lead to produce risk estimates. At 

this time, we do not treat the problem of probabilistically estimating exposure to lead 

under alternative lead NAAQS, but simply take the NHANES-II data to represent current 

blood-lead distributions. 

Probability judgments about dose-response functions for lead-induced hemoglobin 

decrements have been obtained from two pilot subjects as part of an exercise to test the 

viability of a protocol to obtain such judgments from recognized health experts some 

time in the near future. This paper presents the method for obtaining the judgments as 

well as the pilot judgments themselves, and illustrates how (1) such judgments can be 

represented in mathematical form; (2) they can be combined with exposure information 

to generate risk estimates; and (3) the risk information can be summarized in a form 

understandable by a decision maker. 

1.3 OUTLINE 

Section 2 presents the pilot probability encodings. It includes a description of 

the method by which the judgments were obtained, the results for both subjects, and a 

discussion of those results indicating why it is desirable to represent them 

mathematically. 

Section 3 summarizes the mathematics employed to represent the judgments and 

to calculate risk estimates. A full development of the mathematics is provided in an 

appendix to this report. 

Section 4 applies the tools presented in the previous section to the judgments of 

the pilot subjects and of experts who had participated in the study by Wallsten, Forsyth, 

and Budescu (1983). It presents the fits of equal-variance normal-on-log-odds 
P.10 

distributions to the judgments, and illustrative risk calculations based on combining these 

judgmental distributions with NHANES-II blood-lead distributions. Detailed discussion of 

alternative representations of the judgments is contained in an appendix. 

Section 5 illustrates how such risk estimates may be communicated to a decision 

maker, and Section 6 contains observations about the pilot encoding sessions, the 

mathematical representations, the subsequent risk estimates, and lessons learned which 

are relevant for future encoding efforts. 
P.11 

2 PROBABILITY ENCODING 

2.1 GOALS OF THE ENCODING SESSIONS 

The primary goal of the encoding sessions is to obtain experts' probabilistic 

judgments about dose-response functions for lead-induced hemoglobin decrements. 

Specifically, the experts will be aided in encoding for each of various blood-lead levels 

their subjective probabilities about the fraction of a population with hemoglobin levels 

less than or equal to a specified value. 

Because the shape and location of any dose-response function depends on 

numerous population and exposure condition parameters as well as on the exact definition 

of the adverse health effect, it is necessary first to specify these factors. Such 

specification precisely defines the dose-response function, and assures that any 

differences between experts in probability judgments is due to true differences of opinion 

and not merely to consideration of different conditions. 

A secondary goal of the encoding sessions is to discuss the literature on leadinduced hemoglobin decrements. This is to be done prior to the actual probability 

encoding, and serves three purposes. First, it aids the expert in reviewing the literature 

in his or her own mind before beginning the formal judgment process. Second, it provides 

a basis for possibly understanding conflicting judgments among experts, should they 

arise. Third, it provides an opportunity for the expert to make any points he or she feels 

may be useful to the users of the subsequent risk assessments. 

The actual probability encoding with recognized health experts is planned to take 

two sessions per expert, with two to four weeks intervening. The first session will last 

approximately eight hours and the second about four hours. It is anticipated that roughly 

one-third to one-half of the first session will be devoted to preparation and discussion, as 

indicated above, with the remainder spent on the actual probability encoding. The 

preparatory material will be reviewed in the second session, but most of the time will be 
P.12 

given to probability encoding. Based on the earlier work by Wallsten, Forsyth, and 

Budescu (1983), the probability judgments obtained in the second session will be the 

better ones to use in calculating risk estimates. However, for the present pilot study it 

was deemed sufficient to work with each expert in a single session of four hours. 

2.2 METHOD 

2.2.1 Preparation 

A complete protocol for encoding the probability judgments is attached as 

Appendix A to this report. This protocol was sent to the pilot subjects about one week 

prior to the scheduled sessions so that they could familiarize themselves with it. 

Sections 1-7 of the protocol discuss the hemoglobin level to be considered adverse for 

regulatory purposes, the population at risk, the population exposure conditions, the 

population physiological conditions, features of the relevant literature, and factors to be 

aware of when making probability judgments. (Section 8 of the protocol specifies how 

the probability judgments are to be encoded; the subjects were asked to ignore that 

section prior to the session. This section will not be sent out to the health experts.) 

Approximately one-and-a-half to two hours was spent with each subject 

reviewing the material in Sections 1 through 7 of the protocol, reviewing the literature, 

and discussing procedures for encoding judgmental probabilities. The stipulation of 

factors in those sections which define the dose-response functions about which 

probability judgments were to be elicited are as follows. 

Two hemoglobin levels were specified as adverse, 9.5 and 11 g/dl, but in this pilot 

study judgments were obtained only with respect to the function defined for hemoglobins 

less than or equal to 11 g/dl. The population at risk is all U.S. children from conception 

through their seventh birthdays. The subjects were given the option of judging separate 

dose-response curves for children aged 0-3 and 4-7, if they thought that appropriate. 
P.13 

They were further to assume that the blood-lead level under consideration for a given 

judgment was in equilibrium as a result of a sufficiently long-term constant lead 

exposure without gross excursions above or below the stated level. They were also to 

assume that iron, zinc, copper, vitamins A, C, D, and E, calcium, phosphorus, and 

magnesium were distributed in the population as they believe these minerals and vitamins 

in fact are. 

2.2.2 Probability Encoding 

Probability encoding commenced after the relevant literature and the factors 

defining the dose-response function had been thoroughly discussed and the expert had 

indicated whether he thought children aged 0-3 and 4-7 were described by the same or 

different functions. In this pilot study only a single function was encoded per expert, but 

in the actual sessions judgments will be obtained with respect to two or four functions 

(one each for 9.5 and 11 g/dl either for all children to their seventh birthdays or 

separately for the two age groups). 

Probability judgments were encoded regarding possible population response rates 

at blood-lead levels of 5, 15, 25, 35, 45, and 55 ug/dl. Only pilot subject 1 also provided 

judgments regarding possible blood-lead levels supporting response rates of 0.20 and 0.65. 

To determine the useful points in the dose-response space for encoding 

probabilities, each subject was first asked to supply broad "99% confidence interval" 

ranges for possible response rates at each of the six blood-lead levels. The analyst aided 

the expert in determining these ranges by posing suitable questions (e.g., following initial 

specification of a range the analyst would ask whether the expert could construct a 

plausible explanation if he were to learn one day that the true response rate was outside 

that range). The ranges for each blood-lead level were then divided into five equally 

spaced intervals for detailed encoding at six points. For example, if the response rate 

range for a particular blood-lead level was 0.20 to 0.70, subsequent encodings at that 

lead level would occur for response rates of 0.20, 0.30, 0.40, 0.50, 0.60, and 0.70. 
P.M 

8 

For pilot subject 1, probabilities were encoded only at the points predetermined 

in the manner just described. As will become apparent when the results are presented, a 

more dynamic approach was required. Thus, for pilot subject 2 additional points outside 

the specified ranges were encoded when that became necessary. Pilot subject 1 was 

encoded by a single analyst (T.S.W.), who was sufficiently busy during the session that he 

did not notice the need for dynamic adjustment. Pilot subject 2 was encoded by two 

analysts (T.S.W. and R.G.W.), and dynamic adjustment was easily accomplished. 

Within a blood-lead level, response rates were presented for encoding in an 

unsystematic order (but not selected with the aid of a randomization device). Encoding 

was accomplished with the use of a probability wheel. This is a wheel radially divided 

into two sectors of adjustable relative areas, one orange and one blue. The wheel can be 

spun so that it randomly stops with either sector under a pointer. For a particular 

response rate R at a given blood-lead level L, the wheel was set at some relative area of 

blue, and the expert was asked to consider the following question: 

"Do you consider it more probable that the true response rate at blood 

lead L is less than R or that on a random spin the wheel would stop 

with the pointer on blue?" 

Thus, the expert was asked to weigh two probabilities and decide which in his mind was 

greater. Depending on his answer, the relative area of blue was increased or decreased, 

and the question was posed again. This procedure continued until a wheel setting was 

reached such that the expert considered it equally likely that the wheel would randomly 

land on blue as that the true response rate is less than R for blood-lead L. The relative 

area of blue was then taken provisionally as the judged cumulative probability Fn/, . The 

procedure was repeated for another response rate until an entire cumulative probability 

distribution over R for 0.0 < R < 1.0 had been encoded for blood-lead level L. 

At that point, the distribution was graphed, shown to the expert, and discussed 

with him. Inconsistencies, if any, were pointed out and resolved, generally with the aid 

of the wheel. An inconsistency occurs when, for any pair of response rates Rj and R:, 
P.15 

Rj > R-, but FR./T < FR./L* ^n inconsistency quite simply as a dip in the graph which 

should otherwise rise monotonically as response rate goes from 0.0 to 1.0 (See Wallsten, 

Forsyth, and Budescu, 1983, for details.) 

Implications of the encoded distribution were also discussed. As just one 

example, if for Rj > Rj, F R ,r - 0.25 and F^/ L = 0.75, this implies that the expert 

considers it equally likely that the true response rate falls inside the (Rj, Rj) interval as 

outside it. If the expert disagreed with this or any of various other implications, 

adjustments were made, again generally with the aid of the wheel. When the expert was 

satisfied, and considered the encoded distribution to represent his probabilistic judgment 

another blood-lead level was selected and the process was repeated. 

When distributions were encoded for all selected blood-lead levels, these were 

plotted on a single graph and discussed with the expert. Inconsistencies between 

distributions, if any, were resolved, again with the wheel. An inconsistency between 

distributions occurs whenever two distributions cross, i.e., for blood leads Lj and L- such 

that Lj > Lj, and for response rates Rj and Rj, such that Rj > R-, F R /L< > F^ « . but 

FR-/L- * ^R/L- ^see Wailsten

» Forsyth, and Budescu, 1983). When the expert was 

satisfied that the entire set of distributions represented his probabilistic judgments, the 

encoding was completed. 

Additional consistency checks were carried out with pilot subject 1. He also 

encoded subjective probability distributions over blood-lead levels at two response rates, 

in a manner analogous to that described, resulting in cumulative distributions Fj/^ . As 

Feagans and Biller (1981) have proved, consistency requires that FJWT + F, ,-o - 1. 

Inconsistencies with this requirement were pointed out to the subject, and he was aided 

in resolving them to his satisfaction. 

For pilot subject 1, response-rate ranges were obtained, and probability 

judgments over response rates were subsequently encoded, for blood-lead levels in the 

order 15, 55, 35, 5, 45, and 25 ug/dl. In addition, following the 55 ug/dl condition, 
P.16 

10 

probability judgments over blood-lead levels were encoded for.response rates 0.65 and 

0.20, in that order. 

For pilot subject 2, response rate ranges were obtained for blood-lead levels in 

the order 55, 15, 25, 45, 5, and 35 ug/dl. Due to time limitations, probabilities were 

encoded over response rates only for blood-lead levels 55, 15, and 35 ug/dl, in that order. 

2.3 SUBJECTS 

The two pilot subjects are EPA environmental health scientists involved with 

analysis and interpretation of scientific information on lead, who are familiar with the 

literature on lead's hematopoietic effects. 

2.4 RESULTS 

2.4.1 Pilot Subject 1 

In the opinion of subject 1, children aged 0-3 and 4-7 have different doseresponse curves, primarily due to differences in iron status. Judgments were encoded 

only for the function pertaining to the younger age group. 

The probability judgments are presented in Table 2.1 and depicted in Fig. 2.1. 

They may be considered cumulative judgmental probability distribution functions 

(CDFs). That is, for each blood-lead level L, the ordinate in Fig. 2.1 and the column 

labeled F in Table 2.1 represent the judged probability that the true response rate Rrr. is 

less than the rate R shown as the abcissa of the figure and in the labeled column of the 

table. 

Note in Fig. 2.1 the difference in CDFs for L = 5 and 15 ug/dl versus the CDFs 

for the remaining lead levels. This subject is rather certain that population response 

rates are low for the two lowest lead levels employed. He considers the response rates 

to be higher at the higher lead levels, but is more uncertain as to their actual values, as 
TABLE 2.1 Subject l's Probability Judgments, F, about Fraction, R, of Population of U.S. 

Children Aged 0-3 Showing Hemoglobin Levels Less Than or Equal to 11 g/dl at Each 

Indicated Lead Level, L 

Blood-Lead Levels 

LA = 5 u/dl L1 = 15 u/dl L6 = 25 p/dl L3 = 35 g/dl L5 = 45 p/dl L2 =5 5 u/dl 

R

a

F

b RFRFRFR F R F 

0.00 0.00 0.00 0.00 0.2 0.27 0.10 0.07 0.35 0.20 0.30 0.03 

0.02 0.15 0.02 0.05 0.3 0.52 0.26 0.28 0.48 0.48 0.44 0.24 

0.04 0.65 0.04 0.38 0.4 0.87 0.42 0.46 0.61 0.71 0.58 0.45 

0.06 0.79 0.06 0.55 0.5 0.97 0.58 0.80 0.74 0.98 0.72 0.72 

0.08 0.89 0.08 0.70 0.6 1.00 0.74 1.00 0.87 1.00 0.86 0.99 

0.10 0.98 0.10 0.83 0.7 1.00 1.00 1.00 1.00 1.00 1.00 1.00 

aR is population response rate as a decimal fraction; R was presented as a 

percentage to the subject to avoid confusion with probability. 

^P is the judgmental probability that the response rate is less than or equal 

to R. 
P.18 

12 

Xi 

to 

X. 

o 

u 

Cu 

CU 

> 

e 

m. 

Population Response Rate, Expressed As 

Fraction of Population Affected, R 

FIGURE 2.1 Subject l's Probability Judgments about Fraction of Population of U.S. 

Children Aged 0-3 Showing Hemoglobin Levels Less Than or Equal to 11 g/dl at 

Each Indicated Blood-Lead Level, L 
P.19 

13 

indicated by the more gradually rising CDFs. This pattern suggests that the subject 

believes there to be a threshold between 15 and 25 ug/dl, below which lead-induced 

hemoglobin reduction is minimal. Upon seeing these distributions, he independently 

remarked that they were consistent with his belief in such a threshold. 

Note also in Fig. 2.1 that the encoded CDFs for L = 25 and 45 ug/dl do not begin 

sufficiently close to 0.0. Encoding took place for the response rate ranges established 

initially for those two lead levels, but despite all attempts the ranges were not extended 

low enough. This is the problem alluded to earlier, in which dynamic adjustment of the 

points to be encoded was required, but overlooked. The entire set of CDFs is internally 

consistent in that none of them dip and there are no crossovers. However, if the CDFs 

for L = 25 and 45 ug/dl had been encoded at their lower ranges, crossovers may have 

occurred and the subject would have been aided in eliminating them. 

The encoded judgments for blood-lead levels conditional on each of response 

rates 0.20 and 0.65 will not be reproduced here. They were intended only as further 

consistency checks on the judgments just presented. It turned out that whenever these 

judgments conflicted with the others, it was these that the subject changed. 

2.4.2 Pilot Subject 2 

This subject also considered there to be different dose-response curves for the 

two age groups. Thus, as with the other subject, judgments were encoded only for a 

dose-response function for children aged 0-3. These judgments are shown in Table 2.2 

and Fig. 2.2. 

Figure 2.2 gives no suggestion that this subject believes there to be a threshold, 

nor did the existence of one come up in discussion. The three CDFs were encoded over 

their full ranges and are internally consistent. 
P.20 

14 

TABLE 2.2 Subject 2's Probability Judgments, F, 

about Fraction, R, of Population of U.S. 

Children Aged 0-3 Showing Hemoglobin Levels 

Less Than or Equal to 11 g/dl at Each 

Indicated Lead Level, L 

Blood-Lead Levels 

L2 = 15 u/dl L3 = 35 u/dl L1 = 55 u/dl 

R F R F 

0.10 0.00 0.17 0.00 

0.21 0.18 0.20 0.03 

0.32 0.45 0.33 0.32 

0.43 0.68 0.46 0.50 

0.54 0.90 0.59 0.70 

0.65 0.97 0.77 0.92 

0.85 0.95 

R is population response rat e as a decimal 

fraction ; R was presented as a percentage 

to the subject to avoid confusion with 

probability . 

F is the judgmental probabilit y tha t the 

response rat e is less than or equal to R. 

2.5 DISCUSSION 

First, it is of interest to compare judgments from the two subjects. As already 

noted, subject 1 believes there to be a threshold for lead-induced hemoglobin reduction, 

whereas subject 2 gave no evidence of such a belief. Consistent with this difference, 

subject 1 believes population response rates to be lower at 15 ug/dl than does subject 2. 

The probability judgments of the two subjects cover overlapping response rate ranges for 

L = 35 and 55 ug/dl, however. 

R

a 

0.05 

0.14 

0.23 

0.30 

0.40 

0.50 

F

b 

0.00 

0.11 

0.55 

0.90 

0.95 

0.99 
P.21 

15 

5 0,6 -

Xi 

m 

Xi 

o 

cu 

> 

3 E 

3 

U 

S 0.4 -

Population Response Rate , Expressed As 

Fraction of Population Affected, R 

FIGURE 2.2 Subject 2's Probability Judgments about Fraction of Population of U.S. 

Children Aged 0-3 Showing Hemoglobin Levels Less Than or Equal to 11 g/dl 

at Each Indicated Blood-Lead Level, L 
P.22 

16 

But note also that subject 2's CDFs are shallower than subject l's, i.e., they span 

a broader range. Thus, the judgments of subject 2 represent more uncertainty about the 

dose-response function than do those of subject 1. 

Beyond comparisons of this sort, however, the judgments of the two subjects are 

to be used for the purpose of calculating risk estimates. This will be done by using the 

probabilistic judgments to generate a set of possible dose-response functions. These will 

be isoprobability dose-response functions in the sense that the probability is constant 

that a given level of L will generate a response rate less than or equal to R. This set of 

functions can then be combined with suitable estimates of blood-lead distributions to 

yield a set of probabilistic estimates for number of people affected. 

Since numerical judgments are be obtained only at discrete points for a given 

lead level, response rates corresponding to particular probabilities will be obtained in 

general by interpolating between encoded points. This is to be done by fitting a suitable 

probability distribution to the encoded values. 

The criteria for selecting a particular distribution are discussed in the next 

section, as are the techniques for fitting a distribution to an individual's probability 

judgments. The techniques used to interpolate between distributions at particular bloodlevels to obtain isoprobability dose-response relationships are also discussed. Section 4 

then evaluates these techniques by applying them to the judgments of each subject and 

deriving risk estimates. Section 5 indicates how such estimates can be communicated 

understandably to a decision maker, and Section 6 summarizes what has been learned 

from the pilot encoding sessions and the subsequent analysis. 
P.23 

17 

3 MATHEMATICAL FORMULATION 

3.1 CRITERIA 

The first requirement is to specify a family of distributions whose shape, or 

range of shapes, match those of the judgments. In addition, the distributions must be 

ones for which parameters can be estimated such that the judgments are fitted as closely 

as possible, in a well-defined sense, and such that the fit can be assessed. Finally, the 

fitted distributions for an expert must not demonstrate internal inconsistencies. 

Specifically, this means that the fitted distributions must not cross at any point. 

The family of equal-variance normal-on-log-odds distributions meets all these 

requirements, and is presented next. Following that, a method is presented for deriving 

risk estimates by combining a specified blood-lead distribution with an expert's 

probability judgments about dose-response relationships, where the latter is represented 

by normal-on-log-odds distributions. The full mathematical formulation is given in 

Appendix B, and only a summary is presented in this section. 

3.2 THE BEST-FITTING AND EQUAL-VARIANCE NORMAL-ON-LOG-ODDS 

DISTRIBUTION FUNCTIONS 

The uncertain quantities of interest in this paper are population response rates, 

denoted R, at each of several blood-lead levels, denoted L. Through encoding sessions 

with recognized experts, judgments will be elicited representing a cumulative density 

function (CDF) for a population response. rate R given a blood-lead level L. Taken 

together, these judgments will provide a family of probabilistic relationships as idealized 

in Fig. 3.1, and already presented more realistically in Figs. 2.1 and 2.2. Each curve in 

Fig. 3.1 represents the judgmental probability that the population response rate (a 

fraction between 0 and 1) will be less than or equal to a particular value R, given a 

blood-lead level of L, for a well-defined population of interest, under well-defined 

exposure conditions. 
P.24 

18 

X. 

ra 

X) 

o 

u 

o-

> 

3 

E 

3 

CJ 

l . U 

0 . 8 

0.6 

0 . 4 

0 . 2 

0 . 0 

1 I 

*J / 

1 

«o / 

*J / 

1/ 

" / 

' 1/ 

" / 

' l/ 

«o / 

» / 

1 

• 1/ 

«o 7 

I 

' 1/ 

1 

' !/ ' 

I 

1 

.— 

— 

— 

1 1 1 

0.0 O.S 0.4 0 . 6 0.8 1.0 

Population Response Rate, Expressed As 

Fraction of Population Affected, R 

FIGURE 3.1 A Hypothetical Family of Probabilistic Judgments about the Fraction of 

a Well-Defined Population That Will Suffer a Well-Defined Adverse Health Effect at 

Each Indicated Blood-Lead Level, L 
P.25 

19 

As already discussed, it will generally be necessary to fit a distribution to the 

assessed points in order to perform risk calculations. Because the probabilistic 

judgments about the dose-response relationships appear generally to be "S-shaped" over 

the closed [0,1] interval, as demonstrated in Figs. 2.1 and 2.2, as well as by the judgments 

of all the health experts encoded by Wallsten, Forsyth, and Budescu (1983), it can be 

difficult to represent them with simple mathematical functions. One particularly useful 

function that is relatively easy to work with is the normal-on-log-odds distribution 

function. 

The normal-on-log-odds distribution is obtained by fitting a normal distribution 

to the natural log of the odds implied by the population response rates R. Thus, the 

following relationships are defined: 

X = JL, 0 < X < » 

Y = ln(X), - « < Y < -

where X is the odds variable and Y is the log-odds variable. It is assumed that Y is 

normally distributed with mean u and variance o . The degree to which the assumption is 

appropriate can be tested with each set of judgments. If Y is normally distributed, then 

X is lognormally distributed and the distribution of interest induced on R is called a 

normal-on-log-odds distribution. Although the normal-on-log-odds distribution is not 

expressable in a closed form, all probabilities of interest are easily derived from the 

normal distribution over Y, as discussed in Appendix B. 

A separate normal-on-log-odds distribution function can be fit to the elicited 

CDFs (probability judgments over response rate R) at each blood-lead level LJ, j=l, ..., 

m. In other words, a function is fit to each set of points depicted in Fig. 3.1. A unique 

normal-on-log-odds distribution is determined at each lead level by deriving least-squares 

estimates of the parameters UJ and aj, denoted uj and a;, for the normal distribution 

applied to the log-odds transformed variable Y, in the manner developed in Appendix B. 

The set of distributions, each having separate least-squares estimates for u: and aj, is 
P.26 

20 

referred to as the best-fitting normal-on-log-odds distributions. Furthermore, a measure 

is available of how well the probability judgments at a given lead level are represented 

by the derived distribution. This measure, discussed in Appendix B, is the standard 

regression r2 statistic for the goodness of fit of the normal distribution with parameters 

UJ and a- to the CDF over the transformed variable Y. 

As presented fully in Appendix C, and summarized in Sec. 4, the normal-on-logodds distribution describes the encoded judgments very well, with r values generally 

exceeding 0.95. (The maximum possible r2 value is 1.0, which would indicate a perfect 

fit.) Nevertheless, it may be necessary to sacrifice goodness of fit to a small degree by 

fitting an expert's probability judgments with a family of equal-variance normal-on-logodds distributions. Doing so would assure that the derived distributions at two different 

lead levels never cross one another. (Crossing would imply that exceeding a specified 

response rate is judged more probable at blood-lead level Lj than at L2, where L2 is 

greater than Lj.) Because of the manner in which judgments were obtained from the 

pilot subjects, crossings did not occur. Furthermore, judgments will be obtained from the 

health experts over a wider probability range so that crossings would only be possible at 

cumulative probabilities very near to 0 and 1. However, even crossings such as these 

would represent an inconsistency and therefore should not be allowed. 

Equal-variance normal-on-log-odds distributions are derived by first obtaining a 

least-squares estimate of the standard deviation pooled over lead levels, a', then setting 

all the 0' = a' and finding new least squares estimates of the means, u' , as detailed in 

Appendix B. The result is a family of normal-on-log-odds distributions, one member for 

each blood-lead level L-, that have a common variance and differ only in the mean. 

Their goodness of fit can also be assessed by means of r 2 calculated for the fit of the 

variable Y. 

If a functional relationship can be established between blood-lead levels and the 

u', a judgmental probability density function over R can be calculated for any blood-lead 
P.27 

21 

level, should that be necessary for interpolating between blood-lead levels. This 

continuum of density functions in essence is a probability surface over the dose-response 

plane. 

If for each blood-lead level L-, probability judgments are elicited over virtually 

the entire [0,1] interval, and if these judgments exhibit no crossings, then it would be 

expected that the goodness of fit of the equal-variance normal-on-log-odds distributions 

would be close to that of the best-fitting normal-on-log-odds distributions. Since the 

equal-variance distributions could not give rise to inconsistencies, they would be taken as 

the preferred representation of his or her judgments for purposes of developing risk 

estimates. 

3.3 COMBINING BLOOD-LEAD DISTRIBUTIONS WITH PROBABILITY JUDGMENTS 

ABOUT DOSE-RESPONSE FUNCTIONS 

Using either equal-variance or best-fitting normal-on-log-odds functions fit to 

the judgments in the manner described above, it is possible to estimate dose-response 

functions that correspond to any given judgmental cumulative probability. For example, 

a median dose-response curve based on an expert's probability judgments is obtained by 

determining the response rate that corresponds to a cumulative probability of 0.50 at 

each blood-lead L. The dose-response curve, such that there is a 5% chance of the true 

response rate lying above the curve at any given blood lead, is obtained by determining 

the response rate that corresponds to a cumulative probability of 0.95 at each L, and so 

forth. 

As shown in Appendix B, any blood-lead distribution can then be combined with 

the dose-response curve corresponding to judgmental cumulative probability F to yield a 

probability estimate for the overall fraction of the population R suffering the adverse 

health effect. If there is also uncertainty about the blood-lead distribution, that must be 

taken into account as well, but that problem will not be treated in this report. 
P.28 

22 

We will consider here only the empirical blood-lead distribution from the 

NHANES-II survey. The distribution is presented in discrete form, with intervals of 

0-9 ug/dl, 10-19 ug/dl, ..., 60-69 ug/dl. Appendix B shows, in particular, how to combine 

the NHANES-II blood-lead distribution with a dose-response curve corresponding to any 

judgmental probability F, for all values of F between 0.0 and 1.0. 

For example, if the NHANES-II distribution is combined with the dose-response 

curve corresponding to a judged cumulative probability of 0.50, the result is an estimate 

of the fraction of the population suffering the adverse health effect, such that the true 

fraction is equally likely to be above or below it. In general, combining the blood-lead 

distribution with the dose-response curve corresponding to judgmental cumulative 

probability F yields an estimate of the fraction of the population R suffering the health 

effect, such that the true value lies below R with probability F. By carrying out this 

calculation for all values of F, 0 < F < 1, and plotting each F as a function of the derived 

R, a judgmental CDF is obtained over affected fraction of the population, as illustrated 

in Fig. 3.2. We call this CDF a derived risk distribution. 

Figure 3.2 represents a hypothetical derived risk distribution for a particular 

population and a particular adverse health effect, derived from an expert's probability 

judgments about dose-response functions combined with a blood-lead distribution. One 

can read from it, for example, that with probability 0.90, less than about 62% of the 

population suffer the adverse health effect (and with probability 0.10, more than 62% 

do), with probability 0.75, less than about 47% suffer it (and with probability 0.25, more 

than 47% do), and so forth. 

A separate risk distribution is derived by combining each expert's probability 

judgments about dose-response curves with a blood-lead distribution. This gives rise to a 

family of risk distributions that represents the range of uncertainty about the prevalence 

of the adverse health effect. This family can be shown and explained to the decision 

maker. 
P.29 

23 

1.0 

a. 

m-A 

x . 

ra 

Xi 

o 

u 

P. 

> 

3 

E 3 O 

0*4 0.6 1.0 

Population Response Rate, Expressed As 

Fraction of Ponulation Affected.. R 

FIGURE 3.2 A Hypothetical Derived Probability Distribution about the Fraction of 

a WeU-Defined Population That Will Suffer a Well-Defined Adverse Health Effect, 

Obtained by Combining Probability Judgments Such as in Fig. 3.1 with a 

Blood-Lead Distribution Such as the NHANES-II 
P.30 

24 

If alternative blood-lead distributions are hypothesized (say, under alternative 

NAAQSs), and there is uncertainty about them, it is a small matter to generalize the 

present procedures to incorporate this richer informational structure. 
P.31 

25 

4 ILLUSTRATIVE CALCULATIONS 

The analytical methods derived in Appendix B and summarized in the last section 

should be applied to the judgments of health experts only after the methods have been 

tested and shown to describe the judgments with sufficient accuracy. In Appendix C 

best-fitting normal-on-log-odds distributions and equal-variance normal-on-log-odds 

distributions are derived for the two pilot subjects, and their fits are assessed. In 

addition, risk estimates are derived based on the probability judgments from each pilot 

subject. This section summarizes the material in Appendix C. Also, to establish the 

generality of the approach, distributions have been fit to the probability judgments of the 

four lead experts encoded by Wallsten, Forsyth, and Budescu (1983), and the results are 

presented in this section. 

4.1 FITTING NORMAL-ON-LOG-ODDS DISTRIBUTIONS TO JUDGMENTS 

4.1.1 Pilot Subject 1 

Recall that the probability judgments from this subject are shown in Table 2.1 

and Fig. 2.1. The best-fitting and equal-variance normal-on-log-odds distributions both 

fit this subject's probability judgments very well, as indicated by the r-square statistics 

summarized in Table 4.1 from Tables C.2 and C.3 of Appendix C. 

Furthermore, the r-square values for the equal-variance normal-on-log-odds 

distributions are almost as great as those for the best-fitting ones, except at L = 35 

ug/dl. This suggests that with the possible exception of the fit at L = 35 ug/dl, little 

descriptive power is lost by relying on the equal-variance functions. 

The reason for this pattern of results is that the variances of the best-fitting 

distributions were in fact nearly equal, except for that of the function fit at the bloodlead level L = 35 ug/dl, which greatly exceeded the other variances. This outcome is 
P.32 

26 

TABLE 4.1 R-Square Values for the BestFitting and Equal-Variance Normal-OnLog-Odds Distributions Fitted to the 

Probability Judgments of Subjects 1 

and 2 Conditional on the Indicated 

Blood-Lead Levels 

Blood Lead 

5 ug/dl 

15 ug/dl 

25 ug/dl 

35 ug/dl 

45 ug/dl 

55 ug/dl 

aBest Fitti 

Subject 1 

B.F.a E.V.b 

0.97 0.95 

0.99 0.99 

0.98 0.93 

0.97 0.62 

0.97 0.94 

0.98 0.98 

ngSubj< 

B.F. 

NAC 

0.97 

NA 

0.99 

NA 

0.95 

>ct 2 

E.V. 

NA 

0.87 

NA 

0.98 

NA 

0.91 

Equal Variance. 

cNot Appropriate. Judgments were not 

obtained at this blood-lead level. 

explored in detail in Appendix C and can be traced to the fact that the CDFs encoded for 

the neighboring blood-lead levels of L = 25 and 45 ug/dl did not begin sufficiently close to 

0.0, a problem previously discussed in Sec. 2.4.1. The large variance at L = 35 ug/dl 

caused that normal-on-log-odds distribution to cross those at L = 25 and 45 ug/dl and to 

have a mean less than the mean at L = 25 ug/dl. (See Table C.2 and Fig. C.l for 

details.) These are clearly inconsistencies that subject 1 sought to avoid and that do not 

occur with the equal-variance normal-on-log-odds distributions. Thus, all things 

considered, the equal-variance distributions as a group appear to provide the better 

descriptions of subject l's probability judgments. 
P.33 

27 

Figure 4.1 shows the CDFs from the equal-variance normal-on-log-odds 

distributions along with the encbded probability judgments of subject 1. It can be seen 

that the fit is quite satisfactory. 

Another interesting representation of the subject's judgments, derived through 

the equal-variance normal-on-log-odds function, is the family of dose-response curves 

shown in Fig. 4.2. The 0.50 cumulative probability function shows the median judged 

response rate, R, at each lead level, L. The 0.95 and 0.05 functions show the response 

rates, R, at each lead level, L, such that there is a 5% chance of the true response rate 

being, respectively, above or below the indicated value. Thus, in a sense, the two 

extreme dose-response functions map out 90% credible intervals for response rate at 

each blood-lead level, based on subject l's probability judgments. 

4.1.2 Pilot Subject 2 

The probability judgments for this subject, listed in Table 2.2 and displayed in 

Fig. 2.2, were also fit with best-fitting and equal-variance normal-on-log-odds 

distributions. The r-square statistics from Tables C.4 and C.6 are reproduced in Table 

4.1, where it can be seen that the best-fitting functions do an excellent job, and the 

equal-variance ones are almost as good. The greatest discrepancy occurs at L = 15 ug/dl, 

but even here the lesser r-square value is quite high. 

Unlike the results with subject 1, the variances of the best-fitting distributions 

for subject 2 increased with lead level (cf. Table C.4). If there had been time in the 

encoding session to elicit judgments for more than the three lead levels of 15, 35, and 55 

ug/dl, so that adjacent levels would have been more closely spaced, then the consistency 

requirement that CDFs not cross would have caused the variances to be more nearly 

equal. 

An ancillary point pursued in Appendix C concerns the use of linear regression 

techniques to estimate normal-on-log-odds functions at intermediate lead levels where 
28 

P.34 

1.0 

ta. 

>A 

Xi 

ra 

x 

o 

OJ 

> 

3 

e 

3 

u 

Population Response Rate, Expressed As 

Fraction of Population Affected, R 

FIGURE 4.1 Equal-Variance Normal-On-Log-Odds CDFs for the Probability 

Judgments of Subject 1 That Were Previously Shown in Fig. 2.1 
P.35 

29 

t3 

01 

U 

01 

c 

o 

3 

a 

o 

c 

o 

u 

ra 

u 

l.U 

0.8 

0.6 

0.4 

0.2 

n.o 

1 

— 

1 : 

1 1 1 1 

0.95 Cumulative ^ ^ 

Probability *>^ 

/ 0.50 Cumulative ^s' 

/ Probability ^ ^ 

/ ^0^m. " 0.05 Cumulative 

/ ^** Probability 

1 1 I 

1 

• ^ — 

^— 

1 

15 25 35 45 55 

Blood-Lead Level, L (ug/dl) 

FIGURE 4.2 The 0.05-, 0.50-, and 0.95-Cumulative Probability Estimates of Response 

Rate R at Various Blood-Lead Levels L, Based on the Equal-Variance Normal-On-LogOdds Distributions Fitted to the Judgments of Subject 1 
P.36 

30 

encoding did not take place. Such estimation becomes necessary to carry through the 

subsequent risk calculations. Figure 4.3 shows the regressed equal-variance normal-onlog-odds distribution CDFs for all six lead levels along with the encoded probability 

judgments for the levels at which they were obtained. Figure 4.4 shows the 0.05-, 0.50-, 

and 0.95-cumulative probability dose-response functions based on the CDFs in the 

preceding figure. These differ from those for subject 1 (Fig. 4.2) primarily at the lower 

lead levels. 

4.1.3 Judgments of Other Experts 

As a further test of the generality of the techniques developed here, best-fitting 

and equal-variance normal-on-log-odds distributions were fit to the probability judgments 

of the lead experts previously encoded by Wallsten, Forsyth, and Budescu (1983). As with 

the pilot subjects, these experts provided probability judgments over response rates at 

five blood-lead levels regarding lead-induced hemoglobin reduction. Upper and lower, 

rather than sharp, probabilities were encoded at each dose-response point. Distributions 

were fit to the means of the upper and lower probabilities, which can be taken as 

approximations of the values that would have been obtained had sharp probabilities been 

sought. 

Table 4.2 shows the means of the five r-squares for the best-fitting normal-onlog-odds distributions for each expert, as well as the pooled r-square for the fit of the 

equal-variance distributions. It is apparent from Table 4.2 that the individual normal-onlog-odds distributions provide excellent fits to the judgments. The equal-variance 

distribution provide respectable fits, but not as good as those seen previously. However, 

the ranges of probabilities that were encoded for each distribution in this study were 

quite restricted, averaging only 0.58 over the four experts. Had more complete ranges 

been encoded, maintaining no distribution crossovers, the equal-variance functions would 

have fit much better. 
31 

P.37 

1.0 

5 0.6 -

X) 

ra 

XI 

o u a. 

oi 

> 

3 

E 

3 

CJ 

Population Response Rate, Expressed As 

Fraction of Population Affected, R 

FIGURE 4.3 Regressed Equal-Variance Normal-On-Log-Odds CDFs for the Probability 

Judgments of Subject 2 That Were Previously Shown in Fig. 2.2, as Well as for 

Intermediate Lead Levels 
P.38 

32 

1.0 

0.8 

01 

u 

01 

c 

o 

3 

c 

o 

p0.6 -

: 0.4 -

c 

o 

u 

ra 

0.2 -

0.0 

0.95 Cumulative 

Probability 

0.50 Cumulative 

Probability 

Blood-Lead Level, L (ug/dl) 

FIGURE 4.4 The 0.05-, 0.50-, and 0.95-Cumulative Probability Estimates of Response 

Rates R at Various Blood-Lead Levels L Based on the Regressed Equal-Variance 

Normal-On-Log-Odds Distributions Fitted to the Judgments of Subject 2 
P.39 

33 

TABLE 4.2 Regression r2 Values for the Fit of 

Best-Fitting and Equal-Variance Normal-On-LogOdds Distributions to Judgments of Experts 

Encoded by Wallsten, Forsyth, and Budescu (1983) 

Expert 

r

2

 E F G H 

Best-Fitting 0.96 0.96 0.95 0.95 

Equal-Variance 0.92 0.85 0.85 0.83 

To summarize all the results, the normal-on-log-odds function applied to 

individual subjective probability distributions generally provides excellent fits, with rsquares on the order of 0.95 and higher. Constraining the normal-on-log-odds 

distributions to have equal variance for an entire set of distributions still yields very 

good fitting functions, but with, of course, somewhat lower r-squares, while 

simultaneously avoiding the possibility of inconsistencies. Since, as discussed previously, 

probability judgments will be obtained from the health experts over virtually the full 

[0,1] interval for six lead levels, it can be expected that equal-variance functions will be 

quite accurate. 

4.2 ILLUSTRATIVE RISK CALCULATIONS 

Having established that the normal-on-log-odds distribution describes the 

probability judgments very accurately, the next step is to combine the judgment 

distributions with the NHANES-II blood-lead distribution to yield derived risk 

distributions in the manner described in Appendix B and summarized in Sec. 3.3. This is 

done in Appendix C with the judgments of both pilot subjects using the NHANES-II 

distribution for children aged 6 months to 5 years, to illustrate the method. The results 

are summarized here. 
P.40 

34 

Derived risk distributions of the sort depicted in Fig. 3.2 were calculated using 

both the best-fitting and the equal-variance normal-on-log-odds distribution 

representations of each subject's judgments. The results for subject 1 were virtually 

identical for the two representatives (Fig. C.5 in the appendix), but some differences 

occurred for subject 2 (Fig. C.6). In the latter case, the risk distribution was slightly 

more shallow (i.e., spans a larger range) based on the equal-variance than on the bestfitting representation. However, the two curves were very close in the upper third of the 

cumulative distribution, which would be the points of most interest to a decision maker. 

Figure 4.5 shows the derived risk distributions based on the equal-variance 

normal-on-log-odds distribution representation of each subject's probability judgments. 

Utilizing subject l's judgments, the median estimate is that 10.5% of U.S. children aged 

0-3 have hemoglobin levels less than or equal to 11 g/dl. The median estimate utilizing 

subject 2's judgments is 20.1%. Based on subject l's judgments, there is a 0.10 

probability that more than 20.5% of the population suffer the adverse health effect, 

while subject 2's judgments lead to the statement that there is a 0.10 probability that 

more than 39.1% of the population do so. Other values can be read similarly. From 

another perspective, one could focus on particular population response rates (e.g., 10%) 

and read the probabilities that the true response rates are less than that value based on 

each expert's judgment. 

The difference between the derived risk distributions based on the two subject's 

judgments is easily traced back to particular features of their probability judgments. 

Subject 1 considered response rate to be much lower at 15 ug/dl blood lead than did 

subject 2 (cf. Figs. 2.1 and 2.2 or 4.2 and 4.4), and according to the NHANES-II bloodlead distribution, 63% of the population have blood leads in the corresponding interval of 

10-19 ug/dl. Thus, quite naturally, subject l's judgments lead to lower risk estimates 

than do subject 2's. 
P.41 

35 

1.0 

0.8 

u. 

x 

ra 

x 

o 

u 

a. 

0) 

> 

3 6 3 U 

0.6 -

0 . 4 

0.2 

0.0 

Subject 1 

0 . 0 0.1 0. 2 0. 3 0. 4 0.5 0.6 0.7 

Population Response Rate, Expressed As 

Fraction of Population Affected, R 

FIGURE 4.5 Derived Risk Distributions Based on Equal-Variance Normal-On-Log-Odds 

Distributions Fit to Subject l's and Subject 2's Judgments, Showing in Each Case 

a Cumulative ProbabUity F that Fraction R of the Population of U.S. Children 

Aged 0-3 Have Hemoglobin Levels Less Than or Equal to 11 g/dl 
P.42 

36 

5 COMMUNICATING THE RISK ASSESSMENT RESULTS TO A DECISION MAKER 

Although the mathematics to this point are of moderate complexity, they result 

in probabilistic statements about dose-response curves, derived risk distributions and risk 

estimates based on the judgments of experts (two pilot subjects were used here) that can 

be clearly communicated to a decision maker. A graph such as Fig. 4.5 to represent 

derived risk distributions based on each expert's judgments should, with some 

explanation, be understandable by a decision maker and his or her staff. 

For example, from Fig. 4.5 it is relatively easy to see that, with probability of 

0.05, at least 24% of the population have hemoglobins less than or equal to 11 g/dl 

according to subject l's judgments, and at least 46% do so according to subject 2's. From 

another perspective, if the decision maker wishes to focus on a particular population 

response rate, because in his or her opinion that is the maximum allowable rate 

consistent with protecting the public health, then he or she can read the probability that 

it is exceeded given each expert's judgments. 

For example, if the decision maker chose to focus on a population response rate 

of 10%, then he or she would see that this rate is currently exceeded with probability 

0.55 according to subject l's judgments and with probability 0.85 according to subject 2's. 

Derived risk distributions such as those in Fig. 4.5 for each definition of the 

adverse health effect (9.5 and 11 g/dl of hemoglobin) for each of the two age groups (0-3 

and 4-7 years old), or simply combined over both age groups, will give the decision maker 

a good picture of the current risk estimates. While illustrative values read from the 

graphs should be included in accompanying text, the decision maker determines what 

definition of the adverse health effect, what population response rates, and what 

probabilities constitute "protecting the public health with an adequate margin of 

safety." Having made these decisions, he or she can then consult the derived risk 

distributions to determine whether the range of expert opinion suggests that the criteria 
P.43 

37 

are currently being met. If estimated alternative blood-lead distributions are available 

given alternative NAAQS (an issue not addressed in this report), then similar risk 

distributions can be derived for the alternative NAAQS, and the decision maker can 

consult those as well. 

It is likely that the judgments of the actual experts will be more similar to each 

other than were those of the pilot subjects, leading to more similar derived risk 

distributions. But should that not be the case, then the decision maker will be well 

served by seeing the range of risk estimates, rather than a simple "average" estimate. 

A second kind of output that may be understandable by a decision maker and his 

or her staff is the family of equal-probability dose-response functions, such as those 

shown in Figs. 4.2 and 4.4. If such individuals are familiar with the usual empirical doseresponse curves, including associated standard error bars, then these figures are a small 

conceptual leap. They would serve to illustrate the range of expert judgment regarding 

the dose-response relationship between lead and the adverse health effect for the 

population of interest, without requiring the estimation of blood-lead distributions. 
P.44 

38 

6 OBSERVATIONS AND RECOMMENDATIONS 

We have found the pilot encoding sessions and analysis to be very valuable. They 

have enabled us to work out many small difficulties that could only be discovered by 

experience, and make us well prepared to begin work with actual experts. 

The foUowing sections contain observations and recommendations based on these 

two experiences described in Sees. 2 and 4. 

1. The initial broad ranges on response rate given by subject 1 proved 

to be inappropriate in many cases. Because of limited time and 

the fact that only one analyst was present, this went unnoticed 

during the session. This problem should be overcome by using two 

analysts, provided that they are careful to obtain probability 

judgments for each lead level that adequately span the 0.0 to 1.0 

probability interval. 

2. If the normal-on-log-odds distribution is to be used, probabilities 

and response rates of 0 and 1 are not helpful. Instead, values 

around the extremes of 0.01 and 0.99 should be sought. 

3. When inconsistencies arose during the encodings conditional on 

response rates, subject 1 tended to change his judgments 

conditional on response rate (R) rather than those conditional on • 

blood-lead level (L). Analysts should be aware of this possible 

reaction and, if it occurs, should consider spending more time on 

obtaining judgments conditioned on L, or possibly not obtain 

judgments conditional on R at all.* 

4. Subject 1 remarked that starting at 15 ug/dl was not very useful to 

him because there is essentially nothing happening at this level. 

This may have led to the large spread between the L = 15 ug/dl and 

L = 25 ug/dl judgments. If subjects seem to have trouble getting 

started at low lead levels, the analyst should shift to a higher level 

and then return later to the lower levels. 

5. Subject 1 judged that separate functions were appropriate for 

children aged 0-3 years and 4-7 years. This would have required 

encoding four sets of probabilities (since the plan is to encode at 

two levels of hemoglobin). The judgments were encoded in about 

four hours, including discussion of the literature, an explanation of 

•Because encodings conditional on R are potentially very valuable, they provide a rich 

topic for extensive research some time in the future. 
P.45 

39 

the probability wheel, etc'. It would have taken at least two more 

hours to complete the other three sets of probability judgments, 

indicating that a fuU day should be reserved for the first round of 

encodings to enhance the chances of obtaining all of the needed 

judgments. 

6. Only three sets of judgments could be obtained in the time 

available with subject 2. However, the ability to develop a 

complete set of probabilistic judgments (needed to complete 

calculations of risk distributions) is crucial and should be 

guaranteed. 

7. Because more than 95% of the children in the NHANES-II survey 

had blood-lead levels below 30 ug/dl, it might have been better to 

have encoded on L = 25 ug/dl instead of L = 35 ug/dl (which was 

the third curve encoded) for subject 2. If time appears to be a 

constraint in the future, analysts should try to maximize the 

chances of getting the most critical judgments (if they are known 

at the time of the encodings), even if it requires changing the 

planned (possibly randomized) order. 

8. Equal-variance normal-on-log-odds functions fit very closely the 

encoded judgments of subjects 1 and 2. 

9. The normal-on-log-odds distribution is appealing because it permits 

an analytical method for accurately representing encoded judgments. However, if any such mathematical distribution is used, it 

and its implications must be thoroughly explained to the expert 

whose judgments are used. Ultimately, the results must accurately 

portray and summarize each expert's judgments. Therefore, 

ideally the distributions wiU be fit while with the expert to assure 

that he or she agrees that they accurately represent his or her 

judgments. 
P.46 

40 

7 REFERENCES 

1. Feagans, T.B., and W.F. Biller, A General Method for Assessing Health Risks 

Associated with Primary National Ambient Air Quality Standards, U.S. EPA Draft 

Report (May 1981). 

2. Wallsten, T.S., B.H. Forsyth, and D.V. Budescu, Stability and Coherence of Health 

Experts! Upper and Lower Subjective Probabilities about Dose-Response Functions, 

Organizational Behavior and Human Performance, 31:277-302 (1983). 
P.47 

41 

APPENDIX A 

DRAFT PROTOCOL FOR ENCODING PROBABILITY JUDGMENTS 

ABOUT DOSE-RESPONSE CURVES FOR 

LEAD-INDUCED HEMOGLOBIN DECREMENTS 

OUTLINE 

A.l. INTRODUCTION 

A.2. LEAD-INDUCED HEMOGLOBIN DECREMENTS 

A.2.1 Definition of Hemoglobin Decrement as an Adverse Health Effect 

A.3. POPULATION AT RISK 

A.4. EXPOSURE CONDITIONS 

A.5. PHYSIOLOGICAL CONDITIONS 

A.6. FACTORS TO CONSIDER 

A.7. FACTORS TO KEEP IN MIND WHEN MAKING PROBABILITY JUDGMENTS 

A.7.1 Sequential Consideration of Information 

A.7.2 Effects of Memory on Judgment 

A.7.3 Estimating Reliability of Information 

A.7.4 Relation Between Event Importance and Probability 

A.7.5 Estimation of Probabilities 

A.7.6 Recommendations 

A.8. FINAL PREPARATION FOR ELICITATION OF PROBABILITY JUDGMENTS 

Version 84.6.5 
P.48 

42 

APPENDIX A 

DRAFT PROTOCOL FOR ENCODING PROBABILITY JUDGMENTS 

ABOUT DOSE-RESPONSE CURVES FOR 

LEAD-INDUCED HEMOGLOBIN DECREMENTS 

A.1. INTRODUCTION 

The Environmental Protection Agency (EPA) is charged by the Clean Air Act 

with setting and revising National Ambient Air Quality Standards (NAAQS) for selected 

pollutants, at levels sufficient to protect the public health with an adequate margin of 

safety. As you know, the scientific bases for NAAQS are presented and reviewed in 

criteria documents. In support of the forthcoming revision of the lead NAAQS, EPA has 

just prepared a new Air Quality Criteria for Lead. It presents scientific evidence from 

which the most susceptible populations can be determined and from which various 

adverse health effects can be identified. The criteria document summarizes and 

evaluates the available clinical, epidemiological, and animal or toxicological laboratory 

evidence with regard to the physiological and adverse health effects of lead, and 

therefore represents our most up to date knowledge on lead effects. 

As one aspect of the review process, EPA assesses health risks by identifying the 

most sensitive populations for each poUutant, and estimating probabilistically the 

numbers of people in the populations who may suffer each of various well-defined 

adverse health effects attributable to the poUutant. It is believed that information about 

the health risks associated with various potential standards will aid the EPA 

administrator in selecting that standard which, in his judgment, protects the public 

health with an adequate margin of safety. 

Because the risk estimates that EPA seeks are often based in part on doseresponse relationships and lead exposure estimates which are uncertain, it is necessary to 

make probability judgments about relevant dose-response functions based on the 

available evidence, and to probabilistically estimate lead exposure under alternative 
P.49 

43 

NAAQS. Obtaining the health risk estimates then involves combining dose-response and 

exposure probability estimates. 

The problem of estimating dose-response relationships is similar to that which 

exists in clinical medicine when there do not exist data that bear precisely on the 

patient's problem. In that case it is necessary to use scientific judgment to extrapolate 

from the data to make a best decision for the patient. Here, too, it is necessary to use 

scientific judgment to extrapolate from the available data. The extrapolation is not 

certain and, therefore, we will aid you to represent your opinion probabilisticaUy. 

Furthermore, since the extrapolation depends on one's interpretation of the literature, 

different people will have different judgments. For each health effect we intend to 

obtain the probabilistic judgments of about five experts to sample the range of respected 

opinions. The model for estimating risks wiU not merge these judgments into a single 

average judgment, but rather wiU estimate the range of risks based on the range of 

judgments. If we as risk analysts do our job properly, then not only will we be able to 

show the EPA administrator the range of estimates based on the range of judgments, but 

we wiU also be able to show some of the sources of the disagreements. Indeed, a side 

benefit of this exercise in which we probe your knowledge in a structured manner may be 

to help identify sources of greatest disagreement. 

Based on the evidence in the lead criteria document, two populations have been 

identified as being most susceptible to the effects of lead intoxication. One is children 

from birth through the seventh birthday, and the other is pregnant women, or more 

precisely, the fetuses being carried by the pregnant women. A number of adverse health 

effects have been identified for which we would like to estimate dose-response functions 

including the one discussed below. 

A.2. LEAD-INDUCED HEMOGLOBIN DECREMENTS 

We would like to represent in probabilistic form your opinion about the location 

and shape of dose-response functions for certain weU defined levels of lead-induced 
P.50 

44 

hemoglobin reduction in the population of U.S. children from conception through the 

seventh birthday. We realize that available data do not fully define these functions; if 

they did, then it would be unnecessary to obtain your and other experts' opinions about 

them. Nevertheless, if the population, the exposure conditions, and the health effects 

are aU precisely defined, then such functions in fact exist and we would like your best 

judgment about what they would look like if the data could be collected. 

A.2.1 Definition of Hemoglobin Decrement as an Adverse Health Effect 

There are differences of opinion, of course, as to what degree of reduction in 

hemoglobin level constitutes a health risk. The Clean Air Act makes it clear that EPA 

should set its standards to protect against adverse health effects. However, the 

reduction in hemoglobin level considered to be adverse may be different for regulatory 

purposes than for clinical action. Since it is generally agreed for children that a 

hemoglobin level of about 12 grams per deciliter is normal and about 9 grams per 

deciliter is anemic, we wiU specify two hemoglobin levels in that interval, namely Hg and 

Hj (current candidates are 9.5 and 11 gm/dl), and treat each as the physiological effect 

for purposes of specifying a dose-response function. We will elicit from you probability 

judgments about the shape and location of the dose-response curve for each of the two 

levels of effects in the sensitive population. 

A.3. POPULATION AT RISK 

We are defining the most susceptible group as all United States children from 

conception through their seventh birthday. However, you may believe on various grounds 

that younger and older children in this group have different dose-response functions for 

lead-induced hemoglobin reduction. If so, then we will elicit your judgments separately 

for the age groups 0-3 and 4-6. 
P.51 

45 

A.4. EXPOSURE CONDITIONS 

We wiU be asking your judgment about population response rates at various 

blood-lead levels. Assume that the blood-lead level under consideration for a given 

judgment is in equilibrium as a result of a sufficiently long-term constant lead exposure 

without gross excursions above or below the stated level. 

A.5. PHYSIOLOGICAL CONDITIONS 

Because the effect of lead on hemoglobin level depends on many parameters of a 

person's system, it is necessary to specify assumptions about those parameters in the 

population. The effect of lead in the system depends on the person's nutritional and 

metabolic status. The incidence of iron deficiency is greater in children than in adults, 

and greater yet in children age 0-3 than age 4-6, with a wide range of iron levels in 

children up to their seventh birthday. Assume that the distribution of iron levels is that 

which you believe it actuaUy to be in the age groups 0-3 and 4-6. The effect of lead also 

depends on the levels of zinc, copper, vitamins A, C, D, and E, calcium, phosphorous and 

magnesium. Assume in all cases that these nutrients are distributed in the two age 

groups as you believe they in fact are, taking into account the wide range of diets and 

nutritional levels of U.S. children. 

A.6. FACTORS TO CONSIDER 

In order to help you bring to mind the relevant evidence so that you may consider 

it systematicaUy, and also in order to help us to interpret your judgments, we would like 

to ask you to discuss briefly your interpretations of various aspects of the literature. 

First of all, could you tell us something about how, in your judgment, lead and iron 

interact to reduce hemoglobin levels and something about the relation between iron and 

lead levels in the body? Similarly, we would be interested in your opinions about the 

same questions with regard to zinc and lead and to calcium and lead. It appears that lead 
P.52 

46 

affects hemoglobin levels through three somewhat separate routes: interference with 

heme synthesis, interference with globin synthesis, and decreased life expectancy of 

erythrocytes. Briefly, what is your interpretation of the literature on these various 

effects of lead? Finally, we know that some groups of children are at greater risk of 

exposure to lead due to their living in deteriorating pre-1950 housing or in urban areas 

with large amounts of vehicular traffic. These children would tend to have high lead 

levels, but do you believe that there are also independent reasons to think that their 

dose-effect curves wiU be different from children who live in other circumstances? For 

example, these may be the same children who tend to have poorer diets and less access 

to medical care. What is your opinion regarding the possibility of a correlation between 

increased exposure and increased susceptibility? Are there other factors to consider in 

thinking about the dose-response functions for lead-induced hemoglobin decrements that 

we should discuss now? 

A.7. FACTORS TO KEEP IN MIND WHEN MAKING PROBABILITY JUDGMENTS 

There is usually uncertainty associated with conclusions that we draw from 

research, and more generally in our everyday thinking. However, not everyone is aware 

of aU the sources that should contribute to their uncertainty, nor are most people 

familiar with the process of actuaUy expressing their uncertainty in probabilistic terms. 

When an expert is asked to make probability judgments on socially important matters, it 

is particularly important that he or she consider the relevant evidence in a systematic 

and effective manner and provide judgments that represent his or her opinions well. 

Experimental psychologists and decision analysts have amassed a considerable 

amount of data concerning the way people form and express probabilistic judgments. The 

evidence suggests that when considering large amounts of complex information, most 

people employ simplifying heuristics and demonstrate certain systematic distortions of 

thought, i.e., cognitive biases, which adversely affect their judgments. The purpose of 
P.53 

47 

this section is to make you aware of these biases and heuristics so that, as much as 

possible, you can avoid them in making probability judgments. We will first review the 

most widespread biases and heuristics, and then offer some suggestions to help you 

mitigate their effects. 

A.7.1 Sequential Consideration of Information 

GeneraUy, the order in which evidence is considered influences the final 

judgment, although logicaUy that should not be the case. Of necessity, pieces of 

information are considered one by one in a sequential fashion. However, those 

considered first and last tend to dominate judgment. In part, initial information has its 

undue influence because it provides the framework which subsequent information is then 

tailored to fit. For example, people usually search for evidence to confirm their initial 

hypotheses; they rarely look for evidence that weighs against them. The later evidence 

has its undue effect simply because it is fresher in memory. 

Related to these sequential effects is the phenomenon of anchoring and 

adjustment. Based on early partial information, one forms an initial probability estimate 

regarding the event in question. This anchor judgment is then adjusted as subsequent 

information is considered. Unfortunately, such adjustments tend to be too 

conservative. In other words, too little weight is attached to information considered 

subsequent to the formation of the initial judgment. 

A.7.2 Effects of Memory on Judgment 

It is difficult for most people to conceptualize and make judgments about large, 

abstract universes or populations. A natural tendency is to recall specific members and 

then to consider them representative of the population as a whole. However, the specific 

instances often are recalled precisely because they stand out in some way, such as being 

familiar, unusual, especially concrete, or of personal significance. Unfortunately, the 
P.54 

48 

specific characteristics of these singular examples are then attributed, often incorrectly, 

to all the members of the population of interest. Moreover, these memory effects are 

often combined with the sequential phenomena discussed earlier. For example, in 

considering the evidence regarding the dose-response curve of a particular poUutant, you 

might naturally first think of a study you or a personal friend recently completed. Or 

you might think of a study you recently read, or one that was unusual and therefore 

stands out. The tendency might then be to treat the recaUed studies as typical of the 

population of relevant research, ignoring important differences among studies. 

Subsequent attempts to recall information could result in thinking primarily of evidence 

consistent with the initial items you thought of. 

A.7.3 Estimating ReliabUity of Information 

People tend to overestimate the reliability of information, ignoring factors such 

as sampling error and imprecision of measurement. Rather they summarize evidence in 

terms of simple and definite conclusions, causing them to be overconfident in their 

judgments. This tendency is stronger when one has a considerable amount of intellectual 

and/or personal involvement in a particular field. In such cases, information is often 

interpreted in a way which is consistent with one's beliefs and expectations, results are 

overgeneralized, and contradictory evidence is ignored or underestimated. 

A.7.4 Relation Between Event Importance and ProbabUity 

Sometimes the importance of events, or their possible costs or benefits, 

influence judgments about the certainty of the events when, rationally, importance 

should not affect probability. In other words, one's attitudes towards risk tend to affect 

one's ability to make accurate probability judgments. For example, many physicians tend 

to overestimate the probability of very severe diseases, because they feel it is important 

to detect and treat them; and similarly, many smokers underestimate the probability of 
P.55 

49 

adverse consequences of smoking, because they feel that the odds do not apply to 

themselves personally. 

A.7.5 Estimation of Probabilities 

Another limitation is related to one's ability to discriminate between levels of 

uncertainty, and to use appropriate criteria of discrimination for different ranges of 

probability. One result of this fact is that people tend to estimate both extreme and 

mid-range probabilities in the same fashion, usually doing a poor job in the extremes. It 

helps here to think in terms of odds as well as probabilities. Thus, for example, changing 

a probability estimate from 0.510 to 0.501 is equivalent to a change in odds from 1.041:1 

to 1.004:1, but a change from an estimate of 0.999 to 0.990 changes the odds by a factor 

of about 10 from 999:1 to 99:1 The closer to the extremes (either 0 or 1) that one is 

estimating probabilities, the greater the impact of small changes. 

A.7.6 Recommendations 

Although extensive and careful training would be necessary to eliminate all the 

problems mentioned above, some relatively simple suggestions can help minimize them. 

Most important is to be aware of one's natural cognitive biases, and to try consciously to 

avoid them. 

To avoid sequential effects keep in mind that the order in which you think of 

information should not influence your final judgment. It may be helpful to actually note 

on paper the important facts you are considering and then to reconsider them in two or 

more sequences, checking the consistency of your judgments. Try to keep an open mind 

until you have gone through aU the evidence, and don't let the early information you 

consider sway you more than is appropriate. 

To avoid adverse memory effects, define various classes of information that you 

deem relevant, and then search your memory for examples of each. Don't restrict your 
P.56 

50 

thinking only to items that stand out for specific reasons. Make a special attempt to 

consider conflicting evidence, and to think of data that may be inconsistent with a 

particular theory. Also, be careful to concentrate on the given probability judgment and 

do not let your own values (how you would make the decision yourself) affect those 

judgments. 

To accurately estimate the reliability of information, pay attention to such 

matters as sample size and power of the statistical tests. Keep in mind that data are 

probabilistic in nature, subject to elements of random error, imprecise measurements, 

and subjective evaluation and interpretation. In addition, the further one must 

extrapolate, or generalize, from a particular study to a situation of interest, the less 

reliable is the conclusion and the less certainty should be attributed to it. Rely more 

heavily on information which you consider more reliable, but do not treat it as "absolute 

truth." 

Keep in mind that the importance of an event or an outcome should not influence 

its judged probability, it is rational to let the costliness or severity of an outcome 

influence the point at which action is taken with respect to it, but not the judgment that 

is made about the outcome's likelihood. 

FinaUy, in making probability judgments, think primarily in terms of the measure 

(probability or odds) with which you feel more comfortable, but sometimes translate to 

the alternative scale, or even to measures of other events (e.g,, the probability of the 

event not happening). When estimating very small or very large likelihoods, it is usually 

best to think in terms of odds, which are unbounded, instead of probabilities, which are 

bounded. For example, one can more easily conceptualize odds of 1:200 than a 

probability of .005. 
P.57 

51 

A.8. FINAL PREPARATION FOR ELICITATION OF PROBABILITY JUDGMENTS 

The shape and location of the dose-response curve as we defined it above is 

uncertain, because the existing data do not determine the dose-response relationship 

exactly. Yet, if we have defined the dose-response curve precisely, in a mathematical 

sense such a relationship does exist. Our goal is to have you represent probabilistically 

your own uncertainty about the location and shape of this mathematically existent 

function, based on your expertise and the available knowledge. In responding to the 

questions we wiU ask you, please think carefuUy about the relevant information reviewed 

in the criteria document, and consult it, the literature, or your files as you deem 

appropriate. 

The previous section suggests ways to think about the relevant data. The purpose 

of that section is to help minimize the biasing effects that frequently accompany the 

information overload naturally resulting from rapid consideration of large amounts of 

complex evidence. You may find it helpful to review the section or raise questions about 

the points made in it before we begin. 

Uncertainty about the location of a dose response relationship can be represented 

probabilisticaUy in two different ways. 

1. Uncertainty about the percentages, or fractiles, of the defined 

population that would be affected by given blood-lead 

concentrations can be represented probabilistically. 

2. Uncertainty about the blood-lead concentrations that would be 

required to affect given percentages, or fractiles of the defined 

population can be represented probabilistically. 

We wiU concentrate on one way at a time, focusing primarily on the first one. 

For these purposes it is helpful to imagine that everyone in the population has a 

specified blood-lead level that has become stabilized in the manner described above 

under Exposure Conditions. Then, as a result, some percentage of the population will 
P.58 

52 

suffer the hemoglobin response. Since the members of the population can be rank 

ordered according to their susceptibility, when we speak of a particular percentage of 

the population being affected, say 10%, we mean precisely that everyone in the 

population up to and including the person in that fractile, in this case the .10 fractile, has 

been affected. 

Now, in order for us to determine your uncertainty about the fractiles of the 

population that would be affected by given blood lead levels we must introduce a 

definition. Let C be the given blood-lead concentration in question. 

Definition: The R(C)-benchmark is the fractile of the population for which a 

blood-lead level of C would cause the defined hemoglobin health effect. 

Thus, R(C) is precisely the fraction of the population that would show a response if the 

entire population had blood-lead levels of C under the conditions defined above. 

The value of R(C) for a given C is uncertain, and we would like to obtain 

probability judgments from you about its possible values. We will elicit your judgments 

about the possible values of R(C) by specifying a particular percentage or fractile r, and 

having you consider how likely it is that the R(C)-benchmark is less than that value. To 

help you make your probability judgments, we wiU make use of a device called a 

probability wheel, which has adjustable sectors of blue and orange. We can read on the 

back of the wheel the percentage of the wheel that is each color. In making your 

judgments you are to imagine that the wheel is a perfectly fair random device, and that 

therefore the probability of its stopping with the pointer on blue is exactly represented 

by the relative area that is blue. 

We wiU then proceed as follows. For each blood lead concentration C, we will 

specify a particular fractile r, and also set the probability wheel to have a specific 

relative area of blue. Then you are to consider carefuUy the question: 
P.59 

53 

Do you consider it more probable that the R(C)-benchmark is less than r or that 

the wheel would stop with the pointer on blue (on a random spin)? 

You can give one of three responses: 

(1) You judge it to be more probable that R(C) is less than r; 

(2) You judge it to be more probable that the wheel would stop with 

the pointer in blue; 

(3) You cannot judge either event as more probable than the other. 

For a particular concentration C and fractile r, some wheel settings will have a 

small enough relative area of blue that you will feel confident making response (1). 

Other wheel settings will have a large enough relative area of blue that you wiU feel 

confident making response (2). The intermediate settings wiU be more difficult to 

judge. However, for a fixed blood lead C, and population response rate, r, we will 

manipulate the wheel settings to find the one for which you feel most comfortable 

making response (3). 

Once we have determined the point at which you are most comfortable with 

response (3), and still focusing on the given blood lead C, we wiU specify a new fractile 

r', reset the wheel to a new relative area of blue, and proceed to obtain your judgments 

regarding the probabilistic relation between r' and the R(C)-benchmark. This will 

continue for the given C until we have obtained comparisons of various fractiles to the 

benchmark R(C). Then we will have elicited one of the probabilistic representations of 

your uncertainty about the dose-response relationship we need. We will obtain the next 

probabilistic representation by specifying a new C and continuing as before. We will do 

this for a number of values of C. 

It frequently happens that an expert's judgments alter somewhat over the course 

of a session such as this, as he or she considers the evidence from various perspectives 

and thinks about the various responses called for. Hence, we will graph your responses 

and, at appropriate times, show them to you for your consideration and comparison. At 

these times you may wish to change some of the judgments you gave earlier. 
P.60 

54 

Also, although the probability judgments are entirely your own, we must 

introduce one logical constraint: namely, that your final judgments are coherent in a 

sense that we can explain as we go along. It is quite common for initial judgments to 

exhibit some incoherencies. That is another reason we wiU consider together the graphs 

of your judgments. Our objective is to obtain a coherent set of judgments that represent 

your opinions weU by the end of this elicitation process. You are not expected to give us 

such a set immediately. All of this wiU become clearer as we go along. 

We should emphasize that the judgments we are asking you to make are not 

simple ones, nor of course are there known correct answers. Rather, we want your best 

and most considered judgment in light of the available relevant scientific data, 

Therefore, please reflect on the available data carefuUy, feeling free to consult the lead 

criteria document or other sources as you wish as you formulate your judgments. 

(Encode judgments for two concentrations of one dose-response function, then 

continue with instructions.) 

Recall that uncertainty about the dose-response function can also be expressed in 

terms of the blood-lead concentration necessary to produce the effect in a given 

percentage or fractile of the population r, if the entire population had the same bloodlead concentration, stabilized as the result of exposure in the manner discussed earlier. 

More specifically, proceeding as before, let us introduce a definition: 

Definition: The C(R)-benchmark is the blood-lead level that would cause the 

defined hemoglobin health effect in the person at fractile R of the population. 

The value of C(R) is uncertain for a given R, and we want to obtain your 

judgments about its possible values. Analogously to what we have already done in 

obtaining the R(C)-benchmark, for a given R we wiU specify concentrations c and ask you 
P.61 

55 

to consider how likely it is that the C(R)-benchmark is less than the specified c. More 

specificaUy, utilizing the wheel as before, we wiU ask you the question: 

Do you consider it more likely that the C(R)-benchmark is less than c, or that the 

wheel would stop with the pointer on blue (on a random spin)? 

You can give one of three responses: 

(1) You judge it more probable that C(R) is less than c; 

(2) You judge it more probable that the wheel would stop with the 

pointer in blue; 

(3) You cannot judge either event as more probable than the other. 

After determining the wheel setting at which you feel most comfortable with 

response (3), we will specify a new c and repeat the process. As before, for each R we 

wiU elicit your comparative judgments for various values of c. 
56 

APPENDIX B 

DETAILED MATHEMATICAL FORMULATION 

B.1 OUTLINE 

This appendix presents detailed discussions of the mathematical techniques that 

are summarized in Sec. 3 of the report. The techniques are used to represent the 

expert's probability judgments about dose-response relationships, and to combine these 

judgments with a blood-lead distribution to obtain a risk estimate. 

First, basic definitions and notation are introduced. Then the normal-on-log-odds 

function is discussed as a distribution to be fitted to the expert's probability judgments 

that meets certain of the criteria specified in Sec. 3.1. Methods are presented for 

obtaining least-squares estimates of the parameters of a normal-on-log-odds distribution 

and for assessing the goodness of fit. A family of equal-variance normal-on-log-odds 

distributions may sacrifice something in terms of fit to the judgments, but it meets all 

the criteria specified in Sec. 3.1. This family is presented next, along with methods for 

obtaining least-squares estimates of its parameters and for assessing its goodness of fit. 

Finally, techniques are presented for deriving risk estimates by combining a blood-lead 

distribution with an expert's probability judgments about dose-response relationships, 

where the latter is represented either by independent normal-on-log-odds distributions or 

by a family of equal-variance normal-on-log-odds distributions. 

B.2 BASIC DEFINITIONS AND NOTATION 

This subsection presents definitions and notation. It is assumed that the reader is 

familiar with the basic concepts of probability theory (i.e., definitions of random 

variables, probability density functions, expectation and higher moments, etc.). 

For uncertain quantity X, let 

f x 00 = judgmental probability density function (PDF) for X 

Fx(x0) = cumulative judgmental density function (CDF) for X 
P.63 

57 

x 

= J ° fx(x)dx. 

Thus Fv(xQ) is the judgmental probability* that X is less than or equal to xQ. 

Furthermore, let 

u„ = E(X) = expected value of X 

f x fv(x )dx J

 o X o o 

o£ = V(X) = variance of X 

= J (x - E(X)) 2f v ( x )dx . J

 o X o o 

The uncertain quantities of interest in of this paper are population response 

rates, denoted R, at each of several blood-lead levels, denoted L. Through encoding 

sessions with recognized experts, judgments representing a cumulative density function 

(CDF) for a response rate R given a population blood-lead level L wiU be elicited (i.e., 

judgments representing FR/L(R)) f° r a number of blood-lead levels. Taken together, 

these wiU provide a family of probabilistic relationships as idealized in Fig. 3.1, and 

already presented more realisticaUy in Figs. 2.1 and 2.2. Each curve in Fig. 3.1 

represents the judgmental probability that the population response rate (a fraction 

between 0 and 1) will be less than or equal to a particular value R, given a blood-lead 

level of L, for a weU-defined adverse health affect, a well-defined population of interest, 

and under weU-defined exposure conditions. 

As already discussed, it wiU generally be necessary to interpolate between 

specifically assessed points in performing risk calculations. Such interpolation is best 

•Hereafter, aU probabilities referred to shall mean judgmental probabilities, even if not 

explicitly stated. 
P.64 

58 

accomplished by fitting a function to the judged probabilities. The next subsection 

describes a method for doing this. 

B.3 THE NORMAL-ON-LOG-ODDS FUNCTION 

Because the probabilistic judgments about the dose-response relationships appear 

generally to be "S-shaped" over the closed [0,1] interval, as demonstrated in Figs. 2.1 and 

2.2, as weU as by the judgments of all the health experts encoded by Wallsten, Forsyth, 

and Budescu (1983), it can be difficult to represent them with closed-form mathematical 

functions. One particularly useful function (which is relatively easy to work with) is the 

normal-on-log-odds distribution. The normal-on-log-odds distribution is obtained by 

4 

fitting a normal distribution to the natural log of the odds implied by the population 

response rates R. Thus, the following relationships are defined: 

x = x^ , o < x < 

Y = ln(X), -» < Y < » 

where X is the odds variable and Y is the log-odds variable. It is assumed that Y is 

normally distributed with mean u and variance o . The degree to which the assumption is 

appropriate can be tested with each set of judgments. If Y is normally distributed, then 

X is lognormally distributed; and while there is no closed-form expression readily 

available for the distribution on R, aU probabilistic and statistical results of interest on 

R can be obtained through the distribution on Y since 

pr(R < r) = pr(X < ^ ) = pr(Y < In^) ) 

and 

pr(Y < y ) = pr(Z < ^ !l_) = *{I° !ll) 

— o v — a,, ^ OA, 
P.65 

59 

where pr(-) denotes probability and Z is the unit normal random deviate for which *(Z) is 

extensively tabulated and available in computer libraries. Thus 

y 

Table B.l summarizes quantities of interest for these variables. 

The parameters u and a can be estimated in a least-squares sense from an 

assessed distribution. Let the n assessed points for a CDF be denoted (Rj, Fj) for i=l, ..., 

n where Rj is the population response rate and Fj is the associated cumulative judgmental 

probability. 

Least squares estimates u and o for u and a can be obtained by defining 

.2 rf

 y i " *\2 

E' = l[z

, - _i y 

taking derivatives with respect to u and o and equating the results to zero, 

3U 

3Ei 

3a 

. = 0 

u = u 

. = 0 

o=o 
P.66 

60 

TABLE B.1 Summary of Variables Pertaining to the Normal-On-Log-Odds Function 

Defining Equations 

Distribution3 

Meanb 

Variance 

Median 

Mode 

Y 

Y = ln(X) 

N(u,o2) 

u 

o

2 

u 

u 

2 

a 

X 

Variable 

X 

X = eY 

X = R/(l-R) 

A(u,o ) 

_ u+a2/2 

u = e 

X 

2 2 

2u+o r ° = e le -

e» 

2 

eu-a 

•1) 

R 

R »-5_ 

R 1+X 

7 

2 

u o 

X X 

X (1+u / 

X 

2 f 1 ^4 

°x ^1+u

 ] 

X 

u 

e 

l*eu 

2 u-o 

e 

2 

l*eM~° 

aN(u,a2) denotes the normal PDF; A(u',o2) denotes the lognormal PDF; a 

closed-form expression for the PDF for R is not obtainable. 

Mean and variance expressions for R are approximate; they were derived via 

the method of moments; other estimates can be obtained via numerical methods 

on FY = N(u,o2). 
P.67 

61 

and solving for u and a, where 

z. = •" 1(F.) 

1 1 

is the value of the unit normal variate that corresponds to a cumulative probability of Fj, 

and y} = ln(Rj/(l-Rj)). The results are 

(I v\) il «L) - (I /-)( ! JO 

u = 

a = 

(I y j - "M 

il'i) 

(B.l) 

(B.2) 

where aU of the summations are for i=l, ..., n. 

These estimates can also be obtained by linearly regressing the Zj on the yj; o is 

the reciprocal of the slope of the regression equation and G is the y-intercept, i.e., the 

value of y corresponding to z = 0 in the regression equation. Since m judgmental 

distributions are assessed, one for each blood-lead level j for j = 1, ..., m, there are m 

means (ui, ... , um) and standard deviations (a^, ... , am), one pair for each of the 

judged distributions. 

The goodness of fit of each transformed, assessed distribution is given by the 

standard regression r2 statistic 

2 

r 

LK l l ' 1 - — ^ 3^ - (B.3) 

K«i - z) 

where z denotes the estimated Z- values {i.e. , z^ = (y^ - u)/o} and Z is the mean 

of Z. If r" is sufficiently high, then it can be claimed that the normal-on-log-odds 

distribution with parameters u; and o; describes the judgments well for lead level j . 

This distribution can then be used for interpolation. 
P.68 

62 

To summarize, formulas were presented to calculate the parameters of a normalon-log-odds distribution which can be used to represent a judged probability distribution 

over population response rate for a given blood-lead level. Goodness of fit is given by 

the regression r2 statistic. The normal-on-log-odds distribution variable and its PDF can 

be used to estimate judgmental probabilities for the response rate variable R in order to 

calculate risk estimates. 

B.4 REPRESENTING A PROBABILISTIC DOSE-RESPONSE SURFACE WITH A FAMILY 

OF EQUAL VARIANCE NORMAL-ON-LOG-ODDS DISTRIBUTIONS 

If the o; are approximately equal, it is desirable to set them equal to a single 

pooled value before proceeding further (i.e., set o: = o' = a = o for all j), because this 

assures that the derived distributions at two different lead levels never cross one 

another. (Crossing would imply that exceeding a specified response rate is judged more 

probable at blood-lead level L, than at L«, where L? is greater than L,.) Because of the 

manner in which judgments were obtained from the pilot subjects, crossings did not 

occur. Furthermore, judgments will be obtained from the health experts over a wider 

probability range so that crossings would only be possible at cumulative probabilities very 

near to 0 and 1. However, even crossings such as these would represent an inconsistency 

the experts ought to avoid, and therefore should not be allowed. 

A pooled estimate of o, denoted a' can be obtained by first subtracting off the 

mean u.- for each set of assessed points, 

y . . = y . . - u. , i =l » ... , n. , j-1 , ... , m 

where n- is the number of assessed points for Fn/, and y- j is an assessed point on 

^R/L-" ^"he y ' • ar e the n use d t 0 ca l

c u la t e a mean (which should be very close to zero) 

and a variance (which wiU be the pooled, least squares estimate a* ) using Eqs. B.l and 
P.69 

63 

B.2, respectively. FinaUy, the individual, least-squares estimates of the means are 

recalculated by noting that Eq. B.2 can be repeatedly solved for each ui in terms of o' . 

Denote the results of these last steps as uT, •••, u' . Goodness of fit again can be 

1 m 

calculated with Eq. B.3. 

If a functional relationship can be established between blood-lead levels and the 

u ' , a judgmental probability density function over R can be calculated for any blood-lead 

level, should that be necessary for interpolating between blood-lead levels. This 

continuum of density functions in essence is a probability surface over the dose-response 

plane. 

B.5 COMBINING BLOOD-LEAD DISTRIBUTIONS WITH PROBABILITY JUDGMENTS 

ABOUT DOSE-RESPONSE FUNCTIONS 

Using either the equal-variance or the best-fitting normal-on-log-odds functions 

fit to the judgments in the manner described above, it is possible to estimate doseresponse functions that correspond to any given judgmental cumulative probability. For 

example, a median dose-response curve based on an expert's probability judgments is 

obtained by determining the response rate that corresponds to F^ ^ = 0.50 at each blood 

lead L. The dose-response curve, such that there is a 5% chance of the true response 

rate lying above the curve at any given blood lead, is obtained by determining the 

response rate that corresponds to F^ ^ = 0.95 at each L, and so forth. Let Dp(L) 

represent the dose-response curve that corresponds to cumulative probability F. 

Any blood-lead distribution can then be combined with the dose-response curve 

corresponding to judgmental cumulative probability F, Dp(L), to yield an estimate of the 

overall fraction of the population R suffering the adverse health effect with cumulative 

probability F. If there is also uncertainty about the blood-lead distribution, that must be 

taken into account as well, but that problem wiU not be treated in this report. 
P.70 

64 

We wiU consider here only the empirical blood-lead distribution from the 

NHANES-II survey. The distribution is presented in discrete form with intervals of 0-10 

ug/dl, 10-19 ug/dl, ..., 60-69 ug/dl. Let 

P(L:) = proportion of the population in the interval centered at Lfor j=l, ..., 6. If judgmental probabilities for dose-response relationships are encoded 

conditional on blood-lead levels L:, then the NHANES-II distribution can be combined 

with any DF(L) to obtain the overall estimate of R with cumulative probability. F, 

according to 

i

 6 

R = F X(R) = I P(L.)-D (L.) (B.4) 

j = 1 J J 

where F^ 1 (R) is the inverse of F^(R), the derived cumulative probability distribution for 

R like that indicated in Fig. 3.2. If for some reason judgmental probabilities are not 

encoded conditional on a particular Ls, Dp(L:) can be obtained by interpolation on the 

probability surface derived over the dose-response plane.* 

For example, if the P(L:) are combined with Dg 5Q(L:) in Eq. B.4, the result is a 

median estimate of the fraction of the population suffering the adverse health effect. In 

general, combining the blood-lead distribution with a dose-response curve corresponding 

to judgmental cumulative probability F yields an F-fractile estimate of the fraction of 

the population suffering the health effect. By carrying out this calculation for all values 

of F, 0 < F < 1, and plotting each F as a function of the derived R, a judgmental CDF is 

obtained over affected fraction of the population, as illustrated in Fig. 3.2. We call this 

CDF a derived risk distribution. 

•It should also be noted that Eq. B.4 is easily generalized should a blood-lead distribution 

be presented in a continuous or a different discrete form. 
P.71 

65 

Figure 3.2 represents a hypothetical derived risk distribution for a particular 

population and a particular adverse health effect, derived from an expert's probability 

judgments about dose-response functions combined with a blood-lead distribution 

according to Eq. B.4 for all values of F, 0 < F < 1. One can read from it a median 

estimate of the affected fraction of the population, a 95% credible interval for that 

fraction, or whatever is desired. 
P.72 

66 

APPENDIX C 

DETAILED ILLUSTRATIVE CALCULATIONS 

The analytical methods derived in Appendix B should be applied to the judgments 

of health experts only after the methods have been tested and shown to describe the 

judgments with sufficient accuracy. In this appendix best-fitting normal-on-log-odds 

distributions and equal-variance normal-on-log-odds distributions are derived for the two 

pilot subjects, and their fits are assessed. In addition, risk estimates are derived based 

on the probability judgments from each pilot subject. 

C.l FITTING NORMAL-ON-LOG-ODDS DISTRIBUTIONS TO JUDGMENTS 

C.l.l PUot Subject 1 

Recall that the probability judgments from this subject are shown in Table 2.1 

and Fig. 2.1. Transforming the judgments via the normal-on-log-odds function yields the 

results in Table C.l. Applying Eqs. B.l and B.2 to these data yields the estimates UJ 

and OJ (j = 1, ..., 6) listed in Table C.2, as well as the regression r values shown in the 

table. Figure C.l shows the CDFs from the best-fitting normal-on-log-odds distributions 

and the assessed values. 

The high regression r-squares in Table C.2 (the smallest is 0.97) indicate very 

good fits of the separate normal-on-log-odds functions. If the transformations departed 

appreciably from normaUy distributed random variables, then the regression r-squares 

would be much lower. These good fits can be seen visually in Fig. C.l, where the normalon-log-odds functions closely match the assessed probabilities. 

However, note in Fig. C.l that the CDF for L = 35 crosses those for L = 25, 45, 

and 55 ug/dl. This result occurred, because, as already discussed in Sec. 2.4.1, 

probabilities were not encoded for a sufficiently full range for L = 25 and 45 ug/dl. If the 

lower probabilities had been encoded for these two lead levels, then the subject would 
P.73 

67 

TABLE C.l Normal-On-Log-Odds Distribution Transformation of 

ProbabUity Judgments in Table 2.1 for Suoject 1 

Blood-Lead Levels 

R

a

 Yb Zc R Y Z 

5 ug/dl 15 ug/dl 25 ug/dl 

0.02 -3.89 -1.04 0.02 -3.89 -1.65 0.2 -1.39 -0.61 

0.04 -3.18 0.39 0.04 -3.18 -0.31 0.3 -0.85 0.05 

0.06 -2.75 0.81 0.06 -2.75 0.13 0.4 -0.41 1.13 

0.08 -2.44 1.23 0.08 -2.44 0.52 0.5 0.00 1.88 

0.10 -2.20 2.05 0.10 -2.20 0.95 

35 ug/dl 45 ug/dl 55 ug/dl 

0.10 -2.20 -1.47 0.35 -0.62 -0.84 0.30 -0.85 -1.88 

0.26 -1.05 -0.58 0.48 -0.08 -0.05 0.44 -0.24 -0.71 

0.42 -0.32 -0.10 0.61 0.45 0.55 0.58 0.32 -0.13 

0.58 0.32 0.84 0.74 1.05 2.05 0.72 0.94 0.58 

0.86 1.82 2.33 

aR is response rate. 

CZ = 4 (F), where F corresponds to R for each blood-lead 

level as listed in Table 2.1. 
P.74 

68 

TABLE C.2 Parameter Estimates, Goodness of Fit 

Statistics, and Mean Response Rates for BestFitting Normal-On-Log-Odds Distributions 

Fitted to the Judgments of Subject 1 

Blood Regression 

" 2 ~ a 

Lead uj Oj r R1 5 ug/dl -3.30 0.59 0.97 0.04 

2 15 ug/dl -2.84 0.67 0.99 0.07 

3 25 ug/dl -0.99 0.54 0.98 0.28 

4 35 ug/dl -0.44 1.13 0.97 0.19 

5 45 ug/dl -0.06 0.59 0.97 0.47 

6 55 ug/dl 0.37 0.67 0.98 0.56 

aR: represents the mean of the response rate 

distribution at each of the lead levels; the 

formula for calculating R- is listed in 

Table B.l.

 J 

have modified his judgments to avoid crossovers, possibly lowering the judgments at L = 

35 ug/dl, and the modeled distributions would also not have crossed (except, as already 

indicated, possibly at values very close to 0 and 1). 

Equal-variance (i.e., the pooled-variance estimate) normal-on-log-odds 

distributions will not demonstrate the crossovers, and indeed appear rather consistent 

with the probability judgments other than those at L = 35 ug/dl. It can be seen in Table 

C.2 that, except at L = 35 ug/dl, the OJ range from 0.54 to 0.67. Thus, there is good 

reason to determine a pooled estimate for o2 and to fit the equal-variance normal-onlog-odds distributions. 

The pooled estimate o' , calculated as described in Sec. B.4, is 0.71, with a 

P 

pooled regression r* value of 0.94. The revised least-squares estimates of the means, 

u ' , based on a pooled variance are listed in Table C.3. As expected, the two estimates 
69 

P.75 

J2 

ra 

ja 

o 

u 

a. 

Si 

> 

3 E 3 U 

0.4 0.8 1.0 

Population Response Rate , Expressed As 

Fraction of Population Affected, R 

FIGURE C.l Best-Fitting Normal-On-Log-Odds CDFs and the ProbabUity Judgments 

of Subject 1 That Were Previously Shown in Fig. 2.1 
P.76 

70 

TABLE C.3 Parameter Estimates, Goodness of Fit 

Statistics, and Mean Response Rates for the EqualVariance Normal-On-Log-Odds Distributions Fitted 

to the Judgments of Subject 1 

j 

1 

2 

3 ' 

4 

5 

6 

Blood 

Lead 

5 ug/dl 

15 ug/dl 

25 ug/dl 

35 ug/dl 

45 ug/dl 

55 ug/dl. 

u' 

J 

-3.38 

-2.84 

-1.09 

-0.58 

-0.11 

0.37 

0' 

P 

0.71 

0.71 

0.71 

0.71 

0.71 

0.71 

Revised3 

2 

r Values 

0.95 

0.99 

0.93 

0.62 

0.94 

0.98 

I. 

J 

0.04 

0.07 

0.26 

0.35 

0.45 

0.55 

aThat is, for the equal-variance normal-on-logodds functions. 

R' represents the mean of the response rate 

distribution at each of the lead levels for 

the equal-variance normal-on-log-odds _ 

functions; the formula for calculating R' 

is listed in Table B.l.

 J 

of u (comparing Tables C.2 and C.3) do not differ substantially from each other except 

for the L = 35 ug/dl case. Similarly, regression r-squares for the equal-variance normalon-log-odds functions are about the same as those for the best-fitting normal-on-log-odds 

functions except for L = 35 ug/dl. The CDFs from the fitted equal-variance distributions 

have been shown in the body of the report as Fig. 4.1. 

Also listed in Tables C.2 and C.3- are the mean values of the response rate 

variables (R and R', respectively) for the best-fitting and equal-variance normal-onlog-odds functions. Note that in Table C.2, the best-fitting functions result in a mean at 

L = 35 ug/dl that is lower than at L = 25 ug/dl. However, for the equal-variance fits, the 

mean values, the R' in Table C.3, increase monotonically with increasing blood-lead 

levels, as they should on rational grounds. These considerations support the 

appropriateness of the equal-variance model for describing this subject's judgments. 
71 

A final interesting representation of this subject's judgments was shown as Fig. 

4.2, based on the equal-variance normal-on-log-odds distributions in Table C.3. The 

curve labeled 0.50-cumulative probability is the dose-response curve Dg -Q(L), plotting 

median response rate r ' as a function of blood lead level L;, where r' = s—, and 

-»

 ] J u1 + e J 

the u' are those listed in Table C.3. This is a dose-response curve representing the 

median response rate at each lead level, based on the judgments of subject 1. 

The" dose-response curves for 0.05- and 0.95-cumulative probabilities, DQ.OS^) 

and Dg gc(L), respectively, are calculated in an analogous fashion to that for On.50^ 

and together indicate 90% credible intervals on response rates. That is, the two curves 

indicate the range of response rates at any given lead level that has a 0.90 probability of 

containing the true response rate based on subject l's judgments. Note the S shape of all 

three dose-response functions (especially the top two), mirroring the subject's belief, 

discussed earlier, of a threshold. 

To summarize, both the best-fitting and the equal-variance normal-on-log-odds 

distributions fit the subject's judgments very well. The latter has no inconsistent 

implications and fits the judgments almost as well as does the former, suggesting that it 

is suitable for use in obtaining risk estimates. Calculation of risk estimates based on 

pilot subject l's judgments wiU be iUustrated foUowing further evaluation of the normalon-log-odds distribution on the other subject's judgments. 

C.1.2 PUot Subject 2 

As for subject 1, Eqs. B.l and B.2 were used to fit normal-on-log-odds 

distributions to the encoded probability judgments for subject 2 that are listed in Table 

2.2 and shown in Fig. 2.2. The resulting estimates u; and a;, as well as the regression 

r values are shown in Table C.4. Figure C.2 shows the CDFs from the fitted normal-onlog-odds distributions and the assessed values. Recall that judgments were obtained only 
P.78 

1.0 

72 

0.8 0.4 0.6 0.8 1.0 

Population Response Rate, Expressed As 

Fraction of Population Affected. R 

FIGURE C.2 Best-Fitting Normal-On-Log-Odds CDFs for the ProbabUity Judgments 

of Subject 2 That Were Previously Shown in Fig. 2.2 
73 

TABLE C.4 Parameter Estimates, Goodness of Fit 

Statistics, and Mean Response Rates for the 

Normal-On-Log-Odds Distributions Fitted to the 

Judgments of Subject 2 l-l. 

1 

2 

3 

Blood 

Lead 

15 ug/dl 

35 ug/dl 

55 ug/dl 

h 

-1.28 

-0.67 

-0.05 

h 

0.51 

0.69 

0.89 

Regression 

r 

0.97 

0.99 

0.95 

V 

0.23 

0.34 

0.41 

aR- represents the mean of the response rate 

distribution at each of the lead levels; 

the formula for calculating R- is listed in 

Table B.l. 

for blood-lead levels of 15, 35, and 55 ug/dl. Again, the high r-square values indicate 

good fits of normal-on-log-odds functions to the judgments. Note the strong linear 

relationship between blood-lead level and both u and a , which was not the case for 

subject 1. 

Because judgments were not obtained at three of the desired blood-lead levels 

(due to a lack of time with subject 2), it was necessary to estimate them in order to 

calculate the desired risk estimates. This was done by linearly regressing the u j and o; 

in Table C.4 onto the blood lead levels L: to obtain the revised estimates 

v'. and a', shown at each of the six blood-lead levels listed in Table C.5. Figure C.3 

shows the full set of normal-on-log-odds distributions corresponding to the parameter 

estimates in Table C.5. Also listed in Table C.5 are the regression r-squares for the 

fitted functions at 15, 35, and 55 ug/dl; they are essentially the same as the r-square 

values listed in Table C.4 because of the high degree of linearity between blood-lead 

level and the u and a values. The mean values of the population response rates at each 

blood-lead level, based on the fitted normal-on-log-odds functions, are also listed in 

Table C.5. They range from 0.18 at L = 5 ug/dl to 0.41 at L = 55 ug/dl. 
P.80 

74 

TABLE C.5 Parameter Estimates, Goodness of Fit 

Statistics and Mean Response Rates for the 

Linearly Regressed Normal-On-Log-Odds 

Distributions, Applied to the Judgments 

of Subject 2 

j 

1 

2 

3 

4 

5 

6 

Blood 

Lead 

(ug/dl) 

5 

15 

25 

35 

45 

55 

A 

.a 

U. 

J 

-1.59 

-1.28 

-0.97 

-0.67 

-0.36 

-0.05 

3 

0.41 

0.50 

0.60 

0.69 

0.79 

0.89 

2 

r Values 

NAC 

0.97 

NAC 

0.99 

NAC 

0.95 

R-b 

0.18 

0.23 

0.28 

0.34 

0.38 

0.41 

aLinear regression models of the results in 

Table C.4 were used to estimate the u' 

and a", values.

 J 

J 

Estimated mean values of the modeled 

distributions for population response rate 

conditional on blood-lead level. 

cNot appropriate; judgments were not 

obtained at this blood-lead level. 

However, as already discussed, equal-variance normal-on-log-odds distribution 

representations of the probability judgments are preferable on rational grounds. These 

were derived by combining all of the encoded judgments to yield a pooled estimate of a, 

denoted a', in the manner described in Appendix B. The estimated pooled variance is 

o' = 0.74, with a pooled regression r-square value of 0.92. The corresponding leastsquares estimates for the u' , calculated as described in Appendix B, are -1.28, -0.67, 

and -0.05 for L = 15, 35, and 55 ug/dl, respectively. Regressing these estimated values on 

blood-lead level produces the ~ . , values for equal-variance normal-on-log-odds functions 
P.81 

75 

1.0 

ra 

JO 

O 

> 

3 

E 

3 

U 

Population Response Rate, Expressed As 

Fraction of Population Affected, P. 

FIGURE C.3 A Complete Set of Six Modeled Normal-On-Log-Odds CDFs for the 

Response Rate Variable Based on the Parameter Estimates in Table C.5 
P.82 

76 

that are listed in Table C.6. Also listed in the table are the r-square values for the 

associated normal-on-log-odds functions and the mean population response rate at each 

of the six blood-lead levels of interest. These are slightly lower than the r-square values 

in Table C.4, but stiU quite respectable. 

The normal-on-log-odds distributions corresponding to the parameters in Table 

C.6 were shown in the report as Fig. 4.3, along with judgments elicited at the three 

blood-lead levels. Figure 4.4 showed the corresponding 0.05-, 0.50-(or median) and 

TABLE C.6 Parameter Estimates, Goodness of Fit 

Statistics, and Mean Response Rates for Normal-OnLog-Odds Functions with Equal Variance Fitted to 

the Encoded Judgments of Subject 2 Through 

the Use of Linear Regression Techniques as 

Described in the Text 

j 

1 

2 

3 

4 

5 

6 

Blood 

Lead 

(ug/dl) 

5 

15 

25 

35 

45 

55 

u . 

J 

-1.81 

-1.46 

-1.09 

-0.73 

-0.37 

-0.00 

a' 

P 

0.74 

0.74 

0.74 

0.74 

0.74 

0.74 

2 

r Values 

NAC 

0.87 

NAC 

0.99 

NAC 

0.91 

R "b 

0.16 

0.20 

0.26 

0.32 

0.39 

0.47 

Result from a linear regression of the u' 

values (listed in the text, from an equalvariance representation) on blood-lead level, 

Estimated mean values of the modeled 

distributions for population response rate 

conditional on blood-lead level. 

cNot appropriate; judgments were not 

obtained at this blood-lead level. 
P.83 

77 

0.95-cumulative probability estimates of response rate at each of the six blood-lead 

levels. Unlike the curves for subject 1, the curves of equal probability for subject 2 are 

very nearly straight lines. 

To summarize these and the previous results, the normal-on-log-odds function 

applied to individual subjective probability distributions generally provides excellent fits, 

with r-squares on the order of 0.95 and higher. Constraining the normal-on-log-odds 

distributions to have equal variance for an entire set of distributions still yields very 

good fitting functions, but with, of course, somewhat lower r-squares, while 

simultaneously avoiding the possibility of inconsistencies. Since, as discussed previously, 

probability judgments will be obtained from the health experts over virtually the full 

[0,1] interval for six lead levels, it can be expected that equal-variance functions will be 

quite accurate. 

C.2 ILLUSTRATIVE RISK CALCULATIONS 

Having established that the normal-on-log-odds distribution describes the 

probability judgments very accurately, the next step is to combine the judgment 

distributions with the NHANES-II blood lead distribution to yield derived risk 

distributions in the manner described in Sec. B.5. This is done here with the judgments of 

both pilot subjects using the NHANES-II distribution for children aged 6 months to 5 

years, to illustrate the method. 

C.2.1 PUot Subject 1 

Table C.7 iUustrates the calculations required to obtain a median estimate of the 

percent of children aged 0-3 with hemoglobins less than or equal to 11 g/dl. These 

calculations are based on the best-fitting, not the equal-variance, normal-on-log-odds 

distributions. For each lead level, L: = 5, 15, ..., 55 ug/dl, the table shows the mean of 

the normal-on-log-odds distribution, uj, the associated response rate, r^, the NHANES-II 
78 

TABLE C.7 Calculations for a Median Estimate of the 

Fraction of the Population Affected, Based on BestFitting Normal-On-Log-Odds Distributions Fitted to 

the Judgments of Subject 1 

j 

1 

2 

3 

4 

5 

6 

Lj(ug/dl) 

5 

15 

25 

35 

45 

55 

*j 

-3.30 

-2.84 

-0.99 

-0.44 

-0.06 

0.37 

°j 

0.59 

0.67 

0.54 

1.13 

0.59 

0.67 

' / 

0.04 

0.05 

0.27 

0.39 

0.49 

0.59 

P(Lj)

b 

0.122 

0.633 

0.205 

0.035 

0.003 

0.001 

1.000c 

P(Lj)»J 

0.005 

0.032 

0.055 

0.014 

0.001 

0.001 

0.110c 

a r . = e"j/(l • eU j. 

bP(L:) denotes the fraction of NHANES-II children 

witn blood leads in the range of L- ± 5 ug/dl. 

CoLumns do not always sum due to rounding. 

fraction of children with that blood lead, P(L:), and the product, P(L.-)r: for each bloodlead level. Summing over the product and multiplying by 100% yields the median 

estimate that 11.0% of the children aged 0-3 have hemoglobins less than or equal to 11 

g/dl. 

As discussed in Sec. B.5, combining the dose-response curves corresponding to all 

cumulative probabilities F, 0 < F < 1, with a blood-lead distribution yields a judgmental 

CDF over affected fraction of the population, which is a derived risk distribution based 

on the expert's judgments. The derived risk distribution calculated from this subject's 

probability judgments is given in Fig. C.4. It can be seen from this risk distribution 

that: the median estimate of the percent of the population that are affected is 11.0% (as 

already discussed); with probability 0.05 more than 23.4% of the population are affected; 
P.85 

79 

ra 

Xi 

o 

u 

a. 

0) 

> 

E 

3 

U 

0.00 0.05 0*10 0.15 0.B0 0.25 0.30 0.35 

Population Response Rate, Expressed As 

Fraction of Population Affected, R 

FIGURE C.4 Derived Risk Distribution Based on Best-Fitting Normal-On-Log-Odds 

Distributions for Subject l's Judgments, Showing the Cumulative Probability F 

That Fraction R of the Population of U.S. ChUdren Aged 0-3 Have Hemoglobin 

Levels Less Than or Equal to 11 g/dl 
P.86 

80 

and with probability 0.01 more than 30.5% are affected. The estimate of the mean of 

the distribution is a response rate of 11.3%. 

The preceding risk calculations used the best-fitting normal-on-log-odds 

distributions. Repeating the calculations utilizing the equal-variance normal-on-log-odds 

distributions yields almost identical results. The median risk estimate is that 10.5% of 

the population suffer the health effect (compared to the previous estimate of 11.0%). 

With probability 0.05, more than 24.3% are affected (compared to 23.4%), and with 

probability 0.01, more than 32.4% are affected (compared to 30.5%). The mean estimate 

is 11.5%. Figure C.5 compares the two risk estimates, showing their essential identity. 

Since the equal-variance distributions are better on theoretical grounds, their use seems 

preferable. 

C.2.2 PUot Subject 2 

Combining F-fractile dose-response curves based on the judgments of subject 2 

with NHANES-II data on blood-lead distribution for children aged 6 months - 5 years 

produces a different set of risk estimates. The solid curve in Fig. C.6 iUustrates the 

derived risk distribution using the regressed but best-fitting normal-on-log-odds 

distributions in Table C.5. According to this risk distribution, the median percent of the 

population affected is 22.8%, while with probability 0.05, more than 40.9% of the 

population suffer the health effect, and with probability 0.01, more than 49.5% do so. 

The estimated mean of the distribution is 23.6%. 

The broken curve in Fig. C.6 shows the derived risk distribution obtained by 

combining the NHANES-II blood-lead distribution with the F-fractile dose-response 

curves based on the equal-variance normal-on-log-odds distributions in Table C.6. This 

risk distribution yields a slightly lower median estimate (20.1%) and slightly higher 0.95-

and 0.99-fractile estimates (45.6% and 58.0%, respectively) than does the other one. The 

estimated mean is a response rate of 21.4%. 
81 

C.3 COMPARISON OF RESULTS FOR SUBJECTS 1 AND 2 

Only the risk distributions based on the equal-variance normal-on-log-odds 

distributions will be compared here. Figure 4.5 showed the derived risk distributions 

based on the equal-variance normal-on-log-odds functions for subject 1 (the parameters 

of which are summarized in Table C.3) and subject 2 (the parameters of which are 

summarized in Table C.6). This figure was obtained by using the appropriate curves from 

Figs. C.5 and C.6. Subject l's probability judgments combined with the NHANES-II 

blood-lead distribution lead to lower and tighter risk estimates than do subject 2's. The 

estimates are said to be lower because the distribution based on subject 1 is entirely to 

the left of that based on subject 2, resulting in lower estimates of fraction of the 

population affected for any given probability level. For example, the median fraction 

affected, corresponding to a cumulative probability of 0.05, is 10.5% according to subject 

l's judgments and 20.1% according to subject 2's. The risk estimates based on subject 1 

judgments are tighter because the distribution is steeper. Thus, for example, based on 

subject l's estimates, with probability 0.90 the fraction affected is between 4% and 24%; 

based on subject 2's estimates, it is between 7% and 46%, with probability 0.90. 

There is considerable agreement in the two sets of risk estimates, however, as is 

apparent in the plot of the probability density functions (PDFs), shown in Fig. C.7, which 

were derived from the CDFs of Fig. 4.5 by differentiating with respect to response 

rate. A measure of the degree of agreement between the two risk estimates is obtained 

as foUows. For any pair of PDFs, such as in Fig. C.7, call the left-most PDF 1 and the 

right-most 2. Then there is exactly one fraction of the population, R, such that area 

under 1 to the right of R equals the area under 2 to the left of R. The sum of these two 

areas is a measure of agreement. If they agree completely, the sum is 1.0, and if they do 

not agree at all, the sum is 0.0. In Fig. C.7, this fraction R is 0.145, and the sum is 0.60. 

This measure of agreement may be useful if it becomes necessary to quantify the 

degree of concensus among experts. One measure of agreement would thus be calculated 

for every possible pair of experts. 
82 

1.0 

0.8 -

5 0.6 

J3 

ra 

x> 

o 

01 

i 0.4 

3 

E 

3 

U 

0.2 

0.0 

Best Fitting 

Equal Variance 

1 

0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 

Population Response Rate, Expressed As 

Fraction of Population Affected, R 

FIGURE C.5 Derived Risk Distributions Based on the Best-Fitting and Equal-Variance 

Normal-On-Log-Odds Distributions for Subject l's Judgments, Showing the Cumulative 

ProbabUity F That Fraction R of the U.S. ChUdren Aged 0-3 Have Hemoglobin Levels 

Less Than or Equal to 11 g/dl 
P.89 

83 

1.0 

0.8 

X. 

ra 

x. 

o 

u 

CL 

01 

> 

E 

3 

u 

0.6 

0.4 

0.2 

0.0 

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 

Population Response Rate, Expressed As 

Fraction of Population Affected. R 

FIGURE C.6 Derived Risk Distributions Based on the Equal-Variance and Regressed 

Best-Fitting Normal-On-Log-Odds Distributions for Subject 2's Judgments, Showing 

the Cumulative ProbabUity F That Fraction R of the U.S. ChUdren Aged 0-3 

Have Hemoglobin Levels Less Than or Equal to 11 g/dl 
84 

8 

CA 

c 

01 

Q 

J2 

ra 

o 

U 

PL. 

Subject 1 

0.1 0.2 0.3 0.4 0.5 0.6 0.7 

Population Response Rate, Expressed As 

Fraction of Population Affected, R 

FIGURE C.7 ProbabUity Density Functions for Derived Risk Distributions Based on 

the Equal-Variance Normal-On-Log-Odds Distribution Applied to the Judgments 

of Subjects 1 and 2