# OAR Box 1112 Prepped by Ryan Dugan



### Document

Number: 

# 98 ) II-B-2 J Docket Number: 

# A-83-2 2

Printed 4/16/2008 10:34:18 AM Sheet 98 of 98 p. 1P. 2 

### A -<f-7 - 2 1

OUTLINE PROTOCOL FORENCODING PROBABILIT Y JUDGMENT S ABOU T OUTCOMES OF EXPERIMENT S TO ASSES S LEAD-INDUCE D I

Q DECREMENT S
1 . INTRODUCTIO N 1 2 . LEAD-INDUCE D I Q DECREMENT S 3 2. 1 Centra l Nervou s Syste man dBehaviora l Effect s of Lea d3 2. 2 Definitio n of

I Q Decremen t

a s an Advers e Effec t

4 2. 3 Hypothetica l Idea l Experiment s 4 3 . POPULATIO N A

T RIS K 6 4 . EXPOSUR E CONDITION S 7 5 . PHYSIOLOGICA LAN D ENVIRONMENTALCONDITION S 7 5. 1 Physiologica l Condition s 7 5. 2 Environmenta l Condition s 7 6 . FACTOR S T O CONSIDE R8 Y FACTOR S T O KEE P I N MIN D WHE N MAKIN

GPROBABILIT Y JUDGMENT S 9 7. 1 Sequentia l Consideratio n of Informatio n 9 7. 2 Effect s of

Memor y o n Judgmen t

1 0 7. 3 Estimatin g Reliabilit y of Informatio n 1 0 7. 4 Relatio n Betwee n Even t Importanc e an dProbabilit y 1 1 7. 5 Estimatio n of

Probabilitie s 1 1 7. 6 Recommendation s 1 2 8 . FINA LPREPARATIO N FO RELICITATIO N O FPROBABILIT Y JUDGMENT S 1 4 (T o b e provide ddurin g initia l visit. )

Versio n 85.3.2 9P. 3

PROTOCOL FORENCODING PROBABILIT Y JUDGMENT S ABOUT OUTCOMES OF EXPERIMENT S TO ASSES S LEAD-INDUCE D I Q DECREMENT S 1 . INTRODUCTION Th e Environmenta l Protectio n Agenc y (EPA ) i s charge db y th e Clea n Ai r

Ac t wit hsettin g an drevisin g Nationa l Ambien t Ai r Qualit y Standard s (NAAQS ) fo r selecte d pollutants , a t

level s sufficien t

t o protec t

th e publi c healt hwit han adequat e margi n of safety . A s you know , th e scientifi c base s fo r NAAQ S ar e presente dan dreviewe din * criteri adocuments . I

n suppor t

of th e forthcomin g revie w of th e lea dNAAQS , EP A ha s jus t prepare dane w Ai r

Quality Criteria for

Lead. I t

present s scientifi c evidenc e fro m whic hth e mos t

susceptibl e population s ca n b e determine dan dfro mwhic hvariou s advers e healt heffect s ca n b e identified . Th e criteri adocumen t summarize s an d evaluate s th e availabl e clinical , epidemiological , an danima l o r toxicologica l laborator y evidenc e wit hregar dt o th e physiologica l an dadvers e healt heffect s o f lead , an d therefor e represent s ou r mos t up t o dat e knowledg e on lea deffects . A s on e aspec t

of th e revie w process , EP A assesse s healt hrisk s by identifyin g th e mos t sensitiv e population s fo r eac hpollutant , an destimatin g probabilisticall y th e number s of

peopl e in th e population s wh o may suffe r eac hof

variou s well-define d advers e healt heffect s attributabl e t o th e pollutant . I t i s believe dtha t

informatio n abou t th e healt hrisk s associate dwit hvariou s potentia l standard s wil l ai dth e EP A administrato r

i n selectin g tha t

standar dwhich , i n hi s judgment , protect s th e publi c healt hwit ha n adequat e margi n o f safety . Becaus e th e ris k

estimate s tha t

EP A seek s ar e ofte n base di n par t o n dose

respons e relationship s an dlea dexposur e estimate s whic har e uncertain , i t i s necessar y t o mak e probabilit y judgment s abou t relevan t

dose-respons e function s base do n th e availabl e evidence , an dt o probabilisticall y estimat e lea dexposur e unde r alternativ e NAAQS . Obtainin g th e healt hris k estimate s the n involve s combinin g dose-respons e an d exposur e probabilit y estimates .P. 4

Th e proble mo

f estimatin g dose-respons e relationship s i s simila r t o tha t whic h exist s i n clinica l medicin e whe n ther e d o no t exis t dat atha t

bea r precisel y o n th e patient' s problem . I

n tha t cas e i t i s necessar y t o us e scientifi c judgmen t t o extrapolat e fro mth e dat at o mak e abes t decisio n fo r

th e patient . Here , too , i t i s necessar y t o us e scientifi c judgmen t

t o extrapolat e fro mth e availabl e data . Th e extrapolatio n i s no t certai n and , therefore , w e wil l ai dyou t o represen t you r opinio n probabilistically . Furthermore , sinc e th e extrapolatio n depend s on one' s interpretatio n of th e literature , differen t peopl e wil l hav e differen t judgments . Fo r

eac hhealt heffec t

w e inten dt o obtai n th e probabilisti c judgment s o f abou t fiv e expert s t o sampl e th e rang e of respecte d opinions . Th e model fo r estimatin g risk s wil l no t merg e thes e judgment s int o asingl e averag e judgment , bu t rathe r wil l estimat e th e rang e o f risk s base do n th e rang e of judgments . I

f w e a s ris k

analyst s d o ou r

jo b properly , the n no t onl y wil l w e b e abl e t o sho w th e EPA administrato r th e rang e o

f estimate s base don th e rang e o f judgments , bu t w e wil l als o b e abl e t o sho w som e of th e source s o

f th e disagreements . Indeed , asid e benefi t of thi s exercis e i

n whic hw e prob e you r

knowledg e i

n astructure dmanne r

may b e t o hel p identif

y source s of

greates t disagreement . Base don th e evidenc e i n th e lea dcriteri adocument , tw o population s hav e bee n identifie da s bein g mos t susceptibl e t o th e effect s of lea dintoxication . On e i s childre n fro mbirt hthroug hth e sevent hbirthday , an dth e othe r i s pregnan t

women , o r

mor e precisely , th e fetuse s bein g carrie db y th e pregnan t women . A numbe r

of advers e healt h effect s hav e bee n identifie dfo r whic hw e woul dlik e t o estimat e dose-respons e function s includin g th e on e discusse dbelow . 2 . LEAD-INDUCE D I

Q DECREMENT S 2. 1 Centra l Nervou s Syste m and Behaviora l Effect s o f Lea d A larg e numbe r

o f studie s reviewe din th e Criteri aDocumen t sugges t tha t ther e ar e numerou s centra l nervou s syste m(CNS ) an dbehaviora l effect s of lea dexposure .P. 5

Considerabl e uncertaint y surround s al l of thes e effects , however , becaus e of th e enormou s difficult y i

n definin g an dmeasurin g them , an din isolatin g the mfro mth e effect s o f covariates . Sinc e ou r goa l i s t o obtai n probabilisti c estimate s abou t th e shap e of adose-respons e curv e fo r aparticula r

well-define deffec t

i f asufficien t

amoun t o f pertinen t dat acoul db e collected , ou r firs t tas k i s t o selec t

on e CN S o r

behaviora l effec t tha t i s of acknowledge dimportanc e an dtha t ca n b e wel l specified . Th e searc hfo r

lead-induce deffect s ha s include dstudie s of : EEGeffects; , sensor y motor , perceptual , an dattentiona l deficits ; cognitiv e decrement s of variou s sorts ; hyperactivity ; negativ e classroo mbehaviors ; an dothe r effect s tha t

s o fa r hav e bee n studie donl y i n anima l models . Fro mthi s larg e assortmen t of effects , w e hav e selecte dIQ decremen t

a s th e advers e effec t

upo n whic ht o focus . W e ar e no t considerin g IQ t o b e th e onl y no r

necessaril y th e bes t

measur e o

f cognitiv e abilities . No r i s i t bein g considere da s asurrogat e fo r th e othe r

system s i

n whic heffect s hav e bee n explored . (Becaus e o

f it s multifacete dnatur e i t probabl y involve s man y o f them. ) Rather , w e hav e selecte dIQ decremen t

becaus e ther e ar e mor e dat ao n thi s effec t

tha n o n an y other , an d becaus e it s functiona l o r "clinical " significanc e i s clear . I t i s quit e conceivabl e tha t a s researc hcontinue s i

n futur e years , othe r

mor e "pure " CN S o r

behaviora l effect s wil l emerg e tha t

ar e obviousl y advers e an dfo r whic hasubstantia l bod y of dat aha s developed . Unti l suc ha n even t occurs , i t

appear s tha t

lead-induce dI Q decremen t i s th e mos t appropriat e effec t

t o consider . 2. 2 Definitio n o f

I Q Decremen t Becaus e i t

i s generall y though t

tha t CN S effect s of

lea dmay b e cumulative , i t i s necessar y t o specif

y th e children' s age s a t whic hth e I Q decrement s ar e estimated . I n al l cases , assum e tha t

I Q test s ar e give n o n th e children' s sevent hbirthdays . Assum e furthe r tha t th e WISC- Ri s th e tes t employed .P. 6

A lead-induce dI Q chang e canno t b e measure ddirectl y fo r agive n child , an d therefor e th e chang e woul dhav e t o b e estimate dstatisticall y fro msuitabl e data . Thus , rathe r tha n as k

you directl y abou t

dose-respons e functions , w e wil l encod e you r judgment s abou t

th e outcome s of

hypothetica l idea l experiments . I f you agre e wit hth e assumption s on whic hth e hypothetica l experiment s ar e based , the n you r judgment s abou t th e potentia l outcome s wil l lea dnaturall y t o probabilisti c estimate s abou t dose-respons e function s fo r

IQ decrements . Ther e ar e difference s of opinion , of

course , a s t o wha t

IQ decrement s shoul db e considere dadverse . Th e Clea n Ai r Ac t make s i t

clea r tha t EPA shoul dse t

standard s t o protec t agains t advers e healt heffects , bu t th e leve l define da s advers e ma y b e differen t fo r regulator y tha n fo r

clinica l o r

remedia l purposes . W e wil l no t

focu s o n particula r magnitude s of

I Q decrements , bu t

rathe r wil l encod e you r

probabilisti c judgment s abou t I Q distribution s give n variou s blood-lea dlevels . The n w e wil l us e you r judgment s t o deriv e probabilisti c statement s abou t th e percen t

of childre n a t

eac hlea dleve l whos e I Q score s ar e belo w an y valu e o f interest , suc has , fo r example , belo w IQ s o f 70 , 80 , 90 , o r 100 . 2. 3 Hypothetica l Ideal Experiment s Assum e tha t

a t conception , subject s ar e randoml y assigne dt o group s differentiate db y bloo dlea dleve l targete dfo r th e thir dbirthday . Member s of eac h treatmen t

grou p ar e expose dt o lea dfro mconceptio n unti l thei r sevent hbirthdays , whil e member s o f

acontro l grou p ar e sheltere dfro mlea dfro mconceptio n unti l thei r sevent h birthdays . Th e leve l of environmenta l lea dassigne dt o achil di s roughl y constan t fo r th e seve n years , however , eac hchild' s lea duptak e i s not constant , du e t o th e change s wit h ag e o f hi s o r he r physiolog y an dbehavior . Nevertheless , th e experimenta l condition s ar e suc htha t a t thei r third birthdays , al l th e childre n i n eac hgrou p hav e essentiall y th e sam e measure dbloo dlead . Thus , group s diffe r i n term s of

mea n bloo dlea don th e thir dP. 7

birthday . Environmenta l lea dlevel s necessar y t o yiel dagive n blood-lea dleve l a t ag e thre e i n aparticula r

chil dremai n constan t

throug hth e sevent hbirthday . Th e experimenta l manipulatio n affect s onl y lea dexposure , an dn o othe r aspec t of th e children' s lives . The n th e WISC- RI Q tes t i s administere dt o al l childre n o n thei r seventh birthdays . Th e childre n in eac hgrou p hav e adistributio n o f I Q score s wit hsom e mea n an dstandar ddeviation . W e ar e intereste din you r probabilisti c judgment s abou t th e I Q distribution s fo r group s of

childre n o n thei r

sevent hbirthdays , al l of who mha d specifi c blood-lea dlevel s o n thei r thir dbirthdays . Not e thre e feature s abou t thi s hypothetica l experiment . First , i t i s simila r

t o a longitudina l study ; childre n ar e in agrou p fro mconceptio n unti l thei r sevent hbirthdays . Second , i t

involve s rando massignmen t of childre n t o groups , makin g i t unnecessar y t o worr y abou t

covariates . Third , bloo dlea di s measure da t ag e three, an dI Q a t

ag e seven. I f yo u believ e I Q effect s o f lea dt o b e differen t

fo r lowe r SES tha n fo r

middl e an duppe r

SES subpopulation s o f childre n unde r

thes e experimenta l conditions , the n w e wil l conside r

separat e hypothetica l experiment s fo r th e tw o subpopulations . Otherwise , w e wil l conside r

onl y asingl e suc hexperiment . SE S i s acomple x variable . Fo r simplicity , i

f w e divid e th e populatio n int o lo w SE S an dmiddl e o r

uppe r SE S categories , le t

u s defin e lo w SE S a s thos e childre n comin g fro mhousehold s wit hincome s i n th e lowes t

1 5 percent . Unles s yo u disagree , w e wil l als o assum e tha t th e distributio n of

I Q score s i s norma l withi n eac hgroup . (Th e WISC- Rwa s develope dt o yiel dnormall y distribute dI Q score s fo r

th e genera l population , wit hmea n I Q equa l t o 10 0 an dastandar ddeviatio n of 15. ) Furthermore , i f w e conside r

onl y asingl e hypothetica l experiment , samplin g fro mth e ful l populatio n of children , then , unles s yo u disagree , w e wil l assum e tha t th e I Q standar ddeviatio n i s 1 5 a t eac hblood-lea dleve l an dw e wil l onl y hav e t o encod e you r probabilisti c judgmen t

abou t th e mea n I Q a t

ag e seve n fo r

eac ho f severa l blood-lea d level s a t ag e three .P- 8 ,

I f w e conside r

tw o hypothetica l experiments , eac hsamplin g fro madifferen t SE S subpopulation , the n th e I Q standar ddeviatio n wil l possibl y b e les s tha n 15 . Assumin g tha t I Q standar ddeviatio n i s th e sam e a t al l blood-lea dlevel s withi n SE S subpopulations , w e wil l hav e t o encod e you r probabilisti c judgmen t abou t th e standar ddeviatio n fo r eac h SE S group . Then , separatel y fo r

eac hSES grou p o r

fo r th e populatio n a s awhole , a s appropriate , w e wil l encod e you r probabilisti c judgmen t abou t th e mea n I Q a t

ag e seven , of th e contro l group . Finally , w e wil l encod e you r probabilisti c judgment s abou t th e difference s in mea n I Q a t ag e seve n betwee n childre n wit hnegligibl e bloo dlea d(th e contro l group ) an dchildre n a t eac hof severa l elevate dbloo dlead s a t ag e thre e (th e treatmen t

groups) . Th e followin g section s specif y furthe r

th e condition s t o b e assume di n thi s hypothetica l experiment . 3 . POPULATIO N A T RIS K Base do n th e Criteri aDocument , w e ca n specif y th e mos t

susceptibl e populatio n a s al l Unite dState s childre n fro mconceptio n throug hthei r sevent hbirthdays . W e hav e alread y define dth e effec t a s bein g measure don th e sevent hbirthday , bu t o n th e assump tio n tha t younge r childre n wil l reac hthei r sevent hbirthdays , th e effec t i s well-define d fo r th e whol e population . However , a s alread y discussed , yo u may conside r I Q effect s of

lea dt o b e dif

feren t fo r lowe r SE S tha n fo r

middl e an duppe r

SE S subpopulation s of children , resultin g in differen t

experimenta l outcome s fo r th e tw o groups . I

f so , the n w e wil l elici t you r judgment s separatel y fo r lowe r SE S childre n an dfo r

middl e an duppe r

SE S children .P. 9

4 . EXPOSUR

E CONDITIONS W e wil l b e askin g you r

judgmen t

abou t mea n I Q value s o n th e sevent hbirthday s a t variou s blood-lea dlevels . Assum e tha t th e blood-lea dlevel s unde r consideratio n fo r

a give n judgmen t

hav e bee n measure do n th e children' s thir dbirthdays . Assum e furthe r tha t externa l environmenta l condition s supportin g thos e level s hav e bee n mor e o r les s constan t sinc e birth , an dtha t

in interactin g wit hth e environment , th e childre n exhibite d th e usua l rang e of

behavior s a t

eac hage . Thus , blood-lea dlevel s wer e no t necessarily , constan t fro mbirt ht o ag e seven , bu t exposur e an dbehaviora l factor s wer e suc htha t a t ag e three , bloo dlea dwa s a t aspecifi c level . Finally , assum e tha t th e change s i n bloo d lea dlevel s fro mbirt hunti l th e sevent hbirthda y ar e distribute da s you believ e the y actuall y are . 5 . PHYSIOLOGICAL AND ENVIRONMENTA L CONDITIONS Becaus e th e effect s of lea ddepen do n man y parameters , i t i s necessar y t o specif y assumption s abou t

thos e parameter s in th e population(s) . 5. 1 Physiologica l Condition s Th e effec t

of lea di n th e syste mdepend s o n th e person' s nutritiona l an dmeta boli c status . Assum e tha t th e level s of

iron , zinc , copper , vitamin s A , C , D , an dE , calcium , phosphorus , an dmagnesiu mar e distribute dwithi n th e populatio n o r withi n eac h o f th e tw o SE S group s a s you believ e the y in fac t are , takin g int o accoun t th e wid e rang e o f diet s an dnutritiona l level s of childre n withi n th e populatio n o r withi n eac hof

th e tw o SE S groups . 5. 2 Environmenta l Condition s Th e effec t

of lea do n I Q als o may depen don numerou s environmenta l an dcare givin g factor s tha t

var y withi n SE S level , suc ha s thos e assesse di n th e HOM E scal eP.1 0

(Hom e Observatio n fo r Measuremen t

o f th e Environment) , parenta l IQ , an ds o forth . Assum e tha t

thes e factor s ar e distribute dwithi n th e populatio n o r withi n eac ho

f th e tw o SE S group s a s yo u believ e the y i n fac t are . 6 . FACTORS TO CONSIDE R I n orde r t o hel p you brin g t o min dth e relevan t evidenc e s o tha t

you may conside r i t systematically , an dals o i n orde r t o hel p u s t o interpre t you r judgments , w e woul dlike , t o as k you t o discus s briefl y you r interpretation s of

variou s aspect s of

th e literature . How d o yo u evaluat e th e researc hconcernin g th e effect s of lo w leve l lea dexposur e o n cognitiv e an dbehaviora l development ? I s i t you r feelin g tha t

lea dha s adeleteriou s effec t on thi s developmen t

ove r an dabov e tha t

whic hca n b e explaine db y othe r factor s frequentl y associate dwit hlea dexposure ? I f so , d o you believ e tha t th e effect s of

lea d simpl y ad dt o th e effect s of othe r factors , o r

tha t th e effect s of

lea ddepen do n th e level s o f othe r factors ? I

n th e latte r

case , wha t

ar e th e variable s tha t lea dexposur e interact s with ? A relate dquestio n concern s you r opinio n abou t

th e relationshi p betwee n exposur e an dsusceptibility . Tha t is , d o yo u thin k

tha t thos e childre n wh o are . a t greates t ris k of exposur e du e t o thei r livin g in deterioratin g pre-195 0 housin g o r i n urba n area s wit hlarg e amount s of

vehicula r

traffi c o r

du e t o othe r

reasons , ar e als o mos t susceptibl e t o th e effect s of

lead ? Fo r

example , thes e ma y b e th e sam e childre n wh o als o hav e poore r diets , les s acces s t o medica l care , poore r care-givin g environments , an dles s intellectua l resource s t o fal l bac k on . What i s you r opinio n abou t

th e tim e cours e o

f lea dexposur e an do

f lea deffect s o n CNS , cognitiv e an dbehaviora l development ? Wha t ar e th e relativ e effect s of cumula

tiv e versu s curren t

exposure . Ar e th e effect s reversible ? I s ther e athreshol dfo r th e effects ? What

ar e th e implication s of th e anima l model research , bot hth e behaviora l an dth e morphological , fo r huma n CNS , cognitive , an dbehaviora l effect s of lead ? Ar e ther e othe r

factor s t o conside r

i n thinkin g abou t

th e dose-respons e function s fo r lead

induce dIQ effect s tha t w e shoul ddiscus s now ?P.1 1

7 . FACTORS TO KEE

PI N MIN D WHE N MAKIN

G PROBABILIT

Y JUDGMENT S Ther e i s usuall y uncertaint y associate dwit hconclusion s tha t w e dra w fro m research , an dmor e generall y in ou r everyda y thinking . However , no t everyon e i s awar e o f al l th e source s tha t shoul dcontribut e t o thei r uncertainty , no r

ar e mos t

peopl e familia r wit hth e proces s o f

actuall y expressin g thei r uncertaint y i n probabilisti c terms . Whe n a n exper t i s aske dt o mak e probabilit y judgment s on sociall y importan t matters , i t i s particularl y importan t tha t h e o r sh e conside r

th e relevan t

evidenc e i

n asystematic

an deffectiv e manne r

an dprovid e judgment s tha t represen t

hi s o r he r opinion s well . Experimenta l psychologist s an ddecisio n analyst s hav e amasse daconsiderabl e amoun t of dat aconcernin g th e way peopl e forman dexpres s probabilisti c judgments . Th e evidenc e suggest s tha t whe n considerin g larg e amount s of comple x information , mos t peopl e emplo y simplifyin g heuristic s an ddemonstrat e certai n systemati c distortion s of thought , i.e. , cognitiv e biases , whic hadversel y affec t thei r judgments . Th e purpos e o f thi s sectio n i s t o mak e you awar e of thes e biase s an dheuristic s s o that , a s muc ha s possible , yo u ca n avoi dthe mi n makin g probabilit y judgments . W e wil l firs t

revie w th e mos t widesprea dbiase s an dheuristics , an dthe n offe r som e suggestion s t o hel p yo u mitigat e thei r

effects . 7. 1 Sequentia l Consideratio n o f Informatio n Generally , th e orde r

i n whic hevidenc e i s considere dinfluence s th e fina l judgment , althoug hlogicall y tha t shoul dno t

b e th e case . O

f necessity , piece s of informatio n ar e considere don e b y on e i n asequentia l fashion . However , thos e considere dfirs t

an dlas t

ten dt o dominat e judgment . I n part , initia l informatio n ha s it s undu e influenc e becaus e i t provide s th e framewor k tha t subsequen t

informatio n i s the n tailore dt o fit . Fo r

example , peopl e usuall y searc hfo r evidenc e t o confirm thei r initia l hypotheses ; the y rarel y loo k fo r evidenc e tha t

weighs against

them . Th e late r

evidenc e ha s it s undu e effec t

simpl y becaus e i t

i s freshe r

in memory .P.1 2 1 0 Relate dt o thes e sequentia l effect s i s th e phenomeno n of anchoring and adjustment. Base don earl y partia l information , on e form s a n initia l probabilit y estimat e regardin g th e even t

in question . Thi s ancho r judgmen t

i s the n adjuste da s subsequen t informatio n i s considered . Unfortunately , suc hadjustment s ten dt o b e to o conservative . I

n othe r words , to o littl e weigh t i s attache dt o informatio n considere d subsequen t

t o th e formatio n of

th e initia l judgment . 7. 2 Effect s o f

Memor y o n Judgmen t I t i s difficul t

fo r mos t peopl e t o conceptualiz e an dmak e judgment s abou t large , abstrac t universe s o r

populations . A natura l tendenc y i s t o recal l specifi c member s an d the n t o conside r

the mrepresentativ e of th e populatio n a s awhole . However , th e specifi c instance s ofte n ar e recalle dprecisel y becaus e the y stan dou t i n som e way , suc ha s bein g familiar , unusual , especiall y concrete , o r o f persona l significance . Unfortunately , th e specifi c characteristic s o f thes e singula r

example s ar e the n attributed , ofte n incorrectly , t o al l th e member s of

th e populatio n of

interest . Moreover , thes e memor y effect s ar e ofte n combine dwit hth e sequentia l phenomen adiscusse dearlier . Fo r example , in considerin g th e evidenc e regardin g th e dose-respons e curv e of aparticula r

pollutant , yo u migh t naturall y firs t

thin k of astud y yo u o r

apersona l frien drecentl y completed . O r yo u migh t

thin k of astud y yo u recentl y read , o r on e tha t

wa s unusua l an dtherefor e stand s out . Th e tendenc y migh t the n b e t o trea t

th e recalle dstudie s a s typica l o f th e populatio n of

relevan t

research , ignorin g importan t difference s amon g studies . Subsequen t

attempt s t o recal l informatio n coul dresul t in thinkin g primaril y of evidenc e consisten t

wit hth e initia l item s yo u though t of. 7. 3 Estimatin g Reliabilit y o f Informatio n Peopl e ten dt o overestimat e th e reliabilit y of information , ignorin g factor s suc h a s samplin g erro r

an dimprecisio n of

measurement . Rathe r the y summariz e evidenc e i nP.1 3 1 1

term s o f simpl e an ddefinit e conclusions , causin g the mt o b e overconfiden t in thei r judgments . Thi s tendenc y i s stronge r whe n on e ha s aconsiderabl e amoun t o f intellectua l and/o r persona l involvemen t i n aparticula r

field . I

n suc hcases , informatio n i s ofte n interprete di

n away whic hi s consisten t wit hone' s belief s an dexpectations , result s ar e over-generalized , an dcontradictor y evidenc e i s ignore do r underestimated . 7. 4 Relatio n Betwee n Even tImportanc e and Probabilit y Sometime s th e importanc e o f events , o r

thei r possibl e cost s o r

benefits , influenc e judgment s abou t th e certaint y o

f th e event s when , rationally , importanc e shoul dnot

affec t probability . I

n othe r words , one' s attitude s toward s ris k ten dt o affec t one' s abilit y t o mak e accurat e probabilit y judgments . Fo r example , man y physician s ten d t o overestimat e th e probabilit y of ver y sever e diseases , becaus e the y fee l i t i s importan t t o detec t an dtrea t

them ; an dsimilarly , man y smoker s underestimat e th e probabilit y of advers e consequence s of smoking , becaus e the y fee l tha t th e odd s d o no t

appl y t o themselve s personally . 7. 5 Estimatio n o f

Probabilitie s Anothe r limitatio n i s relate dt o one' s abilit y t o discriminat e betwee n level s of uncertainty , an dt o us e appropriat e criteri ao f discriminatio n fo r

differen t

range s of probability . On e resul t of thi s fac t

i s tha t peopl e ten dt o estimat e bot hextrem e an d mid-rang e probabilitie s i n th e sam e fashion , usuall y doin g apoo r jo b in th e extremes . I t help s her e t o thin k

in term s o

f odd s a s wel l a s probabilities . Thus , fo r example , changin g aprobabilit y estimat e fro m0.51 0 t o 0.50 1 i s equivalen t t o achang e in odd s fro m1.041: 1 t o 1.004:1 , bu t

achang e fro man estimat e of 0.99 9 t o 0.99 0 change s th e odd s b y afacto r of abou t 1 0 fro m999: 1 t o 99: 1 Th e close r t o th e extreme s (eithe r 0 o r 1 ) tha t on e i s estimatin g probabilities , th e greate r th e impac t

of smal l changes .P.1 4 1 2
7. 6 Recommendation s Althoug hextensiv e an dcarefu l trainin g woul db e necessar y t o eliminat e al l th e problem s mentione dabove , som e relativel y simpl e suggestion s ca n hel p minimiz e them . Mos t importan t

i s t o b e awar e of

one' s natura l cognitiv e biases , an dt o tr y consciousl y t o avoi dthem . T o avoi dsequential

effects kee p i n min dtha t th e orde r

in whic hyo u thin k

of informatio n shoul dno t

influenc e you r

fina l judgment . I t

ma y b e helpfu l t o actuall y not e o n pape r th e importan t

fact s you ar e considerin g an dthe n t o reconside r the min tw o o r mor e sequences , checkin g th e consistenc y of you r judgments . Try t o kee p a n ope n min d unti l you hav e gon e throug hal l th e evidence , an ddon' t le t th e earl y informatio n yo u conside r swa y you mor e tha n i s appropriate . To avoi dadvers e memory effects, defin e variou s classe s o f informatio n tha t

yo u dee mrelevant , an dthe n searc hyou r memor y fo r

example s of

each . Don' t

restric t you r thinkin g onl y t o item s tha t stan dou t

fo r specifi c reasons . Make aspecia l attemp t t o conside r conflictin g evidence , an dt o thin k o f dat atha t

ma y b e inconsisten t

wit ha particula r

theory . Also , b e carefu l t o concentrat e o n th e give n probabilit y judgmen t an d do no t le t you r ow n value s (ho w yo u woul dmak e th e decisio n yourself ) affec t thos e judgments . T o accuratel y estimate the reliability of information, pa y attentio n t o suc h matter s a s sampl e siz e an dpowe r of th e statistica l tests . Kee p i n min dtha t dat aar e probabilisti c i

n nature , subjec t

t o element s of

rando merror , imprecis e measurements , an dsubjectiv e evaluatio n an dinterpretation . I n addition , th e furthe r on e mus t extrapolate , o r

generalize , fro maparticula r stud y t o asituatio n of interest , th e les s reliabl e i s th e conclusio n an dth e les s certaint y shoul db e attribute dt o it . Rel y mor e heavil y on informatio n whic hyou conside r mor e reliable , bu t

d o no t trea t i t a s "absolut e truth. "P.1 5 1 3
Kee p in min dtha t

the importance of

an event or an outcome should

not influence its judged

probability. I t

i s rationa l t o le t

th e costlines s o r

severit y o f

an outcom e influenc e th e poin t

a t whic hactio n i s take n wit hrespec t t o it , bu t

no t th e judgmen t

tha t i s mad e abou t

th e outcome' s likelihood . Finally , in makin g probabilit y judgments , thin k primaril y i n term s of

th e measur e (probabilit y o r

odds ) wit hwhic hyo u fee l mor e comfortable , bu t sometime s translat e t o th e alternativ e scale , o r eve n t o measure s of

othe r event s (e.g. , th e probabilit y of th e even t no t happening) . Whe

n estimatin g ver y smal l o r ver y larg e likelihoods , i t i s usuall y bes t t o thin k i n term s of

odds , whic har e unbounded , instea dof probabilities , whic har e bounded . Fo r

example , on e ca n mor e easil y conceptualiz e odd s of 1:20 0 tha n a probabilit y of

.005 .P.1 6 1 4 8 . FINAL PREPARATIO N FO RELICITATIO

N OF PROBABILIT

Y JUDGMENT S Th e outcome s o

f th e idea l experiment s describe dabov e ar e uncertain . Existin g dat aar e relevant , bu t

d o no t allo w exac t

predictions . Our

goa l i s t o hav e yo u represen t probabilisticall y you r

ow n uncertaint y abou t th e experimenta l outcome s base don you r expertis e an dth e availabl e knowledge . I n respondin g t o th e question s w e wil l as k you , pleas e thin k

carefull y abou t

th e relevan t

informatio n reviewe di n th e criteri adocument , an dconsul t

i t a s wel l a s th e literatur e o r you r file s a s yo u dee mappropriate . Th e previou s sectio n suggest s ways t o thin k abou t th e relevan t

data . Th e purpos e o f tha t sectio n i s t o hel p minimiz e th e biasin g effect s tha t frequentl y accompan y th e informatio n overloa dnaturall y resultin g fro mrapi dconsideratio n of larg e amount s o

f comple x evidence . Yo u may fin di t

helpfu l t o revie w th e sectio n o r rais e question s abou t th e point s mad e i

n i t befor e w e begin . Uncertaint y abou t

th e effect s of

lea dexposur e on mea n decremen t i n IQ (relativ e t o mea n IQ a t zer o bloo dlead ) ca n b e represente dprobabilisticall y i n tw o differen t ways . 1 . Uncertaint y abou t

th e mea n I Q decremen t tha t woul dresul t

whe n th e expose dgrou p ha s agiven blood-lead concentration. 2 . Uncertaint y abou t th e blood-lea dconcentratio n tha t woul db e require dt o cause a given mean IQ decrement . W e wil l concentrat e o n on e way a t atime , focusin g primaril y o n th e firs t one . Fo r thes e purpose s recal l tha t everyon e i

n th e expose dpopulatio n ha s aspecifie d blood-lea dleve l whil e everyon e in th e zer o blood-lea dpopulatio n ha s negligibl e bloo d lead , a s describe dabov e unde r Exposur e Conditions . Then , i n orde r fo r u s t o determin e you r uncertaint y abou t

th e mea n I Q decremen t o f th e expose dgrou p wit hagiven bloodlead level relativ e t o th e negligibl e blood-lea dgroup , w e mus t introduc e adefinition . Le t Lb e th e give n blood-lea dconcentratio n in question .P.1 7 1 5 Definition: D(L) i s th e mea n IQ decremen t of th e expose dgrou p wit h blood-lea dleve l Lrelativ e t o th e negligibl e group . Th e valu e of

D(L) fo r

agive n Li s uncertain , an dw e woul dlik e t o obtai n probabilit y judgment s fro myou abou t it s possibl e values . W e wil l elici t

you r judgment s abou t th e possibl e value s of D(L ) by specifyin g aparticula r mea n I Q decremen t

dan d havin g you conside r

ho w likel y i t

i s tha t D(L) i s les s tha n tha t

value . T o hel p yo u mak e you r probabilit y judgments , w e wil l mak e us e o f adevic e calle daprobabilit y wheel , whic hha s adjustabl e sector s o f blu e an dorange . W

e ca n rea do

n th e bac k

of th e whee l th e percentag e of

th e whee l tha t

i s eac hcolor . I

n makin g you r

judgment s yo u ar e t o imagin e tha t

th e whee l i s aperfectl y fai r rando mdevice , an dtha t therefor e th e probabilit y o

f it s stoppin g wit hth e pointe r o n blu e i s exactl y represente db y th e relativ e are atha t i s blue . W e wil l the n procee da s follows . Fo r eac hbloo dlea dconcentratio n L , w e wil l specif y aparticula r

mea n I Q decremen t

d , an dals o se t

th e probabilit y whee l t o hav e a specifi c relativ e are aof blue . The n yo u ar e t o conside r carefull y th e question: Do you conside r

i t mor e probabl e tha t

D(L) i s les s tha n do r tha t th e whee l woul dsto p wit hth e pointe r on blu e (on arando mspin) ? You ca n giv e on e o

f thre e responses : (1 ) Yo u judg e i t

t o b e mor e probabl e tha t D(L ) i s les s tha n d ; (2 ) You judg e i t

t o b e mor e probabl e tha t th e whee l woul dsto p wit h th e pointe r

o n blue ; (3 ) You canno t

judg e eithe r

even t a s mor e probabl e tha n th e other . Fo r aparticula r

concentratio n Lan dmea n I Q decremen t d , som e whee l setting s wil l hav e asmal l enoug hrelativ e are aof blu e tha t

yo u wil l fee l confiden t makin g respons e (1) . Othe r

whee l setting s wil l hav e alarg e enoug hrelativ e are aof blu e tha t

yo u wil l fee l confiden t

makin g respons e (2) . Th e intermediat e setting s wil l b e mor e difficul t t o judge . However , fo r afixe dbloo dlea dL , an dmea n IQ decremen t d , w e wil lP.1 8 1 6 manipulat e th e whee l setting s t o fin dth e on e fo r whic hyou fee l mos t comfortabl e makin g respons e (3) . Onc e w e hav e determine dth e poin t a t whic hyou ar e mos t

comfortabl e wit h respons e (3) , an dstil l focusin g on th e give n bloo dlea dL , w e wil l specif y ane w mea n IQ decremen t

d' , rese t

th e whee l t o ane w relativ e are ao f blue , an dprocee dt o obtai n you r judgment s regardin g th e probabilisti c relatio n betwee n d ' an dD(L) . Thi s wil l continu e fo r th e give n Lunti l w e hav e obtaine dcomparison s of variou s mea n value s t o D(L) . The n we * wil l hav e elicite dth e probabilisti c representatio n of you r uncertaint y abou t

th e mea n I Q decremen t

fo r on e blood-lea dlevel . W e wil l obtai n th e probabilisti c representatio n of you r uncertaint y fo r

th e outcom e of

anothe r experimen t

by specifyin g ane w L an d continuin g a s before . W e wil l d o thi s fo r

anumbe r

o f value s of

L . I t frequentl y happen s tha t an expert' s judgment s alte r somewha t

ove r th e cours e of asessio n suc ha s this , a s h e o r sh e consider s th e evidenc e fro mvariou s perspective s an dthink s abou t

th e variou s response s calle dfor . Hence , w e wil l grap hyou r response s and , a t appropriat e times , sho w the mt o you fo r you r consideratio n an dcomparison . A t thes e time s yo u may wis ht o chang e som e of th e judgment s yo u giv e earlier . Also , althoug hth e probabilit y judgment s ar e entirel y you r own , w e mus t introduc e on e logica l constraint : namely , tha t you r final judgment s ar e coheren t in a sens e tha t

w e ca n explai n a s w e g o along . I t i s quit e commo n fo r

initia l judgment s t o exhibi t som e incoherencies . Tha t i s anothe r

reaso n w e wil l conside r togethe r

th e graph s o f you r judgments . Ou r

objectiv e i s t o obtai n acoheren t se t of judgment s tha t

represen t you r opinion s wel l b y th e en do f thi s elicitatio n process . You ar e no t expecte dt o giv e u s suc hase t

immediately . Al l of

thi s wil l becom e cleare r a s w e g o along . W e shoul demphasiz e tha t th e judgment s w e ar e askin g yo u t o mak e ar e no t simpl e ones , no r

o f cours e ar e ther e know n correc t answers . Rather , w e wan t you r bes t an dmos t

considere djudgmen t i n ligh t of th e availabl e relevan t scientifi c data .P.1 9 1 7 Therefore , pleas e reflec t o n th e availabl e dat acarefully , feelin g fre e t o consul t th e lea d criteri adocumen t

o r othe r source s a s yo u wis ha s yo u formulat e you r judgments . (Encod e judgment s fo r tw o blood-lea dconcentrations , the n continu e wit h instructions. ) Recal l tha t

uncertaint y abou t

th e experimenta l outcome s ca n als o b e expresse d i n term s o

f th e blood-lea dconcentratio n necessar y t o produc e agive n mea n I Q decremen t

D , i f th e entir e expose dpopulatio n ha dth e sam e blood-lea dconcentration , establishe di n th e manne r discusse dearlier . More specifically , proceedin g a s before , le t u s introduc e adefinition : Definition: L(D ) i s th e blood-lea dleve l tha t woul dcaus e amea n IQ decremen t

D in an expose dgrou p relativ e t o th e negligibl e group . Th e valu e o

f L(D ) i s uncertai n fo r agive n D , an dw e wan t t o obtai n you r judgment s abou t

it s possibl e values . Analogousl y t o wha t w e hav e alread y don e in obtainin g th e D(L) judgments , fo r agive n D w e wil l specif y concentration s 8

, an das k yo u I t o conside r

how likel y i t

i s tha t th e L(D ) i s les s tha n th e specifie d£. . More specifically , utilizin g th e whee l a s before , w e wil l as k yo u th e question: D o you conside r

i t mor e likel y tha t

L(D ) i s les s tha n i, or

tha t th e whee l woul dsto p wit hth e pointe r o n blu e (on arando mspin) ? You ca n giv e on e of

thre e responses : (1 ) Yo u judg e i t

mor e probabl e tha t

L(D ) i s les s tha n 8, ; (2 ) You judg e i t

mor e probabl e tha t

th e whee l woul dsto p wit hth e pointe r on blue ; (3 ) Yo u canno t judg e eithe r

even t a s mor e probabl e tha n th e other . Afte r determinin g th e whee l settin g a t whic hyou fee l mos t

comfortabl e wit h respons e (3) , w e wil l specif y ane w 9. an drepea t

th e process . A s before , fo r eac hD w e wil l elici t

you r comparativ e judgment s fo r variou s value s of

a.P.2 0 1 8

m W e mus t encod e you r

probabilisti c judgmen t abou t th e mean.I Q of

childre n in th e population , o r

in th e tw o SES groups , who hav e negligibl e blood-leve l unde r th e exposur e condition s describe dpreviously . W e star t wit hadefinition : Definition: Le t

p b e th e tru e populatio n mea n I Q fo r childre n wit hnegligibl e bloo dlead . (Or , le t

y • b e th e tru e populatio n mea n I Q fo r childre n wit hnegligibl e bloo d lea din SE

S grou p i. ) Th e valu e o

f y o r u • i s uncertain , an dw e wil l obtai n you r judgmen t

abou t it s possibl e values . Analogousl y t o wha t w e hav e alread y done , w e wil l specif y amea n y an d

* **

k as k yo u t o conside r

how likel y i t

i s tha t u o r y . i s les s tha n y . W e wil l elici t

you r judgment s abou t

variou s value s of

u wit hth e ai dof

th e probabilit y wheel , i n th e sam e manne r a s wa s don e previously . I f yo u fel t

i t necessar y t o conside r separat e experiment s fo r tw o SE S groups , the n you may believ e tha t th e within-grou p I Q standar ddeviatio n i s differen t fro m15 . I f so , w e mus t

encod e you r

probabilisti c judgmen t abou t tha t value . I

f yo u agree , w e wil l assum e tha t

th e standar ddeviatio n i s unaffecte db y lead-level . However , yo u ma y fee l tha t th e standar ddeviatio n i s differen t fo r th e tw o SES groups . I f so , w e mus t

encod e you r opinio n separatel y abou t eac hcase . A s before , w e star t

wit hadefinition : Definition: Le t

oj * b e th e tru e I Q standar ddeviatio n fo r SE S grou p i . Th e valu e o

f a-*i s uncertain , an dw e wil l obtai n you r judgmen t

abou t it s possibl e values . Analogousl y t o wha t w e hav e alread y done , w e wil l specif y astandar ddeviatio n o an das k yo u t o conside r

how likel y i t

i s tha t oj * i s les s tha n o . W e wil l elici t

you r judgment s abou t

variou s value s of

o wit hth e ai dof

th e probabilit y wheel , i n th e sam e manne r a s was don e previously .