OFFICE OF CHEMICAL SAFETY

AND POLLUTION PREVENTION

UNITED STATES ENVIRONMENTAL PROTECTION AGENCY

WASHINGTON, D.C. 20460

RE: Federal Letter-Review of Chlorpyrifos Epidemiology Studies 

Summary

Between July and September 2012, EPA solicited input from several individuals in the federal 

government with expertise related to (1) the assessment of neurodevelopmental outcomes and (2) the use 

of magnetic resonance imaging (MRI) for the purpose of research in order to review Brain Anomalies in 

Children Exposed Prenatally to a Common Organophosphate Pesticide (2012) study. This 

document contains a list of the Federal reviewers, the questions EPA asked the reviewers to 

consider during their review of the Rauh et al. study, EPA’s brief synopsis of Rauh et al. 2012 

and its evaluation of the epidemiological aspects of the MRI study, the letters sent to each 

reviewer soliciting their input, and their review.

List of Documents

1. List of Reviewers Page 2

2. Questions for Reviewers Page 3

3. EPA’s Synopsis of Rauh et al. and Evaluation of Epidemiological Aspects of MRI Study Page 6

4. Dr. Basser Letter and Review Page 16

5. Dr. Liachenko Letter and Review Page 21

6. Dr. Freund Letter and Review Page 24

7. Dr. Bitsko Letter and Review Page 28

8. Dr. Chelonis Letter and Review Page 33
Federal External Review Panel Members

Peter J. Basser, Ph.D.

NICHD (National Institute of Child Health and Human Development)

TISSUE BIOPHYSICS & BIOMIMETICS

Rebecca Bitsko

CDC/ DHHS/CDC

Lisa Freund, Ph.D. (NICHD)

NICHD (National Institute of Child Health and Human Development)

CHILD DEVELOPMENT & BEHAVIOR BRANCH (HNT3D)

Serguei Liachenko, MD, PhD,

Director of Bioimaging,

National Center for Toxicological Research

Food and Drug Administration

John J. Chelonis 

Division of Neurotoxicology,

National Center for Toxicological Research

Jefferson, AR
Questions for External Peer Review of MRI Paper

1) In April 2012, the FIFRA SAP1

, an external review body designated by Congress to advise EPA 

concerning science policy issues, reviewed the epidemiology studies produced by researchers 

with the Columbia Center on Children’s Environmental Health (CCCEH) regarding the relation 

between prenatal chlorpyrifos exposure and adverse neurodevelopmental consequences 

through childhood (FIFRA Scientific Advisory Panel, 2012). In May 2012, researchers with the 

CCCEH produced the paper under evaluation in this written peer review; the paper on brain 

abnormalities using MRI technology by (Rauh et al., 20122

). Because Rauh et al (2012) was 

available after the April 2012 SAP review, this study was not considered by the Panel. For the 

current peer review, EPA is soliciting comments on the epidemiological aspects of the study 

design which are unique to Rauh et al (2012) (e.g., sample selection and inclusion/exclusion 

criteria; socioeconomic status (SES) considerations; sample size and statistical power, etc.) 

EPA’s has reviewed of the epidemiological context of the MRI study by Rauh et al (2012) (see 

attached). Please comment on the Agency’s review with respect to the epidemiological aspects 

of the study.

2) The authors used sophisticated MRI imaging studies to measure difference in 3D size of cerebral 

surface and cortical thickness by chlorpyrifos exposure group (high/low). Please comment on 

the MRI imaging procedures. In addition to other issues you identify as relevant for EPA to 

consider, please include in your comments a consideration of the following: 

a. Authors report that 20 of 28 eligible participants in the “high” exposure group were 

enrolled with usable MRIs, and 20 of 38 eligible participants in the “low” exposure group 

were enrolled with usable MRIs in the study. Is this a reasonable eligibility rate, i.e., is 

the rate of lack of usable MRI remarkable in this study?

b. Did authors process and pre-process images appropriately according to standard 

practice in research studies? Were there deviations to common practice in MRI studies, 

and if so, what are the potential impacts on the interpretation of the findings?

c. Did authors use appropriate methods to categorize “grey matter” thickness by 

gradations of color of the cerebral cortex across the MRI images? If not, how could 

methods be improved and what are the implications?

d. Authors utilized a two-step process to identify an individual “template” brain against 

which to compare other brain images by CPF exposure status. Please comment as to the 

use of an individual versus an average or amalgamated “template.” 

3) Please comment on the statistical procedures used in Rauh et al (2012). 

 1 FIFRA Scientific Advisory Panel. (2012). Draft Issue Paper Scientific Issues Concerning the Health Effects of Chlorpyrifos. 

2 Rauh, V. A., Perera, F. P., Horton, M. K., Whyatt, R. M., Bansal, R., Hao, X., . . . Peterson, B. S. (2012). Brain anomalies in 

children exposed prenatally to a common organophosphate pesticide. Proc Natl Acad Sci U S A, 109(20), 7871-7876. doi: 

1203396109 [pii] 10.1073/pnas.1203396109
4) The authors reported several findings (i.e., areas of brain enlargement, qualitative statistical 

interaction between full scale intelligence quotient (FSIQ) and brain morphology; aberrations in 

the normal sexual differentiation of the brain, and sporadic cortical thinning). Please provide 

your interpretation of the results reported by Rauh et al (2012). Based upon your interpretation 

of the study, what, if any, neurodevelopmental consequences would you anticipate in these 

children? 

5) What are the uncertainties around the MRI imaging findings in Rauh et al (2012)? Child age 

range in the MRI study was 6-11 years. Are there changes in child brain development in this 

period of rapid brain growth that could affect study results and interpretation?

6) In the developmental neurotoxicity study conducted in laboratory rats (Hoberman, 1998a, 

1998b, 1999a, 1999b3

) EPA’s Data Evaluation Review (DER) of this study, pp. 23-25 and 29 of 

DER for MRID 44556901 & pp. 3-7 and 13-15 of DER for MRID 447873014

], effects in brain 

morphometry were observed in rats exposed to chlorpyrifos during gestation. To what degree, 

if any, are the results of Rauh et al (2012) comparable with Hohmann? 

7) What additional comments would you like to provide us that may assist us in better interpreting 

the results and characterizing the uncertainty?

8) Are there any additional analyses that should be conducted? And what would be the value or 

implications of these analyses?

 3 Hoberman, A. M. (1998a) Developmental neurotoxicity study of chlorpyrifos administered orally via gavage to Crl:CD®BR 

VAF/Plus® presumed pregnant rats. Argus Research Laboratories, Inc., Horsham, Pennsylvania, Laboratory study No. 304-001, 

Sponsor study No. K-044793-109, May 1, 1998. MRID 44556901. Unpublished.

Hoberman, A.M. (1998b) Developmental neurotoxicity study of chlorpyrifos administered orally via gavage to Crl:CD®BR 

VAF/Plus® presumed pregnant rats; Report Supplement 1 (pathology data). Argus Research Laboratories, Inc., Horsham, 

Pennsylvania, Laboratory study No. 304-001, Sponsor study No. K-044793-109, September 23, 1998. MRID 44661001. 

Unpublished.

Hoberman, A.M. (1999a) Developmental neurotoxicity study of chlorpyrifos administered orally via gavage to Crl:CD®BR 

VAF/Plus® presumed pregnant rats; SAS Statistical Output. Argus Research Laboratories, Inc., Horsham, Pennsylvania, 

Laboratory study No. 304-001, Sponsor study No. K-044793-109, Undated. MRID not assigned. Unpublished.

Hoberman, A.M. (1999b) Developmental neurotoxicity study of chlorpyrifos administered orally via gavage to Crl:CD®BR 

VAF/Plus® presumed pregnant rats; Report Supplement 2 - Reanalysis of Morphometric Data. Argus Research Laboratories, 

Inc., Horsham, Pennsylvania, Laboratory study No. 304-001, Sponsor study No. K-044793-109, March 19, 1999. MRID 

44787301. Unpublished.

4

 Makris, S. Chlorpyrifos Toxicology Data Review. 

1

Summary and Critical Review of Rauh et al. (2012) “Brain anomalies in children exposed prenatally to 

a common organophosphate pesticide”

This document includes a brief summary of the research paper “Brain anomalies in children 

exposed prenatally to a common organophosphate pesticide” published by Rauh et al. (2012) as 

well as EPA’s critical analysis of the epidemiologic context of this study. The study utilizes 

magnetic resonance imaging (MRI) technology to identify and contrast anatomical 

characteristics of the cortical region of the brain including cortical surface area and thickness in 

relation to prenatal pesticide exposure. EPA is seeking additional expert input as to the quality 

of the MRI data collection procedures used in this study as well as the appropriateness of the 

authors’ interpretation of the MRI studies in association with the environmental exposure of 

interest. EPA’s critical analysis follows the brief synopsis of the study below. 

Study Summary

In this study, authors with the Columbia Center for Children’s Environmental Health (CCCEH) birth 

cohort study evaluated whether there were areas of morphological change in the pediatric brain in 

regions of the brain known to be associated with learning, cognition and social behavior in association 

with prenatal chlorpyrifos (CPF) exposure (V. A. Rauh et al., 2012). These are functional 

neurodevelopmental outcomes that have been adversely associated with chlorpyrifos exposure in 

previous observational research (V. A. Rauh et al., 2006; V. Rauh et al., 2011), as well as some 

experimental studies, as noted by the authors. Specifically, in this study, authors evaluated 1) whether 

morphological changes in the brain including cortical thickness and differences in the cerebral brain 

surface area and volume were related to prenatal chlorpyrifos exposure (the main effect), and, 2) 

whether the relation between the exposure and morphological changes in the brain were modified by 

either full scale intelligence quotient (FSIQ) score (evidence of effect modification by IQ) and, 3) in 

preliminary analyses, whether changes in the normal sexual differentiation of the brain exist by 

exposure-level (V. A. Rauh et al., 2012). 

Authors concluded that while overall brain size did not differ by prenatal chlorpyrifos exposure, certain 

areas of the brain were enlarged among those more highly exposed to chlorpyrifos including superior 

temporal, posterior middle temporal, and inferior post-central gyri, and superior frontal gyrus, gyrus 

rectus, cuneus, and pre-cuneus in the mesial views of the right hemisphere of the cerebrum. Further, 

they noted a difference exposure-response relation between cerebral surface distances and CPF 

exposure, by FSIQ wherein the relation between brain area enlargements and FSIQ were positively 

correlated in the low CPF group (greater regional brain size correlated with greater IQ), but an inverse 

relation among the high CPF group. In supplemental analyses, authors illustrated that areas of regional 

brain enlargement among those more highly exposed to chlorpyrifos were due to underlying white 

matter enlargements. Researchers also noted sporadic differences in cortical thickness wherein higher 

CPF exposure was related to reduced cortical thickness. Preliminary analyses also displayed a disruption 

of normal male-larger-than-female areas as well as female-larger-than-male in certain areas of the brain 

known to differ by child sex in normal brain. Overall, researchers concluded that differences in brain 

structure (regional cerebral size and thickness) exist between CPF exposure groups, and the differences 
2

(high>low CPF) in regional brain size may be due to enlargement of underlying white matter. However, 

authors did not speculate as to the underlying biological explanation of the observed qualitative 

interaction between CPF exposure and FSIQ with morphological changes in key areas of the brain linked 

with learning, cognition and social behavior.

Design and Methods 

Children selected for this MRI study (n=40) were participants in the larger Columbia Center for 

Children’s Environmental Health (CCCEH) birth cohort study (n=369). The study was initiated in the late 

1990’s to evaluate the relation between air pollution and pesticide exposure and neurodevelopmental 

effects in children. The cohort is comprised of a low income, inner-city population primarily of 

Dominican and African American ethnicity. Mothers were recruited among those who sought prenatal

care by 20 weeks of pregnancy at one of the participating study hospitals. Participants were nonsmoking women, ages 18-35, with no other co-morbidities including diabetes, hypertension, known HIV 

infection, and no documented drug use. Therefore, the cohort represents a population of women at 

low-risk for adverse neurodevelopmental outcomes among their children. Participants were residents of 

NYC for at least one year prior to enrollment in the research study. EPA’s review of all the 

epidemiological studies from the CCCEH (the Columbia studies) is contained in a recent presentation to 

the Federal Insecticide, Fungicide, and Rodenticide Act (FIFRA) Science Advisory Panel (FIFRA SAP), April 

2012 (FIFRA Scientific Advisory Panel, 2012). 

Authors measured chlorpyrifos and other organophosphate pesticides, cotinine, lead and mercury in 

umbilical cord blood samples taken at delivery using CDC analytic methods with published quality 

assurance and control procedures. Prenatal environmental tobacco smoke (ETS) was measured as 

maternal self-report, and verified through cotinine levels (≤15 ng/mL). Polycyclic aromatic hydrocarbon 

(PAH), another environmental chemical possibly related to adverse neurodevelopmental outcomes, was 

measured through personal air monitoring in the third trimester of pregnancy. In the current (MRI) 

study, high chlorpyrifos exposure was defined as the upper tertile of the exposed group, ≥4.39 pg/g, and 

low chlorpyrifos exposure was defined as values below the upper tertile, <4.39 pg/g, including those not 

exposed to CPF. 

To measure intelligence among school aged children, authors used the Wechsler Intelligence Scale for 

Children, 4th edition (WISC-IV). The instrument measures four areas of mental functioning: the Verbal 

Comprehension Index, the Perceptual Reasoning Index, the Working Memory Index, and the Processing 

Speed Index. These indices are associated with, but distinct from, overall intelligence quotient (IQ) and 

are sensitive to cognitive deficits related to learning and working memory. A Full-Scale IQ score 

combines the four composite indices. A General Ability Index score is a summary score of general 

intelligence, similar to Full-Scale IQ, but excludes contributions from both Working Memory Index and 

Processing Speed Index. WISC-IV scores are standardized against U.S. population-based norms for 

English and Spanish-speaking children.
3

MRI 

This section provides some broad detail as to the method of MRI data collection employed in this 

research paper; this is not a critical review of methods. Health outcomes of interest in this study were 

morphological changes in the brain in regions of the pediatric brain that are related to learning, 

cognition and social behavior, as noted in the traditional epidemiology studies. Researchers performed 

brain imaging studies using Magnetic Resonance Imaging (MRI) technology among a sub-sample of the 

cohort, and measured and compared morphological characteristics by chlorpyrifos exposure status. 

Details concerning the MRI imaging procedures, processing and pre-processing image modifications, 

methods to measure cortical thickness and methods used to identify the “template” brain against which 

individual child participant brain images were compared are presented in the Supplemental Information. 

Briefly, researchers used a high-resolution MRI, whole body scanner, with 8-channel head scanner for 

this study. Authors processed MRI images using biomedical imaging software (ANALYZE 8.0) while 

technicians were blinded to participant characteristics, and whether left or right hemisphere of the brain 

was imaged. Before morphological analyses were performed, images in the dataset were re-sliced to 

correct for residual head rotation, tilt, and flexion/extension by participants during imaging. MRI images 

were pre-processed to correct for large-scale variation in color intensity. Extra-cerebral tissues captured 

in the MRI brain image were manually removed from the cerebral image before analyzing differences in 

brain morphology among participants by chlorpyrifos exposure; authors hypothesized difference in 

regional brain areas of the cerebrum only, other parts of the brain were therefore excised from the 

image before processing. Other similar modifications to the brain image were made to enhance 

measurement of key/pre-identified endpoints (cerebral surface morphology and cortex thickness) (See 

Supplemental Information). Variation in cortical grey matter was assigned by authors as four categories 

of color changes across images; authors selected color categories that adequately grouped the 

participants as to degree of cortical grey matter (i.e., creation of four-level categorical variable to 

measure cortical thickness) and to allow for a robust statistical analysis, particularly considering the 

small study sample (n=40). 

To measure morphological characteristics within cerebral areas associated with learning, cognition and 

social behavior, authors identified a “template” brain image against which all participant images were 

compared and measured. Using a two-step process, authors initially identified the individual in the study 

sample that most closely approximated average demographic features (child age, height and weight). 

Using this brain image, all other participant brain image studies were co-registered, i.e., similar points on 

the cerebral surface were matched according to the International Consortium on Brain Imaging. 

Researchers then measured distances of correlated points on the cerebral surface between the 

“template” brain image and all participant images. Using these measured distances, authors then 

identified a final “template” brain based upon the brain image series which most closely matched the 

average distances among cortical surface measures. Authors advocated that use of one “template” brain 

as opposed to an amalgamated “average” brain is superior because of the enhanced clarity and more 

well defined tissue interface with which to make the measurements for the study. All sets of brain image 

studies were then re-calibrated to the final “template” brain image for statistical analyses.
4

Statistical Analyses

Regarding the statistical analyses employed in this paper, the analyses involved comparison of the 

Euclidean distances measured between correlated points on the cerebral surface or cortical thickness by 

chlorpyrifos (high/low) exposure level, adjusting for age and sex. Supplemental information reported 

that thousands of statistical tests were performed. Authors used generalized linear equations and linear 

regression modeling to estimate average differences in morphological characteristics by change in 

chlorpyrifos exposure level (high/low). Because authors performed multiple statistical tests, they utilized 

the False Discovery Rate (FDR) methods to adjust the p-value against Type I error. Authors tested the 

statistical significance of interaction terms of cerebral surface measures and FSIQ, and by child sex at 

p<0.05, adjusted for multiple testing. 

Study Results & Authors’ Interpretation 

For this study, authors selected child participants who had complete measures for CPF, PAH, ETS, and 

who completed the WISC IV testing. There were 70 participants who met these criteria and who also 

had cord blood levels in the upper tertile of 4.39 pg/g (High CPF exposure), and among these 28 met the 

additional criteria of having low or no exposure to PAH and ETS. Twenty of these participants completed 

the MRI series (20 of 28 eligible participants enrolled in study). Similarly, there were 99 participants who 

met the enrollment criteria (completed the environmental exposure and IQ testing battery) and were in 

the lower exposure group (<4.39 pg/g) and had no or low ETS and PAH. Among this group of low CPF 

exposed participants, 38 were randomly selected to participate in the MRI study, and 20 participants 

completed the series (20 of 38 eligible participants enrolled in study). 

Authors reported no significant differences between those in the high and low chlorpyrifos exposure 

group, or among the larger cohort according the common socioeconomic indicators including maternal 

income and education, maternal IQ, race, gestational age and birth weight of child, and blood lead 

measure (See Table 1, main text). The correlation between chlorpyrifos and blood lead was low (ρ<0.20) 

and non-significant (p>0.20). 

Authors concluded that while overall brain size did not differ by prenatal chlorpyrifos exposure, certain 

areas of the brain were enlarged among those more highly exposed to chlorpyrifos including superior 

temporal, posterior middle temporal, and inferior post-central gyri, and superior frontal gyrus, gyrus 

rectus, cuneus, and pre-cuneus in the mesial views of the right hemisphere of the cerebrum. Further, 

they noted interaction, or a different exposure response relation between cerebral surface distances 

and FSIQ, by CPF exposure wherein the relation between brain area enlargements and FSIQ were 

positively correlated in the low CPF group (greater regional brain size equated to greater IQ), but an 

inverse relation among the high CPF group (greater regional brain size related to lower FSIQ scores). 

Authors noted statistical interactions in the superior temporal, inferior frontal, inferior pre-central, and 

inferior post central gyri bilaterally, and the pre-cuneus of the left hemisphere. 

In supplemental analyses, authors illustrated that areas of regional brain enlargement among those 

more highly exposed to chlorpyrifos were due to underlying white matter enlargements. This conclusion 

was supported by the observation of sporadic differences in cortical thickness wherein higher CPF 
5

exposure was related to reduced cortical thickness, or thinning of the cortex in certain areas. The 

qualitative interaction between CPF exposure and FSIQ with cerebral brain region enlargements may be 

explained by size of underlying white matter of the cerebrum, but authors did not speculate as to a 

biological explanation of the observed qualitative interaction. Preliminary analyses also displayed a 

disruption of normal male-larger-than-female areas as well as female-larger-than-male in certain areas 

of the brain known to differ by child sex in normal brain. 

Overall, researchers conclude that differences in brain structure (regional cerebral size and thickness) 

exist between CPF exposure groups, and the differences (high>low CPF) in regional brain size is likely 

due to enlargement of underlying white matter. Authors also observed an inverse relation between 

regional brain enlargements and IQ among those highly exposed to CPF, and positive association 

between regional brain size and IQ among those in the low/no exposed group. While authors do not 

speculate as to a possibly biological explanation of this observed statistical interaction, EPA notes that 

given that total brain size did not differ by exposure group, the enlargement of the white matter in 

certain areas could have been at the decrement of the size, quality or other morphological changes in 

the grey matter in other areas; neurons in grey matter of cerebrum affect learning and cognition. 

Observed inward deformations (reduced brain area) in the dorsal and mesial surfaces of the left 

superior frontal gyrus as also reported the authors suggest may also be part of the explanation of the 

qualitative interaction. However, the authors indicate that additional study is needed to elucidate these 

possibilities. 
6

EPA Review

This study by Rauh et al. 2012 follows upon both previous epidemiological research and experimental 

toxicology studies that suggest an association between chlorpyrifos exposure in the pre-natal and early 

life periods with adverse neurodevelopmental outcomes observed in children through school age, 

including intelligence deficits. Within this CCCEH cohort, authors reported links between prenatal 

chlorpyrifos exposure (concentration of CPF in cord blood at delivery (pg/g)) with attentional difficulties, 

behavioral problems and measures of intelligence particularly working memory (V. A. Rauh et al., 2006; 

V. Rauh et al., 2011). EPA’s review of these previous epidemiology studies from the CCCEH cohort were 

presented to the FIFRA SAP in April 2012 (FIFRA Scientific Advisory Panel, 2012). Regarding this study, 

EPA notes a limited and somewhat unbalanced depiction of the available rodent experimental data was 

presented in the paper. A brief review of references seems to provide some support for the authors’ 

assertion that the epidemiological findings are consistent with those in the experimental literature; 

however, a fuller evaluation of the data concerning animal to human concordance is needed. For 

example, the authors’ stated support for their finding regarding enlargements in underlying white 

matter is not strongly supported by existing experimental data concerning CPF induced glial scarring. 

In this study, authors evaluated morphological characteristics of cortical brain regions which have been 

linked to disrupted learning, cognition and social behavior in previous epidemiologic and experimental 

studies. The cerebrum has been described as the “seat of intelligence” and provides the ability to read, 

write, and speak; to make calculations ,and to compose music; and to remember the past, to plan for 

the future, and to imagine things that have never existed before (Tortora & Grabowski, 2003). However, 

while the cerebrum’s influence upon human intelligence is generally supported, the study lacked specific 

hypotheses regarding particular areas of the cerebrum affected (the largest of the four components of 

the brain). Therefore, as noted in supplemental information, many thousands of statistical tests were 

performed evaluating areas of the cerebrum that have been mapped. Authors appropriately employed 

methods to control for Type I error, however in many ways this research requires replication, as noted 

by the authors. Therefore, use of these data to inform specific underlying biological mechanisms or 

pathways of the potential toxic action of CPF is limited. 

EPA is seeking external peer review of this specific aspect of the current study to verify that the MRI 

methods are sound and appropriate. Briefly, however, the main text and supplemental information 

states that technicians (MRI image readers) were blinded as to exposure status, as well as right or left 

hemisphere. This design feature adds to the validity and reliability of the study. However, EPA notes 

authors do not state whether multiple MRI image readers were employed in identifying and measuring 

cortical surface thickness and cerebral surface area, etc. of interest in this study, and whether quality 

control statistics such as the inter-rater reliability coefficient were calculated and considered. 

Supplemental information notes that validation studies were performed, however does not describe the 

aspects of methods that were validated. EPA particularly seeks input from experts in this area as to 

whether these methods and validation studies, if known, are sound. 

Authors appropriately described the inclusion and exclusion criteria used in this MRI study. Authors 

excluded participants for who measureable levels of both PAH and ETS were reported in the total cohort 
7

study. In this way, by design, authors eliminated the chance of confounding by some other 

environmental chemicals known to effect neurodevelopment in the relation between chlorpyrifos and 

brain morphology. Authors also included only those participants who provided full information 

regarding prenatal chlorpyrifos exposure, other environmental chemical exposure and who completed 

the WISC intelligence testing. Given the relatively small sample size (n=40) for this relatively expensive 

imaging study these decisions eliminated the issue of missing data, and the need to add degrees of 

freedom to statistical models for control of other measured environmental chemicals. These decisions 

added to the validity and statistical power of the study. Although inclusion and exclusion criteria were 

somewhat rigorous in this study, authors still achieved a 60% participation rate among those eligible to 

participant (HIGH: 20/28, LOW: 20/38). Further, Table 1 indicates adequate comparability between the 

exposure groups and the total cohort, although those in the high exposed group were slightly albeit not 

statistically significantly more likely to be in families in which maternal education and income were 

slightly greater than the low exposure group. Therefore, in general, EPA believes that authors were able 

to identify an unbiased sample of cohort, representative of population sample of low income, inner city 

kids. However, given the rigorous enrollment criteria, the external validity of the study findings may be 

limited if the organophosphate pesticide exposure-brain morphology association is modified by factors 

that are more, or less, prevalent in the study populations relative to the population(s) to which 

inference is being made, or to populations with a substantially different exposure range. 

In the larger cohort, authors measured concentration of chlorpyrifos parent compound in the cord blood 

of infants at delivery. Generally, biomarkers are considered superior measures than questionnaire based 

self report or environmental monitoring. The assays used in the study were highly sensitive allowing 

detection of very low levels of chlorpyrifos, in the pg/g range (CDC published methods). However, these 

measures are comparable to those in other samples of the general population, as noted by authors. 

Uncertainties remain however, as to the extent the use of one biomarker measurement reflects 

exposure(s) over critical windows of development during pregnancy, the temporality or periodicity of 

exposure, over the relevant time period of gestation. These cord blood measures have been used in 

several studies to estimate pre-natal exposure to chlorpyrifos, i.e., during the entire period of gestation. 

Validations studies performed within the CCCEH cohort support that use of a one-time measure is an 

accurate and reliable measure with which to rank participants (high/low). In this study, authors 

categorized chlorpyrifos exposure as high (greater than or equal to the upper tertile of 4.39 pg/g) and 

low (all values below 4.39 pg/g including those values below the LOD) which exposure methodology can 

accurately and reliably support. PAH in the ambient air was measured using stationary air monitors and 

ETS was measured through maternal self-report, validated by cotinine level. A noted limitation in the 

measurement of pesticide exposure in these studies is the lack of measurement during the post-natal 

period through the time period of the epidemiologic evaluations, i.e., school age. Chemical exposure in 

the post-natal period may confound or alter the relation between prenatal chlorpyrifos and 

neurodevelopmental outcomes, as noted in EPA’s evaluation (FIFRA Scientific Advisory Panel, 2012). In 

this study, child age and sex were the only co-variables considered; these variables are simple to 

accurately and reliably measure using linkage with birth/medical records. 
8

The selection of few co-variables for the statistical models used to evaluate the relation between 

prenatal chlorpyrifos exposure and brain morphology changes is appropriate. For another variable to be 

a confounder in this relation, the other variable would have to be related to both chlorpyrifos exposure 

and (causally) related to brain morphology (e.g., surface area, cortical thickness, gyri dimensionality). 

Child age and sex are appropriate confounders as these factors would influence brain size, growth and 

differentiation. While some research has noted a link between social characteristics and brain 

development and possibly morphological characteristics, the authors’ decision to restrict CCCEH 

participation to those with very similar SES characteristics reduces that possibility of confounding the 

association of interest in this MRI study by SES. Further, the small sample size supports the use of few 

additional co-variables in the statistical model to maximize statistical power. Similar, within the MRI 

study, authors restricted participation to those without other co-exposures such as PAH and ETS thus 

eliminating the need to consider co-adjustment for these variables. Authors additionally measured 

blood lead in this sample and the overall, total cohort and observe the correlation between chlorpyrifos 

and blood lead (necessary factor for positive confounding bias to occur) was very low (ρ<0.20) and not 

statistically significant (p>0.20). 

Other biases that may be present in epidemiology studies include selection and information biases. 

Selection bias in prospective cohort studies may occur if participants drop out of a study and the 

decision to drop out was related to either the CPF exposure or neurodevelopmental outcome. In the 

CCCEH study the retention rate is high for a birth cohort study. The participation rate in the MRI study 

was good (60%). Authors illustrated that those included and excluded were comparable, and the sample 

and total cohort were also comparable. This ameliorates the concern. Temporal bias, a question as to 

whether exposure preceded disease, is not a large concern in a prospective study. The prospective 

nature of the cohort study also reduces that chance of differential information bias or the 

misclassification of exposure, confounder, or outcome information by key study factors. Non-differential 

exposure measurement error may have occurred: 1) the single measurement chlorpyrifos during the 

third trimester, 2) error arising from differences between measured biomarker levels and actual 

chlorpyrifos at the one-time point, 3) unmeasured time- and space- dependent patterns of chlorpyrifos 

exposure, 4) uncertainty regarding the critical period for chlorpyrifos effects on development, 5) missing 

exposure data, 6) laboratory errors, and 7) imputation of missing exposure levels. Measurement errors 

in the ascertainment of chlorpyrifos exposure are likely to have occurred. However, because of the 

prospective study design employed, the errors are unlikely to result in falsely positive findings, because 

the probability and magnitude of these errors are likely to be independent of the outcome status of 

participants (MRI characteristics). The degree to which measurement error in the imaging studies 

performed using MRI technology are of keen interest to EPA.

Authors employed generalized linear models to estimate the association between CPF exposure 

(high/low) and “voxels” or three-dimensional data reflecting morphological characteristics of the 

cerebral cortex, specifically cortical thickness, distances on the cerebrum, size and dimension of key gyri 

important to learning, cognition, and social behavioral, adjusting for child age and sex. Use of a twolevel exposure variable is preferable as it conserves study power, and is supported by the one-time 

measure of CPF at delivery (cord blood CPF concentration pg/g). Researchers utilized a method to 
9

correct for multiple statistical tests (FDR) which was reasonable and appropriate in this instance. 

Authors acknowledged low statistical power to evaluate statistical interaction was likely including gene

by environment interactions. Linear modeling is a preferred method when sample size is small as the 

method conserves statistical power. Authors did not present a table of model results, only graphical 

depictions of areas of morphological differences of the cerebrum by CPF exposure, color coded to reflect 

the statistical significance of the differences in specific, region brain size or volume or area. Additional 

information reflecting model fit statistics, as well as display of evidence of effect modification in both 

exposure strata (high and low CPF exposure) would enhance clarity and presentation of the research 

results. While authors appropriately characterized the evaluation of the interaction with child sex and 

CPF and brain morphology as preliminary, given the number of statistical tests performed and lack of 

hypotheses regarding particular areas of the cerebrum and cortical surface, results require replication 

before firm conclusions regarding underlying biological mechanisms at play can be posited. 

Authors concluded that the evidence from the study illustrated changes in brain morphology in 

association with higher CPF exposure, and that changes observed were in areas of the brain that subserve those learning, cognition and social behavioral, supported by previous observational and 

experimental literature. 
10

References

FIFRA Scientific Advisory Panel. (2012). Draft Issue Paper Scientific Issues Concerning the Health Effects 

of Chlorpyrifos. http://www.regulations.gov/#!documentDetail;D=EPA-HQ-OPP-2012-0040-

0002.

Rauh, V. A., Garfinkel, R., Perera, F. P., Andrews, H. F., Hoepner, L., Barr, D. B., . . . Whyatt, R. W. (2006). 

Impact of prenatal chlorpyrifos exposure on neurodevelopment in the first 3 years of life among 

inner-city children. Pediatrics, 118(6), e1845-1859. doi: 10.1542/peds.2006-0338

Rauh, V. A., Perera, F. P., Horton, M. K., Whyatt, R. M., Bansal, R., Hao, X., . . . Peterson, B. S. (2012). 

Brain anomalies in children exposed prenatally to a common organophosphate pesticide. Proc 

Natl Acad Sci U S A, 109(20), 7871-7876. doi: 1203396109 [pii] 10.1073/pnas.1203396109

Rauh, V., Arunajadai, S., Horton, M., Perera, F., Hoepner, L., Barr, D. B., & Whyatt, R. (2011). Seven-year 

neurodevelopmental scores and prenatal exposure to chlorpyrifos, a common agricultural 

pesticide. Environ Health Perspect, 119(8), 1196-1201. doi: 10.1289/ehp.1003160

Tortora, Gerard J., & Grabowski, Sandra Reynolds. (2003). Principles of Anatomy and Physiology (pp. 

page 467). United States of America: John Wiley & Sons.


1. I have read and reviewed the material presented in the Summary and Critical 

Review of Rauh et al. (2012) “Brain anomalies in children exposed prenatally 

to a common organophosphate pesticide”, as well as the Rauh et al. (2012) 

paper itself. This Summary fairly represents the methods and findings 

reported in the Ruah (2012) paper. The Critical Review portion seems fair and 

balanced.

2. The Ruah paper uses well-established image acquisition and processing

techniques. It is important to point out that MRI atlasing is not an exact 

science, for instance, as is the determination of weights and measures. In the 

latter case sources of systematic experimental artifacts and instrumental 

biases are known; these can be accounted for and corrected, and accurate 

and precise measurements can be made. MRI atlasing is an evolving science 

in which a particular MRI acquisition protocol is used to produces imaging 

data that is processed by a “pipeline” with which this MRI data is analyzed 

and from which inferences are drawn. Systematic and non-systematic 

artifacts and bias can be introduced at any point in this pipeline. To streamline 

and help standardize processing of multi-subject or multi-site brain MRI data, 

several neuroimaging research centers (e.g., Massachusetts General 

Hospital (MGH), UCLA-LONI, Oxford University, Montreal Neurological 

Institute) have developed and disseminated integrated MRI processing 

pipelines. These are typically based on different assumptions and use 

different mathematical and computational strategies to create brain atlases 

from MRI data and report the statistical significance of their similarities or 

differences. Based on the assumptions and routines described in the Ruah 

paper to analyze their MRI data, it appears that Ruah et al. relied on their own

image-processing pipeline that uses some visualization routines provided in

Analyze 8.0. The use of their own software for this purpose is in itself, no 

particular source of concern.

a) The inclusion of 20 out of 28 eligible participants in the high exposure and 

20 out of 38 of the low risk subjects does not raise any raise any red flags. 

Young children present many challenges to radiologists. Unless sedated, 

they are often non-compliant and a complete patient series may not be 

acquired. Subject motion during the MRI acquisition can be a major 

confound when scanning children. Termination of MRI scanning session

before an entire data set specified in the protocol can be acquired could 

explain why as little as half the data could be salvaged.

b) The authors’ pre-processing and processing of MR images conformed to 

standard methods and practices used in this field. There did not appear to 

be deviations to common practice in the MRI studies. That being said, this 

reviewer would have preferred to see the data analyzed using one or two 

standard processing pipelines. I suspect that the qualitative findings 

would have been the same but that there would have been quantitative 

differences among different approaches.

c) The set of steps used in the acquisition and image-processing pipeline are 

all reasonable. T1-weighted MRI brain data using a fast spoiled gradient 
recalled echo (SPGR) MRI sequence (referred to incorrectly in the text) is 

the standard in the field providing both excellent anatomical fidelity and 

high spatial resolution. Segmentation of gray and white matter areas using 

this MRI data was performed on a slice by slice basis using differences in

signal intensity values. This segmentation step, by itself leads to 

inaccuracies, and is notoriously non-robust, but it is followed up by a 

manual correction step, which in principle should be adequate. The large 

deformation fluid registration method is considered as one of the best 

approaches. While there was an error in their stated formula of the 

Navier-Stokes equation in the supplementary material (missing plus sign) 

this was deemed to be a typo rather than an error in the computer 

software used to implement this approach. It would have been preferable 

for the authors to compare their registration to other methods, such as 

Freesurfer, which was not done here, but this should not be considered an 

omission and is not necessarily a flaw in the study. Many people in the 

community use FreeSurfer or CARET to perform cortical segmentation 

and thickness extraction. However, such methods depend on the 

existence of an average brain, and such an average is often difficult to 

obtain in pediatric subjects.

d) Some people in the community construct an average brain from the group 

of brains within the study. This can be problematic when there is 

significant variability in this sample. The authors choose a representative 

brain and then refined their average. The first step is more similar to 

finding a median brain. There does not appear to be anything problematic 

with this approach.

3. In determining statistical significance, the authors used the False Discovery 

Rate (FDR) test. This test is more permissive than the standard Bonferroni 

multiple comparison test, and is widely used in atlasing applications. The 

entire problem of assessing the significance of similarities or differences 

between signal intensities found within a large number of voxels in MRI is a 

developing area of statistical analysis.

4. I don’t possess the knowledge of the social and psychological aspects of child 

development to comment on this question.

5. There are significant changes in brain structure occurring during this period in 

development; however, many of the primary white matter pathways that one 

sees in the brain of a ten year old are present in skeletonized form already at 

birth. The topology or relative location of different brain structures does not 

appear to change. The large change in brain size can make it difficult to find 

an average brain to which to register other brains.

6. No comment here.

7. This appears to be a competently done study by a qualified group. Most of 

my reservations are not with a particular method or approach used, but with 

the overall problem of validating a particular image processing pipeline. 

8. As I alluded to above, it would be valuable to reanalyze the MRI data using 

Freesurfer or CARET, which use different methods and assumptions, as a 

way to establish consistency in the results, at least in the qualitative findings. 
Sadly, in such atlasing studies, as of yet, there is no “ground truth”. In this 

case consistency and convergence of findings can help improve their 

credibility. 


2) a. Authors report that 20 of 28 eligible participants in the “high” exposure group and 

20 of 38 eligible participants in the “low” exposure group had usable MRIs. Such high 

rate of exclusion due to the quality of MRI scans is not unusual when young children are 

the patients. The quality of MRI data is strongly dependant on the motion stability of the 

subject. Any kind of movement during scan is creating artifacts. Low amplitude repetitive 

motion (heart beats, respiration) could be often dealt with using variety of approaches, 

however, larger amplitude sporadic motion (especially related to claustrophobia or panic 

inside the closed space of the MRI) can easily render MRI data useless. It would be very 

helpful if authors had described the MRI sequence in more detail especially concerning 

the duration of the scan and steps taken to reduce motion artifacts if any.

b. There is no defined ‘standard practice’ in research studies dealing with MRI 

morphometric analysis. The methods of image process and pre-process described in 

Rauh et al (2012) seem reasonable and justified. The analysis software (ANALYZE 

8.0) used in this study is well accepted and widely used in biomedical imaging 

research.

c. Authors used appropriate methods for gray matter segmentation and cortical 

thickness measurements.

d. Authors used elaborate procedure to select the final template brain from the studied 

population. While it is not obvious why the selection of the most representative brain 

might benefit the statistical analysis, authors rightfully point out that the use of 

individual brain as a template is preferable to averaged template due to better 

resolved white matter/gray matter, CSF/gray matter, and gyri/sulci interfaces. The 

bottom line is that the laborious method of two-step selection of the template brain 

might have been overkill, but this did not adversely affect the actual image analysis.

6) It is difficult to compare the developmental neurotoxicity study conducted in 

laboratory rats (Hoberman, 1998a, 1998b, 1999a, 1999b3) with the data presented in 

Rauh et al (2012). Hoberman’s preclinical study does not seem to support clinical 

Rauh’s data, however there are significant differences in the methodology (dosing, 

morphometric methods, rats vs. humans, imaging vs. histology). Rats showed 

decreased brain size as well as the thickness of hippocampus gyrus and parietal cortex 

in high dose group of rats. Rauh et al did not see change in the size of the brain and 

the cortex thickness. The changes in the brain surface morphology were majorly 

attributed to the changes in underlying white matter and the morphology of 

hippocampus was not analyzed. 


Response to Specific questions: 

4) The authors reported several findings (i.e., areas of brain enlargement, qualitative statistical interaction 

between full scale intelligence quotient (FSIQ) and brain morphology; aberrations in the normal sexual 

differentiation of the brain, and sporadic cortical thinning). Please provide your interpretation of the 

results reported by Rauh et al (2012). Based upon your interpretation of the study, what, if any, 

neurodevelopmental consequences would you anticipate in these children? 

The Rauh et al (2012) reported results show a diffuse pattern of morphological differences in the surface 

areas of the brains of the children exposed prenatally to high levels of CPF when compared to a 

representative template brain. Both significant enlargements, bilaterally, occurred across temporal, parietal, 

and frontal regions of the brain found to be primarily the result of enlarged white matter. Some areas in the 

left superior frontal gyrus of high level CPF brains showed depressed deformations of matter (presumeably 

less white matter). The high level CPF group showed a significant positive dose response between exposure 

level and enlargement of the superior frontal gyrus, bilaterally suggesting that as CPF exposure levels 

increased, so did the enlarged white matter in the superior frontal region of the brain. This pattern of 

morphological differences was not reported for the children exposed to low levels of CPF prenatally. When 

the groups were compared in overall cortical thickness, there was the presence of reduced cortical thickness 

in scattered areas across the brain surface in the high CPF group. Overall the pattern of results suggests a 

diffuse or generalized morphological effect on the brain white matter integrity and cortical thickness of 

children exposed to higher levels of CPF. 

When IQ scores were added to the analyses, the low CPF group showed that lower IQs were associated with 

greater depressed deformations while higher IQs were closer to the template standard in areas of pre and 

post central gyri, the inferior frontal gyrus, the precuneus and the fusiform gyrus. Alternatively, the high CPF 

group showed morphological enlargements associated with lower IQs in the these same areas. These results 

support the contention that exposure to CPF, even for some within the low CPF exposure level group, is 

related to general cognitive deficits. 

When sex was analyzed by CPF exposure level in sexually dimorphic brain regions, some reversal of usual 

female vs. male differences occurred. Specifically, females typically have larger right parietal lobes than 

males and this pattern was shown in the low CPF group; however, the high CPF group showed the reverse 

pattern with a larger right parietal lobe (and greater surface distance from the template) in males when 

compared to females. It is not clear what statistical analyses were used to determine this result. The results 

must be interpreted very cautiously since there were only 6 males in the high exposure group and 9 in the 

low exposure group (total males = 15 vs. total females = 25). Thus CPF exposure may also affect typical sex 

dimorphic development in the brain. 

Overall the interpretation of the data from this study appears correct but when the authors list the various 

cognitive processes associated with the CPF effects on specific brain regions, the interpretation is reasonable 

but still an extension beyond what the results actually show. The study only used a general IQ measure to 

quantify cognition in this study and more specific cognitive and behavioral tests would be needed to pinpoint 
specific cognitive processes affected by CPF exposure. In general, among the children with higher CPF 

exposure, and perhaps to some degree those with the lower CPF exposure, a range of cognitive deficits or 

differences might be expected depending upon the level of brain function integrity or general intelligence the 

child’s genetics provide. Those children with greater brain function integrity but higher CPF exposure might 

show soft signs of neurological impairment or non-specific cerebral dysfunction. Those with less functional 

integrity or other, non-CPF-related brain insults, lesions, or genetic predispositions, may show more severe 

but non-specific cerebral dysfunction even with lower CPF exposure. The results of this study suggest that 

one might expect that the most common effects of CPF exposure would be similar to the most common 

effects associated with a range of developmental brain insults, effects such as attention deficits, learning 

disabilities, and deficits in social development. 

6) In the developmental neurotoxicity study conducted in laboratory rats (Hoberman, 1998a, 1998b, 1999a, 

1999b3) EPA’s Data Evaluation Review (DER) of this study, pp. 23-25 and 29 of DER for MRID 44556901 & 

pp. 3-7 and 13-15 of DER for MRID 447873014], effects in brain morphometry were observed in rats 

exposed to chlorpyrifos during gestation. To what degree, if any, are the results of Rauh et al (2012) 

comparable with Hoberman? 

The only indirect comparison between the Hoberman study and the Rauh study is in regard to cortical 

thickness of the dorsal parietal cortex. The Hoberman study found that both males and female rats with a 

high dose of prenatal exposure to CPF showed decreased thickness of the parietal cortex compared to 

controls with no prenatal exposure to CPF. There was no analysis of the low dose prenatal exposure to CPF in 

the Hoberman study. In the Rauh study, the high CPF group included males and females and within that 

group there was a negative correlation such that greater CPF exposure was associated with thinner dorsal 

parietal cortex, bilaterally. The Rauh and Hoberman studies are not completely comparable but they provide 

some convergent evidence of a post-natal effect of higher CPF exposure on cortical thickness in the parietal 

lobe. 


Question 1: For the current peer review, EPA is soliciting comments on the epidemiological aspects of the study 

design which are unique to Rauh et al (2012) (e.g., sample selection and inclusion/exclusion criteria; 

socioeconomic status (SES) considerations; sample size and statistical power, etc.) EPA’s has reviewed of the 

epidemiological context of the MRI study by Rauh et al (2012). Please comment on the Agency’s review with 

respect to the epidemiological aspects of the study. 

- Sample selection and generalizability of the findings: Two racial/ethnic groups were represented: 

African American and Dominican. The sample of mothers is described as “low risk” based on being 

free of diabetes, hypertension, known HIV, and documented drug abuse, as well as being nonsmokers. The sample was described as low-income and urban. The average IQ of the full cohort 

was 85.0. The authors state the findings are generalizable “to a larger urban population because it 

was a representative community-based sample.”

- In agreement with the EPA review, the homogeneity of the sample in terms of 

race/ethnicity and poverty/urban status is appropriate for detecting differences in a small 

sample of women/children, but may not be appropriate for generalizing to a larger urban 

population. 

- The sample is not “low risk” based on the association between maternal poverty (or 

SES/environment) and child neural development.[2-4] This further impacts the 

generalizability of the findings, and need for future research with larger samples and other 

populations. 

- It is not clear how representative the sample is of the community, because of possibly low 

participation rate, especially among the low exposure group. It is stated that 20/28 

(71.4%) eligible individuals in the high exposure group completed the MRI and had 

useable data, while 20/38 (52.6%) in the low exposure completed the MRI and had usable 

data. It isn’t stated specifically how many individual’s chose not to participate or had 

unusable data, and whether that was different between the two groups. It is stated based 

on data in table 1 that there were no differences between those who participated and the 

overall cohort; however, based on small sample size (see below), it is not possible to 

conclude this with certaintly. 

- Sample size. 

- The sample size (20/group) appeared adequate for detecting differences in specific brain 

areas. 

- It is not clear whether the sample size is adequate for concluding that the high 

chlorpyrifos exposure caused the changes in the brain. In table 1, the sociodemographic 

characteristics of the study population are presented, showing no significant differences 

between the high and low exposure groups, as well as between the full MRI sample and 

the larger cohort from which they were selected. It would have been informative to 

include the actual p-values, rather than the notation NS for non-significant. In general, 

the high exposure group seems to have greater risk, and when comparing the high and 

low exposure groups, the sample sizes are small and power is low to detect what may be 

meaningful differences between the groups. Specifically, values in the high exposure 

group are lower for maternal education, maternal IQ, and birth weight, but higher for 

prenatal lead. Power analyses suggest the sample size was not adequate to detect 

whether the differences observed on these measures were significant. The high 
chlorpyrifos group had average prenatal lead exposure of 1.4 μg/dl compared to 0.8 μg/dl 

in the low exposure group. While these levels are not considered elevated, there is 

evidence that even low levels of lead can impact neurodevelopment, and even that the 

observed neurobehavioral deficits are more pronounced at lower blood lead levels when 

compared with higher blood lead levels. [5-8] 

Question 3: Please comment on the statistical procedures used in Rauh et al (2012). 

- The authors included age and sex of the child as covariates in the analyses, which is appropriate. It was 

stated that there was no association between lead and chlorpyrifos, but any association between 

lead and brain differences was not studied or reported. In addition, it seems that one analysis 

used a continuous measure of chlorpyrifos exposure, while all others used a dichotomous 

variable (high/low, with no medium group). It isn’t clear why the continuous measure wasn’t 

used, or whether this would be more informative to demonstrate a relationship between 

chlorpyrifos exposure and neurodevelopment. 

Question 7: What additional comments would you like to provide us that may assist us in better interpreting the 

results and characterizing the uncertainty? 

- A recent publication by the same group suggested there might be differences in the relationship 

between prenatal exposure to chlorpyrifos and working memory by sex, with males being more 

susceptible.[9] The authors controlled for sex in their analyses, and were also able to present 

preliminary results by sex, even with only 6 males in the high exposure group. Additional studies with 

larger samples are needed to better understand differences by sex. 

Question 8: Are there any additional analyses that should be conducted? And what would be the value or 

implications of these analyses? 

- Differences in prenatal lead exposure should be considered, even though they were not statistically 

different between the two groups. Again, this may be due to inadequate sample size to detect a 

difference. The high chlorpyrifos group had almost twice the prenatal lead exposure as the low group. 

It is reported that lead was not associated with chlorpyrifos; however, given the sample size, this may 

not be a reliable finding. Therefore, lead should be included as a covariate, or analyses run determine 

whether lead levels are independently associated with the observed brain differences, in order to tease 

apart a potential role of lead versus chlorpyrifos. 

- Also, if there are data available on other environmental exposures (other pesticides, etc.), the 

association between these exposures and the outcomes should be studied. 

- In general, it is agreed that the research requires replication (as stated by authors and in the EPA 

review). 

REFERENCES 

1. Rauh, V.A., et al., Brain anomalies in children exposed prenatally to a common organophosphate 

pesticide. Proc Natl Acad Sci U S A, 2012. 109(20): p. 7871-6.

2. Farah, M.J., et al., Childhood poverty: specific associations with neurocognitive development. Brain Res, 

2006. 1110(1): p. 166-74. 

3. Hackman, D.A. and M.J. Farah, Socioeconomic status and the developing brain. Trends in Cognitive 

Science, 2009. 13(2): p. 65-73.
4. Tucker-Drob, E.M., et al., Emergence of a gene x socioeconomic status interaction on infant mental 

ability between 10 months and 2 years Psychological Science, 2010: p. 1-9.

5. Bellinger, D.C., Very low lead exposures and children's neurodevelopment. Curr Opin Pediatr, 2008. 

20(2): p. 172-7.

6. Canfield, R.L., et al., Intellectual impairment in children with blood lead concentrations below 10 microg 

per deciliter. N Engl J Med, 2003. 348(16): p. 1517-26.

7. Chiodo, L.M., S.W. Jacobson, and J.L. Jacobson, Neurodevelopmental effects of postnatal lead exposure 

at very low levels. Neurotoxicol Teratol, 2004. 26(3): p. 359-71. 

8. Tellez-Rojo, M.M., et al., Longitudinal associations between blood lead concentrations lower than 10 

microg/dL and neurobehavioral development in environmentally exposed children in Mexico City.

Pediatrics, 2006. 118(2): p. e323-30. 

9. Horton, M.K., et al., Does the home environment and the sex of the child modify the adverse effects of 

prenatal exposure to chlorpyrifos on child working memory? Neurotoxicol Teratol, 2012. 


Answers to Questions for Federal Child Development Experts 

1a. Was the type of testing age appropriate for the outcome measures of memory and IQ testing in 

children reported in the Columbia Mothers and Newborn Study as discussed in the papers by Rauh 

et al., (2006 and 2011)? 

There were no direct measures of memory or IQ in the Rauh et al., 2006 paper. Children were tested at 

the ages of 12, 24, and 36 months. Two of the three age groups were younger than the youngest 

appropriate age for many of the more common measures that are used to assess IQ since many of these IQ 

tests require the child to be able to use and comprehend language. The youngest age to assess IQ for four 

widely used measures of IQ are as follows: The Wechsler Intelligence Scale for Children-Fourth Edition

age 6 years; The Kaufman Brief Intelligence Test, Second Edition age 4 years; The Wechsler Abbreviated Scale 

of Intelligence age 6 years, and The Wechsler Preschool and Primary Scale of Intelligence-Third Edition age 2 

years 6 months. While the Wechsler Preschool and Primary Scale of Intelligence-Third Edition may be 

administered to children as young as age 2 years 6 months, children in the younger age group of 2 years 6 months 

to 3 years 11 months are tested on only four core subsets as opposed to seven core subsets administered to children 

ages 4 years to 7 years 3 months.

The Rauh et al., 2011 paper did directly assess memory and IQ using the Wechsler Intelligence Scale for 

Children-Fourth Edition (WISC-IV). The use of this test was appropriate for the sample because all 265 

subjects had reached the age of seven and the WISC-IV was designed for examinees aged 6 years 0 

months to 16 years 11 months. 

1b. Was the type of testing conducted correctly for the outcome measures of memory and IQ testing 

in children reported in the Columbia Mothers and Newborn Study as discussed in the papers by 

Rauh et al., (2006 and 2011)? 

The Rauh et al., 2006 paper provided extensive detail regarding how the psychological tests were 

administered, and the efforts made to maximize reliability of testing and scoring. However, no 

information was provided regarding the qualifications of the individuals who administered and scored the 

tests. The Rauh, et al., 2011 paper did not provide any information regarding the administration, scoring, 

or qualifications of the individuals who administered the IQ test. Although it should be mentioned that 

the WISC-IV generally has very understandable administration and scoring instructions and that the 

manuals are easy to use, the manual does suggest that examiners be familiar with the Standards for 

Educational and Psychological Testing and that examiners complete graduate training in assessment or be 

supervised by such a person. In addition the manual suggests that examiners have specialized training 

when testing persons from unique linguistic, cultural, or clinical backgrounds. Unfortunately, none of this 

information about the examiners was provided in either study. 

1c. Are there population norms associated with these measures? 

There are population norms associated with the measures used in both Rauh, et al. studies. Below is some 

information regarding the population norms for each of these measures. 

Bayley Scales of Infant Development, Second Edition (BSID-II)

The BSID-II was standardized on 1,700 infants and children between the ages of 1 and 42 months. The 

sample spanned 17 age levels and contained 100 individuals (50 males and 50 females) at each level. Age 
intervals occur monthly up to 12 months (with the exception of ages 7, 9, and 11) and every 3 months 

thereafter to the age of 30 months, and then at 6-month intervals. The standardization sample closely 

approximates the regional, ethnicity, and parent education characteristics of children 1 to 42 months of 

age as determined by the 1988 U.S. Census data. The test developers made an effort to limit the 

standardization sample to children who were not diagnosed with medical, psychological, or behavioral 

problems in order to allow examiners to compare individual child performance to that of normal 

developing peers (Gyurke, 1994). 

Wechsler Intelligence Scale for Children-Fourth Edition (WISC-IV)

The standardization sample used for the WISC-IV included 2,200 examinees divided into 11 age groups, 

200 per group, but the Arithmetic subtest was normed on a sample of 1,100 children. The sample was 

representative of the March 2000 U.S. Bureau of the Census data in terms of age, gender, race ethnicity, 

parent education level, and four geographical regions. 

1d. If so, how can these be used to help characterize the findings in the (Rauh et al., 2006 and 2011) 

studies?

The answer to this question depends on the question that is specifically being addressed in the research 

studies. The BSID-II and the WISC-IV can be used in the current study to determine if mental 

development, psychomotor development, and IQ are worse in children who are exposed to higher levels 

of chlorpyrifos compared to children who are exposed to lower levels of chlorpyrifos. However, given 

that the sample of children used in the Rauh et al. studies had very different characteristics than that of the 

samples on which these measures were standardized, caution should be used when describing the results, 

especially when attempting to generalize these findings to the general population. 

For example, here is some information regarding the approximate percentages from the U.S. Census 

March 2000 Population Survey: School Enrollment Supplemental File which is similar to the sample used 

to standardize the WISC-IV compared to the sample in the Rauh et al. studies. A similar comparison 

could be made for the Rauh et al. studies compared to the BSID-II that were standardized based on the 

1988 U.S. census data. 

Race Ethnicity: 

Population Survey = White –63%, Hispanic – 16%, African American – 16%, Asian – 3.5%, Other – 

1.5%. 

Rauh studies = Dominican – 55%, African American – 45%. 

Parent Education Level: 

Population Survey = High School Graduate or higher – 85% 

Rauh studies = High School Graduate or higher – 69%. 

Geographic Regions: 

Population Survey = Northeast – 18%, South – 34%, Midwest – 24%, West – 24%. 

Rauh studies = Northeast – 100%. 

Household Income 

Population Survey = < $20,000 – 22.1%. >= $20,000 – 77.9% 

Rauh studies = < $20,000 – 52%. >= $20,000 – 48% 
2a. For the outcome measures in the selected papers, how do the distributions compare to 

population norms or other studies using these outcome measures (memory, IQ) in the Columbia 

cohort? 

The distributions of the BSID-II scores for both psychomotor and mental development are reported in the 

Rauh et al., 2006 paper for both the low and high exposure groups. The distributions are based on 

standard scores that have a mean of 100 and a standard deviation of 15. The distribution of the scores for 

the Rauh sample for the psychomotor development index appear to be relatively similar to those of the 

population norms for this instrument. In contrast, the mental development index scores for the lower 

exposure group range from about 6 points less at age 12 months to almost 1 standard deviation (15 points) 

below the mean for the test at age 24 months. Therefore, this sample of children are already below (and at 

24 months well below) the average score for this measure based on the sample on which the measure was 

standardized. Unfortunately, it is impossible to compare the WISC-IV scores to the population norms 

because no information was provided regarding the distribution of WISC-IV scores (i.e., mean and 

standard deviation) in the Rauh et al., 2011 paper. There is a scatterplot showing the raw scores for 

overall IQ and for each of the subtests, but it is not possible to obtain the necessary information to 

compare the distributions of these scores with the norms for the test or with any other study sample. 

Ideally, the means and standard deviations for these scores should be presented for either a non-exposed 

or a non-exposed combined with low exposed group and these should be compared to a moderate or highexposed group as was done for the BSID-II in the Rauh et al., 2006 paper. It should also be noted that the 

IQ composite scores were reported as natural logs of the obtained composite scores. Hence in order to 

obtain the IQ scores the values that were reported would need to be converted back to their original 

composite scores. 

2b. With respect to the domains that make up the outcome measures, how do they compare to 

population norms or other studies populations? 

See 2a. above. It should be mentioned that for each of the subtests of the WISC-IV no information was

reported regarding the means or standard deviations of the samples, hence it is not possible to compare 

these data to the population norms for this measure. 

2c. Are there domains which are considered to be the more/less important in cognitive 

development? 

This is a difficult question to answer because there is little or no consensus regarding what domains are or 

are not as important in cognitive development. However, the assessments that were used in the Rauh et 

al. research to assess infant development and IQ have been widely used in a variety of studies. 

3a. Were the tests used to measure the outcome valid 

The Bayley Scales of Infant Development II (BSID-II) manual presents extensive information regarding the 

construct and criterion validity of this instrument. The strong correlations between the Wechsler Preschool and 

Primary Scale of Intelligence-Third Edition (WPPSI-III) (.73), and the McCarthy Scales of Children's Abilities 
(MSCA) (.79) provide evidence of criterion validity for children at the older age range (36 to 42 months) of the 

BSID-II. The manual presents data from a series of studies with children demonstrating a variety of developmental 

disabilities. These results seem to support the discriminant validity of the BSID-II. Collectively, the validity 

information provided in the manual tends to support the construct and criterion-related validity of the BSID-II.

The Wechsler Intelligence Scale for Children-Fourth Edition (WISC-IV) has been found to have good 

correlations with a variety of other measures of IQ and memory including the Wechsler Preschool and 

Primary Scale of Intelligence-Third Edition (WPPSI-III), the Wechsler Abbreviated Scale of Intelligence 

(WASI), the Children's Memory Scale (CMS), and the second edition of the Adaptive Behavior 

Assessment System (ABAS-II).

3b. Were these tests applicable to the research study question? 

Both the Bayley Scales of Infant Development II and the Wechsler Intelligence Scale for Children IV are 

widely used research tools for measuring mental development and IQ. A text search on the PsychInfo 

database for the Bayley Scales of Infant Development yielded 188 results. A similar search for the 

Wechsler Intelligence Scale for Children IV yielded 341 results. This indicates that both measures have 

been, and are still widely used to assess mental development and IQ, which were both outcomes of the 

Rauh et al. research studies. 

4a. What are the uncertainties around the outcome measurements in this cohort? 

The main uncertainties around the outcome measures stem from the fact that those measured were 

standardized on a sample that was very different than the sample that was used in the Rauh et al. research. 

This is not necessarily problematic when the means for the composite scores (i.e., standard scores) are 

directly compared between two groups within the Rauh et al. sample (as was done for the Bayley Scales 

of Infant Development II in the Rauh et al., 2006 paper). However, caution should be taken when 

interpreting the data from analyses that rely heavily on these standard scores. For example, in the Rauh et 

al., 2006 paper, one potentially interesting and important finding was that a significantly greater 

percentage of children in the higher exposed group had scores of less than or equal to 85 (i.e., 1 standard 

deviation below the mean) for the Mental Development Index (MDI) component of the BSID-II at 36 

months of age and this value was also nearly significant at 24 months of age. This was despite the fact 

that the actual scores on this measure were not significantly lower for the higher exposed than for the 

lower exposed group. This inconsistency may have been due to the fact that the scores for the MDI 

section of the BSID-II for the lower exposed group were near the threshold of 85 to begin at both the 24 

and 36 month age groups so that even a small decrease in scores would cause more children to be below 

this threshold than if the scores on this measure were closer to 100. It is interesting that for the 

Psychomotor Development Index (PDI) scores at the ages at which significant differences occured is 

similar whether one compares the means of the scales between the exposure groups or the percentage of 

children who scored below 85. This may be due to the fact these scores were closer to the standardized 

mean of 100 on this scale to begin with. 

Unfortunately, the Rauh et al., 2011 study did not report any means or standard deviations for overall IQ 

or for any of the subtests. Here the uncertainties stem from the assumptions that are made when 

regression analyses are performed. The main issue here is that outliers can greatly influence the slope of 

the function. For this study the bulk of the data were obtained from children who had exposures of less 

than 10 pg/g. In addition only 7 children had exposures of 15 pg/g or greater, and 2 had exposures of 
20pg/g or greater. Hence the regression (i.e., the slope) of the function would be greatly influenced by the 

data from these children. A second issue regarding the analysis of the data from the Rauh et al., 2011 

paper is that although the correlation between full scale IQ and exposure was significant, as was that of 

working memory and exposure, the correlations were not particularly high (-.13 for full scale IQ, and -.21 

for working memory). In fact, it has been suggested that a correlation should be .5 or higher to be large, 

.3 or higher to be moderate, and .1 to be small (Cohen, 1988). And in fact, the coefficient of 

determination which is a measure of the proportion of variability in the data that is explained by the 

relationship between the two variables would be .0169 for IQ and .0441 for working memory which, 

according to Gravetter and Wallnau (2007), would be indicative of small correlations (a value of .09 (i.e., 

r =.3 or -.3) or greater would be required for medium correlations and .21 or greater for large 

correlations). 

4b. Are socio-economic status and other exposures adequately addressed? 

 

The investigators did an excellent job of addressing the issues of other exposures and socio-economic 

status of the children in their sample. 

5. What additional comments would you like to provide us that may assist us in better interpreting 

the results and characterizing the uncertainty? 

It should be mentioned that the results from the Child Behavior Checklist in the Rauh et al., 2006 study 

were analyzed in a manner that was similar to the analysis conducted on the Bayley Scales of Infant 

Development II with regard to the proportion of children who had a score at or below a certain value. The 

data for the Child Behavior Checklist are reported as the percentage of children who met criteria for being 

in the clinical problem range on one of several dimensions with higher scores indicating a greater problem 

in that domain. Once again, this sample of children may have had scores that were already well above the 

mean for this measure such that even a small increase in scores would be enough to be above the critical 

point on that measure. Unfortunately, this could not be determined because the authors did not provide 

the means and standard deviations for the children in their sample on this measure for either the lower or 

higher exposed group. 

6. Are there any additional analyses that should be conducted? And what would be the value or 

implications of these analyses? 

A between group analysis using inferential statistics, as was done for the Bayley Scales of Infant 

Development II in the Rauh et al., 2006 paper, should be performed on each variable in both studies (i.e., 

the Child Behavior Checklist in Rauh et al., 2006, and the full scale IQ and subscales for the WISC-IV in 

the Rauh et al., 2011 study). This would be the most direct and least problematic method for determining 

whether exposure to chlorpyrifos resulted in significant decreases in IQ or significant increases in 

behavioral problems. If significant differences were found between the groups, this would provide much 

stronger evidence than the evidence that is currently provided by the analyses that were presented in the 

Rauh, et al., 2006 and the Rauh, et al., 2011 papers. 
References: 

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). New Jersey: Lawrence 

Erlbaum. 

Gravetter, F.J. & Wallnau, L.B. (2007). Statistics for the behavioral sciences (7th ed.). California: 

Thomson Wadsworth. 

Gyurke, J. S. (1994). A reply to Bagnato and Neisworth: intelligent versus intelligence testing of preschoolers. 

School Psychology Quarterly, 9(2), 109-112.