**Documentation of Peer Review for EPA's Estimation of**

**an Integrated Dose-Response Function for**

**Intelligence Quotient (IQ) and Prenatal Exposure to Mercury**

**U.S. Environmental Protection Agency**

**National Center for Environmental Economics**

**March 2005**

EPA's analysis of the dose-response relationship between prenatal
mercury exposure and childhood IQ involves a statistical integration of
data from three epidemiological studies conducted in the Faroe Islands,
New Zealand, and the Seychelles Islands. The analysis consists of the
following three documents:

-   *Neurobehavioral Assessments Conducted in the New Zealand, Faroe
    Islands, and Seychelles Islands Studies of Methylmercury
    Neurotoxicity in Children*, by David C. Bellinger, March 2005.

-   *Effects of Prenatal Methylmercury on Childhood IQ: A Synthesis of
    Three Studies,* by Louise M. Ryan, March 2005.

-   *Adverse mercury effects in 7-year old children expressed as loss in
    \"IQ\"*, by Esben Budtz-Jorgensen et al., March 2005.

Drafts of these reports were peer reviewed in accordance with EPA's
*Peer Review Handbook, 2^nd^ Edition*. EPA contracted with Industrial
Economics, Incorporated (IEc) to coordinate the peer review. IEc
identified the peer reviewers and provided the reviewers with the
documents to be reviewed, along with a set of charge questions and
background documents. IEc was the recipient of the written comments, and
subsequently provided these to EPA. There was no direct contact between
EPA or the authors of the three reports being reviewed and any of the
peer reviewers. The peer reviewers selected by IEc were:

-   Dr. Thomas A. Burke, Johns Hopkins University, Bloomberg School of
    Public Health

-   Dr. John Bailar, University of Chicago (emeritus)

-   Dr. David B. Dunson, National Institutes for Environmental Health
    Sciences, Biostatistics Branch

-   Dr. Joseph Jacobson, Wayne State University, Department of
    Psychology

In addition to drafts of the three reports, the reviewers were provided
several critical papers as background. These papers were:

-   Budtz-Jorgensen E, Keiding N, Grandjean P (2001). Benchmark dose
    calculation from epidemiological data. *Biometrics*, 57: 698-706.

-   Budtz-Jorgensen E, Keiding N, Grandjean P, Weihe P (2002).
    Estimation of health effects of prenatal methylmercury exposure
    using structural equation models. *Environmental Health*. 14;1(1):2.

-   Coull BA, Mezzetti M, Ryan LM. (2003). A Bayesian hierarchical model
    for risk assessment of methylmercury. *J Agricultural, Biological &
    Environmental Statistics*, 8:253-270.

-   Crump KS, Kjellstrom T, Shipp AM, Silvers A, Stewart A. (1998).
    Influence of prenatal mercury exposure upon scholastic and
    psychological test performance: Benchmark analysis of a New Zealand
    cohort. *Risk Analysis*, 18:701-713.

-   Grandjean P, Weihe P, White RF, Debes F, Araki S, Yokoyama K, Murata
    K, Sorensen N, Dahl R, Jorgensen PJ. (1997). Cognitive deficit in
    7-year-old children with prenatal exposure to methylmercury.
    *Neurotoxic & Teratol*, 19:417-428.

-   Myers GJ, Davidson PW, Cox C, Shamlaye CF, Palumbo D, Cernichiari E,
    Sloane-Reeves J, Wilding GE, Kost J, Huang LS, Clarkson TW. (2003).
    Prenatal methylmercury exposure from ocean fish consumption in the
    Seychelles child development study. *Lancet*, 361:1686-1692.

-   Kjellstrom T, Kennedy P, Wallis S, Stewart A, Friberg L, Lind B,
    Wutherspoon T, Mantell C. (1989). Physical and mental development of
    children with prenatal exposure to mercury from fish. Stage 2:
    interviews and psychological tests at age 6. Solna, Sweden: National
    Swedish Environmental Board Report 3642. \[selected pages\]

-   Myers GJ, Davidson PW, Cox C, Shamlaye CF, Palumbo D, Cernichiari E,
    Sloane-Reeves J, Wilding GE, Kost J, Huang LS, Clarkson TW. (2003).
    Prenatal methylmercury exposure from ocean fish consumption in the
    Seychelles child development study. *Lancet*, 361:1686-1692.

The reviewers were also provided with the following set of questions to
focus their reviews.

> ***General Topics***

-   Please comment on the robustness of the methods, models and data
    presented and used in this research.

-   Does the analysis incorporate all relevant studies?

-   Given the scope and intended purpose of the methodology, are the
    analytical framework, assumptions, and application of data
    appropriate? Are the scientific uncertainties clearly identified and
    characterized throughout the analysis?

-   Are the methods applied in this study appropriate for quantifying IQ
    decrements in the range of current U.S. mercury exposures? If not,
    what methods would you recommend?

-   What are the overall major strengths and weaknesses of this
    analysis?

-   Are all of the essential elements included in the final report? Is
    the report clear and well-written? What additional documentation, if
    any, do you feel is needed to ensure transparency?

> ***Specific Topics***

-   This analysis focuses on IQ as the neurodevelopmental outcome for
    mercury economic benefits analysis. Are there other
    neurodevelopmental endpoints for mercury that could be quantified?
    Are there advantages and disadvantages of using IQ to represent
    neurodevelopmental effects of prenatal mercury exposure in addition
    to those identified?

-   The approach to quantifying the IQ dose-response relationship
    integrates data from studies conducted in the Faroes, Seychelles and
    New Zealand. Is it appropriate to combine results from these three
    studies for this analysis? Do differences in the version of the IQ
    test (Wechsler Intelligence Scales for Children, or WISC)
    administered in the three studies raise any issues that are of
    concern to you?

-   The authors of this analysis had to select dose-response
    coefficients from the three studies for use in their statistical
    modeling. Please comment on the choice of dose-response coefficients
    from the three studies for use in this analysis. Are there alternate
    coefficients or other data from these three studies that should be
    used in place or, or in addition to, those used in this analysis?

-   This analysis generally relies on coefficients from linear
    dose-response models. Given the available information, is this an
    appropriate approach, and applicable to the full range of exposures
    experienced by the U.S. population, including exposures below those
    in the range of empirical observation?

-   Full-scale IQ was not measured in the Faroe Islands study; however,
    three IQ subtests were conducted. At the request of EPA, the Faroe
    Islands research team conducted a statistical analysis to estimate a
    dose-response relationship. Is the rationale for extrapolating
    full-scale IQ from the three subtests clearly explained and
    justified? Is the approach to estimating a full-scale IQ
    dose-response relationship for the Faroe Islands appropriate?

-   Integrated dose-response coefficients are estimated first using only
    the IQ dose-response coefficient from the three studies, then also
    incorporating data for other neurodevelopmental endpoints (e.g.,
    Boston Naming Test) in a Bayesian hierarchical model. Is the
    rationale for each approach clearly explained?

-   Only certain non-IQ neurodevelopmental endpoints reported in the
    Faroes, New Zealand and Seychelles studies could be included in the
    Bayesian model. Is the rationale for selection of endpoints for
    inclusion/exclusion in the Bayesian model clear and reasonable? Do
    you agree with the rationale?

-   In order to combine data from different studies and different
    neurodevelopmental endpoints, it was necessary to rescale the
    reported dose-response coefficients. Data from the Faroes is
    converted from terms of cord blood mercury to hair mercury. All
    endpoints other than IQ are converted to the IQ scale. Is the
    rescaling of the coefficients clearly explained and appropriately
    executed?

-   Please comment on the implementation of the Bayesian model and
    interpretation of the results of this model. Is this portion of the
    study clearly explained, and appropriately executed and interpreted?
    Has all relevant information been considered in the model?

The remainder of this document presents the comments of the reviewers
and EPA's responses.

**U.S. EPA** **Mercury IQ Dose-Response Analysis:**

**Peer Review Comments and Responses**

**[Reviewer 1]{.underline}**

Review of the USEPA funded analysis of methylmercury dose-response

Submitted by:

Thomas A. Burke, Ph.D., M.P.H.

Johns Hopkins University

February 13, 2005

**General Comments**

This review examines the following three reports: 1) *Effects of
Prenatal Methylmercury on Childhood IQ: A Synthesis of Three Studies,*
by Louise M. Ryan, December 2004; *Neurobehavioral Assessments Conducted
in the New Zealand, Faroe Islands, and Seychelles Islands Studies of
Methylmercury Neurotoxicity in Children*, by David C. Bellinger,
December 2004; and *Adverse mercury effects in 7-year old children
expressed as loss in \"IQ\"*, by Esben Budtz-Jorgensen et al., draft
report of December 3, 2004. The overall goal is to provide feedback on
the utility of the papers and appropriateness of their methods and
findings to support the efforts of EPA to estimate the public health
impacts of maternal exposure to methylmercury and to conduct a benefits
assessment.

The review has been challenging, since there is no single unifying
document that unifies the goals, methods and findings of each component.
Each of the included papers provides unique, albeit complementary,
perspectives. However, the separation of the components into the three
distinct papers leads to a degree of fragmentation and repetition of
limitations and uncertainties that detracts from the strength of the
methods and findings. In addition, the complexity of the computational
methods contributes to a lack of clarity in the Ryan and Faroe team
papers. It is therefore recommended that a short synthesis document be
developed to explain the specific goals, findings and conclusions of
each paper, and characterizes the weight of evidence, biological basis,
and adequacy of the methods used to measure the dose-response
relationship between maternal exposure and childhood IQ.

> **RESPONSE:** Text that summarizes and synthesizes the overall
> analysis has been prepared and is included in EPA's benefit-cost
> analysis document.

Each of the papers are very well done, and taken together provide a
complementary examination of the weight of the evidence for using the
childhood IQ in the assessment of the health impacts of mercury
exposure. They also provide support for conclusions in the Bellinger
paper that "it is possible to estimate Full-Scale IQ score from the
scores on the subtests that were administered in the Faroe Islands".
Taken collectively they also provide support for the conclusion of the
Ryan paper that "prenatal exposure to MeHg results in a significant
decrease in full scale IQ". The models presented in the Ryan paper
include many outcomes in addition to IQ. This integrated analysis of the
results from the three major studies provides many new insights into
common findings among the studies and strengthens the overall weight of
the scientific evidence of adverse neurodevelopmental effects of
prenatal mercury exposures.

There results of the papers must be interpreted with some caution. There
are numerous limitations to these analyses which are described in each
paper. In addition, the magnitude of the IQ effect is small (-0.1 to
-0.25 IQ points per 1ug/g increase in maternal hair mercury) and the
relevance to the lower U.S. population exposure levels will be debated.
In light of the statement of Bellinger that "Full-Scale IQ may not be
the cognitive endpoint that is most sensitive to prenatal MeHg exposure"
it is possible that basing the benefits assessment on IQ effects may
underestimate the true population public health impact.

> **RESPONSE:** We have added a brief discussion on this point and also
> describe how one could alternatively use the estimate based on the
> cognition/achievement domain, rather than focusing just on one
> endpoint (IQ).

The fundamental question for the benefits analysis may well be "Is IQ
the critical endpoint that should drive the benefits analysis?" To
address this question it is recommended that the report include a
stronger explanation of the rationale for selection of IQ as the
critical endpoint for the benefits analysis. This examination should
include the feasibility of including other neurodevelopmental endpoints,
and also examine the emerging evidence of the relationship between Hg
exposure and cardiovascular effects for the total population.
Cardiovascular disease is the nation's leading cause of mortality.
Although the science is still in the formative stages, an examination of
the potential public health impacts of mercury related cardiovascular
risk should be included in the EPA benefits assessment.

> **RESPONSE:** Further rationale for using IQ has been added, along
> with discussion of possible approaches for using other
> neurodevelopmental endpoints. Cardiovascular effects are outside the
> scope of this effort.

**Charge Questions**

***General Topics***

*Please comment on the robustness of the methods, models and data
presented and used in this research.*

The data utilized for this research have been extensively reviewed, and
are drawn from published and carefully reviewed studies. The integrated
analysis of Ryan includes a broad range of endpoints and included
sensitivity analyses to demonstrate the robustness of the modeling
approaches. While there are inherent uncertainties in the models,
including issues such as low-dose effects or thresholds, the methods
have been applied in accordance with generally accepted assumptions. The
authors have included explanations of the strengths and weaknesses.

*Does the analysis incorporate all relevant studies?*

The analysis is based upon the three key epidemiological studies of
maternal exposures to mercury and childhood neurodevelopmental effects,
with a focus on the Faroe Islands study. While there are many studies of
the health impacts of mercury these are most relevant to assessing
potential neurodevelopmental impacts. This approach is consistent with
the NAS Committee on the Toxicological Effects of Methylmercury.

*Given the scope and intended purpose of the methodology, are the
analytical framework, assumptions, and application of data appropriate?
Are the scientific uncertainties clearly identified and characterized
throughout the analysis?*

In each paper the authors provide a detailed discussion of the
assumptions of the models, and their potential limitations. They also
discuss the limitations of the data, including a thorough examination of
the inter-study differences in test measures and exposure measures. The
analyses could be strengthened if a more detailed discussion of the
biological basis for assumptions used in the models, as written the
methods focus on the computational basis more than the neurological
rationale.

> **RESPONSE:** Discussion of the neurological rationale has been
> expanded in the report by Bellinger. Text in the report by Ryan on
> rescaling of data has been expanded.

Since the benefits assessment has important regulatory implications it
is essential to include a thorough explanation of these uncertainties.
It may be important for EPA to examine the collective uncertainties in
each step and provide approaches for their consideration in the
assessment.

> **RESPONSE:** Discussion of uncertainties in the report by Ryan has
> been expanded.

*Are the methods applied in this study appropriate for quantifying IQ
decrements in the range of current U.S. mercury exposures? If not, what
methods would you recommend?*

According to NHANES hair measurements, the mean population exposure
levels for U.S. women are 20 times lower that the mean in the Faroe
study (4.3 vs. 0.2 ppm). Ryan estimates a -0.1 to -0.25 IQ point
decrease for every increase of 1ppm in hair. Even a doubling of the
population mean level would have a very small effect on individual level
IQ scores. While the study methods may be appropriate for the highly
exposed U.S. women, they may not be appropriate to estimate true
population impacts on IQ. In light of the many other neurological
endpoints it may be more appropriate to develop an alternative
neurodevelopmental index. Since this is not my area of expertise I
cannot recommend methods.

> **RESPONSE:** The modeling approach used in this analysis can provide
> a coefficient for the overall cognitive/achievement domain, and other
> model outputs may be informative as to broader neurodevelopmental
> impacts. It is not clear, however, how such outputs could be used in
> quantitative benefit-cost analysis at this time, though this may be
> feasible with further economic research. Discussion of these points
> has been added.

*What are the overall major strengths and weaknesses of this analysis?*

This is discussed in the initial general comments section. The analysis
is very thorough and strong, but IQ may not be the most appropriate
measure of population public health effects on which to base a benefits
assessment. There are a number of uncertainties that each of the authors
point out and there will be debate concerning the relevance of the
findings to the U.S. population. Perhaps most importantly, IQ alone may
underestimate the broader public health effects of population mercury
exposures.

> **RESPONSE:** The discussion of the limitations of using IQ has been
> expanded.

*Are all of the essential elements included in the final report? Is the
report clear and well-written? What additional documentation, if any, do
you feel is needed to ensure transparency?*

As mentioned in the general comments section, the papers should somehow
be pulled together through a summary document that clarifies the goals,
findings, and implications of each component. Each author should also
provide a more complete statistical and biological explanation for
assumptions that have been applied in selecting or eliminating
variables. The conclusions of each paper should be presented more
clearly, including a discussion of population public health
implications. While the papers are well written, they address very
complex issues and methods and are difficult to read and it is
challenging to recognize their important interconnections.

> **RESPONSE:** The documents have been revised as suggested.

***Specific Topics***

*This analysis focuses on IQ as the neurodevelopmental outcome for
mercury economic benefits analysis. Are there other neurodevelopmental
endpoints for mercury that could be quantified? Are there advantages and
disadvantages of using IQ to represent neurodevelopmental effects of
prenatal mercury exposure in addition to those identified?*

The major disadvantage is described in the Belllinger paper, that is IQ
may not be the most sensitive cognitive endpoint. Therefore the
disadvantage of using IQ to represent neurodevelopmental effects is that
it may underestimate true population impacts. Other potentially
quantifiable outcomes should be examined and included, perhaps using
methods of risk assessment to address uncertainties. *\
*

> **RESPONSE:** See responses above. *\
> *

*The approach to quantifying the IQ dose-response relationship
integrates data from studies conducted in the Faroes, Seychelles and New
Zealand. Is it appropriate to combine results from these three studies
for this analysis? Do* differences *in the version of the IQ test
(Wechsler Intelligence Scales for Children, or WISC) administered in the
three studies raise any issues that are of concern to you?*

The integration of the data from the three major studies is appropriate
and provides important insights into the overall similarities of
effects. It also contributes to the weight of the evidence of the
association between maternal exposures and childhood neurodevelopment.
The combination of these data is appropriate, and the author takes great
care in describing similarities and differences, and potential sources
of uncertainty. All three papers address the relevance and potential
limitations of combining the results, including the differing measures
of IQ. This is not my specific area of expertise, however I have
consulted colleagues about this issue and they have reaffirmed the
findings of the authors that the combinations are appropriate.

The differences in testing will undoubtedly be a major focus of policy
debate. It is unfortunate that the Faroe study, the critical study for
the development of the RfD, has the most limited measures for
quantifying IQ.

*The authors of this analysis had to select dose-response coefficients
from the three studies for use in their statistical modeling. Please
comment on the choice of dose-response coefficients from the three
studies for use in this analysis. Are there alternate coefficients or
other data from these three studies that should be used in place or, or
in addition to, those used in this analysis?*

Ryan does a good job of describing the selection of the doe-response
coefficients from a computational perspective, the neurological basis
for these selections are not addressed. In general I do not feel
qualified to address this question.

> **RESPONSE:** The neurological effects of mercury are not sufficiently
> well understood at a mechanistic level to provide a basis for
> selecting tests. All three studies employed a broad range of tests
> because they were not exactly sure what it was they would find. We
> focused our analyses on the test score for which the most data are
> available regarding long-term implications (i.e., IQ). Since this is a
> cognitive outcome, we chose results for other cognitive tests for
> inclusion in the Bayesian model for our main analysis; the tests
> selected would be widely recognized as representing neurological
> outcomes in the cognitive domain; rationale for this grouping is
> provided in the text.

*This analysis generally relies on coefficients from linear
dose-response models. Given the available information, is this an
appropriate approach, and applicable to the full range of exposures
experienced by the U.S. population, including exposures below those in
the range of empirical observation?*

This may be more a policy question than a scientific one. There are
inherent uncertainties at low dose levels. However, the
neurodevelopmental endpoints are continuous measures and there is no
current recognition of a threshold level for MeHg. If the goal of EPA is
to be protective of public health the use of coefficients from linear
dose-response models is appropriate as long as the uncertainties and
policy judgments are documented.

*Full scale IQ was not measured in the Faroe Islands study; however,
three IQ subtests were conducted. At the request of EPA, the Faroe
Islands research team conducted a statistical analysis to estimate a
dose-response relationship. Is the rationale for extrapolating full
scale IQ from the three subtests clearly explained and justified? Is the
approach to estimating a full-scale IQ dose-response relationship for
the Faroe Islands appropriate?*

The discussion section of the Faroes research team paper should be
clarified to more clearly describe the implications of their findings
for estimating Full-Scale IQ from the subtests. This paper should answer
these questions, yet as currently written the conclusions of the authors
are unclear and provide only marginal support for the Ryan modeling. In
general I feel the Bellinger and Ryan papers provide a good examination
of the issues that justifies the use of the subtests as estimates.

> **RESPONSE:** The comment has been shared with the Faroes research
> team.

*Integrated dose-response coefficients are estimated first using only
the IQ dose-response coefficient from the three studies, then also
incorporating data for other neurodevelopmental endpoints (e.g., Boston
Naming Test) in a Bayesian hierarchical model. Is the rationale for each
approach clearly explained?*

The rational is well described, and the results provide important
evidence of the comparability of the studies across a broad range of
endpoints. However this approach is intended as a computational tool and
the underlying scientific basis should be further explained. With
further development this approach may have applicability for the
development of quantifiable estimates of indicators of population
neurodevelopment impacts beyond IQ. One suggestion, this paper could
benefit from editing to improve the clarity for readers with limited
expertise in Bayesian models.

> **RESPONSE:** The text on the Bayesian model has been revised to make
> it clearer. Other text has been added to further describe the
> limitations of focusing on IQ and discussing other model outputs that
> quantify impacts beyond IQ.

*Only certain non-IQ neurodevelopmental endpoints reported in the
Faroes, New Zealand and Seychelles studies could be included in the
Bayesian model. Is the rationale for selection of endpoints for
inclusion/exclusion in the Bayesian model clear and reasonable? Do you
agree with the rationale?*

This question is more appropriately answered by those with expertise in
neurodevelopmental testing and Bayesian statistics. It may be
appropriate to conduct further sensitivity analysis to examine the
influence of the inclusion/exclusion decisions.

> **RESPONSE:** Sensitivity analysis has been added to assess the effect
> of adding in endpoints that were excluded from the primary analysis.

*In order to combine data from different studies and different
neurodevelomental endpoints, it was necessary to rescale the reported
dose-response coefficients. Data from the Faroes is converted from terms
of cord blood mercury to hair mercury. All endpoints other than IQ are
converted to the IQ scale. Is the rescaling of the coefficients clearly
explained and appropriately executed?*

This is an important section of the analysis that could provide
important support for the overall weight of evidence of
neurodevelopmental effects of MeHg. There is well documented data
supporting the conversion of cord blood to hair mercury. However, this
section should provide further discussion of the scientific (biological
and public health) basis for endpoint rescaling.

> **RESPONSE:** Our use of rescaled estimates is analogous to the use of
> "effect sizes," a well-established way to think about comparability of
> dose response coefficients across different scales. We have added
> discussion to this effect in the conclusion section. A substantial
> portion of the conversion is a simple mathematical conversion of a
> number expressed on one scale to an equivalent value on a different
> scale -- similar to converting a temperature from the Fahrenheit scale
> to the Celsius scale. Another aspect of rescaling, which applies only
> to the Faroes, is converting from cord blood units to hair units. This
> rescaling is discussed in the text in some detail.

*Please comment on the implementation of the Bayesian model and
interpretation of the results of this model. Is this portion of the
study clearly explained, and appropriately executed and interpreted? Has
all relevant information been considered in the model?*

My expertise in this area is quite limited. The Bayesian approach
provides a robust tool for examining the results across multiple
endpoints and studies. That being said, the conclusions section of the
paper needs to be strengthened if the results are to provide a useful
and defensible tool for the benefits analysis. Throughout the three
papers there are multiple messages concerning uncertainty, yet the
strategy lacks any statement addressing the point at which the
uncertainty becomes unacceptable for the benefits assessment. Perhaps
this issue should be addressed in the suggested overview document that
would serve to tie the research together and form the basis for EPA
decision making on IQ and maternal mercury exposure.

> **RESPONSE:** This is addressed by the overview and synthesis text
> described above.

**[Reviewer 2]{.underline}**

Review of

*Effects of Prenatal Methylmercury on Childhood IQ: A Synthesis of Three
Studies;*

*Neurobehavioral Assessments Conducted in the New Zealand, Faroe
Islands, and*

*Seychelles Islands Studies of Methylmercury Neurotoxicity in Children;*
and

*Adverse mercury effects in 7-year old children expressed as loss in
\"IQ\"*

Review prepared by

Dr. David B. Dunson

Tenured Senior Investigator, Biostatistics Branch, NIEHS

Adjunct Associate Professor, Institute of Statistics and Decision
Sciences,

Duke University

Adjunct Associate Professor, Department of Biostatistics,

Univ. of North Carolina at Chapel Hill

February 17, 2005

**General Comments:**

My comments are focused on the meta analysis of Professor Ryan. She
combined data from three epidemiologic studies (Seychelles child
development study, Faroe islands study, and New Zealand study) to assess
the effects of prenatal methylmercury exposure on childhood IQ. The
statistical analysis is closely related to the approach described by
Coull et al. (2003). Although the focus is on IQ, Ryan also considered
related endpoints, arguing convincingly that IQ is not measured
perfectly and incorporating related outcomes can potentially improve
estimates of dose response for IQ. In fact, such an idea is well support
in the literature on seemingly unrelated regression. I may add that
intelligence is intrinsically a latent variable or latent trait, so it
may be overly simplistic and misleading to use a single measure of IQ.
Ideally, one would use structural equation models (SEMs), a widely used
approach in psychometrics and social sciences, to formally allow for an
"intelligence" latent trait, which is measured by an item response
battery and may be related to other latent variables (poverty, quality
of childhood education, etc). Of course, analyses that adjust for the
potential confounding effects of childhood environment would require
detailed data on other factors.

Unfortunately, Ryan does not have access to the subject-specific data in
the studies under consideration, and hence is limited in the flexibility
of approaches she can consider to methods, which combine summary
statistics for the different studies. Given the importance of these
results, it would be very appealing to conduct a meta analyses using all
the data. Without this information, it is very difficult to fully
account for uncertainty and to evaluate the appropriateness of modeling
assumptions. In addition, she was forced to select out those endpoints
"for which population means and standard deviations were either provided
or readily available." Certainly, this selection may result in a loss of
information \[refer to bottom of page 4 of her report for the list of
endpoints excluded\].

Because, the different endpoints have different measurement scales, it
is important to standardize in combining the endpoints and in performing
quantitative risk assessment. Ryan uses the approach of expressing the
exposure effect in terms of a percentage of the standard deviation of
the outcome, which seems to be a reasonable strategy. An alternative
strategy is to relate the different endpoints to a latent variable,
which is on some pre-defined scale (e.g., equivalent to the scale of one
of the endpoints or set equal to an arbitrary constant). Such an
approach can be implemented in a SEM analysis. An SEM analysis of the
Faroes data was published previously by Budtz-Jorgensen et al. (2002;
2004). In combining the three IQ subscales, Ryan considers a simplified
SEM.

Although the Budtz-Jorgensen et al. (2002) SEM seems more realistic, the
Ryan simplification seems appropriate for the purposes of combining
information and simplifying presentation of results.

Overall, the analysis is well done and the difficult issues involved
have been carefully considered and dealt with adequately. The main
change I would recommend pertains to the key issue of combining
information from the different outcomes and studies (see point 7 below).
In particular, facing problems with the approach of Coull et al. (2003)
for vague priors due to the small number of studies, Ryan abandons the
original formulation, which was conceptually quite appealing. These
problems likely arose due to the use of vague priors for the parameters
-- certainly the data do not dominant the prior in this case, so it is
not surprising the bad results occur when choosing an unrealistic prior.
For example, a flat prior for the variance components may effective
assign most of its mass to values favoring huge variance among studies
in the regression coefficients. Actually, even if the quality of the
studies varies widely, I would be very surprised to find more than a
modest amount of variability in the coefficients (e.g., changes from
--10 to +15 certainly are implausible). It should be easy to choose a
range of plausible values, particularly given the standardized scale.

> **RESPONSE:** We agree with the commenter regarding the specification
> of priors. To run the model, it was necessary to provide some
> constraint on the range of the prior distributions, but these
> constraints did not impose any strong judgment that would overly
> influence the results; the approach was consistent with the intent of
> using vague priors.

**Specific Comments:**

1\. Step 3, page 5: Excluding tests and endpoints for which the study
means and standard deviations were substantially different from the
population norms seems questionable. One reason for the discrepancies
may be a difference between the study populations and the general
population with respect to important predictors, possibly including the
level of methylmercury exposure. If such predictors are adjusted for in
the analyses, then the results should still be valid. In addition, if
the study sample had a different exposure distribution compared to the
general population and exposure had an effect, then you may be
discarding those endpoints that are most sensitive to the exposure
effect. Another reason for the discrepancy may be variation in the
administrator of the test -- neurobehavioral assessments can be
notoriously variable. Perhaps a compromise would be to conduct a
sensitivity analyses with and without the tests in question.

> **RESPONSE:** A sensitivity analysis has been added considering the
> effect of tests which were excluded from the primary analysis. A
> sensitivity analysis that looks at data for the two Faroes
> Similarities examiners separately, as well as excluding data for
> Similarities entirely, has also been added.

2\. Step 4, page 5: An alternative to arbitrarily selecting one score
for tests with multiple variants would be to include a measurement error
model. For example, use a model with the variants randomly distributed
about a test score latent variable. Such models can be easily fitted in
WinBUGS using a Bayesian approach.

> **RESPONSE:** This is a good idea. However, in light of the challenges
> experienced already with including two levels in our model, due to our
> relatively small sample sizes, such an approach was not practically
> feasible in our setting. Discussion has been added to the report.

3\. Page 8, scale of the latent IQ variable. By assigning a normal
distribution to the latent IQ variable with unit variance, one can use
the regression coefficients to obtain easily interpretable quantities,
such as where typical individuals at any given dose fall in the
population distribution for unexposed (or low exposed) individuals. One
worry with these types of analyses, including both the SEMs and the
rescaling based on estimated standard deviations approach, is the
reliance on the idea that dose results in a shift in the mean of a
normally distributed distribution with constant variance. Certainly,
heterogeneity in the effect of exposure may lead to increasing variance
at higher exposure levels. Inferences on the proportions of individuals
in the tails of the distribution may be particularly sensitive to this
type of violation of the modeling assumptions.

> **RESPONSE:** The primary intended use of the output of the model is
> to estimate the mean IQ decrement in an exposed population for any
> given level of exposure.  We agree that these issues would need to be
> considered if the results were extended to analyze shifts in the tails
> of the distribution.

4\. Discussion of K-power model and linearity assumption: Given the
clear uncertainty in the dose response shape and the large impact
assumptions on the dose response curve have on the results -- e.g.,
estimated dose response, benchmark dose, etc -- the focus on a linear
model is unappealing. The K-power model is not really a great
alternative because the value of K may be driven primarily by data
outside of the range of primary interest. For example, to better fit the
curve at higher exposure levels we may choose K=1, which may be a poor
choice at lower exposure levels. A more flexible approach, such as a
spline seems in order. A monotone regression spline, which enforces the
increasing dose response curve constraint, would be appealing in
estimating the BMD, because it avoid the problem of obtaining an
undefined estimate for negative or zero slopes. A simple alternative is
a Bayesian model averaging approach -- for example, estimate a variety
of dose response curves and assign weights based on a BIC-approximation
to the posterior model probabilities.

> **RESPONSE:** There is insufficient data to explore alternative
> dose-response curves. Some discussion has been added.

5\. Rescaling of methylmercury levels (page 13): Because there is
clearly uncertainty in converting between hair levels and cord blood
levels, it would be appealing to formally account for this uncertainty
using a measurement error model or prior distribution. The sensitivity
analysis approach used here and elsewhere (i.e. repeat analyses plugging
in different plausible values) simplifies the presentation and
implementation. However, a worry is that even though the parameter
estimates may not differ hugely across the different analyses, ignoring
the associated uncertainty may lead to underestimation of the width of
credible intervals. This is particularly problematic when several
unknowns are treated in this manner, which is the case in the current
analysis.

> **RESPONSE:** Based on this suggestion, treatment of the hair:blood
> ratio as a random variable was attempted. This was less easily
> accomplished in the structure of the model than it originally appeared
> -- in part because this conversion applies to only one of the three
> studies. We may revisit this issue at a later time. However, the
> extent to which this variable is uncertain should not be
> overemphasized, as the ratio used is specific to the population to
> which it is applied. The sensitivity analysis using an alternate ratio
> is intended to illustrate how the results might change, but the case
> for using the ratio of 200 in preference to other values is quite
> strong.

6\. Hierarchical model, page 16: This model seems to be a reasonable and
simple way to combine information across the endpoints and studies under
consideration. I worry a bit about the prior specification. Given that
there are only 3 studies under consideration, it would seem difficult to
estimate the degree of heterogeneity among studies based on the data
alone. This may also be the case for the degree of heterogeneity in the
coefficients for different outcomes. It seems that the prior chosen may
have an important impact on the degree of shrinkage of the coefficients
towards the averaged coefficient. I also worry a bit about mixing and
convergence of the MCMC algorithm in WinBUGS, particularly if diffuse
priors are chosen for the variance components.

> **RESPONSE:** Text has been added to the discussion to address this
> issue.

7\. The discussion on page 19 shows that Ryan is well aware of the
issues, and the approach she proposed seems to be reasonable. I would
prefer use of informative prior distributions. It seems that one could
narrow down the range of plausible values for the parameters
considerably -- certainly a very large heterogeneity among studies and
endpoints in the regression parameters seems unreasonable. The
standardized scale helps in choosing plausible values for the variance
components. In addition, mixing may be improved by a hierarchical
centering parameterization or even by the commonly used ad hoc approach
of centering latent variables about their means at each iteration of the
Gibbs sampler. The issue is not entirely the small sample size, because
vague inverse gamma priors for the variance components have well known
problems due to "near impropriety". This has lead to alternative vague
priors, which could be used here. However, the informative prior
approach -- possibly even plugging in an intelligent guess at the
variance components (as is used in the so-called semi-Bayes approach)
seems more reasonable.

**Response to Questions:**

***General Topics***

-   *Please comment on the robustness of the methods, models and data
    presented and used in this research.*

Given the complexity of the data under consideration, it is very
difficult to fully address the robustness issue. However, Ryan has done
a very good job consider a variety of analyses and data to be included.

-   *Does the analysis incorporate all relevant studies?*

Although other relevant studies could potentially be included, the focus
on the 3 studies judged to be of adequate size and quality seems
reasonable.

-   *Given the scope and intended purpose of the methodology, are the
    analytical framework, assumptions, and application of data
    appropriate? Are the scientific uncertainties clearly identified and
    characterized throughout the analysis?*

Overall yes -- see additional comments above

-   *Are the methods applied in this study appropriate for quantifying
    IQ decrements in the range of current U.S. mercury exposures? If
    not, what methods would you recommend?*

Overall yes -- there are many simplifying assumptions, but these seem
reasonable in general. I would recommend changes to the borrowing of
information approach as described above.

> **RESPONSE:** See responses above.

-   *What are the overall major strengths and weaknesses of this
    analysis?*

The strengths of the analysis include the careful consideration of data
to be included and of a variety of approaches for dealing with the
complex multivariate outcomes having different measurement scales. The
weaknesses mainly related to the method for combining information as
described above.

-   *Are all of the essential elements included in the final report? Is
    the report clear and well written? What additional documentation, if
    any, do you feel is needed to ensure transparency?*

The report is very well written and thorough. Many of my suggestions
above (with the exception of the approach for combining information)
would necessitate a more complex analysis and presentation, both of
which may decrease the clarity of the presentation.

***Specific Topics***

-   *This analysis focuses on IQ as the neurodevelopmental outcome for
    mercury economic benefits analysis. Are there other
    neurodevelopmental endpoints for mercury that could be quantified?
    Are there advantages and disadvantages of using IQ to represent
    neurodevelopmental effects of prenatal mercury exposure in addition
    to those identified?\
    *

Other endpoints were considered here. Overall, I think it is a bad idea
to focus sole attention on IQ. Though public interest and concern may be
less for other neurobehavioral outcomes, it would be very appealing to
consider different functional domains jointly in a single analysis.

> **RESPONSE:** We agree that there is value in providing more
> quantitative information for other neurodevelopmental endpoints from
> the studies. Providing these results can lead to expanded qualitative
> or quantitative treatment of these endpoints in benefit-cost analyses.
> As an example of additional outputs that the model can produce, the
> report now provides a coefficient for the overall
> cognitive/achievement domain, which can be interpreted as an index of
> the results for all cognitive endpoints in the model. We will explore
> this type of output and others in future work.

-   *The approach to quantifying the IQ dose-response relationship
    integrates data from studies conducted in the Faroes, Seychelles and
    New Zealand. Is it appropriate to combine results from these three
    studies for this analysis? Do differences in the version of the IQ
    test (Wechsler Intelligence Scales for Children, or WISC)
    administered in the three studies raise any issues that are of
    concern to you?*

I do think that it is appropriate and necessary to combine information
from the studies. Differences in the version of the IQ test have been
adequately addressed in the analysis. These differences may actually be
an advantage, because the combined results may be less sensitive to
limitations in a particular version of the test.*\
*

-   *The authors of this analysis had to select dose-response
    coefficients from the three studies for use in their statistical
    modeling. Please comment on the choice of dose-response coefficients
    from the three studies for use in this analysis. Are there alternate
    coefficients or other data from these three studies that should be
    used in place or, or in addition to, those used in this analysis?*

As I mentioned above, I find it somewhat unappealing to focus on a
selected subset of the dose response coefficients. However, the reasons
presented for doing so in this context seem justified.

> **RESPONSE:** The original stated reasons for focusing on a narrower
> set of endpoints in the main analysis, which were based on the nature
> of the data available, still apply. However, as noted above,
> sensitivity analysis has been added to incorporate a broader selection
> of coefficients.

-   *This analysis generally relies on coefficients from linear
    dose-response models. Given the available information, is this an
    appropriate approach, and applicable to the full range of exposures
    experienced by the U.S. population, including exposures below those
    in the range of empirical observation?*

As mentioned above, the reliance on linearity assumptions does not seem
appropriate. However, given the limitations in terms of data
availability and simplicity of presentations of results and combining
analyses, a good argument can be made in favor of linearity. I certainly
would not recommend the linear model in general, particularly for
estimating the BMD and for quantitative risk assessment.

-   *Full scale IQ was not measured in the Faroe Islands study; however,
    three IQ subtests were conducted. At the request of EPA, the Faroe
    Islands research team conducted a statistical analysis to estimate a
    dose-response relationship. Is the rationale for extrapolating full
    scale IQ from the three subtests clearly explained and justified? Is
    the approach to estimating a full-scale IQ dose-response
    relationship for the Faroe Islands appropriate?*

(Reviewer did not respond to this question)

-   *Integrated dose-response coefficients are estimated first using
    only the IQ dose-response coefficient from the three studies, then
    also incorporating data for other neurodevelopmental endpoints
    (e.g., Boston Naming Test) in a Bayesian hierarchical model. Is the
    rationale for each approach clearly explained?*

(Reviewer did not respond to this question)

*\
*

-   *Only certain non-IQ neurodevelopmental endpoints reported in the
    Faroes, New Zealand and Seychelles studies could be included in the
    Bayesian model. Is the rationale for selection of endpoints for
    inclusion/exclusion in the Bayesian model clear and reasonable? Do
    you agree with the rationale?*

(Reviewer did not respond to this question)

-   *In order to combine data from different studies and different
    neurodevelomental endpoints, it was necessary to rescale the
    reported dose-response coefficients. Data from the Faroes is
    converted from terms of cord blood mercury to hair mercury. All
    endpoints other than IQ are converted to the IQ scale. Is the
    rescaling of the coefficients clearly explained and appropriately
    executed?*

(Reviewer did not respond to this question)

-   *Please comment on the implementation of the Bayesian model and
    interpretation of the results of this model. Is this portion of the
    study clearly explained, and appropriately executed and interpreted?
    Has all relevant information been considered in the model?*

(Reviewer did not respond to this question)

**[Reviewer 3]{.underline}**

Review of

*Effects of Prenatal Methylmercury on Childhood IQ: A Synthesis of Three
Studies;*

*Neurobehavioral Assessments Conducted in the New Zealand, Faroe
Islands, and*

*Seychelles Islands Studies of Methylmercury Neurotoxicity in Children;*
and

*Adverse mercury effects in 7-year old children expressed as loss in
\"IQ\"*

Review prepared by

Dr. Joseph Jacobson

Professor and Chair, Department of Psychology, Wayne State University

February 22, 2005

The three papers provided for this review are very well-written.
Particularly impressive is the paper by Louise Ryan, which lucidly takes
the reader step-by-step through complex statistical analyses that are
not likely to be familiar to most investigators and regulators. The
meta-analytic technique that Dr. Ryan uses is state-of-the-art.

The principal weakness of the dose-response analysis provided by these
papers is the failure of the Faroes investigators to have administered
an IQ test to the children. This limitation is dealt with in these
papers by using the three IQ subtests that were administered in the
Faroes---Similarities, Block Design, and Digit Span---to provide an
estimate of overall IQ. Bellinger is correct that Sattler and other
well-respected scholars have endorsed using 2-5 IQ subtests to estimate
full-scale IQ, particularly when the purpose of the analysis is to
compare particular groups within populations rather than to provide
clinical evaluations of individual patients. However, it is important to
recognize that in the Faroes study the Similarities subtest was much
less strongly related to prenatal methylmercury (MeHg) exposure than two
other verbal tests, Boston Naming and California Verbal Learning. The
MeHg effects detected in those tests would probably have been
represented more adequately by the WISC-III Vocabulary subtest than by
the Similarities subtest that was administered. For this reason, basing
the IQ estimate on the three subtests that were administered in the
Faroes probably led to an underestimate the impact of MeHg exposure on
IQ and, therefore, an underestimate of the slope of the dose-response
curve. Given that the Seychelles study failed to find a relation between
prenatal MeHg exposure and IQ, the evidence for an adverse effect in
this meta-analysis depends primarily on the data from the Faroes and New
Zealand studies. If the IQ analysis understates the impact of MeHg
exposure in one of these two cohorts, the integrative analysis probably
understates its overall impact on IQ. Given the limitation in the data
available from the Faroes, the approach taken here is a reasonable one;
however, the authors need to make clear the likelihood that the slope of
the dose-response function is underestimated.

> **RESPONSE:** We agree that the absence of full-scale IQ testing is a
> limitation for this analysis. We believe that our approach of using
> all of the available data from the WISC, including all three subtests
> and both Similarities examiners, is the most appropriate approach to
> simulating results of a full-scale IQ test.
>
> We do not agree that there is a clear basis for concluding that an
> estimate of full-scale IQ derived from the three available subtests is
> an underestimate. Regarding the suggestion that the WISC-R Vocabulary
> test would be expected to capture the functions represented by the
> Faroes results for the Boston Naming Test (BNT) and the California
> Verbal Learning Test (CVLT-C): the correlation between WISC-R
> Vocabulary subtest score and CVLT-C List A Trials 1-5 Total is 0.33 in
> 5-8 year olds. So the WISC- R Vocabulary subtest probably would not
> have been a very good surrogate for the effects captured in the Faroes
> study by the CVLT-C and, likely, Boston Naming Test. The slope for the
> association between mercury and CVLT or BNT is steeper than the slope
> for IQ, but it is conjecture to assert that the slope for IQ would
> have been steeper had the Vocabulary subtest replaced the Similarities
> subtest, or had the full WISC battery been administered. The analysis
> made use of the best information available for estimating an IQ effect
> in the Faroes, and there is no basis to presume any bias in those
> data.
>
> We have added sensitivity analysis that looks at data for the two
> examiners separately, as well as excluding data for Similarities
> entirely. The results with Similarities excluded are very close to
> those based on using all three subtests.

[Bellinger paper]{.underline}

I agree with Bellinger that the integration of data from the WISC-R and
the WISC-III should not be a problem given how similar the two tests are
in terms of the constructs measured and the distributions of the test
scores. The 5-point difference in the standardization norms should not
be a problem since the integrative analysis is based on within-cohort
correlations. I also agree with Dr. Bellinger that, if the effects of
MeHg are highly focal---affecting only specific cognitive functions---IQ
might underestimate the MeHg effects. However, although the Faroes study
hypothesized and tested for focal deficits, it actually found deficits
on a broad range of cognitive endpoints. Moreover, the magnitude of the
deficits in these focal endpoints did not exceed the magnitude of the
effects found on more global IQ measures administered in New Zealand.
It, therefore, seems unlikely that the results of a full-scale IQ test
would have substantially underestimated the impact of the MeHg
dose-response.

One issue that Bellinger notes, which is not addressed in the
meta-analysis presented by Ryan, is the degree to which a relatively
small effect at the mean of the distribution may translate into a
clinically meaningful deficit at the tails of the dose-response
relationship. Bellinger cites data by Rose and Day showing that the
correlation is often very high between the mean value of a health
indicator within a population and the percentage of members in the
population who meet criteria for disease, i.e., those at one tail of the
distribution. He also refers to data from a randomized control trial
that confirm this same principle but needs to provide a reference for
the latter finding. The principal focus in the meta-analysis presented
here is on the slope of the dose-response curve, which is clearly the
important initial issue that needs to be addressed. However, additional
analyses focusing on the tails of these distributions are also important
for understanding the potential impact of this exposure on those
individuals who are most severely affected. Another important issue
cited by Bellinger that needs to receive more attention is the potential
impact of this exposure on particularly sensitive individuals within the
population. One problem is that one needs to identify the factors that
render particular individuals more sensitive to MeHg exposure, a line of
research that is critical to providing a realistic assessment of the
true cost of this exposure within a given population.

> **RESPONSE:** We agree that additional analysis and research on the
> tails of the distribution and factors related to individual
> sensitivity would be informative. Further analysis of the tails of the
> distribution would require access to the raw data from the three
> studies. The reference for the randomized control trial has been added
> to the Bellinger report:
>
> Laaser U, Breckenkamp J, Ullrich A, Hoffmann B. Can a decline in the
> population means of cardiovascular risk factors reduce the number of
> people at risk? Journal of Epidemiology and Community Health 2001; 55:
> 179-184.

[Budtz-Jorgensen]{.underline}

Budtz-Jorgensen has done an excellent job in providing the reanalyses of
the Faroes data that made Ryan's integrative analysis possible. The
initial decision of the Faroes investigators to use log transformation
of prenatal MeHg exposure to reduce the impact of a few highly exposed
outliers was sound. The National Research Council (2000) report
expressed a concern that the log transformation distorted the Faroes
dose-response curve at the lower end of the distribution, where the
number of cases was relatively sparse. In response to this concern, Dr.
Budtz-Jorgensen reanalyzed the Faroes data using two different
approaches---robust regression and deleting all cases [\>]{.underline}
10 ppm---both of which provide reasonable alternative approaches for
reducing the impact of the outliers. Of the two approaches, the robust
regression is preferable since it makes use of the information from the
full range of the exposure distribution, reducing the impact of the
highest exposed cases without discarding them altogether.

The structural equation modeling (SEM) approach used by Budtz-Jorgenson
for evaluating the dose-response relation between MeHg exposure and
estimated IQ is also very sound in that it permits each of the IQ
subtests to be weighted optimally in relation to exposure. One
additional limitation of the Faroes study explained by Dr.
Budtz-Jorgensen in this paper is that the data on Similarities collected
during Year 2 of the study, when a relatively inexperienced examiner
administered the test, are of questionable validity. Thus, MeHg
predicted reduced Similarities scores during Year 1, when the test was
administered by a more experienced neuropychologist, but not during Year
2. Budtz-Jorgensen handles this problem by controlling statistically for
examiner in the SEM, which is a standard and acceptable approach. The
alternative would be to perform the SEM analysis only on the Year 1
children, which is how Grandjean et al. (1997) handled the problem that
the continuous performance test (CPT) data collected during Year 2 did
not appear to be valid. The advantage of the approach taken with the CPT
data is that the Year 2 data, which are probably not valid, are omitted
from the analysis. The disadvantage is that the sample size is smaller.
It would be of interest to see the analyses performed both ways to
determine whether the dose-response coefficient is stronger when only
the presumably more valid Year 1 data are considered. It should be noted
that Figure 2 in the Ryan paper does not show that examiner was included
in the analysis as a covariate of Similarities; the figure should be
modified to show this important covariate.

> **RESPONSE:** We agree that the difference in results between the two
> Similarities examiners is important to consider and have added
> sensitivity analysis that looks at data for the two examiners
> separately, as well as excluding data for Similarities entirely. The
> results with Similarities excluded are very close to those based on
> using all three subtests.
>
> However, we do not believe it is warranted to disregard the
> Similarities data for Examiner B. The only basis for concluding that
> they are invalid is that the scores were not related to mercury in the
> same way as the scores collected by Examiner A. We do not have the
> information that would be needed to conclude that Examiner B data are
> invalid. The Faroes investigators have not concluded that the Examiner
> B data are invalid.

My only reservation regarding the Budtz-Jorgenson paper relates to
statements made at the end of the middle paragraph on p.3, where in my
view the degree of confounding with PCB exposure is understated. The
issue of confounding with PCBs in relation to certain of the most
important verbal performance endpoints is a difficult one. Dr.
Budtz-Jorgenson has published some analyses that provide some
reassurance that the developmental deficits found to be associated with
MeHg are probably not attributable to PCB exposure. However, in my view,
it is somewhat of an overstatement to characterize the confounding as
"limited."

> **RESPONSE:** We believe the characterization is consistent with the
> published findings and the interpretation offered by the NAS
> methylmercury panel.

[Ryan]{.underline}

As noted above, Dr. Ryan has done an impressive job in integrating the
data from the Faroes, Seychelles, and New Zealand studies and has
provided an exceptionally clear explanation of these meta-analyses. She
does an excellent job of explaining and justifying critical choices at
each step in the analysis; for example, the decisions to exclude the
extreme outlier from the primary New Zealand analyses and to focus on
the whole Faroes sample. The decision to base the scaling factor for the
Faroes IQ measure on the standard deviation of the latent variable
rather than on the scaling factor for Digit Span is also well justified.
The use of sensitivity analyses, such as the simultaneous examination of
the 200 vs. 250 conversion factors for cord blood mercury, is also
impressive and adds to the reader's confidence in the results of the
meta-analysis.

On p. 12, lines 14-16, it would be helpful if the author would explain
the concern about the relatively few observations at the lower end of
the exposure distribution in the Faroes cohort, particularly the
suggestion that the dearth of observations in this range may limit
generalizeability to the U.S. population. Table 3 is confusing. Why are
the Seychelles data listed in the "subset" column? Weren't all the
available data included? I was also confused by Table 10. The note to
Table 10 states that lower values of DIC indicate a better model fit,
but the text at the top of p. 22 states that the optimal value of R
appears to be in the range of 1 to 1.5, where the DIC values are
highest.

> **RESPONSE:** Additional discussion added on page 12, as suggested.
> Seychelles is included in the "subset" column because these
> investigators only report regression analyses for a subset of data
> obtained by excluding some extreme observations found to be highly
> influential (see p 1689 of Myers 2003). DIC values are indeed lowest
> for values of R in the range 1 to 1.5, since they are negative.

The data in Figure 5 warrant some discussion in the text. It should be
noted that in this analysis, the Faroes estimated IQ data overlap
completely with the Seychelles data. Since the Seychelles study found no
evidence of an adverse effect of prenatal MeHg on developmental outcome,
by implication the Faroes estimated IQ data also indicate virtually no
effect. Given that other endpoints in the Faroes study indicate
significant adverse effects, the data in Fig. 5 confirm my concern noted
above that, because IQ was estimated based on three IQ subtests that
were only weakly related to MeHg exposure in this cohort, the estimated
IQ score does not adequately reflect the adverse effect found on other
endpoints. Thus, the dose-response coefficient that emerges from the
integrated cross-study analysis is likely to be underestimated since the
MeHg impact on the Faroes cohort is understated. The principal problem,
which is presented clearly by Bellinger, is that an IQ estimate is
needed to evaluate the societal costs of this exposure. The Boston
Naming Test and the California Verbal Learning Test, two endpoints that
were particularly sensitive to MeHg exposure in the Faroes cohort, are
probably fairly strongly correlated with the WISC-III Vocabulary subtest
and/or Verbal IQ score. (The data in Table 7 illustrate how much more
strongly MeHg was related to Boston Naming than to the estimated IQ
derived from the SEM.) Instead of relying on the Similarities subtest,
which was only very weakly related to MeHg and apparently not validly
assessed in half the sample, it might be better to estimate Vocabulary
or Verbal IQ based on the endpoints that were most sensitive to MeHg and
to include that estimate in addition to or instead of Similarities in
the SEM.

> **RESPONSE:** We respectfully disagree that the estimated effect of
> methyl mercury on IQ is underestimated for the Faroes study. While it
> is true that we did not have access to full scale IQ, the SEM analysis
> does in fact adjust for some of the uncertainty associated with the
> poor measurement. We agree that there is likely to be variability
> among the WISC subtests in sensitivity to the effects of mercury.
> Perhaps Similarities is less sensitive than other subtests that were
> not administered to the Faroes cohort, but there is no way of knowing
> this. The objective of this analysis is to produce a best estimate of
> the IQ dose-response relationship with mercury. We do not believe
> there is a defensible alternative to using all information from the
> WISC subtests that were administered in the Faroes to develop this
> best estimate. It would not be justified to exclude certain tests
> because they displayed a weaker response to mercury in the Faroes
> study.
>
> We do not agree with the inference drawn by the commenter based on
> comparison of the Faroes and Seychelles dose-response coefficients.
> The Seychelles analysis (Myers et al. 2003) found an IQ/methyl mercury
> response of -0.13 IQ points per ppm hair mercury (SE = 0.10). The
> response found in the Seychelles analysis is elevated, though not
> statistically significant (p value from the study is 0.20). One
> interpretation is that there is a response between IQ and
> methylmercury, but the data are insufficient to sufficiently resolve
> the response. It is not uncommon in environmental epidemiology to have
> a number of factors impinge of the ability to fully detect a
> relationship (e.g. issues such as measurement error in the exposure,
> statistical power, etc). It is premature to implicate the Faroes
> results based on the Seychelles response. In addition, the NRC has
> evaluated the neurological responses from the three studies and noted
> why there may be differences in the results among them, some of which
> could influence the findings here.

I generally disagree with the characterization of the results of these
analyses presented in pp. 26-27. To characterize the dose-response
pattern across these three studies as "fairly consistent" strikes me as
an overstatement. Similarly, the statement that the estimates are
"somewhat" sensitive to the assumed values of the variance components,
how the coefficient from the SEM analysis is scaled, and whether the
outlier is included in the New Zealand analysis is, in my opinion, an
understatement. The statement on p. 27 that it is "unfortunate" that IQ
was not measured directly in the Faroes study is also an understatement.
In my view, Dr. Ryan needs to acknowledge clearly that the failure to
measure IQ directly in the Faroes may well have led to an underestimate
of the magnitude of the dose-response coefficient.

> **RESPONSE:** Comments regarding the qualitative characterizations
> will be considered as final versions of the reports are prepared. The
> issue regarding possible IQ underestimation is addressed above.

***General Topics***

-   Please comment on the robustness of the methods, models and data
    presented and used in this research.

The methods, models, and data used in this research are generally very
robust, with the important exception of the data used to estimate IQ for
the Faroes cohort. The problem with these data derives from the failure
of the Faroes investigators to have administered an IQ test to the
children. This limitation is dealt with in these papers by using the
three IQ subtests that were administered in the Faroes---Similarities,
Block Design, and Digit Span---to provide an estimate of overall IQ.
Bellinger is correct that Sattler and other well-respected scholars have
endorsed using 2-5 IQ subtests to estimate full-scale IQ, particularly
when the purpose of the analysis is to compare particular groups within
populations rather than to provide clinical evaluations of individual
patients. However, it is important to recognize that in the Faroes study
the Similarities subtest was much less strongly related to prenatal
methylmercury (MeHg) exposure than two other verbal tests, Boston Naming
and California Verbal Learning. The MeHg effects detected in those tests
would probably have been represented more adequately by the WISC-III
Vocabulary subtest than by Similarities, the verbal subtest that was
administered. For this reason, basing the IQ estimate on the three
subtests that were administered in the Faroes may well have led to a
significant underestimate of the impact of MeHg exposure on IQ and,
therefore, an underestimate of the slope of the dose-response curve
derived from the meta-analysis.

> **RESPONSE:** See responses above regarding the three subtests and
> potential underestimation of IQ for the Faroes.

-   Does the analysis incorporate all relevant studies?

Yes.

-   Given the scope and intended purpose of the methodology, are the
    analytical framework, assumptions, and application of data
    appropriate? Are the scientific uncertainties clearly identified and
    characterized throughout the analysis?

The analytic framework, assumptions, and application of the data are
appropriate and, in most respects, the authors do an excellent job in
identifying and clarifying the principal sources of uncertainty.
However, the degree to which reliance on the three IQ subtests
administered in the Faroes is likely to have led to an underestimate of
the dose-response coefficient is not adequately recognized. On pp. 26-27
of Ryan\'s paper, it is, in my view, an overstatement to characterize
the dose-response pattern across these three studies as "fairly
consistent" and probably an understatement that the estimates are
"somewhat" sensitive to the assumed values of the variance components,
how the coefficient from the SEM analysis is scaled, and whether the
outlier is included in the New Zealand analysis. In addition, Dr.
Budtz-Jorgenson\'s statements at the end of the middle paragraph on p.3
of his paper, in my view, understate the degree of confounding with PCB
exposure. Although he has published some analyses that provide some
reassurance that the developmental deficits found to be associated with
MeHg are probably not attributable to PCB exposure, it is an
overstatement to characterize the confounding as "limited."

> **RESPONSE:** See responses above.

-   Are the methods applied in this study appropriate for quantifying IQ
    decrements in the range of current U.S. mercury exposures? If not,
    what methods would you recommend?

Since the range of exposures in these cohorts overlaps with that found
in the U.S., the methods are appropriate for quantifying IQ deficits in
the range of current U.S. mercury exposures. However, it would be
helpful if the authors would explain more clearly the concern raised by
the NRC panel about the relatively few observations at the lower end of
the exposure distribution in the Faroes cohort, particularly the
suggestion that the dearth of observations in this range may limit
generalizeability to the U.S. population.

> **RESPONSE:** The relatively small number of observations at the lower
> end of the exposure distribution is not at all unusual for
> epidemiological studies of environmental contaminants. The shape of
> the dose-response curve was evaluated by the NRC, and they found that
> the linear model provided the best fit (after excluding supralinear
> models from consideration). Lacking any information to the contrary, a
> linear model down to the lowest doses is appropriate as is used in
> other similar cases. There is more uncertainty at lower exposure
> levels, but the available data do not provide a basis for any other
> approach.

-   What are the overall major strengths and weaknesses of this
    analysis?

The major strengths of this analysis include that it is robust,
state-of-the-art, and provides an important opportunity to evaluate the
societal cost of this exposure based on IQ data. Dr. Ryan does an
excellent job of explaining and justifying critical choices at each step
in the analysis; for example, the decisions to exclude the extreme
outlier from the primary New Zealand analyses and to focus on the whole
Faroes sample. The decision to base the scaling factor for the Faroes IQ
measure on the standard deviation of the latent variable rather than on
the scaling factor for Digit Span is also well justified. The use of
sensitivity analyses, such as the simultaneous examination of the 200
vs. 250 conversion factors for cord blood mercury, is impressive and
adds to the reader's confidence in the results of the meta-analysis. The
principal weakness, as noted above, is that the approach used to
estimate IQ in the the Faroes study probably led to an underestimate of
the impact of this exposure on IQ and, therefore, the magnitude of the
dose-response coefficient derived from it.

-   Are all of the essential elements included in the final report? Is
    the report clear and well-written? What additional documentation, if
    any, do you feel is needed to ensure transparency?

These papers do an impressive job in integrating the data from the
Faroes, Seychelles, and New Zealand studies, and Dr. Ryan has provided
an exceptionally clear explanation of these meta-analyses. She does an
excellent job of explaining and justifying critical choices at each step
in the analysis. One additional analysis that I would like to see would
be a Faroes IQ estimate based only on the children who were assessed on
the IQ Similarities subtest during Year 1. As Budtz-Jorgensen explains,
MeHg predicted reduced Similarities scores during Year 1, when the test
was administered by an experienced neuropychologist, but not during Year
2 when a relatively inexperienced examiner administered the test.
Budtz-Jorgensen handles this problem by controlling statistically for
examiner in the SEM, which is a standard and acceptable approach. The
alternative would be to perform the SEM analysis only on the Year 1
children, which is how Grandjean et al. (1997) handled the problem that
the continuous performance test (CPT) data collected during Year 2 also
did not appear to be valid. The advantage of the approach taken with the
CPT data is that the Year 2 data, which are probably not valid, are
omitted from the analysis. The disadvantage is that the sample size is
smaller. The Faroes IQ estimate might be more valid if it is performed
only on those children for whom a valid Similarities test was obtained.

> **RESPONSE:** We have added a sensitivity analysis of the impact of
> the different results for the two Similarities examiners.

***Specific Topics***

-   This analysis focuses on IQ as the neurodevelopmental outcome for
    mercury economic benefits analysis. Are there other
    neurodevelopmental endpoints for mercury that could be quantified?
    Are there advantages and disadvantages of using IQ to represent
    neurodevelopmental effects of prenatal mercury exposure in addition
    to those identified?

As noted above, the principal advantage of using an IQ and/or estimated
IQ measure is that it will permit evaluation of the societal costs of
this exposure. There are no alternative neurodevelopmental measures that
will provide this information. However, other approaches to estimating
IQ, based on performance on Boston Naming and/or California Verbal
Learning, should probably be considered.

> **RESPONSE:** We are not aware of any methods or precedents for using
> BNT or CVLT-C in the estimation of WISC IQ. We do not believe that a
> stronger effect of mercury is a justifiable basis for selecting tests
> to use in estimating the effect of mercury on IQ. The use of BNT and
> CVLT-C, as well as other non-IQ tests, in estimating the benefits of
> reduced mercury exposure should be explored and considered for future
> analyses of the benefits of mercury exposure reduction. Discussion has
> been added to the Bellinger report on the underestimation of benefits
> when relying solely on IQ and excluding other tests/effects.

-   The approach to quantifying the IQ dose-response relationship
    integrates data from studies conducted in the Faroes, Seychelles and
    New Zealand. Is it appropriate to combine results from these three
    studies for this analysis? Do differences in the version of the IQ
    test (Wechsler Intelligence Scales for Children, or WISC)
    administered in the three studies raise any issues that are of
    concern to you?

Combining the data from these two studies using the approach taken in
these papers in valid. The content of the WISC-R and the WISC-III and
the distributions of the scores are sufficiently similar to warrant
combining results from them in the same analysis. The 5-point difference
in the standardization norms should not be a problem since the
integrative analysis is based on within-cohort correlations.

-   The authors of this analysis had to select dose-response
    coefficients from the three studies for use in their statistical
    modeling. Please comment on the choice of dose-response coefficients
    from the three studies for use in this analysis. Are there alternate
    coefficients or other data from these three studies that should be
    used in place or, or in addition to, those used in this analysis?

The choices seem appropriate.

-   This analysis generally relies on coefficients from linear
    dose-response models. Given the available information, is this an
    appropriate approach, and applicable to the full range of exposures
    experienced by the U.S. population, including exposures below those
    in the range of empirical observation?

The linear models seem appropriate; however, as noted above, the authors
should acknowledge that the relative paucity of cases at the lower end
of the distribution adds some degree of uncertainty to the findings.

> **RESPONSE:** See response above. Discussion has been added as
> suggested.

-   Full scale IQ was not measured in the Faroe Islands study; however,
    three IQ subtests were conducted. At the request of EPA, the Faroe
    Islands research team conducted a statistical analysis to estimate a
    dose-response relationship. Is the rationale for extrapolating full
    scale IQ from the three subtests clearly explained and justified? Is
    the approach to estimating a full-scale IQ dose-response
    relationship for the Faroe Islands appropriate?

Please see my comments above detailing my reservations regarding the
approach used to estimate IQ scores for the Faroes cohort. Using the
approach taken in the analysis, the data in Figure 5 of the Ryan paper
show that the Faroes estimated IQ data overlap completely with the
Seychelles data. Since the Seychelles study found no evidence of an
adverse effect of prenatal MeHg on developmental outcome, by implication
the Faroes estimated IQ data also indicate virtually no effect. Given
that other endpoints in the Faroes study indicate significant adverse
effects, the data in Fig. 5 confirm my concern that, because IQ was
estimated based on three IQ subtests that were only weakly related to
MeHg exposure in this cohort, the estimated IQ score does not adequately
reflect the adverse effect found on other endpoints. Thus, the
dose-response coefficient that emerges from the integrated cross-study
analysis is likely to be underestimated since the MeHg impact on the
Faroes cohort is understated.

> **RESPONSE:** See above responses.

-   Integrated dose-response coefficients are estimated first using only
    the IQ dose-response coefficient from the three studies, then also
    incorporating data for other neurodevelopmental endpoints (e.g.,
    Boston Naming Test) in a Bayesian hierarchical model. Is the
    rationale for each approach clearly explained?

Yes, but it is not clear that the data generated by the Bayesian model
can be used directly in estimating the IQ scores.

-   Only certain non-IQ neurodevelopmental endpoints reported in the
    Faroes, New Zealand and Seychelles studies could be included in the
    Bayesian model. Is the rationale for selection of endpoints for
    inclusion/exclusion in the Bayesian model clear and reasonable? Do
    you agree with the rationale?

The rationale for inclusion/exclusion is sound.

-   In order to combine data from different studies and different
    neurodevelomental endpoints, it was necessary to rescale the
    reported dose-response coefficients. Data from the Faroes is
    converted from terms of cord blood mercury to hair mercury. All
    endpoints other than IQ are converted to the IQ scale. Is the
    rescaling of the coefficients clearly explained and appropriately
    executed?

Yes, and the sensitivity analysis comparing the two conversion factors
is particularly impressive.

-   Please comment on the implementation of the Bayesian model and
    interpretation of the results of this model. Is this portion of the
    study clearly explained, and appropriately executed and interpreted?
    Has all relevant information been considered in the model?

The model is clearly explained, but its relevance to estimation of IQ is
not clear.

> **RESPONSE:** The Bayesian model produces an estimate of the IQ
> dose-response regression coefficient, comparable in interpretation to
> those reported for New Zealand in Crump et al. 1998 and Myers et al.
> 2003, that takes into account information from all three studies. This
> estimate is directly applicable to the estimation of decrements in IQ
> associated with different levels of prenatal maternal mercury body
> burdens.

**[Reviewer 4]{.underline}**

Review of selected materials on methylmercury and IQ

Prepared for Industrial Economics, Inc. by

John Bailar, Professor Emeritus, University of Chicago

February 22, 2005

At the request of IEC I have reviewed materials selected by them
relating to the neurological effects of prenatal methylmercury (mehg)
exposure on IQ. I have reviewed three reports in some detail:

> Louise M. Ryan, Effects of Prenatal Methylmercury on Childhood IQ
>
> David C. Bellinger, Neurobehavioral Assessments conducted in the New
> Zealand, Faroe Islands, and Seychelles Islands Studies of
> Methylmercury Neurotoxicity in Children
>
> Esben Budtz-Jorgensen et al., Adverse Mercury Effects in 7-year-old
> Children Expressed as Loss in \"IQ\"

The paper by Ryan is the main basis of this report, with the other two
in supporting roles. In addition, I have looked at selected parts of 7
other research reports supplied by IEC, which are the bases of the three
papers listed above.

I have not reviewed other relevant materials. Also, I have no more than
a general familiarity with three areas critical to the analysis:
neurological testing, structural equation models, and the biologic
effects of methylmercury. Nevertheless, I believe that my comments below
are correct and appropriate. For full transparency, I am Scholar in
Residence at the National Academies, but I had no role in the NAS report
on methylmercury, released in 2000, nor have I read any part of that
report.

Reader should understand that qualified statisticians facing a complex
set of data, such as here, must make very large numbers of choices, and
that they are likely to make their choices differently. I have therefore
accepted Ryan\'s choices and not even commented on them unless I have
been asked to do so, or I think that our different approaches would lead
to different findings and conclusions.

Summary:

Ryan has done a very solid job of presenting the data, presenting a new
statistical analysis, and discussing the strengths and limitations of
both the data and the methods she uses. However, I would have done two
things in substantially different ways. First, Ryan is careful in
discussing the effects of sampling variation (\"random error\") on
confidence bounds and p-values, but I regret the absence of any serious
discussion of the probable or even possible size of the non-random
uncertainties surrounding her analysis. I believe that they are
considerable, which means that the p-values may be substantially too low
and the confidence bounds substantially too narrow. Second, I would not
have gone as far in estimating effects from combinations of data that
are, or might be, fundamentally incompatible.

> **RESPONSE:** Though we have no access to the raw data, the
> coefficients used in our analyses are all from carefully-controlled
> analysis that adjusted for many covariates. Nevertheless, we agree
> that there may be sources of non-sampling error present in these data.
> The Bayesian hierarchical model allows for this by explicitly
> incorporating study-to-study and endpoint-to-endpoint variance
> components. Random effects are often used to help characterize
> important sources of error that cannot be captured by measured
> covariates. That being said, there are important caveats and
> limitations to our analysis. The discussion of these in the final
> section of the Ryan report has been expanded.

These caveats do not detract from the basically high quality of Ryan\'s
report. I largely concur in her overall conclusion that, \"prenatal
exposure to mehg results in a significant decrease in full scale IQ with
a central estimate in the approximate range of -.1 to -0.25 IQ points
for every 1 ug/increase in maternal mehg \". My remaining uncertainties
about Ryan\'s conclusion have to do mainly with the possibility of
post-natal exposures and other confounders, the narrowness of the
confidence bounds around the estimated coefficients, and the assumption
that a linear scale is appropriate.

Overall, there is more uncertainty about the findings than is evident
from the analyses and text presented, and many of the statistically
significant results have confidence bounds that almost include the point
of no effect. My concern is increased by the \"marked degree of both
study-to-study and endpoint-to-endpoint variability\" (Ryan, page 24)
seen in, for example, Figure 10.

General questions

IEC has asked that my review include specific responses to a list of
questions, classified as general or specific. My answers here about the
general questions refer only to the Ryan analysis. I have a few brief
comments on the other analyses later in this review.

Please comment on the robustness of the methods, models, and data
presented and used in this research.

> The methods, models, and data are all somewhat weak for the purposes
> at hand, but I cannot think of any material improvements. Because of
> these problems, I would have stopped with simple analyses of the
> results from each test in each location, and synthesized the results
> in a more subjective way. Ryan recognizes these problems, and has
> pushed the statistical analysis further than I would, though she may
> be more in the mainstream of statistics on this than I am. What she
> has done is fully defensible.

Does the analysis incorporate all relevant studies?

> I am not an expert in this field, but the studies I know about have
> been as substantially higher levels of mercury exposure. I know of no
> relevant studies that have been missed.

Given the scope and intended purpose of the methodology, are the
analytical framework, assumptions, and application of data appropriate?
Are the scientific uncertainties clearly identified and characterized
throughout the analysis?

> Ryan\'s report passes all these standards without difficulty, with one
> exception. There is insufficient attention to the probable effects of
> non-random errors (bias in the data). These necessarily increase
> levels of uncertainty about the findings, and the increases may be
> quite large. In particular, given the interest of EPA and others in
> the shape of the dose-response relationship, the use of linear models
> seems highly restrictive. This is probably not a problem with Ryan\'s
> analysis, but rather a reflection of the serious limitations of the
> data, but the net result is that linearity cannot be examined, and as
> noted below, lack of linearity would not be at all surprising.
>
> **RESPONSE:** The discussion section of the Ryan report has been
> expanded to include discussion of how the Bayesian hierarchical model
> can be helpful in terms of accommodating systematic variability,
> perhaps due to unmeasured factors, for example, through inclusion of a
> study-to-study random effect. Without access to additional, more
> detailed data, we cannot assess the impact of bias due to things such
> as model misspecification. We agree that the available data would not
> support further evaluation of the shape of the dose-response
> relationship. Our use of a linear model is based on the following
> considerations: 1) The NRC's 2000 report on methylmercury used linear
> model results for deriving BMDs, and cautioned against use of
> supralinear models; 2) the Faroes research team reported that K-power
> models (with the constraint of [K\>]{.underline}1, i.e. with
> supralinearity excluded) fit best with the linear specification, i.e.
> K=1; 3) no non-linear model results are available from the three
> studies (except for Faroes log model), and raw data are not available
> to us for conducting analysis of dose-response shape or other issues;
> 4) the lower end of exposures in the Faroes study overlap
> substantially with U.S. exposure range, indicating that there is
> minimal extrapolation involved in applying the observed data to the
> range of exposures in the U.S.; and 5) there is no evidence, given the
> information at hand, that would support use of an alternative model.

Are the methods applied in this study appropriate for quantifying IQ
decrements in the range of current U.S. mercury exposures? If not, what
methods would you recommend?

> I believe that these methods are acceptable.

What are the overall major strengths and weaknesses of this analysis?

> The major weaknesses are in the data available for analysis: the
> samples are small for this sort of thing, and there is no assurance
> that either the methods of measuring function or the methods of
> estimating mercury exposure are sufficiently similar to treat them
> jointly. One unfortunate result is that EPA\'s primary interest in the
> shape of the dose-response relationship cannot be evaluated from the
> human data; there is simply not enough statistical power for a
> head-to-head comparison of the linear model used with reasonable
> alternative.
>
> **RESPONSE:** We agree that the data are limited and ideally one would
> have detailed individual level data available from multiple different
> studies. We do not believe there is reason to expect that methods of
> measuring function or methods of estimating exposure in the three
> studies differ in any meaningful way. It is reasonable to assume that
> the neurological tests in each study were administered correctly and
> carefully, following the instructions in the test manuals. In
> addition, our analysis is focused on an outcome (IQ) that is common to
> all three studies. While there may indeed be some variation in the
> measurement techniques used to assess mercury exposure in the two
> studies, this will be accommodated in our model as part of
> study-to-study variability. Further, the biomarkers for mercury
> exposure used in these studies have been thoroughly studied and
> evaluated over the past 40 years, and are considered highly reliable
> and well-characterized. Thus, uncertainties in exposure measurements
> across studies can be presumed to be small. Uncertainties in exposure
> data for these studies are small compared to epidemiologic studies of
> other pollutants due to these high-quality biomarkers. See above
> response for discussion of dose-response shape.
>
> The major strengths are in the careful, detailed analysis and
> presentation. This report is a model of its kind.

Are all of the essential elements included in the final report?

> All essential elements are included.

Is the report clear and well written? What additional documentation, if
any, do you feel is needed to ensure transparency?

> The report is clear, well written, and transparent for anyone with
> sufficient knowledge of statistical methods. No additions are needed.

**Specific questions**

Nine specific question areas are listed by IEC.

This analysis focuses on IQ as the neurodevelopmental outcome for
mercury economic benefits analysis. Are there other neurodevelopmental
endpoints for mercury that could be quantified? Are there advantages and
disadvantages of using IQ to represent neurodevelopmental effects of
prenatal mercury exposure in addition to those identified?

> I am not qualified to give a professional opinion on this.

The approach to quantifying the IQ dose-response relationship integrates
data from studies conducted in the Faroes, Seychelles and New Zealand.
Is it appropriate to combine results from these three studies for this
analysis? Do differences in the version of the IQ test (Wechsler
Intelligence Scales for Children, or WISC) administered in the three
studies raise any issues that are of concern to you?

> I would not have combined the data from the three studies, for the
> reasons noted above. I am not qualified to give a professional opinion
> about the different versions of the IQ tests.

The authors of this analysis had to select dose-response coefficients
from the three studies for use in their statistical modeling. Please
comment on the choice of dose-response coefficients from the three
studies for use in this analysis. Are there alternate coefficients or
other data from these three studies that should be used in place or, or
in addition to, those used in this analysis?

> I do not know of any defensible and feasible alternatives to the
> choices Ryan made.

This analysis generally relies on coefficients from linear dose-response
models. Given the available information, is this an appropriate
approach, and applicable to the full range of exposures experienced by
the U.S. population, including exposures below those in the range of
empirical observation?

> See my notes above about the linear model. Other models could have
> been used, and some would surely fit as well as the linear model. the
> data here do not , and could not, rule out a threshold, or
> near-threshold; neither do they rule out a substantial degree of
> supralinearity in (say) a small but highly sensitive sub-population.

Full scale IQ was not measured in the Faroe Islands study; however,
three IQ subtests were conducted. At the request of EPA, the Faroe
Islands research team conducted a statistical analysis to estimate a
dose-response relationship. Is the rationale for extrapolating full
scale IQ from the three subtests clearly explained and justified? Is the
approach to estimating a full-scale IQ dose-response relationship for
the Faroe Islands appropriate?

> If one must estimate a full-scale IQ, this is as good as any other
> method I can think of. However, I question the wisdom of trying to
> combine measures of three rather different things. There were reasons
> for developing the three separate scales in the first place, and I
> believe that those reasons apply here. I can think of no reason, *a
> priori*, for thinking that the constructs behind each of the
> sub-scales should all respond in the same way to mercury. I would not
> have combined these measures, now would I have combined the other
> measures that are discussed here.
>
> **RESPONSE:** The basic nature of an IQ score is that it is an
> aggregate index of several different subscales. The subscales do not
> each assess a single independent neurological function, but represent
> different aspects of many overlapping functions -- and are therefore
> designed to be complementary of one another. In this estimation of
> full scale-IQ for the Faroes, three subscales are being combined --
> many fewer than the ten that are combined when a complete WISC IQ test
> is administered. The Bellinger report discusses the high validity of
> IQ predictions based on the subscales administered in the Faroes. IQ
> is a well-established metric that has been thoroughly evaluated and,
> as discussed in the Bellinger text, has a demonstrated relationship to
> long-term outcomes. Further, there are many studies establishing that
> IQ is sensitive to many chemical and biological insults. We agree with
> the commenter that mercury might affect the underlying constructs
> differently. We have expanded the text noting that IQ is not expected
> to capture all neurodevelopmental effects of mercury.

Integrated dose-response coefficients are estimated first using only the
IQ dose-response coefficient from the three studies, then also
incorporating data for other neurodevelopmental endpoints (e.g., Boston
Naming Test) in a Bayesian hierarchical model. Is the rationale for each
approach clearly explained?

> I am not qualified to give a professional opinion on this.

Only certain non-IQ neurodevelopmental endpoints reported in the Faroes,
New Zealand and Seychelles studies could be included in the Bayesian
model. Is the rationale for selection of endpoints for
inclusion/exclusion in the Bayesian model clear and reasonable? Do you
agree with the rationale?

> I do not agree with the selection of just certain endpoints in this
> context. However, the presentation of what was done is quite clear.
>
> **RESPONSE:** The original stated reasons for focusing on a narrower
> set of endpoints in the main analysis, which were based on the nature
> of the data available, still apply. We have added a sensitivity
> analysis to examine inclusion of a broader set of endpoints in the
> model.

In order to combine data from different studies and different
neurodevelopmental endpoints, it was necessary to rescale the reported
dose-response coefficients. Data from the Faroes is converted from terms
of cord blood mercury to hair mercury. All endpoints other than IQ are
converted to the IQ scale. Is the rescaling of the coefficients clearly
explained and appropriately executed?

> It is clearly explained and appears to be appropriately executed.
> However, each such rescaling inevitably introduces some additional
> uncertainty about eh final results, sometimes a lot of uncertainty,
> and that matter should have been discussed at some length.
>
> **RESPONSE:** The rescaling used in this analysis is a composite of a
> few different mathematical conversions. A substantial portion of the
> conversion is a simple mathematical conversion of a number expressed
> on one scale to an equivalent value on a different scale; the Ryan
> text presents this in the form of multiplying the regression equation
> by the constant 15/σ. This adjustment is similar to converting a
> temperature from the Fahrenheit scale to the Celsius scale - it does
> not add to uncertainty, it is just expressing same values in different
> units. A second aspect of rescaling is use of the 'observed' standard
> deviation for each cohort, rather than the 'expected' or population
> standard deviation, which does introduce some uncertainty. We have
> added discussion of this point to the Ryan report. A final aspect of
> rescaling, which applies only to the Faroes, is converting from cord
> blood units to hair units. This rescaling is discussed in the text in
> some detail. While this aspect does add some uncertainty, we believe
> this is relatively small since the value used in the rescaling is
> specific to the population to which it is applied (i.e. the maternal
> hair: cord blood mercury ratio for the Faroes cohort).

Please comment on the implementation of the Bayesian model and
interpretation of the results of this model. Is this portion of the
study clearly explained, and appropriately executed and interpreted? Has
all relevant information been considered in the model?

> I believe that the Bayesian model is adequately explained and that it
> was appropriately executed and interpreted.

**Further Comments:**

Methylmercury in relatively high doses is beyond question a serious
toxicant for the developing human neurological system, as shown in Japan
by the Minimoto studies and elsewhere. I take the analysis here to be
focused on possible effects at lower, and much more common, doses.
(Throughout this report, I use \"dose\" very loosely, to mean any
measure of the amount of methylmercury in the close environment of the
fetus or infant). Thus major attention should be directed at the shape
of the dose-response relationship in the lower dose range. This involves
two steps: first, determining whether there is evidence of any effect at
all on some outcome; then, if an effect is demonstrated, estimating its
size at various dose levels. Three reports in the scientific literature
have been selected for detailed study because of their \"careful
epidemiologic designs that measured a variety of important potential
confounders such as maternal age and education\" (Ryan, page 26). There
are other important strengths: The samples in each are adequate for
reasonably detailed study, and some of the original data could be used
in lieu of the published summary statistics. Ryan also notes some
weaknesses, including some of those noted below.

Ryan\'s title and much of her analysis are focused on IQ, with other
neurological outcomes (such as other aspects of cognition/attainment,
motor deficits, and attention/behavior) in a supporting role. Reasons
for this emphasis on IQ are not given despite the general doubts and
uncertainties regarding IQ measurement that have been prominent in
recent years.

> **RESPONSE:** The primary objective of this work is to support
> benefit-cost analysis of mercury exposure reductions. The focus on IQ
> is due to the fact that there are established methods for monetizing
> IQ decrements in benefit-cost analysis, and such methods do not exist
> for the other neurological endpoints studies for mercury. The report
> by David Bellinger explains the emphasis on IQ and the associated
> issues. We agree that other outcomes are important. Text has been
> added to the Ryan and Bellinger reports discussing other outputs from
> this modeling that may be informative for quantifying the
> neurodevelopmental effects of mercury (e.g. a coefficient for the
> Cognitive/Achievement domain), even though they can not be used in
> benefit-cost analysis at the current time. Providing these other
> outputs may set the stage for related economic valuation research to
> support future benefit-cost analyses.

Ryan discusses five steps she used to select the data for analysis from
a larger set. While other investigators (including myself) might have
made other choices, each of her decisions seems quite defensible.

One point where I would depart from Ryan\'s analysis is the decision to
combine the measurements of two examiners in one of the studies. Results
from the examiners were quite clearly incompatible, so the combination
is of uncertain meaning. A better approach would have been to present
the data from just the one examiner considered most reliable (despite
the loss of sample size), or to omit that item entirely, or to present
the data from each examiner in parallel analyses without trying to
combine them.

> **RESPONSE:** We believe that our approach of using all of the
> available data, rather than using data for just one examiner or the
> other, is the most appropriate approach. We do not have the
> information that would be needed to conclude that data from one
> examiner or the other are more reliable. We do agree that the
> difference in results between the two examiners may be important and
> have added sensitivity analysis that looks at data for the two
> examiners separately, as well as excluding data for Similarities
> entirely.

Despite the central importance of the shape of the dose-response
relationship to public policy that may be based on this analysis, there
is no real discussion here of that shape, and no attempt to fit and
compare alternative shapes. To compare different shapes would be
difficult and highly uncertain because of the smallness of the available
samples for this purpose, but the absence of discussion of this central
issue is troubling.

> **RESPONSE:** We have expanded our discussion on this point. The main
> problem is that we cannot undertake any analysis based on alternative
> dose response models without access to the raw data or going to the
> original investigators and asking them to run different models. See
> additional discussion of dose-response shape above.

I agree with Ryan\'s decision to use a Bayesian analysis.

I am distinctly uncomfortable with the structural equation analyses,
which use combined exposure metrics to estimate a hypothesized, more
fundamental \"latent variable\". Neither Ryan nor Budtz-Jorgensen
discusses at length the numerous assumptions underlying structural
equation analysis (though Budtz-Jorgensen lists many of them), nor do
they present reasons to believe that structural equations are an
appropriate tool here. This is an important matter because of two
things:

> It is not clear that mercury was the only toxicant affecting these
> children, so that a latent variable would be an abstraction of unknown
> meaning, and
>
> It is not clear that all or even most of the mercury exposure was from
> prenatal exposures. Surely some of these children were breast-fed by
> mercury-bearing mothers, and surely some of the children were eating
> fish or other potentially contaminated foods well before the age at
> which the neurological testing was done.
>
> **RESPONSE:** These are valid concerns. However, they are not specific
> to the SEM analysis and would be present even if full scale IQ had
> been measured. Regarding other toxicant exposures, the question of
> confounding by PCBs in the Faroes study has been examined by the study
> investigators, who found an effect for mercury independent of PCBs
> (see Budtz-Jorgensen et al. 2002). This was also addressed by the NRC
> panel, which concluded that data on the effect of mercury could be
> used assuming no confounding. In addition, blood lead levels were
> measured in Seychelles (at least initially) and were low. Regarding
> potential impacts on neurological outcomes from postnatal/early
> childhood exposures, the Faroes group found little association between
> the children's body burden at age 7 and performance on neurological
> tests at the same age (see Grandjean et al. 1999).
>
> Budtz-Jorgensen E, Keiding N, Grandjean P, Weihe P. Estimation of
> health effects of prenatal methylmercury exposure using structural
> equation models. Environ Health. 2002 Oct 14;1(1):2.
>
> Grandjean P, Budtz-Jorgensen E, White RF, Jorgensen PJ, Weihe P, Debes
> F, Keiding N. Methylmercury exposure biomarkers as indicators of
> neurotoxicity in children aged 7 years. Am J Epidemiol. 1999 Aug
> 1;150(3):301-5.

I am a bit concerned that some of the analyses, in Budtz-Jorgensen as
well as Ryan, were driven more by mathematical and statistical
convenience than by what the problem needs. Examples: Reducing the
K-power model to K = 1 because that fit better than any larger K, though
other evidence suggests that K might be less than one. Using
log-transformed data because the data become more like the standard
normal distribution, though the transformation distorts the critical
dose-response relationship. Omitting one child with very high estimated
exposure, though there was no evidence provided to suggest that that
particular measurement was in error or that a biologic abnormality might
account for the measurement, rather than a failure of the model adopted.
Assuming that everything is linear on the scales adopted, without
providing physical or biological evidence of that, or considering the
high frequency of non-linear (including numerous supra-linear) dose
response relationships.

> **RESPONSE:** Regarding possible values of K\<1, this analysis
> followed the conclusions of the NAS methylmercury panel which
> recommended excluding K values less than 1. In addition, no data are
> available from the New Zealand or Seychelles studies for models with K
> values less than 1, thus such models could not be considered in this
> analysis. Regarding log-transformed data; although the Faroe Islands
> investigators have found that such data provide a better fit than
> un-transformed data, the Ryan analysis does not use the
> log-transformed data. Regarding the omission of the child with very
> high estimated exposure, this analysis followed the recommendation of
> the NAS panel on this issue, but also provides a sensitivity analysis
> with inclusion of data for that child. The neurological effects of
> mercury are not sufficiently well understood at a mechanistic level to
> provide a basis for assumptions regarding the shape of the
> dose-response function or other aspects of the analysis.

The data are highly heterogeneous, coming from different parts of the
world with different populations (including two that are highly inbred,
I understand, which could introduce internal correlations), involve
different neurobehavioral domains and different tests (that were in fact
designed to measure different things) administered by different
investigators with different training (and with evidence of substantial
heterogeneity in the one place where a comparison is possible), with
different details of exposure and slightly different ages and test
conditions. To assume that all of these things should share important
statistical properties seems to me to be inappropriate.

> **RESPONSE:** See responses above regarding differences in the
> neurological tests, examiners and measurement methods. It is true that
> the studies were conducted in different parts of the world with
> different populations -- we believe this is a strength of the analysis
> and the underlying data. The important points are that all three
> studies administered the WISC (albeit in different variations); that
> this is a well-established and well-characterized test; and there is
> no reason to assume that it was not administered appropriately in each
> study. The alternatives to the approach we have taken would be to use
> estimates from each of the studies independently, or to combine
> estimates from the study using a more traditional meta-analysis
> approach, which would not take into account the variability from the
> studies as extensively as the Bayesian model. We believe that our
> approach, which integrates the studies and accounts for variability in
> multiple forms, is the most suitable and defensible way for
> incorporating all available information into the analysis.

I do not understand why differences in the population distribution of IQ
should be rescaled away. I wonder if they might not reflect a situation
that should be preserved in the analysis, though there may have been
good but unstated reasons for the rescaling. The fact that rescaling
makes them a bit more similar seems to me to be irrelevant.

> **RESPONSE:** The rescaling is necessary so as to get the estimated
> coefficients for different tests on a comparable scale. See discussion
> above.

Ryan cites some sensitivity analyses that seem to be right on target.

The similarities in results in the tests taken one test and one location
at a time are encouraging in some ways, but quite puzzling in that one
might expect mehg to act differently on different parts of the nervous
system. Thus I would take this, overall, to be a sign of possible
weakness in the data unless there is other evidence (including
laboratory experiments) showing that various functional capabilities do
in fact move in parallel under the influence of mehg. In my experience,
this is not common in toxicology. Despite my concerns about data
problems, I would have stopped a good bit sooner than these
investigators in trying to find commonalities.

> **RESPONSE:** The neuropsychological tests administered in the three
> studies do not each assess a single circumscribed neural structure. A
> child has to do quite a few things reasonably well to perform well on
> each test. So there is always at least a modest correlation between
> scores on different tests, and if mercury affects performance on one
> test, it is likely to have at least a bit of an effect on another
> test. By aggregating a group of such tests, we might actually end up
> with a more sensitive index of mercury's effect than if we looked at
> each test separately (i.e., a little effect on each constituent test
> might add up to a larger effect on an aggregated index). Our main
> analysis uses only tests in the cognitive domain for precisely the
> reasons outlined in this comment. All the endpoints in our primary
> analysis have been identified as belonging to the
> cognition/achievement domain so that effects are expected to be fairly
> similar for all of them. Our sensitivity analysis using all endpoints
> from additional domains shows more variability, consistent with the
> expectations of this comment, and we agree that this would be a less
> reliable and appropriate basis for estimating the IQ effect.

Ryan notes (page 21) that there was not enough information in the
relatively sparse data available to obtain separate estimates of the
study-to-study and endpoint-to-endpoint variances. This could happen if
the two variance components were highly correlated, and I would
encourage her to check on this. If they are in fact highly correlated,
further study to determine the reasons would be in order.

> **RESPONSE:** The two variance components are computed using very
> different aspects of the data, so it is hard to see that they would be
> correlated. In any case, as discussed in the report, we found that
> there was relatively little information available to reliably estimate
> the two variance components. We instead performed our analysis for a
> variety of different values for the ratio between the two.

The comment at the bottom of Ryan, page 25, about symmetry in mle-based
confidence intervals puzzles me, because there are well-recognized means
for producing asymmetric mle bounds by just altering the probabilities
in the two tails.

> **RESPONSE:** We agree that there are means for producing asymmetric
> confidence intervals for MLEs, and have clarified the text to
> emphasize that "standard" MLE-based confidence intervals are
> symmetric. The purpose of this text was to explain to readers with
> less statistical expertise why the Bayesian confidence intervals are
> not symmetric.

Figure 4 is troubling, because it shows in a very clear way that most of
the differences among subjects within any one location are from factors
unrelated to mehg exposure. The remaining signal is tiny, and could be
accounted for by biases in the data, including biases not yet imagined.
I believe from the evidence here that one would find similar
distributions for the other locations and other tests. It would not take
much to make these data points random around the horizontal line. I
recognize that the point of this whole exercise was, in a sense, to try
to identify and measure a weak signal in the presence of a lot of noise,
and I think that the evidence overall does show a signal, but I am not
entirely convinced.

> **RESPONSE:** We agree that the signal is small, however, as noted by
> the NRC, the epidemiology studies were extremely well conducted,
> including extensive information on potential confounders and included
> reliable individual measurements of exposure. The NRC concluded that
> "The weight of the evidence of developmental neurotoxic effects from
> exposure to MeHg is strong" (p. 326). In addition, there are other
> factors that could have biased the relationship toward the null, such
> as measurement error.