Document ID: EPA-HQ-OPP-2009-0104-0054
Agency: epa
Document Type: Supporting & Related Material
Title: 
Posted Date: 2009-05-19T04:00Z

Submission for the FIFRA SAP on the “Ecological Significance of
Atrazine Effects on Primary Producers in the Corn and Sorghum Growing
Region of the United States (Part II), May 12-14, 2009

Preliminary response to presentation by Syngenta regarding probabilistic
CASMATZ2

Russell Erickson 

Mid-Continent Ecology Division

Office of Research and Development

U.S. Environmental Protection Agency

May 14, 2009

This additional written submission is intended to identify EPA areas of
agreement and concern regarding the presentation by Syngenta in the
public comment period on May 12 regarding probabilistic applications of
CASMATZ2 for assessing atrazine levels of concern.  This is provided in
the hope that it will be of some use in the development of SAP comments
on whether and how CASM should be further developed.  Because this oral
presentation was EPA’s first exposure to the Syngenta’s methodology
and because a variety of important details on the methodology and
analysis are not available, these comments are general and preliminary.

(A) Points of Agreement Between EPA and Syngenta

(1) “Generic” Nature of CASM Application

Syngenta gave particular emphasis to the fact that they were describing
a generic application of CASM, as desired by EPA.  This was presumably
in response to the EPA’s conclusion that CASM was better suited for
system-specific applications and entailed too much uncertainty for the
desired generic application, but might also reflect some confusion on
EPA’s definition of “generic”.  This comment is to clarify, if
there is any question about this, that what Syngenta described is
generic under EPA’s definition, in that an actual risk determination
is not a function of attributes of the specific system being assessment.
 A generic application requires that the model parameters be the same
for all model calculations used to derive an individual LOCMEI (i.e.,
model simulations for all the cosm studies) and for any application of
this specific LOCMEI (i.e., model simulations of field exposures of
interest).  This does not mean that only one set of parameters is ever
used, which an EPA comment in response to a question on May 12 might
have otherwise implied.   Rather, for uncertainty or sensitivity
analyses, the parameters sets are varied to produce alternative LOCMEIs
and EPA analyses did this in a similar fashion to the analyses of
Syngenta. 

(2) Modifications of CASM Parameterization for Uncertainty
Analyses/Probabilistic Application

The Syngenta presentation discussed variation of model parameterization
with respect to toxicity relationships, physicochemical exposure
conditions, and bioenergetics parameters, including consideration of
relevant ranges for these factors and correlations among certain
factors.  With respect to this, they were consistent with the strategy
already described by EPA on page 50-51 of the briefing document:

“For final application of either of these models, estimates of
uncertainties for the risk factors due to model parameterization would
be desired.  This would be accomplished by repeated evaluations of the
risk factor using random assignments of model parameters from
uncertainty distributions assigned to each parameter (Monte Carlo
analyses).  This requires identification of each model parameter to
which risk factors are sensitive, specification of an uncertainty
distribution for the parameter, and consideration of the correlations
among the distributions for different parameters.”  

In fact, EPA also provided an example uncertainty analysis for
CASM-based assessments in the briefing document using the same general
approach to parameter variation as Syngenta, although it was explicitly
stated that this analysis was incomplete because EPA felt that
uncertainty had been demonstrated to be large enough not to warrant
finalizing this effort.  Whether the large sensitivity of CASM-based
risk assessments to its parameterization contraindicates CASM’s use is
in fact the central point at issue, not the general approach to
modifying parameters to assess model variability.  

(B) EPA Questions Regarding Syngenta Probabilistic CASMATZ2 Analyses

(1) Suitability of Distribution Comparisons 

The Syngenta probabilistic CASM method compares the probability
distribution of LOCMEI s (for 200 different CASM parameterizations
applied to cosm studies consistent with EPA approaches) to the
probability distribution of MEIs for a field exposure of interest for
the same 200 different parameterizations, and addresses whether the MEI
distribution is at statistically significant higher concentrations than
the LOCMEI distribution.  At first glance, this provides the desired
comparison of whether the MEI is greater than the LOCMEI, for an
aggregate comparison across multiple parameter sets.  However there are
limitations to and problems with this approach, some of which have
already been pointed out by panel members.

(a) The use of a significance test is questionable.  As pointed out by
the panel, the ability to detect distributional differences is a
function of a somewhat arbitrary decision on sample size.  Furthermore,
because the null hypothesis here is of no effect, this results in
allowing the LOC to be exceeded to some extent as a function of this
sample size.  Although the sample size is large, presumably resulting in
a low detection level, it still must be asked whether this might be
partly responsible for Syngenta differing with EPA regarding whether the
LOC is exceeded at some sites.  This factor should be one of several
examined in better evaluations of these differences.  In the end,
because these are model-generated results, sufficient runs should be
done so that the distributions are adequately defined for direct
comparison, without significance testing.    

(b) The data here are inherently paired (i.e., for a particular
parameter set, the MEI is compared to a specific LOCMEI).  Rather than
comparing distributions as if arising from independent sampling, a
single distribution should be derived (some measure of the difference
between the MEI and the LOCMEI, as already suggested by one panel
member).  The question would then be whether this difference is greater
than zero.  Furthermore, such a distribution would have some useful
information not evident in looking at two separate cumulative
distributions, in that it would show whether the MEI is sometimes above
and sometimes below the LOCMEI, which EPA analyses showed can be the
case and is an important element of the method uncertainty.

(c) If CASM inferences about the variability arising from
parameterization options are to be considered accurate, then a major
question is where within an uncertainty range to position risk
decisions.   Simply determining whether a shift of these distributions
exists, although probabilistic, does not provide the needed information
on uncertainties to address this question.  The required information on
risk uncertainty could be obtained by looking at the distribution of the
paired results.  Even better information would be obtained if results
were expressed in terms of the risk factors used in the EPA analyses.

(2) Adequacy of  parameter variations

  

Syngenta criticized EPA’s sensitivity analysis as incomplete (in not
looking at correlations, except for assuming strong correlations among
nutrients), and presumably thus not adequate to justify our conclusion
that CASM’s sensitivity to parameterization is too large.  EPA did
concede that correlations were not included and that not all the factors
were considered that were intended for a complete uncertainty analysis. 
However, the point of our analysis was whether enough sensitivity was
demonstrated so that this complete analysis was not warranted.  Our
results for the effects of toxicity parameters (which are not correlated
to other parameters), the large effects of other parameters that would
only be partly mitigated, if at all, by correlations, and our
uncertainty analysis did demonstrate high enough sensitivity to be of
concern, even if the analysis was not complete.  The more complete
analysis by Syngenta confirm our conclusions about sensitivity.  For
example, in Syngenta’s slide 38, the distribution of LOCs veries from
2 to 65, and even the 10th to 90th percentile differ by more than
10-fold.  

Because Syngenta’s method does require providing comprehensive
parameter variation, the adequacy of this must be examined, which thus
far are not available.  In general, does Syngenta’s method include all
the important factors, correlations, and ranges?  In particular, EPA did
identify some needs regarding parameter variation.  For example, was the
exposure start date for simulations using the cosm exposures included? 
Was the possible interaction between temperature, start date, and
community bioenergetics noted in EPA’s talk addressed?  Is the
nutrient issue (modeled versus fixed) addressed?

(3) Validity of CASM inferences about parameter effects on atrazine
assessments 

Although CASM includes various accepted formulations for certain
ecological processes, if this large variation of results from CASM is to
be accepted, the validity of the net overall CASM predictions should be
subject to scrutiny.  How credible and accurate are CASM’s predictions
about the effects of varying certain parameters?  For example, EPA noted
the large sensitivity to lower temperature.  Even if certain CASM
equations include a temperature effect, are all important implications
of temperature (and some of the parameter interactions noted above)
included so that the net effect of temperature can be defended by EPA? 
Upon what basis can we validate and defend such results? 

(4) Implications of variability to risk determinations

Assuming the variability of results from CASM with regard to parameter
selection are adequately valid, there are substantial implications to
risk assessments that were not recognized in Syngenta’s probabilistic
CASM analyses.  The first question is, if risk does vary this much among
different systems, can a generic application be justified –
shouldn’t the lower or higher risk at different sites be addressed if
CASM’s inferences are true?  And if a generic application is still
used, the variability of CASM’s results de facto become part of the
uncertainty, which is not adequately characterized by the simple
comparison of probability distributions in Syngenta’s methodology.  If
a median result within this uncertainty is applied, this provides a low
level of protection to a large number of sites.  EPA must consider where
risks are positioned in this uncertainty to meet risk management
decisions and this methodology must provide better information to
support such considerations.  To protect a large majority of sites would
require more conservative LOCs.  How should the inferences from CASM be
treated in this regard?

In concept, there is merit to using CASM to consider how various
community processes might affect risks of atrazine.  However, the use of
the cosm data already account for community processes in the absolute
effects.  As used in the general methodology, only information from CASM
regarding extrapolations among exposure time series, not the absolute
effects, is actually used.  To justify the use of CASM, better
demonstration is needed that this information better informs the risk
assessment, especially to justify the considerably greater complexity,
implementation difficulties, and uncertainties associated with a generic
application of CASM.