Page 1 of 4 12/1/14

Summary of EPA’s Responses to the June 25-28, 2013 FIFRA SAP’s Recommendations:

Tier 2 Non-Mammalian Tests

SAP comment Agency Position EPA Action

SUMMARY OF SAP COMMENTS ON STATISTICS*

General comments on statistics relevant to all assays:

Statistical methods and analyses need more detailed description. Agreed. Statistics workshop held to assist in preparing more 

statistical guidance for each of the test methods.

Additional details for randomization in all studies especially Japanese quail 

and medaka studies (as in mysid and LAGDA).

Agreed. Language in the test guidelines adjusted to help clarify.

Avoid data-driven statistics (e.g., different datasets lead to different 

analyses).

Agreed. Statistical guidance improved to avoid data-driven 

approach.

Specific recommendations:

• Protocols should include a statistical guidance manual for each 

Tier 2 test including a discussion of randomization, power 

analyses (sufficient replicates), and blinded treatments.

A statistical guidance manual is unnecessary as OECD’s 

2007 document, Current Approaches in the Statistical 

Analysis of Ecotoxicity Data: A Guidance to Application

(No. 54), is referenced and more guidance added to the 

statistical analysis section of each test guideline.

• Recommended that the selection of probability distributions be 

appropriate to the data (e.g., Continuous; discrete random; 

ordinal categorical variables) rather than transform data to be 

normal and homoscedastic. Failure to do so may result in error 

due to the tests applied by individual labs and conclusions drawn.

Agreed. Statistical analysis guidance adjusted accordingly.

• Statistical models used should be based on a priori knowledge of 

the types of endpoints measured (use ph – typically normally 

distributed, versus H+ concentration – usually long-normally 

distributed as an example).

Agreed. Statistical analysis guidance adjusted accordingly.

• Recommended using Shapiro-Wilks test for normality, but 

recognize the influence of sample size on power of the test (i.e. 

Small samples sizes are likely to fail to reject null), suggest using a 

visual review (e.g., Quantile-quantile plot of residuals) in addition 

to test, and recommend review of normality should be on fitted 

model residuals not raw data.

Agreed. Statistical analysis guidance adjusted accordingly.

• Recommended using Leven’s test of heterogeneous variances. 

When variances are unequal, the panel cautions against 

transforming data (as one might to achieve normality) as 

statistical packages can analyze data with unequal variance.

Agreed. Revised guidance.

• Caution should be taken when using non-parametric tests (e.g.,

Kruskal-Wallis alternative to ANOVA) due to low power, potential 

Agreed. A caution added to the guidance.
Page 2 of 4 12/1/14

SAP comment Agency Position EPA Action

failure to recognize assumptions of non-parametric analysis (i.e.,

Equal variances and comparable data distributions among effect 

levels), and limitations testing interactions between effects that 

can be especially important in multigenerational tests. Using 

rank-based non-parametric analyses will be further complicated 

when analyzing ordinal data (e.g. Histopathology severity grades) 

with frequent tied ranks.

• ANOVA should not be performed on ranked transformed data as 

the Type I error increases. This is particularly true when the pre 

and post-test scores are correlated, as is the case in repeated 

measures designs.

Agreed. Statistical analysis guidance adjusted accordingly.

• Recommended using confidence intervals for the mean difference 

between control and each dose level to characterize the data 

rather than post-hoc power analyses and cvs which are less 

informative. 

Agreed. The guidance adjusted to clarify reporting confidence 

intervals instead of CVs.

• Commented that the current experimental designs are underpowered for testing intended hypotheses. If sample sizes cannot 

be increased, alterations to the statistical methods and/or 

experimental design may be appropriate, based on the variability 

observed in data from earlier experiments.

Agreed. Sample sizes increased in all tests to improve power.

• Commented that the Fisher’s LDS approach to comparing 

treatment levels for main effects (when F-test rejects the H0) 

should be avoided, as should any test (e.g., Jonckheere-Terpstra) 

that is based on observed results unless the test is designed for 

sequential review.

Agreed. Fisher’s LDS not recommended.

Note: For histopathology, panel strongly recommended use of quantitative 

analysis.

Acknowledged, but not 

accepted.

A histopathology expert workshop was held to discuss 

quantitative analysis.

Comments on Japanese Quail Two Generation Test (JQQTT)

• Select a "Standard Japanese quail strain" and take efforts to 

reduce genetic drift due to continued colonization.

Generally agreed. A specific “standard” strain is not available, but a defined 

size range is recommended.

• Reduce differences in humidity, incubation temperature, and egg 

storage temperature among labs

Agreed. Recommended ranges for these conditions are provided.

• Standardize the time of day of sampling. Acknowledged, but not 

accepted.

The sampling is logistically challenging and occurs 

throughout the day. Guidance to ensure consistent 

sampling across treatments is provided. 

• Standardize light intensity and photoperiod. Agreed.

• Include blood corticosterone in response to capture and handling 

stress to evaluate potential biases due to differential response 

across groups.

Acknowledged, but not 

accepted.

Research and development are needed to standardize 

and demonstrate the utility of these measures.
Page 3 of 4 12/1/14

SAP comment Agency Position EPA Action

• Provide more specific guidance and standardization of tissue 

collection (e.g., Tissue section selected and histological analysis 

used).

Agreed. A histopathology guidance document was completed to 

provide more detailed guidance.

• Provide more detailed guidance and standardized methods of 

collecting behavioral data.

Agreed. Additional guidance provided.

• Increase in sample sizes for hormone analyses, tissue samples, 

and behavioral endpoints.

Agreed. Sample sizes increased slightly to improve power, but 

not make logistically problematic. 

• Increasing the number of replicates to improve statistical powe.r Agreed. The minimum number of pairs was increased to improve 

overall power.

• Maintaining the F2 generation. Agreed. The F2 generation is retained.

• Continuing to analyze histopathology samples from all organs 

currently identified in the JQTT with "blind" treatments.

Acknowledged, but not 

accepted.

The histopathology experts did not feel this was 

necessary, but secondary review of histopathology

readings is recommended.

Comments on Medaka Test (MMT/MRT)

• Describe randomization of fish and provide more detailed 

experimental design (e.g.,How to handle the tanks at each 

generation) to improve reproducibility.

Agreed. Additional guidance provided.

• MRT protocol is an improvement of MMT statistical power at the 

cost of reduced assessment of F0 and F2 generations.

Agreed. The medaka extended one-generation reproduction test 

(MEOGRT) was developed to provide assessment of F0 

and an option for F2, when indicated.

• Other comments covered in “general comments concerning 

statistical issues.”

Acknowledged. Additional statistical guidance provided based on 

recommendations from the statistics expert workshop.

Comments on Larval Amphibian Growth and Development Assay 

(LAGDA)

• Suggested statistical methods should be updated and described 

for each endpoint.

Agreed. Additional statistical guidance provided based on 

recommendations from the statistics expert workshop

• Correct treatment of categorical or ordinal variables. Agreed.

• Include appropriate random or repeated measures effects in 

models.

Acknowledged, but not 

accepted.

The LAGDA endpoints are not conducive to repeated 

measure analysis as collected at a single point.

• Suggested examining interlaboratory data to identify problematic 

aspects of testing methods and endpoints.

Agreed. The interlaboratory data were examined to determine 

what aspects need additional guidance

• Other comments include in “general” section. n/a

Comments on Mysid Two Generation Test (MTTT)

• A threshold (minimum) number of replicates are needed to 

continue with F1 exposures.

Agreed. The recommended number of replicates increased to 4, 

with performance criteria that 3 replicates be available in 

controls at end of treatment.

• The number of replicates may be increased by reducing the 

number of individuals/tank and increasing the number of tanks.

Agreed. The recommended number of replicates increased to 4, 

with performance criteria that 3 replicates be available in 
Page 4 of 4 12/1/14

SAP comment Agency Position EPA Action

controls at end of treatment.

• Examine effect of sample size on power using random generated 

data sets.

Acknowledged, but not 

accepted.

A power analysis was completed for the empirical data 

sets and replication already increased to practical 

limitations.

SUMMARY OF SAP COMMENTS ON HISTOLOGY

General comments on statistics relevant to all assays:

• The Panel strongly recommended use of a quantitative analysis as 

a means to enhance analytical power and consistency among 

laboratories. 

Acknowledged, but not 

accepted.

The histopathology guidance for each test was improved 

and expanded, based on recommendations provided by 

the histopathology experts from the workshop.

• Where possible, histopathology studies should be conducted 

using a blind approach coupled with morphometric analysis.

Acknowledged, but not 

accepted.

The histopathology guidance for each test was improved 

and expanded, based on recommendations provided by 

the histopathology experts.

• Use a systematic series of sections for each tissue conducted 

using analysis of sections either in a systematic or random basis. 

o In the systematic approach, every 3rd or 5th (or some 

specific number) section is reviewed. 

o In the random approach, randomization is used to select 

which sections are reviewed.

Accepted. The histopathology guidance for each test was improved 

and expanded. 

• Image analysis can be readily automated. Acknowledged, but not 

accepted.

The implication is that image analysis should be 

recommended, but the histopathology experts dismissed 

this recommendation as not needed.

• For the Larval Amphibian Development and Growth Assay 

(LAGDA), the Panel recommended objective measures such as 

thyroid area, thyroid follicular cell height, and thyroid follicular 

cell number using computer-assisted methods.

Acknowledged, but not 

accepted.

The implication is that image analysis should be 

recommended, but the histopathology experts dismissed 

this recommendation as not needed.

• Clear guidance on the selection of animals, tissue, and sections 

for histological and subsequent statistical analysis should be 

developed. 

Agreed. Additional guidance provided based on 

recommendations from the histopathology experts.

• Define a “representative” section. Agreed. Additional guidance provided to clarify.

• Increase the number of replicates to at least n=8 (quail). Agreed. Revised guideline to increase the replicates.

* It is important to note that several of the SAP’s criticisms and recommendations of tier 2 statistics were specific to the statistical analyses of interlaboratory data. The 

goal of the workshop was to develop consistent, rigorous, and appropriate guidance for statistical analyses of data for a single Tier 2 EDSP test submission (i.e., a single 

study from a specific lab).