1

February 4, 2011

Don Bergfelt

Office of Science Coordination and Policy (7203M)

Environmental Protection Agency

1200 Pennsylvania Ave., NW.

Washington, DC 20460–0001

Re: 75 FR 67964, November 4, 2010, Docket ID number EPA–HQ–OPPT–2010–

0877; Endocrine Disruptor Screening Program; Draft for Weight-of-Evidence 

Guidance Document: Evaluating Results of EDSP Tier 1 Screening to Identify 

Candidate Chemicals for Tier 2 Testing

Dear Mr. Bergfelt:

These comments are submitted on behalf of People for the Ethical Treatment of Animals and 

the Physicians Committee for Responsible Medicine whose members are national animal 

protection, health, and scientific advocacy organizations with a combined constituency of 

more than two million Americans. We share the common goal of promoting reliable and 

relevant regulatory testing methods and strategies that protect human health and the 

environment while reducing, and ultimately eliminating, the use of animals.

In the Federal Register on November 4, 2010, the Environmental Protection Agency (EPA), 

announced the availability of a Draft Weight-of-Evidence (WoE) Guidance Document: 

Evaluating Results of EDSP Tier 1 Screening to Identify Candidate Chemicals for Tier 2 

Testing to be used for evaluating results of Tier 1 screening and determining whether a 

chemical should undergo Tier 2 analysis. The purpose of this document “is to set forth some 

of the general principles, criteria, and considerations EPA generally believes to be relevant 

under a WoE approach for evaluating data submitted as part of EPA’s two-tiered paradigm for 

screening and testing chemicals for endocrine activity (i.e., estrogen, androgen, and thyroid 

hormonal systems; E, A, and T) under the EDSP (75 FR 67964).” It is also noted in the 

Federal Register that principles articulated in the draft WoE are applicable not only to 

evaluation of data from individual assays with multiple endpoints, but across the whole suite 

of assays in the Tier 1 screening battery. Additionally, these same principles will be applied 

to review of Other Scientifically Relevant Information (OSRI) submitted as potential 

substitutes/equivalents for designated Tier 1 screening assays.
2

General Comments

During the review of the EDSP Phase I and Tier 1 assays, several stakeholders expressed 

concern over a lack of clarity in EPA’s decision-making process, either for determining 

positive/negative outcomes of each Tier 1 assay, or in determining how decisions will be 

made to proceed to Tier 2 testing (available at Regulations.gov in Docket no. EPA-HQOPPT-2007-1081). As a result, in its Terms of Clearance to EPA regarding the original 

Information Collection Request, the Office of Management and Budget (OMB) requested that 

EPA “ensure sufficient opportunity prior to submission of any revision to this collection for 

public comment and peer review of the EPA tools to be developed to guide agency decisions 

on whether a chemical must proceed to Tier II, including the Weight of the Evidence 

Approach and Standard Evaluation Procedures (emphasis added).” In addition, in report 

language, the House Appropriations Committee for the Interior and Environment (FY 2010)

directed EPA to: “develop and publish criteria for evaluating the results of Tier 1 screening 

and determining whether a chemical should undergo Tier 2 analysis within one year of 

enactment. The process should allow for public input (US Congress, 2010) (emphasis 

added).” EPA’s Draft WoE Guidance Document (GD) is intended to fulfill these instructions; 

however, the content is inadequate to satisfy either the letter or intent of both OMB and 

Congress.

The draft WoE GD contains very little information that is specific to Tier 1 test results; rather, 

the draft describes the history of the EDSP, outlines WoE approaches in general and reiterates 

its definition of Other Scientifically Relevant Information (OSRI). Out of a nine page 

document, only a single paragraph relates to evaluation of Tier 1 studies specifically and 

includes the following points: 

 In general, however, Tier 1 in vivo results would carry greater weight than in vitro

results 

 Concordant effects found in multiple interrelated endpoints generally imply a 

compromise in endocrine function, in contrast to isolated or discordant effects. 

 “Generally, consistent positive or negative effects across studies and taxa increase 

confidence...” 

 “other available information may also be appropriate” and “could include consideration 

of other critical effects (non-endocrine), dose response, what is understood about the 

underlying basis (i.e., toxicity MOA) of these critical effects (i.e., non-endocrine or an 

endocrine MOA not covered by Tier 1) and their human relevance, and potential for 

exposure.” 

The draft does not describe specifically what criteria the EPA will use to determine 

whether a chemical does or does not interact with the EAT hormone system(s) or what 

information will be used to determine whether a chemical will proceed to Tier 2 testing. 

Nor has EPA provided “Standard Evaluation Procedures,” satisfying neither the intent 

nor the letter of the OMB Terms of Clearance or the FY 2010 House appropriations 

language. 
3

WoE GD Fails to Provide Stated Goal of Transparency

In a paper developed through the International Life Sciences Institute’s (ILSI) Health and 

Environmental Sciences Risk Assessment Weight of Evidence Project, Schreider et al. (2010) 

state that “[t]ransparency and documentation of the decision process are at the core of a 

credible risk assessment and, in addition, are essential in the presentation of a weight-ofevidence-based approach (emphasis added).” When describing benefits attributable to a 

WoE framework in regulatory decisions, Krimsky (2005) cites Walker’s (1996) three 

objectives of a WoE analysis: (1) it provides a “clear and transparent framework” for 

evaluating the evidence in a risk determination; (2) it offers regulatory agencies a consistent 

and standardized approach to evaluating toxic substances; and (3) it helps to identify the 

discretionary assumptions in risk determinations from experts. Unfortunately, as Krimsky 

goes on to say, most applications of WoE in support of public policy “seem to (by inference 

or lack of specification) use a process methodology that is low on transparency and high on 

subjectivity.” While it is stated by EPA in the Draft WoE that “this document provides a 

transparent scientific approach for broadly evaluating Tier 1 screening data to determine if 

additional Tier 2 testing is necessary”, and later goes on to add that for each chemical “[a] 

summary of the WoE analysis is expected to transparently state and explain conclusions”, we 

find little in the document that transparently and clearly articulates the methods EPA will use 

to determine which OSRI will be considered relevant to the objectives of the EDSP and how 

EPA will evaluate Tier 1 results to determine whether a chemical will have to undergo Tier 2 

testing. 

EDSP-specific WoE Approach is Lacking; Suggestions for Improvement

To create an objective and unbiased assessment of the hypothesis that a particular chemical is 

acting via an endocrine mechanism of action to cause adverse effects on animals in

laboratories, wildlife populations, and humans, all of the relevant information needs to be 

considered in an organized and structured manner (WHO, 2002). In the absence of a single 

definitive study, WoE analyses must rely on a rational integration of multiple sources, 

qualities, and types of data, as well as consideration of uncertainty factors and variability in 

results. As part of a general description of WoE, the WoE GD outlines how individual studies 

are analyzed, for example, by characterizing the “nature of the effects”, specificity and 

sensitivity, dose-and time-dependent changes, potency and magnitude of changes and 

consistency among assays. Also, study “quality” will be considered, i.e., studies performed 

under GLP will be given more weight. At the very least, EPA should develop a scheme for 

grading evidence similar to what the National Toxicology Program (NTP, 2009) has done in 

developing criteria to classify outcomes from non-cancer studies. NTP uses five categories or 

“levels of evidence” to describe non-cancer findings: clear evidence, some evidence, 

equivocal evidence, no evidence and inadequate study, and states the type of results that 

would characterize each level. There is nothing analogous to this in the Draft WoE GD

presented by EPA.

The GD lists a number of factors that contribute to an integrated analysis of data and 

acknowledges, in general terms, that some factors may be considered more important than 

others:
4

 Quality of data and replicability of results

 Strengths and limits of in vitro and in vivo results

 Number and type of effects; potency and magnitude of effects

 Consistency, pattern, range, and interrelationships of effects observed across 

studies, species, strains, and sexes

 Dosage, route, duration conditions under which effects occur

 Understanding of MOA and biological plausibility of responses

While the above factors represent most of those that should be considered in a WoE analysis, 

there is no description of how these general principles will be specifically applied to Tier 1 

assays. Nor is there an explanation or scheme provided which clearly describes how these 

factors will be assessed or integrated, what relevancy weight will be assigned to each, and 

how the final result “score” will be determined. Without a clear explication of how 

evidence will be “weighed” and “weighted”, we find it difficult to see how there can be

consistency between assessors or between assessments of different chemicals. Nor will it 

be clear to the recipients of test orders or the public how decisions have been made.

EPA gives some insight as to the importance of in vitro studies versus in vivo studies, stating 

that “[i]n general, Tier 1 in vivo results would carry greater weight than in vitro results 

because in vitro assays inherently lack physiological conditions associated with whole tissues 

or organs and, therefore, have nil or very limited ability to represent metabolic processes and 

pathways leading to endocrine disruption.” Will all in vivo studies be treated equally and 

always be given greater weight over in vitro studies? For example, will results from specific 

mechanistic tests such as the Hershberger and uterotrophic assays be weighted more heavily 

than if results from several in vitro tests for the same pathway concur? And will mechanistic 

in vivo tests be weighted the same as less specific tests such as the pubertal assays? Are all of 

the endpoints within each assay weighted similarly? For example, are histological changes 

weighted the same as hormone levels? How will EPA differentiate between relevant in vivo

findings and those that are not? For example, how will it be determined that a change in

fecundity seen in the short-term fish assay is due to an endocrine-related mechanism?

These are a few of the many questions that EPA has failed to answer in its Draft WoE GD 

because it has not clearly laid out the steps that must be taken in order to provide a 

transparent, clearly defined decision-making process based on specific results of Tier 1 assays 

and how they relate to an endocrine-related mode or mechanism of action. To meet this goal 

we recommend the following elements be included in the WoE guidance:

1. Clearly state the foundation of data evaluation in relation to endocrine disruption 

based on what we currently know, including relevance, applicability and limitations of 

that knowledge.

2. Define the hypotheses to be tested (e.g., estrogen agonist, estrogen antagonist, 

androgen agonist, etc.) and how each test relates to those hypotheses.
5

3. Clearly state the rules that will be followed when evaluating OSRI and its relevance to 

objectives of EDSP testing.

4. Identify and provide rationale for a priori assumptions or judgments used to weight 

data.

5. Develop an empirically-based quantitative weighting scheme to evaluate OSRI, Tier 1 

and Tier 2 data (see discussion of Calabrese et al., 1997, below).

6. Establish an explicit basis for determining cause and effect.

Calabrese et al. (1997) developed a quantitative weighting system to evaluate estrogen-related 

endocrine effects of chemicals on human health. Criteria providing the predictive framework 

included: (1) endocrine disruption as a multistage process (stages being receptor binding, 

transcription, macromolecule synthesis, cell proliferation, clinical manifestations) with effects 

on the later stages having more weight, (2) phylogenetic relatedness, (3) model system (in 

vitro versus in vivo), and (4) estrogenic potency. When developing such a weighting system, 

it is necessary to consider that scientific evidence for a hypothesis generally appears in 

gradations, and preserving the continuous nature of these data can provide more valuable 

information than turning it into a dichotomous (above or below a threshold) variable. If 

continuous data must be reduced to ordered or dichotomous judgments, then the rationale for 

doing this should be provided. In computing chemical rankings, the authors state that 

estrogenic potency reflects quantitative differences with respect to species and stage in the 

endocrine system, and, as such, was assigned greater importance than the other criteria. 

Similarly, weighting of phylogenetic factors was based on the assumption that organisms 

more closely related phylogenetically were more likely to respond in a similar manner to 

endocrine-disrupting agents than more distantly related ones. Thus, when evaluating assay

results in relation to a particular organism, a test done on a species within the same genus 

would be given much more weight and have a lower uncertainty factor, than a test done on an 

animal of the same family within an order. Using the system proposed by Calabrese et al., a 

high final value would indicate that the study used a more apical endpoint of high predictive 

value, generated data using a closely related species, was performed in a whole animal 

experimental model, and the chemical tested had a potency similar to that of the model highpotency estrogen, estradiol. While this framework may not receive universal agreement from 

experts, it clearly lays out the factors used, their relative weights, any empirically-based and 

judgment-based assumptions that were made, and the rationale behind each element of the 

framework.

In the EDSP, separate WoE assessments will be needed for determining:

a) whether a substance exhibits the potential for interaction with androgen, estrogen, or 

thyroid pathways, based on Tier 1 screening results and OSRI;

b) whether a substance should be further evaluated in Tier 2 tests, based on Tier 1 results 

and OSRI, and if so, which Tier 2 tests;

c) whether a substance exhibits adverse effects based on Tier 2 tests;
6

d) whether the adverse effects observed in Tier 2 tests are a consequence of an 

endocrine-related mode or mechanism of action, based Tier 1 results, OSRI, Tier 2 

tests, and as necessary, additional mode-of-action experiments, and;

e) whether endocrine-mediated adverse effects on humans or wildlife are possible at 

environmentally relevant exposure levels.

The predictive framework of Calabrese et al. (1997) may not be exactly applicable to

evaluating Tier 1 and 2 testing data and OSRI, but it certainly provides an example of the type 

of weighting system that is needed to ensure consistent and transparent evaluation within the 

EDSP. Krimsky (2005) notes that while there may be certain pitfalls and disagreement 

among experts as to how and what to weight in a WoE analysis, “[w]hen WoE is used 

consistently and uniformly by a regulatory body, it enables that body to develop a strong 

comparative approach for assessing the potential health and environmental effects of 

products.”

Without clearly articulated, standardized methods in place for each of the above

determinations, we do not see how rational, transparent decision-making on the part of EPA 

will be achieved. The WoE GD states that it was developed “to provide guidance to EPA 

staff and managers who will be reviewing data submitted in response to Orders for Tier 1 

screening that began October 29, 2009 under the EDSP” and to inform outside parties as to 

how the Tier 1 screening results are being evaluated. It is also expected, by EPA, to comply 

with OMB Terms of Clearance for the ICR as well as the 2010 House Appropriations Report 

language. As currently written, the WoE document fails to achieve any of these objectives.

Thank you for considering our comments.

Sincerely,

Patricia L. Bishop, M.S.

Research Associate

Regulatory Testing Division

People for the Ethical Treatment of Animals

Catherine Willett, PhD

Science Policy Advisor

Regulatory Testing Division 

People for the Ethical Treatment of Animals
7

Kristie Sullivan, M.P.H.

Science & Policy Advisor

Physicians Committee for Responsible Medicine

References

Calabrese, E.J., L.A. Baldwin, P.T. Kostecki, and T.L Potter. (1997). A toxicologically 

based weight-of-evidence methodology for the relative ranking of chemicals of 

endocrine disruption potential. Regulatory Tox. & Pharma. 26: 36-40.

Krimsky, S. (2005) The weight of scientific evidence in policy and law. Am J Public 

Health. 95 (S1): 129-136.

National Toxicology Program. (2009). NTP Non-Cancer Evaluation Criteria. U.S. 

Department of Health and Human Services. Accessed at 

http://ntp.niehs.nih.gov/?objectid=CD5E017E-F1F6-975E-7D86DAA5256F6AB8

Schreider, J., C. Barrow, N. Birchfield, K. Dearfield, D. Devlin, S. Henry, M. Kramer, S. 

Schappelle,. K. Solomon, D. L. Weed, and M. R. Embry (2010). Enhancing the 

credibility of decisions based on scientific conclusions: Transparency is imperative. 

Toxicol. Sci. 116(1): 5-7.

U.S. Congress. House Report 111–180. Department of the Interior, Environment, and Related 

Agencies Appropriations Bill, 2010. pp. 105 – 106. Available at: 

http://www.gpo.gov/fdsys/pkg/CRPT-111hrpt180/pdf/CRPT-111hrpt180.pdf 

(Accessed 10 Jan, 2011)

Walker, V.R. (1996) Risk characterization and the weight of evidence: Adapting 

gatekeeping concepts from the courts. Risk Analysis 14: 793-799.

World Health Organization. (2002) Chapter 7 of Global assessment of the state-of-thescience of endocrine disruptors: Causal Criteria for Assessing Endocrine Disruptors -

A Proposed Framework. WHO/PCS/EDC/02.2. WHO, Geneva Switzerland. 180p.