Endocrine 

Policy 

Forum

November 25, 2014

Docket Identification Number EPA-HQ-OPP-2014-0614

Dr. Fred Jenkins

Office of Science Coordination and Policy (7201M)

Environmental Protection Agency

US EPA Headquarters 

1200 Pennsylvania Ave., NW

Washington, D.C. 20460-0001

Posted to Federal eRulemaking Portal: http://www.regulations.gov

RE: EPF comments submitted with regard to the FIFRA SAP on integrated 

bioactivity and exposure ranking

Dear Dr. Jenkins:

The Endocrine Policy Forum (EPF)1is committed to assisting its members in complying 

with EPA’s Endocrine Disruptor Screening Program (EDSP). EPF has focused on 

complying with the EDSP List 1 Tier 1 Test Orders and is preparing for subsequent Tier 

2 testing as necessary. The EPF supports the development and use of scientifically sound 

methods for priority setting for the EDSP. Herein we consider and provide constructive 

feedback on the EPA’s White Paper developed for public release and SAP review, 

“Integrated bioactivity and exposure ranking: A computational approach for the 

prioritization and screening of chemicals in the Endocrine Disruptor Screening Program” 

(hereafter “SAP White Paper”).

At the conceptual level, we support EPA’s data-driven approach to employ scientifically 

valid methods to combine high-throughput (HTS) bioactivity data via modeling, convert 

in vitro activities to systemic exposures using high-throughput toxicokinetics (HTTK),

and then to compare these data to high-throughput exposure (HTE) predictions to derive 

integrated bioactivity and exposure ranking (IBER) values to aid in the prioritization of 

 

1 The Endocrine Policy Forum is a consortium of CropLife America and American Chemistry Council 

members and non-member stakeholders formed to address the aggregate needs of the group with respect to 

technical guidance and regulatory policy in response to the implementation of the EPA Endocrine 

Disruptor Screening Program. Working with the EPA, our goal is to maximize the efficient and effective 

use of the Agency's and our members' time and resources.
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 2

chemicals for screening under the EDSP program. We generally understand the EPA 

approach to determine the estrogen receptor area under the curve (ER AUC), androgen 

receptor area under the curve (AR AUC), and derivation of the IBER. However, the 

information EPA released in its white paper, and appendices, is insufficient to support an 

independent review. Specifically, EPA's models currently lack enough transparency to 

evaluate their strengths and weaknesses. The data sources, including the identification of 

the set of ToxCast data used (e.g., the public release date, R scripts, and other supporting 

information) should have been provided for stakeholders to better interpret and/or 

recapitulate these models. Further, the step-wise function of these models, and how the 

range of ER and AR AUC values relate to in vitro potency have not been adequately 

defined. These limitations suggest the utility of these models for prioritization cannot 

currently be adequately independently characterized by external scientists. Nevertheless, 

the methods are encouraging and appear to address several key challenges faced in 

interpreting HTS results for determining activity, including potential assay interference 

that may lead to different outcomes across a suite of HTS assays, including ER and AR 

assays.

In the attached set of detailed comments, primarily focused on the charge questions with 

some overarching observations, we provide analysis and perspectives on the SAP White 

Paper as well as recommendations for improving the scientific basis of priority setting for 

chemical screening utilizing high-throughput tools. The work described in the SAP 

White Paper clearly advances the science of prioritization by defining the relative 

distance between predicted exposure and predicted bioactivity, with emphasis on 

estrogen- or androgen-receptor driven bioactivity. However, in their current state, the ER 

and AR AUC models are not described in enough technical and mathematical detail, and 

as such, may appear as “black box” toxicology. In order for stakeholders to responsibly 

review these models for use in decision-making, it will be necessary to describe all terms 

in these models (e.g., sign(slope), how T-scores are applied) and the specific reasons for 

model choices (e.g., linear model). These further explanations and clarifications will be 

critical in not only advancing the science and future application of these high-throughput 

methods, but also in achieving confidence with stakeholders. We view this work as a 

conceptual advance in the science of EDSP prioritization, but future application of these 

models will require critical review by stakeholders, which is not currently possible. 

Further explanation of these models also comports with EPA’s stated objectives of 

advancing data quality and transparency. 

It is important for the SAP to provide clear direction to EPA to guide the Agency's 

research on developing and validating the data sets and prediction models. Thank you in 
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 3

advance for your consideration of our comments. If you have any questions please 

contact me at emihaich@nc.rr.com or 919-479-6947.

Sincerely,

Ellen Mihaich, Ph.D, DABT

Scientific Coordinator

Endocrine Policy Forum
Endocrine 

Policy 

Forum

Endocrine Policy Forum (EPF)

Comments on

Integrated bioactivity and exposure ranking: A computational 

approach for the prioritization and screening of chemicals in the 

Endocrine Disruptor Screening Program

(EPA-HQ-OPP-2014-0614)
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 2

Table of Contents

1. Overarching Comment........................................................................3

A. The ER and AR AUC Models of HTS Endocrine Activity....................... 3

B. Integrated bioactivity exposure ranking.................................................4

C. Application of a Scientific Confidence Framework (e.g., Cox et al., 2014).... 4

D. Abbreviated Time for Review and Analysis of EPA’s White Paper and 

Related Materials ............................................................................................. 5

2. EPF Response to Charge Questions Concerning Estrogen Bioactivity..................6

A. Charge Question 1.a..........................................................................................6

B. Charge Question 1.b........................................................................................ 10

C. Charge Question 1.c.........................................................................................12

D. Charge Question 1.d........................................................................................ 13

E. Charge Question 1.e.........................................................................................16

F. Charge Question 1.f......................................................................................... 17

3. EPF Response to Charge Questions Concerning Androgen Bioactivity................19

A. Charge Question 2.a........................................................................................ 19

B. Charge Question 2.b........................................................................................ 21

C. Charge Question 2.c........................................................................................ 22

D. Charge Question 2.d........................................................................................ 22

4. EPF Response to Charge Questions Concerning the integrated bioactivity 

exposure ranking (IBER).......................................................................................23

A. Charge Question 3.a........................................................................................ 23

B. Charge Question 3.b and 3.c.......................................................................... 23

5. References............................................................................................................. 28

6. Appendix A: Print-out of the iCSS Dashboard Data for DES............................. 30
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 3

1. Overarching Comments

A. The ER and AR AUC Models of HTS Endocrine Activity

The EPA’s data-driven model approach to combine HTS assay activity within the 

estrogen/anti-estrogen or androgen/anti-androgen pathways addresses many of the 

challenges of using HTS in vitro data as prioritization information for EDSP: inclusion of 

all orthogonal or related assay data due to the perhaps unknown limitations and strengths 

of certain technologies; development of a modeling approach that attempts to use all of 

these data (use of a penalty) with fewer, rather than more assumptions; and separation of 

cytotoxicity and assay interference from in vitro endocrine activity driven by ER or AR 

signaling. However, several areas require improvement and/or further explanation before 

the utility of the ER and AR AUC models can be independently evaluated and verified

for their fit-for-purpose use in prioritization for further EDSP screening in the EDSP21 

framework.

1. Transparency, transferability, and user-interface with the model are currently 

lacking. Specifically, the EPA has not described which release of the ToxCast 

data was used to develop the ER and AR AUC scores reported; it is inferred that 

the latest re-release of the ToxCast data was used, and these data have only 

recently been posted by the EPA (end of October 2014), and without appropriate 

guidance and time for stakeholders or independent peer reviewers on the SAP to 

review these data. 

2. The EPA has not provided enough detail of the function of the model to 

recapitulate it, nor have they provided the R scripts or a user interface which are 

necessary to objectively and independently evaluate and verify the model.

3. The ER and AR AUC models do not incorporate potential in vitro interaction with 

steroidogenesis, a mechanism that may drive apical effects on estrogen and 

androgen systems. This is a major toxicological deficiency with current use of 

these models for prioritizing chemicals that may interfere with estrogen and 

androgen signaling.

4. Explanations of how the ER and AR AUC range of values relate to a potency 

distribution for reference chemicals are currently lacking, and are necessary to 

improve understanding of how the model might actually be used for a 

prioritization task. The White Paper would be strengthened by demonstration of 
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 4

how prioritization (or the setting of a benchmark level for delineating "priority" 

from "non-priority") would be accomplished. For example for ER, use of an 

AUC = 0.7 corresponding to phytoestrogen activity or setting the IBER 

benchmark at the IBER of a phytoestrogen or a chemical of low priority would be 

helpful. The proposed cut-off for positivity of AUC = 0.1 by EPA has not been 

scientifically justified; specifically, EPA has not provided enough information 

regarding the pathway potency that might be expected to correspond to such a 

proposed cut-off level.

5. The AR AUC model is too preliminary to evaluate charge question 1d on the use 

of the model to distinguish and prioritize chemicals on the basis of in vitro

androgen bioactivity. No predictivity can be reported for the model because 

comparisons to in vivo or “true positive” data are not available. 

B. Integrated bioactivity exposure ranking 

The IBER approach represents a major advance in systematic prioritization of 

chemicals. The integration of bioactivity results with exposure information provides a 

scientifically sound approach for use in priority setting for the EDSP. The general 

concept is well-conveyed: decreased priority corresponds to increased distance between 

the upper 95% confidence limit on potential oral human exposure and the oral equivalent 

dose corresponding to the 5% bioactivity level. While such an approach can account for 

uncertainty and variability, upper bounds and lower bounds are clearly not as stable as 

central estimates. For comparison across compounds for use in priority setting, central 

estimates of both exposure and activity should be considered. The use of upper bound 

and lower bound estimates likely implies greater precision than is warranted by the 

datasets and models,

What is less obvious is how the IBER distribution will be used to establish further 

priorities for screening, as pharmaceuticals rank quite high in IBER, with a clear break 

from all other chemicals. Demonstration of low and high priority using IBER would 

improve stakeholder capability to evaluate it as a fit-for-purpose prioritization tool. 

C. Application of a Scientific Confidence Framework (e.g., Cox et al., 2014)

The EPA’s program for exposure-activity profiling includes some components of 

the Scientific Confidence Framework described by Cox et al. (2014); however, the 

program would benefit by adopting all aspects of the Scientific Confidence Framework. 

The framework is designed to document scientific confidence in high throughput 
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 5

screening (HTS) assays and subsequent prediction models. The major components of the 

Scientific Framework include: 

1) Analytical validation - the biological basis and analytical performance of each 

assay should be understood, including the chemical domain of applicability and 

assay performance.

2) Qualification – to ensure transparency, a defined algorithm for each prediction 

model is needed that includes measures of goodness-of-fit, robustness and 

predictivity, as well as model limitations. This includes characterization of 

prediction models in sufficient detail to facilitate review, reconstruction and 

independent verification of results. In the SAP White Paper, the models are not 

characterized sufficiently to allow full review and independent verification. This 

aspect of the current program requires additional work. 

3) Utilization - justifying applications where there is sufficient scientific confidence 

to support the use of the prediction model. The level of scientific confidence needed

for HTS assays and prediction models is “use dependent” (i.e., the confidence 

needed for prioritization is less than including HTS data/models in a weight of 

evidence assessment or replacing some EDSP Tier 1 assays). We understand that 

EPA is in the process of implementing this step and we look forward to the 

opportunity to review the Agency's findings. 

When available, the assay validity results, the model code, training and test sets and 

model performance results will enable independent verification and objective and 

thorough peer review. Such information should ensure that the scientific evidence 

and documentation is sufficient to support the proposed uses for EDSP prioritization. 

Unfortunately, since this information has not yet been made available by EPA, a 

thorough and independent scientific evaluation is not possible. 

D. Abbreviated Time for Review and Analysis of EPA’s White Paper and Related 

 Materials 

As with the four SAP meetings held in 2013 and the one held in July 2014, EPA has 

again provided its documents to the public only 3 weeks prior to the SAP, giving 

stakeholders only 14 or 15 days total to comprehensively review the SAP White Paper 

and develop and submit written comments. Moving forward, adequate time for public 

review and comment would result in more meaningful stakeholder involvement for the 

Agency. The short time for review is exacerbated by the “soft” re-release of ToxCast 

data at the end of October, and the use of unpublished HTTK values for ToxCast Phase II 
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 6

chemicals. The models presented incorporate new data, not all of which is publically or 

easily available, and as such the time provided to review the presented models was

insufficient.

2. EPF Response to Charge Questions Concerning Estrogen Bioactivity

Charge Question 1.a: How clearly has EPA described the computational tools in 

Section 2.1 (i.e., high throughput assays and models) used to estimate ER agonist 

and antagonist bioactivity?

EPA's impetus for the described conceptual approach to generate area under the curve 

(AUC) scores for chemicals, corresponding to a profile of potential estrogenic or antiestrogenic activity, is based on the need to employ a method that can combine results 

from different types of assays in a manner that proves useful in prioritizing chemicals for 

further endocrine screening. The general introduction of this model, and the assay data 

input for the model, are well-described. While we commend EPA for its initiative in 

developing such an approach, and believe we understand the main features of the 

modeling, several shortcomings are apparent. For example, the specific function of the 

model to generate the AUC scores, and filtering using the Z-score and T-score, requires

further description and utilities for independent experts, other agencies, the private sector, 

and SAP peer reviewers, among others, to be able to utilize and evaluate the model. The 

current presentation of the model represents an initial work-in-progress rather than a tool 

that other toxicologists and information scientists can use and explore in order to define 

strengths, limitations, and inputs that may have the most significant (or least significant) 

impacts on the model output.

Specific points for clarification that would have significantly improved presentation 

clarity include:

i. A complete and separate section to describe the Z-score approach to 

distinguishing cell stress and cytotoxicity-related assay interference (page 18 of 

111).

We commend the effort that EPA has invested toward distinguishing cytotoxicity from 

apparent estrogenic or anti-estrogenic activity, a major source of assay interference which 

hinders appropriate interpretation of prioritization information. However, the section 

describing the technical details of the Z-score approach requires the inclusion of more 

information before it can be considered a reproducible and reliable approach open to 

adaptation.
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 7

a. The collection of 35 assays in the ToxCast assay battery used to detect 

cytotoxicity and cell stress should be explicitly described in a table or 

appendix, including assay source, assay endpoint name, range of 

concentrations evaluated in screening, technology or assay platform, and 

target information. By providing specific information, others might be able to 

regenerate these Z-scores, and also suggest revision, either deletion or 

addition, of certain assays to this cytotoxicity battery from either assays that 

are already run within the ToxCast screening program, or assays currently 

outside the purview of ToxCast.

b. The global cytotoxicity median absolute deviation (MAD) calculation should 

be specifically demonstrated, perhaps in a separate appendix or via issuance of 

an R program and flat input data file (.csv) such that others could verify this 

value of 0.26 log units.

c. A visual graphic should be provided to demonstrate the distribution and 

meaning of Z-scores; i.e., a typical value range that suggested overlap with 

cytotoxicity should be defined for each assay (as illustrated for 

ATG_ERa_TRANS in a separate section of the document on page 33), or 

across all assays.

d. Example Z-score calculations should be provided.

ii. A complete table of derived numerical values, including the symbol and 

definition. Such a table would include some of the descriptions given in the text, 

and in some cases expand these descriptions. Terms such as the F matrix, Rj

vector (pages 21-22), penalty(R), etc. There are many terms to track, and 

complete definitions provided in one table, in words and mathematically, would 

assist in understanding. The Rj vector is not well-described at all; are these values 

the vector of 26 activity values (in log micromolar units) (page 23). More detail 

on these terms, and examples, would be helpful.

iii. A complete description of how the sign (slope) term is used to discount assays or 

sets of assays active at lower concentrations than the remaining assays. The 

description provided on page 22 is not adequate to understand how the data are 

being discounted mathematically. This "discounting" may significantly impact 

the predicted bioactivity.
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 8

iv. A bulleted summary or workflow diagram of the order of operations in the model. 

A bulleted summary or workflow diagram would help explain the steps, and their 

order, for the model to generate scores for use in prioritization. Expansion of 

Figure 2.7 (page 34 of 111) to include each mathematical step following 

collection of the vector of assay AC50 values would be helpful to understand for 

instance when the penalty(R) is applied in relationship to T-filtering and Z-score 

filtering. Without this logical work flow, it is difficult to recapitulate the model, 

or identify the steps with the greatest impact on the ultimate result.

v. Release of an interactive utility concurrently with the White Paper describing the 

model. It would be useful to understand how different chemical-assay inputs 

affect the model score by having a user interface, like the one described on page 

101 of 111. In the absence of the ability to interact with the model and 

demonstrate how different chemical-assay inputs affect model scores, it is not 

possible to provide a complete assessment of the strengths and weaknesses of the 

model as requested by Charge Question 1b. Thus, the SAP White Paper may be 

somewhat premature based on the progress of the model development and 

presentation.

vi. Release of R-code and sample input files concurrently with the White Paper 

describing the model. Also, EPA should concurrently release R-scripts and easyto-use flat input files to regenerate model scores, at a minimum for the reference 

chemicals, so as to enable a thorough and independent peer review. Lacking 

such information, it would be appropriate to state that the model lacks

transparency, and is more of a “black box” than a verifiable, open source 

program. With the issuance of R-scripts and files for analysis, a technical 

discourse could be developed on the validity and merit of the model. Release of 

pre-named, flat files for call-in to R would be most appropriate, as this would not 

require the installation of additional programs beyond R or experience with 

database management software, e.g. MySQL, or access to a server to run a 

MySQL database. The fewer barriers to assessment of the model, the better for 

interaction with the stakeholder community.

vii. Sample table comparing the results of the AUC model and the Uterotrophic assay 

with a better description of relative potency. Table 2.12 (page 57) is necessarily 

“blinded” for chemical name due to the draft nature of List 1 Tier 1 weight of 

evidence evaluations. However, it would be appropriate to list reference chemical 

activity in the ER AUC model versus the Uterotrophic assay outcome (either in 

the literature or run as positive controls) in order to understand the translation of 

in vitro to in vivo potency of potential estrogenic or anti-estrogenic chemicals. 
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 9

As the majority of reference chemicals used for optimization of the ER AUC 

model were “Inactive” or “Very Weak,” it is important to understand if “Very 

Weak” agonist (and perhaps at some point antagonist) potency in the model will 

ever be relevant for in vivo hazard, regardless of the level of exposure. 

viii. Avoidance of the term pseudo-receptor (page 20 of 111). The term “pseudoreceptor” introduces unnecessary confusion. In drug design, for example, 

“pseudoreceptor models” are used to combine features of receptor and ligand 

modeling techniques (Tanrikulu & Schneider, 2008). These models have nothing 

to do with interference. “Interference target,” “non-ER entrypoint” or some other 

more descriptive term would distinguish receptor-related targets from entrypoints 

into the ER pathway system that have unknown biological importance or reflect 

cytotoxicity or nonselective activity. Calling these activity targets “pseudoreceptors” gives a level of biological relevance that does not reflect current 

biological understanding of estrogen signaling.

ix. Differences and similarities with the published ER AUC model (Rotroff et al. 

2014) should be delineated in the text or in an appendix. It appears that the 

model deviates from Rotroff et al., (2014), at least in minor ways. For full 

transparency, there should be text included to explain in what ways the SAP 

White Paper model and published model may be distinct. The explanation in 

Rotroff et al. (2014) is structured very differently than that in the White Paper.

x. Explanation of next analysis steps in prioritization, e.g. “down-selection” of 

chemicals. Figure 2.7 (page 34) defines populations of chemicals based on ER 

pathway activity, followed by T-score and Z-score filtering. However, it is 

unclear what happens to each population of chemicals. For instance, what 

happens to “Active” chemicals with “Low T?” Are these chemicals of low 

concern, and as such deprioritized for further assessment? There is a lack of clear 

and specific language about what the binning demonstrated in this graph means 

for decision-making, both scientifically and in regulatory applications.

xi. Explanation of how the model was corrected to handle empirical observations, or 

what is expected from the model considering the empirical observations. On 

pages 34-35, causes of crosstalk and cross-reactivity in the ER AUC model are 

described by technology platform, e.g., ATG, NVS, and Tox21 assays in the ER 

AUC model. What is unclear is how these assertions based on analysis of assay 

data should (a) inform interpretation of ER AUC model scores, or (b) change the 

function of the model or suggest model corrections. How are these observations 
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 10

of crosstalk and cross-reactivity translated into revisions to the model or 

interpretation of the results?

Charge Question 1.b: What are the strengths and limitations of the ER AUC 

model’s ability to identify reference chemicals that include a variety of 

structures and have a wide range of in vitro ER bioactivities?

i. Strength: Inclusion of as many assay endpoints as possible to inform an estrogen 

or anti-estrogen hypothesis. Inclusion of multiple assays with inherent design 

limitations is a major strength of the approach. This helps to provide context to 

potential assay false positives.

ii. Strength: Inclusion of a systematic method for consideration of cytotoxicity and 

cell stress. The Z-score approach, or some similar approach, is absolutely needed 

to provide appropriate context for the concentrations required to elicit endocrine 

activity in vitro.

iii. Limitation: Narrow discussion (page 36 of 111) and presentation of structural 

classes evaluated with the ER AUC model. Based on the current presentation of 

structural diversity included in the chemical set used to evaluate the ER AUC 

model, it may not be possible to assess model capability to handle a diversity of 

chemical structures. Several questions arise at review of the structure class 

discussion on page 36, including: 

a. How many structure classes were assigned in the library? This

information does not appear to be provided in Appendix 3. 

b. Do the reference structures evaluated using high-throughput in vitro

bioactivity assays reflect current understanding of the structure classes 

capable of perturbing the ER pathway, i.e. based on the systems like the 

ER Expert System?

c. The AUC model seems to lead to compressed scores for substances that 

have potencies that differ by several orders of magnitude. Although the

model correctly predicts higher AUC scores for very strong agonists, 

including 17β-estradiol, diethylstilbestrol, etc., versus weaker agonists, the

scores do not reflect the vast differences in potencies. For example, the 

model provides AUC scores of 1.01 for 17β-estradiol and 0.77 genistein, 

yet in terms of EC50 values 17β-estradiol is 3-4 orders of magnitude more 

potent in ER binding assays than genistein. Furthermore, the AUC 
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 11

difference between 17β-estradiol and estrone (1.01 versus 0.987) is 

difficult to reconcile with the 2 orders of magnitude difference in EC50 

values reported for these substances typically based on ER standard 

assays. On the other hand, if the difference in AUC values for 17βestradiol versus estrone reflects two orders of magnitude difference in 

EC50 potency, then the AUC difference between 17β-estradiol and 

methyltestosterone (1.01 versus 0.44) would translate to 57 orders of 

magnitude difference in EC50 values, which is far greater than the 

approximately 6 orders of magnitude usually reported. It is thus unclear 

how potency information is conveyed by the AUC scoring scale, whether 

that scale is compressed or expanded relative to standard measurements of 

potency, and hence, whether the model provides a scientifically justified 

approach for portraying the resolution between chemicals of different 

structural class and in vitro or in vivo activity in estrogen activity assays.

These ambiguities might be resolved and the clarity of the information 

conveyed by the model bolstered by a more thorough discussion of the 

AUC scoring scale and the values produced by the different chemistries 

tested.

iv. Limitation: The current model does not include input from effects on 

steroidogenesis pathways. Perturbation of metabolic pathways involved in steroid 

hormone production is not currently included in this model, and it is unclear how 

they will be incorporated. As this represents a major node for xenobiotics to 

modulate estrogen pathways, this is a deficit in the current model that should be 

addressed in subsequent iterations of this general approach. Reference chemicals 

known to perturb estrogen pathways via effects on steroidogenesis should be 

evaluated; thus, any new steroidogenesis model should intersect with estrogen and 

androgen models, rather than being distinct. The Future Directions section of the 

White Paper mentions modeling of the steroidogenesis pathway using a “variant 

of the EDSP Tier 1 T47D guideline assay” (page 99). This may be an error, as 

there is not a T47D guideline assay for steroidogenesis; rather, we believe the 

EPA meant the H295R steroidogenesis assay (OCSPP 890.1550).

v. Limitation: The assays generally lack metabolic capacity; therefore, metabolic 

activation/detoxification are not factored into the scoring system. Some 

substances administered by the oral route may be essentially fully inactivated 

upon absorption, and thus the AUC results, if interpreted as an indicator of 

relative potency in vivo would greatly overestimate the potency of such a 

substance. The opposite would hold for a substance that requires metabolic 

activation to form a moiety that binds to the receptor. 
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 12

Charge Question 1.c: EPA used data from published in vivo studies that are 

methodologically consistent with EDSP Tier 1 guidelines to evaluate concordance 

between ER AUC model scores of in vitro bioactivity and the in vivo uterotrophic 

response studies (Section 2.2.1). What are the strengths and limitations of the 

curation methods and quality standards used for evaluating published in vivo 

studies?

i. Limitation: A thorough description of the test article or chemical administered 

should be a requisite for inclusion in this literature curation of Uterotrophic 

assays. The availability, or unavailability, of any analytical chemistry on the 

diet or administered dosing solution should be noted. A lack of information 

regarding test substance purity and source should decrease the quality rating 

of the study, and should be considered when systematically reviewing the 

validity of any study. Review or reporting regarding test substance purity is a 

primary element of the ToxRTool rating for the systematic review of the 

literature (https://eurl-ecvam.jrc.ec.europa.eu/about-ecvam/archivepublications/toxrtool). GLPs also require that the dose preparations be 

analyzed for composition to validate actual doses administered. We suggest 

that NICEATM attempt to harmonize the systematic curation of the literature 

with ToxRTool to improve international acceptance of this work.

ii. Limitation: Minimum endpoints included in acceptable studies should be clearly 

specified. Minimum endpoints should have included body weight so that both 

relative and absolute uterine weight can be used for evaluation of the assay 

findings. Systemic toxicity, baseline husbandry and experimental conditions 

can influence endocrine function in the estrogen, androgen and thyroid 

systems; thus, sufficient information should be provided to help distinguish

primary endocrine-mediated effects from indirect, secondary effects due to 

alterations in other systems/organs. Also, description of the uterine weights, 

including whether or not the uterus was wet or blotted at necropsy, should be 

annotated. 

iii. Limitation: Criteria for determination of study outcomes should be clearly 

specified. The curation methodology should also include a complete 

description of the criteria for different study outcomes, i.e., positive or 

negative, or equivocal. Were the conclusions of the individual publications 

used, or were the results evaluated by a reviewer using the predetermined 

criteria established in the OCSPP 890.1600 test guideline? Did the reviewers 

evaluate the acceptability of vehicle control values set forth in the OCSPP 
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 13

890.1600 test guideline? Were there limits on the acceptable coefficient of 

variation (CV) values for given endpoints? 

iv. Limitation: Verification of laboratory proficiency. The OECD uterotrophic assay 

test guideline requires that each laboratory conducting the uterotrophic assay 

demonstrate proficiency either by periodically conducting the assay with 

reference compounds (every 6 months) or by running a concurrent positive 

control in each assay. The laboratory is also required to conduct a baseline 

positive control study with a minimum of 4 doses. 

http://www.epa.gov/endo/pubs/uterotrophic_OECD_guideline.pdf

v. Limitation: Specification of the birth date and age of the animals used in the 

immature assay. The OECD test guideline requires that for the uterotrophic 

assay with immature animals the day of birth must be specified. Furthermore, 

the OECD test guideline requires that dosing begin immediately after weaning 

(e.g., postnatal day 18 (with the day of birth being postnatal day 0), and 

completed on postnatal day 21. The test guideline states that dosing should 

cease prior to PND 25 because, after this age, endogenous estrogen levels 

begin to rise leading to an increase in uterine weight and an increase in the 

group standard deviations. 

It is unclear how consideration of these limitations may have affected the summary 

results presented in Table 2.7 (page 45 of 111).

Charge Question 1.d: Based on all the data presented in Section 2 on ER AUC 

model performance including characterization of reference chemicals, and 

concordance with in vivo uterotrophic results, what are strengths and limitations of 

using the ER AUC model to distinguish and prioritize chemicals based on potential 

estrogen bioactivity?

i. Strength: The ER Pathway model performance relative to Uterotrophic reference 

chemicals. A clear strength of this approach is its predictive value; from 

Table 2.9, false positive and false negative rates were extremely low using 

reference chemicals, with excellent sensitivity and specificity scores. 

However, it should be noted that this model will be improved when the highthroughput assay battery is improved via addition of more metabolic capacity. 

The importance of amending the high-throughput battery to include 

metabolism as an input for this model cannot be overstated (as mentioned on 

pages 51 and 54 of the White Paper).
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 14

ii. Limitation: Lack of consideration of a relative potency cut-off and uncertain 

potency value scale. For chemicals with an ER agonist AUC score equal to 1, 

LEL values for Uterotrophic reference chemicals ranged 5 orders of 

magnitude (Figure 2.14, page 52 - of note, the Figure legend should be revised 

to appropriately indicate that the ER AUC scores are on the X-axis). For 

high-level binning of chemicals, compression of the ER agonist AUC scores 

into a smaller window, with a threshold of 0.1, may be appropriate. For 

prioritization of chemicals for further testing, it may be difficult to accurately 

distinguish potent estrogenic activity from relatively weak estrogenic activity. 

Thus, the actual task of prioritizing hundreds or thousands of chemicals for 

further screening using the ER AUC model may represent a highly sensitive 

approach, but not a very specific one with respect to the potential ER- or ARmediated responses of a substance in in vivo models. Further, it may be 

appropriate to consider a relative potency cut-off, i.e. relative potency below 

some value may not be of any concern based on receptor biology (Borgert et 

al. 2013). A demonstration of how to benchmark the model results to 

chemicals with in vitro estrogenic activity that do not demonstrate in vivo 

activity (e.g., no activity in the uterotrophic assay) based on low potency (high 

Kd with ER) should be included. 

a. The importance of this limitation can be appreciated from considering an 

interesting example. Potent androgens have been used as reference 

chemicals to validate various assays for detecting potential estrogenic 

activity, including validation of estrogen receptor transactivation assays 

under ICCVAM and the current estrogen expert system under ToxCast. 

Rough estimates of relative potency can be derived from these as well as 

the published literature, a few examples of which appear in the following 

table.

Reference Chemical Relative Potency 

Estimate1

Assay / [Citation]

Estriol 6.67E-02 ERTA(α)

[Chen et al. 20043]

2-OH-E2 2.00E-02 ERTA(α)

[Chen et al. 20043]

Testosterone 2.00E-05 ERTA(α)

[Chen et al. 20043]

Testosterone 7.01E-06 BG1-Luc ERTA Mean 

EC50

[ICCVAM, 2011]
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 15

19-Nortestosterone22.06E-06 BG1-Luc ERTA Mean 

EC50

[ICCVAM, 2011]

Methyltestosterone 5.24E-06 BG1-Luc ERTA Mean 

EC50

[ICCVAM, 2011]

Tamoxifen 5.05E-05 BG1-Luc ERTA Mean 

EC50

[ICCVAM, 2011]

Genistein 4.00E-04 ERTA(α)

Ranhotra et al., 2005

Genistein 1.13E-05 BG1-Luc ERTA Mean 

EC50

[ICCVAM, 2011]

Estrone 1.57E-02 BG1-Luc ERTA Mean 

EC50

[ICCVAM, 2011]

Ethinyl Estradiol 4.8E-01 BG1-Luc ERTA Mean 

EC50

[ICCVAM, 2011]

DES 1.63E-01 BG1-Luc ERTA Mean 

EC50

[ICCVAM, 2011]

1Mean EC50 Estradiol-17 ÷ Mean EC50 of Test Chemical

2

non-aromatizable androgen

3Data derived from published graphs

From this table, it is apparent that in vitro potencies relative to estradiol-17

range 6 orders of magnitude. The potencies of the two aromatizable 

androgens, testosterone and methyltestosterone, are at the extreme low end of 

this range and no more potent than the non-aromatizable androgen 19-

nortestosterone, indicating that the potency of these steroids at the estrogen 

receptor is not due to a metabolic transformation to an estrogenic molecular 

structure. These and/or similar androgens are weakly uterotrophic; however 

their uterotrophic activity is blocked by anti-androgens but not by antiestrogens, indicating an anabolic effect on uterine tissue mediated via uterine 

androgen receptors (Schmidt et al., 1976; Schmidt & Katzenellenbogen, 1979; 

Beri et al., 1998). Genistein is one to two orders of magnitude more potent 

than the androgens; the endogenous estrogens estrone and estriol are two to 

three orders of magnitude more potent than the phytoestrogens, and; the 

synthetic pharmaceutical estrogens ethinyl estradiol and DES are one order of 

magnitude more potent still. In contrast, Appendix 3 of the White Paper lists 
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 16

estrogen agonist AUC values that differ by less than 3-fold for these or similar 

chemicals, despite much lower AUC values for other chemicals. 

Reference Chemical AUC

Genistein 0.77

Diadzein 0.623

Estradiol-17 1.01

Estrone 0.987

DES 1.01

17-methyltestosterone 0.441

From these values, the AUC methodology seems to suggest little potency 

difference between the primary endogenous estrogen, potent synthetic 

estrogens, phytoestrogens that are marginally active in vivo, and androgens 

that are not estrogenic in vivo. It is thus unclear whether, and if so how, the 

AUC methodology would prioritize substances with "true" potential 

estrogenic activity differently from "true" non-estrogens or low potency 

estrogens for further screening. 

Charge Question 1.e: Based on all the data presented in Section 2 on ER AUC 

model performance including characterization of reference chemicals, concordance 

with in vivo uterotrophic results, and comparison with Tier 1 assay endpoints, what 

are strengths and limitations of the ER AUC model to contribute to the weight of 

evidence determination of a chemical’s potential estrogen bioactivity?

i. Strength: ToxCast bioactivity data, and the ER AUC model, should be considered 

in the weight of evidence for potential estrogen bioactivity because these data 

appear to inform relevant molecular-initiating event hypotheses. In principle,

high-throughput bioactivity data in general should be considered in any 

weight of evidence approach. However, as noted above, we have concerns 

about the AUC methodology and the lack of transparency of model inputs, 

which, at the present time, prevents independent verification. Furthermore, it 

is important to maintain context for these data as screening level information: 

higher levels of information, e.g. in vivo studies, should generally be 

considered more relevant for potential endocrine activity if these data are 

available.

ii. Limitation: Confusion regarding how shifts on the x-axis (potency shifts) affect 

the overall AUC score. It is not clear if two equivalent AUCs, one AUC 

derived from a right-shifted dose-response curve and one AUC derived from a 

left-shifted dose-response curve, would result in the same AUC score. This 
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 17

should be explained explicitly in the SAP White Paper. If these two AUCs 

would result in the same score, then this presents a limitation to consideration 

of relative potency using this method. Using the AUC method would appear 

to conflict with a prioritization method that ideally prioritizes chemicals with 

greater potency at this receptor over chemicals that may act with low potency 

if at all.

iii. Limitation: Transferability of the method. Until stakeholders can interact with 

and modulate the results of the ER AUC model, it is not transferable enough 

to apply to weight of evidence determinations. 

iv. Limitation: It is unclear how an incomplete set of assay data would be used with 

the ER AUC model or independent of the ER AUC model. As no single assay 

consistently drives the ER AUC score, it would appear that it is difficult to 

parse the assay battery down to a fewer number of assays. Thus, how would 

an incomplete high-throughput screening battery for a chemical be handled? 

Can inputs for some subset of the assays be used with the ER AUC model, or 

would a separate consideration of these data be needed for a weight-ofevidence determination? What is the forward applicability of this model in 

circumstances in which not all of these assay data are available for a 

chemical?

Charge Question 1.f: Based on all the data presented in Section 2 on ER AUC 

model performance including characterization of reference chemicals, concordance 

with in vivo uterotrophic results, and comparison with Tier 1 assay endpoints, what 

are strengths and limitations of using the ER AUC model to substitute for EDSP 

Tier 1 ER binding, ER transactivation, or Uterotrophic assays for the purpose of 

characterizing a chemical’s potential estrogen bioactivity?

i. Combined Strength and Limitation: Though there was a high degree of 

concordance between the ER AUC model and the Uterotrophic results, the

extension of the model beyond the reference chemicals focused predominantly 

on chemicals with negative Uterotrophic results (Table 2.12). More 

confidence in the model predictivity would result from consideration of a 

higher number of chemicals which demonstrated some estrogenic activity in 

vivo. 

ii. A notable exception is found in the mention of a single false negative 

substance, octamethylcyclotetrasiloxane, known as D4. The White Paper 

(page 51) carefully notes the possible reasons for the apparent lack of 
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 18

detection in the ER AUC model, including the possibility that due to the high 

volatility of D4, test solutions contained insufficient concentrations of the 

chemical to elicit a response. If this occurred, the result would indicate a true 

false negative. Also mentioned is the possibility that D4 has insufficient 

potency to exhibit estrogenic potential in this system. Here again, it is critical 

to distinguish between in vitro activity as identified by the computational 

system from biologically relevant potency (e.g., uterotrophic response). 

D4 has been evaluated in two-generation reproduction studies considered to

be definitive Tier 2 tests per the EDSP (Dow Corning, 2001). In such 

bioassays, D4 fails to elicit a pattern of responses indicative of estrogen 

agonism. Consistent with this lack of clear estrogenic effect, D4 exhibits very 

low potency in sensitive in vivo and in vitro screening-level assays for 

detecting estrogenic activity, the results of which are summarized in the 

following table.

Assay Positive 

Control

[Positive 

Control]

[D4 at 50% 

response]

Citation

RUA (F344) 

Weight

EE 2.6E-07 McKim et al. 

2001

RUA (F344) 

Weight

DES 5.1E-07 McKim et al. 

2001

RUA (SD) 

Weight

EE 1.71E-06 McKim et al. 

2001

RUA (SD) 

Weight

DES 1.85E-06 McKim et al. 

2001

RUA (SD) 

Weight

Coumestrol 2.17E-02 McKim et al. 

2001

MUA (B6C3F1) EE 6.0E-06 

(Weight)

2.5E-06 

(Peroxidase)

He et al. 

2003

ERTAα 17β-E 5.0E-06 Quinn et al. 

2007
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 19

ERBAα EE 1.14E-04 Quinn et al. 

2007

ERBAα DES 7.4E-04 Quinn et al. 

2007

ERBAβ EE, DES 0 Quinn et al. 

2007

RUA: Rat Uterotrophic Assay

MUA: Mouse Uterotrophic Assay

ERTA: Estrogen receptor transactivation assay

ERBA: Estrogen receptor binding assay

Assuming that D4 was present in the test solutions near nominal concentrations, the 

failure of the computational system to indicate an estrogenic “hit” for D4 would be 

consistent with its lack of in vivo estrogen agonism in the 2-generation reproduction 

studies and its very low potency measured in other in vitro and in vivo screening 

assays. In that case, the term “false negative” may be misleading, as the 

computational system may be correctly distinguishing a chemical that does not 

warrant further screening, much less testing, based on potential estrogenic activity. 

Although androgenic potential was not addressed in the White Paper, D4 also lacks 

any detectable activity in the Hershberger assay (Quinn et al., 2007).

3. EPF Response to Charge Questions Concerning Androgen Bioactivity

Charge Question 2.a: How clearly has EPA described the computational tools in 

Section 3.1 (i.e., high throughput assays and models) used to estimate AR agonist 

and antagonist bioactivity?

Many of the same recommendations and questions are relevant in the AR AUC section as 

cited above in the ER AUC section (1a). 

One point to note, there are some clarifications required in the document that avoid 

confusion. For example:

 The EPA should cite which version of ToxCast was used for the data included in 

this white paper. Given that the ToxCast and ExpoCast are still undergoing 

updates, it is important for stakeholders to refer to the correct version of these 

databases if trying to recreate or verify analyses by the EPA.

 
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 20

 Table 3.1 (page 62) references the 9 androgen receptor (AR) assays used for the 

ToxCast analysis. This list includes two AR binding assays – NVS_NR_hAR 

(human androgen receptor) and NVS_NR_rAR (rat androgen receptor); however, 

on page 67, the legend for the figures lists the two AR binding assays as 

NVS_NR_hAR (human androgen receptor) and NVS_NR_cAR (chimp androgen 

receptor). ToxCast phase 2 contains data on all three AR binding assays, so it is 

unclear which two assays were used in this analysis and why the third assay was 

discarded. 

 It appears that ToxCast Phase 3 data may have been used in the EPA’s analysis. 

For example, for the reference chemicals, diethylstilbestrol is included as an AR 

antagonist (page 63); however, the two antagonist assays (Tox 21_AR betalactamase antagonist reporter gene and Tox21_AR luciferase-MDAKB2 

antagonist reporter gene) were negative for diethylstilbesterol in ToxCast Phase 2 

(see Appendix A). While it is commendable that the EPA has used the most 

recent data and continues to refine their approach, it would be useful to cite the 

version of ToxCast used in this SAP White Paper. This poses a problem for 

stakeholders as ToxCast Phase 3 was released very recently and is not yet 

available in the CSS Dashboard. Thus, the data is not yet in a “user friendly” 

format, making it difficult for outside experts from stakeholder groups or 

independent peer reviewers to verify the data which were the basis for the 

analyses. 

 In Table 3.3 (page 69 of 111), three colors are used in the “AUC agonist (R1)” 

and “AUC antagonist (R2)” columns (green, yellow and red), but there is no 

description in the table legend to disclose the significance of these colors. Red 

appears to indicate negative (inactive) responses, but some AUC = 0 values are 

shaded yellow (yellow includes AUCs from 0 – 0.35; whereas green indicates 

higher AUC values from 0.55 – 1.09).

 In some places in the document, the EPA states (p. 69) that “Overall, there were 

20/1855 chemicals with AR agonist (R1) AUC ≥ 0.1 and 93 chemicals with AR 

antagonist (R2) AUC ≥ 0.1”; however, in other parts of the text (page 74), it 

states: “Chemicals in the EDSP universe with AR agonist (R1) or antagonist (R2) 

scores > 0.05 are higher priority candidates for additional endocrine screening and 

testing.” While these AUC values are not mutually exclusive, the criterion should 

be consistent and scientifically justified to warrant additional screening and 

testing. If these AUC values are referring to different aspects of analysis (i.e., the 

text was meant to refer to two different AUCs), the rationale for using two 

different criteria should be clearly presented.
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 21

 The figure legend for Figure 3.5 (page 75) references “AR agonist (R1) AUC”, 

when the graph appears to illustrate “AR antagonist (R2) AUC.” There is also a 

typographical error on the AUC score in this legend. 

Charge Question 2.b: What are strengths and limitations of the AR AUC model’s 

ability to identify reference chemicals that include a variety of structures and have a 

wide range of in vitro AR bioactivities?

One of the strengths of the EPA’s approach is the inclusion of multiple methods to 

differentiate positive AR-related responses (agonism or antagonism) from assay 

interference in the AR-related in vitro assays (i.e., false positive results due to crossreactivity, cell stress, etc.). This is particularly important with the loss-of-signal responses 

in the antagonist assays, which are sometimes due to cytotoxicity. Appendix 7 was very 

helpful in clarifying the AR results for the reference chemicals as well as identifying 

sources of interference. 

A current limitation to the predictive capability of the AR AUC model is a lack of assay 

biomarkers of steroidogenesis perturbation. Modulation of AR bioactivity is not the only 

mechanism by which xenobiotic exposure may result in apical changes in whole animal 

assays like the Hershberger assay or multigeneration reproduction studies. In the event 

these data are available for a chemical subject to screening, whole animal data with 

endocrine function should be considered more biologically-relevant.

Currently, the reference set of chemicals is too small to build sufficient confidence in this 

approach for the three goals stated by the Agency. Additional reference compounds are 

needed to strengthen the conclusions. Reference chemicals must span a wide range of 

potencies and include a greater number of inactive compounds. Figures such as those in 

Appendix 7 would be useful for future reference compounds too. 

The AR AUC model as presented represents a very encouraging but incomplete effort. 

Without model performance definitions, i.e., representation of true positive, false 

positive, true negative, and false negative findings when compared to the Hershberger 

assay, the model is not in a state that can be fully evaluated for its predictive capability. 

Of the reference chemical set, only three were completely inactive, and two of these three 

were phthalates (same chemical class). A greater chemical diversity should be 

represented in the reference set of chemicals before determination of model success in 

delineating a wide range of in vitro bioactivities.
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 22

Charge Question 2.c: EPA plans to use data from published in vivo studies that are 

methodologically consistent with EDSP Tier 1 guidelines to evaluate concordance 

between AR AUC model scores of in vitro bioactivity, and the in vivo androgenic and 

antiandrogenic responses (Section 3.2.1). What are strengths and limitations of the 

planned curation methods and quality standards for evaluating published in vivo 

studies?

The inclusion of published studies will be beneficial to the EPA’s goal of planning to 

replace in vivo Tier 1 EDSP assays with ToxCast-based in vitro assays. EPA should 

consider using a data reliability tool (e.g., ToxRTool) for a systematic evaluation of 

publications prior to their inclusion in the analysis. Confirmation of chemical identity 

and purity is critical as well as dose concentration confirmatory analyses for in vivo 

studies.

As part of this process, the EPA should pay close attention to systemic toxicity or 

methodological approaches that could jeopardize the relevance and/or reliability of the 

data. Also, as mentioned in the text, the EPA should conduct the analysis (in vitro vs. in 

vivo AR data) using chemicals that span a range of potencies. In addition, it is critical 

that the EPA include more than three inactive compounds in their analysis. Sufficient 

sample sizes are critical to maintain variance within a reasonable range.

In addition, page 73 states that “Tier 1 in vivo Hershberger Assay (OCSPP 890.1400; 

OECD no. 441) is designed to detect androgen agonist and antagonist activities indicated 

by any change in five accessory sex tissue weights." Generally, the Hershberger assay 

requires statistically significant changes in two or more accessory sex tissue weights and 

the appropriate direction change in all accessory sex tissues for the assay to be deemed 

positive. 

Other applicable comments appear in response to charge question 1.c above. The same 

deficiencies in the curation method are present for the approach to the Hershberger assay 

curation project. 

Charge Question 2.d: Based on the data presented in Section 3 on AR AUC model’s 

performance, what are strengths and limitations of using the AR AUC model to 

distinguish and prioritize chemicals based on potential androgen bioactivity?

As stated above for ER AUC outputs, there are some questions about relative potency 

with respect to AR AUCs. The order of AUC Agonist (R1) (or antagonist (R2)) listed in 

Table 3.3 (page 69) is reported to reflect potency (AC50 values – page 22-23). Again, 

compression of scores seems to be operative with the AUC model. If AUC is an 

indication of potency, then it is a bit surprising that the relative potencies of the strong 

AR agonists are so similar: 17beta trenbolone (AUC = 1.09) 17-methyltestosterone 
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 23

(AUC = 1.08), 5alpha-dihydrotestosterone (AUC = 1.09), norethinodrone (AUC = 1.02) 

and testosterone propionate = (AUC = 1.01) in light of results from the literature which 

indicate significant potency differences. In addition, 17alpha-estradiol has an AUC of 

0.61, which is approximately 60% of the strong agonists. Is there any concern that the 

relative potencies will be difficult to compare because different assay systems are used?

Please refer to our previous comments, earlier in this document, on relative potency 

considerations, transferability, and questions regarding the forward utility of the model in 

cases where not all assay data are available for a particular chemical. The AR AUC 

model is preliminary and not at a stage that it can be evaluated for prioritization potential, 

particularly in the absence of model performance information.

4. EPF Response to Charge Questions Concerning the integrated bioactivity 

exposure ranking (IBER)

Charge Question 3.a: How clearly has EPA described the computational tools in 

Section 6 to develop IBER values, including modeling uncertainty and population 

variability?

i. Population variability as modeled by Monte Carlo analysis is explained well in 

the text, especially when referencing July SAP materials on high-throughput 

exposure modeling.

ii. Explanations of uncertainty and general variability (page 94) are far too limited, 

and mostly relegate these issues to “default” from another publication, or 

future research direction. Thus, these areas require further development and 

explanation, and should be considered in a draft stage.

iii. The IBER approach is generally well explained and meets the objective of 

defining the distance between potential predicted hazard and exposure for any 

given chemical. However, it is unclear if the activity comparator, 5% of the 

bioactivity dose range, is for all high-throughput bioactivity, or for just 

estrogen or androgen bioactivity. This should be clearly explained. 

Charge Questions 3.b and 3.c: What are the strengths and limitations of using the 

IBER approach to prioritize chemicals for further EDSP screening based on the 

ratio between ER/AR bioactivity dose range and the expected dose range?

i. Strength: This is an information-based prioritization approach that is generally 

transparent, and EPA is to be commended for the IBER's scientific rigor.
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 24

ii. Strength: There is a clear recognition of different scenarios that might arise from 

the QSAR models and there are statements provided for how each scenario 

will be interpreted, including one in which a chemical is set aside following 

consistent prediction as ER-inactive in a majority of all models. 

iii. Limitation: A practical demonstration of how the prioritization process would 

proceed is unclear because there is a clear separation in IBER scores between 

the top IBER values (for pharmaceuticals) and the rest of the IBER scores. 

The highest priority IBER scores were for steroids and pharmaceuticals, with 

a clear break in the distribution of these scores. 

Example from Appendix 3 of the ER IBER Scores. The steroidal estrogens cluster 

together (IBER < 1). Then, the distribution of IBER values quickly advance in 

magnitude, with daidzein, a dietary phytoestrogen, represented with an ER IBER = 90. 

Including requisite variability and uncertainty, which values in the range of 90 warrant 

consideration for further screening in the EDSP? The EPA should consider providing 

additional rationale to illustrate the prioritization process using IBER. Note: Endosulfan 

(ER IBER 2.055965) and Trifluralin (ER IBER 417.8785) were both List 1 chemicals. 

Perhaps the results of screening these compounds could be taken into consideration in 

how they would have been prioritized.

Chemical Name

ER IBER (Appendix 3), 

the top 30 values sorted 

ascending

Tamoxifen citrate 9.65E-09

17alpha-Estradiol 0.000116

17alpha-Ethinylestradiol 0.000769

Diethylstilbestrol 0.000925

Basic Blue 7 0.005917

2,2-Bis(4-hydroxyphenyl)-1,1,1-

trichloroethane 0.007686

17beta-Estradiol 0.067728

Estrone 0.093178

4-Hydroxytamoxifen 0.121667

Norethindrone 1.779332

Endosulfan 2.055965

Pyridaben 4.813092

Bisphenol AF 27.29529

Prodiamine 47.56313

17-Methyltestosterone 56.21679

Bisphenol A 62.22238

Flumetralin 86.1036

Daidzein 90.40641
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 25

2-Phenylphenol 171.9054

CJ-013610 184.9567

17beta-Trenbolone 190.023

Clorophene 323.2731

Trifluralin 417.8785

SB236057A 467.1451

2,2',4,4'-

Tetrahydroxybenzophenone 570.9987

Propylparaben 951.7878

5HPP-33 1517.262

C8-18-Alkydimethylbenzyl 

ammonium chlorides 1557.882

Chlorethoxyfos 2256.855

Ethalfluralin 2450.251

Additional example of the AR IBER score distribution from Appendix 3. There is a clear 

break between the steroidal androgens and the rest of the chemical set. Using a suitable 

benchmark chemical, EPA should consider highlighting what IBER value would warrant 

consideration for further screening in the EDSP. Note: List 1 chemicals in this table are 

Endosulfan (AR IBER 1101.125), Linuron (AR IBER 4357.942), Resmethrin (AR IBER 

5522.567), and Propiconazole (AR IBER 643056). List 2 chemicals are Triflumizole 

(AR IBER 775.7125) and Coumaphos (AR IBER 722723.4). 

Chemical Name

AR IBER 

(Appendix 3), 

the top 53 

values sorted 

ascending

Testosterone propionate 0.000101

Basic Blue 7 0.00119

17beta-Trenbolone 0.002949

5alpha-Dihydrotestosterone 0.013169

Pyridaben 0.023351

17-Methyltestosterone 0.029335

Norethindrone 0.788686

Triclosan 3.643363

17beta-Estradiol 6.614861

Emamectin benzoate 35.64566

Clorophene 41.2542

Dicloran 68.1402

17alpha-Estradiol 110.4586

2,2',4,4'-

Tetrahydroxybenzophenone 131.8482

MGK-264 168.1495

Estrone 298.3226
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 26

17alpha-Ethinylestradiol 503.5311

Triticonazole 649.4789

Propanil 661.7377

Diethylstilbestrol 770.9295

Triflumizole 775.7125

Oryzalin 844.4524

4-Hydroxytamoxifen 886.5688

Tamoxifen citrate 943.1446

Tetraconazole 954.7428

Bisphenol A 980.4187

Endosulfan 1101.125

Digoxin 1209.293

Quinoxyfen 1724.291

Napropamide 3307.225

Fenitrothion 3358.944

Fipronil 3732.393

Linuron 4357.942

Resmethrin 5522.567

Oxadiazon 22868.31

Fluoxastrobin 35728.56

Cycloheximide 41209.32

Fluazinam 47290.31

2,2-Bis(4-hydroxyphenyl)-1,1,1-

trichloroethane 48511.66

Oxazepam 84784.17

Imazalil 92725.96

Difenoconazole 100189.9

Bisphenol AF 105041.8

Fenbuconazole 139115.1

Butachlor 147440.1

Methoxychlor 203594.8

Dimethomorph 254275.1

4,4',4-Ethane-1,1,1-triyltriphenol 264370.9

Fenhexamid 335429

Fenarimol 463649.7

Phosalone 479440.4

Propiconazole 643056

Coumaphos 722723.4

a. Since an IBER for a given substance is highly dependent upon the 

exposure estimate, EPA should provide an opportunity for submission of a 
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 27

refined exposure estimate as part of OSRI, before requiring further EDSP 

screening. 

b. EPA should anticipate that as part of OSRI data, well conducted published 

and guideline studies will likely be available for many compounds. How 

will this other scientifically-relevant information be integrated with the 

HTS/AUC results for consideration in this prioritization process prior to 

the issuance of test orders? How would ER or AR AUC scores that 

conflict with guideline study information or other scientifically relevant 

information be handled?

c. One of the learnings from List 1 testing confirms negative endocrine 

activity outcomes for chemicals previously tested in higher tier assays. 

That suggests that these higher tier assays (i.e., OECD Test Guideline 416 

with enhanced endpoints, and OECD Test Guideline 443) with negative 

results should be sufficient to assess potential endocrine activity and, thus, 

chemicals with negative data from these assays should not be placed onto 

future lists to go through the screening process.

iv. Limitation: A lack of metabolism in the bioactivity assays and limited 

consideration of metabolism in reverse toxicokinetic approaches. The most 

pressing research need to improve the utility of the IBER approach is better 

consideration of metabolism in both bioactivity and exposure estimates. The 

impact of metabolic clearance in the HTTK reverse dosimetry approach is 

fairly well understood by incorporating test material loss rates determined in 

human hepatocytes (Wetmore et al. 2012). Little can be done to accurately 

determine metabolic loss in the HTS ER/AR assays that have already been 

conducted. One approach, however, to estimate metabolism in these 

bioactivity assays would be to scale the hepatocyte-based metabolic clearance 

determined in the HTTK approach to the relative CYP levels (or other 

relevant metabolic system, i.e.: UDPGT) in bioactivity assay cells vs. liver 

cells. 

v. Limitation: IBER approach is not currently transferable to stakeholders due to a 

lack of access to the ER and AR AUC models and the latest reverse 

toxicokinetic information. While the HTTK approach is well described in the 

July SAP documentation (and related publications), it relies on a fairly highend PBPK modeling tool (SimCYP) to provide probabilistic distributions of 

systemic exposure. Specific details of the PBPK model used (i.e.: model 

code) have not been published. Also, utilization of a more mainstream 
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 28

modeling application would allow for a broader implementation of this 

approach with other scientists. 

vi. Limitation: Use of the upper 95% confidence limit on potential oral human 

exposure and the oral equivalent dose corresponding to the 5% bioactivity 

level has not been shown to be superior to use of central estimates. While use 

of upper bound on exposure and lower bound on potency can account for 

uncertainty and variability, upper bounds and lower bounds are clearly not as 

stable as central estimates. For comparison across compounds for use in 

priority setting, central estimates of both exposure and activity should be 

considered. The use of upper bound and lower bound estimates likely implies 

greater precision than is warranted by the datasets and models. 

References

Beri, R., Kumar, N., Savage, T., Benalcazar, L., and Sundaram, K. (1998). Estrogenic 

and progestational activity of 7alpha-methyl-19-nortestosterone, a synthetic 

androgen. J Steroid Biochem Mol Biol 67, 275-283.

Borgert, C.J., Baker, S.P., and Matthews, J.C. (2013). Potency Matters: Thresholds 

Govern Endocrine Activity. Regul Toxicol Pharmacol 67, 83-88.

Chen, Z., Katzenellenbogen, B.S., Katzenellenbogen, J.A., and Zhao, H. (2004). 

Directed evolution of human estrogen receptor variants with significantly enhanced 

androgen specificity and affinity. J Biol Chem 279, 33855-864.

Cox, L.A., Popken, D., Marty, M.S., Rowlands, J.C., Patlewicz, G., Goyak, K.O., and 

Becker, R.A. (2014). Developing scientific confidence in HTS-derived prediction 

models: Lessons learned from an endocrine case study. Regul. Toxicol. Pharmacol. 

69, 443-450.

Dow Corning Corporation. (2001). A two-generation inhalation reproductive toxicity 

and developmental neurotoxicity study of octamethyltetrasiloxane (D4) in rats. 

Report No. 2001-10000-50855, December 2001.

He, B., Rhodes-Brower, S., Miller, M.R., et al. (2003). Octamethylcyclotetrasiloxane 

exhibits estrogenic activity in mice via ERalpha. Toxicol Appl Pharmacol. 192:254-

261.

ICCVAM (Interagency Coordinating Committee on the Validation of Alternative 

Methods), (2011). ICCVAM Test Method Evaluation Report: The Lumi-CELL® ER 

(BG1Luc ERTA) Test Method: An In Vitro Assay for Identifying Human Estrogen 

Receptor Agonist and Antagonist Activity of Chemicals. NIH Publication Number 

11-7850.

McKim, J.M., Wilga, P.C., Breslin, W.J., Plotzke, K.P., Gallavan, R.H., Meeks, R.G. 

(2001). Potential estrogenic and antiestrogenic activity of the cyclic siloxane 
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 29

octamethylcyclotetrasiloxane (D4) and the linear siloxane hexamethyldisiloxane 

(HMDS) in immature rats using the uterotrophic assay. Toxicol Sci. 63:37-46.

Quinn, A.L., Regan, J.M., Tobin, J.M., et al. (2007). In vitro and in vivo evaluation of 

the estrogenic, androgenic, and progestagenic potential of two cyclic siloxanes. 

Toxicol Sci. 96:145-153.

Ranhotra, H.S., and Teng, C.T. (2005). Assessing the estrogenicity of environmental 

chemicals with a stably transfected lactoferrin gene promoter reporter in HeLa cells. 

Environmental Toxicology and Pharmacology 20, 42-47.

Schmidt, W.N., and Katzenellenbogen, B.S. (1979). Androgen-uterine interactions: an 

assessment of androgen interaction with the testosterone- and estrogen-receptor 

systems and stimulation of uterine growth and progesterone-receptor synthesis. Mol 

Cell Endocrinol 15, 91-108.

Schmidt, W.N., Sadler, M.A., and Katzenellenbogen, B.S. (1976). Androgen-uterine 

interaction: nuclear translocation of the estrogen receptor and induction of the 

synthesis of the uterine-induced protein (IP) by high concentrations of androgens in 

vitro but not in vivo. Endocrinology 98, 702-716.

Tanrikulu, Y., and Schneider, G. (2008). Pseudoreceptor models in drug design: 

bridging ligand- and receptor-based virtual screening. Nat Rev Drug Discov 7, 667-

677.

Wetmore, B.A. et al. (2012). "Integration of dosimetry, exposure, and high-throughput 

screening data in chemical toxicity assessment." Toxicol Sci 125(1): 157-174.
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 30

Appendix A

Print-out of the iCSS Dashboard Data for DES. Accessed 11/14/2014.
EPF Comments for Bioactivity and Exposure Ranking SAP

November 25, 2014

Page 31