Ellen Mihaich, Ph.D., DABT

ER2 on Behalf of the Endocrine Policy Forum

Science Advisory Panel Meeting

December 2-5, 2014

Endocrine Policy Forum Comments on the EPA White Paper on 

“Integrated Bioactivity and Exposure Ranking: 

A Computational Approach for the Prioritization and Screening of 

Chemicals in the Endocrine Disruptor Screening Program”

1
Endocrine Policy Forum

 Consortium of Tier 1 test order recipients and other interested 

stakeholders

 Self-funded

 Represents >95% of List 1 test order recipients 

 Stakeholders include CLA, ACC, ACI, CSPA, and consulting companies

 Objective to address regulatory and policy issues and technical 

guidance

 To undertake collaborations to tackle common areas of interest

 Actively evaluating data from Tier 1, developing methods for WoE, 

publishing in peer-reviewed journals, and providing written and oral 

comments for four SAPs in 2013, the one in July 2014, and now the 

current SAP

2
Overarching Comments

Strengths

 Conceptually we support EPA’s data-driven approach to 

combine high-throughput bioactivity data and exposure 

predictions for prioritization

 The ER model appears to perform well as currently 

parameterized

 There is a reproducible systematic approach to separate in vitro 

bioactivity from non-selective cell stress

 Acknowledgment of the importance of rigorous data quality 

evaluation for model development and evaluation

3
Overarching Comments

Limitations

 Insufficient information was provided to support an 

independent review of the methodology

 Lack of model details (e.g. user interface, R-code) prevented 

full evaluation of the adequacy, predictivity, and transferability 

of the models (although we understand that the dashboard is 

now active)

 AUC/potencies should be anchored to biologically-relevant 

activity in vivo

 Application of IBER scores needs to be demonstrated

 Comparison of the AR AUC model with in vivo data is needed 

(as acknowledged)

4
White Paper Recommendation

 The EPA White Paper should be updated with the 

additional information and important clarifications 

provided in the EPA SAP presentations

 Improve understanding of the methodology

 Important step in completing the document to have an 

accurate and current record

5
List 1 Lessons and 

Data Considerations

 Tier 1 screening of List 1 chemicals confirmed many negative results 

identified by higher-tiered testing

 Negative results from higher-tiered studies should be used to “deprioritize” chemicals

 Opportunity to comment on draft prioritization could enable 

refinements

 Industry could provide information before the issuance of test 

orders for appropriate prioritization

 Other Scientifically Relevant Information 

 Guideline studies and published literature

 Refined exposure estimates

6
Perspectives on Charge Questions:

Estrogen and Androgen Receptor Systems

Sue Yi, Ph.D.

Syngenta on Behalf of the Endocrine Policy Forum

US EPA Science Advisory Panel Meeting

December 2-5, 2014

7
Questions 1.a. and 2.a: How clearly has EPA described the computational tools used to 

estimate ER and AR agonist and antagonist bioactivity?

 The critical need for and the general function of the ER and AR AUC model 

are well-presented and understood.

 Major strengths: EPA is to be commended for pursuing a scientifically-rigorous 

approach to synthesize multiple estrogen and androgen receptor assay inputs, and 

distinguishing in vitro estrogen and androgen receptor activity from cytotoxicity using 

the systematic Z-score approach.

 Primary limitations:

 Technical descriptions of ER and AR AUC model functions are incomplete for 

evaluation.

 The current method of AUC calculation is unclear. Its composite nature complicates 

a true comparison to pharmacological potency.

 A specific example of how the AUC was calculated would be helpful.

 Important to cite which version of ToxCast and ExpoCast are being used so that 

stakeholders can refer to the correct version if trying to recreate or verify analyses

8
More detail on model function would improve ability to evaluate 

the ER and AR AUC model (cont. 1a and 2a).

 Concurrent release of an interactive user-interface and/or R scripts with input data 

files would be helpful in understanding the function of the model.

 Bulleted summary of the order of operations of the model would be helpful.

 Explanation of how the model was corrected to handle the detailed observations of 

crosstalk and cross-reactivity (pages 34-35) would be helpful.

 Further explanation of specific model terms and choices is needed for evaluation, e.g.:

 Z-Scores should be presented in an appendix, with sample calculations, and a list of all assay 

inputs used;

 Penalty(R) terms, includes set levels for α and SR0;

 Sign(slope): “The factor sign(slope) is included to handle cases where one of the assays or sets 

of assays is active at significantly lower concentrations than the remaining assays. The 

corresponding receptor curve will then rise and subsequently fall, and this AUC needs to be 

discounted.” As discounting may affect predicted bioactivity, it is necessary to understand this procedure.

9

Page 22 of SAP White Paper; mathematical description 

of the model on pgs. 21-23 appears incomplete. 
Questions 1.b and 2.b: What are the strengths and limitations of the ER or AR AUC 

model’s ability to identify reference chemicals that include a variety of structures and 

have a wide range of in vitro ER or AR bioactivities?

 Primary strengths:

 Systematic consideration of cytotoxicity and nonselective cell stress as separate from in 

vitro endocrine activity (Z-score filtering).

 Inclusion of many assay endpoints to contextualize potential false positives and false 

negatives due to assay interference.

 Current limitations:

 Effects on steroidogenesis pathways are not currently included.

 Mathematical relationship between potency and AUC is unclear, and as such, it is 

unclear how AUC can be used to prioritize and distinguish chemicals associated with 

different levels of concern.

 Discussion of structural classes evaluated is limited. 

 How many structure classes were assigned in the library of tested chemicals?

 Do the reference structures evaluated reflect understanding of the diversity of 

structures likely to perturb the ER or AR pathway

 Reference chemicals must span a wide range of potencies to strengthen confidence

10
Question 1c. and 2.c“...evaluate concordance between ER AUC model scores (and in 

the future AR AUC model scores) of in vitro bioactivity, and the in vivo uterotrophic (or 

Hershberger) response studies (Sections 2.2.1 or 3.2.1.) What are the strengths and 

limitations of the curation methods and quality standards used for evaluating published 

in vivo studies?

 Systematic review of the literature to evaluate concordance of the 

ER AUC and AR AUC model is a critical step toward acceptance 

of these models.

 Literature must include chemicals that span a range of potencies and 

representing sufficient true positive and negative chemicals

11
Question 1c. and 2.c. (cont.): What are the strengths and limitations of the 

curation methods and quality standards used for evaluating published in vivo 

studies?

 Limitations:

 Availability of test article purity should be included. Suggest following the 

ToxRTool approach (https://eurl-ecvam.jrc.ec.europa.eu/about-ecvam/archivepublications/toxrtool).

 Attention to systemic toxicity or methodological approaches that could jeopardize 

the relevance and/or reliability of the data is needed.

 e.g., Uterotrophic (UT) - body weight should be specified as a minimum 

endpoint to distinguish systemic toxicity from endocrine activity (absolute versus 

relative uterus weight could then be determined).

 Specification of the birth data and age of animals used in the UT immature assay.

 Criteria for determination of study outcomes should be clearly specified (e.g., in 

accordance with OCSPP 890.1600 and OCSPP 890.1400 guidelines).

 e.g., Hershberger requires statistically significant changes in two or more 

accessory sex tissue weights in the appropriate direction, not any change as 

noted on Page 73 of the White Paper

 Was laboratory proficiency determined? 12
1d. Based on all the data presented in Section 2 on ER AUC model performance 

including characterization of reference chemicals, and concordance with in vivo 

uterotrophic results, what are strengths and limitations of using the ER AUC model to 

distinguish and prioritize chemicals based on potential estrogen bioactivity?

 Strength: For the ER AUC, currently the model performance relative 

to the Uterotrophic reference chemicals appears excellent based on 

sensitivity and specificity; improvement would be anticipated based on 

addition of metabolic capacity to the battery.

 Limitations:

 Lack of consideration of a relative potency cut-off, i.e. 0.1 model positive may not relate to 

in vivo receptor activity. 

 Benchmarking to a chemical with in vitro estrogenic activity that does not demonstrate in 

vivo activity based on low potency would be helpful.

 Explanation of ER AUC scores that translate to in vivo relevance would increase 

confidence that the ER AUC model can be used for prioritization.

 Explanation of steps in prioritization, e.g. what happens to “down-selected” chemicals based 

on T-score and Z-score filtering (page 34)?

 Does ER AUC actually distinguish low and high ER potency for prioritization? What is an 

13 example of a priority chemical for further evaluation?
2.d: Based on the data presented in Section 3 on AR AUC model’s 

performance, what are strengths and limitations of using the AR AUC model to 

distinguish and prioritize chemicals based on potential androgen bioactivity?

 The AR AUC model as presented represents a very encouraging 

but incomplete effort.

 Without the comparison to in vivo responses, it is not currently at a 

stage that can be evaluated for prioritization potential

 Model performance information is needed across a wider diversity of 

chemicals

 Relative potency measures must be addressed

14
1e.: “...what are the strengths and limitations of the ER AUC model to contribute to the 

weight of evidence determination of a chemical’s potential estrogen bioactivity?”

 Strength: ToxCast bioactivity data, and the ER AUC model, should be 

considered in the weight of evidence for potential estrogen bioactivity 

because these data appear to inform relevant molecular-initiating event 

hypotheses. 

 Limitations:

 How would an incomplete set of assay data be used with the ER AUC 

model or independent of the ER AUC model?

 How were the 18 assay inputs determined to contribute to model performance?

 Method must be more transferable before application to weight of evidence 

determinations.

 Further explanation of how potency shifts affect the overall AUC score is needed. 

15
1f. “...what are strengths and limitations of using the ER AUC model to 

substitute for EDSP Tier 1 ER binding, ER transactivation, or uterotrophic 

assays for the purpose of characterizing a chemical’s potential estrogen 

bioactivity?”

 Combined strength and limitation: a high degree of concordance 

between the ER AUC model and the uterotrophic results, but 

focused predominantly on negative uterotrophic results.

 Is a uterotrophic assay result the only in vivo endpoint that could be 

useful for comparison?

 Target all the estrogenic/anti-estrogenic hypotheses as relevant for 

assessing this charge question

 More positive controls or chemicals need to be incorporated.

16
The ER AUC and AR AUC models are an encouraging effort, but require more 

accessibility and transferability for evaluation with respect to their utility in 

prioritization.

 Models needs to be further described.

 Relationship between AUC and potency needs to be well-understood.

 Instances of incomplete data sets should be defined.

 Systematic separation of cytotoxicity and endocrine bioactivity is an appropriate step, 

but should be clearly described.

 Steroidogenesis should be considered for estrogen and androgen activity 

prioritization.

 Concordance with in vivo androgenic responses is needed

 Consider benchmarking to a chemical of low to no in vivo concern, or 

determining an AUC threshold, so that stakeholders can understand how 

prioritization would actually proceed.

17
Christopher J. Borgert, Ph.D.

APT on Behalf of the Endocrine Policy Forum

Science Advisory Panel Meeting

December 2-5, 2014

Perspectives on Potency

18
Potency is Under-Utilized by AUC Methodology

19

Reference Chemical AUC

Genistein 0.77

Diadzein 0.623

Estradiol-17 1.01

Estrone 0.987

DES 1.01

17-methyltestosterone 0.441

Example ER AUC Scores from Appendix 3

• AUC methodology appears to compress 

potency ranges and – relative to traditional 

potency comparisons – skews differences 

between primary endogenous estrogens, 

potent synthetic estrogens, phytoestrogens 

that are marginally active in vivo; and 

androgens that are not estrogenic in vivo. 

Potency could be used to greater advantage. Potency comparisons between key in 

vitro assays, uterotrophic assays, and reproduction studies could provide a sound 

basis for potency thresholds below which manifestation of hormonal activity is 

highly unlikely. Further screening for chemicals below the threshold potency 

would be unwarranted and wasteful of animals and resources.

If AUC1 = AUC2, how exactly is the AUC score 

adjusted for potency differences (chemical1

potency >> chemical2 potency)?

Are instances like this possible?

AUC1

ER 

activity

AUC2

concentration
Endocrine Pharmacology: 

Affinity Range Determines Relevance In Vivo

Competitive Binding Assay

Unlabeled test ligand can replace radiolabelled E2 

from the receptor depending on its affinity

E2 and DES, high affinity

Natural Products of 

Metabolism (precursors, 

metabolites, sterols, fatty 

acids, proteins) = 

105- 109 Molar Excess Relative 

to Hormones!

Genistein, lower affinity

20
Endocrine Pharmacology: Potency / Receptor Occupancy

Borgert et al. 2013 Regul Toxicol Pharmacol. 67:83-88.

Endogenous Hormone Concentration = 10 part per quadrillion

Hormone Molecular Weight = 100 mu

Diameter of Eukaryotic cell = 10 – 100 μm (assume 20 μm)

Assume Spherical, so Volume = 3 x 10-11 Liters

Hormone Molecules per Cell = 2

How Much Additional Hormone Needed to Increase Receptor Occupancy?

Hormone Affinity Constant = 1 x 10-11 M

At Hormone Concentration = 1 x 10-13 M, 1% of receptors will be occupied.

1 x 10-13 M additional ligand is required to increase receptor occupancy to 2%

This is equivalent to 2 trillion molecules in the water compartment of an average-size 

woman (33 Liters); more if in a man.

If the ligand has a lower affinity – e.g., most putative EDCs – a greater molar increase 

would be needed to increase receptor occupancy by 1%.

21
False Negative (D4): Potency Can Clarify Interpretation 

Possible Reasons Given for False Negative Result 

(White Paper, page 51)

22

i) “in vivo metabolic conversion to an active compound, which cannot be replicated by the in 

vitro systems;” metabolites are inactive

[but is active in other in vitro systems]

ii) “uterotrophic bioactivity mediated via a non-ER mechanism,”

[but is blocked by ICI]

iii) “weak in vivo activity (high LEL) that would require serum concentrations that are too high 

to be replicated by in vitro systems,”

[D4 fails to elicit an anti- / estrogenic pattern of effects in reproduction studies: 

no uterine weight changes in F0 or F1 females; no effects on F1 males; very weak 

potency in screening-level assays]

iv) “inaccurate assumptions regarding the quantities of test material used during the screening 

process (quantity / purity).”

[possible or likely; material extremely volatile; difficult to maintain aqueous 

concentrations]
23

Test Chemical Mean EC50 Estradiol-17*

Mean EC50 Test Chemical*

Assay / Citation

Estriol 6.67E-02 ERTA(α)

Chen et al. 2004*

2-OH-E2 2.00E-02 ERTA(α)

Chen et al. 2004*

Testosterone 2.00E-05 ERTA(α)

Chen et al. 2004*

Testosterone 7.01E-06 BG1-Luc ERTA Mean EC50 

(ICCVAM, 2011)

19-Nortestosterone

(not aromatized)

2.06E-06 BG1-Luc ERTA Mean EC50 

(ICCVAM, 2011)

Methyltestosterone 5.24E-06 BG1-Luc ERTA Mean EC50 

(ICCVAM, 2011)

Tamoxifen 5.05E-05 BG1-Luc ERTA Mean EC50 

(ICCVAM, 2011)

Genistein 4.00E-04 Ranhotra et al. 2005*

Genistein 1.13E-05 BG1-Luc ERTA Mean EC50 

(ICCVAM, 2011)

Estrone 1.57E-02 BG1-Luc ERTA Mean EC50 

(ICCVAM, 2011)

Ethinyl Estradiol 4.8E-01 BG1-Luc ERTA Mean EC50 

(ICCVAM, 2011)

DES 1.63E-01 BG1-Luc ERTA Mean EC50 

(ICCVAM, 2011)

* Data derived from published graphs
Assay Positive Control [Positive Control]

[D4 at 50% response]

Citation

RUA (F344) Weight EE 2.6E-07 McKim et al. 2001

RUA (F344) Weight DES 5.1E-07 McKim et al. 2001

RUA (SD) Weight EE 1.71E-06 McKim et al. 2001

RUA (SD) Weight DES 1.85E-06 McKim et al. 2001

RUA (SD) Weight Coumestrol 2.17E-02 McKim et al. 2001

MUA (B6C3F1) EE 6.0E-06 (Weight)

2.5E-06 (Peroxidase)

He et al. 2003

ERTAα 17β-E 5.0E-06 Quinn et al. 2007

ERBAα EE 1.14E-04 Quinn et al. 2007

ERBAα DES 7.4E-04 Quinn et al. 2007

ERBAβ EE, DES 0 Quinn et al. 2007

D4 concentrations that elicit effects in reproduction studies (700 ppm vapor) exceed 90% ER 

occupancy based on potency estimates. Lack of effects at lower concentrations (less receptor 

occupancy) is inconsistent with an ER mechanism. 24
Summary on Potency

 Potency is not well utilized by the current AUC methodology.

 Potency could be used to greater benefit, as illustrated by the “false negative” example 

(D4) given in the white paper:

 If D4 operated through an ER mechanism, one should see:

a) typical estrogenic effects in 2-gen studies, and;

b) estrogen-type effects at concentrations that would occupy less than 90% ERs.

 Instead, D4 produces a pattern of effects inconsistent with effects of classical estrogens and 

which does not occur until D4 would be occupying 90% or more of ERs, based on its 

estimated potency from screening-level assays.

 Thus, the AUC methodology could be enhanced by considering that chemicals with low 

potency (e.g., below 1E-04):

c) May be negative by the AUC methodology for a variety of reasons, and;

d) even if positive by the AUC methodology, should not be prioritized for further 

screening.

Establishing clear potency thresholds below which endocrine activity in intact, functioning 

organisms is unlikely, should be possible using the AUC methodology, and would markedly 

enhance the biological consistency of the method and its practical utility.

25
Perspectives on Charge Questions 

IBER

26

Christopher J. Borgert, Ph.D.

APT on Behalf of the Endocrine Policy Forum

Science Advisory Panel Meeting

December 2-5, 2014
3a. “How clearly has EPA described the computational tools in Section 6 to 

develop IBER values, including modeling uncertainty and population 

variability?”

 IBER scores will depend on results of High-throughput exposure 

(HTE) simulations, High-throughput toxicokinetic (HTTK) predictions 

and bioactivity assays pathway activity distributions. 

 HTE: 

 The approaches for exposure estimates have been well defined (see July SAP)

 However, there still exists a wide range of variability in exposure predictions, 

especially for far-field scenarios

 Also, no predictions for non-oral exposure routes are given, which can be 

substantial for many occupational and consumer scenarios

27
3a. “How clearly has EPA described the computational tools in Section 6 to 

develop IBER values, including modeling uncertainty and population 

variability?”

 HTTK: 

 The PK model used to predict systemic exposures (steady-state blood levels) 

was well discussed at the July 2014 SAP

 The use of a high-end PK model tool (SimCYP) allows for a probabilistic 

distribution of systemic exposures

 Specific details of the 3-compartment PK model have not been made public 

(i.e., model code)

 Also, cost issues will limit access to the SimCYP tool by relevant stakeholders

 Explore less costly alternatives 

 The HTTK model, as defined, assumes linear kinetics for absorption, 

metabolism and excretion. Utilization of models that allow for non-linear 

properties (especially for metabolism) would provide more relevant blood level 

predictions at low-level, environmental exposures. 

 Finally, no predictions for non-oral exposure routes are given 

28
3b-3c. “What are strengths and limitations of using the IBER approach to prioritize

chemicals for further EDSP screening based on the ratio between the ER or AR

bioactivity dose ranges, and the expected exposure ranges?”

29

 We agree that the IBER screening approach will provide a means of prioritizing 

compounds with the highest ER/AR biological activity in selected in vitro/in vivo 

assay for subsequent EDSP testing

 Bioactivity vs. Exposure comparisons:

 The use of upper bounds of exposure and lower bounds for bioactivity can account for 

uncertainty and variability

 However, upper and lower bounds are clearly not as stable as central estimates

 Comparisons across compounds within this program should consider use of central 

estimates of exposure and activity

 The use of upper bound and lower bound estimates likely implies greater 

precision than is warranted by the datasets and models – fit for purpose

 Prior to final decisions for EDSP evaluation based on IBER rankings, potential test 

order recipients should be allowed to submit refined IBER assessments that utilize 

measured exposure and or toxicokinetic datasets
Thank you for your attention!

Questions?

30