1

1 SCREENING CHEMICALS FOR ESTROGEN RECEPTOR BIOACTIVITY USING A COMPUTATIONAL MODEL  

Patience Browne1*, Richard S. Judson2

, Warren Casey3, Nicole Kleinstreuer4, Russell S. Thomas2 2

1

US EPA, Office of Chemical Safety and Pollution Prevention, Washington DC; 2 3 US EPA, Office of Research and

Development, Research Triangle Park, NC; 3 4 National Toxicology Program, Interagency Center for the Evaluation

of Alternative Toxicological Methods, Research Triangle Park, NC; 4 5 ILS/NICEATM, Research Triangle Park, NC  

6 The views expressed in this paper are those of the authors and do not necessarily reflect the views or policies of the US Environmental

7 Protection Agency.

8

9 *Address correspondence to P. Browne, US EPA, Office of Science Coordination and Policy, 1200 Pennsylvania

10 Ave NW (7203M), Washington, DC 20460, Telephone: 202‐564‐8474, Fax: 202‐564‐8482, E‐mail:

11 browne.patience@epa.gov.

12

13 Abstract

14 The U.S. Environmental Protection Agency (EPA) is considering high‐throughput and computational methods to

15 evaluate the endocrine bioactivity of environmental chemicals.  Here we describe a multi‐step, performance‐

16 based validation of new methods and demonstrate these new tools are sufficiently robust to be used in the

17 Endocrine Disruptor Screening Program (EDSP).  Results from 18 estrogen receptor (ER) ToxCast high‐throughput

18 screening assays were integrated into a computational model that can discriminate bioactivity from assay‐

19 specific interference and cytotoxicity.  Model scores range from 0 (no activity) to 1 (bioactivity of 17β‐estradiol).  

20 ToxCast ER model performance was evaluated for reference chemicals, as well as results of EDSP Tier 1

21 screening assays in current practice.  ToxCast ER model accuracy was 86 to 93% when compared to reference

22 chemicals, and predicted results of EDSP Tier 1 guideline and other uterotrophic studies with 84 to 100%

23 accuracy.  Performance of high‐throughput assays and ToxCast ER model predictions demonstrates these

24 methods correctly identify active and inactive reference chemicals, provide a measure of relative ER bioactivity,

25 and rapidly identify chemicals with potential endocrine bioactivities for additional screening and testing.  EPA is

26 accepting ToxCast ER model data for 1812 chemicals as alternatives for EDSP Tier 1 ER binding, ER

27 transactivation, and uterotrophic assays.   

28

29 Introduction   

30 The US Environmental Protection Agency’s (EPA) Endocrine Disruptor Screening Program (EDSP) was established

31 in 1999 for the purpose of evaluating potential risk of endocrine disruption in humans and wildlife from

32 exposure to pesticide chemicals and drinking water contaminants.  To screen environmental chemicals for

potential endocrine bioactivity, EPA developed a battery of five in vitro and six in vivo Tier 1 screening assays1 33 .  

34 To test for potential endocrine disruption, EPA developed in vivo multigenerational Tier 2 tests that include

apical endpoints to identify adverse effects, and establish quantitative dose response relationships2 35 .  In 2009,

EPA published a final list of 67 pesticide chemicals (List 1) and issued EDSP Tier 1 test orders on these chemicals3 36 .  

37 Fifteen List 1 chemicals were voluntarily withdrawn from the pesticide market.  EPA is currently reviewing

38 results of EDSP Tier 1 screening assays, along with other scientifically relevant information, and developing
2

39 weight of evidence evaluations of potential endocrine bioactivity with a determination of further testing that

40 may be required for the remaining 52 of the 67 chemicals.  A second list of pesticide and high production

volume chemicals (List 2) proposed for Tier 1 screening was published in June 20134 41 ; however, test orders have

42 yet to be issued.  The remaining EDSP universe of pesticide chemicals and drinking water contaminants includes

43 approximately 10,000 environmental chemicals to be screened for potential endocrine bioactivity in humans

and wildlife5 44 .  

In response to the US National Academy of Sciences report, Toxicity Testing in the 21st Century6 45 , and the US

President’s 2012 proposed budget7 46 , EPA began a multi‐year transition from existing EDSP test methods towards

47 utilizing more rapid, cost‐effective, computational models and high‐throughput assays.  The transition to using

48 computational toxicology approaches to prioritize and screen thousands of EDSP chemicals has been outlined by

the Agency in two strategic planning documents5, 8 49 .  However, to use new computational toxicology approaches

50 in the existing EDSP screening and testing framework, new methods must be validated and perform as well or

51 better than existing methods currently in practice.   

A variety of pesticides and environmental chemicals act as estrogen receptor (ER) agonists9‐12 52 .  Although the

53 scope of the program was expanded to consider androgen and thyroid active environmental chemicals, the EDSP

54 was originally established in response to statutory mandates in the Federal Food Quality Protection and Safe

55 Water Drinking Acts compelling EPA to evaluate potential xenoestrogens.  As a result, the Tier 1 screening

56 battery was weighted towards assays that detect potential ER interactions.  EDSP Tier 1 assay endpoints that

57 measure interaction with the ER only measure agonism and environmental chemicals that act through the ER

58 are primarily expected to act as agonists, therefore we focused this demonstration using computational

59 toxicology tools to evaluate ER agonist bioactivity. Future work will address estrogen antagonism and other

60 endocrine pathways.

61 In this manuscript, we present a performance‐based approach to validate computational toxicology tools for

62 evaluating a chemical’s potential ER agonist bioactivity and as alternative data for existing EDSP Tier 1 in vitro

and in vivo assays.  High‐throughput in vitro screening assay data from EPA’s ToxCast program13 63 were integrated

64 into a computational model of the ER pathway, and model performance was compared with in vitro and in vivo

reference chemicals identified from the peer‐reviewed literature14 65 .  A variety of publications have previously

described the ToxCast program15, 16, the validation of high‐throughput screening assays17 66 , and endocrine‐

relevant ToxCast assays18, 19 67 ; however, this paper is the first description of using ToxCast high‐throughput

68 screening data as an alternative for regulatory guideline studies.  Integrating 21st century toxicology in the EDSP

69 allows for rapid evaluation of potential endocrine bioactivity of thousands of chemicals to which humans and

wildlife may be exposed and this approach is consistent with recommendations of the 2007 NAS report6 70 and

EPA’s strategic plan for evaluating the toxicity of chemicals8 71 , specifically to: 1) provide broad coverage of

72 chemicals examined; 2) reduce the cost and time of toxicity testing; 3) reduce animal use; and 4) develop a

73 robust scientific basis for assessing health effects of environmental agents.

74

75 Methods

76 In Vitro Assays

Details of the in vitro assays are described on EPA's ToxCast website13 and in a variety of publications15, 20 77 .  

78 Briefly, potential ER bioactivity was measured in 18 high‐throughput in vitro assays run in EPA’s ToxCast
3

79 program.  The suite of high throughput assays measure the molecular initiating event (i.e., receptor binding), in

80 addition to several key events (e.g., receptor dimerization, DNA binding, transactivation, gene expression, and

81 cell proliferation) in an adverse outcome pathway.  The 18 ER assays include three cell‐free biochemical

radioligand ER binding assays21, 22 82 ; three protein complementation assays that measure formation of ER dimers

83 and test for activity against ERα and ERβ, each measured at two time points; an assay measuring interaction of

84 the mature transcription factor with DNA at two time points; two reporter gene assays measuring RNA

transcript levels23; two assays measuring reporter protein levels24; an ER‐sensitive cell proliferation assay25 85 ; and

two transactivation antagonist assays24 86 (Table 1).   

87 This combination of biochemical and cell‐based in vitro assays relies on different technologies and probes

88 different key events in the ER signaling pathway (Table 1).  Though the assays are primarily human proteins

89 and/or cell types, the suite of 18 assays include human, murine, and bovine ER binding assays and ER pathway

90 interactions in a variety of human tissue types (Table 1).  Every in vitro assay is potentially subject to technology‐

91 specific interference (e.g., chemicals that denature the receptor protein, are luminescent, are cytotoxic, etc.)

92 that can be mistakenly interpreted as bioactivity. However, combining data from multiple assays and integrating

93 data in a network model of the entire ER pathway allows for detection of false positives and a more confident

94 assessment of the “true” in vitro estrogenic bioactivity of the tested chemical.   

95 Concentration Response Analysis and Computational Modeling

96 Chemicals were run in concentration‐response format in all assays except the cell‐free binding assays.   Cell‐free

97 competitive binding assays were initially run at a single screening concentration (25 μM), and if the test chemical

98 was active in the assay (i.e., radioligand was displaced), the assay was run in concentration‐response format.  All

99 in vitro assays except the assays measuring RNA transcript were normalized to 17‐estradiol.  RNA transcript

100 data were normalized as a fold‐change from the solvent (DMSO) control.  Concentration‐response data from in

101 vitro assays were fit to three models that included a four parameter Hill model, a modified Hill model with gain‐

loss at high concentrations, or a constant (no concentration‐response) model26 102 .  The best model was statistically

103 selected using the Akaike Information Criteria value.  All concentration‐response data were analyzed using the

104 ToxCast data analysis pipeline, which automates the processes of baseline correction, normalization, curve‐

fitting, hit‐calling and detection of a variety of potential confounders27 105 .  To integrate results of the different in

106 vitro assays, concentration‐response curves were generated for each assay across 14 concentrations from 0.01‐

107 100 μM.

108 The concentration‐response curves for all 18 assays were included in a computational model, referred to here as

the ToxCast ER model for bioactivity14 109 .  The computational model integrates data from the 18 in vitro assays

110 measuring ER agonist and antagonist responses in an unweighted manner, while subtracting background and

111 other non‐specific assay interference including cytotoxicity.  The model output includes separate agonist and

112 antagonist area under the curve (AUC) scores, though only the agonist response was considered in this analysis.

113 The ER agonist AUC model scores range from 0 (no activity) to 1 (bioactivity of 17β‐estradiol). The computational

114 model is a simple linear additive model and assumes that the value for a given concentration is a linear sum of

115 the contributions from the ER interaction (e.g., receptor binding, transactivition, mRNA production, etc.)

116 measured in each assay. Each assay contributes equally to the overall score for ER pathway activity if there is

117 direct molecular interaction between a chemical and an assay, and if the assay is not hit by the chemical it

118 provides no contribution to the score.  Therefore, the model assumes lossless transmission of signals from the

119 key events in the signaling pathway to the assays.  For each chemical‐concentration pair, a constrained least‐

120 squares minimization approach is used to reconcile the predicted assay values and the measured values, taking
4

121 into account both potency and efficacy (mathematical details and a complete description of the model can be

found in EPA 2015)14 122 .   

123 Model scores were truncated at values < 0.001, considered to have no ER bioactivity and given a score of 0, as a

124 value <0.001 implies an concentration required to elicit 50% of the maximal response (AC50) greater than 10

125 millimolar which is several orders of magnitude greater than the highest concentrations tested in ToxCast

126 assays.  ToxCast ER agonist scores > 0.1 were considered positive; a model score of 0.1 equates to an AC50 of

127 about 100 μM and approximates the upper limit of bioactivity detected in this approach.  Model scores of 0.1>

128 AUC >0 were considered inconclusive for this validation because these chemicals were active in only one or two

129 ToxCast ER assays and activity was limited to the highest concentrations tested.  Activity in only a few of the 18

130 assays may be due to differences (and thus differential sensitivities) in the various in vitro assays, though all

131 chemicals were tested up to at least 100 μM. It is unlikely that environmental exposures to most chemicals

132 would result in an internal dose at this level, and therefore such limited activity and low potency (i.e., interaction

133 with the ER occurring only at high concentration of test chemical) has questionable relevance to in vivo

134 bioactivity.  We evaluated model performance against all chemicals and after excluding chemicals with

135 inconclusive scores.  

136 Performance‐Based Validation of the ToxCast ER Model

137 To assess the strengths and limitations of the ToxCast ER model, we adopted a performance‐based validation

138 approach consistent with the Organization for Economic and Cooperation and Development (OECD) conceptual

framework for assessing potential endocrine disrupting chemicals28 139 .  In principle, this method can be used to

140 assess the applicability of any test method or set of methods that meets defined performance standards.  The

141 ToxCast ER model was validated using two approaches: comparison of ToxCast ER model scores against sets of

142 reference chemicals with independently confirmed ER bioactivity in a validated test method, and comparison of

143 ToxCast ER model scores with results of EDSP Tier 1 guideline assays currently used to screen chemicals for

144 endocrine bioactivity.   

145 In vitro ER reference chemicals, previously identified using multiple validated low throughput in vitro ER assays,

146 were identified by the Interagency Coordinating Committee on the Validation of Alternative Test Methods  

(ICCVAM) and OECD29 147 for the express purpose of validating novel in vitro assays.  Forty chemicals (28 agonists of

148 differing potencies indicated by a range in AC50 and 12 inactive chemicals) were selected for reproducible results

in in vitro ER binding and transactivation assays, and to include a diverse set of chemical structures (Table 2)29, 30 149 .  

150 All in vitro reference chemicals were run in the 18 high‐throughput ToxCast ER assays and the resulting ToxCast

151 ER model scores for agonist bioactivity were compared with results anticipated from low or medium throughput

in vitro assays29 152 .   

153 In vivo reference chemicals were established from a literature search of short‐term rodent assays that were

methodologically similar to the OECD31 and EDSP Tier 1 battery uterotrophic32 154 assays.  A comprehensive search

and review of uterotrophic studies published in peer‐reviewed literature was performed14 155 .   Chemical name and

156 chemical abstract services registry number (CASRN) were used to search PubMed, the EPA’s Aggregated

Computational Toxicology Resource (ACToR)33 157 , and the US Food and Drug Administration’s Endocrine Disruptor

Knowledge Base (EDKB)34 158 for the 1812 chemicals run in the ToxCast in vitro ER assays with “uterotrophic assay”,

159 “uterotrophic”, “uterotropic”, and “uterine weight” as modifier terms.  Articles identified were reviewed for

methodological consistency with the EDSP uterotrophic assay guidelines32 160 based on: 1) age and species of

161 animals used (immature rat or ovariectomized mouse or rat); 2) number of animals per treatment group; 3)

162 number of treatment groups; 4) route of chemical administration; 5) length of dosing; and 6) time of necropsy.  
5

163 Over 1000 articles were identified, entered into a database, and independently reviewed by two scientists.  Of

the articles identified, 442 studies of 103 chemicals met all six minimum criteria14 164 , were considered "guideline‐

165 like", and were used in this analysis.   

166 Chemical data from guideline‐like uterotrophic studies were considered with two levels of stringency.  First, a

167 chemical was considered positive for potential in vivo ER agonist bioactivity if a significant increase in uterine

168 weight among treated animals was reported and negative if no significant increase in uterine weight was

169 reported in any guideline‐like study.  Second, a subset of chemicals with reproducible results from two or more

170 independent, guideline‐like studies were referred to as in vivo reference chemicals.  Chemicals that resulted in a

171 significant increase in uterine weight in two or more studies were considered active reference chemicals, while

172 those chemicals that showed negative results in all studies (two or more) were considered inactive reference

173 chemicals (Table 3).  

174 ToxCast ER model agonist scores were compared with results of the EDSP Tier 1 assays that directly assess a test

chemicals’ ER bioactivity.  The Tier 1 in vitro ER binding assay35 175 uses rat uterine cytosol (primarily ERα) and

176 cannot distinguish between potential agonist or antagonist bioactivity.  The competitive binding assay measures

test chemical displacement of radioligand ([3 177 H]17β‐estradiol) from the ER across a range of concentrations in

178 three independent runs.  Results of the assay are “positive” if the test chemical displaces >50% of radioligand

179 (and Log(IC50) is calculated), “equivocal” if test chemical displaces <50% but >25% of radioligand, and “negative”

if test chemical displaces <25% of radioligand35 180 .  The Tier 1 in vitro Estrogen Receptor Transcriptional Activation

(ERTA) assay 181 36 measures chemiluminescence in response to an ERα‐mediated increase in luciferase gene

182 expression (i.e., agonist activity).  A test chemical is “positive” if the maximum response induced by the test

183 chemical is > 10% of the maximum response of the positive control (17β‐estradiol; RPCmax) in at least two of

184 three assay runs (i.e., RPCmax ≥10).  If the test chemical fails to achieve at least 10% of the response of the

positive control, a negative response is recorded for the test chemical.  The Tier 1 uterotrophic assay32 185 is a short‐

186 term, in vivo assay designed to detect exogenous estrogen agonist activity indicated by an increase in uterine

187 weight in animals in prepubertal or ovariectomized rodents, in which the hypothalamic‐pituitary‐gonadal axis is

188 not functional.  

189 Application of ToxCast ER model to EDSP Chemicals

190 ER model agonist scores were examined for 1812 chemicals evaluated in all 18 ToxCast ER assays.  These

191 chemicals include 62 EDSP List 1 chemicals for which EPA issued Tier 1 test orders, and 57 List 2 chemicals

192 identified by EPA as candidates to receive the next group of Tier 1 test orders.  The 1812 chemicals include 387

193 pesticide active ingredients, and 364 pesticide inerts; most of the remaining chemicals are relevant to the EDSP,

194 contingent on potential for exposure of substantial human populations through sources of drinking water.

195

196 Results  

197 Concentration‐response curves of the 18 high‐throughput in vitro ER assays were integrated into a model of ER

pathway bioactivity14 198 .  Model outputs include an integrated measure of agonist bioactivity, antagonist

bioactivity, as well as "false positive" signaling due to cytotoxicity or technology‐specific interference14 199 .  For

200 reasons described previously in this paper, only ER agonist bioactivity was considered in these analyses.  

201 Performance metrics (true positives, true negatives, false positives, false negatives, balanced accuracy,

sensitivity, and specificity)37 202 were calculated for ToxCast ER model score performance against reference
6

203 chemicals and guideline studies.   Performance metrics were calculated for chemicals with any indication of

204 ToxCast ER agonist bioactivity (AUC>0) or no activity (AUC=0) and again excluding inconclusive model scores

205 (0<AUC<0.1) for which no call of bioactivity could be determined (Table 4).

206 The performance‐based assessment of the ToxCast ER model for agonist bioactivity relied on in vitro reference

207 chemicals, in vivo reference chemicals, guideline‐like uterotrophic studies, and results of EDSP Tier 1 assays

208 (Figure SI‐1).  For the 40 in vitro agonist reference chemicals, the ToxCast ER model performed very well, with an

209 overall balanced accuracy of 93% (Table 4).  Of the 28 active reference chemicals, 26 of 28 had ToxCast ER model

210 bioactivity (Table 2).  ToxCast ER agonist scores were positive (> 0.1) for all strong, moderate and weak agonist

211 reference chemicals (Table 2).  ToxCast ER model bioactivity was inconclusive (0<AUC<0.1) for one very weak

212 active chemical (di‐n‐butyl phthalate).  Two very weak reference chemicals (diethylhexyl phthalate (DEHP) and

213 dicofol) were false negatives (model score = 0).  Of the 12 inactive reference chemicals, 11 chemicals had no ER

214 model agonist bioactivity.  One inactive chemical (haloperidol) had an inconclusive ToxCast model score

215 (0<AUC<0.1).  If the two chemicals with inconclusive ToxCast ER agonist model scores are excluded from

216 performance metrics, the overall accuracy is 95% (Table 4).    

217 The overall accuracy of the ToxCast ER model agonist bioactivity evaluated for 43 in vivo reference chemicals

218 (Table 3) with independently verified results in two or more guideline‐like uterotrophic studies was 86% (Table

219 4).  Of 30 active reference chemicals, 29 had positive ToxCast ER agonist model scores (AUC > 0.1; Table 3).  The

220 potential false negative chemical, octamethylcyclotetrasiloxane (D4), was positive in multiple uterotrophic

221 studies run in independent labs but negative in the ToxCast ER bioactivity model (Table 3).  Due to the volatility

222 of the chemical (157 Pa /1.18 mmHg at 25°C), it is possible that the concentration of the compound actually

223 tested in the high‐throughput assays was lower than the calculated nominal concentration with these

224 considerations.  Of 13 inactive in vivo reference chemicals, eight had no ToxCast ER agonist bioactivity (AUC=0;

225 Table 3).  Kaempferol was negative in uterotrophic studies but had modest ER agonist model bioactivity

226 (AUC=0.25, Table 3) and scored as a false positive, though the positive result was consistent with other lower

throughput in vitro ER assays29, 38 227 .  Four inactive in vivo reference chemicals (dibutyl phthalate, dicyclohexyl

228 phthalate, dihexyl phthalate, and fenvalerate) had very low ToxCast ER model scores associated with

229 inconclusive calls (0<AUC<0.1; Table 3), This result supports the hypothesis that ToxCast ER model bioactivity in

230 this range has limited in vivo relevance.  If the four inconclusive chemicals were excluded from calculations of

231 performance metrics, the resulting overall accuracy for model performance for in vivo reference chemicals was

232 95% (37/39; Table 4).  

233 To expand the evaluation of the ToxCast ER model, we compared model agonist bioactivity with 103 chemicals

234 run in at least one guideline‐like uterotrophic study.  This larger set of chemicals included the 43 in vivo

235 reference chemicals as defined above, plus 60 additional chemicals that did not meet the stringent criteria for an

236 in vivo reference chemical because they were only run in one study or had discordant results that could not be

237 resolved (e.g., only one guideline‐like positive and at least one guideline‐like negative study for the same

238 chemical; Figure SI‐1).  Of the 55 chemicals with at least one positive uterotrophic response, 49 had ToxCast

239 model bioactivity; nine of the 49 had inconclusive scores (Table SI‐1).  Six chemicals with positive uterotrophic

240 studies (methylparaben, triclosan, reserpine, permethrin, octamethylcyclotetrasiloxane, and gibberellic acid)

241 had no reported bioactivity in the ToxCast model (i.e., AUC = 0).  Forty‐eight chemicals had no significant effect

242 on uterine weight in any study examined.  Of these, 37 had no ToxCast model bioactivity, seven were

243 inconclusive, and four chemicals with ToxCast model scores >0.1 (phenolphthalein, benzoic acid, kaempferol,

244 and benzylbutylphthalate) were potentially false positives.  It is worth noting that three of the four potential
7

245 false positives (phenolphthalein, kaempferol, and benzylbutylphthalate) were identified as in vitro positive

reference chemicals29 246 .  The overall accuracy of the ToxCast ER model when compared to uterotrophic guideline‐

247 like studies was 84% (86/103; Table 4).  If inconclusive calls are excluded from analyses, concordance between

248 the ToxCast ER model bioactivity and the in vivo guideline‐like uterotrophic studies for 75 chemicals with

249 ToxCast positive (>0.1) or negative (0) model scores was 88% (75/85; Table 4).     

250 Comparing ToxCast model scores and EDSP Tier 1 results, three List 1 chemicals did not have ToxCast assay data

251 and none of the remaining 49 chemicals had ToxCast ER model scores > 0.1.  Similarly, none of the chemicals

252 had clear positive agonist activity in the Tier 1 ER in vitro assays (ER binding and ERTA) or in vivo (uterotrophic)

253 assays.  ToxCast ER model scores were inconclusive for eight List 1 Tier 1 chemicals, all of which had limited

254 signal in the EDSP Tier 1 assays, but none of which would be considered positive based on the Tier 1 response

255 (Table 4; Table SI‐1).  All ToxCast ER assay responses for these chemicals were detected at concentrations similar

256 to those that resulted in cytotoxicity and may be explained by cell‐stress or cytotoxicity‐related false positive

257 activity.  Although there were both positive and negative Tier 1 ERTA assays reported for chemical codes 41 and

258 43, there were not clear indications of a positive Tier 1 ER binding, ERTA, or uterotrophic study (or any study

259 submitted to EPA to satisfy a Tier 1 test order) for any chemical.  Similarly ToxCast model scores were negative

260 for the remaining 41 chemicals.  Comparison between computational methods and Tier 1 assays is biased by the

261 lack of positive results, but for this analysis the ToxCast model accuracy against List 1 chemicals with Tier 1 data

262 is 84% (41/49) and 100% if inconclusive results are not included (Table 4).

263 ToxCast ER model scores were used to evaluate potential agonist activity in the 1812 chemicals with data for all

264 18 in vitro ToxCast ER assays, including 57 of 107 List 2 chemicals (Table SI‐2).  Concentration‐response data,

265 curve fits, and AC50 for all 18 ER assay as well as ToxCast ER model agonist scores are available for all 1812

266 chemicals (http://actor.epa.gov/edsp21/).   All of the 57 List 2 chemicals lacked ToxCast ER agonist bioactivity

(i.e., model scores < 0.1; Figure SI‐2)39 267 .  However among the remaining chemicals run in ToxCast high‐throughput

268 ER assays, about 7% (133) chemicals had ToxCast ER model scores indicating positive agonist bioactivity (i.e.,

269 scores > 0.1), 15% (276) were inconclusive,  and 77% (1403) have no observed ER agonist bioactivity (Figure SI‐

2)39 270 .   

271

272 Discussion

273 Before new computational toxicology tools can be used to screen for potential endocrine bioactivity, their utility

274 should be adequately demonstrated for the proposed purpose.  The key aspect of the analysis presented in this

275 paper is the performance‐based validation approach which uses multiple sets of well‐studied reference

276 chemicals to establish specificity and sensitivity of the ToxCast ER computational model, and comparisons of

277 model scores with existing test methods used for regulatory decision making.  Our analyses focused on ER

278 agonism because the EDSP Tier 1 battery assays that measure ER interactions are only capable of detecting

agonism, and most estrogen‐active environmental chemicals act as agonists9‐12 279 .  The ToxCast ER model

accurately predicted the bioactivity of reference chemicals across a range of structures and potencies14 280 ,  and for

281 a relatively large set of 193 chemicals with bioactivities independently confirmed by another test method (i.e., in

282 vitro and in vivo reference chemicals as well as results of guideline‐like uterotrophic studies, results of List 1/Tier

283 1 in vitro and uterotrophic studies; Figure SI‐1). Together, these analyses provide a high degree of confidence in

284 ToxCast ER model predictions and demonstrate the utility of using these computational tools to meet the

285 intended objectives which were to: 1) contribute to the weight of evidence evaluation of a chemical’s ER agonist
8

286 activity; and 2) provide an alternative source of data for specific EDSP Tier 1 endpoints measuring in vitro and in

287 vivo ER interaction.    

288 The time and resource intensive multi‐laboratory approaches traditionally used to validate toxicology assays

289 require seven to 10 years even for simple assays and are not suited to the rapid inclusion of new high‐

290 throughput tools and emerging technologies.  A number of groups have proposed more rapid performance‐

based validation approach for new assays40‐45 291 , including a large number of reference chemicals that spans a

292 range of structures and potencies.  The inclusion of relatively large sets of reference chemicals provides a high

293 degree of confidence in the validation, greatly increases knowledge of which chemicals are active in a given test,

294 helps to define the chemical space for which the assays can accurately predict an outcome (i.e., the domain of

applicability)46, 47 295 , and illustrates a major strength of high‐throughput assays capable of screening a variety of

296 chemical classes.  Furthermore, the performance‐based validation identifies a set of reference chemicals which

297 can be used to validate any assay or group of assays that accurately detect the reference chemicals, and such an

298 approach can and will be adopted for other endocrine pathways.  Given the thousands of chemicals to be

299 screened for potential endocrine bioactivity by the EDSP, validating high throughput approaches to be used in a

300 tiered testing strategy is the only practical path forward.

301 The performance‐based validation approach described in this study used a larger set of reference chemicals

302 than were used to validate the EDSP Tier 1 ER assays and the bioactivities were independently confirmed.  The in

303 vitro reference chemicals were active (or inactive) in several different types of ER assays, with reported

304 potencies ranging five orders of magnitude, the lower end of which can be used to establish a highly sensitive in

305 vitro assay but may have limited in vivo biological relevance.  The ToxCast ER model identified all strong,

306 moderate, and weak reference chemicals (i.e., AC50<1 μM); the only ambiguous results were inconclusive

307 bioactivity for one negative chemical and inconclusive (1 chemical) or no detected bioactivity (2 chemicals) for

308 three very weak reference chemicals.  In contrast to the 40 in vitro reference chemicals used in this analysis, the

309 Tier 1 ER in vitro transactivation and binding assays were initially validated with 12 and 23 chemicals,

respectively28, 48 310 , and results for 35% of the reference chemicals tested in the Tier 1 in vitro binding assay were

311 not consistent with the expected outcomes, either because of lack of agreement among assay results from

different labs or disagreement with observed results and anticipated activity of the selected chemicals48 312 .    The

313 43 in vivo reference chemicals included a similarly diverse range of structures and potencies, and greatly

exceeded the seven chemicals examined in the OECD validation of the in vivo uterotrophic assay49 314 .  The

315 relatively short duration and limited number of animals employed in the standard uterotrophic study design has

been reported to potentially elicit false negative results due to variability in uterus weights of control animals50, 316

51 317 .  The ToxCast ER model also performed better against uterotrophic results than lower throughput in vitro

assays52 318 which showed 66% agreement between results of competitive ER binding and uterotrophic assays

319 results for 65 chemicals.  Though the ToxCast ER model results also agreed with the List 1 Tier 1 uterotrophic

320 assay results, the guideline uterotrophic study includes a single endpoint (uterine weight) and may actually be

less sensitive and reproducible than in vitro assays52 321 , particularly given that 18 ToxCast ER assays targeting

322 multiple key events along the pathway are likely to reduce false negative responses.  

323 Although comparing the ToxCast ER model scores with in vivo uterotrophic results was a critical part of the

324 validation approach, further analysis of the full set of 103 guideline‐like uterotrophic studies indicated a

moderate degree of in vivo inter‐study variability13 325 .  Evaluation of uterotrophic study results for any single

326 chemical often differed with animal model, strain, dose of test chemical, and delivery route used in the study,

and highlighted the inherent variability in uterotrophic “guideline” method14, 53 327 .  Of chemicals with >1 guideline‐
9

like study, 26% had contradictory results with at least one positive and one negative study14 328 , which puts into

329 perspective the 84% accuracy of the ToxCast model when compared with all guideline‐like uterotrophic studies

330 in this analysis.  While much of the variability in the uterotrophic assay can be explained by differences in the

331 experimental design, among the 24 guideline‐like uterotrophic assays conducted for bisphenol A (BPA) delivered

332 by subcutaneous injection to the immature rat, discordant results ranged over three orders of magnitude (e.g., 4

333 mg/kg/d produced a positive response in one study and 1000 mg/kg/d failed to do so in another; Figure 1).  

334 Recognizing this variability is important because it sets realistic expectations as to the performance of any

335 alternative method.  For the uterotrophic assay, it is unrealistic that an alternative method should predict both

336 the true positives as well as account for the associated in vivo experimental variability.    

337 The ToxCast ER model incorporates the 18 high throughput in vitro assays in an unweighted manner.  During the

338 development of the model, multiple approaches for weighting the assays in a variety were investigated,

339 including performance against reference chemicals and levels of baseline or background noise, both of which

340 are indicators of overall reliability.  Analyses indicated there was little change in the model output (AUC score),

341 and the primary consequence was minor shifts in the ranking of chemicals at the expense of adding 18 free

342 parameters to the model whose exact structures and values that could not be well justified.  As a result, the

343 mathematically simpler, unweighted model was used in this analysis.

344 For analyses presented in this manuscript, AUC>0.1 (equivalent to an AC50 of about 100 μM) was considered

345 bioactive, AUC truncated at <0.001 was considered inactive, and chemicals with model scores 0<AUC<0.1 were

346 considered inconclusive.  When all bioactivities (e.g., model scores <0.001 ) of reference chemical and List 1 Tier

347 1 chemicals (which all have independent corroboration of the estrogen agonist activity) are included in the

348 evaluation of model performance, the number of false positive calls increases (Table 2), supporting the

349 hypothesis that very low bioactivity scores in the inconclusive range are not biologically relevant.  Similarly, if

350 the threshold for positive calls was changed from 0.1 to 0.01, two more in vitro reference chemicals (di‐n‐butyl

351 phthalate and haloperidol; Table 3) and four more in vivo reference chemicals (dibutyl phthalate, dicyclohexyl

352 phthatlate, dihexyl phthatlate, and fenvalerate; Table 4) were predicted to be bioactive.  Though this adjustment

353 would detect one very weak active in vitro reference chemical, it would also result in one additional false

354 positive in vitro reference chemical and four additional false negative in vivo reference chemicals.  Further, an

355 additional 139 chemicals of the 1812 run in ToxCast would be considered "active" (Table SI‐2).  None of the eight

356 List 1 Tier 1 chemicals with inconclusive ToxCast model scores had consistent positive outcomes in the EDSP Tier

357 1 assays, further supporting the absence of biologically relevant bioactivity associated with ToxCast ER model

358 scores in this range.  At this point, quantitative uncertainties in the in vitro data and qualitative uncertainties in

359 the uterotrophic assay data reduce the value of over interpreting inconclusive ToxCast ER agonist bioactivities.  

360 It should be noted that inconclusive chemicals would not escape additional testing, but additional data,

361 including results of other EDSP Tier 1 battery assays, would be used to help interpret inconclusive ToxCast model

362 scores (0.1>AUC>0).  Eventually, the goal is to resolve inconclusive calls and distinguish “true” negative from

363 very weak bioactivity, though this resolution may require additional data (e.g., assays, QSARs, or in vivo testing).  

364 Performance‐based validation of computational data demonstrates that the ToxCast ER model performs as well

365 or better than the EDSP Tier 1 ER binding, ERTA, and uterotrophic assays.  Accuracy of the ToxCast ER model is

366 84 to 93% when compared with all data sets described in this analysis (i.e., 40 in vitro reference chemicals, 43 in

367 vivo reference chemicals, the 63 guideline‐like uterotrophic studies, and results of Tier 1 battery assays for 49

368 chemicals), and when inconclusive model scores are removed from the analyses, the accuracy ranges from 88 to

369 100% (Table 4).  The high sensitivity of the model is critical for screening environmental chemicals, since it
10

370 means that few false negatives were observed (Table 4).  Given the redundancy of coverage among the 18

371 ToxCast ER assays (e.g., multiple receptor binding and transactivation assays), it is unlikely that running a single

372 EDSP Tier 1 guideline ER binding or ERTA assay would provide additional insight into a chemical's potential ER

373 bioactivity.  In addition, the 18 assays included in the network model provide a more comprehensive pathway

374 coverage for the biology of the ER signaling pathway (e.g., receptor binding, dimerization, mRNA production,

375 protein production, cell proliferation) and a more robust estimate of a chemical’s potential ER bioactivity than

376 existing Tier 1 ER binding and ERTA assays because the ToxCast model is capable of detecting false positives due

to assay‐specific interference or cytotoxicity that can be discriminated from “true” bioactivity14 377 .  When the

378 performance of the ToxCast ER model is considered in the context of our stated objectives, it is clear that the

379 model has demonstrated utility for contributing to the weight of evidence of a chemical’s potential interaction

380 with the ER pathway, and if ToxCast in vitro assay data were available for a given chemical, no additional

381 information on potential ER bioactivity would be gained by requiring a EDSP Tier 1 ER binding, ERTA or

382 uterotrophic assay.

383 The ToxCast high throughput screening assays and models built on the output of the assays do have some

384 limitations, though many of these are also limitations of the EDSP Tier 1 counterparts.  As with any in vitro assay,

385 highly volatile chemicals or chemicals with low solubility (in DMSO) are difficult to assess.  The current high‐

386 throughput ER assays have limited capacity to address chemicals  that may be biotransformed to active or

387 inactive metabolites, though the ToxCast ER bioactivity model does detect both methoxychlor (AUC=0.254) and

388 its more potent metabolite 2,2‐bis(4‐hydroxyphenyl)‐1,1,1‐trichloroethane (HPTE; AUC=0.568).  Additionally,

389 this is a similar limitation of the EDSP Tier 1 ER in vitro and uterotrophic assays, as the preferred method of

390 administration for the guideline in vivo uterotrophic assay is subcutaneous injection which bypasses first pass

391 hepatic metabolism.  The EDSP Tier 1 ER in vitro assays are specific to ERα and only the ToxCast dimerization and

392 protein production assays detect ERβ bioactivity, therefore both screens may not detect chemicals that are ERβ‐

393 selective or act through non‐classical ER mechanisms.

394 Our intention was to demonstrate the validity and utility of the ToxCast ER model for screening thousands of

395 environmental chemicals for potential ER agonist bioactivity, though these analyses do not support substitution

396 of all EDSP Tier 1 assays relevant to the estrogen pathway.  In the absence of an in vivo uterotrophic assay, the

EDSP Tier 1 in vivo rat pubertal assay54 397 , which includes a variety of endpoints in addition to uterine weight, could

398 detect chemicals that are metabolized to active metabolites through the oral administration of test chemical.  

399 The rat pubertal assay exposes a larger sample of animals (n=16 versus n=6) for a longer duration (20 days

400 versus three days), which may increase the sensitivity of the assay compared with the uterotrophic test.  The

401 EDSP Tier 1 battery is intended to screen potential endocrine effects in the highly conserved ER signally pathway

402 of humans and wildlife.  ToxCast in vitro ER assays measure effects using cells derived from the kidney, cervix,

403 liver, ovary, uterus, and breast; use rodent, bovine, and human receptor proteins; and detect interaction using a

404 variety of technologies (Table 1).  This diversity of the 18 high‐throughput ER assays accounts for estrogenic

405 effects more broadly across cell types, organs, and species than the single human ovarian and two rodent

406 uterine ER assays in the existing Tier 1 battery and may have greater relevance to wildlife.   

407 We demonstrated the ToxCast ER model ability to predict ER bioactivity of in vitro and in vivo reference

408 chemicals, the utility of using the ToxCast ER model bioactivity as an alternative to the EDSP Tier 1 ER binding,

409 ERTA, and uterotrophic endpoints, and the application of ToxCast ER model scores to prioritize chemicals in the

410 EDSP universe for additional screening and testing.  Results of the ER model indicate only about 7% of the 1812

411 chemicals run in ToxCast have potential significant ER agonist bioactivity and this subset does not include any
11

412 List 1 or List 2 chemicals, and the absence of predicted bioactivity among EDSP List 1 chemicals is consistent with

the initial review of Tier 1 battery data for a subset of List 1 chemicals55, 56 413 .  The ToxCast ER model can be used to

414 rapidly screen thousands of chemicals in the EDSP universe, allowing EPA to move away from screening lists of

415 few chemicals with relatively low or no potential endocrine activity, reduce reliance on animal‐based assays, and

416 identify chemicals with the greatest potential endocrine bioactivity that may be high priority candidates for

further screening and testing55, 56 417 .  This approach for using computational toxicology tools in the EDSP only

418 evaluated ER‐mediated bioactivity and it should be noted that while List 1 and List 2 chemicals appear to have

419 limited ER bioactivity, these chemicals may be active in other endocrine pathways.  In the future, we plan to use

420 a performance‐based validation approach of high‐throughput ToxCast ER assays and other computational

421 toxicology tools to compare with the existing “guideline‐like” Tier 1 assays, including the fish and pubertal rat, to

422 determine how well high‐throughput models predict estrogen bioactivity in neuroendocrine‐intact animals.  In

423 addition, EPA will use this performance‐based approach for validating new computational tools to screen for

424 androgen and thyroid effects, taking advantage of both existing and innovative, emerging technologies to

425 implement a scientifically robust and comprehensive chemical prioritization process for EDSP.  The application

426 of these innovative tools to screening chemicals for endocrine bioactivity represents the first step in a paradigm

427 shift for chemical safety testing, a practical approach to rapidly screen thousands of environmental chemicals for

428 potential endocrine bioactivity in humans and wildlife, and the first systematic application of ToxCast data in an

429 EPA regulatory program.

430

431 Acknowledgements

432 We thank D. Bergfelt, A. Dixon, T. Green, K. Hamernik, S. Lynn, K. Markey, L. Touart, and W. Wooge from OSCP;

433 K. Houck, K. Crofton, T. Knudsen, M. Martin, A. Richard, W. Setzer, I. Shah, J. Wambaugh, and T. Bahadori from

434 ORD for their contributions. We are grateful to the NICEATM/ILS team for curation of the uterotrophic database.  

435 We also thank R. Hines from ORD for his technical review of the manuscript.  

436

437 Supporting Information Available

438 Table SI‐1 summarizes List 1 chemical EDSP Tier 1 ER binding, ERTA, and uterotrophic results compared with

439 ToxCast ER model scores.  Table SI‐2 includes ToxCast ER model scores for all 1812 chemicals, along with

440 uterotrophic results.  Figure SI‐1 summarizes the various chemical populations used in the performance based

441 validation approach.  Figure SI‐2 is a graph of ToxCast ER model scores for active and inactive in vitro reference

442 chemicals, EDSP List 1 chemicals, EDSP List 2 chemicals, and the remaining positive ToxCast ER agonist scores

443 among the 1812 chemicals for which all 18 ToxCast high throughput ER assay data are available.  

444

445 This information is available free of charge via the Internet at http://pubs.acs.org .
446 Table 1. Summary of the 18 high‐throughput in vitro estrogen receptor (ER) assays included in the ToxCast ER bioactivity mod

information on assays, please reference the EDSP21 dashboard39 447 . NA = Not Applicable (cell‐free binding assay).

Assay

ID Assay Name Biological Process Target Detection Technology Organism Tissue 

A1 NVS_NR_bER receptor binding radioligand bovine uterus A2 NVS_NR_hER receptor binding radioligand human NA 

A3 NVS_NR_mERa receptor binding radioligand mouse NA 

A4 OT_ER_ERaERa_0480 protein complementation fluorescence human kidney 

A5 OT_ER_ERaERa_1440 protein complementation fluorescence human kidney 

A6 OT_ER_ERaERb_0480 protein complementation fluorescence human kidney 

A7 OT_ER_ERaERb_1440 protein complementation fluorescence human kidney 

A8 OT_ER_ERbERb_0480 protein complementation fluorescence human kidney 

A9 OT_ER_ERbERb_1440 protein complementation fluorescence human kidney 

A10 OT_ERa_EREGFP_0120 protein production fluorescence human cervix 

A11 OT_ERa_EREGFP_0480 protein production fluorescence human cervix 

A12 ATG_ERa_TRANS_up mRNA induction fluorescence human liver 

A13 ATG_ERE_CIS_up mRNA induction fluorescence human liver 

A14 Tox21_ERa_BLA_Agonist_ratio protein production fluorescence human kidney 

A15 Tox21_ERa_LUC_BG1_Agonist protein production bioluminescence human ovary 

A16 ACEA_T47D_80hr_Positive cell proliferation electrical impedance human breast 

A17 Tox21_ERa_BLA_Antagonist_ratio protein production fluorescence human kidney 

A18 Tox21_ERa_LUC_BG1_Antagonist protein production bioluminescence human ovary 

448
13

449 Table 2: In vitro estrogen receptor (ER) agonist reference chemicals.

CASRN Chemical Name Agonist Potency1 ToxCast ER Model

Score

57‐63‐6 17alpha‐Ethinyl estradiol Strong 1

84‐16‐2 meso‐Hexestrol Strong 0.99

56‐53‐1 Diethylstilbestrol (DES) Strong 0.94

50‐28‐2 17beta‐Estradiol Strong 0.94

57‐91‐0 17alpha‐Estradiol Moderate 1.06

53‐16‐7 Estrone Moderate 0.81

140‐66‐9 4‐tert‐Octylphenol Moderate 0.39

446‐72‐0 Genistein Weak 0.54

77‐40‐7 Bisphenol B Weak 0.49

80‐05‐7 Bisphenol A Weak 0.45

486‐66‐8 Daidzein Weak 0.44

521‐18‐6 5alpha‐Dihydrotestosterone Weak 0.40

789‐02‐6 o,p'‐DDT Weak 0.39

599‐64‐4 4‐Cumylphenol Weak 0.38

143‐50‐0 Kepone Weak 0.17

58‐18‐4 17alpha‐Methyltestosterone Very Weak 0.50

520‐36‐5 Apigenin Very Weak 0.31

72‐43‐5 Methoxychlor Very Weak 0.25

520‐18‐3 Kaempferol Very Weak 0.25

85‐68‐7 Butylbenzyl phthalate Very Weak 0.18

480‐40‐0 Chrysin Very Weak 0.13

60168‐88‐9 Fenarimol Very Weak 0.11

104‐40‐5 p‐n‐Nonylphenol Very Weak 0.1

120‐47‐8 Ethylparaben Very Weak 0.1

72‐55‐9 p,p'‐DDE Very Weak 0.1

84‐74‐2 Di‐n‐butyl phthalate Very Weak 0.03

115‐32‐2 Dicofol Very Weak 0

117‐81‐7 Diethylhexyl phthalate Very Weak 0

52‐86‐8 Haloperidol Inactive 0.01

52‐01‐7 Spironolactone Inactive 0

50‐22‐6 Corticosterone Inactive 0

13311‐84‐7 Flutamide Inactive 0

1912‐24‐9 Atrazine Inactive 0

32809‐16‐8 Procymidone Inactive 0

330‐55‐2 Linuron Inactive 0

50‐55‐5 Reserpine Inactive 0

52806‐53‐8 Hydroxyflutamide Inactive 0

57‐30‐7 Phenobarbital Sodium Inactive 0

65277‐42‐1 Ketoconazole Inactive 0

66‐81‐9 Cycloheximide Inactive 0
14

1 450 Reference chemical potency, determined by concentration required to elicit 50% of the maximal response

(AC50), in low throughput in vitro ER assays28, 29 451 .  Strong = AC50 <0.0001 μM, moderate = AC50 < 0.1 μM, weak =

AC50 < 1 μM, very weak = all other activities, inactive = no detected activity29 452 .   
15

453 Table 3: In vivo estrogen receptor (ER) agonist reference chemicals with at least two independent active or

inactive guideline‐like uterotrophic studies14 454 .  The numbers of guideline‐like active and inactive study results are

455 reported for each chemical.

CASRN Name Active Inactive Bioactivity ToxCast ER Model

Score

57‐91‐0 17alpha‐Estradiol 2 0 Active 1.06

57‐63‐6 Ethinyl Estradiol 59 0 Active 1

56‐53‐1 Diethylstilbestrol (DES) 8 1 Active 0.94

50‐28‐2 Estradiol 25 0 Active 0.94

474‐86‐2 Equilin 2 0 Active 0.82

53‐16‐7 Estrone 9 0 Active 0.81

50‐27‐1 Estriol 4 0 Active 0.79

72‐33‐3 Mestranol 3 0 Active 0.74

17924‐92‐4 Zearalenone 4 0 Active 0.71

1478‐61‐1 Bisphenol AF 4 0 Active 0.55

446‐72‐0 Genistein 27 1 Active 0.54

68‐22‐4 Norethindrone 2 0 Active 0.52

58‐18‐4 Methyltestosterone 3 0 Active 0.50

77‐40‐7 Bisphenol B 2 0 Active 0.49

80‐05‐7 Bisphenol A 37 6 Active 0.45

104‐43‐8 4‐Dodecylphenol 3 0 Active 0.41

521‐18‐6 Dihydrotestosterone 3 0 Active 0.4

131‐55‐5 Benzophenone‐2 6 0 Active 0.40

140‐66‐9 4‐(1,1,3,3‐Tetramethylbutyl)phenol 3 1 Active 0.39

789‐02‐6 o,p'‐DDT 15 1 Active 0.39

599‐64‐4 p‐Cumylphenol 2 0 Active 0.38

5153‐25‐3 Benzoic acid, 4‐hydroxy‐, 2‐

ethylhexyl ester

2 0 Active 0.37

80‐46‐6 4‐(1,1‐Dimethylpropyl)phenol 4 0 Active 0.28

131‐56‐6 2,4‐Dihydroxybenzophenone 3 0 Active 0.27

80‐09‐1 Bisphenol S 2 0 Active 0.26

72‐43‐5 Methoxychlor 18 1 Active 0.25

94‐26‐8 Butylparaben 8 2 Active 0.25

98‐54‐4 p‐tert‐Butylphenol 2 0 Active 0.16

104‐40‐5 Nonylphenol 5 4 Active 0.10

556‐67‐2 Octamethylcyclotetrasiloxane 3 0 Active 0

520‐18‐3 Kaempferol 0 3 Inactive 0.25

84‐74‐2 Dibutyl phthalate 0 2 Inactive 0.03

84‐61‐7 Dicyclohexyl phthalate 0 2 Inactive 0.02

84‐75‐3 Dihexyl phthalate 0 2 Inactive 0.01

51630‐58‐1 Fenvalerate 0 2 Inactive 0.01

103‐23‐1 Bis(2‐ethylhexyl)hexanedioate 0 2 Inactive 0

117‐81‐7 Bis(2‐ethylhexyl)phthalate 0 2 Inactive 0
16

CASRN Name Active Inactive Bioactivity ToxCast ER Model

Score

1461‐22‐9 Tributylchlorostannane 0 2 Inactive 0

1912‐24‐9 Atrazine 0 2 Inactive 0

61‐82‐5 Amitrole 0 2 Inactive 0

84‐66‐2 Diethyl phthalate 0 2 Inactive 0

87‐86‐5 Pentachlorophenol 0 2 Inactive 0

99‐96‐7 4‐Hydroxybenzoic acid 0 2 Inactive 0

456    
17

457 Table 4. Performance based validation of the ToxCast ER model based on 18 high‐throughput in vitro assays

458 measuring potential estrogen receptor (ER) agonist activities and in vitro reference chemicals (see text for

459 detailed explanation).  ToxCast ER model scores >0.1 were considered positive, negative scores = 0 (and values

460 <0.001 were truncated as 0), and model scores (0>AUC<0.1) were inconclusive.  Performance metrics were

461 calculated with all chemicals and excluding chemicals with inconclusive model scores (values shown in

462 parentheses).

Performance

In vitro reference

chemicals

In vivo reference

chemicals

GL uterotrophic

studies

Tier 1 studies

# True Pos 26 (25) 29 (29) 49 (38) 0 (0)

# True Neg 11 (11) 8 (8) 37 (37)   41 (41)

# False Pos 1 (0) 5 (1) 11 (4) 8 (0)

# False Neg 2 (2) 1 (1) 6 (6) 0 (0)

Accuracy 0.93 (0.95) 0.86 (0.95) 0.84 (0.88) 0.84 (1.0)

Sensitivity 0.93 (0.93) 0.97 (0.97) 0.89 (0.86) 0 (0)

Specificity 0.92 (1.0) 0.67 (0.89) 0.77 (0.90) 0.84 (1.0)

463    
18

464

465 Figure 1. Variability of results for bisphenol A (BPA) in uterotrophic studies conducted in the immature rat

466 model.  All studies are methodologically similar to the EDSP Tier 1 guideline and considered "guideline‐like", yet

467 have discordant results even with the same route of administration.  LEL = lowest effect level; MDT = maximum

468 dose tested.
19

469 Literature Cited  

470 1. U.S. EPA EDSP Tier 1 Battery of Assays

471 http://www.epa.gov/endo/pubs/assayvalidation/tier1battery.htm (2009),  

472 2. U.S. EPA EDSP Overview. http://www.epa.gov/endo/pubs/edspoverview/components.htm  

473 3. U.S. EPA Final List of Chemicals for Initial Tier 1 Screening.

474 http://www.epa.gov/endo/pubs/prioritysetting/finallist.html  

475 4. U.S. EPA Revised Second List of Chemicals for Tier 1 Screening.

476 http://www.epa.gov/endo/pubs/prioritysetting/revlist2.htm  

477 5. U.S. EPA, Universe of Chemicals and General Validation Principles. In U.S. Environmental Protection

478 Agency:

479 http://www.epa.gov/endo/pubs/edsp_chemical_universe_and_general_validations_white_paper_11_12.pdf,

480 2012.

481 6. NRC, Toxicity testing in the 21st century: A vision and a strategy. National Academies Press: Washington,

482 DC, 2007; Vol. 25, p 136‐138.

483 7. U.S. EPA, FY 2012 EPA Budget In Brief. In Officer, O. o. t. C. F., Ed. U.S. Environmental Protection Agency:

484 http://nepis.epa.gov/Exe/ZyPURL.cgi?Dockey=P100A5RE.txt, 2011; p 128.

485 8. U.S. EPA, The U.S. Environmental Protection Agency's Strategic Plan for Evaluating the Toxicity of

486 Chemicals In U.S. Environmental Protection Agency:

487 http://www.epa.gov/spc/toxicitytesting/docs/toxtest_strategy_032309.pdf, 2009.

488 9. Kojima, H.; Katsura, E.; Takeuchi, S.; Niiyama, K.; Kobayashi, K., Screening for estrogen and androgen

489 receptor activities in 200 pesticides by in vitro reporter gene assays using Chinese hamster ovary cells.

490 Environmental health perspectives 2004, 112, (5), 524‐31.

491 10. Safe, S. H.; Pallaroni, L.; Yoon, K.; Gaido, K.; Ross, S.; Saville, B.; McDonnellc, D., Toxicology of

492 environmental estrogens. Reproduction, fertility, and development 2001, 13, (4), 307‐15.

493 11. Sonnenschein, C.; Soto, A. M., An updated review of environmental estrogen and androgen mimics and

494 antagonists. The Journal of steroid biochemistry and molecular biology 1998, 65, (1‐6), 143‐50.

495 12. Diamanti‐Kandarakis, E.; Bourguignon, J. P.; Giudice, L. C.; Hauser, R.; Prins, G. S.; Soto, A. M.; Zoeller, R.

496 T.; Gore, A. C., Endocrine‐disrupting chemicals: an Endocrine Society scientific statement. Endocrine reviews

497 2009, 30, (4), 293‐342.

498 13. U.S. EPA EPA’s ToxCast Program. http://www.epa.gov/ncct/toxcast/  

499 14. U.S. EPA, Integrated Bioactivity and Exposure Ranking: A computational Approach for the Prioritization

500 and Screening of Chemicals in the Endocrine Disruptor Screening Program. In U.S. Environmental Protection

501 Agency: http://www.regulations.gov/#!documentDetail;D=EPA‐HQ‐OPP‐2014‐0614‐0003, 2014.

502 15. Dix, D. J.; Houck, K. A.; Martin, M. T.; Richard, A. M.; Setzer, R. W.; Kavlock, R. J., The ToxCast program

503 for prioritizing toxicity testing of environmental chemicals. Toxicol. Sci. 2007, 95, (1), 5‐12.

504 16. Houck, K. A.; Kavlock, R. J., Understanding mechanisms of toxicity: Insights from drug discovery research.

505 Toxicology and Applied Pharmacology 2008, 227, 163‐178.

506 17. Martin, M. T.; Knudsen, T. B.; Reif, D. M.; Houck, K. A.; Judson, R. S.; Kavlock, R. J.; Dix, D. J., Predictive

507 model of rat reprodcutive toxicity from ToxCast high throughput screening Biology of Reproduction 2011, 85, (2),

508 327‐339.

509 18. Rotroff, D. M.; Dix, D. J.; Houck, K. A.; Knudsen, T. B.; Martin, M. T.; McLaurin, K. W.; Reif, D. M.; Crofton,

510 K. M.; Singh, A. V.; Xia, M. H.; Huang, R. L.; Judson, R. S., Using in vitro high throughput screening assays to

511 identifiy potential endocrine disrupting chemicals. Environmental health perspectives 2013, 121, 7‐14.

512 19. Reif, D. M.; Martin, M. T.; Tan, S. W.; Houck, K. A.; Judson, R. S.; Richard, A. M.; Knudsen, T. B.; Dix, D. J.;

513 Kavlock, R. J., Endocrine profile and prioritization of environmental chemicals using ToxCast data. Environmental

514 health perspectives 2010, 118, (12), 1714‐1720.
20

515 20. Judson, R. S.; Houck, K. A.; Kavlock, R. J.; Knudsen, T. B.; Martin, M. T.; Mortensen, H. M.; Reif, D. M.;

516 Rotroff, D. M.; Shah, I.; Richard, A. M.; Dix, D. J., In vitro screening of environmental chemicals for targeted

517 testing prioritization: the ToxCast project. Environmental health perspectives 2010, 118, (4), 485‐92.

518 21. Knudsen, T.; Houck, K.; Judson, R. S.; Singh, A.; Mortensen, H.; Reif, D.; Dix, D. J.; Kavlock, R. J.,

519 Biochemical activities of 320 ToxCast chemicals evaluated across 239 functional targets. Toxicology 2011, 282, 1‐

520 15.

521 22. Sipes, N. S.; Martin, M. T.; Kothiya, P.; Reif, D. M.; Judson, R. S.; Richard, A. M.; Houck, K. A.; Dix, D. J.;

522 Kavlock, R. J.; Knudsen, T. B., Profiling 976 ToxCast chemicals across 331 enzymatic and receptor signaling

523 assays. Chemical research in toxicology 2013, 26, (6), 878‐95.

524 23. Martin, M. T.; Dix, D. J.; Judson, R. S.; Kavlock, R. J.; Reif, D. M.; Richard, A. M.; Rotroff, D. M.; Romanov,

525 S.; Medvedev, A.; Poltoratskaya, N.; Gambarian, M.; Moeser, M.; Makarov, S. S.; Houck, K. A., Impact of

526 environmental chemicals on key transcription regulators and correlation to toxicity end points within EPA's

527 ToxCast program. Chemical research in toxicology 2010, 23, (3), 578‐90.

528 24. Huang, R.; Sakamuru, S.; Martin, M. T.; Reif, D. M.; Judson, R. S.; Houck, K. A.; Casey, W.; Hsieh, J.‐H.;

529 Shockley, K. R.; Ceger, P.; Fostel, J.; Witt, K. L.; Tong, W.; Rotroff, D. M.; Zhao, T.; Shinn, P.; Simeonov, A.; Dix, D.

530 J.; Austin, C. P.; Kavlock, R. J.; Tice, R. R.; Xia, M., Profiling of the Tox21 10K compound library for agonists and

531 antagonists of the estrogen receptor alpha signaling pathway. Scientific Reports 2014, 4, (5664).

532 25. Rotroff, D. M.; Dix, D. J.; Houck, K. A.; Kavlock, R. J.; Knudsen, T. B.; Martin, M. T.; Reif, D. M.; Richard, A.

533 M.; Sipes, N. S.; Abassi, Y. A.; Jin, C.; Stampfl, M.; Judson, R. S., Real‐time growth kinetics measuring hormone

534 mimicry for ToxCast chemicals in T‐47D human ductal carcinoma cells. Chemical research in toxicology 2013, 26,

535 (7), 1097‐107.

536 26. U.S. EPA U.S. EPA ToxCast Program Data. http://epa.gov/ncct/toxcast/data.html  

537 27. Filer, D., The ToxCast Analysis Pipeline: An R Package for Processing And Modeling Chemical Screening

538 Data. In U.S. Environmental Protection Agency:

539 http://www.epa.gov/ncct/toxcast/files/MySQL%20Database/Pipeline_Overview.pdf, 2014.

540 28. OECD, Test No. 455: Performance‐Based Test Guideline for Stably Transfected Transactivation In Vitro

541 Assays to Detect Estrogen Receptor Agonists. In Organisation for Economic Co‐operation and Development:

542 http://www.oecd‐ilibrary.org/environment/test‐no‐455‐performance‐based‐test‐guideline‐for‐stably‐

543 transfected‐transactivation‐in‐vitro‐assays‐to‐detect‐estrogen‐receptor‐agonists_9789264185388‐en, 2012.

544 29. ICCVAM The LUMI‐CELL® ER (BG1Luc ER TA) Test Method: An In Vitro Assay for Identifying Human

545 Estrogen Receptor Agonist and Antagonist Activity of Chemicals; NEIHS:

546 http://ntp.niehs.nih.gov/pubhealth/evalatm/iccvam/test‐method‐evaluations/endocrine‐

547 disruptors/bg1luc/tmer/index.html, 2011.

548 30. OECD, OECD Test No. 457: BG1Luc Estrogen Receptor Transactivation Test Method for Identifying

549 Estrogen Receptor Agonists and Antagonists. In http://www.oecd‐ilibrary.org/environment/test‐no‐457‐bg1luc‐

550 estrogen‐receptor‐transactivation‐test‐method‐for‐identifying‐estrogen‐receptor‐agonists‐and‐

551 antagonists_9789264185395‐en, 2012; Vol. 2012.

552 31. OECD, Test No. 440: Uterotrophic Bioassay in Rodents A short‐term screening test for oestrogenic

553 properties In Organisation for Economic Co‐operation and Development: http://www.oecd‐

554 ilibrary.org/environment/test‐no‐440‐uterotrophic‐bioassay‐in‐rodents_9789264067417‐en, 2007.

555 32. U.S. EPA, Endocrine Disruptor Screening Program Test Guidelines OPPTS 890. 1600: Uterotrophic Assay.

556 In U.S. Environmental Protection Agency: http://www.regulations.gov/#!documentDetail;D=EPA‐HQ‐OPPT‐

557 2009‐0576‐0012, 2009.

558 33. U.S. EPA EPA ACToR. http://actor.epa.gov/actor  

559 34. FDA, U. S. US Food and Drug Administration’s Endocrine Disruptor Knowledge Base.

560 http://www.fda.gov/ScienceResearch/BioinformaticsTools/EndocrineDisruptorKnowledgebase/default.htm  
21

561 35. U.S. EPA, Endocrine Disruptor Screening Program Test Guidelines OPPTS 890. 1250: Rat Uterine Cytosol

562 Estrogen Receptor Binding Assay. In U.S. Environmental Protection Agency:

563 http://www.regulations.gov/#!documentDetail;D=EPA‐HQ‐OPPT‐2009‐0576‐0005, 2009.

564 36. U.S. EPA, Endocrine Disruptor Screening Program Test Guidelines OPPTS 890. 1300: Estrogen Receptor

565 Transactivation Assay. In U.S. Environmental Protection Agency:

566 http://www.regulations.gov/#!documentDetail;D=EPA‐HQ‐OPPT‐2009‐0576‐0006, 2009.

567 37. Stehman, S. V., Selecting and interpreting measures of thematic classification accuracy. Remote Sensing

568 of Environment 1997, 62, (1), 13.

569 38. Zava, D. T., Estrogenic and antiproliferative properties of genestine and other flavonoids in human

570 breast cancer cells in vitro. Nutrition and Cancer 1997, 27, (1), 31‐40.

571 39. U.S. EPA EDSP21 Dashboard. http://actor.epa.gov/edsp21/  

572 40. Becker, R. A.; Friedman, K. P.; Simon, T. W.; Marty, M. S.; Patlewicz, G.; Rowlands, J. C., An

573 exposure:activity profiling method for interpreting high‐throughput screening data for estrogenic activity‐‐proof

574 of concept. Regulatory toxicology and pharmacology : RTP 2015, 71, (3), 398‐408.

575 41. Patlewicz, G.; Ball, N.; Boogaard, P. J.; Becker, R. A.; Hubesch, B., Building scientific confidence in the

576 development and evaluation of read‐across. Regulatory toxicology and pharmacology : RTP 2015, 72, (1), 117‐

577 33.

578 42. Patlewicz, G.; Simon, T. W.; Rowlands, J. C.; Budinsky, R. A.; Becker, R. A., Proposing a scientific

579 confidence framework to help support the application of adverse outcome pathways for regulatory purposes.

580 Regulatory toxicology and pharmacology : RTP 2015, 71, (3), 463‐77.

581 43. Judson, R.; Kavlock, R.; Martin, M.; Reif, D.; Houck, K.; Knudsen, T.; Richard, A.; Tice, R. R.; Whelan, M.;

582 Xia, M.; Huang, R.; Austin, C.; Daston, G.; Hartung, T.; Fowle, J. R., 3rd; Wooge, W.; Tong, W.; Dix, D.,

583 Perspectives on validation of high‐throughput assays supporting 21st century toxicity testing. Altex 2013, 30, (1),

584 51‐6.

585 44. Sauer, J. M.; Hartung, T.; Leist, M.; Knudsen, T. B.; Hoeng, J.; Hayes, A. W., Systems Toxicology: The

586 Future of Risk Assessment. International journal of toxicology 2015, [Epub ahead of print].

587 45. Tralau, T.; Oelgeschlager, M.; Gurtler, R.; Heinemeyer, G.; Herzler, M.; Hofer, T.; Itter, H.; Kuhl, T.; Lange,

588 N.; Lorenz, N.; Muller‐Graf, C.; Pabel, U.; Pirow, R.; Ritz, V.; Schafft, H.; Schneider, H.; Schulz, T.; Schumacher, D.;

589 Zellmer, S.; Fleur‐Bol, G.; Greiner, M.; Lahrssen‐Wiederholt, M.; Lampen, A.; Luch, A.; Schonfelder, G.; Solecki, R.;

590 Wittkowski, R.; Hensel, A., Regulatory toxicology in the twenty‐first century: challenges, perspectives and

591 possible solutions. Arch Toxicol 2015, 89, (6), 823‐50.

592 46. OECD, Guidance Document on the Validation of (Quantitative) Structure‐Activity Relationships [(Q)SAR]

593 Models. In Organisation for Economic Co‐operation and Development:

594 http://www.oecd.org/officialdocuments/publicdisplaydocumentpdf/?doclanguage=en&cote=env/jm/mono(200

595 7)2, 2007; p 154.

596 47. OECD, Guidance Document on Developing and Assessing Adverse Outcome Pathways: Series on Testing

597 and Assessment No. 184. In Organisation for Economic Co‐operation and Development:

598 http://www.oecd.org/officialdocuments/publicdisplaydocumentpdf/?cote=env/jm/mono(2013)6&doclanguage

599 =en, 2013.

600 48. U.S. EPA, Integrated Summary Report for Validation of an Estrogen Receptor Binding Assay Using Rat

601 Uterine Cytosol as Source of Receptor as a Potential Screen in the Endocrine Disruptor Screening Program Tier 1

602 Battery. In U.S. Environmental Protection Agency: http://www.epa.gov/endo/pubs/assayvalidation/status.htm,

603 2009.

604 49. OECD, Series on Testing and Assessment. Number 67.  Report of the Validation of the Uterotrophic

605 Bioassay: Additional Data Supporting the TEst Guideline on the Uterotrophic Bioassay in Rodents. . In

606 http://epa.gov/endo/pubs/uterotrophic_OECD_validation_report.pdf, 2007.
22

607 50. Ashby, J.; Odum, J., Gene expression changes in the immature rat uterus: effects of uterotrophic and

608 sub‐uterotrophic doses of bisphenol A. Toxicological Sciences 2004, 82, (2), 458‐467.

609 51. Christian, M. S.; Hoberman, A. M.; Bachmann, S.; Hellwig, J., Variability in the uterotrophic response

610 assay (an in vivo estrogenic response assay) in untreated control and positive control (DES‐DP, 2.5 microG/kg,

611 bid) Wistar and Sprague‐Dawley rats. . Drug and Chemical Toxicology 1998, 21 Supple 1, 50‐101.

612 52. Akahori, Y.; Nakai, M.; Yamasaki, K.; Takatsuki, M.; Shimohigashi, Y.; Ohtaki, M., Relationship between

613 the results of in vitro receptor binding assay to human estrogen receptor α and in vivo uterotrophic assay:

614 comparative study with 65 selected chemicals. Toxicology in Vitro 2008, 22, 225‐332.

615 53. Laws, S. C.; Carey, S. A.; Ferrell, J. M.; Bodman, G. J.; Cooper, R. L., Estrogenic activity of octylphenol,

616 nonylphenol, bisphenol A and methoxychlor in rats. Toxicological sciences : an official journal of the Society of

617 Toxicology 2000, 54, (1), 154‐67.

618 54. U.S. EPA, Endocrine Disruptor Screening Program Test Guidelines: OPPTS 890.1450. Pubertal

619 Development and Thyroid Function in Intact Juvenile/Peripubertal Female Rats. In U.S. Environmental Protection

620 Agency: http://www.regulations.gov/#!documentDetail;D=EPA‐HQ‐OPPT‐2009‐0576‐0009, 2009.

621 55. U.S. EPA, SAP Minutes No: 2013‐05 ‐ Weight‐of‐Evidence: Evaluating Results of EDSP Tier 1 Screening. In

622 U.S. Environmental Protection Agency:

623 http://www.epa.gov/scipoly/sap/meetings/2013/july/073013minutes.pdf, 2013; p 65.

624 56. U.S. EPA, Final EDSP (Endocrine Disruptor Screening Program) White Paper on Tier I Assay and Battery

625 Performance for May 21‐23, 2013 FIFRA SAP Meeting. In U.S. Environmental Protection Agency:

626 http://www.regulations.gov/contentStreamer?objectId=09000064812a3e5d&disposition=attachment&contentT

627 ype=pdf, 2013; p 160.

628