1 SCREENING CHEMICALS FOR ESTROGEN RECEPTOR BIOACTIVITY USING A COMPUTATIONAL MODEL 1

Patience Browne 1*

, Richard S. Judson 2

, Warren Casey 3

, Nicole Kleinstreuer 4

, Russell S. Thomas 2 2

1 US EPA, Office of Chemical Safety and Pollution Prevention, Washington DC; 2 US EPA, Office of Research and 3

Development, Research Triangle Park, NC; 3 National Toxicology Program, Interagency Center for the Evaluation 4

of Alternative Toxicological Methods, Research Triangle Park, NC; 4 ILS/NICEATM, Research Triangle Park, NC 5

The views expressed in this paper are those of the authors and do not necessarily reflect the views or policies of the US Environmental 6

Protection Agency. 7

8

*Address correspondence to P. Browne, US EPA, Office of Science Coordination and Policy, 1200 Pennsylvania 9

Ave NW (7203M), Washington, DC 20460, Telephone: 202

‐ 564

‐ 8474, Fax: 202

‐ 564

‐ 8482, E

‐ mail: 10

browne.patience@epa.gov. 11

12

Abstract 13

The U.S. Environmental Protection Agency (EPA) is considering high

‐ throughput and computational methods to 14

evaluate the endocrine bioactivity of environmental chemicals. Here we describe a multi

‐ step, performance

‐ 15

based validation of new methods and demonstrate these new tools are sufficiently robust to be used in the 16

Endocrine Disruptor Screening Program (EDSP). Results from 18 estrogen receptor (ER) ToxCast high

‐ throughput 17

screening assays were integrated into a computational model that can discriminate bioactivity from assay

‐ 18

specific interference and cytotoxicity. Model scores range from 0 (no activity) to 1 (bioactivity of 17 β‐ estradiol). 19

ToxCast ER model performance was evaluated for reference chemicals, as well as results of EDSP Tier 1 20

screening assays in current practice. ToxCast ER model accuracy was 86 to 93% when compared to reference 21

chemicals, and predicted results of EDSP Tier 1 guideline and other uterotrophic studies with 84 to 100% 22

accuracy. Performance of high

‐ throughput assays and ToxCast ER model predictions demonstrates these 23

methods correctly identify active and inactive reference chemicals, provide a measure of relative ER bioactivity, 24

and rapidly identify chemicals with potential endocrine bioactivities for additional screening and testing. EPA is 25

accepting ToxCast ER model data for 1812 chemicals as alternatives for EDSP Tier 1 ER binding, ER 26

transactivation, and uterotrophic assays. 27

28

Introduction 29

The US Environmental Protection Agency’s (EPA) Endocrine Disruptor Screening Program (EDSP) was established 30

in 1999 for the purpose of evaluating potential risk of endocrine disruption in humans and wildlife from 31

exposure to pesticide chemicals and drinking water contaminants. To screen environmental chemicals for 32

potential endocrine bioactivity, EPA developed a battery of five in vitro and six in vivo Tier 1 screening assays 1

. 33

To test for potential endocrine disruption, EPA developed in vivo multigenerational Tier 2 tests that include 34

apical endpoints to identify adverse effects, and establish quantitative dose response relationships 2

. In 2009, 35

EPA published a final list of 67 pesticide chemicals (List 1) and issued EDSP Tier 1 test orders on these chemicals 3

. 36

Fifteen List 1 chemicals were voluntarily withdrawn from the pesticide market. EPA is currently reviewing 37

results of EDSP Tier 1 screening assays, along with other scientifically relevant information, and developing 382 weight of evidence evaluations of potential endocrine bioactivity with a determination of further testing that 39

may be required for the remaining 52 of the 67 chemicals. A second list of pesticide and high production 40

volume chemicals (List 2) proposed for Tier 1 screening was published in June 2013 4 ; however, test orders have 41

yet to be issued. The remaining EDSP universe of pesticide chemicals and drinking water contaminants includes 42

approximately 10,000 environmental chemicals to be screened for potential endocrine bioactivity in humans 43

and wildlife 5

. 44

In response to the US National Academy of Sciences report, Toxicity Testing in the 21 st Century 6

, and the US 45

President’s 2012 proposed budget 7

, EPA began a multi

‐ year transition from existing EDSP test methods towards 46

utilizing more rapid, cost

‐ effective, computational models and high

‐ throughput assays. The transition to using 47

computational toxicology approaches to prioritize and screen thousands of EDSP chemicals has been outlined by 48

the Agency in two strategic planning documents 5, 8

. However, to use new computational toxicology approaches 49

in the existing EDSP screening and testing framework, new methods must be validated and perform as well or 50

better than existing methods currently in practice. 51

A variety of pesticides and environmental chemicals act as estrogen receptor (ER) agonists 9

‐ 12

. Although the 52

scope of the program was expanded to consider androgen and thyroid active environmental chemicals, the EDSP 53

was originally established in response to statutory mandates in the Federal Food Quality Protection and Safe 54

Water Drinking Acts compelling EPA to evaluate potential xenoestrogens. As a result, the Tier 1 screening 55

battery was weighted towards assays that detect potential ER interactions. EDSP Tier 1 assay endpoints that 56

measure interaction with the ER only measure agonism and environmental chemicals that act through the ER 57

are primarily expected to act as agonists, therefore we focused this demonstration using computational 58

toxicology tools to evaluate ER agonist bioactivity. Future work will address estrogen antagonism and other 59

endocrine pathways. 60

In this manuscript, we present a performance

‐ based approach to validate computational toxicology tools for 61

evaluating a chemical’s potential ER agonist bioactivity and as alternative data for existing EDSP Tier 1 in vitro 62

and in vivo assays. High

‐ throughput in vitro screening assay data from EPA’s ToxCast program 13 were integrated 63

into a computational model of the ER pathway, and model performance was compared with in vitro and in vivo 64

reference chemicals identified from the peer

‐ reviewed literature 14

. A variety of publications have previously 65

described the ToxCast program 15, 16

, the validation of high

‐ throughput screening assays 17

, and endocrine

‐ 66

relevant ToxCast assays 18, 19 ; however, this paper is the first description of using ToxCast high

‐ throughput 67

screening data as an alternative for regulatory guideline studies. Integrating 21st century toxicology in the EDSP 68

allows for rapid evaluation of potential endocrine bioactivity of thousands of chemicals to which humans and 69

wildlife may be exposed and this approach is consistent with recommendations of the 2007 NAS report 6 and 70

EPA’s strategic plan for evaluating the toxicity of chemicals 8

, specifically to: 1) provide broad coverage of 71

chemicals examined; 2) reduce the cost and time of toxicity testing; 3) reduce animal use; and 4) develop a 72

robust scientific basis for assessing health effects of environmental agents. 73

74

Methods 75

In Vitro Assays 76

Details of the in vitro assays are described on EPA's ToxCast website 13 and in a variety of publications 15, 20

. 77

Briefly, potential ER bioactivity was measured in 18 high

‐ throughput in vitro assays run in EPA’s ToxCast 783 program. The suite of high throughput assays measure the molecular initiating event ( i.e., receptor binding), in 79

addition to several key events ( e.g.

, receptor dimerization, DNA binding, transactivation, gene expression, and 80

cell proliferation) in an adverse outcome pathway. The 18 ER assays include three cell

‐ free biochemical 81

radioligand ER binding assays 21, 22 ; three protein complementation assays that measure formation of ER dimers 82

and test for activity against ER α and ER β

, each measured at two time points; an assay measuring interaction of 83

the mature transcription factor with DNA at two time points; two reporter gene assays measuring RNA 84

transcript levels 23 ; two assays measuring reporter protein levels 24 ; an ER

‐ sensitive cell proliferation assay 25 ; and 85

two transactivation antagonist assays 24 (Table 1). 86

This combination of biochemical and cell

‐ based in vitro assays relies on different technologies and probes 87

different key events in the ER signaling pathway (Table 1). Though the assays are primarily human proteins 88

and/or cell types, the suite of 18 assays include human, murine, and bovine ER binding assays and ER pathway 89

interactions in a variety of human tissue types (Table 1). Every in vitro assay is potentially subject to technology

‐ 90

specific interference ( e.g.

, chemicals that denature the receptor protein, are luminescent, are cytotoxic, etc.) 91

that can be mistakenly interpreted as bioactivity. However, combining data from multiple assays and integrating 92

data in a network model of the entire ER pathway allows for detection of false positives and a more confident 93

assessment of the “true” in vitro estrogenic bioactivity of the tested chemical. 94

Concentration Response Analysis and Computational Modeling 95

Chemicals were run in concentration

‐ response format in all assays except the cell

‐ free binding assays. Cell

‐ free 96

competitive binding assays were initially run at a single screening concentration (25 μ M), and if the test chemical 97

was active in the assay ( i.e., radioligand was displaced), the assay was run in concentration

‐ response format. All 98

in vitro assays except the assays measuring RNA transcript were normalized to 17 

‐ estradiol. RNA transcript 99

data were normalized as a fold

‐ change from the solvent (DMSO) control. Concentration

‐ response data from in 100

vitro assays were fit to three models that included a four parameter Hill model, a modified Hill model with gain

‐ 101

loss at high concentrations, or a constant (no concentration

‐ response) model 26

. The best model was statistically 102

selected using the Akaike Information Criteria value. All concentration

‐ response data were analyzed using the 103

ToxCast data analysis pipeline, which automates the processes of baseline correction, normalization, curve

‐ 104

fitting, hit

‐ calling and detection of a variety of potential confounders 27

. To integrate results of the different in 105

vitro assays, concentration

‐ response curves were generated for each assay across 14 concentrations from 0.01

‐ 106

100 μ M. 107

The concentration

‐ response curves for all 18 assays were included in a computational model, referred to here as 108

the ToxCast ER model for bioactivity 14

. The computational model integrates data from the 18 in vitro assays 109

measuring ER agonist and antagonist responses in an unweighted manner, while subtracting background and 110

other non

‐ specific assay interference including cytotoxicity. The model output includes separate agonist and 111

antagonist area under the curve (AUC) scores, though only the agonist response was considered in this analysis. 112

The ER agonist AUC model scores range from 0 (no activity) to 1 (bioactivity of 17 β‐ estradiol). The computational 113

model is a simple linear additive model and assumes that the value for a given concentration is a linear sum of 114

the contributions from the ER interaction ( e.g., receptor binding, transactivition, mRNA production, etc.) 115

measured in each assay. Each assay contributes equally to the overall score for ER pathway activity if there is 116

direct molecular interaction between a chemical and an assay, and if the assay is not hit by the chemical it 117

provides no contribution to the score. Therefore, the model assumes lossless transmission of signals from the 118

key events in the signaling pathway to the assays. For each chemical

‐ concentration pair, a constrained least

‐ 119

squares minimization approach is used to reconcile the predicted assay values and the measured values, taking 1204 into account both potency and efficacy (mathematical details and a complete description of the model can be 121

found in EPA 2015) 14

. 122

Model scores were truncated at values < 0.001, considered to have no ER bioactivity and given a score of 0, as a 123

value <0.001 implies an concentration required to elicit 50% of the maximal response (AC 50 ) greater than 10 124

millimolar which is several orders of magnitude greater than the highest concentrations tested in ToxCast 125

assays. ToxCast ER agonist scores > 0.1 were considered positive; a model score of 0.1 equates to an AC 50 of 126

about 100 μM and approximates the upper limit of bioactivity detected in this approach. Model scores of 0.1> 127

AUC >0 were considered inconclusive for this validation because these chemicals were active in only one or two 128

ToxCast ER assays and activity was limited to the highest concentrations tested. Activity in only a few of the 18 129

assays may be due to differences (and thus differential sensitivities) in the various in vitro assays, though all 130

chemicals were tested up to at least 100 μM. It is unlikely that environmental exposures to most chemicals 131

would result in an internal dose at this level, and therefore such limited activity and low potency ( i.e.

, interaction 132

with the ER occurring only at high concentration of test chemical) has questionable relevance to in vivo 133

bioactivity. We evaluated model performance against all chemicals and after excluding chemicals with 134

inconclusive scores. 135

Performance

‐ Based Validation of the ToxCast ER Model 136

To assess the strengths and limitations of the ToxCast ER model, we adopted a performance

‐ based validation 137

approach consistent with the Organization for Economic and Cooperation and Development (OECD) conceptual 138

framework for assessing potential endocrine disrupting chemicals 28

. In principle, this method can be used to 139

assess the applicability of any test method or set of methods that meets defined performance standards. The 140

ToxCast ER model was validated using two approaches: comparison of ToxCast ER model scores against sets of 141

reference chemicals with independently confirmed ER bioactivity in a validated test method, and comparison of 142

ToxCast ER model scores with results of EDSP Tier 1 guideline assays currently used to screen chemicals for 143

endocrine bioactivity. 144

In vitro ER reference chemicals, previously identified using multiple validated low throughput in vitro ER assays, 145

were identified by the Interagency Coordinating Committee on the Validation of Alternative Test Methods 146

(ICCVAM) and OECD 29 for the express purpose of validating novel in vitro assays. Forty chemicals (28 agonists of 147

differing potencies indicated by a range in AC 50 and 12 inactive chemicals) were selected for reproducible results 148

in in vitro ER binding and transactivation assays, and to include a diverse set of chemical structures (Table 2) 29, 30

. 149

All in vitro reference chemicals were run in the 18 high

‐ throughput ToxCast ER assays and the resulting ToxCast 150

ER model scores for agonist bioactivity were compared with results anticipated from low or medium throughput 151

in vitro assays 29

. 152

In vivo reference chemicals were established from a literature search of short

‐ term rodent assays that were 153

methodologically similar to the OECD 31 and EDSP Tier 1 battery uterotrophic 32 assays. A comprehensive search 154

and review of uterotrophic studies published in peer

‐ reviewed literature was performed 14

. Chemical name and 155

chemical abstract services registry number (CASRN) were used to search PubMed, the EPA’s Aggregated 156

Computational Toxicology Resource (ACToR) 33

, and the US Food and Drug Administration’s Endocrine Disruptor 157

Knowledge Base (EDKB) 34 for the 1812 chemicals run in the ToxCast in vitro ER assays with “uterotrophic assay”, 158

“uterotrophic”, “uterotropic”, and “uterine weight” as modifier terms. Articles identified were reviewed for 159

methodological consistency with the EDSP uterotrophic assay guidelines 32 based on: 1) age and species of 160

animals used (immature rat or ovariectomized mouse or rat); 2) number of animals per treatment group; 3) 161

number of treatment groups; 4) route of chemical administration; 5) length of dosing; and 6) time of necropsy. 1625 Over 1000 articles were identified, entered into a database, and independently reviewed by two scientists. Of 163

the articles identified, 442 studies of 103 chemicals met all six minimum criteria 14

, were considered "guideline

‐ 164

like", and were used in this analysis. 165

Chemical data from guideline

‐ like uterotrophic studies were considered with two levels of stringency. First, a 166

chemical was considered positive for potential in vivo ER agonist bioactivity if a significant increase in uterine 167

weight among treated animals was reported and negative if no significant increase in uterine weight was 168

reported in any guideline

‐ like study. Second, a subset of chemicals with reproducible results from two or more 169

independent, guideline

‐ like studies were referred to as in vivo reference chemicals. Chemicals that resulted in a 170

significant increase in uterine weight in two or more studies were considered active reference chemicals, while 171

those chemicals that showed negative results in all studies (two or more) were considered inactive reference 172

chemicals (Table 3). 173

ToxCast ER model agonist scores were compared with results of the EDSP Tier 1 assays that directly assess a test 174

chemicals’ ER bioactivity. The Tier 1 in vitro ER binding assay 35 uses rat uterine cytosol (primarily ER α ) and 175

cannot distinguish between potential agonist or antagonist bioactivity. The competitive binding assay measures 176

test chemical displacement of radioligand ([ 3 H]17 β‐ estradiol) from the ER across a range of concentrations in 177

three independent runs. Results of the assay are “positive” if the test chemical displaces >50% of radioligand 178

(and Log(IC 50 ) is calculated), “equivocal” if test chemical displaces <50% but >25% of radioligand, and “negative” 179

if test chemical displaces <25% of radioligand 35

. The Tier 1 in vitro Estrogen Receptor Transcriptional Activation 180

(ERTA) assay 36 measures chemiluminescence in response to an ER α‐ mediated increase in luciferase gene 181

expression ( i.e., agonist activity). A test chemical is “positive” if the maximum response induced by the test 182

chemical is > 10% of the maximum response of the positive control (17 β‐ estradiol; RPC

max ) in at least two of 183

three assay runs ( i.e., RPC

max ≥ 10). If the test chemical fails to achieve at least 10% of the response of the 184

positive control, a negative response is recorded for the test chemical. The Tier 1 uterotrophic assay 32 is a short

‐ 185

term, in vivo assay designed to detect exogenous estrogen agonist activity indicated by an increase in uterine 186

weight in animals in prepubertal or ovariectomized rodents, in which the hypothalamic

‐ pituitary

‐ gonadal axis is 187

not functional. 188

Application of ToxCast ER model to EDSP Chemicals 189

ER model agonist scores were examined for 1812 chemicals evaluated in all 18 ToxCast ER assays. These 190

chemicals include 62 EDSP List 1 chemicals for which EPA issued Tier 1 test orders, and 57 List 2 chemicals 191

identified by EPA as candidates to receive the next group of Tier 1 test orders. The 1812 chemicals include 387 192

pesticide active ingredients, and 364 pesticide inerts; most of the remaining chemicals are relevant to the EDSP, 193

contingent on potential for exposure of substantial human populations through sources of drinking water. 194

195

Results 196

Concentration

‐ response curves of the 18 high

‐ throughput in vitro ER assays were integrated into a model of ER 197

pathway bioactivity 14

. Model outputs include an integrated measure of agonist bioactivity, antagonist 198

bioactivity, as well as "false positive" signaling due to cytotoxicity or technology

‐ specific interference 14

. For 199

reasons described previously in this paper, only ER agonist bioactivity was considered in these analyses. 200

Performance metrics (true positives, true negatives, false positives, false negatives, balanced accuracy, 201

sensitivity, and specificity) 37 were calculated for ToxCast ER model score performance against reference 2026 chemicals and guideline studies. Performance metrics were calculated for chemicals with any indication of 203

ToxCast ER agonist bioactivity (AUC>0) or no activity (AUC=0) and again excluding inconclusive model scores 204

(0<AUC<0.1) for which no call of bioactivity could be determined (Table 4). 205

The performance

‐ based assessment of the ToxCast ER model for agonist bioactivity relied on in vitro reference 206

chemicals, in vivo reference chemicals, guideline

‐ like uterotrophic studies, and results of EDSP Tier 1 assays 207

(Figure SI

‐ 1). For the 40 in vitro agonist reference chemicals, the ToxCast ER model performed very well, with an 208

overall balanced accuracy of 93% (Table 4). Of the 28 active reference chemicals, 26 of 28 had ToxCast ER model 209

bioactivity (Table 2). ToxCast ER agonist scores were positive (> 0.1) for all strong, moderate and weak agonist 210

reference chemicals (Table 2). ToxCast ER model bioactivity was inconclusive (0<AUC<0.1) for one very weak 211

active chemical (di

‐ n

‐ butyl phthalate). Two very weak reference chemicals (diethylhexyl phthalate (DEHP) and 212

dicofol) were false negatives (model score = 0). Of the 12 inactive reference chemicals, 11 chemicals had no ER 213

model agonist bioactivity. One inactive chemical (haloperidol) had an inconclusive ToxCast model score 214

(0<AUC<0.1). If the two chemicals with inconclusive ToxCast ER agonist model scores are excluded from 215

performance metrics, the overall accuracy is 95% (Table 4). 216

The overall accuracy of the ToxCast ER model agonist bioactivity evaluated for 43 in vivo reference chemicals 217

(Table 3) with independently verified results in two or more guideline

‐ like uterotrophic studies was 86% (Table 218

4). Of 30 active reference chemicals, 29 had positive ToxCast ER agonist model scores (AUC > 0.1; Table 3). The 219

potential false negative chemical, octamethylcyclotetrasiloxane (D4), was positive in multiple uterotrophic 220

studies run in independent labs but negative in the ToxCast ER bioactivity model (Table 3). Due to the volatility 221

of the chemical (157 Pa /1.18 mmHg at 25°C), it is possible that the concentration of the compound actually 222

tested in the high

‐ throughput assays was lower than the calculated nominal concentration with these 223

considerations. Of 13 inactive in vivo reference chemicals, eight had no ToxCast ER agonist bioactivity (AUC=0; 224

Table 3). Kaempferol was negative in uterotrophic studies but had modest ER agonist model bioactivity 225

(AUC=0.25, Table 3) and scored as a false positive, though the positive result was consistent with other lower 226

throughput in vitro ER assays 29, 38

. Four inactive in vivo reference chemicals (dibutyl phthalate, dicyclohexyl 227

phthalate, dihexyl phthalate, and fenvalerate) had very low ToxCast ER model scores associated with 228

inconclusive calls (0<AUC<0.1; Table 3), This result supports the hypothesis that ToxCast ER model bioactivity in 229

this range has limited in vivo relevance. If the four inconclusive chemicals were excluded from calculations of 230

performance metrics, the resulting overall accuracy for model performance for in vivo reference chemicals was 231

95% (37/39; Table 4). 232

To expand the evaluation of the ToxCast ER model, we compared model agonist bioactivity with 103 chemicals 233

run in at least one guideline

‐ like uterotrophic study. This larger set of chemicals included the 43 in vivo 234

reference chemicals as defined above, plus 60 additional chemicals that did not meet the stringent criteria for an 235

in vivo reference chemical because they were only run in one study or had discordant results that could not be 236

resolved ( e.g.

, only one guideline

‐ like positive and at least one guideline

‐ like negative study for the same 237

chemical; Figure SI

‐ 1). Of the 55 chemicals with at least one positive uterotrophic response, 49 had ToxCast 238

model bioactivity; nine of the 49 had inconclusive scores (Table SI

‐ 1). Six chemicals with positive uterotrophic 239

studies (methylparaben, triclosan, reserpine, permethrin, octamethylcyclotetrasiloxane, and gibberellic acid) 240

had no reported bioactivity in the ToxCast model ( i.e., AUC = 0). Forty

‐ eight chemicals had no significant effect 241

on uterine weight in any study examined. Of these, 37 had no ToxCast model bioactivity, seven were 242

inconclusive, and four chemicals with ToxCast model scores >0.1 (phenolphthalein, benzoic acid, kaempferol, 243

and benzylbutylphthalate) were potentially false positives. It is worth noting that three of the four potential 2447 false positives (phenolphthalein, kaempferol, and benzylbutylphthalate) were identified as in vitro positive 245

reference chemicals 29

. The overall accuracy of the ToxCast ER model when compared to uterotrophic guideline

‐ 246

like studies was 84% (86/103; Table 4). If inconclusive calls are excluded from analyses, concordance between 247

the ToxCast ER model bioactivity and the in vivo guideline

‐ like uterotrophic studies for 75 chemicals with 248

ToxCast positive (>0.1) or negative (0) model scores was 88% (75/85; Table 4). 249

Comparing ToxCast model scores and EDSP Tier 1 results, three List 1 chemicals did not have ToxCast assay data 250

and none of the remaining 49 chemicals had ToxCast ER model scores > 0.1. Similarly, none of the chemicals 251

had clear positive agonist activity in the Tier 1 ER in vitro assays (ER binding and ERTA) or in vivo (uterotrophic) 252

assays. ToxCast ER model scores were inconclusive for eight List 1 Tier 1 chemicals, all of which had limited 253

signal in the EDSP Tier 1 assays, but none of which would be considered positive based on the Tier 1 response 254

(Table 4; Table SI

‐ 1). All ToxCast ER assay responses for these chemicals were detected at concentrations similar 255

to those that resulted in cytotoxicity and may be explained by cell

‐ stress or cytotoxicity

‐ related false positive 256

activity. Although there were both positive and negative Tier 1 ERTA assays reported for chemical codes 41 and 257

43, there were not clear indications of a positive Tier 1 ER binding, ERTA, or uterotrophic study (or any study 258

submitted to EPA to satisfy a Tier 1 test order) for any chemical. Similarly ToxCast model scores were negative 259

for the remaining 41 chemicals. Comparison between computational methods and Tier 1 assays is biased by the 260

lack of positive results, but for this analysis the ToxCast model accuracy against List 1 chemicals with Tier 1 data 261

is 84% (41/49) and 100% if inconclusive results are not included (Table 4). 262

ToxCast ER model scores were used to evaluate potential agonist activity in the 1812 chemicals with data for all 263

18 in vitro ToxCast ER assays, including 57 of 107 List 2 chemicals (Table SI

‐ 2). Concentration

‐ response data, 264

curve fits, and AC 50 for all 18 ER assay as well as ToxCast ER model agonist scores are available for all 1812 265

chemicals (http://actor.epa.gov/edsp21/). All of the 57 List 2 chemicals lacked ToxCast ER agonist bioactivity 266

( i.e.

, model scores < 0.1; Figure SI

‐ 2) 39

. However among the remaining chemicals run in ToxCast high

‐ throughput 267

ER assays, about 7% (133) chemicals had ToxCast ER model scores indicating positive agonist bioactivity ( i.e.

, 268

scores > 0.1), 15% (276) were inconclusive, and 77% (1403) have no observed ER agonist bioactivity (Figure SI

‐ 269

2) 39

. 270

271

Discussion 272

Before new computational toxicology tools can be used to screen for potential endocrine bioactivity, their utility 273

should be adequately demonstrated for the proposed purpose. The key aspect of the analysis presented in this 274

paper is the performance

‐ based validation approach which uses multiple sets of well

‐ studied reference 275

chemicals to establish specificity and sensitivity of the ToxCast ER computational model, and comparisons of 276

model scores with existing test methods used for regulatory decision making. Our analyses focused on ER 277

agonism because the EDSP Tier 1 battery assays that measure ER interactions are only capable of detecting 278

agonism, and most estrogen

‐ active environmental chemicals act as agonists 9

‐ 12

. The ToxCast ER model 279

accurately predicted the bioactivity of reference chemicals across a range of structures and potencies 14

, and for 280

a relatively large set of 193 chemicals with bioactivities independently confirmed by another test method ( i.e.

, in 281

vitro and in vivo reference chemicals as well as results of guideline

‐ like uterotrophic studies, results of List 1/Tier 282

1 in vitro and uterotrophic studies; Figure SI

‐ 1). Together, these analyses provide a high degree of confidence in 283

ToxCast ER model predictions and demonstrate the utility of using these computational tools to meet the 284

intended objectives which were to: 1) contribute to the weight of evidence evaluation of a chemical’s ER agonist 2858 activity; and 2) provide an alternative source of data for specific EDSP Tier 1 endpoints measuring in vitro and in 286

vivo ER interaction. 287

The time and resource intensive multi

‐ laboratory approaches traditionally used to validate toxicology assays 288

require seven to 10 years even for simple assays and are not suited to the rapid inclusion of new high

‐ 289

throughput tools and emerging technologies. A number of groups have proposed more rapid performance

‐ 290

based validation approach for new assays 40

‐ 45

, including a large number of reference chemicals that spans a 291

range of structures and potencies. The inclusion of relatively large sets of reference chemicals provides a high 292

degree of confidence in the validation, greatly increases knowledge of which chemicals are active in a given test, 293

helps to define the chemical space for which the assays can accurately predict an outcome ( i.e., the domain of 294

applicability) 46, 47

, and illustrates a major strength of high

‐ throughput assays capable of screening a variety of 295

chemical classes. Furthermore, the performance

‐ based validation identifies a set of reference chemicals which 296

can be used to validate any assay or group of assays that accurately detect the reference chemicals, and such an 297

approach can and will be adopted for other endocrine pathways. Given the thousands of chemicals to be 298

screened for potential endocrine bioactivity by the EDSP, validating high throughput approaches to be used in a 299

tiered testing strategy is the only practical path forward. 300

The performance

‐ based validation approach described in this study used a larger set of reference chemicals 301

than were used to validate the EDSP Tier 1 ER assays and the bioactivities were independently confirmed. The in 302

vitro reference chemicals were active (or inactive) in several different types of ER assays, with reported 303

potencies ranging five orders of magnitude, the lower end of which can be used to establish a highly sensitive in 304

vitro assay but may have limited in vivo biological relevance. The ToxCast ER model identified all strong, 305

moderate, and weak reference chemicals ( i.e.

, AC50<1 μM); the only ambiguous results were inconclusive 306

bioactivity for one negative chemical and inconclusive (1 chemical) or no detected bioactivity (2 chemicals) for 307

three very weak reference chemicals. In contrast to the 40 in vitro reference chemicals used in this analysis, the 308

Tier 1 ER in vitro transactivation and binding assays were initially validated with 12 and 23 chemicals, 309

respectively 28, 48

, and results for 35% of the reference chemicals tested in the Tier 1 in vitro binding assay were 310

not consistent with the expected outcomes, either because of lack of agreement among assay results from 311

different labs or disagreement with observed results and anticipated activity of the selected chemicals 48

. The 312

43 in vivo reference chemicals included a similarly diverse range of structures and potencies, and greatly 313

exceeded the seven chemicals examined in the OECD validation of the in vivo uterotrophic assay 49

. The 314

relatively short duration and limited number of animals employed in the standard uterotrophic study design has 315

been reported to potentially elicit false negative results due to variability in uterus weights of control animals 50, 316

51

. The ToxCast ER model also performed better against uterotrophic results than lower throughput in vitro 317

assays 52 which showed 66% agreement between results of competitive ER binding and uterotrophic assays 318

results for 65 chemicals. Though the ToxCast ER model results also agreed with the List 1 Tier 1 uterotrophic 319

assay results, the guideline uterotrophic study includes a single endpoint (uterine weight) and may actually be 320

less sensitive and reproducible than in vitro assays 52

, particularly given that 18 ToxCast ER assays targeting 321

multiple key events along the pathway are likely to reduce false negative responses. 322

Although comparing the ToxCast ER model scores with in vivo uterotrophic results was a critical part of the 323

validation approach, further analysis of the full set of 103 guideline

‐ like uterotrophic studies indicated a 324

moderate degree of in vivo inter

‐ study variability 13

. Evaluation of uterotrophic study results for any single 325

chemical often differed with animal model, strain, dose of test chemical, and delivery route used in the study, 326

and highlighted the inherent variability in uterotrophic “guideline” method 14, 53

. Of chemicals with >1 guideline

‐ 3279 like study, 26% had contradictory results with at least one positive and one negative study 14

, which puts into 328

perspective the 84% accuracy of the ToxCast model when compared with all guideline

‐ like uterotrophic studies 329

in this analysis. While much of the variability in the uterotrophic assay can be explained by differences in the 330

experimental design, among the 24 guideline

‐ like uterotrophic assays conducted for bisphenol A (BPA) delivered 331

by subcutaneous injection to the immature rat, discordant results ranged over three orders of magnitude ( e.g., 4 332

mg/kg/d produced a positive response in one study and 1000 mg/kg/d failed to do so in another; Figure 1). 333

Recognizing this variability is important because it sets realistic expectations as to the performance of any 334

alternative method. For the uterotrophic assay, it is unrealistic that an alternative method should predict both 335

the true positives as well as account for the associated in vivo experimental variability. 336

The ToxCast ER model incorporates the 18 high throughput in vitro assays in an unweighted manner. During the 337

development of the model, multiple approaches for weighting the assays in a variety were investigated, 338

including performance against reference chemicals and levels of baseline or background noise, both of which 339

are indicators of overall reliability. Analyses indicated there was little change in the model output (AUC score), 340

and the primary consequence was minor shifts in the ranking of chemicals at the expense of adding 18 free 341

parameters to the model whose exact structures and values that could not be well justified. As a result, the 342

mathematically simpler, unweighted model was used in this analysis. 343

For analyses presented in this manuscript, AUC>0.1 (equivalent to an AC 50 of about 100 μM) was considered 344

bioactive, AUC truncated at <0.001 was considered inactive, and chemicals with model scores 0<AUC<0.1 were 345

considered inconclusive. When all bioactivities ( e.g.

, model scores <0.001 ) of reference chemical and List 1 Tier 346

1 chemicals (which all have independent corroboration of the estrogen agonist activity) are included in the 347

evaluation of model performance, the number of false positive calls increases (Table 2), supporting the 348

hypothesis that very low bioactivity scores in the inconclusive range are not biologically relevant. Similarly, if 349

the threshold for positive calls was changed from 0.1 to 0.01, two more in vitro reference chemicals (di

‐ n

‐ butyl 350

phthalate and haloperidol; Table 3) and four more in vivo reference chemicals (dibutyl phthalate, dicyclohexyl 351

phthatlate, dihexyl phthatlate, and fenvalerate; Table 4) were predicted to be bioactive. Though this adjustment 352

would detect one very weak active in vitro reference chemical, it would also result in one additional false 353

positive in vitro reference chemical and four additional false negative in vivo reference chemicals. Further, an 354

additional 139 chemicals of the 1812 run in ToxCast would be considered "active" (Table SI

‐ 2). None of the eight 355

List 1 Tier 1 chemicals with inconclusive ToxCast model scores had consistent positive outcomes in the EDSP Tier 356

1 assays, further supporting the absence of biologically relevant bioactivity associated with ToxCast ER model 357

scores in this range. At this point, quantitative uncertainties in the in vitro data and qualitative uncertainties in 358

the uterotrophic assay data reduce the value of over interpreting inconclusive ToxCast ER agonist bioactivities. 359

It should be noted that inconclusive chemicals would not escape additional testing, but additional data, 360

including results of other EDSP Tier 1 battery assays, would be used to help interpret inconclusive ToxCast model 361

scores (0.1>AUC>0). Eventually, the goal is to resolve inconclusive calls and distinguish “true” negative from 362

very weak bioactivity, though this resolution may require additional data (e.g., assays, QSARs, or in vivo testing). 363

Performance

‐ based validation of computational data demonstrates that the ToxCast ER model performs as well 364

or better than the EDSP Tier 1 ER binding, ERTA, and uterotrophic assays. Accuracy of the ToxCast ER model is 365

84 to 93% when compared with all data sets described in this analysis ( i.e.

, 40 in vitro reference chemicals, 43 in 366

vivo reference chemicals, the 63 guideline

‐ like uterotrophic studies, and results of Tier 1 battery assays for 49 367

chemicals), and when inconclusive model scores are removed from the analyses, the accuracy ranges from 88 to 368

100% (Table 4). The high sensitivity of the model is critical for screening environmental chemicals, since it 36910 means that few false negatives were observed (Table 4). Given the redundancy of coverage among the 18 370

ToxCast ER assays ( e.g.

, multiple receptor binding and transactivation assays), it is unlikely that running a single 371

EDSP Tier 1 guideline ER binding or ERTA assay would provide additional insight into a chemical's potential ER 372

bioactivity. In addition, the 18 assays included in the network model provide a more comprehensive pathway 373

coverage for the biology of the ER signaling pathway ( e.g.

, receptor binding, dimerization, mRNA production, 374

protein production, cell proliferation) and a more robust estimate of a chemical’s potential ER bioactivity than 375

existing Tier 1 ER binding and ERTA assays because the ToxCast model is capable of detecting false positives due 376

to assay

‐ specific interference or cytotoxicity that can be discriminated from “true” bioactivity 14

. When the 377

performance of the ToxCast ER model is considered in the context of our stated objectives, it is clear that the 378

model has demonstrated utility for contributing to the weight of evidence of a chemical’s potential interaction 379

with the ER pathway, and if ToxCast in vitro assay data were available for a given chemical, no additional 380

information on potential ER bioactivity would be gained by requiring a EDSP Tier 1 ER binding, ERTA or 381

uterotrophic assay. 382

The ToxCast high throughput screening assays and models built on the output of the assays do have some 383

limitations, though many of these are also limitations of the EDSP Tier 1 counterparts. As with any in vitro assay, 384

highly volatile chemicals or chemicals with low solubility (in DMSO) are difficult to assess. The current high

‐ 385

throughput ER assays have limited capacity to address chemicals that may be biotransformed to active or 386

inactive metabolites, though the ToxCast ER bioactivity model does detect both methoxychlor (AUC=0.254) and 387

its more potent metabolite 2,2

‐ bis(4

‐ hydroxyphenyl)

‐ 1,1,1

‐ trichloroethane (HPTE; AUC=0.568). Additionally, 388

this is a similar limitation of the EDSP Tier 1 ER in vitro and uterotrophic assays, as the preferred method of 389

administration for the guideline in vivo uterotrophic assay is subcutaneous injection which bypasses first pass 390

hepatic metabolism. The EDSP Tier 1 ER in vitro assays are specific to ER α and only the ToxCast dimerization and 391

protein production assays detect ER β bioactivity, therefore both screens may not detect chemicals that are ER β‐ 392

selective or act through non

‐ classical ER mechanisms. 393

Our intention was to demonstrate the validity and utility of the ToxCast ER model for screening thousands of 394

environmental chemicals for potential ER agonist bioactivity, though these analyses do not support substitution 395

of all EDSP Tier 1 assays relevant to the estrogen pathway. In the absence of an in vivo uterotrophic assay, the 396

EDSP Tier 1 in vivo rat pubertal assay 54

, which includes a variety of endpoints in addition to uterine weight, could 397

detect chemicals that are metabolized to active metabolites through the oral administration of test chemical. 398

The rat pubertal assay exposes a larger sample of animals (n=16 versus n=6) for a longer duration (20 days 399

versus three days), which may increase the sensitivity of the assay compared with the uterotrophic test. The 400

EDSP Tier 1 battery is intended to screen potential endocrine effects in the highly conserved ER signally pathway 401

of humans and wildlife. ToxCast in vitro ER assays measure effects using cells derived from the kidney, cervix, 402

liver, ovary, uterus, and breast; use rodent, bovine, and human receptor proteins; and detect interaction using a 403

variety of technologies (Table 1). This diversity of the 18 high

‐ throughput ER assays accounts for estrogenic 404

effects more broadly across cell types, organs, and species than the single human ovarian and two rodent 405

uterine ER assays in the existing Tier 1 battery and may have greater relevance to wildlife. 406

We demonstrated the ToxCast ER model ability to predict ER bioactivity of in vitro and in vivo reference 407

chemicals, the utility of using the ToxCast ER model bioactivity as an alternative to the EDSP Tier 1 ER binding, 408

ERTA, and uterotrophic endpoints, and the application of ToxCast ER model scores to prioritize chemicals in the 409

EDSP universe for additional screening and testing. Results of the ER model indicate only about 7% of the 1812 410

chemicals run in ToxCast have potential significant ER agonist bioactivity and this subset does not include any 41111 List 1 or List 2 chemicals, and the absence of predicted bioactivity among EDSP List 1 chemicals is consistent with 412

the initial review of Tier 1 battery data for a subset of List 1 chemicals 55, 56

. The ToxCast ER model can be used to 413

rapidly screen thousands of chemicals in the EDSP universe, allowing EPA to move away from screening lists of 414

few chemicals with relatively low or no potential endocrine activity, reduce reliance on animal

‐ based assays, and 415

identify chemicals with the greatest potential endocrine bioactivity that may be high priority candidates for 416

further screening and testing 55, 56

. This approach for using computational toxicology tools in the EDSP only 417

evaluated ER

‐ mediated bioactivity and it should be noted that while List 1 and List 2 chemicals appear to have 418

limited ER bioactivity, these chemicals may be active in other endocrine pathways. In the future, we plan to use 419

a performance

‐ based validation approach of high

‐ throughput ToxCast ER assays and other computational 420

toxicology tools to compare with the existing “guideline

‐ like” Tier 1 assays, including the fish and pubertal rat, to 421

determine how well high

‐ throughput models predict estrogen bioactivity in neuroendocrine

‐ intact animals. In 422

addition, EPA will use this performance

‐ based approach for validating new computational tools to screen for 423

androgen and thyroid effects, taking advantage of both existing and innovative, emerging technologies to 424

implement a scientifically robust and comprehensive chemical prioritization process for EDSP. The application 425

of these innovative tools to screening chemicals for endocrine bioactivity represents the first step in a paradigm 426

shift for chemical safety testing, a practical approach to rapidly screen thousands of environmental chemicals for 427

potential endocrine bioactivity in humans and wildlife, and the first systematic application of ToxCast data in an 428

EPA regulatory program. 429

430

Acknowledgements 431

We thank D. Bergfelt, A. Dixon, T. Green, K. Hamernik, S. Lynn, K. Markey, L. Touart, and W. Wooge from OSCP; 432

K. Houck, K. Crofton, T. Knudsen, M. Martin, A. Richard, W. Setzer, I. Shah, J. Wambaugh, and T. Bahadori from 433

ORD for their contributions. We are grateful to the NICEATM/ILS team for curation of the uterotrophic database. 434

We also thank R. Hines from ORD for his technical review of the manuscript. 435

436

Supporting Information Available 437

Table SI

‐ 1 summarizes List 1 chemical EDSP Tier 1 ER binding, ERTA, and uterotrophic results compared with 438

ToxCast ER model scores. Table SI

‐ 2 includes ToxCast ER model scores for all 1812 chemicals, along with 439

uterotrophic results. Figure SI

‐ 1 summarizes the various chemical populations used in the performance based 440

validation approach. Figure SI

‐ 2 is a graph of ToxCast ER model scores for active and inactive in vitro reference 441

chemicals, EDSP List 1 chemicals, EDSP List 2 chemicals, and the remaining positive ToxCast ER agonist scores 442

among the 1812 chemicals for which all 18 ToxCast high throughput ER assay data are available. 443

444

This information is available free of charge via the Internet at http://pubs.acs.org

. 44512Table1.Summaryofthe18high‐throughputinvitroestrogenreceptor(ER)assaysincludedintheToxCastERbioactivitymodel.Foradditional446informationonassays,pleasereferencetheEDSP21dashboard39.NA=NotApplicable(cell‐freebindingassay).447AssayIDAssayNameBiologicalProcessTargetDetectionTechnologyOrganismTissueCellLineA1NVS_NR_bERreceptorbindingradioligandbovineuterusNAA2NVS_NR_hERreceptorbindingradioligandhumanNANAA3NVS_NR_mERareceptorbindingradioligandmouseNANAA4OT_ER_ERaERa_0480proteincomplementationfluorescencehumankidneyHEK293TA5OT_ER_ERaERa_1440proteincomplementationfluorescencehumankidneyHEK293TA6OT_ER_ERaERb_0480proteincomplementationfluorescencehumankidneyHEK293TA7OT_ER_ERaERb_1440proteincomplementationfluorescencehumankidneyHEK293TA8OT_ER_ERbERb_0480proteincomplementationfluorescencehumankidneyHEK293TA9OT_ER_ERbERb_1440proteincomplementationfluorescencehumankidneyHEK293TA10OT_ERa_EREGFP_0120proteinproductionfluorescencehumancervixHeLaA11OT_ERa_EREGFP_0480proteinproductionfluorescencehumancervixHeLaA12ATG_ERa_TRANS_upmRNAinductionfluorescencehumanliverHepG2A13ATG_ERE_CIS_upmRNAinductionfluorescencehumanliverHepG2A14Tox21_ERa_BLA_Agonist_ratioproteinproductionfluorescencehumankidneyHEK293TA15Tox21_ERa_LUC_BG1_AgonistproteinproductionbioluminescencehumanovaryBG1A16ACEA_T47D_80hr_PositivecellproliferationelectricalimpedancehumanbreastT47DA17Tox21_ERa_BLA_Antagonist_ratioproteinproductionfluorescencehumankidneyHEK293TA18Tox21_ERa_LUC_BG1_AntagonistproteinproductionbioluminescencehumanovaryBG144813 Table 2: In vitro estrogen receptor (ER) agonist reference chemicals. 449

CASRN Chemical Name Agonist Potency 1 ToxCast ER Model

Score

57

‐ 63

‐ 6 17alpha

‐ Ethinyl estradiol Strong 1

84

‐ 16

‐ 2 meso

‐ Hexestrol Strong 0.99

56

‐ 53

‐ 1 Diethylstilbestrol (DES) Strong 0.94

50

‐ 28

‐ 2 17beta

‐ Estradiol Strong 0.94

57

‐ 91

‐ 0 17alpha

‐ Estradiol Moderate 1.06

53

‐ 16

‐ 7 Estrone Moderate 0.81

140

‐ 66

‐ 9 4

‐ tert

‐ Octylphenol Moderate 0.39

446

‐ 72

‐ 0 Genistein Weak 0.54

77

‐ 40

‐ 7 Bisphenol B Weak 0.49

80

‐ 05

‐ 7 Bisphenol A Weak 0.45

486

‐ 66

‐ 8 Daidzein Weak 0.44

521

‐ 18

‐ 6 5alpha

‐ Dihydrotestosterone Weak 0.40

789

‐ 02

‐ 6 o,p'

‐ DDT Weak 0.39

599

‐ 64

‐ 4 4

‐ Cumylphenol Weak 0.38

143

‐ 50

‐ 0 Kepone Weak 0.17

58

‐ 18

‐ 4 17alpha

‐ Methyltestosterone Very Weak 0.50

520

‐ 36

‐ 5 Apigenin Very Weak 0.31

72

‐ 43

‐ 5 Methoxychlor Very Weak 0.25

520

‐ 18

‐ 3 Kaempferol Very Weak 0.25

85

‐ 68

‐ 7 Butylbenzyl phthalate Very Weak 0.18

480

‐ 40

‐ 0 Chrysin Very Weak 0.13

60168

‐ 88

‐ 9 Fenarimol Very Weak 0.11

104

‐ 40

‐ 5 p

‐ n

‐ Nonylphenol Very Weak 0.1

120

‐ 47

‐ 8 Ethylparaben Very Weak 0.1

72

‐ 55

‐ 9 p,p'

‐ DDE Very Weak 0.1

84

‐ 74

‐ 2 Di

‐ n

‐ butyl phthalate Very Weak 0.03

115

‐ 32

‐ 2 Dicofol Very Weak 0

117

‐ 81

‐ 7 Diethylhexyl phthalate Very Weak 0

52

‐ 86

‐ 8 Haloperidol Inactive 0.01

52

‐ 01

‐ 7 Spironolactone Inactive 0

50

‐ 22

‐ 6 Corticosterone Inactive 0

13311

‐ 84

‐ 7 Flutamide Inactive 0

1912

‐ 24

‐ 9 Atrazine Inactive 0

32809

‐ 16

‐ 8 Procymidone Inactive 0

330

‐ 55

‐ 2 Linuron Inactive 0

50

‐ 55

‐ 5 Reserpine Inactive 0

52806

‐ 53

‐ 8 Hydroxyflutamide Inactive 0

57

‐ 30

‐ 7 Phenobarbital Sodium Inactive 0

65277

‐ 42

‐ 1 Ketoconazole Inactive 0

66

‐ 81

‐ 9 Cycloheximide Inactive 014 1 Reference chemical potency, determined by concentration required to elicit 50% of the maximal response 450

(AC 50 ), in low throughput in vitro ER assays 28, 29

. Strong = AC 50 <0.0001 μ M, moderate

= AC 50 < 0.1 μ M, weak

= 451

AC 50 < 1 μ M, very weak

= all other activities, inactive = no detected activity 29

. 45215 Table 3: In vivo estrogen receptor (ER) agonist reference chemicals with at least two independent active or 453

inactive guideline

‐ like uterotrophic studies 14

. The numbers of guideline

‐ like active and inactive study results are 454

reported for each chemical. 455

CASRN Name Active Inactive Bioactivity ToxCast ER Model

Score

57

‐ 91

‐ 0 17alpha

‐ Estradiol 2 0 Active 1.06

57

‐ 63

‐ 6 Ethinyl Estradiol 59 0 Active 1

56

‐ 53

‐ 1 Diethylstilbestrol (DES) 8 1 Active 0.94

50

‐ 28

‐ 2 Estradiol 25 0 Active 0.94

474

‐ 86

‐ 2 Equilin 2 0 Active 0.82

53

‐ 16

‐ 7 Estrone 9 0 Active 0.81

50

‐ 27

‐ 1 Estriol 4 0 Active 0.79

72

‐ 33

‐ 3 Mestranol 3 0 Active 0.74

17924

‐ 92

‐ 4 Zearalenone 4 0 Active 0.71

1478

‐ 61

‐ 1 Bisphenol AF 4 0 Active 0.55

446

‐ 72

‐ 0 Genistein 27 1 Active 0.54

68

‐ 22

‐ 4 Norethindrone 2 0 Active 0.52

58

‐ 18

‐ 4 Methyltestosterone 3 0 Active 0.50

77

‐ 40

‐ 7 Bisphenol B 2 0 Active 0.49

80

‐ 05

‐ 7 Bisphenol A 37 6 Active 0.45

104

‐ 43

‐ 8 4

‐ Dodecylphenol 3 0 Active 0.41

521

‐ 18

‐ 6 Dihydrotestosterone 3 0 Active 0.4

131

‐ 55

‐ 5 Benzophenone

‐ 2 6 0 Active 0.40

140

‐ 66

‐ 9 4

‐ (1,1,3,3

‐ Tetramethylbutyl)phenol 3 1 Active 0.39

789

‐ 02

‐ 6 o,p'

‐ DDT 15 1 Active 0.39

599

‐ 64

‐ 4 p

‐ Cumylphenol 2 0 Active 0.38

5153

‐ 25

‐ 3 Benzoic acid, 4

‐ hydroxy

‐ , 2

‐ ethylhexyl ester 2 0 Active 0.37

80

‐ 46

‐ 6 4

‐ (1,1

‐ Dimethylpropyl)phenol 4 0 Active 0.28

131

‐ 56

‐ 6 2,4

‐ Dihydroxybenzophenone 3 0 Active 0.27

80

‐ 09

‐ 1 Bisphenol S 2 0 Active 0.26

72

‐ 43

‐ 5 Methoxychlor 18 1 Active 0.25

94

‐ 26

‐ 8 Butylparaben 8 2 Active 0.25

98

‐ 54

‐ 4 p

‐ tert

‐ Butylphenol 2 0 Active 0.16

104

‐ 40

‐ 5 Nonylphenol 5 4 Active 0.10

556

‐ 67

‐ 2 Octamethylcyclotetrasiloxane 3 0 Active 0

520

‐ 18

‐ 3 Kaempferol 0 3 Inactive 0.25

84

‐ 74

‐ 2 Dibutyl phthalate 0 2 Inactive 0.03

84

‐ 61

‐ 7 Dicyclohexyl phthalate 0 2 Inactive 0.02

84

‐ 75

‐ 3 Dihexyl phthalate 0 2 Inactive 0.01

51630

‐ 58

‐ 1 Fenvalerate 0 2 Inactive 0.01

103

‐ 23

‐ 1 Bis(2

‐ ethylhexyl)hexanedioate 0 2 Inactive 0

117

‐ 81

‐ 7 Bis(2

‐ ethylhexyl)phthalate 0 2 Inactive 016 CASRN Name Active Inactive Bioactivity ToxCast ER Model

Score

1461

‐ 22

‐ 9 Tributylchlorostannane 0 2 Inactive 0

1912

‐ 24

‐ 9 Atrazine 0 2 Inactive 0

61

‐ 82

‐ 5 Amitrole 0 2 Inactive 0

84

‐ 66

‐ 2 Diethyl phthalate 0 2 Inactive 0

87

‐ 86

‐ 5 Pentachlorophenol 0 2 Inactive 0

99

‐ 96

‐ 7 4

‐ Hydroxybenzoic acid 0 2 Inactive 0

45617 Table 4. Performance based validation of the ToxCast ER model based on 18 high

‐ throughput in vitro assays 457

measuring potential estrogen receptor (ER) agonist activities and in vitro reference chemicals (see text for 458

detailed explanation). ToxCast ER model scores >0.1 were considered positive, negative scores = 0 (and values 459

<0.001 were truncated as 0), and model scores (0>AUC<0.1) were inconclusive. Performance metrics were 460

calculated with all chemicals and excluding chemicals with inconclusive model scores (values shown in 461

parentheses). 462

Performance In vitro reference

chemicals In vivo reference

chemicals GL uterotrophic

studies Tier 1 studies

# True Pos 26 (25) 29 (29) 49 (38) 0 (0)

# True Neg 11 (11) 8 (8) 37 (37) 41 (41)

# False Pos 1 (0) 5 (1) 11 (4) 8 (0)

# False Neg 2 (2) 1 (1) 6 (6) 0 (0)

Accuracy 0.93 (0.95) 0.86 (0.95) 0.84 (0.88) 0.84 (1.0)

Sensitivity 0.93 (0.93) 0.97 (0.97) 0.89 (0.86) 0 (0)

Specificity 0.92 (1.0) 0.67 (0.89) 0.77 (0.90) 0.84 (1.0)

46318 464

Figure 1. Variability of results for bisphenol A (BPA) in uterotrophic studies conducted in the immature rat 465

model. All studies are methodologically similar to the EDSP Tier 1 guideline and considered "guideline

‐ like", yet 466

have discordant results even with the same route of administration. LEL = lowest effect level; MDT = maximum 467

dose tested. 46819 Literature Cited 469

1. U.S. EPA EDSP Tier 1 Battery of Assays 470

http://www.epa.gov/endo/pubs/assa yvalidation/tier1battery.htm (2009), 471

2. U.S. EPA EDSP Overview. http://www.epa.gov/endo/pubs/e dspoverview/components.htm 472

3. U.S. EPA Final List of Chemicals for Initial Tier 1 Screening. 473

http://www.epa.gov/endo/pubs/p rioritysetting/finallist.html 474

4. U.S. EPA Revised Second List of Chemicals for Tier 1 Screening. 475

http://www.epa.gov/endo/pubs/p rioritysetting/revlist2.htm 476

5. U.S. EPA, Universe of Chemicals and General Validation Principles. In U.S. Environmental Protection 477

Agency: 478

http://www.epa.gov/endo/pubs/eds p_chemical_universe_and_general_ validations_white_paper_11_12.pdf

, 479

2012. 480

6. NRC, Toxicity testing in the 21st century: A vision and a strategy

. National Academies Press: Washington, 481

DC, 2007; Vol. 25, p 136

‐ 138. 482

7. U.S. EPA, FY 2012 EPA Budget In Brief. In Officer, O. o. t. C. F., Ed. U.S. Environmental Protection Agency: 483

http://nepis.epa.gov/Exe/ZyP URL.cgi?Dockey=P100A5RE.txt

, 2011; p 128. 484

8. U.S. EPA, The U.S. Environmental Protection Agency's Strategic Plan for Evaluating the Toxicity of 485

Chemicals In U.S. Environmental Protection Agency: 486

http://www.epa.gov/spc/toxicitytest ing/docs/toxtest_strategy_032309.pdf

, 2009. 487

9. Kojima, H.; Katsura, E.; Takeuchi, S.; Niiyama, K.; Kobayashi, K., Screening for estrogen and androgen 488

receptor activities in 200 pesticides by in vitro reporter gene assays using Chinese hamster ovary cells. 489

Environmental health perspectives 2004, 112

, (5), 524

‐ 31. 490

10. Safe, S. H.; Pallaroni, L.; Yoon, K.; Gaido, K.; Ross, S.; Saville, B.; McDonnellc, D., Toxicology of 491

environmental estrogens. Reproduction, fertility, and development 2001, 13

, (4), 307

‐ 15. 492

11. Sonnenschein, C.; Soto, A. M., An updated review of environmental estrogen and androgen mimics and 493

antagonists. The Journal of steroid biochemistry and molecular biology 1998, 65

, (1

‐ 6), 143

‐ 50. 494

12. Diamanti

‐ Kandarakis, E.; Bourguignon, J. P.; Giudice, L. C.; Hauser, R.; Prins, G. S.; Soto, A. M.; Zoeller, R. 495

T.; Gore, A. C., Endocrine

‐ disrupting chemicals: an Endocrine Society scientific statement. Endocrine reviews 496

2009, 30

, (4), 293

‐ 342. 497

13. U.S. EPA EPA’s ToxCast Program. http://www.epa.gov/ncct/toxcast/ 498

14. U.S. EPA, Integrated Bioactivity and Exposure Ranking: A computational Approach for the Prioritization 499

and Screening of Chemicals in the Endocrine Disruptor Screening Program. In U.S. Environmental Protection 500

Agency: http://www.regulations.gov /#!documentDetail;D=EPA

‐ HQ

‐ OPP

‐ 2014

‐ 0614

‐ 0003

, 2014. 501

15. Dix, D. J.; Houck, K. A.; Martin, M. T.; Richard, A. M.; Setzer, R. W.; Kavlock, R. J., The ToxCast program 502

for prioritizing toxicity testing of environmental chemicals. Toxicol. Sci. 2007, 95

, (1), 5

‐ 12. 503

16. Houck, K. A.; Kavlock, R. J., Understanding mechanisms of toxicity: Insights from drug discovery research. 504

Toxicology and Applied Pharmacology 2008, 227

, 163

‐ 178. 505

17. Martin, M. T.; Knudsen, T. B.; Reif, D. M.; Houck, K. A.; Judson, R. S.; Kavlock, R. J.; Dix, D. J., Predictive 506

model of rat reprodcutive toxicity from ToxCast high throughput screening Biology of Reproduction 2011, 85

, (2), 507

327

‐ 339. 508

18. Rotroff, D. M.; Dix, D. J.; Houck, K. A.; Knudsen, T. B.; Martin, M. T.; McLaurin, K. W.; Reif, D. M.; Crofton, 509

K. M.; Singh, A. V.; Xia, M. H.; Huang, R. L.; Judson, R. S., Using in vitro high throughput screening assays to 510

identifiy potential endocrine disrupting chemicals. Environmental health perspectives 2013, 121

, 7

‐ 14. 511

19. Reif, D. M.; Martin, M. T.; Tan, S. W.; Houck, K. A.; Judson, R. S.; Richard, A. M.; Knudsen, T. B.; Dix, D. J.; 512

Kavlock, R. J., Endocrine profile and prioritization of environmental chemicals using ToxCast data. Environmental 513

health perspectives 2010, 118

, (12), 1714

‐ 1720. 51420 20. Judson, R. S.; Houck, K. A.; Kavlock, R. J.; Knudsen, T. B.; Martin, M. T.; Mortensen, H. M.; Reif, D. M.; 515

Rotroff, D. M.; Shah, I.; Richard, A. M.; Dix, D. J., In vitro screening of environmental chemicals for targeted 516

testing prioritization: the ToxCast project. Environmental health perspectives 2010, 118

, (4), 485

‐ 92. 517

21. Knudsen, T.; Houck, K.; Judson, R. S.; Singh, A.; Mortensen, H.; Reif, D.; Dix, D. J.; Kavlock, R. J., 518

Biochemical activities of 320 ToxCast chemicals evaluated across 239 functional targets. Toxicology 2011, 282

, 1

‐ 519

15. 520

22. Sipes, N. S.; Martin, M. T.; Kothiya, P.; Reif, D. M.; Judson, R. S.; Richard, A. M.; Houck, K. A.; Dix, D. J.; 521

Kavlock, R. J.; Knudsen, T. B., Profiling 976 ToxCast chemicals across 331 enzymatic and receptor signaling 522

assays. Chemical research in toxicology 2013, 26

, (6), 878

‐ 95. 523

23. Martin, M. T.; Dix, D. J.; Judson, R. S.; Kavlock, R. J.; Reif, D. M.; Richard, A. M.; Rotroff, D. M.; Romanov, 524

S.; Medvedev, A.; Poltoratskaya, N.; Gambarian, M.; Moeser, M.; Makarov, S. S.; Houck, K. A., Impact of 525

environmental chemicals on key transcription regulators and correlation to toxicity end points within EPA's 526

ToxCast program. Chemical research in toxicology 2010, 23

, (3), 578

‐ 90. 527

24. Huang, R.; Sakamuru, S.; Martin, M. T.; Reif, D. M.; Judson, R. S.; Houck, K. A.; Casey, W.; Hsieh, J.

‐ H.; 528

Shockley, K. R.; Ceger, P.; Fostel, J.; Witt, K. L.; Tong, W.; Rotroff, D. M.; Zhao, T.; Shinn, P.; Simeonov, A.; Dix, D. 529

J.; Austin, C. P.; Kavlock, R. J.; Tice, R. R.; Xia, M., Profiling of the Tox21 10K compound library for agonists and 530

antagonists of the estrogen receptor alpha signaling pathway. Scientific Reports 2014, 4

, (5664). 531

25. Rotroff, D. M.; Dix, D. J.; Houck, K. A.; Kavlock, R. J.; Knudsen, T. B.; Martin, M. T.; Reif, D. M.; Richard, A. 532

M.; Sipes, N. S.; Abassi, Y. A.; Jin, C.; Stampfl, M.; Judson, R. S., Real

‐ time growth kinetics measuring hormone 533

mimicry for ToxCast chemicals in T

‐ 47D human ductal carcinoma cells. Chemical research in toxicology 2013, 26

, 534

(7), 1097

‐ 107. 535

26. U.S. EPA U.S. EPA ToxCast Program Data. http://epa.gov/ncct/toxcast/data.html 536

27. Filer, D., The ToxCast Analysis Pipeline: An R Package for Processing And Modeling Chemical Screening 537

Data. In U.S. Environmental Protection Agency: 538

http://www.epa.gov/ncct/tox cast/files/MySQL%20Database/Pipeline_Overview.pdf

, 2014. 539

28. OECD, Test No. 455: Performance

‐ Based Test Guideline for Stably Transfected Transactivation In Vitro 540

Assays to Detect Estrogen Receptor Agonists. In Organisation for Economic Co

‐ operation and Development: 541

http://www.oecd

‐ ilibrary.org/environment/test

‐ no

‐ 455

‐ performance

‐ based

‐ test

‐ guideline

‐ for

‐ stably

‐ 542

transfected

‐ transactivation

‐ in

‐ vitro

‐ assays

‐ to

‐ detect

‐ estrogen

‐ receptor

‐ agonists_9789264185388

‐ en

, 2012. 543

29. ICCVAM The LUMI

‐ CELL® ER (BG1Luc ER TA) Test Method: An In Vitro Assay for Identifying Human 544

Estrogen Receptor Agonist and Antagonist Activity of Chemicals ; NEIHS: 545

http://ntp.niehs.nih.gov/pubhealth/evalatm/iccvam/test

‐ method

‐ evaluations/endocrine

‐ 546

disruptors/bg1luc/tmer/index.html

, 2011. 547

30. OECD, OECD Test No. 457: BG1Luc Estrogen Receptor Transactivation Test Method for Identifying 548

Estrogen Receptor Agonists and Antagonists. In http://www.oecd

‐ ilibrary.org/environment/test

‐ no

‐ 457

‐ bg1luc

‐ 549

estrogen

‐ receptor

‐ transactivation

‐ test

‐ method

‐ for

‐ identifying

‐ estrogen

‐ receptor

‐ agonists

‐ and

‐ 550

antagonists_9789264185395

‐ en

, 2012; Vol. 2012. 551

31. OECD, Test No. 440: Uterotrophic Bioassay in Rodents A short

‐ term screening test for oestrogenic 552

properties In Organisation for Economic Co

‐ operation and Development: http://www.oecd

‐ 553

ilibrary.org/environment/test

‐ no

‐ 440

‐ uterotrophic

‐ bioassay

‐ in

‐ rodents_9789264067417

‐ en

, 2007. 554

32. U.S. EPA, Endocrine Disruptor Screening Program Test Guidelines OPPTS 890. 1600: Uterotrophic Assay. 555

In U.S. Environmental Protection Agency: http://www.regulations.gov/#!documentDetail;D=EPA

‐ HQ

‐ OPPT

‐ 556

2009

‐ 0576

‐ 0012

, 2009. 557

33. U.S. EPA EPA ACToR. http://actor.epa.gov/actor 558

34. FDA, U. S. US Food and Drug Administration’s Endocrine Disruptor Knowledge Base. 559

http://www.fda.gov/ScienceResearch/Bioinformatics Tools/EndocrineDisruptorKnowledgebase/default.htm 56021 35. U.S. EPA, Endocrine Disruptor Screening Program Test Guidelines OPPTS 890. 1250: Rat Uterine Cytosol 561

Estrogen Receptor Binding Assay. In U.S. Environmental Protection Agency: 562

http://www.regulations.gov/#!documentDetail;D=EPA

‐ HQ

‐ OPPT

‐ 2009

‐ 0576

‐ 0005

, 2009. 563

36. U.S. EPA, Endocrine Disruptor Screening Program Test Guidelines OPPTS 890. 1300: Estrogen Receptor 564

Transactivation Assay. In U.S. Environmental Protection Agency: 565

http://www.regulations.gov/#!documentDetail;D=EPA

‐ HQ

‐ OPPT

‐ 2009

‐ 0576

‐ 0006

, 2009. 566

37. Stehman, S. V., Selecting and interpreting measures of thematic classification accuracy. Remote Sensing 567

of Environment 1997, 62

, (1), 13. 568

38. Zava, D. T., Estrogenic and antiproliferative properties of genestine and other flavonoids in human 569

breast cancer cells in vitro

. Nutrition and Cancer 1997, 27

, (1), 31

‐ 40. 570

39. U.S. EPA EDSP21 Dashboard. http://actor.epa.gov/edsp21/ 571

40. Becker, R. A.; Friedman, K. P.; Simon, T. W.; Marty, M. S.; Patlewicz, G.; Rowlands, J. C., An 572

exposure:activity profiling method for interpreting high

‐ throughput screening data for estrogenic activity

‐‐ proof 573

of concept. Regulatory toxicology and pharmacology : RTP 2015, 71

, (3), 398

‐ 408. 574

41. Patlewicz, G.; Ball, N.; Boogaard, P. J.; Becker, R. A.; Hubesch, B., Building scientific confidence in the 575

development and evaluation of read

‐ across. Regulatory toxicology and pharmacology : RTP 2015, 72

, (1), 117

‐ 576

33. 577

42. Patlewicz, G.; Simon, T. W.; Rowlands, J. C.; Budinsky, R. A.; Becker, R. A., Proposing a scientific 578

confidence framework to help support the application of adverse outcome pathways for regulatory purposes. 579

Regulatory toxicology and pharmacology : RTP 2015, 71

, (3), 463

‐ 77. 580

43. Judson, R.; Kavlock, R.; Martin, M.; Reif, D.; Houck, K.; Knudsen, T.; Richard, A.; Tice, R. R.; Whelan, M.; 581

Xia, M.; Huang, R.; Austin, C.; Daston, G.; Hartung, T.; Fowle, J. R., 3rd; Wooge, W.; Tong, W.; Dix, D., 582

Perspectives on validation of high

‐ throughput assays supporting 21st century toxicity testing. Altex 2013, 30

, (1), 583

51

‐ 6. 584

44. Sauer, J. M.; Hartung, T.; Leist, M.; Knudsen, T. B.; Hoeng, J.; Hayes, A. W., Systems Toxicology: The 585

Future of Risk Assessment. International journal of toxicology 2015

, [Epub ahead of print]. 586

45. Tralau, T.; Oelgeschlager, M.; Gurtler, R.; Heinemeyer, G.; Herzler, M.; Hofer, T.; Itter, H.; Kuhl, T.; Lange, 587

N.; Lorenz, N.; Muller

‐ Graf, C.; Pabel, U.; Pirow, R.; Ritz, V.; Schafft, H.; Schneider, H.; Schulz, T.; Schumacher, D.; 588

Zellmer, S.; Fleur

‐ Bol, G.; Greiner, M.; Lahrssen

‐ Wiederholt, M.; Lampen, A.; Luch, A.; Schonfelder, G.; Solecki, R.; 589

Wittkowski, R.; Hensel, A., Regulatory toxicology in the twenty

‐ first century: challenges, perspectives and 590

possible solutions. Arch Toxicol 2015, 89

, (6), 823

‐ 50. 591

46. OECD, Guidance Document on the Validation of (Quantitative) Structure

‐ Activity Relationships [(Q)SAR] 592

Models. In Organisation for Economic Co

‐ operation and Development: 593

http://www.oecd.org/officialdocume nts/publicdisplaydocumentpdf/?docl anguage=en&cote= env/jm/mono(200 594

7)2

, 2007; p 154. 595

47. OECD, Guidance Document on Developing and Assessing Adverse Outcome Pathways: Series on Testing 596

and Assessment No. 184. In Organisation for Economic Co

‐ operation and Development: 597

http://www.oecd.org/officialdocume nts/publicdisplaydocumentpdf/?co te=env/jm/mono(2013)6&doclanguage 598

=en

, 2013. 599

48. U.S. EPA, Integrated Summary Report for Validation of an Estrogen Receptor Binding Assay Using Rat 600

Uterine Cytosol as Source of Receptor as a Potential Screen in the Endocrine Disruptor Screening Program Tier 1 601

Battery. In U.S. Environmental Protection Agency: http://www.epa.gov/endo/pubs/assayvalidation/status.htm

, 602

2009. 603

49. OECD, Series on Testing and Assessment. Number 67. Report of the Validation of the Uterotrophic 604

Bioassay: Additional Data Supporting the TEst Guideline on the Uterotrophic Bioassay in Rodents.

. In 605

http://epa.gov/endo/pubs/uterotrophic_OECD_validation_report.pdf

, 2007. 60622 50. Ashby, J.; Odum, J., Gene expression changes in the immature rat uterus: effects of uterotrophic and 607

sub

‐ uterotrophic doses of bisphenol A. Toxicological Sciences 2004, 82

, (2), 458

‐ 467. 608

51. Christian, M. S.; Hoberman, A. M.; Bachmann, S.; Hellwig, J., Variability in the uterotrophic response 609

assay (an in vivo estrogenic response assay) in untreated control and positive control (DES

‐ DP, 2.5 microG/kg, 610

bid) Wistar and Sprague

‐ Dawley rats.

. Drug and Chemical Toxicology 1998, 21 Supple 1

, 50

‐ 101. 611

52. Akahori, Y.; Nakai, M.; Yamasaki, K.; Takatsuki, M.; Shimohigashi, Y.; Ohtaki, M., Relationship between 612

the results of in vitro receptor binding assay to human estrogen receptor α and in vivo uterotrophic assay: 613

comparative study with 65 selected chemicals. Toxicology in Vitro 2008, 22

, 225

‐ 332. 614

53. Laws, S. C.; Carey, S. A.; Ferrell, J. M.; Bodman, G. J.; Cooper, R. L., Estrogenic activity of octylphenol, 615

nonylphenol, bisphenol A and methoxychlor in rats. Toxicological sciences : an official journal of the Society of 616

Toxicology 2000, 54

, (1), 154

‐ 67. 617

54. U.S. EPA, Endocrine Disruptor Screening Program Test Guidelines: OPPTS 890.1450. Pubertal 618

Development and Thyroid Function in Intact Juvenile/Peripubertal Female Rats. In U.S. Environmental Protection 619

Agency: http://www.regulations.gov /#!documentDetail;D=EPA

‐ HQ

‐ OPPT

‐ 2009

‐ 0576

‐ 0009

, 2009. 620

55. U.S. EPA, SAP Minutes No: 2013

‐ 05

‐ Weight

‐ of

‐ Evidence: Evaluating Results of EDSP Tier 1 Screening. In 621

U.S. Environmental Protection Agency: 622

http://www.epa.gov/scipoly/sap/ meetings/2013/july/073013minutes.pdf

, 2013; p 65. 623

56. U.S. EPA, Final EDSP (Endocrine Disruptor Screening Program) White Paper on Tier I Assay and Battery 624

Performance for May 21

‐ 23, 2013 FIFRA SAP Meeting. In U.S. Environmental Protection Agency: 625

http://www.regulations.gov/contentStreamer?objec tId=09000064812a3e5d&disposition=attachment&contentT 626

ype=pdf

, 2013; p 160. 627

628