Section 2.1:

Computational Tools

& Models

Richard Judson

National Center for Computational Toxicology

Office of Research and Development

Slide 1 of 65
Presentation Roadmap

Introduction

Estrogen Receptor 

Bioactivity

Androgen Receptor 

Bioactivity

Exposure

Toxicokinetics/Dosimetry

Integrated Bioactivity 

Exposure Ranking

Future Directions

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 2 of 65
In Vitro Estrogen Receptor Model

Combines results from multiple ToxCast in vitro assays

 Use multiple assays per pathway

• Different technologies

• Different points in pathway

 No assay is perfect

• Assay Interference

• Noise

 Use model to integrate assays

 Evaluate model against reference chemicals

 Methodology being applied to other pathways

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 3 of 65
Appendix Overview

 Appendix 1: ER AUC plots for reference chemicals

• Shows plots of ER reference chemicals, assay and model 

data

 Appendix 2: ER AUC plots for all chemicals

• Shows plots for all 1812 chemicals, assay and model 

data

 Appendix 3: Supplementary Data: ER, AR, and IBER

• IBER data for all chemicals with ER and AR model data, 

HTTK data and exposure estimates

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 4 of 65
Major theme:

All Assays Have False Positives & Negatives

Much of this “noise” is reproducible

- “assay interference”

- Result of interaction of chemical 

with complex biology in the assay

EDSP chemical universe is structurally diverse

-Solvents

-Surfactants

-Intentionally cytotoxic compounds

-Metals

-Inorganics

-Pesticides

-Drugs

Assays cluster by technology,

suggesting technology-specific non-ER 

bioactivity

Section 2.1

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 5 of 65
Section 2.1, Figure 2.1, p. 21

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 6 of 65

ER

Model
Radioligand Biochemical Assays (Novascreen) Assay ID Assay Name Gene Target Organism Tissue

A1 NVS_NR_bER ESR1 bovine uterus

A2 NVS_NR_hER ESR1 human NA

A3 NVS_NR_mERa Esr1 mouse NA

Common Features:

- Receptor binding / competitive binding

- Percent of positive control

- Radioligand binding

- Scintillation counting readout

- 18 hour incubation

- Cell-free biochemical assay

- 96-well format

Section 2.1, Table 2.1, p.16

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 7 of 65
Protein Complementation Assays (Odyssey Thera) Assay ID Assay Name Gene Target Assay Design Type Biological Process Target Detection Technology Timepoint (Hours) Tissue Cell Line Name

A4 OT_ER_ERaERa_0480 ESR1

protein fragment 

complementation assay protein stabilization

Protein-fragment 

Complementation 8 kidney HEK293T

A5 OT_ER_ERaERa_1440 ESR1

protein fragment 

complementation assay protein stabilization

Protein-fragment 

Complementation 24 kidney HEK293T

A6 OT_ER_ERaERb_0480 ESR1

protein fragment 

complementation assay protein stabilization

Protein-fragment 

Complementation 8 kidney HEK293T

A7 OT_ER_ERaERb_1440

ESR1 

ESR2

protein fragment 

complementation assay protein stabilization

Protein-fragment 

Complementation 24 kidney HEK293T

A8 OT_ER_ERbERb_0480 ESR2

protein fragment 

complementation assay protein stabilization

Protein-fragment 

Complementation 8 kidney HEK293T

A9 OT_ER_ERbERb_1440 ESR2

protein fragment 

complementation assay protein stabilization

Protein-fragment 

Complementation 24 kidney HEK293T

A10 OT_ERa_EREGFP_0120 ESR1

fluorescent protein 

induction

regulation of gene 

expression Microscopy 2 cervix HeLa

A11 OT_ERa_EREGFP_0480 ESR1

fluorescent protein 

induction

regulation of gene 

expression Microscopy 8 cervix HeLa

Common Features:

- Percent of positive control

- Fluorescence readout

- Scintillation counting readout

- Cell-based, immortalized cell line

- Human

- 384-well format

Section 2.1, Table 2.1, p.16

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 8 of 65
RNA Transcription Activation Assays (Attagene) Assay ID Assay Name Gene Target

A12 ATG_ERa_TRANS_up ESR1

A13 ATG_ERE_CIS_up ESR1

Common Features:

- Log2 fold induction (scaled x25)

- mRNA induction

- Sequencing readout, RT-PCR

- Capillary electrophoresis

- HepG2 liver immortalized cell line

- Human

- 24 hour incubation

- 24-well format

Section 2.1, Table 2.1, p.16

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 9 of 65
Protein Transcription Activation Assays (Tox21)

Assay ID

Assay Name

Gene Target

Assay Design Type

Detection Technology

Detection Technology Subtype

Measurement Timepoint

(Hours)

Tissue

Cell Line Name

A14 Tox21_ERa_BLA_Agonist_ratio ESR1

beta 

lactamase 

induction

GAL4 b-lactamase 

reporter gene

Fluorescence 

intensity 24 kidney HEK293T

A15 Tox21_ERa_LUC_BG1_Agonist ESR1

luciferase 

induction

Luciferase-coupled 

ATP quantitation Bioluminescence 48 ovary BG1

A17 Tox21_ERa_BLA_Antagonist_ratio ESR1

beta 

lactamase 

induction

GAL4 b-lactamase 

reporter gene

Fluorescence 

intensity 24 kidney HEK293T

A18 Tox21_ERa_LUC_BG1_Antagonist ESR1

luciferase 

induction

Luciferase-coupled 

ATP quantitation Bioluminescence 48 ovary BG1

Common Features:

- Percent of positive control

- Immortalized cell line

- Human

- 1536-well format

- Measure gene express8ion

Section 2.1, Table 2.1, p.16

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 10 of 65
Proliferation Assay (ACEA)

Assay ID

Assay Name

GeneTarget

Normalized Data Type

Assay Design Type

Biological Process Target

Detection Technology

Detection Technology Subtype

Measurement Timepoint

(Hours)

Organism

Tissue

Cell Format

Cell Line Name

Assay Footprint

A16 ACEA_T47D_80hr_Positive ESR1

percent 

activity

real-time 

cell-growth 

kinetics

cell 

proliferation RT-CES

Electrical 

Sensor: 

Impedance 80 human breast

cell 

line T47D

96-well 

plate

Section 2.1, Table 2.1, p.16

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 11 of 65
Assay Bioactivity Patterns Track with Technology

Drives need for model to separate ER from off-target bioactivity

Many assays are active

One or a few assays active

Section 2.1, Figure 2.2, p. 24

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 12 of 65
Section 2.1, Figure 2.1, p. 21

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 13 of 65

ER

Model
ER Computational Model

14

Penalty enforces physical assumption

that chemical will not hit many targets 

simultaneously

AUC Summarizes results

 

j

AiFijRj

Aiis the efficacy of the assay at a given concentration

Rjis the “true” efficacy which is unobservable

F links receptors to assays

( ) ( )

2 2

A A penalty R

i

meas

i

pred

i



   

Solve a constrained least-squares problem 

to minimize difference between the 

measured and predicted assay values

2

0

2

2

( )

[1,0]

SR SR

SR penalty R

A

pred

i













 

Nconc

i

j i

conc

j

sign slope R conc

N

AUC

1

( ) ( )

1

Section 2.1, p. 21

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 14 of 65
ER AUC Scales Monotonically With AC50 

for a Pure Agonist Active in All Assays

AUC=0.1

Equivalent to 

AC50=100 uM

Section 2.1

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 15 of 65
Example 1 – BPA – True Agonist 

ER AUC=0.65

Binding assays active at lowest concentration

AUC “sign” feature will discount this

Blue: agonist “receptor”

Assays Receptors

Cytotoxicity 

Region: red line 

is median 

cytotox AC50

Section 2.1

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 16 of 65
Example ER Curves

True Agonist True Antagonist

Negative-Broad Assay Interference Negative-Narrow Assay Interference

Section 2.1,

Figure 2.3, p. 26

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 17 of 65
Selection of ER Reference Chemicals

 Start with list from OECD TG 455

• “Test No. 455: Performance-Based Test Guideline for Stably Transfected 

Transactivation In Vitro Assays to Detect Estrogen Receptor Agonists”

• http://www.oecd-ilibrary.org/environment/test-no-455-performancebased-test-guideline-for-stably-transfected-transactivation-in-vitroassays-to-detect-estrogen-receptor-agonists_9789264185388-en

 Primarily taken from literature surveys, input from 

NICEATM/ICCVAM, EPA 

 Criteria for selection was mainly consistency across laboratories 

in hit-call and relative potency 

 During cross-laboratory validation of TG 455 assay, there was not 

100% consistency in calls

 Heavily weighted towards agonist positives

Section 2.1, p. 18

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 18 of 65
ER Reference 

Chemical List

CASRN Chemical Name Agonist Potency Antagonist Potency

57-63-6 17alpha-Ethinyl estradiol Strong Inactive

50-28-2 17beta-Estradiol Strong NA

56-53-1 Diethylstilbestrol (DES) Strong Inactive

84-16-2 meso-Hexestrol Strong NA

57-91-0 17alpha-Estradiol Moderate NA

140-66-9 4-tert-Octylphenol Moderate NA

53-16-7 Estrone Moderate NA

599-64-4 4-Cumylphenol Weak NA

521-18-6 5alpha-Dihydrotestosterone Weak Inactive

80-05-7 Bisphenol A Weak Inactive

77-40-7 Bisphenol B Weak NA

486-66-8 Daidzein Weak NA

446-72-0 Genistein Weak Inactive

143-50-0 Kepone Weak Inactive

789-02-6 o,p'-DDT Weak NA

58-18-4 17alpha-Methyltestosterone Very Weak NA

520-36-5 Apigenin Very Weak Inactive

85-68-7 Butylbenzyl phthalate Very Weak Inactive

480-40-0 Chrysin Very Weak Inactive

115-32-2 Dicofol Very Weak Inactive

117-81-7 Diethylhexyl phthalate Very Weak Inactive

84-74-2 Di-n-butyl phthalate Very Weak Inactive

120-47-8 Ethylparaben Very Weak NA

60168-88-9 Fenarimol Very Weak NA

520-18-3 Kaempferol Very Weak NA

72-43-5 Methoxychlor Very Weak NA

72-55-9 p,p’-DDE Very Weak Inactive

104-40-5 p-n-Nonylphenol Very Weak NA

1912-24-9 Atrazine Inactive NA

50-22-6 Corticosterone Inactive NA

66-81-9 Cycloheximide Inactive NA

13311-84-7 Flutamide Inactive NA

52-86-8 Haloperidol Inactive NA

52806-53-8 Hydroxyflutamide Inactive NA

65277-42-1 Ketoconazole Inactive NA

330-55-2 Linuron Inactive NA

57-30-7 Phenobarbital Sodium Inactive NA

32809-16-8 Procymidone Inactive NA

50-55-5 Reserpine Inactive NA

52-01-7 Spironolactone Inactive NA

68392-35-8 4-Hydroxytamoxifen (E/Z) NA Active

82640-04-8 Raloxifene NA Active

10540-29-1 Tamoxifen NA Active

54965-24-1 Tamoxifen citrate NA Active

57-83-0 Progesterone NA Inactive

Note that many but not 

all chemicals are used 

for both agonist and 

antagonist references

Section 2.1, Table 2.2, p. 19

Slide 19 of 65
ER Reference Chemical 

Performance

Section 2.1, Figure 2.4, p. 29

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 20 of 65
Using the ER model

 Goal: Eliminate false positives, but not true / weak ones

 Data contains “true” ER hits and various types of noise and 

assay interference

• Noise: results could be different if we reran the assay tomorrow

• Interference: results are reproducible, but do not arise from 

binding to the ER

 Factors to use in distinguishing ER from non-ER bioactivity

• AUC

• Number of technologies active (0,1,...)

• Efficacy (Top/Emax) (Is the response>>noise threshold?)

• Potency (Is the response at a low concentration?)

• Z-score (?s the response well below cytotoxicity threshold?)

• Chemical structure / class

• Physico-chemical properties

Section 2.1

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 21 of 65
ER Summary for 1812 Chemicals

AUC Range 0.1 to 1 0.01 to 0.1

Agonist 97 196

Antagonist 21 109

R3 10 30

R6 96 199

R8/A16 33 35

R9 54 56

A2 13 41

A3 23 24

A9 10 43

A10 13 31

A12 37 216

A13 66 171

A15 71 93

A17 64 95

A18 28 61

Higher intensity color indicates greater activity (AUC<0.1 set to zero)

Section 2.1, Figure 2.5, p. 31

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 22 of 65
Most chemicals display a “burst” of potentially nonselective bioactivity near cytotoxity concentration

10 100 1000 0.1 1

AC50 (mM)

Number of Hits

Burst 

Region

Cytotoxicity

Range

3 MAD

Tested Concentration Range

21 18 15 12 9 6 3 0 -3 -6 -9

Z

Number of Hits

Burst 

Region

Cytotoxicity

Range

3 MAD

Concentration Space Z- Space

Bioactivity 

inferred Section 2.1

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 23 of 65
Example of Burst Bioactivity by Chemical

Cytotoxicity assays

Section 2.1

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 24 of 65
Burst Is Seen Across Technologies

Cell line (HepG2)

Human Primary Cells

Cell-free Biochemical

Bimodal

Section 2.1

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 25 of 65
Schematic Explanation of the Burst

Oxidative Stress

DNA Reactivity

Protein Reactivity

Mitochondrial stress

ER stress

Cell membrane disruption

Specific apoptosis

...

Specific Non-specific

Section 2.1

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 26 of 65
ER Example – 17beta Estradiol Section 2.1

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 27 of 65
ER Example – 5alpha-Dihydrotestosterone

Section 2.1

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 28 of 65
ER Example – Raloxifene

Section 2.1

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 29 of 65
ER Example – Bisphenol A

Section 2.1

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 30 of 65
ER Example – Branched Nonylphenol

Section 2.1

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 31 of 65
ER Example – Linear Nonylphenol

Section 2.1

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 32 of 65
Section 2.1

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 33 of 65

ER Example – Kepone
Section 2.1

ER Example – Butyl Benzyl Phthalate

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 34 of 65
Section 2.1

ER Example – DEHP

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 35 of 65
Section 2.1

ER Example – Tributyltin

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 36 of 65
Section 2.1

ER Example – Cycloheximide

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 37 of 65
Section 2.1

ER Example – Dimethyl Hydrogen Phosphite

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 38 of 65
Section 2.1

ER Example – alpha-Cyclodextrin

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 39 of 65
ER Antagonists 

CASRN Name Use category AUC 

(Antagonist)

82640-04-8 Raloxifene hydrochloride Pharmaceutical 0.973

68392-35-8 4-Hydroxytamoxifen Pharmaceutical 0.961

129453-61-8 Fulvestrant Pharmaceutical 0.911

50-41-9 Clomiphene citrate Pharmaceutical 0.817

54965-24-1 Tamoxifen citrate Pharmaceutical 0.776

10540-29-1 Tamoxifen Pharmaceutical 0.695

84371-65-3 Mifepristone Pharmaceutical 0.456

13463-41-7 Zinc pyrithione Anti-fungal/anti-bacterial/drug 0.335

26027-38-3 Polyoxyethylene(10)nonylphenyl ether Preservative 0.197

683-18-1 Dibutyltin dichloride Microbicide 0.179

989-38-8 Rhodamine 6G Dye 0.15

4342-36-3 Tributyltin benzoate Microbicide 0.146

2971-36-0 2,2-Bis(4-hydroxyphenyl)-1,1,1-trichloroethane (HPTE) Degradate of Methoxychlor 0.143

56-95-1 Chlorhexidine diacetate Biocide 0.134

90357-06-5 Bicalutamide Pharmaceutical 0.129

66-81-9 Cycloheximide Fungicide 0.121

62-38-4 Phenylmercuric acetate Fungicide 0.115

9004-82-4 Sodium lauryl polyoxyethylene ether sulfate Surfactant 0.107

25155-18-4 Methylbenzethonium chloride Microbiocide 0.105

1034-01-1 Octyl gallate Antioxidant 0.104

57-09-0 Hexadecyltrimethylammonium bromide Algaecide/antiseptic/anti-static agent 0.104

Antagonism is often confounded by cytotoxicity

These need to be examined further

Section 2.1

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 40 of 65
Pseudo-Receptor Diagnostics

 Model separates chemicals into ER active, ER inactive, 

and pseudo-receptor active (individual 

assays/technologies)

 Aim: diagnose cause of pseudo-receptor activity

 Diagnostics:

• Non-normal distribution of baseline noise / T

• Cell-stress / cytotoxicity / Z

• Crosstalk / cross-reactivity

• Pan-active compounds (PAINS)

• Structure class trends

• Physico-chemical property trends

 Work in progress

Section 2.1

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 41 of 65
First Drivers of Pseudo-Receptor Activity:

Low T (non-normal noise)

Low Z (cytotoxicity-related)

PseudoReceptorN |T| |T| 

(Comparator)

p(T) |Z| |Z| 

(Comparator)

p(Z)

R3 10 94 86 NS 9 8.8 NS

R6 96 60 76 2.00E-09 3.8 7.3 2.10E-14

R8 33 101 110 NS 7.7 7.9 NS

R9 54 98 97 NS -0.57 2.8 2.80E-05

A2 13 91 88 NS 7.5 9.4 NS

A3 23 81 80 NS 6.7 7.9 NS

A9 10 47 101 1.40E-06 6.1 5.4 NS

A10 13 72 85 NS 6.8 6.3 NS

A12 37 53 81 2.10E-09 3.3 7.3 1.50E-06

A13 66 33 71 9.10E-23 5.8 7.3 0.021

A15 71 65 116 2.80E-08 4.5 6.2 0.022

A17 64 75 86 0.037 1.4 3.6 0.019

A18 28 95 101 NS 2.6 1.9 NS

Section 2.1, Table 2.3, p. 32

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 42 of 65
T-distributions for T-significant Assays

Section 2.1

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 43 of 65
Z-distributions for Z-significant Assays

Section 2.1

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 44 of 65
Crosstalk / Cross-Reactivity

 Crosstalk – Bioactivity in one target of a 

multiplexed assay causes other readouts to fire

• ATG, ACEA

 Cross-reactivity – A compound potent against 

one receptor is weakly potent against another

• True active, but some other process is probably more 

important

 Can be difficult to distinguish 

Section 2.1, p. 34

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 45 of 65
Quantify Cross-Reactivity

 Are pseudo-receptor actives more likely to 

also be active in many assay readouts than are 

ER-inactives?

 Test for broad bioactivity in ATG CIS, ATG 

TRANS, NVS NR, Tox21 BLA agonist, Tox21 LUC 

agonist, Tox21 BLA antagonist, Tox21 LUC 

antagonist

Section 2.1, p. 34

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 46 of 65
Cross-Reactivity Results

Non-ER Assay Set Expected PseudoReceptorsCorrelated

Pseudo- Receptor

Reference

Mean

PseudoReceptor

mean

p-value

Attagene CIS A13, R6 A12 3.8 15 0.0023

Attagene CIS A13, R6 A18 3.8 18 0.0013

Attagene TRANS A12, R6 A12 1.5 6.7 0.00015

Attagene TRANS A12, R6 A13 1.5 3.4 0.00069

Attagene TRANS A12, R6 A18 1.5 5.9 0.0066

Attagene TRANS A12, R6 R6 1.5 2.7 0.00019

Novascreen NR A1, A2, A3 A2 0.64 2.1 0.01

Novascreen NR A1, A2, A3 Agonist 0.64 1.8 0.00024

Novascreen NR A1, A2, A3 Antagonist 0.64 3.3 0.0015

Tox21 BLA Antagonist A17, R9 A17 0.26 2.2 1.20E-07

Tox21 LUC Agonist A15, R7 A12 0.54 1.1 0.0078

Tox21 LUC Antagonist A18, R9 A17 0.14 1.1 4.30E-06

Tox21 LUC Antagonist A18, R9 A18 0.14 1.7 1.70E-05

Tox21 LUC Antagonist A18, R9 Agonist 0.14 0.41 0.0035

Example: Chemicals active in only A12 (ATG_ERa_TRANS) and no other ER assays have a mean 

of 15 hits in the ATG CIS battery, vs. an mean of 3.8 in chemicals inactive in all ER assays

Interpretation: ER is just a bystander in some process causing all of these assay readouts to fire

Section 2.1,

Table 2.4, p. 35

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 47 of 65
PAINs

Pan-Assay Interference Compounds

 Some compounds are observed to activate many 

assays

 Known problem in pharmaceutical development

 Related to cell-stress issues?

 Public structure-classification model is available

 Test whether pseudo-receptor actives are more 

likely to be active in many ToxCast assays vs. 

those inactive in all ER assays

Section 2.1, p. 35 Baell and Holloway, J. Med. Chem., 2010, 53 (7), pp 2719–2740

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 48 of 65
PAINs Results

Receptor

HI- Z reference 

mean

HI-Z pseudoreceptor

mean P (HI-Z)

LO-Z reference 

mean

LO-Z pseudoreceptor mean P (LO-Z)

A10 0.026 0.0017 NS 0.018 0.025 NS

A12 0.026 0.0039 NS 0.018 0.087 0.0013

A13 0.026 0.054 NS 0.018 0.026 0.0052

A15 0.026 0.001 NS 0.018 0.031 0.0043

A17 0.026 0.0048 NS 0.018 0.079 2.80E-05

A18 0.026 0.045 NS 0.018 0.054 1.80E-05

A2 0.026 0.0019 NS 0.018 0.046 0.032

A3 0.026 0.016 NS 0.018 0.051 0.0012

Agonist 0.026 0.072 1.80E-08 0.018 0.076 3.80E-25

Antagonist 0.026 0.18 6.10E-05 0.018 0.055 4.50E-05

R3 0.026 0.00067 NS 0.018 0.055 0.0041

R6 0.026 0.0047 NS 0.018 0.03 1.80E-06

R8 0.026 0.0081 NS 0.018 0.029 0.014

Example: Chemicals that are model-agonists are active in an average of 7.2% of all non-ER 

ToxCast assays, while chemicals that are active in no ER assays are active in an average of 2.6% 

of all non-ER ToxCast assays, if one excludes all bioactivity for Z<3

Section 2.1, p. 35

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 49 of 65
Chemical Structure Class

 Are particular structure classes more likely to be 

associated with pseudo-receptor activity?

• All ToxCast chemicals are placed into structure classes based 

on hierarchical clustering

 Only significant findings: model-agonist and modelantagonist chemicals

 ER agonist-associated classes: 

• genistein-like, paraben, alkyl-phenol, steroids targeting the 

estrogen receptor and steroids targeting the androgen 

receptor

 ER antagonist-associated class: 

• Tamoxifen analogs

Section 2.1, p. 36

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 50 of 65
Physico-Chemical Properties

 Are chemicals in particular ranges of physicochemical properties more likely to be pseudoreceptor active?

 Weak but statistically significant trends are 

seen, but need further refinement to be useful

• MW, surface area, volume, pKa, polarizability, logP, 

logS, logD

• Overlap between property values for modelactives (including reference chemicals) and 

pseudo-receptor chemicals is significant

Section 2.1, p. 36

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 51 of 65
 Tox21, ~8000 chemicals in 4 assays; 

 FDA EDKB database of ~8000 chemicals from the literature; 

 METI database, ~2000 chemicals;

 ChEMBL database, ~2000 chemicals.

 Training chemicals for specific QSAR models

Experimental / Literature Data for Comparison with 

ER In Vitro Model and QSAR Models

EPA/NCCT, UMEA/Chem, FDA/NCTR, NIH/NCATS, Helmholtz/ISB

60,000 entries for ~15,000 chemicals 

Section 2.1

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 52 of 65
Comparison with ER Literature Data Chemical Set Receptor N (Binding Positive) N (Binding Negative) N (Agonist Positive) N (Agonist Negative) N (Antagonist Positive) N (Antagonist Negative)

Initial Agonist 55 0 25 0 1 0

Initial Antagonist 5 0 0 0 3 0

Initial None 11 3 2 0 2 0

Initial R6 4 1 3 0 0 0

Final Agonist 41 0 19 0 1 0

Final Antagonist 5 0 0 0 3 0

Use training data with at least 4 concordant calls

Remove all Tox21 data before this comparison

Some literature positive appear to be consistent false-positives

- Maneb and Mancozeb

Section 2.1, Table 2.5, p. 38

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 53 of 65
Introduction to the EDSP21 Dashboard

 Goal: To make ER and AR data easily available 

to all stakeholders

• Assay-by-assays concentration-response plots

• Model scores – AUC agonist and antagonist

• ER QSAR calls

• Other relevant data

 First version is “beta” – intended to elicit 

comments, requests for further additions

 http://actor.epa.gov/edsp21

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 54 of 65
http://actor.epa.gov/edsp21

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 55 of 65
ToxCast + QSAR: CERAPP

 Collaborative Estrogen Receptor Activity Prediction Project 

(CERAPP)

 Goals:

• Use ToxCast ER score (or other data) to build many QSAR models

• Use consensus of models to prioritize chemicals for further 

testing

 Assumptions

• ToxCast chemicals cover enough of chemical space to be a good 

“global” training set

• Consensus of many models will be better than any one 

individually

 Process

• Curate chemical structures

• Curate literature data set

• Build many models

• Build consensus model

• Evaluate models and consensus

ER QSAR/ CERAPP, p. 38

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 56 of 65
 DTU/food: Technical University of Denmark/ National Food Institute

 EPA/NCCT: U.S. Environmental Protection Agency / National Center for Computational Toxicology

 FDA/NCTR/DBB: U.S. Food and Drug Administration/ National Center for Toxicological 

Research/Division of Bioinformatics and Biostatistics 

 FDA/NCTR/DSB: U.S. Food and Drug Administration/ National Center for Toxicological 

Research/Division of Systems Biology

 Helmholtz/ISB: Helmholtz Zentrum Muenchen/Institute of Structural Biology

 ILS&EPA/NCCT: ILS Inc & EPA/NCCT

 IRCSS: Istituto di Ricerche Farmacologiche “Mario Negri”

 JRC_Ispra: Joint Research Centre of the European Commission, Ispra.

 LockheedMartin&EPA: Lockheed Martin IS&GS/ High Performance Computing

 NIH/NCATS: National Institutes of Health/ National Center for Advancing Translational Sciences

 NIH/NCI: National Institutes of Health/ National Cancer Institute

 RIFM: Research Institute for Fragrance Materials, Inc

 UMEA/Chemistry: University of UMEA/ Chemistry department

 UNC/MML: University of North Carolina/ Laboratory for Molecular Modeling

 UniBA/Pharma: University of Bari/ Department of Pharmacy

 UNIMIB/Michem: University of Milano-Bicocca/ Milano Chemometrics and QSAR Research Group

 UNISTRA/Infochim: University of Strasbourg/ ChemoInformatique

CERAPP Participants:

ER QSAR/ CERAPP, p. 38

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 57 of 65
INITIAL LIST OF CHEMICALS

File parsing & first check

Check for mixtures

Check for salts & counter ions

Normalization of specific chemotypes

Manual inspection

Removal of duplicates

Treatment of tautomeric forms

CURATED DATASET

Check for inorganics

Scheme of the Structure Curation Workflow

Univ. North Carolina, Danish Technical University, EPA NCCT

Implement in open-source KNIME Workflow

Fourches, Muratov, Tropsha. J Chem Inf Model, 2010, 29, 476 – 488

ER QSAR/ CERAPP, p. 38

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 58 of 65
Chemicals for Prediction:

The Human Exposure Universe

 EDSP Universe (10K)

 Chemicals with known use (40K) (CPCat & ACToR) 

 Canadian Domestic Substances List (DSL) (23K)

 EPA DSSTox – structures of EPA/FDA interest (15K)

 ToxCast and Tox21 (In vitro ER data) (8K)

~55k to ~32K unique set of structures

 Training set (ToxCast): 1,677 chemical structures 

 Prediction Set: 32,464 chemical structures

ER QSAR/ CERAPP, p. 39

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 59 of 65
ER QSAR Evaluation & Consensus

 Qualitative:

• Binding: 22 models

• Agonist: 11 models

• Antagonist: 9 models

 Quantitative:

• Binding: 3 models

• Agonist: 3 models

• Antagonist: 2 models

Models received: Evaluation procedure:

 On the EPA training set (1677)

 On the full evaluation set (~7k)

 Evaluation set with multi-sources

 Remove “Very Weak” 

 Remove single source 

 Remove chemicals outside the AD

Score functions & weights 

for consensus predictions

ER QSAR/ CERAPP, p. 39

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 60 of 65
Concordance of 22 Qualitative ER Models

757 chemicals have >75% positive concordance

Active

Inactive

Prioritization 

Most models predict most chemicals as inactive

ER QSAR/ CERAPP, p. 39

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 61 of 65
Total Database

Binders: 3961

Agonists: 2494

Antagonists: 2793

Consensus Evaluation

Key point: As greater consistency is 

required from literature sources, 

QSAR consensus model performance 

improves

ER QSAR/ CERAPP, p. 39

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 62 of 65
• positive concordance < 0.6 => Potency class= Very weak

• 0.6=<positive concordance<0.75 => Potency class= Weak

• 0.75=<positive concordance<0.9 => Potency class= Moderate

• positive concordance>=0.9 => Potency class= Strong

Positive Concordance & Potency Level

Key Points:

1. Fraction of ER QSAR models called positive is highly correlated with measured potency

2. If strong consensus is required from literature, consensus ER QSAR model performs well

ER QSAR/ CERAPP,

Figure 2.8, p. 40

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 63 of 65
ER QSAR Summary

 Many ER QSAR and docking models built using ER 

model result (AUC) as training data

 Consensus of models gives high balanced accuracy for 

literature data that is internally consistent

 Open-source structure preparation process performed 

on all EDPS universe distinct chemicals (and others)

 5-10% of chemical universe has predicted potential for 

ER bioactivity

 QSAR-model positives are candidates for follow-up in 

vitro testing

ER QSAR/ CERAPP

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 64 of 65
ER Model Conclusions

 1812 Chemicals run in 18 ToxCast ER assays

 Model developed to quantify agonist or antagonist 

bioactivity

 Assays and model perform well against reference 

chemicals

 A small number of chemicals have consistent ER 

bioactivity

• 97 agonist

• 21 antagonists

• Most of these are in well-studied chemical classes 

(phenols, steroids)

Section 2.1

FIFRA SAP: Integrated Bioactivity Exposure Ranking Dec 2-5, 2014 Slide 65 of 65