1 

# FIFRA Scientific Advisory Panel Minutes No.

2015

- 0 1

A Set of Scientific Issues Being Considered

by the
Environmental Protection Agency

Regarding
Integrated Endocrine Bioactivity and
Exposure

- Based Prioritization and Screening

December 2

- 4, 2014

FIFRA Scie ntific Advisory Panel Meeting

Held at the

One Potomac Yard
Arlington, VA2 TABLE OF CONTENTS

PANEL ROSTER 5

LIST OF COMMON ACRONYMS USED 8

INTRODUCTION 9

PUBLIC COMMENT ER S 9

OVERALL SUMMARY 10

EXECUTIV E SUMMARY OF PANEL DISCUSSION AND

RECOMMENDATIONS 11

DETAILED PANEL DELIBERATIONS AND RESPONSE

TO CHARGE 2 0

REFERENCES 433 NOTICE

The Federal Insecticide, Fungicide, and Rodenticide Act (FIFRA), Scientific Advis ory Panel (SAP) is a Federal advisory committee operating in accordance with the Federal Advisory Committee Act and established under the provisions of FIFRA as amended by the Food Quality Protection Act (FQPA) of 1996. The FIFRA SAP provides advice, infor mation, and recommendations to the Agency Administrator on pesticides and pesticide

- related issues regarding the impact of regulatory actions on health and the environment. The Panel serves as the primary scientific peer review mechanism of the Environment al Protection Agency (EPA), Office of Pesticide Programs (OPP), and is structured to provide balanced expert assessment of pesticide and pesticide

- related matters facing the Agency. FQPA Science Review Board

members serve the FIFRA SAP on an ad hoc basis t o assist in reviews conducted by the FIFRA SAP. The meeting minutes have been written as part of the activities of the FIFRA SAP.

The FIFRA SAP carefully considered all information provided and presented by EPA, as w ell as information presented by the pu blic . The minutes represent the views and r ecommendations of the FIFRA SAP and do not necessarily represent the views and policies of the EPA, nor of other agencies in the Executive Branch of the Federal government. Mention of trade names or commercial pr oducts does not constitute an endorsement or recommendation for use. The meeting minutes do not create or confer legal rights or impose any legally binding requirements on EPA or any party. The meeting minutes of the December 2

- 4, 2014 FIFRA SAP meeting

h eld to consider and review scientific issues associated with “ Integrated Endocrine Bioactivity and Exposure

- Based Prioritization and Screening ” were certified by James McManaman, Ph.D., FIFRA SAP Session Chair, and Fred Jenkins, Ph.D., FIFRA SAP Designated Federal Official,

on March 2, 2015

. The meeting report was reviewed by Laura E. Bailey, M.S., FIFRA SAP Executive Secretary. The minutes are publicly available on the SAP website ( http://www.epa.gov/scipoly/sap/ ) under the heading of “Meetings” and in the public e

- docket,

Docket No. EPA

- HQ

- OPP

- 2014

- 0 614 , accessible through the docket portal: http://www.regulations.gov

. Further information about FIFRA SAP reports and activities can be

obtained from its website at htt p://www.epa.gov/scipoly/sap/

. Interested persons are invited to contact Fred Jenkins, Ph.D., SAP Designated Federal Official, via e

- mail at jenkins.fred@epa.gov.5 PA NEL ROSTER

FIFRA SAP Chair

James L. McManaman, Ph.D.

Professor and Chief Section of Basic Reproductive Sciences Department of Obstetrics and Gynecology and Physiology and Biophysics University of Colorado Denver
Aurora, CO

Designated Feder al Official

Fred Jenkins, Jr., Ph.D. US Environmental Protection Agency Office of Science Coordination and Policy FIFRA Scientific Advisory Panel EPA East Building, MC 7201M 1200 Pennsylvania Avenue, NW Washington, DC 20460 Phone (202) 564

- 3327 Fa x (202) 564

- 8382 jenkins.fred@epa.gov

FIFRA Scientific Advisory Panel Members

Dana Boyd Barr, Ph.D.

Research Professor Department of Environmental and Occupational

Health

Rollins School of Public Health Emory University
Atlanta, GA

Kenneth Del clos

, Ph.D. Research Pharmacologist Division of Biochemistry Toxicology National Center for Toxicological Research US Food and Drug Administration
Jefferson, AR6

David Jett, Ph.D.

Director National Institute of Neurological Disorders and

Stro ke

National Institutes of Health
Bethesda, MD

FQPA Science Review Board Members

Veronica Berrocal, Ph.D.

Assistant Professor of Biostatistics Department of Biostatistics

School of Public Health University of Michigan Ann Arbor, MI

Terry Brown, Ph.D.

Professor Department of Biochemistry and Molecular Biology Division of Reproductive Biology The Johns Hopkins Bloomberg School of Public Health Baltimore, MD

Robert Denver, Ph.D.

Professor and Chair Department of Molecular, Cellular, and Developmental Bi ology The University of Michigan Ann Arbor, MI

Daniel Doerge, Ph.D. Food and Drug Administration National Center for Toxicological Research (NCTR) Jefferson, AR

Edward Perkins, Ph.D.

Senior Scientist (ST) Environmental Laboratory US Army Engineers Rese arch and Development Center Vicksburg, MS7

Thomas Potter, Ph.D.

Research Chemist US Department of Agriculture Agricultural Research Station (USDA

- ARS)

USDA

- ARS, Coastal Plain Experiment Station

Southeast Watershed Research Lab
Tifton, GA

C at herine Propper, Ph.D.

Professor Biological Services Northern Arizona University Flagstaff, Arizona

Daniel Schlenk, Ph.D. Professor of Aquatic Ecotoxicology and Environmental Toxicology Department of Environmental Sciences University of California, R iverside
Riverside, CA

Grant Weller, Ph.D.

Research Scientist Savvysherpa, Inc. Minneapolis, MN8 LIST OF COMMONLY USED ACRONYMS AND

ABBREVIATIONS

AIC: Akaike Information Criterion

AOP: Adverse Outcome Pathway

AR: Androgen receptor

AUC: Area Under the Curve

BW: Body weight

CDC: the United States Centers for Disease Control

EDSP: the Endocrine Disrupter Screening Program

EDSTAC: Endocrine Disruptors Screening and Testing Advisory Committee

EEC: Expected Environmental Concentration

EPA: the United States Environmental Protection Agency

EPI Suite: EPA’s Estimation Program Interface Suite

ER: Estrogen receptor ExpoCast: EPA’s Exposure fore c ast prioritization research program

FIFRA: Federal Insecticide, Fungicide, and Roden ticide Act

HEDS: EPA’s Human Exposure Data System

HPDB: Household Product Database

HPV: High Production Volume

HT: High

- throughput

HTE: High

- throughput exposure

HTS: High

- throughput screening

HTTK: High Throughput Toxicokinetics

IBER: I ntegrated (RTK) Bioactivity Exposure Ranking

IEMS: Integrated Exposure Modeling System

IUR: EPA’s Inventory Update Reporting and Chemical Data Reporting

list (now CDR)

LEL: Lowest effect level

LOD: Limit of detection

NHANES: National Health and Nutrition Examination Survey

NTP: the NIH National Toxicology Program

November 3, 2014 Page 6 of 111

SAP: Scientific Advisory Panel

SEEM: Systematic Empirical Evaluation of Models framework

ToxCast: EPA’s Toxicity foreCast prioritization researc h program

Tox21: Toxicology in the 21st Century

– the NTP/NCGC/EPA/FDA

consortium for chemical hazard HTS

USEtox: United Nations Environment Program and Society for

Environmental Toxicology and Chemistry toxicity model9 I NTRODUCTION

On December 2

- 4, 2014 the US EPA Federal Insecticide, Fungicide, and Rodenticide Act Scientific Advisory Panel (FIFRA SAP) met in Arlington , VA to consider and review scientific

issues associated with “ Integrated Endocrine Bioactivity and Exposure

- Based Prioritization and Screening ”. The Panel was charged with advising the Agency on its proposed method s of using computational toxicology and exposure tools (i.e., high throughput screening (HTS) assays
, and computational models ), and other data streams for integrated bio activity and exposure based prioritiz ation and screening of the universe of E ndocrine D isruptor S creening P rogram (EDSP)

chemicals

. The Panel provided recommendations to EPA regarding this proposal. Specifically, the Panel addressed the EPA’s charge to the m on questions concerning the following three topic

areas associated with integrat ing bioactivity and exposure : 1) estrogen bioactivity , 2) androgen bioactivity, and 3) I ntegrated Bioactivity Exposure Ranking (IBER) . Opening remarks at the meeting were pro vided by Dr. David Dix, Director of the Office of Science Coordination and Policy. US EPA presentations were provided by the following EPA staff Steven Knott, Patience Browne, Richard Judson, and John Wambaugh

.

Presentations were also provided by Warren C asey and Nicole Kleinstreur, respectively staff and

affiliate of the US National Toxicology Program's Interagency Center for the Evaluation of Alternative Toxicological Methods :

PUBLIC COMMENT ERS

Oral public comments are listed in the order presented du ring the meeting :

Patricia Bishop

, People on behalf of Ethical Treatment of Animals Richard A. Becker Ph.D., DABT for the American Chemistry Council Ellen Mihaich, Ph.D., DABT of the Environmental and Regulatory Resources on behalf of the

Endocrine

Sue Yi, Ph.D. of Syngenta on behalf of the Endocrine Policy Forum Chris Borgert, Ph.D. of Applied Pharmacology and Toxicology on behalf of the Endocrine Policy

Forum

Katie Paul Friedman , Ph.D. on behalf of B ayer CropScience and Dow Agrosciences Sue Marty, P h.D. D.A.B.T on behalf of Dow Chemical Company Anna Mazzucco, Ph.D. on behalf of the National Center for Health Research

Written statements were provided by (listed in alphabetical order):

The American Chemistry Council The Endocrine Policy Forum People for Et hical Treatment of Animals, Physicians Committee for Responsible Medicine, and the Humane Society of the United States (Jointly written comment)10 OVERALL SUMMARY

The Panel was charged with advising the Agency on the following three topic areas associated with integrating bioactivity and exposure: 1) estrogen bioactivity , 2) androgen bioactivity, and 3) Integrated Bioactivity Exposure Ranking (IBER)

.

In general

, the Panel agreed the ER Model for assessing estrogen bioactivity had several

stren gths. For example, they believed that the ER AUC approach was a computationally, time

resourceful, and insightful approach to determine the estrogenic bioactivity of a chemical. They

also noted the AUC ranking had a strong utility for ranking chemicals in association with potential estrogen bioactivity. The Panel concurred that the ER AUC model demonstrated outstanding performance in the characterization of reference chemicals, and in concordance with

the selected chemicals for in vivo uterotrophic response s.

Although the Panel highlighted several strengths of the ER Model, they also pointed out several limitations and/or weaknesses of the model. For instance, they noted that the models d id not incorporate uncertainty or sensitivity analyses. They recommen ded that the Agency explore the inclusion of such analyses. The Panel also noted that the Agency provide more transparency in describing details about the underpinnings of the model.

In regard to the Agency’s efforts to assess androgen bioactivity, the Pa nel noted that c urrent knowledge suggests that chemicals which impact androgen bioactivity act as relatively weak antagonists rather than agonists. Based upon these largely antagonistic activities of chemicals, they advised that it is critical that the HTS AR bioactivity assay battery include careful assessments and attention to the potential cytotoxic effects of chemicals that may otherwise appear as false positives due to assay interference. This applies to effects generated by the test

chemical as well a s solvents used in the formulation of chemical doses. The range of chemical structures tested in the assay battery should be expanded to maximize the screening potential. Furthermore, the AR bioactivity battery should include methods to assess the potenti al effects of chemicals, as well as their metabolites formed by enzymatic conversion in biological systems. The AR bioactivity battery specifically addresses the classical AR nuclear receptor/genomic

actions of chemicals

, but does not consider other potent ial non

- classical/non

- genomic mechanisms that mimic or inhibit androgen bioactivity.

Concerning the Agency’s proposed Integrated Bioactivity Exposure Ranking (IBER) approach

, the Panel noted that it was rationally developed and laid a foundation for a the oretical basis with the potential to prioritize further EDSP screening of compounds with estrogenic activity. The Panel especially highlighted the practicality of the statistical modeling of the IBER approach. Although the Panel positively remarked about t he IBER approach, they cautioned that the approach needed further refinement before it is employed by the Agency’s Endocrine Disruption Screening P rogram . Suggested areas of exploration to further refine the IBER approach included gaining a better understa nding of how monitoring data would be to strengthen the approach. The Panel also expressed concern about the limited amount of NHANES exposure data integrated

into the IBER model.11

EXECUTIVE SUMMARY OF PANEL DISCUSSION AND RECOMMENDATIONS

TOPIC: ESTROGEN BIOACTIVITY

1) EPA’s proposed approach for quantifying a chemical’s potential estrogen bioactivity is based on a computational model integrating data from 18 high throughput ToxCast assays measuring several endpoints along the estrogen receptor (ER) sign aling pathway. The computational model outputs are expressed as area under the curve (AUC) scores for ER agonist (R1) and antagonist (R2) bioactivity. Before routinely using the ER computational model in the Endocrine Disruptor Screening Program (EDSP) fra mework, EPA is reviewing the scientific strengths and limitations

of the ER model described in the White Paper to: i) prioritize chemicals for further EDSP screening and testing based on estimated bioactivity, ii) contribute to the weight of evidence

evalu ation of a chemical’s potential bioactivity, and iii) substitute for specific endpoints in the EDSP Tier 1 battery. Please address the following charge questions relevant to Section 2 of the White Paper and estrogen bioactivity:

a . How clearly has EPA de scribed the computational tools in Section 2.1 ( i.e. , highthroughput assays and models) used to estimate ER agonist and antagonist bioactivity?

Summary

In general

, the EPA has clearly described the computational models used to estimate ER agonist

and ant agonist bioactivity. However, several areas were noted as providing insufficient

information regarding the development and application of the proposed tool s

. The Panel noted

that it would be beneficial if the White Paper provided more information on bioas say performance and if q uality c ontrol/ a ssurance procedures were made available, either via an addendum or otherwise easily accessible documents. M ore information is needed regarding the

model details such as modified Z

- scores, effect of weighting on summa ry scores, construction of a consensus model, and the penalized least

- squares minimization procedure. Additionally, a discussion of the necessity for, and importance of redundancy in terms of orthogonal assays would clarify why so many assays are utilized.

b . What are strengths and limitations of the ER AUC model’s ability to identify reference chemicals that include a variety of structures and have a wide range of in vitro ER bioactivities?

Summary

Overall, the Panel believe d that the ER AUC approach i s a computational ly time

- efficient and intuitive approach to determine the estrogenic bioactivity of a chemical. Its strengths are the use of a combination of assays, its simple and parsimonious mathematical formulation, and its performance on reference ch emicals ( particularly agonist chemicals ) . Despite a slighter poorer performance on antagonist chemicals, in general, the ER AUC model provides good or better results than the three assays that it is intended to replace in the existing Tier 1 screening batt ery.

The limitations of the ER AUC model can be grouped into two general categories: computational/inferential and generalizability. With respect to computational/inferential12

limitations, the Panel believe d that: (i) efforts should be undertaken to accoun t for uncertainty in the formulation of the statistical model relating receptor signal with assays; and (ii) sensitivity analyses should be conducted to assess how results change under different choices on the model parameters (e.g. the activation threshol d or the penalty term in the ER AUC model). Relative to generalizability, the Panel acknowledge d the possibility that the reference chemicals examined to assess the performance of the ER AUC model may not cover all the structural classes that are present in the entire EDSP universe of chemicals.

c. EPA used data from published in vivo studies that are methodologically consistent with EDSP Tier 1 guidelines to evaluate concordance between ER AUC model scores of in vitro bioactivity,

and the in vivo uterotr ophic response studies (Section 2.2.1). What are strengths and limitations

of the curation methods and quality standards used for evaluating published in vivo studies?

Summary

The Panel concluded that the methods used to curate and standardize the litera ture studies for comparing the in vivo uterotrophic endpoints to the ER AUC model scores were in most cases

sufficient. The ER AUC provided remarkable sensitivity of estimates of uterotrophic responses particularly for low potency compounds with balanced a ccuracy of 95%, positive predictive value of 97%, and negative predictive value of 92%.

In regard to the limitations of these methods, while the literature evaluations indicated strength of comparisons ( in vivo to ER AUC) for high and low potency compoun ds, variability (18/70

chemicals or 26% discordant) was still significant. The Panel agree d with the Agency that this is likely due to differences in design ( e.g. sample size) and the number of studies conducted for

each chemical.

Overall, with the high d egree of variability, and only having one false positive and one false negative out of 42 compounds (Figure 2.12) the ER AUC model is surprisingly accurate. However, additional studies should focus on metabolically activated compounds (e.g. pyrethroids, an d methoxychlor) that should be more appropriately detected in vivo . In addition, literature curation should also include comparisons with other in vivo assays such as the FSTRA and rat pubertal assays.

d . Based on all the data presented in Section 2 on E R AUC model performance including characterization of reference chemicals, and concordance with in vivo uterotrophic results, what are strengths and limitations of using the ER AUC Page 2 of 3 model to distinguish and prioritize chemicals based on potentia l estrogen bioactivity?

Summary

T he Panel concurred (assuming linear hER binding and/or activation through nuclear receptors) the AUC model is a relatively simple, but elegant, approach for integrating different bioassay data into a single value that is proportional to the bioactivity or potency of a chemical. The AUC ranking clearly has a strong applicability for ranking chemicals in terms of potential estrogen bioactivity.13

Additional s trengths of the method include: e xcellent performance for reference chemicals;

consistent outcomes with active and non

- active descriptions for List 1

, List 2

, and the EDSP

Universe of compounds ; redundancy and power of multiple HTS signals for confirmation of

“hits” ; strong predictive accuracy of greater than 90%; and str ong correlation across potency categories for reference chemicals.

The Panel also noted several limitations including a l imited evaluation of variation; a need to use of oral dose equivalents to compare AUC values to the l owest effect l evel (LEL) in uter otrophic studies; m etabolic/biotransformation capacities not included; exclusive AOP focus on nuclear ER without evaluation of alternative receptors/pathways (i.e. GPER); and vague description of potency categories

.

Overall, with minor limitations for c ompounds that require metabolic activation or have targets other than nuclear receptors, the ER AUC appears to be an appropriate tool for chemical prioritization for L ist 1, L ist 2 and the EDSP universe compounds.

e . Based on all the data presented in Sec tion 2 on ER AUC model performance including characterization of reference chemicals, concordance with in vivo uterotrophic results, and comparison with Tier 1 assay endpoints, what are strengths and limitations of the ER AUC

model to contribute to the wei ght of evidence determination of a chemical’s potential estrogen bioactivity?

Summary

The Panel noted that the ER AUC model had several strengths. They concluded that ER AUC model showed outstanding performance in the characterization of reference chemic als, and in

concordance with the selected chemicals for in vivo uterotrophic responses (literature and Tier 1 endpoints). The use of the battery of ER AUC fits the AOP for the linear pathway of ER activation, DNA binding of receptor

, and trans

- activation o f the receptor. Use of the AOP paradigm will eventually allow the ability to assess mixtures on combined modes of action or antagonistic pathways once complete molecular pathways have been identified. Use of redundancy and complementary endpoints of ER act ivation and pathway provides qualitative confirmation of activity.

The Panel also noted several limitations of the ER AUC model which included : an o mission of

the r ainbow trout liver slice data (ER expert system); a general failure to “weigh” modular eve nts in the AOP process; a lack of comparisons between ER AUC data and other in vivo FSTRA and Pubertal assays; a lack of consideration for non

- monotonicity in ER evaluations; a narrow focus of nuclear ER activation, and lack of metabolic capabilities in th e in vitro systems.

f. Based on all the data presented in Section 2 on ER AUC model performance including characterization of reference chemicals, concordance with in vivo uterotrophic results, and comparison with Tier 1 assay endpoints, what are strength s and limitations of us i ng the ER AUC

model to substitute for EDSP Tier 1 ER binding, ER transactivation, or u terotrophic assays for the purpose of characterizing a chemical’s potential estrogen bioactivity?14

Summary

The need to rapidly evaluate anthropog enic compounds for estrogen

- like activity is critical given that several have been identified as potentially estrogenic

. T he EPA’s efforts to develop a process accommodating this need is commendable. In general

, the ER AUC model will capture many, though n ot all, of the estrogenic compounds in the “Universe of Estrogen Disrupting Compounds.” Overall, because both the ER AUC model and the Tier 1 in vitro assays capture either nuclear receptor binding and/or transactivation, replacement of the Tier 1 in vitro ER endpoints (ER binding and ERTA) with the ER AUC model will likely be a more effective and sensitive measure for the occurrence of estrogenic activity that occurs through nuclear receptor binding and activation.

The Panel found that the data comparing the ER AUC model to the uterotrophic assay were strong for the reference compounds that were clearly estrogenic (high AUC values) or unmistakably not estrogenic (very low AUC values) . However , the model outcomes were less straight forward when the ER AUC model for non

- reference chemicals was compared to uterotrophic studies where the data were limited or discordant. Furthermore, the data for all of the assays utilizing the List 1 and List 2 battery were below the AUC cut off of 0.1 and were consistently n egative for the uterotropic assay. This finding suggests a very low risk of false negatives in this data set, but was limited by the fact that there were no chemicals with either high or intermediate AUC model values available for functional comparison. A second, but related concern remains regarding the ability of ToxCast in vitro methods to assess estrogenic effects that may be impacted by absorption and metabolism or may result through non

- nuclear estrogen receptor signaling pathways.

For these reasons , the Panel did not recommend that the uterotrophic assay be substituted by the AUC model at this time. The Panel suggested that the EPA considers: 1) conducting limited uterotrophic and other Tier 1 in vivo assay testing, using the original Tier 1 Guideli nes (and/or through literature curation), for a limited number of chemicals in the low and middle AUC value range from the “Universe of Chemicals” shown in Fig. 2.16 of the Agency’s White Paper . Such testing would then add weight

- of

- evidence supporting or refuting the hypothesis that the ToxCast suite of assays is predictive of in vivo estrogenic responsiveness, and 2) performing similar comparisons of the ER AUC model to the other in vivo assays in the Tier 1 estrogenic battery

.

TOPIC: ANDROGEN BIOACTIVIT Y

2) EPA’s proposed approach for quantifying a chemical’s potential androgen bioactivity is based on a computational model integrating data from nine high throughput ToxCast assays measuring several endpoints along the androgen receptor (AR) signaling pat hway. The computational model outputs are expressed as area under the curve (AUC) scores for AR agonist (R1) and antagonist (R2) bioactivity. Before routinely using the AR computational models in the EDSP framework, EPA is reviewing the scientific strength s and limitations of the AR AUC model

described in this White Paper to: i) prioritize chemicals for further EDSP screening and testing

based on estimated bioactivity, and ii) contribute to the weight of evidence evaluation of a15

chemical’s potential bioacti vity. Please address the following charge questions relevant to

Section 3 of the White Paper and androgen bioactivity:

a . How clearly has EPA described the computational tools in Section 3.1 ( i.e. , high throughput assays and models) used to estimate AR ag onist and antagonist bioactivity?

Summary

Results from the battery of assays to estimate AR agonist and antagonist bioactivity of chemicals was presented as preliminary and thereby represents a work in progress. The overall consensus was that the assay b attery and computational tools represent a major step forward in defining and quantifying chemical agonist and antagonist activities mediated via the nuclear androgen receptor pathway. In particular, the implementation of high throughput in vitro recombina nt protein and cell based bioactivity assays that provide an integrated assessment of chemicals affecting the androgen receptor activity pathway was viewed as a significant advancement intended to supplant the narrowly focused, highly variable, labor inten sive

, and animal

- based in

vitro rat prostate cytosol androgen receptor binding assay in Tier 1 testing. Further refinement of the assay battery was encouraged to optimize the assessment of chemical activities affecting the AR pathway, especially in the con text of antagonism of AR bioactivity which defines the action of a majority of chemicals tested to date. Specifically, the assays and computational tools must be sufficiently robust so as to distinguish true AR antagonism from cellular/molecular toxicity. Although the AR test battery is under development, future efforts should be guided by lessons learned from parallel development of the high throughput test battery and computational methods applied to chemical effects on ER agonist and antagonist bioactivi ty as described in Section 1, Estrogen Bioactivity.

b . What are strengths and limitations of the AR AUC model’s ability to identify reference chemicals that include a variety of structures and have a wide range of in vitro AR bioactivities?

Summary

The overall strengths of the AR AUC model outweigh the limitations. The use of complementary

in vitro recombinant protein and cell based assays provide a high throughput modality for the rapid screening of chemicals through an integrated battery of assays for the assessment of AR bioactivity with the benefit of reducing the need for animals. The assays significantly improve the reliability of assessing AR binding (replacing the rat prostate cytosol AR binding assay) and provide new measures of AR transcriptiona l activity previously not included in the Tier 1 screening. In general, the assays and computational model are designed to distinguish AR agonist and antagonist bioactivities. However, results reported to date cover a rather narrow range of chemical struct ures and the distribution of calculated values for AR bioactivity fall within a limited range of AR bioactivities. AR AUC values primarily represent known pharmaceuticals with strong agonist activities that appear at the top of the AUC scale or test

chemic als with antagonist activities that appear at the bottom of the AUC scale, thus the AUC computational model places these chemicals at disparate ends of the value range. Because chemicals with AR antagonist activity have low values according to the AUC com putational model, there is an inherent risk for misinterpretation of cytotoxicity as AR antagonist activity.16

c . EPA plans to use data from published in vivo studies that are methodologically consistent with EDSP Tier 1 guidelines to evaluate concordance between AR Page 3 of 3 AUC model scores of

in vitro bioactivity, and the in vivo androgenic and antiandrogenic responses (Section 3.2.1). What are strengths and limitations of the planned curation methods and quality standards for evaluating published in v ivo studies?

Summary

Preliminary data with the assay battery for assessing the in vitro AR bioactivity of chemicals

correlated well with the in vivo Hershberger androgen bioactivity assay data from the literature. This provides initial support that in vi tro AR bioactivity assays will replicate in vivo chemical activities and will be especially beneficial as a rapid, high throughput screen in Tier 1 testing. An

obvious limitation of in vitro screening assays i s the inability to fully appreciate or replicat e the multiplicity of biological actions that chemicals produce in vivo . Included among these limitations are differences in routes of exposure and dosing, activation or inactivation of chemicals via metabolism, non

- genomic androgenic effects and potential off

- target effects of

chemicals that occur in vivo but are not appreciated within in vitro assays.

d . Based on the data presented in Section 3 on AR AUC model’s performance, what are strengths and limitations of using the AR AUC model to distinguish and prioritize chemicals based on potential androgen bioactivity?

Summary

The AUC model integrates a high throughput battery of assays to probe a linear array of steps in the AR pathway that provides for significant improvement in efficiencies and effectivel y complements other aspects of the Tier 1 testing program. The test battery incorporates redundancy among different species and different assay methodologies to increase confidence in the values assigned by the AUC model calculations. The downside of assay redundancy lies in the balance that each individual assay contributes to the AUC model and how the so

- called “penalty term” is applied to the AUC calculation. The AUC model reliably predicts the agonist and antagonist properties of reference chemicals whi le minimizing false positives due to overt cellular chemical cytotoxicity. The AUC calculation distinguishes between chemicals with agonist activity at the top of the range and those with antagonist activity at the bottom of the range, however the AUC valu es are so closely grouped within each category of agonist or antagonist activity that it fails to clearly differentiate between potencies of individual chemicals. The AUC model does not effectively account for the possibility of non

- monotonic responses. A broader range of chemicals with structure variation should be explored using the AUC model to provide confidence and demonstrate the robust nature of the model.17

TOPIC: INTEGRATED BIOACTIVITY EXPOSURE RANKING (IBER)

3) For Endocrine Disruptor Screen ing Program (EDSP) chemicals with ToxCast estrogen receptor (ER) and androgen receptor (AR) bioactivity scores (Section 2 and 3), and ExpoCast high throughput toxicokinetics and exposure estimates (Sections 4 and 5), the IBER approach

was used to rank chem icals based on the ratio between the bioactivity dose range, and the expected exposure range (Section 6). The IBER approach extends point estimates of bioactivity, toxicokinetics, and exposure for a chemical, to distribution ranges based on uncertainty and population variability. Chemical rankings are based on the ratio of the lower range of the bioactive dose, to the upper range of the exposure estimate. Please address the following charge questions relevant to Section 6 of the White Paper and the IBER app roach:

a . How clearly has EPA described the computational tools in Section 6 to develop IBER values, including modeling uncertainty and population variability?

Summary

The Panel noted that the description of the IBER approach to prioritize chemicals wa s adequately

clear. The Panel commend ed the Agency for developing a metric that aims at capturing the “worst case scenario”, while attempting to account for uncertainty and variability in both chemical bioactivity and population exposure. While this effort is a good starting point, the Panel

believe d that there are several areas of further development, particularly with respect to modeling and accounting for sources of uncertainty and variability. T he Panel encourage d the Agency to develop approaches that a ccount for all the sources of uncertainty and variability and model them jointly rather than via multi

- step approaches which fail to propagate the uncertainty with a resulting underestimation of uncertainty. More specific comments and suggestions on

approa ches to properly account for uncertainty and variability are provided in the Detailed Panel Recommendation portion of this Report.

b . What are strengths and limitations of using the IBER approach to prioritize chemicals for further EDSP screening based on the ratio between the ER bioactivity dose range, and the expected exposure range?

Summary

The Systematic Empirical Evaluation of Models (SEEM) effort was logically developed . It also provided a theoretical framework with the potential to prioritize furt her EDSP screening of compounds with estrogenic activity through the IBER approach. The statistical models and methods used are complex enough to capture many of the potential sources of variability in bioactivity and exposure . However, they are simple eno ugh to allow for straightforward scientific interpretation, model validation , and most importantly further development. In practice, the applicability and reliability of the IBER approach (f or the purpose of identifying really high priority chemicals ) dep ends on how efficiently and appropriate ly the distribution of bioactivity levels and exposure concentrations were derived. The IBER approach provides a

means to take into account population variability via Monte Carlo simulations and in the future through application of the SHEDS model. Therefore, the Panel noted that the approach was18

theoretically sound for us ing data and predictive models to produce estimates with respective

uncertainties f

or both exposure and bioactivity. Using the IBER ratio value is in theory sufficiently “ fit

- for

- purpose ” for potentially prioritizing EDSP chemicals.

However, the Panel also expressed several concerns regarding the IBER model that suggest that it will need refinement prior to utilization. There were several limitations that the Panel and the

EPA acknowledged and identified. The Panel ’s concerns regarding the bioactivity interpretations were already addressed in Question 3a . However, the Panel concurred that there is a need to have better exposure monitoring data availa ble to strengthen the model. The Panel was concerned about the limited amount of NHANES exposure data incorporated into the model. This data set was much less robust in comparison to the data available through ToxCast for bioactivity. The Panel was uneasy with predictions for chemicals for which biomonitoring data

were not available or w ere limited. Not having any or limited data to validate the predictions, the HTE/SEEM models appeared to be more an extrapolation exercise given that the large majority

of c hemicals do not have measurements available for validation of the model. The Panel

commented that the assumptions and parameters used to develop the exposure model had potential problems. The assumptions (such as for example whether or not the pesticide ac tive or inert ingredients is modeled) provide the potential for poorly estimating the exposure data values. Also, the model may need to incorporate more parameters, and/or weight the parameters in place, to help strengthen the model output. The Panel was a lso concerned that specific human populations such as agricultural workers, chemical formulators and pregnant women, who may have the highest exposure levels for specific compounds were not always taken into account.

The Panel had several recommendation s to improve the model

. Such as they noted that the

i nclusion of a Bayesian model formulation that combines the pharmacodynamics and pharmacokinetics together is warranted. Model uncertainty might be addressed by combining predictions from different models together with population variability through Monte Carlo sampling. One Panel member suggested another, and perhaps statistically more appropriate, approach to SEEM, which would be to develop a framework similar to the Collaborative Estrogen Receptor Activ ity Prediction Project ( CERAPP ) and combine multiple statistical models together. Another possible alternative approach is to propagate the uncertainty in estimates of exposure and bioactivity forward to produce probability statements about the entire

dist ribution of the ratio of these two values. Last ly the Panel commented that the model should undergo a validation process as what w as done with the AUC model approach.

c. What are strengths and limitations of using the IBER approach to prioritize chemical s for further EDSP screening based on the ratio between the AR bioactivity dose range, and the expected exposure range?19

Summary

The Panel determined the strengths and weaknesses of the IBER approach to prioritize AR active

chemicals for further ED SP screening were largely the same as those identified in their response to Charge Question 3b for ER active compounds. Theoretically, the model is sound for this approach, but it needs refinement followed by validation prior to implementation by EDSP.

How ever, the Panel found that the IBER process utilizing the androgenic assays and data for the exposure modeling were not as far along in their development as they were for the estrogenic

IBER . Thus, it was difficult to evaluate whether the approach will wor k for this set of compounds. In particular, the Panel thought there would be an increased risk of chemicals not becoming prioritized when they should be because of failure of the exposure model component of the IBER approach.

The Panel also noted that whe n using creatinine adjusted values racial, sex

, and age differences

in creatinine excretion should be considered. Also, biotransformation products should be

considered. Last ly , the Panel emphasized , across all charge questions, that it will be critical to incorporate exposure to complex chemical mixtures in the model to ultimately understand the real exposure risks.20 D ETAILED PANEL RECOMMENDATIONS

TOPIC: ESTROGEN BIOACTIVITY

1) EPA’s proposed approach for quantifying a chemical ’s potential estrogen bioactivity is based on a computational model integrating data from 18 high throughput ToxCast assays measuring several endpoints along the estrogen receptor (ER) signaling pathway. The computational model outputs are expressed as are a under the curve (AUC) scores for ER agonist (R1) and antagonist (R2) bioactivity. Before routinely using the ER computational model in the Endocrine Disruptor Screening Program (EDSP) framework, EPA is reviewing the scientific strengths and limitations

o f the ER model described in the White Paper to: i) prioritize chemicals for further EDSP screening and testing based on estimated bioactivity, ii) contribute to the weight of evidence evaluation of a chemical’s potential bioactivity, and iii) substitute fo r specific endpoints in the EDSP Tier 1 battery. Please address the following charge questions relevant to Section 2 of the White Paper and estrogen bioactivity:

Question 1 a . How clearly has EPA described the computational tools in Section 2.1 ( i.e. , hig h

throughput assays and models) used to estimate ER agonist and antagonist bioactivity?

Overall, the White Paper clearly describe d the conceptual approach that EPA is taking to estimate ER agonist and antagonist activity by assessing 18 assays that measur e various responses triggered by interaction with nuclear estrogen receptors. The model described attempts to evaluate activity across a 10,000

- fold dose range and take into account interfering effects of cytotoxicity or other non

- receptor

- mediated trigge rs of the measured responses. The issue presented for discussion was not whether there are other models that might perform better, but how well this model served its purpose in the prioritization of chemicals that have the potential to interact with nucle ar estrogen receptors for further testing. The Agency’s use of reference chemicals to evaluate the performance of the model was a very important step

. EPA

was careful in its description of the process for selecting this set and evaluating the models on it.

The EPA should be commended for making ToxCast data and previous model codes publicly

available

, and the Panel recommended that the R code for the IBER models be made publically

available as well. Careful editing of the White Paper would greatly improve i ts readability and clarity. It was the Panel’s understanding that the White Paper is a work in progress; however

there were many cases where poor grammar, imprecise wording, and repetition made the

document difficult to read.

T he Agency state d in their W hite Paper that “Rather than summarizing or reconsidering the research that has been presented previously, the aim of this document and the subsequent SAP meeting is to request scientific input on EPA’s proposed method for applying computational

tools” . Ho wever the Panel believed that a more inclusive background on the relationship of the proposed tools and assays to previously presented work would add greatly to the understanding of the approach and tools. It appear ed to the Panel that the Agency was reque sting that the computational procedures described in the White Paper be evaluated without consideration of

these or other efforts. Presenting the current work in the context of other tools would provide more clarity . As a result, the White paper would be s trengthened and more transparent by including a critical discussion of related work. For instance a background or supporting21

documentation would compensate for the very limited description of the ToxCast High Throughput Screening assays and their validatio n in the White Paper

. In addition, example it was unclear how the current EPA EDSP Tox C ast effort fits into prior Agency work (e.g., Reif et al. 2010; Filer et al. 2014) and other U.S. Tox21 collaborative programs or prior SAP reviews on in

vitro assays an d new high throughput exposure models (e.g., Jan 2013 SAP; July 2014 SAP; Kavlock et al. 2012, Rotroff et al. 2013, Judson et al. 2014; Rotroff et al. 2010, Wetmore et al. 2013, Wetmore et al. 2014; Wetmore et al. 2011, Wambaugh et al. 2013, Wambaugh et al

. 2014)

.

Due to the lack of assay background information, SAP members had several questions regarding the nature, reproducibility and validation of the assays . They were unable to find complete information on these assays via website/database/literature s earches to evaluate their strengths and weaknesses. For example, the ‘Assay Providers’ link on the EPA’s ToxCast webpage was

nonfunctional at the time of the SAP public meeting. Based on the information presented, it was difficult to tell how these assays were validated, if at all. Since data generated from these 18

assays were used to develop the ER AUC model, the lack of assay information made it difficult to determine if these assays are appropriate to replace Tier 1 assays in the EDSP. Tier 1 EDSP

assay s were extensively validated prior to evaluation by a prior SAP. It is unclear whether this

w as done for the ToxCast assays used in the model

. If the assays were not validated a rational e should be given in the White P aper. Because of the superficial treat ment of the assays, it was not clear that the chosen HTP assays represent the best methodology or that they are indeed state

- of

the

- art.

It was noted by one Panel ist that there were problems with assay interference, and that this

varied depending on the t ype of assay. As a result, the time points chosen for the assays may be too long (8 and 24 hours) to identify direct actions, or may lead to detection of indirect effects on the signaling pathways assayed.

The Panel suggested that the Agency add a summary discussion in the White Paper o n the Q uality C ontrol/ A ssurance (QAQC) procedures used. At a minimum, citations of on

- going efforts should be described. For example, some information was available for download from the

ToxCast web site. This site provided a summary (as of Oct. 2014) of chemical purity and Molecular Weight validation of chemicals in the ToxCast Universe. There were 12,779 entries with 57% flagged as passed, 36 % as “not determined” (analyses in progress) and 6% as

“caution”. Similar metrics on the subset of chemicals examined in in vitro estrogen and androgen activity screening programs were not available. Such data would add confidence to results. Other QA/QC metrics that are also critical include results of the replicate analyses with the e strogen and androgen screening matri c es. It was noted that area under the curve (AUC) values presented in the text are presented as unit values in most cases. Clarity of these results would be improved by reporting error bounds linked to analysis of blind replicates in the ToxCast batteries.

Although the overall approach was described in the White Paper

, the Panel indicated that there

were several aspects that were vague

. The presentations given by EPA during the public SAP meeting were helpful in clarify ing some aspects that were unclear in the written material. For example i n the discussion surrounding Equation (2.1), it would have be en helpful to have more description regarding the meaning and justification for the use of the modified Z

- score, the22

penal ty term, and the explanation of “discounting” if one receptor was activated at lower doses

than others. In particular, a n explanation justifying the use of the median and MAD, as opposed to mean and standard deviation, was warranted. Further discussion of the effect of estimation error in the penalized regression model (2.2) would also have be en useful. The Panel

recommended that the Model goodness

- of

- fit be investigated.

The Panel also recommended that the description of the penalized least

- squares minimi zation procedure be improved. In particular, the estimation procedure use d a method similar to a ridge regression penalty (Hoerl and Kennard, 1970). Th is method ology appear ed to be impromptu

, particularly with respect to the turning parameters SR 0 and α. While EPA noted that selected
values of these parameters result in sensible results (satisfying the “fit

- for

- purpose” criterion), the Panel questioned the sensitivity of these results (e.g., R j estimates, AUC calculations) and the

selection of these parameters. The Panel advised the Agency to explore the use of cross

validation to select tuning parameter values

. It may also be possible to adapt a hierarchical group LASSO (Zhou and Zhu, 2010) approach for this problem to lend more credibility to the c urve

fitting procedure.

The Panel advised that it would be important to add mo re d etails to the White Paper regarding the effects of weighting (or the lack thereof ) on the integration of different assay results. It is

well

- known in the statistical resear ch literature that using a weighted combination of multiple models generally improves predictive performance . Thus, the Panel commended EPA for its

efforts with the Collaborative Estrogen Receptor Activity Prediction Project ( CERAPP ) program. However, the description of the consensus model construction warrants more clarity and explanation

. There are many different ways to combine the results of multiple predictive models, such as model averaging, decision forests, cross

- validation, etc. It is unclear which specific methods were used by the Agency

. The White P aper does not disclose which methods the Agency used to combine model predictions

. Several Panel ists noted that the White Paper description of the us e of Z

- scores to quantify cytotoxicity was vague (tho ugh more clarity was provided in EPA presentations at the public meeting ). A specific example for illustration would

have be en helpful. Additionally, a more thorough explanation of the relationship of dose to AUC

calculations would minimize the potential f or confusion

.

The Panel remarked that it would have also be en useful if the Agency provided a more detailed explanation of the potential shortcomings of the estimation methodology (e.g., non

- unique solutions), and how these shortcoming might affect “downs tream” uses of the estimators (e.g., producing AUC values). In particular, while EPA attempts to quantify modeling uncertainty in IBER in Section 6, it does not appear that all uncertainty is carried forward.

The Panel advised that a section in the White Paper be added to describe and emphasize the importance of redundancy, especially with regards to orthogonal assays even though presentations by EPA during the public SAP meeting clarified many of the Panel ’ s questions

. Although the main goal of the prese ntations was to assess the utility of this model, it was stated

in the White Paper that it would be un likely that all 18 assays would be available for the

assessment of all chemicals in the future. Thus, the Panel noted that some discussion of what

subset of assays would be most useful would be helpful.23

Question 1 b . What are strengths and limitations of the ER AUC model’s ability to identify reference chemicals that include a variety of structures and have a wide range of in vitro ER

bioactivities?

The E R AUC model was found to be a commendable attempt by the EPA to develop a statistical model and a computational tool based on a series of in vitro and cell

- based assays. The model allows for prediction of potential estrogenic activity of chemicals in vitro , thus reducing the need for more expensive animal testing. By analyzing several chemicals in parallel and using a combination of assays, the ER AUC is both computationally and time efficient . It also provides a

robust assessment of the in vitro estrogenic ity of a chemical by allowing detection of technology

specific false positives and negatives.

The ER AUC model has a very simple mathematical formulation based on two main assumptions: (i) the assay signal represents a true, nuclear receptor

- mediated proc ess and (ii) there is a linear relationship and no

- loss of information in the transmission from the receptor to the assay signal. Despite its simplicity, the results presented in the White Paper indicate d that the

ER AUC model was able to detect reference chemicals that act as agonists. In particular, results

indicate d that the model was able to detect weak agonists better than the Tier 1 EDSP assays.

With respect to antagonist reference chemicals, the Panel observed that the model had a slightly

less ro bust capacity of detection when compared to agonists due to the confounding effects of cytotoxicity and cellular stress. Nevertheless , in general, the results produced by the ER AUC

model were as good or better than results obtained using the three assays that it is intended to replace in the existing Tier 1 screening battery.

The simplicity, parsimony, and interpretability of the ER AUC model were identified as strengths . Several competing statistical models for predicting estrogenic activities of chemic als

based on cell

- based assay data might be formulated, assuming different physiological relationships between the cellular signaling pathway and assay signal. It is possible that more complex models provide a slightly better performance when evaluated via the reference chemicals, especially with respect to antagonist chemicals. However, as the priority of the EDSP chemical screening is on estrogen agonists, the Panel noted that it was unlikely that more complex models would yield significantly better resul ts than the ER AUC model. In the presence of an equivalent predictive ability on reference chemicals yielded by various alternative and competing models, the Panel recommended that priority be given to models that are more parsimonious and succinct, as ide ntified by Occam razor’s principle (Jefferies and Berger, 1992)

.

The Panel recognized some limitations with the ER AUC modeling approach, and suggested that the EPA investigate the sensitivity of the ER AUC results to modeling assumptions and

estimation m ethods as well as consider different modeling strategies in an effort to improve the ER AUC predictive performance for antagonist chemicals. T he Panel noted that the results of the ER AUC’s model for the receptor pathways depend on the pathway signals, whi ch are in turn estimated using a constrained penalized least square approach. It is well reported in the statistical literature that parameter estimates derived using a penalized least squares approach are influenced by the penalization constant. In partic ular, model parameters “shrink” towards zero as the magnitude of the penalty term changes.24

The Panel noted that the penalty term in the ER AUC model has been determined in an ad

- hoc way and not via cross

- validation as is standard practice in this type of problem ( Golub et al. 1979; Hastie et al. 2009). As the choice of the penalty term might have an effect on the estimated pathway signal and thus on the ER AUC predictive performance, the Panel encouraged EPA to carry out a sensitivity analysis to determin e how dependent the predicted ER AUC values are on the penalty term adopted. The Panel hypothesized that large ER AUC values are likely to be robust to changes in the penalty term, and thus results for high activity chemicals may not change significantly. However, the ER AUC results for moderate or low activity chemicals with ER AUC closer to the activity threshold may be affected by changes in the penalty term.

The adoption of a threshold of 0.1 for ER AUC values to identify highly active chemical s is no t clearly defined in the White Paper . The classification of chemicals as true/false positive and, respectively, true/false negatives depends on the threshold value of ER AUC adopted. Given the uncertainty in the estimated receptor signals, which in turn pr opagated to uncertainty in the ER AUC values, it is plausible that reference chemicals with ER AUC values near the threshold might be incorrectly misclassified (ref. Figure 2.4 in the White Paper ) . This would chang e the accuracy of the ER AUC model on the reference chemicals. Although, the Panel recognized that the uncertainty in the estimated receptor signals will likely affect the predicted classification of

the moderate/low

- activity chemicals which are not the priority chemicals at the moment, the

Panel believed that uncertainty in the threshold and consequent classification of chemicals should be acknowledged.

The Panel also noticed that the ER AUC values were derived not by using raw data arising from replicates of assay experiments, but on the best

- fi tting Hill, Gain

- Loss or constant models fitted to the raw data selected using model fitting criteria such as the Akaike Information Criteria (AIC). While Hill or Gain

- Loss models have been previously used in the literature, the Panel acknowledged that the model fitting strategy used in the ER AUC model introduces additional uncertainty. The Panel suggested that data arising from each assay replicate be used to derive several curves relating a chemical concentration to an assay signal (each corresponding to a replicate), yielding in turn a range of ER AUC values from which mean/median and confidence intervals could be derived, communicating the uncertainty in the estimated bioactivity of the

chemical

.

As mentioned above, the ER AUC model was based on the as sumption of a linear relationship between assay signal and receptor signal. While the Panel recognize d that this is a good starting assumption, there may exist different functional relationship s between assay and receptor signals. The Panel encourage d EPA to investigate different type s of functional relationship s
, especially if this endeavor might improve the ER AUC model ’ s predictive performance for antagonist chemicals.

In conclusion, the ER AUC model yielded good predictive results on the reference chem icals, which, constituting the training set, have possibly guided the selection of the assays mostly focused on nuclear estrogen receptors. As discussed in the public presentations, there is uncertainty with how well the chemicals examined to this point su fficiently cover the structural classes that are present in the entire universe of chemicals to be assessed.25

Question 1 c . EPA used data from published in vivo studies that are methodologically consistent with EDSP Tier 1 guidelines to evaluate concordance between ER AUC model scores of in vitro bioactivity, and the in vivo uterotrophic response studies (Section 2.2.1). What are strengths and

limitations of the curation methods and quality standards used for evaluating published in vivo

studies?

The Panel concluded that the methods used to curate and standardize the literature studies for comparing the in vivo uterotrophic endpoints to the ER AUC model scores were in most cases sufficient. The separation of studies between injection and oral dosing strategi es to identify potential study design discordance was valid

. The separation of studies using immature and adult ovariectomized rodent protocols was also warranted to detect design discrepancies. The Panel

noted that future studies should focus on using rat s given the insensitivity of m ice

.

The ER AUC provided remarkable sensitivity of estimates of uterotrophic responses particularly for low potency compounds with balanced accuracy of 95%, positive predictive value of 97%, and negative predictive value of 9 2%.

In regard to limitations of these methods, w hile the literature evaluations indicated strength of comparisons (i n vivo to ER AUC) for high and low potency compounds

, variability (18/70

chemicals or 26% discordant) was significant. Although the Panel agreed with the Agency that this is likely due to differences in design and the limited number of studies conducted for each chemical, it could also be due to the following errors, which in some cases may or may not be

determined from stric t literature eva luation but whenever possible should be evaluated:

1. Chemical purity and availability in dosing materials . This item is not a subject specific to literature evaluations, but should be considered in all HTS assays. Recent HTS studies with zebrafish embryos in dicated that a contaminant from a flame retardant that was present in less than 1% was responsible for significant adverse effects (Bugel and Tanguay 2015).

2. Site of injection . While it was stated that 99% of injections were performed with

subcutaneous meth ods, the site of injection was not available. The site can either be in the neck or abdomen which could likely affect distribution to the portal venous system and first pass metabolism.

3. Differences in solvent/carrier controls

. It was stated that some studi es used different carriers (ethanol vs. saline) and this could lead to variability between studies.

4. Specifics on strain differences between animals

. While strain differences between animals

was stated in the oral presentation; it was not discussed in the A gency’s White Paper ( i.e. how many “rat” studies used ; what strain of rat specimen was used i.e. Sprague Dawley?)

. 5. Limited sample size s . While the presentation stated that a minimum treatment number was 5,

the text indicated control treatment values of 3. This low value is unacceptable and likely

contributes to the variance between studies. The original Tier 1 guidelines suggest a minimum sample size of 6 for all treatments. Given the number of treatments (4), a sample size of three in the control group pr ovides very little statistical power.

6. Systemic toxicity evaluations

. The Panel suggested that it is imperative to maintain strict observations of body weight during exposure as this is a primary indication of overt toxicity which can clearly confound endo crine re s ponses.26

The Panel noted that additional methodological limitations included the following : 1. The uterotrophic responses were lacking for LELs between 0.01 and 1 (slide 16/22 Casey Presentation). To have more points on the curve, additional studies that compared uterotrophic responses with LELs are necessary.

2. Figure 2.12

-- For Reference compounds there were only 3 values between LELs of 0.2 and 0.001. Similarly with only values provided between higher potency (0.2) and lower potency (0.001) metrics, additional data between these would enhance certainty and may reduce discordancy.

3. Figure 2.15

-- F or non

- reference compounds there was only one value between LELs of 0.02

and 0.001. As mentioned above, additional data points are needed.

4. Concern that liter ature based curation was not performed on other in vivo Tier 1 responses including the FSTRA, rat pubertal

.

Overall, with the high degree of variability, and only having one false positive and one false negative out of 42 compounds (Figure 2.12) the ER AU C model was surprisingly accurate. However, additional studies are needed that focus on metabolically activated compounds ( e.g. pyrethroids, and methoxychlor which should be more appropriately dectected in vivo )

.

Question 1 d . Based on all the data presen ted in Section 2 on ER AUC model performance including characterization of reference chemicals, and concordance with in vivo uterotrophic results, what are strengths and limitations of using the ER AUC Page 2 of 3 model to distinguish and prioritize chemic als based on potential estrogen bioactivity?

T he Panel concurred ( assuming linear hER binding and/or activation through nuclear receptors )

that the AUC model is a relatively simple, but elegant, approach for integrating different bioassay data into a sing le value that is proportional to the bioactivity or potency of a chemical. The AUC ranking clearly has a strong applicability for ranking chemicals in terms of potential estrogen bioactivity. It seems that more information from the bioassay AC50 curves cou ld be utilized to understand uncertainty surrounding the bioactivity AUC value. The Panel noted that

s pecific strengths of the model include:

1. Excellen t performance for reference chemicals

. 2. Outcomes for List s 1 and 2 and Universe seemed consistent with acti ve and non

- active descriptions.

3. Redundancy and power of multiple HTS signals for confirmation of “hits

. ”

4. Strong predictive accuracy of greater than 90% and

5. Strong correlation across potency categories for reference chemicals

.

The Panel also noted the fol lowing l imitations:

1. Loss in Variation in AUCs: Variation in AUCs between different bioassays was lost with the use of a mean, or median, AUC score for bioactivity from the active assays in the total 18 assays examined. This results in losing measures o f uncertainty surrounding the range of active assays for a chemical. Since the potency relationship of the different in vitro tests to

in vivo results may be different, using the lowest AUC for a chemical as a lower bound for bioactivity and the highest a s the upper bound may give a better approximation of the in

vitro bioactivity impact relative to in vivo effects. A similar approach is used in S ection 6 of27

the White Paper where the median and minimum in vitro active concentration at the cutoff

was calcul ated.

2. Comparison of a single number (mean?) AUC value to the Lowest effect Level in an uterotrophic study

. Comparing the mean AUC value to the l owest effect level, appears to be similar to comparing apples to oranges and has a lack of biological releva nce between the measures. A more meaningful comparison may be to compare the range of BMD in oral dose equivalents (mg/kg/d) derived from AUCs via biological pathway activation concentrations and HTTK approaches as developed in Section 5 and applied in S ec tion 6 of

the Agency’s W hite P aper to BMD values derived from uterotrophic studies (assuming that such studies had appropriate data to calculate a BMD). This may result in a more relevant comparison, as it would include some rudimentary toxicokinetics.

3. Metabolism not included : Reference chemicals should include those that are metabolically activated. Metabolic activation can be evaluated using incubations with human and rat s

- 9 prior to cell treatment. Examples of metabolically active reference chemic als include methoxychlor, permethrin, PCBs, other PAHs. While uterotrophic assays may enhance clearance with greater metabolism, activation to estrogenic metabolites may be significantly important in sensitive developmental windows.

4. Nuclear receptor foc us : The Panel recommended that the Agency explore alternative receptors/pathways (i.e

. GPER) instead of focusing primarily on nuclear receptors

. 5. Selection of potency categories: The Agency’s s election and differentiation of potency categories was vague ( i.e. the Panel was unclear about what the term “modest potency”

mean s numerically? (Refer to s lide 63/65 in the EPA presentation during the public meeting entitled “Section 2.1:Computational Tools & Models” was much more quantitative .)

Overall the Pan el noted that with minor limitations for compounds that require metabolic

activation

, ER AUC appears to be an appropriate tool for chemical prioritization for List 1, L ist 2 and EDSP universe compounds.

Question 1e

. Based on all the data presented in Sect ion 2 on ER AUC model performance including characterization of reference chemicals, concordance with in vivo uterotrophic results, and comparison with Tier 1 assay endpoints, what are strengths and limitations of the ER AUC model to contribute to the weig ht of evidence determination of a chemical’s potential estrogen bioactivity?

The Panel conclude d that ER AUC model had several s trengths

. For example

, the ER AUC showed outstanding performance in the characterization of reference chemicals, and concordanc e with the selected chemicals for i n vivo uterotrophic responses (literature and Tier 1 endpoints)

. In addition, u se of the battery of ER AUC fits the AOP pathway for the linear pathway of ER activation, and DNA binding of receptor and trans

- activation of the receptor. Also u se of the AOP paradigm will eventually allow the ability to assess mixtures. This may occur additively on combined modes of action or assess antagonistic chemical pathways once complete molecular pathways have been identified. They Pane l also positively noted that u se of redundancy as well

as complementary endpoints of ER activation and pathway provides qualitative confirmation of activity. This attribute was initially an objective of EDSTAC ( http://epa.gov/endo/pubs/edspoverview/finalrpt.htm ) . This should be maintained unless

advances in science dictate otherwise.28

In regard to the l imitations of the ER AUC model the Panel noted that the r ainbow trout liver slice data (E R expert system) should be implemented given the multiple nodes in the AOP

process using this tool provides including metabolic activation prior to molecular initiation as well as cellular and protein responses . This tool is not only effective for ecologic al and wildlife evaluations, it is also effective for overall AOP evaluations for estrogenic activity. In addition, i n follow up to the July 2013 FIFRA SAP meeting on “ Weight

- of

- Evidence (WoE): Evaluating Results of EDSP Tier 1 Screening” a more transparen t and objective description of the WoE process for prioritization was found necessary. For example, how will various endpoints and potencies be “weighted” in evaluation. Thus, at some point, it will be necessary to quantify modular relationships in the AOP process. ER AUC data should be compared against additional

in vivo fish short

- term reproduction assay ( FSTRA ) and p ubertal assays as these assays have demonstrated significant sensitivity in previous EDSP evaluations (Ankley and Gray 2013).

There wa s a ls o lack of consideration for non

- monotonicity in ER evaluations. Multiple studies

have observed non

- monotonic responses within in vitro and in vivo systems at low chemical concentrations (Vandenberg 2012 Bergman et al. 2013). This contrasts ToxCast data tha t had

limited examples of non

- monotonic responses , and these were exclusively related to cytotoxicity.

There wa s clearly a discrepancy here

, and additional evaluations are necessary to determine why

it exists. Overall, estrogenic activity is narrowly defin ed as nuclear ER activation, and it is recommended that additional assays be employed as they become available to assess other estrogenic activity pathways that may not occur through nuclear ER binding or activation. An often stated deficiency with the in vitro assays is the lack of metabolic capability to assess the potential for in vivo inactivation or activation of the chemical. However, this is for the most part a tolerable deficiency if the question is limited to the potential for nuclear receptor

- med iated activity. It would not be tolerable in cases where only a metabolite of the compound was biologically active.

Question 1 f . Based on all the data presented in Section 2 on ER AUC model performance including characterization of reference chemicals, c oncordance with in vivo uterotrophic results, and comparison with Tier 1 assay endpoints, what are strengths and limitations of us i ng the ER

AUC model to substitute for EDSP Tier 1 ER binding, ER transactivation, or Uterotrophic assays for the purpose of c haracterizing a chemical’s potential estrogen bioactivity?

Strengths

In general

, the ER AUC model will capture many, though not all, of the estrogenic compounds in the “Universe of Estrogen Disrupting Compounds.” Overall both the ER AUC model and the

Tie r 1 in vitro assays capture either nuclear receptor binding and/or transactivation . Thus

, replacement of the Tier 1 in vitro ER endpoints (ER binding and ERTA) with the ER AUC model will likely be a more effective and sensitive measure for the occurrence o f estrogenic activity that occurs through nuclear receptor binding and activation.

Questions r egarding whether the ER AUC model can replace the Tier 1 uterotrophic assay are more complicated. The replacement of this in vivo assay is desirable from both a nimal welfare and cost perspectives. The data comparing the ER AUC model to the uterotrophic assay were strong for the reference compounds that were clearly estrogenic (high AUC values) or29

unmistakably not estrogenic (very low AUC values), although there were a few false negatives and positives (Fig. 2.12). EPA provided appropriate explanations for these false outcomes. Also,

there was some concern expressed in public comments that the compression at the top of the AUC curve may be problematic, the Panel d id not have similar concerns. They noted that the compounds at the top of the curve were largely well known estrogen

- related reference chemicals, and were validated through both the Tier 1 in vitro and in vivo assays. The Panel recognized that

the concorda nce of the model at the high and low end of the AUC spectrum was outstanding.

One of the strengths of the ER AUC model was consideration of potency in comparisons to the uterotropic assay results (Fig. 2.14). If the compounds in the intermediate AUC ranges only experienced active detections at a very high low effect level ( LEL ) of a compound, the potential for the compound to have exposure and biological relevancy may be questioned. As EPA pointed out, this information can then be taken into account w hen trying to determine the model’s potential for use in place of the uterotropic assays. Such an approach was provided to the Panel

for the curated non

- reference chemicals with AUC values below 0.1, but with positive uterotrophic assay results (see Table 2.10 and discussion below), but not for those with positive uterotrophic assay results between 0.1 and 0.4. On the other hand, for the chemicals that provided a negative uterotrophic assay result and AUC model values between 0.1 and 0.4, the data suggeste d that these may be true negatives since the no effect level ( NEL ) was high. Results emphasize that taking potency into account may help fine tune the model

.

Limitations

The P anel also found limitations to the model. Specifically, model outcomes were less straight

forward when the ER AUC model for non

- reference chemicals was compared to uterotrophic studies where the data were limited or discordant (Fig. 2.15 in White Paper). In particular, when the AUC was below about 0.4, and even below the AUC cut off of 0.1, there was a mix of active and inactive results. This suggests that the predictability of the model based on the curation approach was limited when the AUC values were lower than 0.4.

Furthermore, the data for all of the assays utilizing t he List 1 and List 2 battery were below the AUC cut off of 0.1 and were consistently negative for the uterotropic assay. This finding has

suggest ed a very low risk of false negatives in this data set, but was limited by the fact that there

were no chemical s with either high or intermediate AUC model values available for functional comparison. Specifically, while this data set supported the finding that very low AUC values were predictive of no estrogenic response in vivo , they still did not offer insights i nto the functionality of the model for those compounds in the intermediate ER AUC value range. To summarize this concern statistically, the data presented comparing the ER AUC model to the uterotropic assay represents “Spearman Karber

- like” statistics wher e doses were symmetrically active and inactive, but data were limited in between.

A related concern was discussed regarding the ability of ToxCast in vitro methods to assess estrogenic effects that may be impacted by absorption and metabolism or may resu lt through

non

- nuclear estrogen receptor signaling pathways. The lat t er are important physiological

processes that are well understood for a few chemicals, but relatively little is understood about many of the other compounds in the “Universe.” For this re ason, the Panel encouraged the30

iteration of this process by comparing ER AUC with the FSTRA and pubertal assays. Furthermore, evaluating concordance of all three in vivo assays may provide insights into potential nonnuclear receptor estrogenic action, espe cially for those compounds with low AUC values (see Table 2.10 and discussion below) when conducted through the ToxCast suite of

assays . The P anel recognized that EPA stated that it is currently working towards similar ER

AUC model evaluations with the oth er Tier 1 in vivo assays and encouraged this process through comparing ER AUC with the FSTRA and pubertal assays.

For these reasons, the P anel would not recommend that the uterotrophic assay be substituted by

the AUC model at this time. The P anel suggeste d that EPA considers: 1) conducting limited uterotrophic and other Tier 1 in vivo assay testing, using the original Tier 1 Guidelines (and/or through literature curation), for a limited number of chemicals in the low and middle AUC value range from the “Un iverse of Chemicals” shown in Fig. 2.16. Such testing would l i kely add weight

- of

- evidence supporting or refuting the hypothesis that the ToxCast suite of assays is predictive of in vivo estrogenic responsiveness, and 2) the EPA does similar comparisons of the

ER AUC model to the other in vivo assays in the Tier 1 estrogenic battery

.

TOPIC: ANDROGEN BIOACTIVITY

2) EPA’s proposed approach for quantifying a chemical’s potential androgen bioactivity is based on a computational model integrating data from nine high throughput ToxCast assays measuring several endpoints along the androgen receptor (AR) signaling pathway. The computational model outputs are expressed as area under the curve (AUC) scores for AR agonist (R1) and antagonist (R2) bioactivity. Befo re routinely using the AR computational models in the EDSP framework, EPA is reviewing the scientific strengths and limitations of the AR AUC model

described in this White Paper to: i) prioritize chemicals for further EDSP screening and testing

based on es timated bioactivity, and ii) contribute to the weight of evidence evaluation of a

chemical’s potential bioactivity. Please address the following charge questions relevant to

Section 3 of the White Paper and androgen bioactivity:

Question 2 a . How clearly h as EPA described the computational tools in Section 3.1 ( i.e. , high throughput assays and models) used to estimate AR agonist and antagonist bioactivity?

The SAP commended the EPA collaborative team on the work they have performed to provide

the informati on presented in the White Paper describing the early phases of development and evaluation of androgen bioactivity assays relevant to the Tier 1 EDSP (Endocrine Disruptor Screening Program). Importantly, the EPA team provided further explanations during th e SAP public meeting that afforded a better understanding of the underlying methodologies and analytical approaches. Specifically, the adaptation and implementation of a battery of in vitro HT AR (high throughput androgen receptor) bioactivity pathway assa ys are a major advancement in the EDSP and represent a response to concepts advocated by previously convened SAP s

.

The Panel recommended that the Estrogen Receptor bioactivity discussion of the White Paper be

better elucidated similar to the level of cl arity expressed during the oral presentations regarding the AR bioactivity assays and model assumptions

. They advised that this information be included

in a revised version of the White Paper. Testing and model development should be written in the31

appropri ate historic and scientific context within the White Paper by briefly discussing and summarizing prior efforts that are described in the published literature. A major shortcoming

was that t h ere was a limited description of the ToxCast HTS assays in the W hite Paper. Members of the SAP were unable to find complete information on these assays via website/database/literature searches to evaluate the strengths and weaknesses of the assays. Also

, it could not be determined w hether or how these assays were vali dated. Data from eight assays were used to develop the AR AUC model; however, based on the information provided it was difficult to know if these assays were appropriate to replace Tier 1 assays in the EDSP. Other

recommendations included a need to identif y and discuss quality metrics, including but not limited to purity of chemicals, that were tested in the HTS system and to interrogate reproducibility of results by blind replicate testing of selected compounds with high, medium and

low AUC scores. All sco res were reported as point values and the uncertainty of these metrics need to be specified. An example of an AUC50 calculation and related discussion of deliberations involving “weighting” or the lack of it for individual tests should be included. The

co ncept of redundancy and/or lack of it in the test battery should also be discussed. Finally it was suggested that f

actor analysis and/or other computational approaches could be used to determine whether or not it is appropriate to parse the testing tools/a ssays to a limited set of tests.

Understandably, in contrast to the more extensive evaluation and application of ER bioactivity assays, the AR bioactivity assessments remain a work in progress. The EPA team was encouraged by the Panel to build on the ba ttery of AR bioactivity assays to maximize representation of individual steps in the AR signal transduction pathway as illustrated in the White Paper. This should include AR binding and AR transactivation as in the current repertoire

as well as other relev ant and definable steps in the pathway. This implies continuing assessment and refinement of the model with particular attention to the contribution of each component assay to the final assignment of the relative AR bioactivity as determined by the AUC mod el. There was uncertainty among the Panel regarding whether each assay should be assigned an equal value in the AUC model and the calculation of AR bioactivity and what effect, positive or negative, redundant assays (e.g. AR binding assays in multiple spec ies, hAR and cAR) have on the final calculation of AUC and thus on the value for AR bioactivity.

Questions also persisted on how much redundancy in assay design is beneficial and when does redundancy lead to skewing of the data which may result when one step in the AR pathway is

over or under

- represented in the final AUC calculation

. Also

, d o the current assays capture direct actions of AR bioactivity (e.g. AR binding) as opposed to potential indirect effects, such as in the

time

- dependent cell

- based tran scription assays that may measure secondary rather than primary

AR

- dependent gene regulatory events? The relative absence of information in the White Paper to describe the methodologies and proficiency evaluations for each of the AR bioactivity assays led to some misunderstanding and lack of clarity during the SAP meeting (e.g. the radioligand used in the AR binding assays is the synthetic androgenic ligand, methyltrienolone (R1881) rather than the endogenous androgen ligands, testosterone or dihydrotestost erone). Additional information than is currently available in the White Paper regarding validation of the individual AR bioactivity assays should be readily available. The transparency provided by the EDSP21

dash board is critical to the overall success of the program. It will likely promote confidence of stakeholders and the scientific public, while also providing opportunities for other interested32

scientists to offer further input and refinement for the methods, models and approaches employed

in these ana lyses

.

Particular attention should be given to issues related to assay interference and to the factors and

chemicals that contribute to cytotoxicity and cell stress. Careful assessment of the general properties of solvent and test chemical s in in vitro as says should be considered. These factors are critical for the AR bioactivity assays due to the predominance of chemicals, aside from the known pharmaceutical AR agonists, that predominantly express antagonist activity rather than agonist activity.

Wher eas the current focus is on the AR nuclear receptor genomic activity pathway, attention should also be given to the development of alternative AR

- related assays that do not follow the classical genomic/nuclear receptor pathway.

Question 2 b . What are stren gths and limitations of the AR AUC model’s ability to identify reference chemicals that include a variety of structures and have a wide range of in vitro AR

bioactivities?

Strengths

Concerning the strengths, the battery of assays described by the Agency are high th roughput and probe a linear array of AR dependent bioactivities in a framework that integrates different steps in the AR pathway. The approach efficiently utilizes in vitro reconstituted/recombinant protein and cell based assays with a relevant and signifi cant reduction in in vivo animal studies. The use of various cell lines with different inherent properties may allow subtle differences in chemical properties to be ascertained by careful assessment of assay performance characteristics and data analyses. T he test battery includes assays of AR binding that parallel EDSP Tier 1 studies of AR binding conducted with rat prostate cytosol and these assays are complemented by investigation of additional steps within the AR pathway leading to regulation of target g ene transcriptional activity. The assay battery is built on the key aspects of redundancy among different species and the inclusion of different methodologies that employ various protein components and biochemical/molecular detection and quantification too ls. In particular, the assay battery demonstrates an ability to distinguish between agonist and antagonist properties of chemicals and attempts to minimize false positives due to overt cellular chemical cytotoxicity.

Limitations

A potential limitation is that the AUC is derived from a network of assays with some redundancy in assay endpoints and some variation in degree of specificity for the AR

- dependent pathway that may contribute unequally or disproportionately to the mean calculated AUC value. As presented

to the Panel, the AUC value range is narrow and lacks significant magnitude/range for discriminating between AR bioactivity values/scores that assigned to specific chemicals. There was a general lack of clarity regarding how the penalty term is applied to the AUC calculation which may be significant when arriving at a final AUC value for each chemical. The Panel encourages the inclusion of a wider range of chemicals among different structural classes to inform the future studies using these methodologies. The majo rity of chemicals interacting with33

AR have antagonist activity so assays

, and AUC values must be able to distinguish between cell toxicity/cell stress and authentic AR antagonism. Efforts should be made to include in the model, assays that address the meta bolism of chemicals as well as non

– classical AR mechanisms. Further consideration should be given to the ligand of choice for the AR binding assays because methyltrienolone (R1881), although it is a high affinity, non

- metabolizable synthetic AR ligand,

ma y possess subtle differences in its binding properties and the AR structural conformations that it induces as compared to the naturally occurring ligands, testosterone and DHT. Specifically, protein

- protein interactions of the AR with other regulatory prot eins (e.g. coactivators or corepressors) may be sensitive to the reference AR ligand (i.e. R1881, T or DHT). As an example within the current assay battery, interaction of the coactivator, SRC1 with AR is specific to AR bioactivity by agonists, whereas cor epressor interactions with AR are more relevant to chemicals with antagonist activity that are prominent among those examined to date.

Other considerations, to date show the relative absence of non

- monotonic responses detected in the AR pathway analyses. The Pa nel recommend ed that the potential for non

- monotonic responses not be excluded from consideration. The current observations may be limited by the predominance of AR antagonism by the chemicals tested and the coincidence of antagonist activity with higher ch emical doses whereas receptor

- mediated biological effects that are inherently sensitive to lower doses of chemicals may exhibit non

- monotonic properties.

Question 2 c . EPA plans to use data from published in vivo studies that are methodologically

consiste nt with EDSP Tier 1 guidelines to evaluate concordance between AR Page 3 of 3 AUC

model scores of in vitro bioactivity, and the in vivo androgenic and antiandrogenic responses (Section 3.2.1). What are strengths and limitations of the planned curation meth ods and quality standards for evaluating published in vivo studies?

The preliminary data provided by the Agency correlated well with the Hershberger androgen bioactivity assay and thus provides support that in vitro AR bioactivity assays will replicate

pu blished in vivo chemical bioactivities. The curation methods and quality standards were well considered and strong. According to the information provided to the Panel, correlative data will be available on approximately 40 chemicals from articles currently undergoing secondary analyses, and this will provide a significant and relevant but somewhat limited information for

evaluation of in vitro AR bioactivity data. Both the Hershberger in vivo assay and the proposed

in vitro AR bioactivity assays are designe d with parallel objectives to detect either agonist or antagonist activity.

In parallel with the nature and design of many in vivo assays, Hershberger in vivo assays are more complex than single endpoint in vitro assays. The Hershberger assay involves ass essments of multiple androgen dependent tissue weights in either the agonist or antagonist mode. The general expectation is that in vitro AR bioactivity assays will measure consistent agonist or antagonist activity of chemicals whereas the evaluation of pu blished results from Hershberger assays may show degrees of variation in response among the multiple androgen

- responsive target tissues. These variable tissue responses could be related to SARM activity as well as to the diversity of in vivo physiological effects generated by a chemical as observed in the Hershberger assay. The numbers of curated studies involving the Hershberger assay may be low due to the relative scarcity of chemicals with defined agonist or antagonist androgenic activity. Because the34

de sign and utilization of the Hershberger Assay dates back several decades, the congruence

among in vivo study designs for the Hershberger assay may vary significantly due to changes or difference in rat strains, animal care conditions and laboratory environ ments. In vivo test chemical purity, route of administration, dosage, age and strain of rats, may be among the factors that vary significantly between studies and complicate data interpretation for Hershberger assay data published in the literature over th e time course of several decades. Systemic toxicities due to chemical exposures may affect results of Hershberger assays and their interpretation. Pertinent study information about animal body weights, feeding behavior, and general tolerance of the dosing schedule will be essential in evaluating Hershberger assay data to distinguish AR dependency from overt toxicity. Metabolism and in vivo conversion of parent chemical compounds to active metabolites remains a concern with the current battery of in vitro as says. With regard to pesticide metabolites and how AR bioactivity can be affected by metabolic conversion, analyses of vinclozolin and its bioactive metabolites, M1 and M2, are suggested as reference compounds for the ToxCast screen. 1

Question 2 d . Based o n the data presented in Section 3 on AR AUC model’s performance

, what are strengths and limitations of using the AR AUC model to distinguish and prioritize chemicals based on potential androgen bioactivity?

Strengths

Regarding the model strengths, the AR AUC model as presented by the Agency displayed considerable value in preliminary evaluations of AR bioactivity with a number of standard chemicals. By making the data available to the scientific public and providing transparency for the computational mode l(s) used to generate the data, it can be anticipated that additional scientific input will contribute toward the refinement of the model design and its assessments from both within the organization (EPA) as well as by interested external scientists. The A UC model as currently constructed incorporates contributions from a variety of assay designs and endpoint analyses as well as providing filters for cytotoxicity and cell stress. As presented in Section 3, the current data on AR AUC model performance recomm end its potential as a tool for prioritization. The data shown in Fig. 3.5, in particular, may serve as a starting point for further validation of the utility of the AR AUC model approach.

Limitations

In regard to the model’s limitations due

, to the prepo nderance of chemicals with AR antagonist activities, moving forward with the AR AUC model will continue to require careful discrimination between true AR antagonist activity and effects due to genotoxicity and cell stress. The Panel expressed some concern over the distinction and discrimination of potency defined by the AUC model compared to traditional assays of AR bioactivity. Chemicals tested to date primarily include androgen pharmaceuticals that fall into either the very high (potent agonist) or very low (antagonist) value range according to the AR bioactivity AUC model, with only a small range of AUC values assigned to these chemicals. It should be noted that the data provided by the Agency are preliminary and lack adequate representation of diverse c hemical structures. In the opinion of the Panel, the current data set is small and insufficient to adequately validate the approach. Because the current results are themselves preliminary, further35

comparisons and correlations with curated in vivo responses and data will be necessary. The Panel suggested additional in vitro assays, such as HTS for protein structural alterations, be explored to evaluate false positives for AR antagonism that result from chemical effects such as protein denaturation.

TO PIC: INTEGRATED BIOACTIVITY EXPOSURE RANKING (IBER)

3) For Endocrine Disruptor Screening Program (EDSP) chemicals with ToxCast estrogen receptor (ER) and androgen receptor (AR) bioactivity scores (Section 2 and 3), and ExpoCast high throughput toxicokinet ics and exposure estimates (Sections 4 and 5), the IBER approach was used to rank chemicals based on the ratio between the bioactivity dose range, and the expected exposure range (Section 6). The IBER approach extends point estimates of bioactivity,

toxico kinetics, and exposure for a chemical, to distribution ranges based on uncertainty and population variability. Chemical rankings are based on the ratio of the lower range of the bioactive dose, to the upper range of the exposure estimate. Please address th e following charge questions relevant to Section 6 of the White Paper and the IBER approach:

Question 3 a . How clearly has EPA described the computational tools in Section 6 to develop IBER values, including modeling uncertainty and population variability?

The Panel agreed that, at a general level, Section 6 illustrated clearly and thoroughly the Agency’s IBER approach

. In particular, the motivation for and the description of the IBER

metric as a “worst

- case scenario” were well described in the White Paper . Figure 1.3 also provided a good representation of the fact that the IBER approach accounts for uncertainty and variability in both a chemical bioactivity and population exposure.

With regard to Figure 1.3, Panel members indicated that it would be benef icial to incorporate

text in the figure to clearly indicate that IBER values are based on the ratios of the 5

- th percentile of the bioactivity distribution over the 95

- th percentile of the exposure distribution and not on the overlap of the two distributi ons. Although Section 6 clearly stated how the Agency derived IBER values, explicitly representing the ratio within Figure 1.3 may alleviate some confusion in the readership.

While the Panel recognized that the IBER approach relies on the computational a nd statistical

methods described in Sections 2

- 5 for both bioactivity and exposure prediction, the Panel found that as the outputs of those methods are used to construct the IBER ranking, little clarity was provided with respect to which uncertainties and sources of variability are accounted for in the

IBER metric. The clarity of the IBER approach would be enhanced by a dding more detail on which sources of uncertainty are accounted for, which ones are modeled and how, and which

ones are not accounted for an d why not. Table 6.1 attempted to provide a very concise description of the approaches used to handle uncertainty and variability, but the Panel indicated that a more in depth elaboration would be helpful.

The Panel made numerous suggestions to the Agency r elative to the uncertainty described and assessed in the IBER approach. The suggestions were noted as follows

.36

It is clear, from the adoption of a Monte Carlo approach that while individual heterogeneity and population variability are accounted for in th e pharamacokinetics component of the IBER approach, model uncertainty is currently not addressed (as Table 6.1 indicates). A n obvious question is how the Agency intends to account for this. The Panel recommended that the Agency develop strategies to a ccount also for this source of uncertainty.

As Table 6.1 indicated, both model uncertainty and population variability were accounted for in the pharmacodynamics aspect of bioactivity using the “default” method. Although references to Judson (2011) outlin ing the “default” approach were provided, the Panel noted that providing more detail about this “default” approach would be a valuable addition to the Section.

Generally

, the Panel commended the Agency for its efforts in accounting for uncertainty in both the pharmacokinetics and pharmacodynamics aspect of bioactivity ; however

, the Panel encouraged EPA to formulate a joint Bayesian model for pharmacokinetics and pharmacodynamics that also takes into account uncertainty in the AUC values. This joint model

wo uld allow a propagation of the uncertainty from the AUC values, to the pharmacokinetics and pharmacodynamics component of bioactivity providing a more faithful representation of mod el uncertainty in bioactivity. Finally, a Monte

- Carlo approach, as currentl y used and described in Section 5, would account for population variability.

With respect to the exposure prediction component of the IBER approach, the P anel stated that description of the methods utilized to derive exposure are too general and d id not p rovide enough

detailed information. For example, more details on the HTE and SEEM models could be provided to address questions such as: What is the equation that links parent chemical

concentration to measured chemical biomass concentration? W hat prior as sumptions are utilized to link parent chemical concentration to concentration of chemical in plasma? What routes of exposure and degrees of biotransformation are considered and used in the reverse prediction model and why? How are creatinine adjusted value s used in SEEM? Have other approaches such as urinary excretion rate (UER) considered since these data (or data needed to calculate UER)

are now collected in NHANES? Are exposure mixtures considered and/or will they? This information should be provided to enable more transparency on the Agency’s approach.

A lthough Table 6.1 indicated that model uncertainty had been accounted for in the exposure prediction component, the P anel ha d concerns that it was not properly accounted for. In particular, the P anel fou nd Figure 4.4 troubling as it indicates that the same level of uncertainty in exposure is predicted for those chemicals for which information exists through NHANES and

for those for which no NHANES data are available. Conceptually, this result implies that adding NHANES exposure adds no value to the ExpoCast estimates. The P anel suspected that the unexpected results presented in Figure 4.4 were due to the way the predictions from the HTE are used in the SEEM model, with the chemical parent concentration pre dicted by the Bayesian model for HTE probably incorporated in the SEEM only through their posterior summary. A better way to propagate uncertainty from the HTE to the SEEM predictions would be to combine

the two models in a single Bayesian model. This woul d allow the predictions for those chemicals

for which no data are available to have greater uncertainty.37

As Table 6. 1 also indicate d , population variability was not accounted for in the exposure component. However, as discussed during the public meeting, the Agency is planning to address this by using SHEDS

- HT in the next phase. As this is a major and important innovation that

would allow the Agency to account for racial, age and sex differences in chemical excretion (e.g. creatinine), the P anel recommend e d inclusion of this discussion in the W hite P aper.

While the P anel commended the Agency for its effort to take into consideration uncertainty in the IBER approach, it would also like to point out that there are several sources of uncertainties

and variab ility at play: “ parameter estimation uncertainty ” (which can be accounted and communicated via confidence intervals), “model uncertainty” (that is, the uncertainty in the formulations of the statistical models used to make predictions, e.g. the linear mode l in SEEM, the HTE model, the ER AUC model, etc.) and “population variability”. Currently, the IBER ranking approach is accounting, in part, for estimat ion uncertainty and population variability, but not model uncertainty . T he predictions of both bioactivi ty and exposure are derived conditional ly on the adopted model formulation, e.g. the ER AUC model and the HTE/SEEM model. The

Panel encourage d the Agency to clarify that the model uncertainty accounted for in IBER is in reality “parameter/estimate uncertai nty

. ”

Model uncertainty could be addressed in each aspect of the IBER approach ( including both the bioactivity and exposure component) by considering competing statistical models simultaneously and combining their predictions together using approaches suc h as super learner ( Van der Laan

et al. 2007 ) , Bayesian model averaging ( Hoeting et al.

, 1999 ) , or boosting ( Friedman et al.

, 2000 ; Freund

, 2001 ; Freund and Schapire ; 1997) similar to the efforts the EPA is undertaking through Collaborative Estrogen Recept or Activity Prediction Project ( CERAPP )

.

In summary, the Panel recognized that the IBER approach is a scientifically sound approach to provide a single metric to prioritize EDSP chemicals for further study. The Panel also indicated that increased transpa rency is needed regarding how the ranked chemicals will be tested and

whether IBER values will be re

- evaluated as the IBER approach undergoes refinement, e.g. will the IBER be performed on a specified timeline, or will it be performed as new data become

available ?

Finally, the Panel advised the EPA to provide further discussion about the confidence in the performance of the IBER inputs for the extreme cases (e.g., the obviously bioactive chemicals or very low

- exposure chemicals). The Agency suggest ed during the public SAP meeting that these

cases a re not of high interest because there is a high scientific confidence in the priority levels of these chemicals (very high or very low). Given this position, the Panel noted that the practical impacts (for ranking purpo ses) of issues such as the “compression" at the upper end of the forecast distribution of bioactivity ( discussed during the public SAP meeting ) likely become negligible.

Question 3 b. What are strengths and limitations of using the IBER approach to priorit ize chemicals for further EDSP screening based on the ratio between the ER bioactivity dose range, and the expected exposure range?38

The Systematic Empirical Evaluation of Models (SEEM) effort was logically developed and has provided a theoretical framewor k with potential to prioritize further EDSP screening of compounds with estrogenic activity through the IBER approach. The problem the EPA is trying to address is extremely complex, especially given the lack of information on a vast majority of

the chemica ls in the EDSP, as well as the large number of chemicals to be considered in the

“ EDSP Universe.” Furthermore, the EPA’s utilization of uncertainty and variation to identify the lower bioactivity ranges and higher exposure ranges experienced by the general population suggests that the prioritization will be conservative for inclusion of compounds that may have endocrine disrupting capacity. The statistical models and methods used were complex enough to capture many of the potential sources of variability in bioactivity and exposure, yet simple enough to allow for straightforward scientific interpretation, model checking, and most importantly, further development. In practic e
, the applicability and reliability of the IBER approach at identifying high priority chemicals depends on how well and appropriate ly the distribution of bioactivity and exposure are derived. Both components of IBER (bioactivity and exposure level determinations) require the EPA to thoroughly account for all sources of uncertainty in the bio activity and exposure aspect of IBER including model parameter uncertainty, model uncertainty and population variability. The IBER approach accounts for population variability via Monte Carlo simulations and, in the future, through application of the

SHED S model. Therefore, the P anel found that in principal, t he proposed approach to use data and predictive models to produce estimates of both exposure and bioactivity, as well as their respective uncertainties, is sound. In sum mary
, using the IBER ratio to pr ioritize EDSP chemicals likely meets the “fit

- for

- purpose” criteria proposed by the A gency

. However, the

Panel also expressed the following concerns regarding the IBER model that suggest that it will need refinement prior to utilization.

The Panel’s co ncerns regarding the bioactivity interpretations were already addressed in Question 1, but there was consensus that there is a need to have better exposure monitoring data

available to strengthen model outcomes. The P anel was concerned about the limit ed amount of

NHANES

- mined exposure data incorporated into the model. This data set appears to be much less robust in comparison to the data available through ToxCast for bioactivity

.

The Panel understood that the universe of ESDP chemicals is large and t hat there is a need to derive predictions of exposure so that risks can be assessed. However, the P anel was not confident with predictions for chemicals for which biomonitoring data is not available or was limited. Not having any or limited data to validat e the predictions, the HTE/SEEM models appeared to be more an extrapolation exercise given that the large majority of chemicals do not

have measurements available for validation of the model. The m odel ’s uncertainty led to a number of questions by Panel me mbers regarding the functionality of the model : is the relationship in the SEEM prediction model deriving the parent chemical exposure really linear? Is there a threshold effect? Are the indicators in the model the SEEM model clearly defined?

One place w here uncertainty exists in the model that should be perhaps be addressed within the model framework, is the fact that the uncertainty in the SEEM predictions should be different for chemicals for which the HTE/SEEM predictions utilizes some information fro m biomonitoring data or for those which do not. Furthermore, it appears that some effort should be devoted to determine whether biomonitoring data for other chemicals beyond the NHANES data set are39

available in the literature. Additional data would likely improve confidence in the SEEM predictions.

The Panel was also concerned that specific human populations who may have the highest exposure levels for specific compounds were not taken into account. In particular it was recommended that workplace exposure s should be addressed. In the case of pesticides, agricultural workers likely have much higher exposure risks. Numerous studies have shown that formulators, farm workers and their families may be exposed to higher levels of pesticides (for example, McKon e et al., 2007; McKelvey et al., 2013; Johnson et al., 2014; Morgan et al., 2014). As pointed out by the public commenters, worker/applicator exposure is routinely addressed in deliberations involving pesticide registration, and therefore, this group is of particular concern. Curating data, such as those referenced above, could be used to calibrate the model and may be particularly important in identifying the high end of the exposure distribution range. A similar issue came up with specific groups who may be more vulnerable to lower exposure levels. It was recommended that efforts be made to define exposures that are characteristic of pregnant and lactating women and infants. It is widely recognized that these populations may be at highest risk due to expos ure of endocrine active chemicals. One approach may be to create subclasses with

use classifications within the models that delineate chemicals of concern.

The assumptions and parameters used to develop the exposure model had potential problems.

The ass umptions (such as pesticide active or not, for example) provide the potential for poorly estimating e xposure data values. Also, the model may need to incorporate more parameters, and/or weight the parameters in place, to help strengthen the model output. In this case , the triclosan example demonstrated that wrong assumptions in only one parameter can lead to large discrepancies that negatively impact prioritization decisions. Along these lines, within the White Paper, exposure computations involved classif ying chemicals according to use patterns and weighting production volumes on this basis. Pesticides were given a low ranking since most are used in agricultural settings where exposure potential to the general population is limited.

However

, there are many pesticides used in an d around homes, schools, etc. wher e “near

- field” exposures are likely. In this further analysis of study data, the Panel quantified the potential exposures and intake doses of 129 preschool children, ages 20 to 66 months, to 16 pesticid es (eight organochlorines, two organophosphates, three pyrethroids, and three acid herbicides). In conclusion, these children were likely exposed daily to several pesticides from several sources and routes at their homes and daycares.

It was strongly reco mmended at the July of 2014 SAP that the pesticide group, and other groupings as necessary, be further subdivided and weighted by near

- field and far

- field exposure potential. The Panel supported this recommendation. Again, and as noted by the Agency during the public SAP meeting

, such an approach would likely have improved computational results for

triclosan that were described at the Panel meeting.

With regard to the exposure model, the FIFRA SAP that convened in July of 201 4 also felt that the assumpti on of a log

- normal distribution in computations for values less than the detection limit may not be appropriate. Alternate approaches were identified including the possibility that “range

- finding” calculations could be used by assuming all LOD were zero or equivalent to the40

LOD of detection in separate computations. The later approach would provide more confidence

in results.

The Panel believed that the IBER approach in its current formulation did not appropriately account for model uncertainty. The IBER a pproach used the ER AUC model to determine the in

vitro ER activity of a chemical, as well as a particular pharmacodynamics and pharmacokinetics models to estimate exposure data. The exposure model assumed linearity of the toxicokinetics

. This may not be e asily defende d for many compounds. Given that there are both kidney and liver associated parameters involved in the toxicokinetics, the Panel questioned the assumption of linearity in the model. Furthermore, the assumptions of the model in general did not take age and the associated shifts in toxicokinetics that may occur into account. Another limitation of the approach was that the derived IBER quantity is a point estimate, whereas a full distribution or confidence interval may be more informative. One asp ect of uncertainty addressed by the EPA was population variability, and the Panel suggested that 1) model parameter uncertainty, 2) overall model uncertainty and 3) different modeling formulations could be considered in model development.

The Panel foun d it encouraging that the IBER mod el appropriately classified the pharmaceuticals. However, when these results were removed, the classification of other chemicals based on the distance between exposure and toxic equivalents is not as convincing.

Detailed a ssessments of these chemicals within the IBER context need to be conducted to determine if the IBER system can be used in classifications schemes that nay be used to prioritize chemicals for further EDSP testing.

T his model is predicated on human exposure data. In the future, the Panel suggest ed that the lessons learned regarding the application of the process to human health risk be applied to the comparative Tier 1 assays and potentially incorporated with wildlife exposure data.

To summarize, overall t he Panel was appreciative that the EPA recognized that uncertainty and variability should be incorporated into the model for the IBER approach prior to launching the platform for prioritization. The Panel emphasized that these issues need to be further eva luated and especially fine

- tuned by including the uncertainty and variation that may occur to the most vulnerable groups.

One suggestion that the Panel proposed was to include a Bayesian model formulation that combines the pharmacodynamics and pharmacok inetics together. This approach could propagate the uncertainty through the two components and address model parameter uncertainty. Model uncertainty may be addressed by combining predictions from different models together and population variability throug h Monte Carlo sampling. Given that there exists many potential models and no model is perfect, nevertheless, one p anel member suggested another, perhaps statistically more appropriate, approach to SEEM which would be to develop a framework

similar to CERAP P as mentioned in the White Paper . This model combines multiple statistical models together. Some methods that could be used to create this combined model include Bayesian model averaging, super learner, random forest, boosting, etc.). There was some

discu ssion about how to construct the IBER metric. As derived, it is a ratio of extreme points of the bioactivity (lower confidence interval bound) and exposure (upper confidence bound), but41

measures of central tendency were also discussed. The Panel noted that a more encompassing approach is to propagate the uncertainty in estimates of exposure and bioactivity forward to produce probability statements about the entire distribution of the ratio of these two values.

The model as it relates to the EDSP would need the next step s of the validation process as has been done with the AUC model approach including : choos ing certain reference compounds,

evaluat ing them through the ToxCast and ExpoCast, get ting a ratio value, and compar ing the

results to literature and Tie r 1 in vivo output for compounds that are suggestive of ER action in each of the prioritization categories. In theory, the highest priority compounds identified through the IBER model approach should provide the highest estrogenic outcomes from the in vivo

assays.

Question 3 c . What are strengths and limitations of using the IBER approach to prioritize chemicals for further EDSP screening based on the ratio between the AR bioactivity dose range, and the expected exposure range?

The Panel determined th at strengths of the IBER approach to prioritize chemicals for further EDSP screening were largely the same as those identified in 3b for ER active compounds. Theoretically, the model is sound, but it needs refinement followed by validation prior to use

. Regar dless of imperfections in the IBER approach that make the data appear less robust, the purpose of this approach is for prioritization of testing, not for making regulatory decisions. One

Panel member thought as such, it would be unwise to invest a great d eal of money into refining this approach if less than 50% improvement were to be made in the estimates.

The Panel believed the current versions of the models for both androgens and estrogens demonstrate the potential for predictability only for those c ompounds that are very strongly androgenic or estrogenic. Those in the intermediate range of biological activity will need validation through all stages of model development. The Panel found the process of model development occurring appropriately . Thus, a s reevaluations continue to be performed appropriate chemicals will be captured for prioritization as needed.

Specifically, regarding the IBER process for the AR evaluations, the Panel found that the androgenic assays and data for the exposure modeling were not as far along well developed making it more difficult to evaluate whether the approach will work for this set of compounds. In particular, the Panel thought there would be an increased risk of chemicals not becoming prioritized when they should be because of the failure of the exposure model side of the IBER approach.

The Panel indicated that the use of biomonitoring data in the SEEM approach was a strength, however, the Agency should keep in mind the limitations of NHANES data and seek to obtain data from other studies/resources that specifically : (1) target susceptible populations like pregnant women, fetuses, children and the elderly and (2) populations at higher exposure risk like pesticide formulators and agricultural workers. Also, NHANES d oes a good job at capturing baseline exposures (e.g., dietary or continual exposures) but does a poor job at capturing episodic exposures, especially those during critical windows of development. In addition, NHANES sampling logistics prevent the collecti on of data in less dense population areas (some areas where many chemicals are used widely) and do not necessarily capture seasonal exposures (note: these42

concerns were also applicable to the questions regarding the IBER approach for ER bioactivity). Also , the long times spent in the MEC units collecting samples and biometry and anthropometric data may allow for the elimination of short lived chemicals (or create MEC specific exposures) thus underestimating exposures. Thus, inclusion of studies without th ese limitations, although they likely have smaller number of participants were considered imperative to get accurate picture of US chemical exposures.

Other concerns regarding the use of the available exposure data were expressed by the Panel that apply to both the ER and AR models. For example, when using creatinine adjusted values, racial, sex and age differences in creatinine excretion should be considered. A better approach may be to use urinary excretion rate. Also, biotransformation products shoul d be considered. For example, the data showed unremarkable results for several phthalates, but many reports suggest that the monoesters (simple hydrolytic metabolites) are the bioactive form. As the assay is now performed, such bioactivity would not be d emonstrated and may be overlooked in the IBER process/ranking. One Panel member thought this issue to be the biggest weakness in the approach.

Last ly , the Panel

, across all charge questions, emphasized that it will be critical to incorporate

exposure to complex chemical mixtures in the model to ultimately understand the real exposure

risks. C umulative and mixture were not exposures addressed. Since these chemicals are all undergoing the same bioactivity assays, and we know that chemical exposures do not occur in isolation, cumulative exposures to multiple chemicals should be considered in the ranking

process. For example, if BPA and DEHP both have a low IBER ratio, they may not be ranked high in the priority process. But if people are commonly exposed t o both, which they are, then this may increase the likelihood that pharmacologically relevant effects could be seen by these

exposures.43 R eferences

Eskenazi, B , Bradman, A. , Gladstone, E. , Jaramillo, S. , Birch, K. , Holland, N. ( 2003) CHAMACOS, A Longitudinal Birth Cohort Study: Lessons from the Fields Journal of

Children's Health , , Vol. 1, No. 1 : Pages 3

- 27

Freund,Y. (2001). An adaptive version of the boost by majority algorithm. Machine Learning 43,

293

– 318.

Friedman, J. Hastie, T. and Tibshirani R. (2000). Addit ive logistic regression: a statistical view of boosting. The Annals of Statistics 28, 337

- 407.

Giffe T. Johnson, Steve Morris, James D. McCluskey, Ping Xu, Raymond D. Harbison (2014). Pesticide Biomonitoring in Florida Agricu ltural Workers. Occupational Diseases and

Environmental Medicine , , 2, 30

- 38.

Freund, Y. and Schapire, R.E. (1997). A decision

- theor etic generalization of online learning and an application to boosting. Journal of Computer and System Sciences 55, 119

– 139.

Hoeting,J.A. Madigan, D. Raftery,A.E. and Volinsky, C.T. (1999). Bayesian model averaging: a

tutorial. Statistical Science 14, 382

- 417.

Jefferys, W. and Berger, (1992). Ockham 's razor and Bayesian analysis. , `American Scientist', (80), 64

- 72.

Judson, R.S., Houck, K., Martin, M., Knudsen, T., Thomas, R.S., Sipes, N., Shah, I., Wambaugh, J. and Crofton, K. (2014). "In vitro and modelling approaches to risk assessment from the U .S. Environmental Protection Agency ToxCast program." Basic & Clinical Pharmacology & Toxicology 115(1): 69

- 76

Kelce WR, Monosson E, Gamcsik MP, Laws SC, Gray LE Jr. (1994) Environmental hormone disruptors: evidence that vinclozolin developmental toxicity is mediated by antiandrogenic

metabolites. Toxicol

. Appl

. Pharmacol . Jun;126(2):276

- 85. PMID: 8209380

M.J. Van der Laan, E.C. Polley, and A.E. Hubbard (2007). Super learner. Statistical Applications in Genetics and Molecular Biology 6(25), Article 25.

Mckone, T, Castorina, R., Arnly M., Yukuwarbara, Eskenazi, Bradma A, N. (2007). Merging Models and Biomonitoring to Characterize Sources and Pathways of Human Exposure to Organophosphorus Pesticides in the Salinas Valley of California. Environ. Sci. Tech nol ., 41,

3233

- 3240.44

Morgan M., Wilson, Nancy K. and Chuang, J. C. ( 2014). Exposures of 129 Preschool Children to Organochlorines, Organophosphates, Pyrethroids, and Acid Herbicides at Their Homes and Daycares in North Carolina. Int. J. Environ. Res . Public Health , 11, 3743

- 3764.

Reif, D.M., Martin, M.T, Tan, S.W., Houck, K.A., Judson, R.S., Richard, A.M., Knudsen, T.B., Dix, D.J. and Kavlock, R.J. (2010). "Endocrine profiling and prioritization of environmental chemicals using ToxCast data." Enviro nmental Health Perspectives 118 (12): 1714

- 20.

Rotroff, D.M., Dix, D.J., Houck, K.A., Kavlock, R.J., Knudsen, T.B., Martin, M.T., Reif, D.M., Richard, A.M., Sipes, N.S., Abassi, Y.A., Jin, C., Stampfl, M. and Judson, R.S. ( 2013 ) . "Real

Time Growth Kinetics Measuring Hormone Mimicry for ToxCast Chemicals in T

- 47D Human

Ductal Carcinoma Cells." Chemical Research in Toxicology

.

Rotroff, D.M., Wetmore, B.A., Dix, D.J., Ferguson, S.S., Clewell, H.J., Houck, K.A., Lecluyse, E.L., Andersen, M.E., Judson, R.S., Sm ith, C.M., Sochaski, M.A, Kavlock, R.J., Boellmann, F., Martin, M.T., Reif, D.M., Wambaugh, J.F. and Thomas, R.S. (2010). "Incorporating human dosimetry and exposure into high

- throughput in vitro toxicity screening." Toxicological Sciences

117 (2): 348

- 358

Wambaugh, J.F., Wang, A., Dionisio, K.L., Frame, A., Egeghy, P., Judson, R., Setzer, R.W. (2014). "High Throughput Heuristics for Prioritizing Human Exposure to Environmental

Chemicals." Environmental Science & Technology 2014 Oct 24.

Wendy McKelvey,. Br yan Jacobson, Daniel Kass, Dana Boyd Barr, Mark Davis, Antonia M. Calafat, and Kenneth M. Aldous. (2013). Population

- Based Biomonitoring of Exposure to Organophosphate and Pyrethroid Pesticides in New York City. Environmental Health Perspectives • volume 121 | number 11

- 12 | November

- December. 1349

- 1356.

Wetmore, B.A., Wambaugh, J.F., Ferguson, S.S., Li, L., Clewell, H.J., Judson, R.S., Freeman, K., Bao, W., Sochaski, M.A., Chu, T.M., Black, M.B., Healy, E., Allen, B., Andersen, M.E., Wolfinger, R.D. and Thomas, R.S. (2013). "Relative Impact of Incorporating Pharmacokinetics on Predicting In vivo Hazard and Mode of Action from High

- Throughput In vitro Toxicity Assays." Toxicological Sciences 132(2): 327

- 346.

William H. Jefferys, James O. Berger (1992)

. “ Ockham’s razor and Bayesian analysis”,

American Scientis t 80, 64

- 72