November 25, 2014

Dr. Fred Jenkins

Office of Science Coordination and Policy (7201M)

Environmental Protection Agency

US EPA Headquarters

1200 Pennsylvania Ave., NW

Washington, D.C. 20460

Posted to Federal eRulemaking Portal: http://www.regulations.gov, Docket EPA-HQ-OPP-2014-0614

Dear Dr. Jenkins:

The American Chemistry Council (ACC) appreciates the opportunity to provide comments to EPA and 

EPA’s Science Advisory Panel on EPA’s White Paper “ Integrated Bioactivity and Exposure Ranking: A 

Computational Approach for the Prioritization and Screening of Chemicals in the Endocrine Disruptor 

Screening Program”1 (hereafter referred to as EPA’s IBER Profiling White Paper). 

ACC2 has played a constructive role in the Agency’s development and implementation of the Endocrine 

Disruptor Screening Program (EDSP) for many years. ACC has sponsored technical studies to supply 

needed data for endocrine test development and validation, and has participated with the broader 

scientific and stakeholder communities on EPA’s standardization and validation advisory committees. 

As a member of the Endocrine Policy Forum (EPF), we have contributed to the development of 

methodologies for integrating EDSP data across the Tier 1 assays, based on transparent and robust 

weight of evidence (WoE) procedures, to evaluate the potential of a substance to interact with 

components of the estrogen, androgen and thyroid pathways. These work products have been published 

in the scientific literature and communicated to EPA. In addition, we have supported basic and applied 

research to develop: methods for high throughput exposure modeling; in vitro to in vivo extrapolation;

and, (as presented at EPA’s September 2014 ToxCast Data Summit), exposure activity profiling methods 

to evaluate endocrine assay activity in the context of environmentally relevant human exposures. Our 

objective, through the years, has been to constructively contribute to the development and 

implementation of a scientifically sound and transparent EDSP program.

1 Dated 11/05/2014; ID: EPA-HQ-OPP-2014-0614-0003. 2 The American Chemistry Council (ACC) represents the leading companies engaged in the business of chemistry. ACC members apply the

science of chemistry to make innovative products and services that make people's lives better, healthier and safer. ACC is committed to 

improved environmental, health and safety performance through Responsible Care®, common sense advocacy designed to address major 

public policy issues, and health and environmental research and product testing. The business of chemistry is an $812 billion enterprise and a 

key element of the nation's economy. It is the nation’s largest exporter, accounting for twelve percent of all U.S. exports. Chemistry 

companies are among the largest investors in research and development. Safety and security have always been primary concerns of ACC 

members, and they have intensified their efforts, working closely with government agencies to improve security and to defend against any 

threat to the nation’s critical infrastructure.

americanchemistry.com® 700 Second St., NE | Washington, DC | 20002 | (202) 249-7000 

 
November 25, 2014

Page 2

We support EPA’s conceptual approach to employ scientifically valid methods to determine putative 

activity in screening assays measuring endocrine-related endpoints and to combine this with exposure 

information to provide an improved scientific basis for priority setting. The approach presented in EPA’s 

IBER Profiling White Paper represents a significant advance in bringing such methods to the forefront.

Additional work by EPA, however, should be considered to improve the scientific basis of the methods 

used, and to fully document model development and performance. We understand that the Agency 

intends to release model codes in the future, but the lack of such transparency at the present time renders 

a comprehensive independent verification of modeling by outside experts and peer reviewers 

impossible. In the attached report, prepared on behalf of ACC by Dr. Tony Cox, specific comments on 

the strengths and limitations of the area under the curve (AUC) approach for activity profiling are noted, 

and recommendations for improvement are made. We request that the Agency give due attention to Dr. 

Cox’ report as the Agency moves forward to revise the IBER Profiling method for EDSP priority 

setting. 

In July 2014 comments to the Agency, we indicated a need for EPA to include appropriate reference 

substances in such exposure and activity prioritization to provide needed context. We have demonstrated 

such methodology in those comments and in scientific communications. We note that in the EPA IBER 

Profiling White paper, EPA proposes cut off levels for AUC values, but has not included use of activity:

exposure IBER values for benchmarking. Unless EPA adopts suitable benchmarks, such as genistein, for 

comparison for activity and exposure, the Agency will be ignoring critical biologically relevant scientific 

information concerning potency and exposures. This could lead to unnecessary EDSP screening/testing.

We have also participated in the development of the Endocrine Policy Forum’s (EPF) analysis and 

comments on EPA’s IBER Profiling White Paper, and we fully endorse the findings and 

recommendations in those comments. As noted in the EPF comments , the AUC methodology is not 

sufficiently transparent to understand why substances such as potent estrogenic substances (e.g., 

pharmaceuticals, DES) result in calculated AUCs that are only 1.3-fold or so greater than the AUC for 

genistein and diazedin, when in the literature values for relative potency in estrogen assays (see EPF 

comments) indicate there are typically many orders of magnitude difference (e.g. genistein is 10,000- to 

100,000-fold less potent than DES). The scientific rationale for this compression of AUC values is not 

adequately described or discussed in the EPA IBER White Paper. EPA should address this in the 

Agency’s response to the SAP review. 

If you have any questions about these comments or need clarification of any of our conclusions or 

recommendations or if you would like to further discuss our perspectives on exposure:activity profiling 

for the EDSP, please contact me by e-mail at Rick_Becker@americanchemistry.com or by phone at 202-

249-6405.

Sincerely

Richard A. Becker Ph.D., DABT

Science and Research Division

Attachment

americanchemistry.com® 700 Second St., NE | Washington, DC 20002 | (202) 249.7000 
Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303-388-1778; Fax 303-388-0609 

www.cox-associates.com

Comments on “Integrated Bioactivity and Exposure Ranking: A Computational Approach for the 

Prioritization and Screening of Chemicals in the Endocrine Disruptor Screening Program.”

Tony Cox, tcoxdenver@aol.com, 11-24-2014

EXECUTIVE SUMMARY

These comments discuss the approaches to model development taken in the U.S. EPA 

Endocrine Disruptor Screening Program report entitled “Integrated Bioactivity and Exposure 

Ranking: A Computational Approach for the Prioritization and Screening of Chemicals in the 

Endocrine Disruptor Screening Program.” The report offers a highly optimistic assessment of 

the computational methods that it describes – primarily, an “AUC [area-under-curve] model” 

that adds up weighted sums of assay results at different concentrations to get a final score –

leading to the key conclusions that “ER agonist (R1) and antagonist (R2) AUC scores compare 

very well with reported bioactivity of reference chemicals across a range of structures and 

potencies (Section 2.1.1). The agonist (R1) AUC model also accurately predicted in vivo ER 

agonist activity for a relatively large set (~150) of chemicals (Section 2.2.2). Together, these 

analyses provide a high degree of scientific confidence in the ability of ToxCast ER models to 

predict bioactivity and demonstrate the utility of using these computational tools to prioritize 

chemicals in the EDSP Universe.” A precisely similar approach is taken to AR agonist and 

antagonist modeling (Section 3.1). 

The following comments question whether the evidence presented really justifies “a high 

degree of scientific confidence in the ability of ToxCast ER models to predict bioactivity and 

demonstrate the utility of using these computational tools to prioritize chemicals in the EDSP 

Universe.” We note that the computational models and methods are not presented with the 

standard model assessment information needed for an independent expert or peer reviewer to 

critically evaluate model construction and performance, such as results of model diagnostics, 

uncertainty characterizations, and sensitivity analyses. Nor has it actually been shown that the 

models do a “good” job, relative to other approaches, in prioritizing EDSP chemicals. The 

models are simply assumed and used, and their results reported, without any critical comparison 

to alternative approaches. The possibility that much simpler methods would work much better 

1
Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303-388-1778; Fax 303-388-0609 

www.cox-associates.com

cannot be ruled out by the evidence presented. Although it is true (and encouraging) that false 

positive and false negative rates are low when assessed on the specific sets of reference 

chemicals considered, this may reflect the fact that most of the right answers are easy to predict. 

For example, “No activity” will usually be correct for in vivo results unless there is a broad 

pattern of activity in the in vivo assays. Little sophisticated analysis or computational modeling 

are needed to see and exploit this simple pattern. The fact that the AUC models get this much 

right for the reference chemicals (some of which were used both to train and to test the AUC 

models) is no guarantee that they are likely to make accurate predictions for other chemicals, 

particularly those with structures that are outside the training/test sets. EPA should specify the 

domain of applicability and indicate structures of the classes of substances that the models can be 

used for, and the classes for which model use may be limited.

CRITIQUE OF MODELING ASSUMPTIONS

The following key modeling assumptions and methods were used in developing all of the 

AUC models (for ER agonists, ER antagonists, AR agonists, and AR antagonists).

• Hill models. Concentration-response curves can be described by any of three models: “a 

constant model (no activity), a Hill model, and a ‘Gain-Loss’ model, which is a combination 

of two Hill models, one increasing in signal, and the other decreasing at higher 

concentration.”

• Simple linear additive model: “The computational model assumes that the value (the 

efficacy, A) returned by an assay at a given concentration is a linear sum of the contributions 

from the receptors that it measures, i.e., it is a simple linear additive model”

• Curve-fitting: “The goal is then to... minimize the difference between the predicted assay 

values (AAiipprreedd) and the measured ones (AAiimmeeaass) for each chemical and concentration. For 

each chemical-concentration pair, a constrained least-squares minimization approach is 

used...” In addition, a “penalty term helps stabilize the solutions and enforces a reasonable 

physical assumption about chemical promiscuity, i.e., that it is unlikely that most chemicals 

will strongly and specifically interact with many dissimilar molecular targets.”

2
Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303-388-1778; Fax 303-388-0609 

www.cox-associates.com

These methods and models are simply assumed and applied, without derivation or justification. 

The best practices in predictive modeling recognize that selection of a model may profoundly 

influence results, and provide the following information so that others can understand this 

influence and assess the likely predictive performance of the models in new applications. 

• Model diagnostic results: How well do the assumed models describe the data to which they 

are applied? Are there patterns of systematic differences between predicted and observed 

values? Do model specification errors bias the conclusions?

• Sensitivity analysis and model uncertainty results: How do the results change if different 

plausible models are used instead? 

• Model cross-validation. How does the modeling approach perform if it is repeatedly trained 

on some subsets of cases and tested on other, disjoint subsets?

Without such results, it is impossible to determine whether the particular modeling choices made 

are appropriate for the data to which they are applied, or to assess whether they will produce 

credible, unbiased, accurate, or useful results for the chemicals to which they will be applied in 

future, compared to other models and methods. Thus, a key recommendation for improving the 

current report is that the authors should provide model diagnostic results, sensitivity analysis and 

model uncertainty characterization results, and cross-validation results so that the predictive 

performance of the proposed method can be better assessed.

The use of a simple linear additive model (Equation 2.2) is particularly troubling, insofar 

as previous modeling work has established that receptor-mediated responses typically exhibit 

highly nonlinear (e.g., switch-like or threshold) behaviors (e.g., Andersen and Barton, 1999, 

http://toxsci.oxfordjournals.org/content/48/1/38.full.pdf; Zhang et al., 2014, 

http://ehp.niehs.nih.gov/1408244/.) . The linear additive model assumed in Equation (2.2) is a 

very special case of a standard statistical regression model (a main-effects only linear model with 

no error term). This simplification, presented without justification, appears to be a huge step 

backward compared to the use of modern regression models (e.g., generalized additive models, 

generalized linear models, etc.) Including a well-modeled noise term, and checking for 

important interactions and nonlinearities, and then modeling them where needed, are well within 

the current state-of-the-art of statistical practice, and could potentially improve the predictive 

3
Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303-388-1778; Fax 303-388-0609 

www.cox-associates.com

performance, generalizability, and trustworthiness of model-based predictions using relatively 

well-established and readily available statistical prediction methods.

Similarly, the area-under-curve (AUC) obscures difference between concentrationresponse curves that may be important for sensible priority-setting. For example, consider two 

hypothetical chemicals, one with activity only for concentrations above a certain threshold, and 

the other with activity only below that threshold level. A risk manager who can afford to reduce 

exposure concentrations for only one of the chemicals in the near term might strongly prefer to 

reduce the concentration of the first, rather than the second. Yet, there is no reason that the AUC 

for the first must be higher than that for the second. In practice, for the reference chemicals 

studied, AUC works pretty well. But other approaches might work better (e.g., approaches that 

pay attention not only to the AUC, but also to the shape of the concentration-response curve). 

Moreover, for other chemicals, the AUC model may perform poorly or perversely, as in this 

hypothetical example. There is no statistical or mathematical justification in the work presented 

for the belief that the AUC model will have high predictive accuracy in general for biological 

activity, especially for chemicals for which assay results are discordant. Nor is it clear that the 

implicit value judgments incorporated in the AUC metric (e.g., that a concentration-response 

curve with many small values should have the same priority as an equal-AUC curve with a few 

large values) are those that a policy-maker should want to use to set priorities. Discussion of 

such examples and limitations of the AUC metric would be helpful. 

TECHNICAL DIRECTIONS FOR IMPROVING PREDICTIVE POWER AND CONFIDENCE

The current state-of-the-art in predictive modeling suggests the following main technical

directions for improving the current AUC model. 

1. Compare the current results to those from standard machine-learning and statistical 

algorithms (e.g., logistic regression models, GAMS and GLIM models, Random Forest and 

MARS models and algorithms). Use methods such as boosting to combine the results from 

the most successful predictive models to yield final predictions.

2. Use model cross-validation to correct for over-fitting bias and to better assess the true 

predictive (rather than descriptive) performance of models.

4
Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303-388-1778; Fax 303-388-0609 

www.cox-associates.com

3. Evaluate whether a small subset of ToxCast assays can predict at least as well as the full set 

of 18 assays used in the current ER AUC model. In particular, it seems very plausible that 

many of these assays are redundant. Modern methods of feature selection (such as mutual 

information and conditional entropy reduction methods) might well reveal that a small subset 

of assays can achieve equal or superior predictive performance compared to the currently 

proposed 18 assays and the AUC models.

4. Allow for missing data. Computational statistical methods and algorithms (ranging from 

data-augmentation, multiple imputation, and expectation-maximization algorithms to the use 

of “surrogate splits” in classification and regression trees) make it practical to perform highquality predictions in many settings similar to this one, even when some chemicals have 

missing values for one or more assays.

These technical directions are very likely to be able to build on the promising start made in the 

current report to provide more robust (e.g., missing data-tolerant), defensible, better-validated, 

and more accurate and trustworthy predictions of in vivo ER and AR biological activities based 

on the results of HTS assays. 

The preceding points and recommendations are developed further in the detailed 

comments that follow.

5
Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303-388-1778; Fax 303-388-0609 

www.cox-associates.com

DETAILED COMMENTS

The following comments take a deliberately, systematically skeptical view of the claims 

advanced in Sections 2 and 3.1 of the recent report entitled “Integrated Bioactivity and Exposure 

Ranking: A Computational Approach for the Prioritization and Screening of Chemicals in the 

Endocrine Disruptor Screening Program.” A few comments are also made on Section 1. The 

purpose of these comments is not to attack or undermine this admittedly very interesting and 

promising work, but to help improve its exposition and approach by challenging its assertions, 

questioning its methods, and suggesting less optimistic interpretations for its key findings and 

conclusions. This is intended as a “Devil’s Advocate” piece to help make the work stronger. 

“The U.S. Environmental Protection Agency (EPA) is continuing a series of scientific peer 

reviews focused on computational approaches for prioritization and screening of chemicals 

in the Endocrine Disruptor Screening Program (EDSP).”

COMMENT: It might be useful to explain the decision problem(s) to which “prioritization and 

screening of chemicals in the Endocrine Disruptor Screening Program (EDSP)” is intended to be 

a solution. For example, to support risk management decisions about which chemicals are most 

worth focusing on first to reduce undesirable impacts, it might be necessary to consider not just 

how large risks are, or how likely different chemicals are to have endocrine disruptor activity, 

but rather to consider by how much the risks from different chemicals could be reduced by 

allocating available resources. For example, if regulating chemical A would reduce risk of an 

adverse outcome from exposure to A from 0.8 to 0, and if regulating chemical B would reduce 

risk from exposure to B from 0.95 to 0.94 then a risk manager who could only afford one of 

these as top priority might well prefer to regulate chemical A instead of chemical B, even though 

B has a larger risk of adverse outcome. To support effective risk management, therefore, 

“prioritization and screening of chemicals” would have to model potential changes in risk from 

alternative resource allocation prioritization and screening rules, so that the most effective 

alternative could be selected. Explicitly stating the purpose of the prioritization and screening 

effort (e.g., to help inform allocation of risk management resources to reduce risks) could help 

make clear whether the current design of the EDSP supports the stated purpose, or whether 

6
Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303-388-1778; Fax 303-388-0609 

www.cox-associates.com

different information and considerations are required, such as estimation of the change in risk 

that would be produced by an affordable change in exposures.

“This document describes innovative approaches for determining estrogen and androgen 

bioactivity based on a series of computational models integrating data from ToxCast high 

throughput screening (HTS) assays.”

COMMENT: The models in this document are not capable of “determining estrogen and 

androgen bioactivity,” but only of informing guesses about whether such activity will be present

in specific systems. Thus, this sentence seems to promise more than is delivered in asserting that 

it describes “approaches for determining estrogen and androgen bioactivity.”

“This document describes a series of generalizable chemical prioritization and screening 

methodologies based on endocrine bioactivity and exposure which could be applied and 

adapted to address other endocrine pathways and EDSP chemicals. The focus for review by 

the Federal Insecticide, Fungicide, and Rodenticide Act (FIFRA) Scientific Advisory Panel 

(SAP) is on the concepts and methodologies, and not on results for specific chemicals that 

may be prioritized for EDSP screening. It is the intention of the EPA to complete the 

prioritization methodology and process after receiving recommendations from the SAP on 

the proposed computational approaches.”

COMMENT: In general, prioritization is not an effective approach for allocating limited risk 

management resources. For example, suppose that an agency must allocate a limited budget to 

reduce risks from endocrine disruptors. Three opportunities have been identified to reduce risks, 

as follows:

• Regulating chemical A reduces risk from exposure to A from 100 to 80 (on some 

appropriate scale). It costs $30. 

• Regulating chemical B reduces risk from 50 to 10. It costs $40.

• Regulating chemical C reduces risk from 25 to 0. It costs $20.

7
Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303-388-1778; Fax 303-388-0609 

www.cox-associates.com

The goal for resource allocation is to achieve the largest possible total risk reduction for the 

available budget. What priority ranking of A, B, and C achieves this goal, if resources are 

allocated from the top of the priority list down until they are exhausted? The answer is that no 

priority ranking achieves this goal. Instead, the correct answer depends on the budget. For a 

budget of $49, the largest feasible risk reduction is achieved by regulating B, so the best priority 

order puts B first. If the budget is $50, then regulating A and C achieves the greatest risk 

reduction, so B should be ranked last. At $60, the best investment is to fund regulation of B and 

C, so now A should be ranked last. Thus, no priority-ordering of A, B, and C optimizes resource 

allocations independent of the budget. For example, no priority order of A, B, and C is optimal 

for budgets of both $49 and $50. The difficulty illustrated here is that optimal resourceconstrained investments in reducing risks, or in Tier 1 screening, or on other desired activities,

cannot in general be represented by priority-rankings. Since, in general, no prioritization of 

chemicals can guide optimal allocation of resources, it would be helpful for the report to explain 

in greater detail exactly what the prioritization is intended to accomplish, and to consider 

whether different approaches (such as mathematical optimization) might better accomplish what 

is intended.

“Figure 1.3: A graphical representation of the Integrated Bioactivity-Exposure Ranking 

(IBER) approach indicating no priority chemicals (those that either have no potential 

bioactivity or no potential exposure), low (chemicals for which bioactive concentrations are 

considerably higher than potential exposures), medium, and high priority (chemicals for 

which bioactivity and exposure concentrations are close or overlapping).”

COMMENT: The y axis in Figure 1.3 has units of mg/kg/day. However, the bioactivity of a 

chemical may depend on how exposures are distributed over time (e.g., one large exposure every 

7 days vs. 1/7 of that exposure on each of 7 consecutive days). This information is lost, for both 

exposure and bioactivity, in the conceptual model shown in Figure 1.3. 

“In this section, we describe an approach to combine data from multiple in vitro assays 

measuring estrogen receptor activity to make a weight of evidence prediction concerning 

“true” activity.” 

8
Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303-388-1778; Fax 303-388-0609 

www.cox-associates.com

COMMENT: The term “weight of evidence prediction” should be precisely defined. In general, 

predictions made by sound principles (e.g., by conditioning on all available evidence using 

Bayes’ Rule) cannot necessarily be represented by combining items of evidence (e.g., values of 

predictors) using any system of weights or additive scoring models. The term “weight of 

evidence” is often used to describe processes that seek to use professional judgments and expert 

opinions to make guesses or to reach consensus conclusions based on ambiguous data. Such 

conclusions are often mistaken (see e.g., Kahneman, Thinking, Fast and Slow, for a discussion of 

expert judgments). Is that what is intended here? 

“Therefore, a key aspect of the analysis given below is the use of a reasonably large set of 

well-studied reference chemicals.”

COMMENT: The report should be much more precise about technical terms, and not rely on 

vague or undefined terms (such as “reasonably large set”) for key aspects of the analysis. The 

term “reasonably large set” should be clearly defined. The statistical literature on design of 

experiments and response surface evaluation shows that even a very large set of well-studied 

reference chemicals will not provide the information needed to make accurate predictions for 

other chemicals outside that set. Instead, the set of reference chemicals must also be sufficiently 

diverse, in a sense that can be made precise using formal definitions and techniques for design of 

experiments. Use of such precise definitions and statistical design criteria (such as D-optimality)

may be useful in explaining what criteria the authors intend to advocate for determining why and 

whether the set of reference chemicals used is sufficient to permit confident predictions for 

chemicals not in the set. 

“By combining data from multiple assays, and using a model that detects these technologyspecific false positives, we can arrive at a more confident assessment of the in vitro 

estrogenicity of a chemical.”

COMMENT: What, precisely, is being asserted here? Does “we can arrive at a more confident 

assessment” mean something different from “we will feel better about our assessment of the in 

9
Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303-388-1778; Fax 303-388-0609 

www.cox-associates.com

vitro estrogenicity of a chemical”? If so, what exactly is the claim that is being made? Is it that 

confidence intervals will be tighter, or that probabilities will be closer to 0 and 1, or something 

else?

“Using a suite of assays to detect pathway perturbations overcomes limitations of a single 

assay.” 

COMMENT: Could using a suite of assays introduce further limitations (e.g., additional 

opportunities for false-positive results, additional noise and error rates in the conclusions) that 

using only one or a few best assays would avoid or reduce? Is the intended claim here that using 

a single assay can never give superior performance to using a suite of assays? EPA implies that 

no single assay drove the AUC score, which seems to imply that no assay was most predictive 

for the reference set. But EPA has not shown the results to indicate whether certain subsets 

together were more predictive than the selected suite. Again, it would be helpful to readers if the 

claim being made were stated very clearly, and any conditions required to prove or justify it were 

stated explicitly.

“To navigate this complexity, we developed mathematical and statistical models to infer 

whether chemicals that activate specific patterns of the in vitro assays were more likely to 

be ER agonists or antagonists, or were more likely to be causing assay activity through 

specific types of assay interference.”

COMMENT: The mathematical and statistical models used deal only with associations and 

patterns, and do not provide any formal causal analysis or modeling (e.g., using DAG models, 

SEM and path models, Granger causality tests of activation patterns over time, etc. based on the 

conceptual networks provided.) They thus do not appear to be appropriate models for 

quantifying or comparing likelihoods of alternative causal explanations, as this sentence 

suggests. Additional discussion of how likelihoods of different causal explanations for observed 

patterns (e.g., ER agonist, antagonist, or assay interference) were (or should be) assessed and 

validated would therefore be useful. Does the report mean to imply that sound causal 

10
Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303-388-1778; Fax 303-388-0609 

www.cox-associates.com

conclusions can be drawn using the mathematical and statistical models developed, without need 

for formal causal analysis methods?

“Overall approach: The input data for the model includes chemical structures and 

concentration-response data for 18 ER-related in vitro assays, plus data for many, non-ER 

in vitro assay endpoints (ranging from 186-821 assays, depending on the chemical).”

COMMENT: How sensitive are the results to the selection of these 18 ER-related in vitro

assays? Would other, equally valid, selections produce different results? If so, then how should 

performance results be attributed to choice of inputs vs. to the methods used? 

COMMENT: How do these choices of input data affect performance quality (e.g., false 

positives, false negatives, and balanced accuracy (BA)? Might different selections of inputs lead 

to improved performance? For example, Cox et al. (2014) 

(www.sciencedirect.com/science/article/pii/S0273230014000920) report the results of analyses 

suggesting that the selection of these 18 assays needlessly reduces performance: “The main 

results of applying the Random Forest algorithm to the full set of HTS (ToxCastTM assay) 

indicators are presented in Table 3. Significant improvements in fit are noted, with BA scores of 

1 for classes of guideline-E and -A studies (Table 3), compared to 0.94 and 0.92 (Table 2) 

achieved using 18 endocrine assays for prediction model development.... Thus, looking at BA, 

fully automated methods gave significantly better BA results than manual selection for both 

guideline and non-guideline studies, and for all classes.”

COMMENT: How well can subsets of these 18 assays be used to make predictions, e.g., in 

cases of missing data? 

“The model produces scores for agonist activity, antagonist activity and for ‘false’ activity 

in each assay technology. These scores can then be compared with reference chemicals, 

structure classes, and other properties.”

11
Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303-388-1778; Fax 303-388-0609 

www.cox-associates.com

COMMENT: It is unclear what is being suggested here. What does it mean to say that “scores 

can then be compared with reference chemicals?” (A score is a number; a chemical is not. What 

would a comparison between them consist of, and what could it reveal that would be useful?) 

COMMENT: Scores can be generated in an infinite number of possible ways. What formal 

criteria have been used, or should be used, to decide whether the scores produced by these 

models are useful for making predictions, for supporting decisions, or for other purposes? How 

are the scores intended to be used, and what guarantees, if any, can be provided that they are 

useful for the purposes for which they are intended? 

“The 1812 chemicals used in this model were selected because they had data for the full set 

of 18 ER assays.”

COMMENT: Is there an assumption that chemicals with complete data for the 18 ER assays are 

representative of the wider universe of chemicals? If yes, how is this justified? If no, then is 

there any guarantee that results can be generalized?

“All of the concentration-response data were analyzed using the ToxCast data analysis 

pipeline, which automates the processes of baseline correction, normalization, curve-fitting, 

hit-calling and detection of a variety of potential confounders, which are annotated as 

“caution flags”. This pipeline, along with all of the raw and processed data, and 

annotations is publicly available [http://epa.gov/ncct/toxcast/data.html].”

COMMENT: As noted by Cox et al. (2014), “Full details of how the dose–response data were 

processed were not given in the Supplementary Materials, and despite several helpful discussions 

with the authors of the HTS-derived EDSP T1S endocrine prediction models, we could not 

replicate them. ...None of the published prediction model algorithms were transparent, nor were 

the algorithms clearly defined and documented. Input from the first author was helpful, but 

significant efforts were still needed to reverse engineer the prediction models. For the most part, 

the underlying data sets were available to support this reverse engineering activity, however, the 

12
Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303-388-1778; Fax 303-388-0609 

www.cox-associates.com

precise details of how dose–response data were re-coded as binary and response indicators (using 

an unspecified Hill function with some manual adjustment) were not documented or readily 

available in the Supplementary Materials.” 

“The concentration-response data for each in vitro assay were fit to 3 models: a constant 

model (no activity), a Hill model, and a “Gain-Loss” model, which is a combination of two 

Hill models, one increasing in signal, and the other decreasing at higher concentration.”

COMMENT: The selection of a model may profoundly influence results. To reveal this 

influence and to allow others to assess the quality of a model, the following information should 

be provided.

• Model diagnostic results: How well do the assumed models describe the data to which they 

are applied? Are there patterns of systematic differences between predicted and observed 

values? Do model specification errors bias the conclusions? How confident can we be in the 

answers to these questions?

• Sensitivity analysis results: How do the results change if different plausible parametric 

models are used instead? For example, as previously noted, Cox et al. (2014) find that a

different, entirely automated procedure using standard R routines improves classification of 

chemicals (to BA = 100% for E class chemicals). How would the results change if a nonparametric curve were fit to each concentration-response data set?

• Model uncertainty results: If statistical techniques such as Bayesian Model Averaging were 

used to account for model uncertainty, then would the same predictors still be retained? How 

sensitive are results to over-fitting using Hill and Gain-Loss models, e.g., as assessed by kfold model cross-validation?

• Results from model cross-validation, out-of-sample performance, and correction for overfitting. How would the use of k-fold cross-validation to correct for model over-fitting change 

reported results?

Without such results, it is impossible to determine whether the particular modeling choices made 

(e.g., selection of Hill and Gain-Loss models) are appropriate for the data to which they are 

13
Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303-388-1778; Fax 303-388-0609 

www.cox-associates.com

applied, or to assess whether they produce credible, unbiased, accurate, or useful results

compared to other methods and models . 

“The following scheme is used to filter out these non-selective, cell-stress or cytotoxicityrelated assay hits. For chemicals with two or more positive responses in cytotoxicity assays, 

we calculate the mean logAC50 (cytotoxicity) and the median absolute deviation (MAD) of 

the logAC50(cytotoxicity) hits. Next, we calculate the median of the MAD of the 

logAC50(cytotoxicity) distributions across all chemicals to define the global cytotoxicity 

MAD. A new value (the Z-score) is then assigned to each assay hit... (Equation 2.1). If 

fewer than 2 cytotoxicity assays are hit, the median cytotoxicity concentration is arbitrarily 

set to 1000 μM, which simply sets all Z-values for assay hits to a value > 3. A hit with a 

large value of Z occurs at concentrations significantly below where cytotoxicity is 

occurring. This hit is both unlikely to be caused by cell-stress or cytotoxicity-related 

processes, and is more likely to be able to cause toxicity through a target-selective

mechanism.”

COMMENT: What other procedures were considered, if any, and how do results from this 

procedure compare to the results from other, equally well justified, alternative procedures? 

COMMENT: Can the following assertions be proved? “A hit with a large value of Z occurs at 

concentrations significantly below where cytotoxicity is occurring. This hit is both unlikely to be 

caused by cell-stress or cytotoxicity-related processes, and is more likely to be able to cause 

toxicity through a target-selective mechanism.” Evaluating cytotoxicity and correcting for it in

predictive modeling is indeed desirable, but the available data and information presented are not 

sufficient to independently verify these statements. The Z-score method deserves its own section 

and appendix of assay data.

“Reference chemicals: A set of 45 positive and negative reference chemicals were used to 

evaluate the performance of the model.”

14
Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303-388-1778; Fax 303-388-0609 

www.cox-associates.com

COMMENT: Is there any guarantee that performance results on this set of 45 reference 

chemicals can be extrapolated to performance on other chemicals?

COMMENT: How would the evaluated performance of the model change if different sets of 

reference chemicals were used? For example, if many random subsets of size 30 were used to 

evaluate the model, then what would be the distribution of its performance statistics?

“Structure of the Network: Figure 2.1 is a graphical representation of the network used to 

evaluate the integrated in vitro assay responses.... The model assumes that a chemical that 

interacts with the ER will bind in one or both of the agonist or antagonist conformations 

and that this will trigger activity in the appropriate pathway. ... The details of the process 

connecting the pseudo-receptors to the assays were simplified to a single connection 

because, in general, we do not know the intermediate details of these processes or even the 

identity of the pseudo-receptors. Note that the pseudo-receptors may not be what a 

biologist would define as a receptor, but are surrogates for generic processes such as cellstress or cytotoxicity that can lead to non-ER mediated assay activity. It is possible to 

describe many possible alternative pathways, but in general, the data are not sufficient to 

distinguish between alternate models. Pseudo-receptors are assigned to each group of 

assays (technology group) and to each assay individually.”

COMMENT: From the purely pragmatic standpoint of creating and validating useful predictive 

outputs, the whole discussion of pathways, “pseudo-receptors,” networks, etc. seems 

unnecessary: what was actually done is summarized in the statement “The computational model 

assumes that the value (the efficacy, A) returned by an assay at a given concentration is a linear 

sum of the contributions from the receptors that it measures, i.e., it is a simple linear additive 

model.” 

COMMENT: As noted in Cox et al. (2014), using a fully automated technique with no attempt 

to impose constraints or assumptions about underlying biological networks gives improved 

(perfect) classification performance for E chemicals. 

15
Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303-388-1778; Fax 303-388-0609 

www.cox-associates.com

“Mathematical Model: The computational model assumes that the value (the efficacy, A) 

returned by an assay at a given concentration is a linear sum of the contributions from the 

receptors that it measures, i.e., it is a simple linear additive model: 

AAii=ΣFFiijjRRjjjj Equation (2.2) 

where the elements of the F matrix are 1 if there is a connection between a receptor and an 

assay, and zero otherwise.”

COMMENT: Systems biology models and previous studies show that receptor-mediated 

responses for endocrine disruptors typically exhibit switch-like and threshold behaviors (e.g., 

Andersen and Barton, 1999, http://toxsci.oxfordjournals.org/content/48/1/38.full.pdf; Zhang et 

al., 2014, http://ehp.niehs.nih.gov/1408244/. No justification is given for a simple linear 

additive model, as in Equation (2.2), as being realistic.

COMMENT: All standard linear and nonlinear regression models include error terms, yet there 

is no error term in the model in Equation (2.2), and no discussion of its properties (e.g.,

homoscedasticity, residuals, need for transformations to stabilize error variance, etc.) The use of 

what amounts to a main-effects only linear model with no error term appears to be a huge step 

backward compared to the use of modern regression models (e.g., GAMS, GLIM, etc.) This 

topic is mentioned on page 31, as follows: “All assays have a background level of noise, and one 

must define a threshold to decide between a hit and a non-hit.” Yet, the noise is not included in 

the model in Equation (2.2) and is not explicitly modeled. It should be.

COMMENT: To justify use of Equation (2.2), the authors should provide evidence that:

(a) Model diagnostics indicate that the model is appropriate for the data

(b) Analyses using techniques such as CHAID, CART, or MARS indicate that there are no 

important interactions (or, if there are, the model should be revised to include them)

(c) Analyses using techniques such as polynomial regression, GAMS, or smoothing splines 

indicate no significant departure from linearity and no need to model nonlinearities

No such results have been provided, leaving unclear the rationale for Equation (2.2) and its 

acceptability as a useful model (preferable to, e.g., standard nonlinear regression models).

16
Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303-388-1778; Fax 303-388-0609 

www.cox-associates.com

“The goal is then to find a set of RRjj values that minimize the difference between the 

predicted assay values (AAiipprreedd) and the measured ones (AAiimmeeaass) for each chemical and 

concentration. For each chemical-concentration pair, a constrained least-squares 

minimization approach is used...”

COMMENT: Why is this “the goal”? (The goal for achieving what purpose?) This is a curvefitting goal. As stated, it encourages over-fitting (i.e., choosing Rj values to optimize fit to the 

training data, rather than to optimize predictive accuracy outside the training set). A better goal, 

for purposes of predictive modeling, would be to select Rj values to minimize k-fold crossvalidation error.

COMMENT: As shown in the following example, it is possible to have a completely accurate fit 

between observed and measured values in a training set, and yet to have no predictive power 

outside the training set. Suppose that cases 1-6 are “reference chemicals” available for training, 

with 4 predictors (analogous to the 18 ER assays) and one predicted outcome (the right-most) 

column. The predictive challenge is to predict the outcome for case 7 (the value, 0 or 1, in the 

“?” cell in the lower right of the table.) The following two models each describe the training 

data in rows 1-6 perfectly, yet they make opposite prediction for case 7:

Model 1: Predicted value = 1 if sum of predictors 2, 3, and 4 > 1, else 0

Model 2: Predicted value = value of Predictor 3.

Case # Predictor 1 Predictor 2 Predictor 3 Predictor 4 Predicted

1 1 1 1 1 1

2 0 0 0 0 0

3 0 1 1 0 1

4 1 1 0 0 0

5 0 0 0 1 0

6 1 0 1 1 1

7 1 1 0 1 ?

17
Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303-388-1778; Fax 303-388-0609 

www.cox-associates.com

Likewise, they would make opposite predictions for a chemical with predictor values of (0, 0, 1, 

1, 0). This illustrates that successful classification or description of reference chemicals with 

respect to biological activities is a different task from successful prediction of the biological 

activities of chemicals outside the training set. As in these examples, it is possible for a 

computational procedure to have up to 100% balanced accuracy on the former task, while 

making predictions with no better than a random (50-50) probability of being correct for the 

latter task. Seeking “RRjj values that minimize the difference between the predicted assay values 

(AAiipprreedd) and the measured ones (AAiimmeeaass) for each chemical and concentration” addresses the 

former task, but not the latter. Yet, it is the latter that should be the goal of chief interest to 

practitioners who want to apply the model to chemicals other than those used in building the 

model.

“This penalty term helps stabilize the solutions and enforces a reasonable physical 

assumption about chemical promiscuity, i.e., that it is unlikely that most chemicals will 

strongly and specifically interact with many dissimilar molecular targets. Values used here 

are αα = 0.05 and SSRR0 = 0.5.”

COMMENT: This penalty term seems ad hoc, insofar as no derivation or justification is given 

for this specific mathematical formula. What alternative penalty terms were considered, and 

how sensitive are results to different, equally plausible, choices of the penalty term? 

COMMENT: How sensitive are the results to the selected values αα = 0.05 and SSRR0 = 0.5?

COMMENT: Is it necessarily the case that different receptors (or other molecular targets) are so 

dissimilar as to make it unlikely that chemicals will strongly and specifically interact with two or 

more of them? What is the specific evidence supporting this assumption?

“The model results in a response value (between 0 and 1) for each receptor at each 

concentration. These results are summarized as area under the curve (AUC), which is the 

integral across the concentration range... ”

18
Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303-388-1778; Fax 303-388-0609 

www.cox-associates.com

COMMENT: The use of the AUC obscures potentially important differences in concentrationresponse curves. For example, it appears that Equation 2.6 can assign identical values to three 

concentration-response curves that are flat, increasing (perhaps with a threshold), and decreasing 

in concentration, respectively, and yet these three curves might have very different implications 

for risk assessment and risk management priorities.

COMMENT: There is no explanation of how sign(slope) factor is computed, or why it is 

considered an appropriate way to “discount” AUCs for assays that are activated at relatively low 

concentrations.

COMMENT: No formal justification for the use of AUC has been provided. Does it provably 

optimize some criterion, or support good decisions, or have any other desirable properties that 

make it superior to other measures? If there is some explicit justification for using the AUC, it 

should be described. The current assertions that “The values of AUC can be related to the 

potency (AC50), and are useful in interpreting different magnitudes” are not given any rigorous 

explanation or support. The text correctly notes that “One issue that arises with this approach is 

how to interpret these receptor AUC values” and explains that “Qualitatively, we interpret these 

R values as concentration-specific probabilities that the chemical is interacting with the 

corresponding (pseudo) receptor.” But this means that the R values (and the AUC derived from 

them) are interpreted in terms of probabilities of interactions with fictitious constructs – hardly a 

clear or reassuring basis for use. Moreover, no justification is provided for such a probabilistic 

interpretation. It is not specified what information these “probabilities that the chemical is 

interacting with the corresponding (pseudo) receptor” are conditioned on, or how they would or 

should change as evidence from similar chemicals is considered. In short, it is not clear why or 

whether the AUC is a useful measure. The hand-waving discussion provided raises more 

questions than it answers. For example, how do we know that the R values are not the squares or 

square roots of the “concentration-specific probabilities that the chemical is interacting with the 

corresponding (pseudo) receptor”? Or, if indeed “we interpret these R values as concentrationspecific probabilities that the chemical is interacting with the corresponding (pseudo) receptor,” 

and if arbitrarily weak interactions are allowed, then why are not all the R values equal to 1, 

19
Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303-388-1778; Fax 303-388-0609 

www.cox-associates.com

assuming that at least some (perhaps undetectably weak or extremely rare) interaction is always 

possible? In short, the suggested interpretation in terms of probabilities has not been justified, 

and does not appear to be obviously plausible or correct.

“Note that BPA has activity in one of the antagonist assays (A17, gold), consistent with 

Selective Estrogen Response Modulator (SERM) activity at high concentrations. However, 

the results of the network model do not reflect this weak antagonist behavior.”

COMMENT: The report should discuss further the implications of such discrepancies between 

model predictions and assay results. Do they signal that a different model should be used? 

Would using a nonlinear model produce more accurate results? 

“Model Results for Reference Chemicals: The AUC values for the ER reference chemicals 

are shown in Figure 2.4. For the positive agonist chemicals, all but two have non-zero AUC 

values, these being diethylhexyl phthalate (DEHP) and dicofol.”

COMMENT: It should be noted that these “model results” actually refer to the ability of the 

model to fit the reference chemical data used to construct it. Thus, they are closer to model fits –

answering whether the model can at least reproduce the data used to build it – than to what is

often meant by “results,” i.e., results of applying the model to new cases, which might be used to 

assess its predictive validity rather than its descriptive validity, as in Section 2.2.2.

“We explore several potential causes of pseudo-receptor activity. The first potential cause 

has to do with assay noise. All assays have a background level of noise, and one must define 

a threshold to decide between a hit and a non-hit. To be called a hit in the data analysis 

pipeline, the maximum efficacy must be above a defined statistical criteria, which is 

typically 10 MAD above baseline. Given this relatively high threshold (10 MAD, essentially 

10 standard deviations), one would expect few noise-related false positives if the noise was 

approximately normally distributed. In particular, for full agonists in an assay with only 

normally distributed noise, one should see T values of ~100% for the actives, and T values 

for inactive chemicals should be small (<<10 MAD). We compared the T distributions 

20
Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303-388-1778; Fax 303-388-0609 

www.cox-associates.com

between the strong agonists and the chemicals in each of the pseudo-receptor groups using 

a t-test. Table 2.3 lists the mean T for chemicals active in the pseudo-receptors and in the 

agonists, and a t-test p-value. (Note that for the antagonist assays and R9, the comparator 

group was the antagonist chemicals, AUC(R2) ≥0.1. We see that the actives (AUC≥0.1) in 

R6, A9, A12, A13, A15 and A17 have significantly lower T values than the comparator 

groups, but are still above the statistical threshold, implying that at least some of this 

pseudo-receptor activity is associated with non-normally-distributed assay noise.”

COMMENT: As previously noted, assay noise should be explicitly included in the modeling 

(instead of the noiseless Equation (2.2)) so that standard regression techniques such as 

transformations to stabilize variance can be applied if needed. It is not necessarily plausible that 

noise is normally distributed (since negative values are not allowed, homoscedasticity seems 

implausible for concentration-sensitive responses, and phenomena such as zero-inflation or finite 

mixture distributions in concentration-response relations may require discrete probabilities to 

model accurately). 

COMMENT: The conditions for use of the t-test have not been verified.

“There are 11 compounds that we classify as negative for which there are multiple 

literature reports of being positive binders. ... ER Model Summary: This section has 

demonstrated a method to combine data from multiple in vitro assays in the ER pathway in 

order to generate a consensus call of ER activity. The need for this approach is driven by 

the presence of false positive and false negative calls in all assays. An ongoing topic of 

research is trying to use chemical structure and properties to understand exactly why 

specific chemicals cause this (mainly) false positive activity.”

COMMENT: Since the simple linear additive model in Equation (2.2) has been assumed 

without any explicit justification or validation, it is not clear that the “consensus call of ER 

activity” that it generates is trustworthy for chemicals outside the reference sets, or that it avoids 

the problems of false classifications (“calls”) that it was introduced to solve.

21
Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303-388-1778; Fax 303-388-0609 

www.cox-associates.com

“The following criteria, which characterize the overall potential of chemicals in the UTDB 

to show bioactivity in the uterotrophic assay, were used to develop a list of reference 

chemicals:

Active – a chemical was classified as active if two or more independent GL studies 

classified the chemical as active in the uterotrophic bioassay, regardless of the number of 

inactive studies.

Inactive – a chemical was classified as inactive if two or more independent GL studies 

classified the chemical as inactive in the uterotrophic bioassay, and there were no other GL 

studies reporting the chemical as being active in the uterotrophic bioassay. The resulting 

list of 42 reference chemicals for uterotrophic bioactivity (29 active, 13 inactive) is provided 

in Table 2.8.”

COMMENT: Why are the definitions for Active and Inactive not symmetric? How does this 

asymmetry in definitions affect the reported performance of the methods used? 

“The 42 chemicals listed in Table 2.8 (29 active, 13 inactive) were used to assess the 

performance of the of the ER Pathway model for predicting uterotrophic bioactivity. A 

clear separation of reference chemical bioactivity (active/inactive) is apparent when viewed 

as a function of model score, as shown in Figure 2.12. When using an ER agonist AUC 

value of >=0.1 as the lower limit for bioactivity, the model scores served as a good 

surrogate for the qualitative assessment of uterotrophic bioactivity, with a balanced 

accuracy of 95% (Table 2.9). ... When using this list of reference chemicals (30 Active, 13 

inactive), the ER agonist (R1) AUC model demonstrated a balanced accuracy of 95%, 

active predictive value of 97%, and inactive predictive value of 92%. Discordant results 

between the ER agonist (R1) AUC and uterotrophic outcomes were most likely due to lack 

of bioavailability of the chemical in vivo (“false positives”) or insufficient concentrations of 

the chemical, or its metabolite, in the ToxCast assays necessary to approximate blood/tissue 

concentrations that result in bioactivity (“false negatives”). There appears to be a strong 

relationship between in vitro AUC values and doses causing in vivo bioactivity, raising the 

possibility of both qualitative and quantitative utility for the ER (agonist) pathway model. 

22
Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303-388-1778; Fax 303-388-0609 

www.cox-associates.com

for prioritizing and screening chemicals that may have bioactivity in the ER signaling 

pathway.”

COMMENT: It has not been shown whether the high BA (balanced accuracy) truly reflects “the 

performance of the of the ER Pathway model for predicting uterotrophic bioactivity,” or whether 

even higher performance would have been achieved without using the ER Pathway model, e.g., 

by applying straight-forward statistical or machine learning algorithms to the same data. In 

related work, Cox et al. (2014) noted that simply classifying outputs from inputs using a fully 

automated machine-learning approach, without any ER Pathway model or biological 

interpretation, could actually increase BA to 100%. Here, it is true, and admirable, that the BA is 

high (although largely because of the many “easy” cases in the left and right tails of the curve in 

Figure 2.12). But it is not clear that this implies that “the model scores served as a good 

surrogate for the qualitative assessment of uterotrophic bioactivity” compared to other 

approaches, such as a standard statistical or machine-learning model, or use of just one or a few 

“best” assays. If EPA has conducted modeling using other approaches, these should be 

presented. If not, EPA should provide a rationale, underpinned by scientific evidence, for ruling 

out other approaches.

COMMENT: Since the chemicals in Tables 2.2 and 2.8 overlap, there is potential for selection 

of the chemicals to artificially increase performance scores such as balanced accuracy. For 

example, if the reference chemicals tend to be those for which in vivo and in vitro results agree, 

then a high balanced accuracy might be expected for the reference chemicals, even if the 

balanced accuracy for other chemicals would be much smaller. This may be challenging for EPA 

to address if the overlap of the chemicals in these tables reflects the fact that they are among the 

most-studied chemicals. 

“Analyses presented in Section 2 are meant to provide scientific support for the utility of 

ToxCast models for predicting estrogen receptor bioactivity. ToxCast high throughput 

assays measure multiple endpoints in the ER signaling pathway using different 

technologies and the network models integrate assay results into an estimate of ER agonist 

or antagonist bioactivity strength. ER agonist (R1) and antagonist (R2) AUC scores 

23
Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303-388-1778; Fax 303-388-0609 

www.cox-associates.com

compare very well with reported bioactivity of reference chemicals across a range of

structures and potencies (Section 2.1.1). The agonist (R1) AUC model also accurately 

predicted in vivo ER agonist activity for a relatively large set (~150) of chemicals (Section 

2.2.2). Together, these analyses provide a high degree of scientific confidence in the ability 

of ToxCast ER models to predict bioactivity and demonstrate the utility of using these 

computational tools to prioritize chemicals in the EDSP Universe. ... Analyses in this 

section provide a high degree of scientific confidence in the computational tools presented.”

COMMENT: A different interpretation of the evidence presented is that the “prediction” 

problem on which performance is being evaluated is trivially easy: almost any method for 

making predictions that reflects the information in the in vitro assays, including the AUC model, 

will perform well when evaluated for a set of chemicals that are mainly either positive on almost 

all assays (both in vivo and in vitro) or negative on almost all assays. For this problem, it is easy 

to predict the results of some assays from the results of others. The AUC scores do indeed 

perform well. But other methods (e.g., using any small subset of assays to “vote” for the results 

that will be revealed by the rest) might perform as well or better, while requiring less data and 

less effort. From this skeptical perspective, the admittedly good performance of the AUC scores 

reflects the easy nature of the test problems used to evaluate performance (with many chemicals 

in the tails of the curves, where results are the same for all or almost all assays), rather than 

providing “a high degree of scientific confidence in the ability of ToxCast ER models to predict 

bioactivity” on less easy cases outside the evaluation set. Similarly, the claim that “Analyses in 

this section provide a high degree of scientific confidence in the computational tools presented”

is not justified if the computational tools presented simply detract from the performance 

produced by simpler methods, e.g., by simple voting methods using a subset of assays, or by 

standard statistical or machine-learning methods. In this case, the performance results should 

reflect high confidence that the evaluation task is an easy one on which many methods will 

perform well, rather than high confidence in the specific computational tools presented. To show 

that the computational tools themselves are worthwhile, it would be necessary to show that they 

out-perform simpler methods or standard algorithms such as logistic regression modeling or 

Random Forest. 

24
Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303-388-1778; Fax 303-388-0609 

www.cox-associates.com

COMMENT: The terms “scientific confidence” and “high degree of scientific confidence” 

should be defined and, if possible, quantified. How does scientific confidence differ from 

statistical confidence, or subjective confidence? 

“In particular, the AUC calculation tends to down-weight one receptor if it is overwhelmed 

by another at higher concentrations, which is what we see for most of these chemicals.”

COMMENT: This suggests that alternatives to the AUC calculation might perform better in this 

context.

 

25