November 25, 2014
Dr. Fred Jenkins Office of Science Coordination and Policy (7201M) Environmental Protection Agency US EPA Headquarters 1200 Pennsylvania Ave., NW Washington, D.C. 20460
Posted to Federal eRulemaking Portal: http://www.regulations.gov , Docket EPA

- HQ

- OPP

- 2014

- 0614
Dear Dr. Jenkins: The American Chemistry Council (ACC) appreciates the opportunity to provide comments to EPA and EPA’s Science Advisory Panel on EPA’s White Paper “ Integrated Bioac tivity and Exposure Ranking: A Computational Approach for the Prioritization and Screening of Chemicals in the Endocrine Disruptor Screening Program” 1 (hereafter referred to as EPA’s IBER Profiling White Paper).
ACC 2 has played a constructive role in th e Agency’s development and implementation of the Endocrine Disruptor Screening Program ( EDSP ) for many years . ACC has sponsored technical studies to supply needed data for endocrine test development and validation, and has participated with the broader
sc ientific and stakeholder communities on EPA’s standardization and validation advisory committees. As a member of the Endocrine Policy Forum (EPF), we have contributed to the development of methodologies for integrating EDSP data across the Tier 1 assays, based on transparent and robust weight of evidence (WoE) procedures, to evaluate the potential of a substance to interact with components of the estrogen, androgen and thyroid pathways. These work products have been published in the scientific literature and communicated to EPA. In addition, we have supported basic and applied research to develop : methods for high throughput exposure modeling ; in vitro to in vivo extrapolation ; and, ( as presented at EPA’s September 2014 ToxCast Data Summit ) , exposure acti vity profiling methods to evaluate endocrine assay activity in the context of environmentally relevant human exposures. Our objective , through the years, has been to constructively contribute to the development and impl eme ntation of a scientifically sound and transparent EDSP program.

1 Dated 11/05/2014; ID: EPA

- HQ

- OPP

- 2014

- 0614

- 0003. 2 The American Chemistry Council (ACC) represents the leading companies engaged in the business of chemistry. ACC members apply the science of chemistry to make innovative products and services that make people's lives better, healthier and safer. ACC is co mmit ted to improved environmental, health and safety performance through Responsible Care®, common sense advocacy designed to address ma jor public policy issues, and health and environmental research and product testing. The business of chemistry is an $812 bi llion enterprise and a key element of the nation's economy. It is the nation’s largest exporter, accounting for twelve percent of all U.S. exports. Chemistry companies are among the largest investors in research and development. Safety and security have al ways been primary concerns of ACC members, and they have intensified their efforts, working closely with government agencies to improve security and to defend against any threat to the nation’s critical infrastructure. americanchemistry.com ® 700 Second St., NE | Washington, DC | 20002 | (202) 249

- 7000November 25, 2014 Page 2 We support EPA’s conceptual approach to employ scientifically valid methods to determine putative activity in screening assays measuring endocrine

- related endpoints and to combine this with exposure information to provide an improved scientific basis for priority setting. T he approach presented in EPA’s IBER Profiling White Paper represents a significant advance in bringing such methods to the forefront

. A dditional work by EPA , however, should be considered to improve the scie ntific basis of the methods used, and to fully document model development and performance. W e understand that the Agency intends to release model codes in the future, but the lack of such transparency at the present time renders a comprehensive independent verification of modeling by outside experts and peer reviewers impossible. In the attached report, prepared on behalf of ACC by Dr. Tony Cox, specific comments on the strengths and limitations of the area under the curve ( AUC ) approach for activity profil ing are noted, and recommendations for improvement are made. We request that the Agency give due attention to Dr. Cox’ report as the Agency moves forward to revise the IBER Profiling method for EDSP priority setting. In July 2014 comments to the Agency, we indicated a need for EPA to include appropriate reference substances in such exposure and activity prioritization to provide needed context. We have demonstrated such methodology in those comments and in scientific communications. We note that in the EPA IBER Profiling White paper, EPA proposes cut off levels for AUC values, but has not included use of activity: exposure IBER values for benchmarking. Unless EPA adopts suitable benchmarks, such as genistein, for comparison for activity and exposure, the Agency will be ignoring critical biologically relevant scientific information concerning potency and exposures

. T his could lead to unnecessary EDSP screening/testing. We have also participated in the development of the Endocrine Policy Forum’s (EPF) anal ysis and comments on EPA’s IBER Profiling White Paper, and we fully endorse the findings and recommendations in those comments. As noted in the EPF comments , the AUC methodology is not sufficiently transparent to understand why substances such as potent e strogenic substances (e.g., pharmaceuticals, DES) result in calculated AUCs that are only 1.3

- fold or so greater than the AUC for genistein and diazedin, when in the literature values for relative potency in estrogen assays (see EPF comments) indicate ther e are typically many orders of magnitude difference (e.g. genistein is 10,000

- to 100,000

- fold less potent than DES). Th e scientific rationale for this compression of AUC values is not adequately described or discussed in the EPA IBER White Paper. EPA shou ld address this in the Agency’s response to the SAP review. If you have any questions about these comments or need clarification of any of our conclusions or recommendations or if you would like to further discuss our perspectives on exposure:activity pr ofiling for the EDSP, please contact me by e

- mail at Rick_Becker@americanchemistry.com or by phone at 202

- 249

- 6405. Sincerely

Richard A. Becker Ph.D., DABT

Science and Research Division Attachment

americanchemistry.com ® 700 Second St., NE | Washington, DC 20002 | (202) 249.7000Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303

- 388

- 1778; Fax 303

- 388

- 0609 www.cox
- associates.com
Comments on “Integrated Bioactivity and Exposure Ranking: A Computational Approach for the Prioritization and Screening of Chemicals in the Endocrine Disruptor Screening Program.”
Tony Cox, tcoxdenver@aol.com , 11

- 2 4

- 2014

EXECUTIVE SUMMARY

These comments discuss the approaches to model development taken in the U.S. EPA Endocrine Disruptor Screening Program report entitled “Integrated Bioactivity and Exposure Ranking: A Computational Approach for the Prioritizat ion and Screening of Chemicals in the Endocrine Disruptor Screening Program.” The report offers a highly optimistic assessment of the computational methods that it describes

– primarily, an “AUC [area

- under

- curve] model” that adds up weighted sums of assay results at different concentrations to get a final score

– leading to the key conclusion s that “ER agonist (R1) and antagonist (R2) AUC scores compare very well with reporte d bioactivity of reference chemicals across a range of structures and potencies (Section 2.1.1). The agonist (R1) AUC model also accurately predicted in vivo ER agonist activity for a relatively large set (~150) of chemicals (Section 2.2.2). Together, thes e analyses provide a high degree of scientific confidence in the ability of ToxCast ER models to predict bioactivity and demonstrate the utility of using these computational tools to prioritize chemicals in the EDSP Universe.” A precisely similar approach is taken to AR agonist and antagonist modeling (Section 3.1). The following comments question whether the evidence presented really justifies “a high degree of scientific confidence in the ability of ToxCast ER models to predict bioactivity and demonstr ate the utility of using these computational tools to prioritize chemicals in the EDSP Universe.” We note that the computational models and methods are not presented with the standard model assessment information needed for an independent expert or peer r eviewer to critically evaluate model construction and performance, such as results of model diagnostic s, uncertainty characterizations, and sensitivity analyses. Nor has it actually been shown that the models do a “good” job, relative to other approaches , in prioritizing EDSP chemicals

. The models are simply assumed and used , and the ir results reported, without any critical comparison to alternative approaches . The possibility that much simpler methods would work much better
1Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303

- 388

- 1778; Fax 303

- 388

- 0609 www.cox
- associates.com
cannot be ruled out by the evidence presented. Although it is true (and encouraging) that false positive and false negative rates are low when assessed on the specific sets of reference chemicals considered, this may reflect the fact that most of the righ t answers are easy to predict. For example, “No activity” will usually be correct for in vivo results unless there is a broad pattern of activity in the in vivo assays. Little sophisticat ed analysis or computational modeling

are needed to see and exploit this simple pattern . The fact that the AUC models get this much right for the reference chemicals (some of which were used both to train and to test the AUC models) is no guarantee that they are likely to make accurate predictions for other chemicals , part icularly those with structures that are outside the training/test sets

. EPA should specify the domain of applicability and indicate structures of the classes of substances that the models can be used for, and the classes for which model use may be limited.

CRITIQUE OF MODELING ASSUMPTIONS
The following key modeling assumptions and methods were used in developing all of the AUC models ( for ER agonist s , ER antagonist s , AR agonist s , and AR antagonist s)

.

• Hill models. Concentration

- response curves can be described by any of three models: “ a constant model (no activity), a Hill model, and a ‘ Gain

- Loss ’ model, which is a combination of two Hill models, one increasing in signal, and the other decreasing at higher concent ration.”

• Simple linear additive model: “ The computational model assumes that the value (the efficacy, A ) returned by an assay at a given concentration is a linear sum of the contributions from the receptors that it measures, i.e., it is a simple linear additive model ”

• Curve

- fitting: “ The goal is then to

... minimize the difference between the predicted assay values ( AA ii pp rr ee dd ) and the measured ones ( AA ii mm ee aa ss ) for each chemical and concentration. For each chemical

- concentration pair, a constrained le ast

- squares minimization approach is used... ” In addition, a “ penalty term helps stabilize the solutions and enforces a reasonable physical assumption about chemical promiscuity, i.e., that it is unlikely that most chemicals will strongly and specifically interact with many dissimilar molecular targets.”

2Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303

- 388

- 1778; Fax 303

- 388

- 0609 www.cox
- associates.com
These methods and models are simply assumed and applied, without derivation or justification. The best practices in predictive modeling recognize that selection of a model may profoundly influence results , and provide the following information so that others can understand this influence and assess the likely predictive performance of the models in new applications.

• Model diagnostic results: How well do the assumed models describe the data to which they

a re applied? Are there patterns of systematic differences between predicted and observed values? Do model specification errors bias the conclusions?

• Sensitivity analysis and model uncertainty results: How do the results change if different plausible models are used instead?

• Model cross

- validation

. How does the modeling approach perform if it is repeatedly trained on some subsets of cases and tested on other, disjoint subsets?

Without such results, it is impossible to determine whether the p articular modeling choices made are appropriate for the data to which they are applied, or to assess whether they will produce credible, unbiased, accurate, or useful results for the chemicals to which they will be applied in future , compared to other models and methods

. Thus, a key recommendation for improving the current report is that the authors should provide model diagnostic results, sensitivity analysis and model uncertainty characterization results, and c ross - validation results so that the predictive performance of the proposed method can be better assessed. The use of a simple linear additive model (Equation 2.2) is particularly troubling, insofar as previous modeling work has established that receptor

- me diated responses typically exhibit highly nonlinear (e.g., switch

- like or threshold ) behaviors (e.g., Andersen and Barton, 1999, http://toxsci.oxfordjournals.org/content/48/1/38.full.pdf ; Zhang et al., 2014, http://ehp.niehs.nih.gov/1408244/

. )

. The linear additive model assumed in Equation (2.2) is a very special case of a standard statistical regression model (a main

- effects only linear model with no error term). This simplification, presented without justification, appears to be a huge step backward compared to the use of modern regression models (e.g., generalized additive models, generalized linear models, etc.) Including a w ell

- modeled noise term, and checking for important interactions and nonlinearities , and then modeling them where needed, are well within the current state

- of

- the

- art of statistical practice, and could potentially improve the predictive

3Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303

- 388

- 1778; Fax 303

- 388

- 0609 www.cox
- associates.com
performance, general izability, and trustworthiness of model

- based predictions using relatively well

- established and readily available statistical prediction methods. Similarly, the area

- under

- curve (AUC) obscures difference between concentration

response curves that may be im portant for sensible priority

- setting. For example, consider two hypothetical chemicals, one with activity only for concentrations above a certain threshold, and the other with activity only below that threshold level. A risk manager who can afford to re duce exposure concentrations for only one of the chemicals in the near term might strongly prefer to reduce the concentration of the first, rather than the second. Yet, there is no reason that the AUC for the first must be higher than that for the second. In practice, for the reference chemicals studied, AUC works pretty well. But other approaches might work better (e.g., approaches that pay attention not only to the AUC, but also to the shape of the concentration

- response curve). Moreover, for other ch emicals, the AUC model may perform poorly or perversely, as in this hypothetical example. There is no statistical or mathematical justification in the work presented for the belief that the AUC model will have high predictive accuracy in general for biolo gical activity , especially for chemicals for which assay results are discordant. Nor is it clear that the implicit value judgments incorporated in the AUC metric (e.g., that a concentration

- response curve with many small values should have the same priori ty as an equal

- AUC curve with a few large values) are those that a policy

- maker should want to use to set priorities. Discussion of such examples and limitations of the AUC metric would be helpful.

TECHNICAL DIRECTIONS FOR IMPROV ING PREDICTIVE POWER AND CONFIDENCE

The current state

- of

- the

- art in predictive modeling suggests the following main technical directions for improving the current AUC model. 1. Compare the current results to those from standard machine

- learning and statistical algorithms (e.g., logistic regression models, GAMS and GLIM models, Random Forest and MARS models and algorithms ) . Use methods such as boosting to combine the results from the most successful predictive models to yield final predictions. 2. Use model cross

- validation to corre ct for over

- fitting bias and to better assess the true predictive (rather than descriptive) performance of models.

4Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303

- 388

- 1778; Fax 303

- 388

- 0609 www.cox
- associates.com

3. Evaluate whether a small subset of ToxCast assays can predict at least as well as the full set of 18 assays used in the current ER AUC model. In particular, it seems very plausible that many of these assays are redundant. Modern methods of feature selection (such as mutual information and conditional entropy reduction methods) might well reveal that a small subset of assays can achieve equal or superior predictive performance compared to the currently proposed 18 assays and the AUC models

. 4. Allow for missing data. Computational statistical methods and algorithms (ranging from data

- augmentation, multiple imputation, and expectation

- maximization algorithms to the use of “surrogate splits” in classification and regression trees) make it practical to perform high

quality predictions in many settings similar to this one, even when some chemicals have missing values for one or more assays.

These te chnical directions are very likely to be able to build on the promising start made in the current report to provide more robust (e.g., missing data

- tolerant), defensible, better

- validated, and more accurate and trustworthy predictions of in vivo ER and AR biological activities based on the results of HTS assays. The preceding points and recommendations are developed further in the detailed comments that follow.

5Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303

- 388

- 1778; Fax 303

- 388

- 0609 www.cox
- associates.com

DETAILED COMMENTS
The following comments take a deliberately , systematically sk eptical view of the claims advanced in Sections 2 and 3.1 of the recent report entitled “ Integrated Bioactivity and Exposure Ranking: A Computational Approach for the Prioritization and Screening of Chemicals in the Endocrine Disruptor Screening Program. ” A few comments are also made on Section 1. The purpose of these comments is not to attack or undermine this admittedly very interesting and promising work, but to help improve its exposition and approach by challenging its assertions, questioning its met hods, and suggesting less optimistic interpretations for its key findings and conclusions. This is intended as a “Devil’s Advocate” piece to help make the work stronger.
“The U.S. Environmental Protection Agency (EPA) is continuing a series of scientific peer reviews focused on computational approaches for prioritization and screening of chemicals in the Endocrine Disruptor Screening Program (EDSP).”
COMMENT: It might be useful to explain the decision problem (s) to which “prioritization and screening of chemicals in the Endocrine Disruptor Screening Program (EDSP)” is intended to be a solution. For example, to support risk management decisions about which chemicals are most worth focusing on first to reduce unde sirable impacts, it might be necessary to consider not just how large risks are, or how likely different chemicals are to have endocrine disruptor activity, but rather to consider by how much the risks from different chemicals could be reduced by allocatin g available resources. For example, if regulating chemical A w ould reduce risk of an adverse outcome from exposure to A from 0. 8 to 0 , and if regulating chemical B w ould reduce risk from exposure to B from 0.95 to 0.94 then a risk manager who could only a fford one of these as top priority might well prefer to regulate chemical A instead of chemical B, even though B has a larger risk of adverse outcome. To support effective risk management, therefore, “prioritization and screening of chemicals” would have to model potential changes in risk from alternative resource allocation prioritization and screening rules, so that the most effective alternative could be selected. Explicitly stating the purpose of the prioritization and screening effort (e.g., to help inform allocation of risk management resources to reduce risks) could help make clear whether the current design of the EDSP supports the stated purpose, or whether
6Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303

- 388

- 1778; Fax 303

- 388

- 0609 www.cox
- associates.com

different information and considerations are required, such as estimation of the change in risk that would be produced by an affordable change in exposures.
“This document describes innovative approaches for determining estrogen and androgen bioactivity based on a series of computational models integrating data from ToxCast high throughput scre ening (HTS) assays.”
COMMENT: The models in this document are not capable of “determining estrogen and androgen bioactivity,” but only of informing guesses about whether such activity will be present in specific systems . Th us, this sentence seems to pro mise more than is delivered in asserting that it describes “approaches for determining estrogen and androgen bioactivity.”
“This document describes a series of generalizable chemical prioritization and screening methodologies based on endocrine bioactivity and exposure which could be applied and adapted to address other endocrine pathways and EDSP chemicals. The focus for review by the Federal Insecticide, Fungicide, and Rodenticide Act (FIFRA) Scientific Advisory Panel (SAP) is on the concepts a nd methodologies, and not on results for specific chemicals that may be prioritized for EDSP screening. It is the intention of the EPA to complete the prioritization methodology and process after receiving recommendations from the SAP on the proposed compu tational approaches.”
COMMENT: In general, prioritization is not an effective approach for allocating limited risk management resources. For example, suppose that an agency must allocate a limited budget to reduce risks from endocrine disruptors. Three opportunities have been identified to reduce risks, as follows:

• Regulating chemical A reduces risk from exposure to A from 100 to 80 (on some appropriate scale) . It costs $30

.

• Regulating chemical B reduces risk from 50 to 10. It costs $40.

• Regulating c hemical C reduces risk from 25 to 0. It costs $20.

7Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303

- 388

- 1778; Fax 303

- 388

- 0609 www.cox
- associates.com
The goal for resource allocation is to achieve the largest possible total risk reduction for the available budget. What priority ranking of A, B, and C achieves this goal , if resources are allocated from the top of th e priority list down until they are exhausted? The answer is that no priority ranking achieves this goal . Instead, the correct answer depends on the budget. For a budget of $49, the largest feasible risk reduction is achieved by regulating B, so the best priority order puts B first. If the budget is $50, then regulating A and C achieves the greatest risk reduction, so B should be ranked last. At $60, the best investment is to fund regulation of B and C, so now A should be ranked last. Th us, no priority

- ordering of A, B, and C optimizes resource allocations independent of the budget. For example, no priority order of A, B, and C is optimal for budgets of both $49 and $50. The difficulty illustrated here is that optimal resource
constrain ed investments in reducing risks , or in Tier 1 screening, or on other desired activities, cannot in general be represented by priority

- rankings. Since, in general, no prioritization of chemicals can guide optimal allocation of resources, it would be helpf ul for the report to explain in greater detail exactly what the prioritization is intended to accomplish, and to consider whether different approaches (such as mathematical optimization) might better accomplish what is intended.

“ Figure 1.3 : A graphical representation of the Integrated Bioactivity
- Exposure Ranking (IBER) approach indicating no priority chemicals (those that either have no potential bioactivity or no potential exposure), low (chemicals for which bioactive concentrations are considerably hi gher than potential exposures), medium, and high priority (chemicals for which bioactivity and exposure concentrations are close or overlapping). ”
COMMENT: The y axis in Figure 1.3 has units of mg/kg/day. However, the bioactivity of a chemical may depen d on how exposures are distributed over time (e.g., one large exposure every 7 days vs. 1/7 of that exposure on each of 7 consecutive days). This information is lost, for both exposure and bioactivity, in the conceptual model shown in Figure 1.3.

“In th is section, we describe an approach to combine data from multiple in vitro assays measuring estrogen receptor activity to make a weight of evidence prediction concerning “true” activity.”
8Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303

- 388

- 1778; Fax 303

- 388

- 0609 www.cox
- associates.com

COMMENT: The term “weight of evidence prediction” should be preci sely defined. In general, predictions made by sound principles (e.g., by conditioning on all available evidence using Bayes’ Rule) cannot necessarily be represented by combining items of evidence (e.g., values of predictors) using any system of weights or additive scoring models. The term “weight of evidence” is often used to describe processes that seek to use professional judgments and expert opinions to make guesses or to reach consensus conclusions based on ambiguous data. Such conclusions are often m istaken (see e.g., Kahneman, Thinking, Fast and Slow , for a discussion of expert judgments). Is that what is intended here?
“Therefore, a key aspect of the analysis given below is the use of a reasonably large set of well
- studied reference chemicals.”

COMME N T: The report should be much more precise about technical terms, and not rely on vague or undefined terms (such as “reasonably large set”) for key aspects of the analysis. The term “reasonably large set ” should be clearly defined. The statistical literature on design of experiments and response surface evaluation shows that even a very large set of well

- studied reference chemicals will not provide the information needed to make accurate predictions for other chemicals outside that set . Instead, t he set of reference chemicals must also be sufficiently diverse , in a sense that can be made precise using f ormal definitions and techniques for design of experiments. Use of such precise definitions and statistical design criteria ( such as D

- optimality ) may be useful in explaining what criteria the authors intend to advocate for determining why and whether the set of reference chemicals used is sufficient to permit confident predictions for chemicals not in the set

.
“By combining data from multiple ass ays, and using a model that detects these technology
specific false positives, we can arrive at a more confident assessment of the in vitro estrogenicity of a chemical.”
COMMENT: What, precisely, is being asserted here? Does “we can arrive at a more confident assessment” mean something different from “we will feel better about our assessment of the in
9Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303

- 388

- 1778; Fax 303

- 388

- 0609 www.cox
- associates.com

vitro estrogenicity of a chemical”? If so, what exactly is the claim that is being made? Is it that confidence intervals will be tighter, or that probabilities will be closer to 0 and 1, or something else?
“Using a suite of assays to detect pathway perturbations overcomes limitations of a single assay . ”

COMMENT: Could using a suite of assays introduce further limitations (e.g., additional opportunities for false

- positive results, additional noise and error rates in the conclusions) that using only one or a few best assays would avoid or reduce? Is the intended claim her e that using a single assay can never give superior performance to using a suite of assays? EPA implies that no single assay drove the AUC score, which seems to imply that no assay was most predictive for the reference set. But EPA has not shown the result s to indicate whether certain subsets together were more predictive than the selected suite

. Again, it would be helpful to readers if the claim being made were stated very clearly, and any conditions required to prove or justify it were stated explicitly.
“To navigate this complexity, we developed mathematical and statistical models to infer whether chemicals that activate specific patterns of the in vitro assays were more likely to be ER agonists or antagonists, or were more likely to be causing assay activity through specific types of assay interference.”
COMMENT: The mathematical and statistical models used deal only with associations and patterns, and do not provide any formal causal analysis or modeling (e.g., using DAG models, SEM and path model s, Granger causality tests of activation patterns over time, etc. based on the conceptual networks provided. ) They thus do not appear to be appropriate models for quantifying or comparing likelihoods of alternative causal explanations, as this sentence

su ggests. Additional discussion of how likelihoods of different causal explanations for observed patterns (e.g., ER agonist, antagonist, or assay interference) were (or should be) assessed and validated would therefore be useful. Does the report mean to imp ly that sound causal

10Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303

- 388

- 1778; Fax 303

- 388

- 0609 www.cox
- associates.com
conclusions can be drawn using the mathematical and statistical models developed, without need for formal causal analysis methods?

“ Overall approach : The input data for the model includes chemical structures and concentration
- response data for 18 ER
- related in vitro assays, plus data for many, non
- ER in vitro assay endpoints (ranging from 186
- 821 assays, depending on the chemical). ”

COMMENT: How sensitive are the results to the selection of these 18 ER

- related in vitro assays? Would other, equally valid, selections produce different results? If so, then how should performance results be a ttributed to choice of inputs vs. to the methods used?
COMMENT: How do these choices of input data affect performance quality (e .g., false positives, false negatives, and balanced accuracy (BA) ? Might different selections of inputs lead to improved performance? For example, Cox et al. (2014) ( www.sc iencedirect.com/science/article/pii/S0273230014000920 ) report the results of analyses suggesting that the selection of these 18 assays needlessly reduces performance: “The main results of applying the Random Forest algorithm to the full set of HTS (ToxCast TM assay) indicators are presented in Table 3. Significant improvements in fit are noted, with BA scores of 1 for classes of guideline

- E and

- A studies (Table 3), compared to 0.94 and 0.92 (Table 2) achieved using 18 endocrine assays for prediction model de velopment.... Thus, looking at BA, fully automated methods gave significantly better BA results than manual selection for both guideline and non

- guideline studies, and for all classes.”
COMMENT: How well can subsets of these 18 assays be used to make predi ctions, e.g., in cases of missing data?
“The model produces scores for agonist activity, antagonist activity and for ‘ false ’ activity in each assay technology. These scores can then be compared with reference chemicals, structure classes, and other prop erties.”

11Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303

- 388

- 1778; Fax 303

- 388

- 0609 www.cox
- associates.com

COMMENT: It is unclear what is being suggested here. What does it mean to say that “scores can then be compared with reference chemicals?” (A score is a number; a chemical is not. What would a comparison between them consist of, and what cou ld it reveal that would be useful?)
COMMENT: Scores can be generated in an infinite number of possible ways. What formal criteria have been used, or should be used, to decide whether the scores produced by these models are useful for making predictions, for supporting decisions, or for oth er purposes? How are the scores intended to be used, and what guarantees, if any, can be provided that they are useful for the purposes for which they are intended?

“The 1812 chemicals used in this model were selected because they had data for the full set of 18 ER assays.”
COMMENT: Is there an assumption that chemicals with complete data for the 18 ER assays are representative of the wi der universe of chemicals? If yes, how is this justified? If no, then is there any guarantee that results can be ge neralized?

“All of the concentration
- response data were analyzed using the ToxCast data analysis pipeline, which automates the processes of baseline correction, normalization, curve
- fitting, hit
- calling and detection of a variety of potential confounders, which are annotated as “caution flags”. This pipeline, along with all of the raw and processed data, and annotations is publicly available [http://epa.gov/ncct/toxcast/data.html].”

COMMENT: As noted by Cox et al. (2014), “Full details of how the dose

– re sponse data were processed were not given in the Supplementary Materials, and despite several helpful discussions with the authors of the HTS

- derived EDSP T1S endocrine prediction models, we could not replicate them. ...None of the published prediction model algorithms were transparent, nor were the algorithms clearly defined and documented. Input from the first author was helpful, but significant efforts were still needed to reverse engineer the prediction models. For the most part, the underlying data sets were available to support this reverse engineering activity, however, the
12Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303

- 388

- 1778; Fax 303

- 388

- 0609 www.cox
- associates.com
precise details of how dose

– response data were re

- coded as binary and response indicators (using an unspecified Hill function with some manual adjustment) were not documented or read ily available in the Supplementary Materials.”

“The concentration
- response data for each in vitro assay were fit to 3 models: a constant model (no activity), a Hill model, and a “Gain
- Loss” model, which is a combination of two Hill models, one increasing in signal, and the other decreasing at higher concentration.”

COMMENT: The selection of a model may profoundly influence results. To reveal this influence and to allow others to assess the quality of a model, the following information should be provided

.

• Model diagnostic results: How well do the assumed models describe the data to which they are applied? Are there patterns of systematic differences between predicted and observed values? Do model specification errors bias the conclusions? How confident can we be in the answers to these questions?

• Sensitivity analysis results : How do the results change if different plausible parametric models ar e used instead? For example, as previously noted, Cox et al. (2014) find that a different, entirely automated procedu re using standard R routines improves classification of chemicals ( to BA = 100% for E class chemicals). How would the results change if a non

parametric curve were fit to each concentration

- response data set?

• Model uncertainty re sults: If statistical techniques such as Bayesian Model Averaging were used to account for model uncertainty, then would the same predictors still be retained? How sensitive are results to over

- fitting using Hill and Gain

- Loss models , e.g., as assessed by k

fold model cross

- validation ?

• Results from model c ross - validation, out

- of

- sample performance, and correction for over

fitting. How would the use of k

- fold cross

- validation to correct for model over

- fitting change reported results?

Without such results, it is impossible to determine whether the particular modeling choices made (e.g., selection of Hill and Gain

- Loss models) are appropriate for the data to which they are

13Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303

- 388

- 1778; Fax 303

- 388

- 0609 www.cox
- associates.com
applied, or to assess whether they produce credible, unbiased, accurate , or useful res ults compared to other methods and models

.

“ The following scheme is used to filter out these non
- selective, cell
- stress or cytotoxicity
related assay hits. For chemicals with two or more positive responses in cytotoxicity assays, we calculate the mean logAC50 (cytotoxicity) and the median absolute deviation (MAD) of the logAC50(cytotoxicity) hits. Next, we calculate the median of the MAD of the logAC50(cytotoxicity) distributions across all chemicals to define the global cytotoxicity MAD. A new value (t he Z
- score) is then assigned to each assay hit... (Equation 2.1). If fewer than 2 cytotoxicity assays are hit, the median cytotoxicity concentration is arbitrarily set to 1000 μM, which simply sets all Z
- values for assay hits to a value > 3. A hit with a la rge value of Z occurs at concentrations significantly below where cytotoxicity is occurring. This hit is both unlikely to be caused by cell
- stress or cytotoxicity
- related processes, and is more likely to be able to cause toxicity through a target
- selective mechanism. ”

COMMENT: What other procedures were considered, if any, and how do results from this procedure compare to the results from other, equally well justified, alternative procedure s?
COMMENT: Can the following assertions be proved? “A hit with a large value of Z occurs at concentrations significantly below where cytotoxicity is occurring. This hit is both unlikely to be caused by cell

- stress or cytotoxicity

- related processes, and is more likely to be able to cause toxicity through a target

- selective mechanism.” Evaluating cytotoxicity and correcting for it in predicti ve modeling is indeed desirable , but the available data and information presented are not sufficient to independently verify these statements. T he Z

- score method deserves its own section and appendix of assay data

.

“

Reference chemicals : A set of 45 positive and negative reference chemicals were used to evaluate the performance of the model.”

14Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303

- 388

- 1778; Fax 303

- 388

- 0609 www.cox
- associates.com
COMMENT: Is there any guarantee that performance results on this set of 45 reference chemicals can be extrapolated to performance on other chemicals?

COMMENT: How would the evaluated performance of the model change if different sets of reference chemicals were used? For exam ple, if many random subsets of size 30 were used to evaluate the model, then what would be the distribution of its performance statistics?
“Structure of the Network: Figure 2.1 is a graphical representation of the network used to evaluate the integrated i n vitro assay responses.... The model assumes that a chemical that interacts with the ER will bind in one or both of the agonist or antagonist conformations and that this will trigger activity in the appropriate pathway. ... The details of the process connecti ng the pseudo
- receptors to the assays were simplified to a single connection because, in general, we do not know the intermediate details of these processes or even the identity of the pseudo
- receptors. Note that the pseudo
- receptors may not be what a biol ogist would define as a receptor, but are surrogates for generic processes such as cell
stress or cytotoxicity that can lead to non
- ER mediated assay activity. It is possible to describe many possible alternative pathways, but in general, the data are not sufficient to distinguish between alternate models. Pseudo
- receptors are assigned to each group of assays (technology group) and to each assay individually.”

COMMENT: From the purely pragmatic standpoint of creating and validating useful predictive output s, the whole discussion of pathways, “pseudo

- receptors,” networks, etc. seems unnecessary: what was actually done is summarized in the statement “The computational model assumes that the value (the efficacy, A ) returned by an assay at a given concentration is a linear sum of the contributions from the receptors that it measures, i.e., it is a simple linear additive model.”
COMMENT: As noted in Cox et al. (2014), using a fully automated technique with no atte mpt to impose constraints or assumptions about underlying biological networks gives improved (perfect) classification performance for E chemicals.

15Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303

- 388

- 1778; Fax 303

- 388

- 0609 www.cox
- associates.com

“

Mathematical Model: The computational model assumes that the value (the efficacy, A ) returned by an assay at a given concentration is a linear sum of the contributions from the receptors that it measures, i.e., it is a simple linear additive model: AA ii =Σ FF ii jj RR jj

jj Equation (2.2) where the elements of the F matrix are 1 if there is a connection between a receptor and an assay, and zero otherwise.”
COMMENT: Systems biology models and previous studies show that receptor

- mediated responses for endocrine disruptors typically exhibit switch

- like and threshold behaviors (e.g., Andersen and Barton, 19 99, http://toxsci.oxfordjournals.org/content/48/1/38.full.pdf ; Zhang et al., 2014, http://ehp.niehs.nih.gov/1408244/

. No justification is given for a simple linear additive model, as in Equation (2.2) , as being realistic.
COMMENT: All standard linear and nonlinear regression models include error terms, yet there is no error term in the model in Equation (2

. 2), and no discussion of its properties (e.g., homoscedasticity , residuals, need for transformations to stabilize error variance, etc. ) The use of what amounts to a main

- effects only linear model with no error term appears to be a huge step backward compared to the use of modern regression models (e.g., GAMS, GLIM, etc.) This to pic is mentioned on page 31, as follows: “All assays have a background level of noise, and one must define a threshold to decide between a hit and a non

- hit.” Yet, the noise is not included in the model in Equation (2.2) and is not explicitly modeled. It should be.
COMMENT: To justify use of Equation (2.2), the authors should provide evidence that: (a) Model diagnostics indicate that the model is appropriate for the data (b) Analyses using techniques such as CHAID, CART, or MARS indicate that there are no important interactions (or, if there are, the model should be revised to include them) (c) Analyses using techniques such as polynomial regression, GAMS, or smoothing splines indicate no significant departure from linearity and no need to model nonlinearities
No such results have been provided, leaving unclear the rationale for Equation (2.2) and its acceptability as a useful model (preferable to, e.g., standard nonlinear regression models).
16Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303

- 388

- 1778; Fax 303

- 388

- 0609 www.cox
- associates.com

“The goal is then to find a set of RR jj values that minimize the difference between the predicted assay values ( AA ii pp rr ee dd ) and the measured ones ( AA ii mm ee aa ss ) for each chemical and concentration. For each chemical
- concentration pair, a constrained least
- squares minimization approach is used ...”
COMMENT: Why is this “t he goal ” ? (The goal for achieving what purpose?) This is a curve fitting goal . As stated, it encourages over - fitting (i.e., choosing R j values to optimize fit to the training data, rather than to optimize predictive accuracy o utside the training set). A better goal, for purposes of predictive modeling, would be to select R j values to minimize k

- fold cross

validation error.
COMMENT: As shown in the following example, it is possible to have a completely accurate fit between ob served and measured values in a training set, and yet to have no predictive power outside the training set. Suppose that cases 1

- 6 are “reference chemicals” available for training, with 4 predictors (analogous to the 18 ER assays) and one predicted outcome (the right

- most) column. The predictive challenge is to predict the outcome for case 7 (the value, 0 or 1, in the “?” cell in the lower right of the table.) The following two models each describe the training data in rows 1

- 6 perfectly, yet they make op posite prediction for case 7: Model 1: Predicted value = 1 if sum of predictors 2, 3, and 4 > 1, else 0 Model 2: Predicted value = value of Predictor 3.

Case # Predictor 1 Predictor 2 Predictor 3 Predictor 4 Predicted

1 1 1 1 1 1 2 0 0 0 0 0 3 0 1 1 0 1 4 1 1 0 0 0 5 0 0 0 1 0 6 1 0 1 1 1 7 1 1 0 1 ?

17Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303

- 388

- 1778; Fax 303

- 388

- 0609 www.cox
- associates.com
Likewise, they would make opposite predictions for a chemical with predictor values of (0, 0, 1, 1, 0). This illustrates that successful classification or description of reference chemicals with respect to biological activities is a different task from successful prediction of the biological activities of chemicals outside the training set

. As in these examples, it is possible for a computational procedure to have up to 100% balanced accuracy on the former task, while making predictions with no better than a random (50

- 50) probability of being correct for the latter task. Seeking “ RR jj val ues that minimize the difference between the predicted assay values ( AA ii pp rr ee dd ) and the measured ones ( AA ii mm ee aa ss ) for each chemical and concentration” addresses the former task, but not the latter. Yet, it is the latter that should be the goal of chi ef interest to practitioners who want to apply the model to chemicals other than those used in building the model.
“This penalty term helps stabilize the solutions and enforces a reasonable physical assumption about chemical promiscuity, i.e., that it is unlikely that most chemicals will strongly and specifically interact with many dissimilar molecular targets. Values used here

are αα = 0.05 and SS RR 0 = 0.5.”
COMMENT: This penalty term seems ad hoc , insofar as no derivation or justification is given for this specific mathematical formula . What alternative penalty terms were considered, and how sensitive are results to different, equally plausible, choices of the penalty term?

COMMENT: How sensitive are the results to the selected values αα = 0.05 and SS RR 0 = 0.5?
COMMENT: Is it necessarily the case that different receptors (or other molecular targets) are so dissimilar as to make it unlikely that chemicals will strongly and specifically interact with two or more of them? What is the specific eviden ce supporting this assumption?
“The model results in a response value (between 0 and 1) for each receptor at each concentration. These results are summarized as area under the curve (AUC), which is the integral across the concentration range... ”

18Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303

- 388

- 1778; Fax 303

- 388

- 0609 www.cox
- associates.com

COMMENT: The use of the AUC obscures potentially important differences in concentration

response curves. For example, it appears that Equation 2.6 can assign identical values to three concentration

- response curves that are flat, increasing (perhaps with a threshold), and decreasing in concentration, respectively, and yet these three curves might have very different implications for risk assessment and risk management priorities.

COMMENT: There is no explanation of how sign(slope) factor is computed, or w hy it is considered an appropriate way to “discount” AUCs for assays that are activated at relatively low concentrations.

COMMENT: No formal jus tification for the use of AUC has been provided. Does it provably optimize some criterion, or support good de cisions, or have any other desirable properties that make it superior to other measures? If there is some explicit justification for using the AUC, it should be de scribed. The current assertions that “The values of AUC can be related to the potency (AC50 ), and are useful in interpreting different magnitudes” are not given any rigorous explanation or support. T he text correctly notes that “One issue that arises with this approach is how to interpret these receptor AUC values” and explains that “Qualitativ ely, we interpret these R values as concentration

- specific probabilities that the chemical is interacting with the corresponding (pseudo) receptor.” But this means that the R values (and the AUC derived from them) are interpreted in terms of probabilities of interactions with fictitious constructs

– hardly a clear or reassuring basis for use. Moreover, no justification is provided for such a probabilistic interpretation . It is not specified what information these “probabilities that the chemical is intera cting with the corresponding (pseudo) receptor” are conditioned on, or how they would or should change as evidence from similar chemicals is considered. In short, it is not clear why or whether the AUC is a useful measure . The hand

- waving discussion provi ded raises more questions than it answers

. For example, how do we know that the R values are not the squares or square roots of the “concentration

- specific probabilities that the chemical is interacting with the corresponding (pseudo) receptor”? Or, if i ndeed “we interpret these R values as concentration

specific probabilities that the chemical is interacting with the corresponding (pseudo) receptor,” and if arbitrarily weak interactions are allowed, then why are not all the R values equal to 1,
19Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303

- 388

- 1778; Fax 303

- 388

- 0609 www.cox
- associates.com
assuming that at least some (perhaps undetectably weak or extremely rare) interaction is always possi ble? In short, t he suggested interpretation in terms of probabilities has not been justified , and does not appear to be obviously plausible or correct

.

“Note that BPA has activity in one of the antagonist assays (A17, gold), consistent with Selective Estrogen Response Modulator (SERM) activity at high concentrations. However, the results of the network model do not reflect this weak antagonist behavior.”

COMMENT: The report should discuss further the implications of such discrepancies between model predictions and assay results . Do they signal that a different model should be used? Would using a nonlinear model produce more accurate results?

“

Model Results for Reference Chemicals : The AUC values for the ER reference chemicals are shown in Figure 2.4 . For the positive agonist chemicals, all but two have non
- zero AUC values, these being diethylhexyl phthalate (DEHP) and dicofol.”

COMMENT: It should be noted that these “model results” actually refer to the ability of the model to fit the reference chemical data used to construct it. Thus, they are closer to model fits

– answering whether the model can at least reproduce the data used to build it

– than to what is often meant by “results,” i.e., results of applying the model to new cases , which might be used to assess its predictive validity rather than its descriptive validity , as in Section 2.2.2

.
“We explore several potential causes of pseudo
- receptor activity. The first potential cause has to do with assay noise. All assays have a background level of noise, and one must define a threshold to decide between a hit and a non
- hit. To be called a hit i n the data analysis pipeline, the maximum efficacy must be above a defined statistical criteria, which is typically 10 MAD above baseline. Given this relatively high threshold (10 MAD, essentially 10 standard deviations), one would expect few noise
- related false positives if the noise was approximately normally distributed. In particular, for full agonists in an assay with only normally distributed noise, one should see T values of ~100% for the actives, and T values for inactive chemicals should be small ( <<10 MAD). We compared the T distributions
20Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303

- 388

- 1778; Fax 303

- 388

- 0609 www.cox
- associates.com
between the strong agonists and the chemicals in each of the pseudo
- receptor groups using a t - test. Table 2.3 lists the mean T for chemicals active in the pseudo
- receptors and in the agonists, and a t
- test p - value . (Note that for the antagonist assays and R9, the comparator group was the antagonist chemicals, AUC(R2) ≥0.1. We see that the actives (AUC≥0.1) in R6, A9, A12, A13, A15 and A17 have significantly lower T values than the comparator groups, but are still a bove the statistical threshold, implying that at least some of this pseudo
- receptor activity is associated with non
- normally
- distributed assay noise.”
COMMENT: As previously noted, assay noise should be explicitly included in the modeling (instead o f the noiseless Equation (2.2)) so that standard regression techniques such as transformations to stabilize variance c an be applied if needed. It is not necessarily plausible that noise is normally distributed (since negative values are not allowed, homoscedas ticity seems implausible for concentration

- sensitive responses, and phenomena such as zero

- inflation or finite mixture distributions in concentration

- response relations may require discrete probabilities to model accurately).

COMMENT: The conditions fo r use of the t

- test have not been verified.
“There are 11 compounds that we classify as negative for which there are multiple literature reports of being positive binders. ... ER Model Summary : This section has demonstrated a method to combine data from multiple in vitro assays in the ER pathway in order to generate a consensus call of ER activity. The need for this approach is driven by the presence of false positive and false negative calls i n all assays. An ongoing topic of research is trying to use chemical structure and properties to understand exactly why specific chemicals cause this (mainly) false positive activity.”
COMMENT: Since the simple linear additive model in Equation (2.2) has been assumed without any explicit justification or validation, it is not clear that the “consensus call of ER activity” that it generates is trustworthy for chemicals outside the reference sets , or that it avoids the problems of false classifications (“ca lls”) that it was introduced to solve.

21Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303

- 388

- 1778; Fax 303

- 388

- 0609 www.cox
- associates.com
“The following criteria, which characterize the overall potential of chemicals in the UTDB to show bioactivity in the uterotrophic assay, were used to develop a list of reference chemicals: Active

– a chemical was cl assified as active if two or more independent GL studies classified the chemical as active in the uterotrophic bioassay, regardless of the number of inactive studies. Inactive

– a chemical was classified as inactive if two or more independent GL studies classified the chemical as inactive in the uterotrophic bioassay, and there were no other GL studies reporting the chemical as being active in the uterotrophic bioassay. The resulting list of 42 reference chemicals for uterotrophic bioactivity (29 active, 13 inactive) is provided in Table 2.8.”

COM MENT: Why are the definitions for Active and Inactive not symmetric? How does this asymmetry in definitions affect the reported performance of the methods used?

“The 42 chemicals listed in Table 2.8 (29 active, 13 inactive) were used to assess the performance of the of the ER Pathway model for predicting uterotrophic bi oactivity. A clear separation of reference chemical bioactivity (active/inactive) is apparent when viewed as a function of model score, as shown in Figure 2.12. When using an ER agonist AUC value of >=0.1 as the lower limit for bioactivity, the model score s served as a good surrogate for the qualitative assessment of uterotrophic bioactivity, with a balanced accuracy of 95% (Table 2.9). ... When using this list of reference chemicals (30 Active, 13 inactive), the ER agonist (R1) AUC model demonstrated a balan ced accuracy of 95%, active predictive value of 97%, and inactive predictive value of 92%. Discordant results between the ER agonist (R1) AUC and uterotrophic outcomes were most likely due to lack of bioavailability of the chemical in vivo (“false positive s”) or insufficient concentrations of the chemical, or its metabolite, in the ToxCast assays necessary to approximate blood/tissue concentrations that result in bioactivity (“false negatives”). There appears to be a strong relationship between in vitro AUC values and doses causing in vivo bioactivity, raising the possibility of both qualitative and quantitative utility for the ER (agonist) pathway model.

22Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303

- 388

- 1778; Fax 303

- 388

- 0609 www.cox
- associates.com
for prioritizing and screening chemicals that may have bioactivity in the ER signaling pathway. ”

COMMEN T: It has not been shown whether the high BA (balanced accuracy) truly reflects “the performance of the of the ER Pathway model for predicting uterotrophic bioactivity,” or whether even higher performance would have been achieved without using the ER Path way model, e.g., by applying straight

- forward statistical or machine learning algorithms to the same data. In related work, Cox et al. (2014) noted that simply classifying outputs from inputs using a fully automated machine

- learning approach, without any ER Pathway model or biological interpretation, could actually increase BA to 100%. Here, it is true, and admirable, that the BA is high (although largely because of the many “easy” cases in the left and right tails of the curve in Figure 2.12). But it is not clear that this implies that “the model scores served as a good surrogate for the qualitative assessment of uterotrophic bioactivity” compared to other approaches, such as a st andard statistical or machine

- learning model , or use of just one or a few “best” assays. If EPA has conducted modeling us ing other approaches, these should be presented. If not, EPA should provide a rationale, underpinned by scientific evidence, for ruling out other approaches.

COMMENT: Since the chemicals in Tables 2.2 and 2. 8 overlap, there is potential for selection of the chemicals to artificially increase performance scores such as balanced accuracy . For example, if the reference chemicals tend to be those for which in vivo and in vitro results agree, then a high balanced accuracy might be expected for the reference chemicals, even if the balanced accuracy for other chemicals would be much smaller. This may be challenging for EPA to address if the overlap of the chemicals in these tables reflects the fact that they are amo ng the most

- studied chemicals

.

“ Analyses presented in Section 2 are meant to provide scientific support for the utility of ToxCast models for predicting estrogen receptor bioactivity. ToxCast high throughput assays measure multiple endpoints in the ER signaling pathway using different te chnologies and the network models integrate assay results into an estimate of ER agonist or antagonist bioactivity strength. ER agonist (R1) and antagonist (R2) AUC scores
23Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303

- 388

- 1778; Fax 303

- 388

- 0609 www.cox
- associates.com
compare very well with reported bioactivity of reference chemicals across a range of structures and potencies (Section 2.1.1). The agonist (R1) AUC model also accurately predicted in vivo ER agonist activity for a relatively large set (~150) of chemicals (Section 2.2.2). Together, these analyses provide a high degree of scientific confide nce in the ability of ToxCast ER models to predict bioactivity and demonstrate the utility of using these computational tools to prioritize chemicals in the EDSP Universe. ... Analyses in this section provide a high degree of scientific confidence in the com putational tools presented. ”
COMMENT: A different interpretation of the evidence presented is that the “prediction” problem on which perf ormance is being evaluated is trivially easy: almost any method for making predictions that reflects the information in the in vitro assays, including the AUC model, will perform well when evaluated for a set of chemicals that are mainly either positive on almost all assays (both in vivo and in vitro ) or negative on almost all assays. For this problem, it is easy to pre dict the results of some assays from the results of others. The AUC scores do indeed perform well . But other methods (e.g., using any small subset of assays to “vote” for the results that will be revealed by the rest) might perform as well or better, whil e requiring less data and less effort. From this skeptical perspective , the admittedly good performance of the AUC scores reflects the easy nature of the test problems used to evaluate performance (with many chemicals in the tails of the curves, where res ults are the same for all or almost all assays), rather than providing “a high degree of scientific confidence in the ability of ToxCast ER models to predict bioactivity” on less easy cases outside the evaluation set. Similarly, the claim that “Analyses i n this section provide a high degree of scientific confidence in th e computational tools presented ” is not justified if the computational tools presented simply detract from the performance produced by simpler methods, e.g., by simple voting methods using a subset of assays, or by standard statistical or machine

- learning methods. In this case, the performance results should reflect high confidence that the evaluation task is an easy one on which many methods will perform well, rather than high confidence i n the specific computational tools presented. To show that the computational tools themselves are worthwhile, it would be necessary to show that they out

- perform simpler methods or standard algorithms such as logistic regression modeling or Random Forest.

24Cox Associates. 503 Franklin St., Denver, CO, 80218. Ph 303

- 388

- 1778; Fax 303

- 388

- 0609 www.cox
- associates.com
COMMENT: The terms “scientific confidence” and “high degree of scientific confidence” should be defined and, if possible, quantified. How d oes scientific confidence differ from statistical confidence, or subjective confidence?
“In particular, the AUC calculation tends to down
- weight one receptor if it is overwhelmed by another at higher concentrations, which is what we see for most of these chemicals.”
COMMENT: This suggests that alternatives to the AUC calculation might perform better in this

co ntext

.

25