December 12 - 13, 2013

COMMENTS AT THE EPA BI-MONTHLY LISTENING 

SESSION: RDX, ETBE, and tert-Butanol

On behalf of ACC and the Center for Advancing Risk 

Assessment Science and Policy

Nancy Beck, PhD, DABT

Senior Director, ACC
ACC and the Center for Advancing Risk 

Assessment Science and Policy (ARASP)

ACC:

• Represents the leading companies engaged in the business of 

chemistry.

• Committed to improved environmental, health and safety 

performance through Responsible Care®.

ARASP:

• Coalition of 19 organizations focused on development and 

application of scientifically sound methods for conducting 

chemical assessments.

• Members include chemical specific panels and other trade 

associations. See: http://arasp.americanchemistry.com/
IRIS Enhancements

 July 2013 IRIS enhancements are a constructive step forward.

 Early stakeholder engagement, particularly before a draft is 

developed will help strengthen assessments and move them to 

completion in a more timely manner.

 Planning and scoping will help in understanding parameters that 

will be assessed.

 Complex scientific issues will ideally be discussed earlier in the 

process.

 Identification of critical studies and their summaries should help 

stakeholders understand the direction the agency is heading.

 Appropriate exposure response tables will help provide context.

 The release of evidence tables, while a helpful start, is not 

sufficiently consistent with the IRIS enhancements.

 Further improvements are necessary.
IRIS Process Step 1

 The revised IRIS process documentation includes the following in the 

details about Step 1:

 Begins with planning and scoping, including public meeting on technical 

problem formulation and release of planning and scoping summary.

 Conducts literature search and critical study selection.

 Develops evidence tables that succinctly summarize the critical studies to 

be considered in developing the assessment.

 Publicly releases literature search, literature search strategy, critical study 

selection criteria, evidence tables for critical studies, and exposureresponse figures (which graphically depict responses at different exposure 

levels for studies in evidence tables).

 Convenes public meeting to discuss literature search, evidence tables, 

exposure-response figures, and key issues.

 Only one of these five elements is complete in the RDX, tert-butanol and 

ETBE releases.
Knowing is not enough; we must apply.

Willing is not enough; we must do.

-Johann Wolfgang von Goethe
Evidence Table Releases Fall Short (1)

 No planning and scoping summary is provided.

 There is no understanding of the questions being asked or issues to be 

addressed.

 No context is provided in the released evidence tables.

 EPA’s definition of a systematic review is related only to the literature 

search.

 Document entitled “Systematic review of the ETBE literature” on IRIS 

takes readers to a HERO webpage for identified studies.

 Systematic review must be more than a first step literature search strategy.

 All studies identified in the literature search, that provide an endpoint 

where a change is seen, are deemed ‘critical studies’.

 There are no ‘critical study’ identification criteria.

 Study quality must be an essential element of ‘critical study’

identification.
Evidence Table Releases Fall Short (2)

 Evidence tables present all studies that show a change in an 

endpoint, not necessarily ‘critical studies’.

 There is no discussion of which studies should be treated as ‘critical 

studies’ due to quality, methodology, adversity of effect, or any other 

criteria.

 Studies negative for statistical changes are excluded from evidence tables, 

therefore making them incomplete and misleading.

 While an approach like this may work where there are limited studies 

identified in a literature review, as more complex chemicals are reviewed, 

the EPA approach will be unworkable.

 Burden is on stakeholders to review and comment on every study 

identified in the literature review, not just those in the table.

 Evidence tables focus only on endpoints, ignoring all ‘critical 

studies’ that relate to mode of action (MOA).
Evidence Table Releases Fall Short (3)

 Exposure-response arrays are misleading

 All studies are treated as being of equal quality, when this is 

not the case.

 Exposure-response arrays are limited to positive endpoints, 

ignoring studies with negative endpoint findings and also 

ignoring all mode of action information.
Improving Evidence Tables

1) Release planning and scoping summary along with evidence tables.

2) Don’t confuse a literature search with a systematic review. Clarify 

terminology and use it consistently.

3) Determine, a priori, criteria for ‘critical study’identification. This should 

include not only endpoint specific data, but also mode of action 

information. 

• It should be more specific than including all relevant studies identified in 

the literature review.

4) Critical study criteria should include, at a minimum, a review of study 

quality, methodology, relevance, and adversity of effect (if relevant).

• This review should be released by EPA along with the evidence tables.

5) Evidence tables and exposure-response arrays should be developed for 

mode of action information (see Kushman et. al., for example).

6) Exposure-response arrays should include only those studies of sufficient 

quality (e.g. those meeting critical study criteria) and quality of studies 

should be clear in the arrays.
December 12 - 13, 2013

COMMENTS AT THE EPA BI-MONTHLY LISTENING 

SESSION: RDX

On behalf of ACC and the Center for Advancing Risk 

Assessment Science and Policy

Nancy Beck, PhD, DABT

Senior Director, ACC
RDX General Recommendations

1) Release planning and scoping summary along with evidence tables.

2) Don’t confuse a literature search with a systematic review. Clarify 

terminology and use it consistently.

3) Determine, a priori, criteria for ‘critical study’identification. This should 

include not only endpoint specific data, but also mode of action 

information. 

• It should be more specific than including all relevant studies identified in 

the literature review.

4) Critical study criteria should include, at a minimum, a review of study 

quality, methodology, relevance, and adversity of effect (if relevant).

• This review should be released by EPA along with the evidence tables.

5) Evidence tables and exposure-response arrays should be developed for 

mode of action information (see Kushman et. al., for example).

6) Exposure-response arrays should include only those studies of sufficient 

quality (e.g. those meeting critical study criteria) and quality of studies 

should be clear in the arrays.
RDX Specific Concerns (1) 

 Literature search approach needs improvement

 EPA apparently excluded mechanistic, genotoxicity, and toxicokinetics

data. These were kept as additional data but not primary sources. Thus none 

of these studies are considered ‘critical’.

 However, EPA created tables using genotoxicity studies (table 2-14). Use of 

additional data, in evidence tables, seems inconsistent at this stage and also 

not consistent with EPA description of what is displayed in bold in section 

1.2.1. Is all genotox information in the tables? If not, why not?

 How and when will EPA use all additional studies in the assessment?

 All studies should be evaluated for quality and relevance, consistent with 

developed criteria.

 When EPA searched DTIC, one study was not brought forward due to study 

quality considerations.

 What were the criteria used? How were these criteria applied to all other 

studies in the evidence tables?

 Seems like an inconsistent approach as Table 2-1 contains at least one study 

with no exposure measurements. Shouldn’t this also have been excluded?
RDX Specific Concerns (2) 

 Why are ecosystem effects information maintained for an IRIS 

assessment that is theoretically only evaluating human health?

• Releasing the scoping and planning document may help clarify this.

 Date limits on literature search should be clear and consistent.

• For some searches month, date, and year information is provided; for others 

only a year or month and year. Further transparency is required to improve 

reproducibility.

 References to tables and arrays as including “Key Study Data” 

should be corrected and clarified.

 All mode of action and mechanistic information, which could be critical to 

the assessment is excluded from tables and arrays.

 Other than a literature review, EPA has done no study evaluation to 

determine which studies should be “key studies”. 
RDX Specific Concerns (3) 

 EPA states that tables include references “to identify those most 

pertinent for evaluating human health effects..”

• Mode of action information and mechanistic information MUST be 

considered pertinent.

• Evidence tables for mechanistic information can be created (see, for 

example, Kushman et al. 2013, which includes NCEA authors).

 Endpoints for which no effects are seen in any studies are completely 

excluded from evidence tables.

• Negative information should be considered pertinent and informative to 

IRIS assessments. Should not be excluded from the toxicological review. At 

a minimum, this information can help inform the database uncertainty 

factor. 
RDX Specific Concerns (4) 

 For mortality and neurological endpoints, EPA identified effects 

based on biological significance, rather than statistical significance.

• EPA stated that for studies with a low number of animals, this may 

preclude identifying a statistically significant change. EPA implies that 

statistical significance could be achieved with a higher number of animals, 

however this is not known, nor is it known whether there is any causal 

relationship (see Goodman et al., 2010 in Reg Tox and Pharm).

• EPA is missing any clear definition of what constitutes “biological 

significance.”

• Is this consistent with common practice?

 EPA includes studies with no exposure measurements in the tables

• Its not clear how these data can be ‘critical studies’. Some evaluation of 

study quality and relevance is necessary.
RDX Specific Concerns (5) 

 Evidence tables don’t present all the information.

• Key information about endpoints or timepoints that were evaluated, but 

not statistically significant are missing from the tables.

• Clarity about which changes are statistically significant is missing from 

the tables, although it is noted in some arrays.

• Is this consistent with common practice?

 EPA conducted their own statistical analysis

• Is this appropriate at this stage? Shouldn’t study quality be considered first?

• For other evidence tables, no further analysis was conducted. Why did EPA 

decide to do their own analyses here?
December 12 - 13, 2013

COMMENTS AT THE EPA BI-MONTHLY LISTENING 

SESSION: ETBE AND Tert-BUTANOL

On behalf of ACC and the Center for Advancing Risk 

Assessment Science and Policy

Nancy Beck, PhD, DABT

Senior Director, ACC
ETBE and t-Butanol General Recommendations

1) Release planning and scoping summary along with evidence tables.

2) Don’t confuse a literature search with a systematic review. Clarify 

terminology and use it consistently.

3) Determine, a priori, criteria for ‘critical study’identification. This should 

include not only endpoint specific data, but also mode of action 

information. 

• It should be more specific than including all relevant studies identified in 

the literature review.

4) Critical study criteria should include, at a minimum, a review of study 

quality, methodology, relevance, and adversity of effect (if relevant).

• This review should be released by EPA along with the evidence tables.

5) Evidence tables and exposure-response arrays should be developed for 

mode of action information (see Kushman et. al., for example).

6) Exposure-response arrays should include only those studies of sufficient 

quality (e.g. those meeting critical study criteria) and quality of studies 

should be clear in the arrays.
ETBE and tert-Butanol Specific Concerns (1) 

 Literature search approach needs improvement.

 EPA apparently excluded structure activity information. Why?

 Toxicokinetic data were kept as additional data but not primary 

sources. Thus none of these studies are considered ‘critical’.

 Specific rational for excluding individual studies is not clear. 

More documentation is necessary.

• How and when will EPA use all additional studies in the 

assessment? If a study is ‘pertinent’ but not ‘primary’ what exactly 

does this mean?

 All studies should be evaluated for quality and relevance, 

consistent with necessary criteria.

 Date limits on literature search should be clear and consistent.

 Range of years searched is not clear.
ETBE and tert-butanol Specific Concerns (2) 

 Referrals to tables and arrays as including most pertinent 

references should be corrected and clarified.

 Mode of action and mechanistic information, which could be critical 

to the assessment is excluded from tables and arrays (unless it is in 

an endpoint study).

 Other than a literature review, EPA has done no study evaluation to 

determine which studies should be most important.

 Endpoints for which no effects are seen in any studies are 

unfortunately excluded from evidence tables.

 Negative information should be considered pertinent and 

informative to IRIS assessments. Should not be excluded from the 

toxicological review. At a minimum, this information can help 

inform the database uncertainty factor. 
ETBE and tert-butanol Specific Concerns (3) 

 More clarity on endpoints in tables is needed. 

 It is not know if all endpoints from a positive study are 

included or if only the positive endpoints are included.

 Criteria for extraction of information into the tables is 

necessary.

 Cancer is treated differently in each assessment. Should it 

consistently get its own table and section?

 When percent change is shown, should also show the mean 

and standard deviation for controls.

 Exposure-arrays are misleading.

 A quality review of the data is necessary before providing 

summary statistics and graphics.

 Criteria for reviewing data should be developed and used 

before presenting information graphically.
Thank You!

 EPA has taken a strong first step. However improvements are needed.

 Consistency with intent of IRIS process enhancements will go a long 

way towards improving the evidence tables.

 Most importantly, EPA must conduct a review of the quality and 

relevance of studies before moving them forward as ‘critical studies’

for an IRIS assessment.

 Without changes and improvements, the utility of evidence tables, 

following the current structure, may not help sufficiently speed the 

finalization of IRIS assessments. 

 Small changes, such as those noted in our general 

recommendations slide will lead to much improvement.
Knowing is not enough; we must apply.

Willing is not enough; we must do.

-Johann Wolfgang von Goethe