OAR Box 1112 

Prepped by Ryan Dugan 

Document Number: 

27) ll-C-1e 

Docket Number: 

A-83-22 

Printed 4/16/2008 9:12:07AM Sheet 27 of 97 

P.1 
' ARGONNE NATIONAL LABORATORY k-c-t 

9700 SouTh CASS AVENUE, ARGPWE, Ittois 60459 "felEphoNE 512/972- 8 4 3 0 

P.2 

September 16, 1982 

Mr. Thomas R. McCurdy 

U.S. Environmental Protection Agency 

Office of Air Quality Planning and Standards 

Research Triangle Park, NC 27791 

Dear Tom: 

Argonne National Laboratory is pleased to have the opportunity to 

review the two risk assessment methodologies developed by SRI International 

(SRI) and Woodard-Clyde Consultants (WCC). The funding of these two 

developments by the Environmental Protection Agency has provided a unique and 

noteworthy set of studies in the risk analysis field. Since the guidelines 

were sufficiently non-restrictive, each was able to develop a solution that 

was essentially Independent of the other. It has been interesting to study 

and compare these reports. 

This letter is structured around the evaulation criteria that EPA 

specified in the scope of work with ANL. However, as you will see, we have 

added another notion which we feel is essential in judging the suitability of 

these risk assessment approaches to the regulatory environment in which EPA 

must operate. In considering the strengths and weaknesses of these 

methodologies, we have tried to put ourselves in the position of the EPA. 

Thus, we used a needs/value structure for EPA which is based on our 

understanding of EPA's role in the standards setting process. 

In a few words, our opinion is that the methodology developed by WCC is 

better suited than the SRI methodology for further development and application 

to the analysis of the risks of alternative national ambient air quality 

standards for criteria pollutants by the Environmental Protection Agency. The 

rational for this conclusion is explained in the remainder of this letter. 

For many reasons it was difficult to make a definitive choice between 

the two contractors. Both are highly qualified, broadly experienced, and do 

good work. We have suppressed what we know generally about both groups in 

doing this evaluation and have tried to focus on the contents of the 

documents. In looking at performance based on their final reports, it appears 

to us that the WCC methodology best meets the requirements of a formalized 

assessment of the health risks due to various criteria pollutants. 

Furthermore, it appears that the WCC group is more aware of and more 

responsive to the standards setting environment in which EPA must operate.* 

Discussion of each of the six categories of evaluation criteria 

follows. 

US. DEppM-rnv-ENT of ENERqy T k LKiKERsiTy of OHJCAQO ARCANE UNIVERSITIES ASSOCIATION 
P.3 

•T. R. McCurdy Page 2 of 7 Sept. 16, 1982 

Openness and Understandability 

The first category of the EPA-supplied criteria is openness and 

understandability. It is important that the methodology be open to review by 

interested parties, that methodology assumptions be explicit, and that outputs 

be understandable by decisionmakers and the public. These are very important 

considerations, perhaps the most important of the six categories supplied by 

EPA. If results are to be accepted by the scientific, environmental health, 

industry, and public communities, they must be reproducible. While neither 

SRI nor WCC provides enough information in their final reports to allow a 

third party to reproduce the results, the WCC approach leaves fewer unanswered 

questions. The principal reason for this is the fact that SRI has introduced 

another layer of complexity to the problem by developing sub-models in order 

to determine the eventual dose-response relationships that are needed. 

ANL agrees that this is the right solution in an ideal world. However, 

it requires more information and experimentation than that required for 

directly quantifying dose-response curves. This also complicates, we think, 

the standards setting process for EPA. While the modeling approach does 

theoretically make the entire process more explicit (a plus for SRI), the 

price is probably prohibitive for a number of reasons. It is not necessarily 

true that additional modeling will make the results "better." When more 

variables are introduced to a model, the possiblity of cross-correlations 

increases. Since there is little data on which to base direct measurement of 

the dose-repsonse curves, these data most certainly would not be very useful 

for investigating cross-correlations. As a result, independence between 

variables would most likely have to be assumed and this usually leads to a 

larger spread in the calculated results. Such a spread could be overly and 

unncessarily cautious. 

SRI developed a number of "sub-models". For example, the coronary 

obstructive pulmonary disease (COPD) model contained a simulation model that 

was only briefly alluded to in the report. Because there is no description of 

this model, it is impossible to reproduce the results of this section. In 

addition, the output of this model was expressed in terms of "life 

shortening". This is a new output measure for the standards setting 

process. That may pose problems. More serious it seems, is the fact that the 

output measures EPA needs (bench-mark and head count risks) do not fall out of 

such a model. 

The WCC approach, which involved direct assessment of the dose-response 

curves, is much easier to follow. There is a very serious question as to 

whether or not it is possible to accurately measure these dose-response curves 

directly. SRI reported that they turned to a modeling approach because the 

health experts they Interviewed were not able to answer questions aimed at 

directly assessing dose-response curves. Since WCC was successful at directly 

measuring dose-response curves, the important question is "How good were these 

assessments?" 
P.4 

.T. R. McCurdy Page 3 of 7 Sept. 16, 1982 

Fortunately there was overlap in the output measures used by SRI and 

WCC, so we can get some indication of the acceptibility of the direct 

assessments. For example, both produced national estimates of excessive 

deaths due to ASHD and excessive angina attacks. After translating the 

results to the same basis (that is, relative to the strictest standard), and 

recognizing that there are differences in other assumptions leading up to 

these results, it appears that: 

• The expected numbers of excess deaths calculated by SRI are 

generally a little less (by a factor of about 0.25) than the lowest 

estimates produced by WCC. 

• WCC's median estimates are about three times higher than SRI's mean 

estimates for national ASHD deaths. 

• SRI estimates of the numbers of excess angina attacks are about the 

same as the highest estimates made by WCC. 

• WCC's median estimates are about 40% lower than SRI's mean estimates 

for excess angina attacks. 

These observations are based on data in SRI's Table 9.2 and WCC's Table 7.17. 

The Los Angeles results (SRI's Tables 5.4 and 5.5, WCC's Table 7.1) 

have fewer sources for differences. The conclusions are essentially the same 

as for the national data: 

• SRI estimates of deaths are generally a little lower (about 10% 

lower) than WCC's lowest estimates. 

• WCC median estimates of deaths are about two times higher than SRI's 

mean estimates. 

• SRI estimates of excess angina attacks are generally about the same 

as the highest estimates made by WCC. 

• WCC median estimates of excess angina attacks are about 45% lower 

than SRI estimates. 

Although it would be difficult to establish a statistical test of the 

hypothsis that these estimates are "the same", it appears that they are of the 

same "order of magnitude". Accepting this, one may reasonably conclude that 

the WCC estimates of the dose-response curves are "acceptable". 

Efficiency -

The second category of evaluation criteria is efficiency. The concerns 

here are that the method: 
P.5 

T. R. McCurdy Page 4 of 7 Sept. 16, 1982 

• make appropriate use of logic, mathematics, and mathematical models; 

• make maximum feasible use of relevant information and expertise; and 

• comprehensively deal with all of the uncertainties 

The WCC method better addresses these issues, although there is a notable 

tradeoff between the first and third points. The key word here 

is"appropriate" because this is a main point which separates the SRI approach 

from the WCC approach. As discussed earlier, the SRI modeling approach is 

more rigorous but probably not appropriate for use in the current standards 

setting process. This approach introduces considerations which are not 

addressed in the CO criteria document. One example is the use of the double 

product instead of "reduced time to onset of pain" to predict angina attacks. 

On the other hand, this more rigorous modeling approach does make 

explicit more of the underlying, uncertain physiological factors that relate 

to health effects. 

If the data existed to support it, and the standards setting process 

were equipped to handle it, then a more extensive modeling approach would be 

appropriate. As things stand, it seems likely that the SRI method would place 

unrealistically heavy demands on available information and expertise. 

Arbitrariness 

The intent of this evaluation category is that the methodologies 

minimize arbitrariness and also consider diverse scientific opinions. While 

there is nothing inherent in either methodology which violates these desirable 

features, the SRI report has many more instances where assumptions were made 

in order to make a calculation, assumptions that were not explained or 

defended. For example, the probability tree in Fig. 7.5 used for the COPD 

model contains probabilities that are not explained; the same is true in 

Fig. 5.5 which is a probability tree for probablistic analysis of the angina 

model; also in the probablistic phase analysis for angina is the assumption 

that MOC is lognormally distributed. If there was a rational for these 

choices, it should have been mentioned. 

WCC did make similar assumptions. The lognormal distribution is at the 

core of their dose-response curve relating the percentage of additional angina 

attacks to angina patients vs. COHb level (Fig. 6.1). However, WCC gave some 

arguments (statistical goodness of fit) in the text for the choice. 

The same figure is also illustrative of WCC's more obvious use of 

diverse expert opinion. The raw data from Table 6.1 were used to derive, the 

dose-response curves, which graphically show that uncertainty exists in the 

response even when the dose is known. The explicit use of diverse expert 

opinion is carried along through the analysis all the way to the output tables 

(7.2 - 7.20), where low, median, and high risk estimates are listed. 
P.6 

T. R. McCurdy Page 5 of 7 Sept. 16, 1982 

Generality 

The generality category requires that the method: 

• handle different kinds of health effects; 

• work for all NAAQS pollutants; 

• work for standards with various averaging times; and 

• handle different types of information (e.g., epidemiological, 

clinical, toxicological evidence). 

ANL knows of no reason why either method should be deficient or 

necessarily better than the other on any of these points. Both methods are 

really approaches or "schools of thought" on how to model and assess 

uncertainty. As such, each is sufficiently adaptable to accomodate a wide 

variety of problems, even in the highly focused arena of setting standards for 

NAAQS pollutants. The most serious weakness seems to be SRI's detailed 

modeling approach, which would most likely introduce considerations which will 

not be addressed in the other criteria documents. But again, this is a point 

raised earlier and should not be belabored. 

Flexibility 

The fifth evaluation category is flexibility. Issues relevant to this 

category include the ability of a methodology to incorporate new 

methodological improvements and new information. Comments on this category 

are the same as those from the previous category concerning generality. Since 

both methods are based on Bayesian statistics, both would incorporate new 

information by updating prior estimates conditional on the new Information. 

Bayes' rule is used to perform the mathematics in such a situation. New 

considerations would be accomodated (in theory) by each methodology by simply 

adding a new variable and adjusting the remainder of the model as necessary. 

Neutrality 

The last category is neutrality. The concerns here are vital 

considering the regulatory environment in which EPA must operate. It is 

important that the risk estimation procedure minimize the incorporation of 

social values and minimize bias'. Both methodologies incorporate assessment 

protocols which do enchance the likelihood of obtaining estimates which are 

value free and unbiased. 

It is almost unavoidable that value judgments are a major factor in 

computing a composite risk estimate. This occurred in Chapter 10 of SRI's 

report which developed an "equivalent days of restricted activities" (EDRA) 

metric for ranking decision alternatives. This chapter should be viewed only 

as an example of what could be done to aggregrate the impacts of many 
P.7 

T. R. McCurdy Page 6 of 7 Sept. 16, 1982 

different health effects; but it is also important to note that the assignment 

of such measures is a value judgment. For example, automobile deaths are 

"more important" than fetal deaths because the EDRAs are 30 years and 7.5 

years respectively. A step in the direction of a composite risk estimate 

approaches the notion of "risk evaluation". If such a measure and assignments 

were actually used, they would have to be chosen with great and careful 

thought. 

A suggestion for future work Is that assessment protocols be varied 

when assessing an expert for a number of probability densities. Research has 

shown that the phrasing and order of questions can influence responses. While 

consistency checks should solve any such problems, if they arise, it is 

possible to design assessment protocols which explicitly account for these 

subtleties. If the asssessment procedure always begins by assessing the 50% 

fractile on the cumulative probabilty curve, a systematic bias called 

"anchoring" may be introduced. The end result is usually that the assessed 

distribution is too narrow, peaked around the mode. It may be helpful to 

begin some assessments at the 95% fractile, some at the 5% fractile, some at 

the 50% fractile, etc. Randomization of the starting point would enhance 

objectivity of the procedure. Then if there are enough assessments, the 

presence of undesirable biases may be detectable. 

Statement of Work 

In addition to these six categories in evaluation criteria, it is also 

appropriate to consider one important issue from the statement of work which 

EPA specified to SRI and WCC. 

The contractors were given a large degree of freedom in deciding what 

output measures of human health risks were appropriate, with some guidance as 

to the minimum needs of the EPA. These minimum needs are critical since the 

measures are specifically used in the standards setting process. These 

measures are: 

• The probability that the number of persons experiencing and the 

expected number of incidences of, each adverse health effect equals 

or exceeds a given value for a specified time period. 

• The expected number of persons experiencing, and the expected number 

of incidences of, each adverse health effect in a given time period. 

• The probability that nZ of the most sensitive group, if exposed to 

air quality just attaining a given standard, will experience an 

adverse health effect in a specified period of time. . 

Only WCC provided estimates for each of these output measures. SRI provided 

some of these plus some new ones of their own creation. Even though the new 

ones are important measures of the risks to human health, the standards 

setting process is not set up to deal with them. Most important is the fact 

that SRI did not meet the minimum needs of the EPA. It also seemed that SRI 

had some difficulty with the notion of who comprises a sensitive group. 
P.8 

T. R. McCurdy Page 7 of 7 Sept. 16, 1982 

Summary 

Many words have been used In the preceding pages to evaluate the SRI 

and WCC risks assessment methodologies. The main points are the following: 

T • WCC is judged to be better than SRI in three of the six evaluation 

categories considered, and they are equal in the other three. 

• The most important difference, which ripples through the entire 

evaluation, is philosophical; SRI's approach is based on detailed 

modeling which ultimately complicates the standard setting process, 

while WCC used direct assessments of critical dose-response 

relationships. 

• WCC better addressed the tasks specified in the statement of work, 

specifically providing the benchmark and head count risk estimates 

needed by EPA. 

The final point is that the EPA should be complimented for sponsoring 

this type of research. It is very important that the best information 

available be used in the standards setting process* Analysis of the risks to 

human health is fraught with uncertainty and incomplete data. A subjective 

view of probability is the best way to tie what is known and what is unknown 

together so that something rational can be said in support of critical 

decisions which must be made regarding the standards for criteria pollutants. 

Respectfully, 

/LM;$>(Mf^ 

Ronald G. Whitfield, Ph.D 

Decision and System Sciences Section 

Energy and Environmental Systems Division 

cc: T. Surles 

T. Wolsko 

'"RGW :sm