From: Lydia Wegman/RTP/USEPA/US

To: "Frey, Nathan J." <Nathan_J._Frey@omb.eop.gov>, "Lee, Amanda I." 

<Amanda_I._Lee@omb.eop.gov>

Cc: Joseph-J Dougherty/DC/USEPA/US@EPA, Erika Sasser/RTP/USEPA/US@EPA, Vicki 

Sandiford/RTP/USEPA/US@EPA, John Hannon/DC/USEPA/US@EPA

Date: 12/12/2012 05:08 PM

Subject: Secondary RTC-- Preference Studies section

Nathan and Amanda, 

Attached is the section of our draft Response to Comments document that address the visibility 

preference studies. 
Interagency Review Draft –Do Note Quote or Cite December 12, 2012

PM NAAQS Response to Comments—Section IV.5 (DRAFT)

2

5. Specific Comments on Visibility Preference Studies

The evidence relied upon by the Administrator in determining an appropriate degree of 

public welfare protection from visibility impairment is predominately derived from a small 

number of urban-focused visibility preference studies. As a result, many comments were 

received on the adequacy and appropriateness of this limited dataset serving as a basis for the 

Administrator’s judgments regarding what is requisite national protection for visibility 

impairment. Both CASAC and the EPA staff, as well as a few commenters, concluded that while 

limited, these studies did form a sufficient basis on which the Administrator could make a 

decision. A majority of commenters, however, expressed the view that the existing preference 

studies provide an insufficient basis for selection by the Administrator of an appropriate level of 

public welfare visibility protection for a national standard. These commenters highlighted a 

number of limitations and uncertainties associated with these studies as support for this view. 

Many of the important and fundamental comments are also discussed in the final rule, with 

further details provided below. As in the final rule, this section organizes and discusses the 

preference study comments under four broad topic areas, including: (a) limitations and 

uncertainties associated with the visibility preference studies; (b) preference study methods and 

design; (c) use of preference study results for determining adversity; (d) the appropriateness of

using regionally varying preference study results to select a single level for a national standard. 

a. Comments on Preference Study Limitations and Uncertainties

A large and diverse number of limitations and uncertainties associated with the visibility 

preference studies has been identified and discussed in the public comments. Many of these same 

limitations and uncertainties were also identified and discussed by the EPA in the various 

documents developed throughout this review such as the Integrated Science Assessment, 

Visibility Assessment, and Policy Assessment, as well as in the proposed rule (see section VI.b.2 

beginning at 77 FR 38973, 38990) and in the final rule (section VI.C.1.e.i). 

(1) Comment: Most commenters expressed concern over the small number of preference 

studies that are available in this review. In particular some note that these preference 

studies cover just four locations, only three of which occur in the U.S., that the two 

studies conducted in Washington, DC were pilot studies, not full preference studies, 

and/or that three of the preference studies were conducted in the West, while only one 

was conducted in the East, thus providing only limited geographic coverage. Typically 

these same commenters also noted the small number of survey participants, which totaled 

852 across the four urban areas, which they assert are too few and unrepresentative 

nationally. These commenters thus concluded that there is insufficient information, both 

geographically and demographically, upon which to select a national level of a visibility 

index for purposes of visibility protection. 

Response: The EPA notes that it is well aware of the limited nature of the information, 

which it has described in great detail in the Integrated Science Assessment, Visibility 

Assessment, and Policy Assessment, as well as in section VI.B.2 of the proposed rule 

(i.e., beginning at 77 FR 38973, 38990). The EPA further notes however that limited 

information does not preclude the Administrator from making judgments based on the 

best available science, taking into account the existing uncertainties and limitations 
Interagency Review Draft –Do Note Quote or Cite December 12, 2012

PM NAAQS Response to Comments—Section IV.5 (DRAFT)

3

associated with that available science. Thus, in reaching judgments based on the science, 

the Administrator appropriately weighs the associated uncertainties. The CASAC 

supported this view and concluded that the available information provided a sufficient 

basis on which the Administrator could form a judgment about requisite PM-related 

public welfare visibility protection. Specifically, CASAC stated “[t]he 20-30 deciview 

range of levels chosen by EPA staff as ‘Candidate Protection Levels’ is adequately 

supported by the evidence presented. (Samet, 2010b, p. iii). As discussed in the proposed

rule (77 FR 38990), the Administrator recognized and explicitly took into account the 

uncertainties and limitations in the science in determining an appropriate degree of 

protection when she proposed a level at the upper end of the recommended range. 

(2) Comment: Some commenters also made the point that the EPA relied on much of this 

same evidence to reach the conclusion in 2006 that the information was too limited to 

allow selection of a national standard. For example, API stated: 

[T]he bulk of the VAQ preference studies were available during the previous PM 

NAAQS review and were considered by the Agency in its establishment of the 

2006 PM secondary NAAQS.... The Proposed Rule does not mention this fact 

and does not explain why many of these same studies now compel EPA to 

propose this new secondary NAAQS....The Proposed Rule notes in passing that,

since the last review of the PM NAAQS, ‘limited information that has become 

available regarding the characterization of public preferences in urban areas has 

provided some new perspectives on the usefulness of this information in 

informing the selection of target levels of urban visibility protection.’ 77 Fed. 

Reg. at 38969/2. It is a serious oversight that the Proposed Rule makes no attempt 

to explain what that information is or how it affects the interpretation of the VAQ 

preference studies. This ‘limited information’ is an apparent reference to 

information provided by Dr. Anne Smith. (API, p. 37)

Response: The EPA agrees that 3 of the four study areas examined in this review were 

available in the previous review (see discussion in section VI.A.1 of the proposal which 

specifically identifies the studies from Denver, Phoenix and British Columbia (77 FR 

38967/2) as being considered in the last review). However, the EPA disagrees that it has 

not explained what information, both preexisting and new, it relied upon in this review to 

inform its views and provide the basis for its proposal, beginning in section VI.A.3 of the 

proposed rule (77 FR 38969). In particular, in addition to the substantial recent PM urban 

air quality information and analyses new to this review, there are three other noteworthy 

sources of information that have specifically “provided some new perspectives on the 

usefulness of” the preference studies “in informing the selection of target levels of urban 

visibility protection.” They include: (1) results from additional urban visibility preference 

study experiments conducted for Washington, DC by Smith and Howell (2009) which 

added to the preference data for that location, provided proof that the 2001 study could be 

replicated, and shed light on the role of location in preference responses (77 FR 38974, 
Interagency Review Draft –Do Note Quote or Cite December 12, 2012

PM NAAQS Response to Comments—Section IV.5 (DRAFT)

4

38976)1; (2) an extensive review and reanalysis of the urban visibility public preference 

studies from the four urban areas, including the newly available Smith and Howell (2009) 

experiments which examined the similarities and differences between the studies and 

evaluated the potential significance of those differences on the study results (77 FR 

38893, 389975-76)2; and (3) additional analyses conducted in response to questions 

raised by the CASAC, most importantly a logit analysis presented in the Visibility 

Assessment (see Appendix J and discussion in Chapter 2) which showed that each city’s 

responses represented unique and statistically different curves (77 FR 38975-76). 3Taken 

together, these sources contributed to an increase in the EPA’s knowledge and 

understanding of the robustness of each survey study’s results, the appropriateness of 

comparing each study’s results to the others, and the key uncertainties relevant to data 

interpretation. In addition, in the last review the decision to not adopt a distinct secondary 

standard was remanded as contrary to law and failing to provide a reasoned explanation 

for the decision. As such it is not appropriate for purposes of comparison with the 

Administrator's judgment and reasoning in this review.

(3) Comment: Other commenters stated support for using the preference studies, concluding 

they provide an adequate basis, in spite of their limited nature. In particular, AMC et al. 

state:

We believe that these studies provide sufficient results to inform setting a national 

visibility standard. While the number of studies is small, they do incorporate 

spatial variation and, in the case of Denver and Phoenix, varied populations.... 

EPA should have confidence, rather than uncertainty, in the fact that these studies 

used different methods and respondents and yield a range of 20 – 24 dv, with one 

outlier of 29. (pp. 6-7)

Response: The EPA agrees that these studies provide a limited but sufficient basis to 

inform the Administrator’s judgments regarding an appropriate level of public welfare 

visibility protection (see also response to (1) above). However, the EPA does not agree 

that the Washington, D.C. results represent an outlier but instead represent a reasonable 

50% acceptability value for that type of urban scene and thus are appropriately included 

in the range identified for the Administrator to consider. 

(1) Comment: A number of other technical features of the studies were identified by 

commenters which they assert limit the usefulness of these studies in representing real 

world urban views and in eliciting appropriate preferences for urban visibility. For 

 

1

Smith, AE; Howell S (2009). An assessment of the robustness of visual air quality preference 

study results. CRA International. Washington, DC. 

http://yosemite.epa.gov/sab/sabproduct.nsf/B55911DF9796E5E385257592006FB737/$File/CRA

+VAQ+Pref+Robustness+Study+3+30+09+final.pdf.

2

Stratus Consulting, 2009. Review of Urban Visibility Public Preference Studies: Final Report. 

EPA-HQ-OAR-2007-0492-0042.

3 Deck L., Lawson M. 2010. Statistical Analysis of Existing Urban Visibility Preference Studies. 

EPA-HQ-OAR-2007-0492-0089.
Interagency Review Draft –Do Note Quote or Cite December 12, 2012

PM NAAQS Response to Comments—Section IV.5 (DRAFT)

5

example, some commenters state that the assumption of light extinction at 550 nm is 

unsupported, technically flawed and invalid for a number of reasons, including that it 

does not account for the mix of pollutants, which varies substantially within and between 

urban areas, or how changes in that mix affect light extinction at different wavelengths of 

light, and it misses the role of sky discoloration from NO2 and carbonaceous particles 

(i.e., Brown Cloud phenomenon) which can be significant in urban areas and influence 

human perception of urban visibility impairment. Other commenters note that pollutant 

concentrations in urban areas are not uniform in all directions over the length of a line of 

sight and that visibility depends upon the site-path integrated concentration of fine PM. 

Studies which used photographic data capture the integrated site path. They further state 

that momentary glances at 2-dimentional pictures cannot represent the actual way that 

visibility is perceived outdoors. 

Response: First, the EPA disagrees that its choice of measuring light extinction at 550 nm 

wavelength is ‘unsupported, technically flawed and invalid’. The EPA’s Integrated 

Science Assessment makes clear that 550 nm represents the “most sensitive portion of the 

spectrum for human vision” and has thus traditionally “been used to characterize light 

extinction and its components” (U.S. EPA, 2009a, p. 9-2). The ISA further states that 

“the convention for visibility monitoring purposes is to make measurement at or near 550 

nm, which is the wavelength of maximum eye response” (U.S. EPA, 2009a, p. 9-8). The 

EPA therefore concludes that it remains logical to select a wavelength of light that is 

most relevant to human observers when evaluating impacts to the public welfare. 

Regarding the comment that 550 nm misses the impacts of NO2 on sky discoloration, the 

EPA notes that this PM NAAQS review is intended to only address visibility impairment 

caused by particles present in the ambient air. Thus, while it is true that sky discoloration 

from NO2 results from its ability to absorb more light in the short wavelength blue 

portion of the spectrum giving a yellow or brown appearance to the pollution layer or 

plume, improvement in the Brown Cloud Phenomenon is not the goal of this NAAQS 

review and would be expected to occur only indirectly from a secondary PM2.5 NAAQS 

for visibility in as much as PM2.5 controls also reduced co-emitted NO2. The EPA further 

notes, however, that not every urban or other area suffers from the Brown Cloud 

Phenomenon. At background concentrations NO2 absorption is generally less than five 

percent of the light scattering by clean air (Rayleigh scattering), making it 

imperceptible.” (ISA, p. 9-5) Regarding the comment that measuring light extinction

using 550 nm “does not account for the mix of pollutants, which varies substantially 

within and between urban areas”, the EPA disagrees, noting that the IMPROVE 

algorithm used to calculate light extinction was designed to specifically take into account 

the variability in the mix of pollutants from place to place and to measure the “photopicweighted absorption efficiency at the 550 nm wavelength” in its calculations of PMrelated light extinction. Thus, given the above, the EPA believes it is still appropriate to 

use the 550 nm wavelength in order to measure the maximum impact of ambient PM on 

light extinction that is perceptible to the human observer.

Second, with respect to the other technical issues raised above, the EPA agrees that 

photographic and WinHaze images cannot duplicate the outdoor, three dimensional 

viewing environment. This limitation of the preference study design does introduce some 
Interagency Review Draft –Do Note Quote or Cite December 12, 2012

PM NAAQS Response to Comments—Section IV.5 (DRAFT)

6

degree of uncertainty into the results. However, the EPA does not believe that this 

uncertainty significantly limits the usefulness of the results. People take pictures of 

aesthetically pleasing vistas so that they can look at them repeatedly, even though they 

are not exact replicas of their initial outdoor viewing experience. This shows that they 

appreciate the photographic view of the scene, commensurate with their appreciation with 

the scene itself. Thus, the EPA believes photos or digital images can be useful surrogates 

for the purpose of eliciting public preferences. Regarding the commenters other concern 

that pollutant concentrations in urban areas are not uniform in all directions over the 

length of a line of sight and that visibility depends upon the site-path integrated 

concentration of fine PM, the EPA notes that both the photos used in two of the 

preference study areas (Denver and British Columbia) and the WinHaze images used in 

two others (Phoenix and Washington, DC) depict visibility impairment over an integrated 

sight path. The difference between the two methods is that a photo can capture the nonuniformity across the sight path while the WinHaze views present a uniform haze 

integrated across the sight path. Photos of the Denver scene therefore would also have 

included any layered haze known as the brown cloud that was present at the time that fell 

within the integrated site path. Despite these variables, the ordering of the results of 

these studies appear to be explainable based predominantly on scene characteristics – i.e. 

people found impairment of the views with the most sensitive scene characteristics 

(Denver and Phoenix) more unacceptable at lower deciview values than those with less 

sensitive scenes (British Columbia and Washington, DC) regardless of the image 

depiction methods used. Thus, the EPA concludes that the concerns over study methods 

and designs identified by commenters are not substantial, and that the preference study 

results are still sufficiently informative to allow selection of candidate protection levels.

a. Comments on Preference Study Methods and Design

(1) Comment: Many commenters asserted that the methodologies used in the preference 

studies are fundamentally flawed. Most of these same commenters also pointed to an 

assessment of the preference study methodology conducted by Smith and Howell (2009)

of Charles River Associates (CRA) as the basis for their views. These commenters 

criticized the EPA for not acknowledging that (in their view) the Smith and Howell 

results undermined the validity of the earlier preference studies. For example:

Smith and Howell (2009) show that VAQ preference study outcomes are 

malleable and depend entirely on the design of the study. Accordingly, such 

studies do not identify any meaningful threshold of acceptable visibility 

conditions. Despite Smith and Howell’s conclusions, EPA continues to assert that 

the VAQ preference studies can be used to identify minimally acceptable 

visibility conditions even though the Agency has never provided any valid 

scientific basis for discounting the Smith and Howell (2009) results. (API, p. 38)

Well-controlled preference studies discussed by Anne Smith of Charles River 

Associates at the March 2010 CASAC meeting demonstrated that the judgment of 

panel members was affected by the order in which photographs were presented 
Interagency Review Draft –Do Note Quote or Cite December 12, 2012

PM NAAQS Response to Comments—Section IV.5 (DRAFT)

7

and tendency to identify the middle of the range of visibility degredation as a 

threshold of acceptability. This points to a potential flaw in these studies and that 

artifacts caused by these tendencies may have influenced study results. 

Dismissing these inherent flaws in the existing preference studies and then using 

these studies to set a secondary NAAQS is arbitrary and capricious. (API, 

attachment 14, p. 12) 

Response: As noted above, the EPA is aware of the issues raised regarding the 

uncertainties and limitations associated with the preference studies, including those 

associated with study methods and design and specifically solicited comment on how 

these uncertainties and limitations should be considered (see 77 FR 38990). However, as 

is explained in the following discussion, the EPA strongly disagrees that the analysis by 

Smith and Howell (2009) supports the commenters’ conclusion that the preference study 

methodologies and design were fundamentally flawed and instead asserts that the Smith 

and Howell (2009) experiments increased the EPA’s knowledge and understanding of the 

findings of the 2001 Washington, DC focus group pilot study (Abt, 2001) in several 

important ways. Smith and Howell (2009) conducted a series of three sets of experiments 

based on the 2001 Washington, DC, focus group pilot study. The first set of experiments 

(Test 1) was designed to replicate the 2001 focus group to explore whether the 2001 

study findings could be reproduced. Tests 2 and 3 varied the range and distribution of 

visibility conditions to examine whether such changes would affect the results. As 

explained below, the EPA asserts that some aspects of the approach used by Smith and 

Howell (2009) in Tests 2 and 3 are flawed.

It is important to note as an initial matter, that while Smith and Howell (2009) claims to 

have replicated the Abt 2001 protocol, there were three distinct differences in the 

methods used by Smith and Howell (2009) and the methods used in the Washington, DC 

2001 pilot study. First, the images of alternative levels of visibility degradation in 

Washington, DC were prepared and presented differently. While both studies used the 

same original photograph of Washington, DC, the 2001 images showing different levels 

of visibility degredation were prepared by John Molenar of Air Resource Specialists, Inc. 

(ARS) using the mainframe computer version of WinHaze available at that time. The 

2009 Smith and Howell experiments, on the other hand, used images prepared in a more 

simple manner using the MS-Windows©-based version of WinHaze (Ver. 2.9.0) available 

by that time from ARS. Second, the 2001 pilot study used high quality photographic 

transparencies (“color slides”) prepared by ARS, which were presented on a large format, 

high quality projection screen. The entire group of participants was seated at the viewing 

angle and distance from the projection screen in a darkened room to provide the optimal 

viewing conditions recommended by John Molenar. The 2009 study used images 

presented to individuals on a large desktop computer monitor. Third, the method used to 

recruit participants differed between the 2001 and 2009 studies. While a group of only 9 

people can never be a statistically valid representative set of any population, the 2001 

focus group participants were recruited by a professional focus group facilitation firm to 

reflect aspects of the demographic distribution of the Washington, DC metropolitan area 

adult population. The recruitment screening method was designed to obtain 9 adults 

resembling major aspects of the demographic gender, age, residence and income 
Interagency Review Draft –Do Note Quote or Cite December 12, 2012

PM NAAQS Response to Comments—Section IV.5 (DRAFT)

8

distribution of the Washington, DC area. At least two of the 2001 participants were in 

each of the following groups: a) Income: at least two people with household incomes 

under $25k, two with incomes between $25k and $60k, and two with incomes above 

$60k; b) Age: at least two between the ages 18 and 25, two between 26 and 55, and two 

over 56; c) Race/Ethnicity: at least two African Americans and two Hispanics; d) 

Residence location: at least two living in each of Washington, DC, Maryland and 

Virginia. In contrast, the Smith and Howell (2009) study did not use a structured 

recruitments process. Due to the schedule of the Smith and Howell (2009) study, all study 

participants were employees of CRA, Inc. (where both Smith and Howell are also 

employed). One variant of Test 1 was conducted with participants who work at the CRA 

Washington, DC office (and who presumably live in the Washington, DC area). The 

other variant of Test 1 was conducted with participants from the CRA Houston office 

(and likely live in the Houston area). The same set of images of Washington, DC was 

shown to both the CRA Washington and Houston participants.

In spite of these differences in study methods, the Test 1 results (Smith and Howell, 

2009, p.11) found a “...remarkable degree of replication, given the differences in the 

samples, and the relatively small sample size of the Abt pilot (nine people).” 

Furthermore, the preferences for Washington, DC visibility provided by the Houston 

participants were effectively the same as both the preferences of the CRA Washington 

office participants and the Abt 2001 participants.

The EPA identified a number of important implications from the similarity between the 

Test 1 (2009) and 2001 study results. 

1) The change in the presentation methods (i.e., from slide projection to 

computer monitor, and mainframe WinHaze images versus the simpler Windows 

WinHaze method) shows no evidence of impacting reported preferences. 

2) The replicability of the 50% acceptable criteria results supports a 

hypothesis that the 2001 results, although based on a small sample of 9, may be a 

fair representation of the preferences for the general population. 

3) The similarity of the Test 1 Houston results to both the Test 1 Washington 

results and the 2001 Washington study suggest that an individual’s preferences 

for visibility in one location (i.e., Washington, DC) may not depend on whether 

they live in that location or in another.

As a result of the similarity between the results of these two DC studies and the above 

implications, the EPA concluded that the Test 1 experiment could legitimately be 

combined with the 2001 study to effectively more than triple the responses to the original 

Washington, DC urban visibility preference study, thus increasing the available 

information from 9 (in the 2001 study) to 31 (with the addition of both the Washington, 

DC and Houston CRA participants). Smith and Howell’s Test 1 thus instigated a shift in 

the EPA’s thinking regarding the usefulness of the available Washington, DC preference 

study data.
Interagency Review Draft –Do Note Quote or Cite December 12, 2012

PM NAAQS Response to Comments—Section IV.5 (DRAFT)

9

The Smith and Howell (2009) Tests 2 and 3 also provided important information relevant 

to understanding the results of visibility preference studies. In Test 2 and 3 the range and 

distribution of visibility conditions shown to participants were substantially different than 

the range and distribution shown in Test 1 and the 2001 Washington, DC study. Test 2 

removed from the range of Test 1 slides those that showed the 11 worst visibility 

conditions (all images above 27.1 dv). Test 3 changed the visibility conditions presented 

in two ways: first, the number of clean or cleaner images were reduced substantially, 

dropping all images in the lower third of the distribution, and adding two new images of 

substantially worse visibility (42 and 45 dv), which were substantially above the previous 

worst scene shown (38.3 dv).

Smith and Howell (2009) found that changing the range of visibility images shown has a 

pronounced impact on the resulting 50% acceptability level. They concluded that this is 

an indication that preferences obtained using the general PS method are not stable, and 

therefore not an accurate representation of the populations pre-existing (or “true”) 

preferences for visibility.

While the Smith and Howell (2009) results suggest such a hypothesis, the Test 2 and Test 

3 results also are consistent with a very different effect in visibility preference survey 

results. The range of visibility conditions shown in Test 1 and the original 2001 

Washington, DC study were selected by EPA to present a good representation of the 

distribution of visibility conditions expected to occur in Washington, DC with thencurrent EPA air quality regulations. In other words, the original range and distribution of 

visibility conditions shown to the participants were near to current conditions, and 

reflected the “policy relevant” range of expected conditions. This fundamental approach 

was used in all the existing studies (i.e. Denver, Phoenix, and British Columbia); the 

range of visibility conditions reflects current real world conditions. While the participants 

were not informed that the images they were being shown were essentially then-current 

conditions, the range of images would have been consistent with the participants own 

experiences.

When the range of images presented was substantially different, as in Tests 2 and 3, the 

participants would very likely recognize that the conditions did not depict the real range 

of conditions. This creates a disconnect between the participants’ own experiences and 

expectations and the hypothetical scenario being presented to them through the use of the 

altered ranges. This creates a form of the well known “scenario rejection bias” that has 

been observed in many types of stated preference studies. Current methods of stated 

preference research on a range of environmental issues find it is very important to present 

as realistic and believable a description as possible of the real world conditions matching 

to participants personal experience and knowledge in order to produce robust, stable and 

reproducible results. The effect Smith and Howell (2009) see in their Test 2 and Test 3 

methods are likely to be due to just such a scenario rejection biasing influence.

Thus, on the basis of the above discussion, the EPA disagrees that the findings of Tests 2 

and 3 in Smith and Howell (2009) make the case that the methodology used in the 

preference studies is fundamentally flawed and therefore disagrees with commenter 

views that the EPA’s use of these studies is somehow inappropriate, scientifically 
Interagency Review Draft –Do Note Quote or Cite December 12, 2012

PM NAAQS Response to Comments—Section IV.5 (DRAFT)

10

unsound, or arbitrary and capricious.

(2) Comment: Many commenters also asserted that preference study participants were not 

given guidance on how to interpret the concept of “acceptable urban visibility”. This lack 

of guidance was considered a detriment by commenters because in their view it renders 

the responses subjective.

Response: EPA notes that while the overall method used in all the preference studies is 

consistent (based on the method and elicitation protocol developed by Ely et al., 1991 in 

the Denver study), each study developed a specific protocol (including the specific 

language on both background information and the wording of the key questions) for 

conducting that study’s survey. The Denver and Phoenix studies each gave the survey 

participants specific guidance on interpreting “acceptability”. 

The Denver study framed the question in the explicit context of setting an urban visibility 

standard for Denver, and instructed the participants that they were being asked to identify 

the VAQ that in their opinion should violate an urban visibility standard, and that 

“...standard violations should be VAQ that is unreasonable/objectionable, and 

unacceptable visually. Participants were asked to discern ‘how much haze was too much’, 

not to indicate standard violations based on whether haze was detectable unless any 

amount of haze was more than they could accept” (Ely et al., 2001, p. 6-7).

The Phoenix study instructions to participants defined acceptable (or unacceptable) as 

“Consider ‘unacceptable’ as visual air quality that is unreasonable or objectionable 

visibility. Please do not mark a slide ‘unacceptable’ just because you can see some haze, 

unless you believe that any amount of haze is more than you would tolerate.”

The British Columbia study (Pryor et al., 1996) followed the survey protocol in the 

Denver survey closely, and gave similar guidance to the participants on the meaning of 

“acceptability” in a standard setting context.

The Washington, DC (2001) study was a focus group designed to investigate a series of 

issues involved with conducting group interview type surveys about preferences for 

urban visibility. A key feature of the 2001 focus group script was to deliberately not give 

the participants guidance on how to interpret “acceptable” or “unacceptable” before 

rating the series of Winhaze images. After the series questions involving rating the 

images, the focus group moderator lead a discussion about how the participants 

interpreted the term “acceptable”, and what issues and information they considered in 

forming their responses. The 2009 series of Washington, DC urban visibility preference 

experiments followed a similar procedure, although apparently conducted a more 

abbreviated focus group follow-on discussion.

Thus, the EPA believes that study participants were given sufficient direction and context 

so that they could understand the nature of the questions being posed and were all on 

equal footing in developing subjective judgments regarding “acceptable” and 

“unacceptable” visual air quality. Since the goal of these studies was to elicit personal 

preferences, which by very nature are subjective, the EPA does not view the subjectivity 
Interagency Review Draft –Do Note Quote or Cite December 12, 2012

PM NAAQS Response to Comments—Section IV.5 (DRAFT)

11

of the responses as a weakness of the study design or protocols.

c. Comments on Preference Study Results and Adversity

A number of comments were received regarding the EPA’s use of preference study 

results to make the determination that adverse PM2.5-related visibility effects on the public 

welfare are occurring. Some of these commenters questioned whether the EPA had made the 

case that unacceptable levels of visual air quality based on preference study results alone can be 

equated with an adverse public welfare effect. Other commenters questioned the EPA’s selection 

of the 50% acceptability criterion as the appropriate benchmark for determining adversity. 

(1) Comment: Several commenters asserted that the EPA had not made a sufficient case for 

the occurrence of adverse public welfare effects in association with ambient levels of 

PM2.5: 

Thus, EPA seemingly was building the foundation for a determination of what 

constitutes an adverse effect on visibility in the context of public welfare. 

However ... EPA subsequently veered toward an oversimplified focus on public 

acceptance of visibility conditions.... EPA’s discussion of visibility in the Policy 

Assessment and its proposed rule in the Federal Register focuses entirely on 

“acceptable” and “unacceptable” visual air quality and make no mention of an 

“adverse effect” in the context of visibility. EPA’s reliance on only 3 urban PS 

represents a paucity of data and a wholesale abandonment of any effort to seek a 

scientifically measurable adverse effect. (Kennecott Utah Copper LLC, p. 26)

These commenters suggested that unless preference study information is linked to 

personal comfort and well-being or other associated welfare effects, it cannot form the 

basis of a determination of adversity.

Response: The EPA first notes that the definition of effects on welfare included in section 

302(h) of the CAA identifies both visibility and the broader category of effects on 

personal comfort and well-being as effects on welfare. In setting a secondary standard to 

address visibility impairment, the EPA considers the effect on the public from 

impairment of visibility as a separate and distinct welfare effect in its own right. The EPA 

is not required to translate this into terms of personal comfort and well-being, as visibility 

impairment is designated explicitly by Congress as an effect on welfare. While there may 

be a large degree of overlap among these different welfare effects, the EPA properly 

focuses on evaluating all of the information before the Agency on the effect visibility 

impairment has on the public, whether or not this impairment would also be categorized 

as having an adverse effect on personal comfort and well-being. It is in the context of all 

of this information that the EPA makes the judgment as to the appropriate degree of 

protection from known and anticipated adverse effects on the public from visibility 

impairment. The EPA recognizes that there is uncertainty about the degree of adversity 

to the public welfare associated with impairment of VAQ. However a secondary standard 

is designed to provide protection from "known or anticipated" adverse effects, and a 

bright line determination of adversity is not required in judging the requisite degree of 

protection under section 109(b)(2). Furthermore, the EPA disagrees that it has 
Interagency Review Draft –Do Note Quote or Cite December 12, 2012

PM NAAQS Response to Comments—Section IV.5 (DRAFT)

12

abandoned its consideration of visibility-related impacts on the welfare effect of personal 

comfort and well-being, as is made clear in the following quote:

Research has demonstrated that people are emotionally affected by low visual air quality, 

that perception of pollution is correlated with stress, annoyance, and symptoms of 

depression, and that visual air quality is deeply intertwined with a ‘‘sense of place,’’ 

affecting people’s sense of the desirability of a neighborhood (U.S. EPA, 2009a, section 

9.2.4). Though it is not known to what extent these emotional effects are linked to 

different periods of exposure to poor visual air quality, providing additional protection 

against short-term exposures to levels of visual air quality considered unacceptable by 

subjects in the context of the preference studies would be expected to provide some 

degree of protection against the risk of loss in the public’s ‘‘sense of well-being.’’ (77 

FR 38973/1, emphasis added)

The approach taken to address such qualitative, but policy-relevant, information in this 

review is the same as in other NAAQS reviews. The review is initiated with a 

comprehensive assessment of all possible public health and welfare effects associated 

with PM in the Integrated Science Assessment. Then policy-relevant effects for which 

there is sufficient quantitative information to allow a determination of the change in risks 

associated with incremental changes in air quality are assessed (in this review, in the 

Visibility Assessment) and used to provide a quantitative basis to inform the selection of 

an appropriate range of levels for further consideration in the Policy Assessment. In the 

Policy Assessment, the EPA considers all important policy-relevant evidence and 

information, both quantitative and qualitative, in making recommendations regarding the 

range of policy options appropriate for the Administrator to consider. It is in the context 

of all of this information that the Administrator makes her final judgment as to the 

appropriate degree of protection from known and anticipated adverse effects on the 

public from visibility impairment..

(2) Comment: Another issue raised in the comments regarding adversity is the EPA’s 

decision to use the 50 percent acceptability criterion from the public preference studies in 

determining candidate protection levels of visibility impairment for the selection of a 

national level of visibility protection. For example, AMC et al. recommended “a 75% 

acceptability criterion as a target that is in line with protecting the broader public from 

the negative effects of visibility impairment” (AMC, et al., p. 9).

Response: In the Visibility Assessment, the EPA noted that the use of the 50 percent 

acceptance level for urban visibility was first presented in Ely et al. (1991) (U.S. EPA, 

2010b, p. 2-5). Ely discussed the use of the 50 percent acceptability criterion as a 

reasonable basis for setting a standard to protect visibility in urban areas.

The standard was determined based on a 50% acceptability criterion, that is, the standard 

was set at the level of extinction that would divide the slides into two groups: those 

judged acceptable and those judged unacceptable by a majority of the people in the study. 

The criterion is politically reasonable because it defines the point where a majority of the 

study participants begin to judge slides as representing unacceptable visibility. It is also 
Interagency Review Draft –Do Note Quote or Cite December 12, 2012

PM NAAQS Response to Comments—Section IV.5 (DRAFT)

13

consistent with psychological scaling theory4 which indicates that a “true score” exceeds 

a standard when more than 50% of the “observed scores” exceed that standard. (Ely et al., 

1991, p. 11)

As Ely described, the 50 percent acceptability criterion and the preference study 

conducted by Ely were used as the basis for setting the level of the Denver Visibility 

Standard in 1990. The same criterion was used in the Phoenix preference study (BBC 

research, 2003) and was also used as the basis for setting the level of the Phoenix 

Visibility Standard in 2003. Thus, after considering the information that served as the 

original basis for its selection as described in Ely et al., 1991, and given its acceptance 

and use in existing visibility programs and the absence of information to support a 

different acceptability criterion, the EPA continues to conclude that it is reasonable to use 

the 50 percent acceptability criterion in determining target levels of protection from 

visibility impairment. 

d. Comments on Using Preference Studies as Basis for Nationally Uniform Standard

A number of commenters raised concerns regarding the bases for and implications of the 

differences observed in the preference study results, concluding that these results were due to 

regionally varying factors and thus could not be used to set a national standard.

(1) Comment: Some commenters asserted that because the confidence intervals around the 

four 50 percent acceptability levels do not overlap at all, and because there are variations 

in preference study designs and inherent differences in the visual setting among cities and 

panels, the four preference curves and their associated 50 percent dv values are cityspecific and statistically different. The commenters concluded, therefore, that it was 

inappropriate to aggregate the 50th percentile dv values from multiple studies and that 

they should instead be evaluated individually. Similarly, other commenters expressed the 

related view that the preference study results cannot be used to set a national standard for 

visibility impairment because the results show that visibility preferences vary regionally 

and/or locally for a variety of reasons. For example: 

The ‘one-size-fits-all’ approach...is not viable because it does not account for 

regional and city-specific factors that have been made evident in the disparity of 

preference study data.... It is well known, for example, that the level of light 

extinction to which people in different areas of the country are accustomed, as 

well as the urban setting, are the primary factors that affect a person’s visual 

perception of an urban vista. Thus, the degree to which extinction threshold can 

be related to human welfare is inevitably regionally-dependent. (API, Attachment 

2, p. 4)

[T]he proposed method falls short because it is not temporally or geographically 

representative enough to have any meaning.... The uncertainty evidenced in these 

studies and the non-uniformity between the western and eastern vistas makes it 

impossible at this time to set an acceptable light extinction value that would 

 

4

J.C. Nunnally, Psychometric Theory, 2nd ed., MacGraw-Hill, New York, 1978.
Interagency Review Draft –Do Note Quote or Cite December 12, 2012

PM NAAQS Response to Comments—Section IV.5 (DRAFT)

14

appropriately address visibility concerns in non-Class I areas. (New York 

DOH/DEC, pp. 5-6)

These commenters suggest that the basis for the differences in preference study

results include regionally varying preferences for acceptable visual air quality 

and/or regionally varying factors, such as relative humidity and other differences 

between Eastern and Western areas. These commenters asserted that it would be 

more appropriate to develop distinct visibility standards at the state or local level, 

since it is not possible to select a single level for a national standard to protect 

visibility across the United States that provides the appropriate degree of 

protection in locations with distinct characteristics, including areas which lack 

“important visibility vistas” that might not need standards at all, since flat areas 

without significant terrain have a limited maximum visual range. (NEDA/CAP, p. 

3)

Response: The EPA agrees that the preference curves and the 50 percent dv levels are 

separate and distinct data points representing four different VAQ preference curves for 

four unique urban scenes. However, the EPA does not agree that the study results 

necessarily support a conclusion that preferences are regionally dependent, nor does the 

EPA consider the fact that the four curves are distinct as a weakness of the approach or a 

reason that the results cannot be compared. In particular, the EPA notes that the results of 

Smith and Howell (2009) which show that participants in Houston and Washington, DC 

did not have significantly different views on acceptable air quality in Washington, DC, 

provide limited support for the conclusion that people’s preferences differ less because of 

where they live and more because of the scene they are viewing. Indeed, these preference 

study results seem to substantiate what is already known, that people’s preferences for 

VAQ depend in large part on the characteristics and sensitivity of the scene being 

viewed. It is to be expected that people looking at four different scenes would have four 

different preference curves, with four different 50% acceptability values, depending on 

how changes in VAQ impact the various scenes. Therefore, it might also be the case that 

if the same scene was shown to participants in different regions of the country, a single 

preference curve for that scene could be drawn. The preference study results showing that 

people find less air quality degredation acceptable for sensitive scenes than for less 

sensitive scenes could be interpreted as validating the appropriateness of the approach for 

eliciting public preferences. It is the EPA’s view that the differences in the preference 

study results appropriately reflect the different sensitivities to visibility impairment 

among the four urban scenes and thus provide useful information about how people’s 

preferences for visual air quality would change when viewing urban scenes of differing 

sensitivities. 

In response to the comments that there are factors that vary regionally that are important 

to take into account when setting a national standard for visibility protection, the EPA 

agrees, and notes that it has clearly taken these into account. Section VI.A above 

regarding the history of the secondary PM NAAQS review discusses the evolution of the 

EPA’s understanding regarding the regional differences in PM concentrations, relative 

humidity and other factors. As a result, the current review has gone to great lengths to 
Interagency Review Draft –Do Note Quote or Cite December 12, 2012

PM NAAQS Response to Comments—Section IV.5 (DRAFT)

15

address these factors, leading to the EPA’s proposal to use the IMPROVE algorithm to 

calculate light extinction in order to take into account the varying effects of relative 

humidity and speciated PM. While this approach does not result in a uniform level of 

ambient PM2.5, it does ensure a nationally uniform level of visibility protection. The EPA 

refers the reader to other sections of the final rule, including sections VI.B.1.a, VI.B.1.c, 

VI.C.1.b and VI.C.1.f, and other sections of the Response to Comments document for a 

more detailed response as to how it is taking these variables into account. 

(2) Comment: The Arizona DEQ stated that it was inappropriate to use the Phoenix study for 

purposes of establishing a level for a national standard: 

The proposed secondary visibility standard does not integrate our measurement 

method or mirror our index scale, essentially making the program incomparable 

with the proposed standard. Four urban visibility studies were cited in the 

proposed rule as a consideration in developing the level of the secondary PM2.5

standard. The recommended level for the proposed standard, 28 or 30 deciviews, 

and corresponding calculation method excludes extinction by coarse particulate 

matter and Rayleigh scattering. The cited studies were conducted considering 

total light extinction; including extinction resulting from particulate matter and 

Rayleigh scattering. Visibility impairment due to coarse particulate matter can be 

an important contributor in Arizona, specifically in the Phoenix area where

ongoing measurements have been made. Therefore, ADEQ believes that the 

proposed levels of the secondary visibility standard are inconsistent with 

applicable urban studies. (Arizona DEQ, p. 2)

Response: The EPA understands that each of the four preference study areas has a 

different mix of pollutants and that across the studies a variety of methods for measuring 

and linking ambient PM concentrations to images (either photos or WinHaze) of differing 

haze levels at a given urban scene were employed. However, as discussed above in (2) in 

subsection i., the EPA conducted a detailed assessment (77 FR 38893, 77 FR 38974-75) 

of the differences and similarities that existed between the four studies and concluded 

that it was still appropriate to compare them and consider their results together. The 

CASAC also reached the same conclusion (Samet, 2010b, iii). In regard to the 

commenter’s concern about contributions from coarse particles and Rayleigh, the EPA 

further notes that participants in the Phoenix and other studies were not asked to break 

out their preferences based on how much of the haze was caused by contributions from 

coarse particles, Rayleigh scattering or fine particles. Instead, as noted above in comment 

(2) in subsection ii., Phoenix study participants were simply asked to identify levels of 

visual air quality that they would consider “unreasonable or objectionable”, no matter 

what the source. Based on this information, Phoenix was able to come up with a target 

level of VAQ, even though pollutant mixes vary temporally and spatially, even within the 

Phoenix urban area. Since the goal of the secondary PM2.5 NAAQS is to provide a 

uniform national level of public welfare protection from PM2.5 related visibility 

impairment, the form of the EPA’s proposed secondary standard was selected to allow 

each area to achieve that level of protection in a way that addresses its own unique mix of 

contributing pollutants. Therefore, the EPA disagrees with the commenter that it is 
Interagency Review Draft –Do Note Quote or Cite December 12, 2012

PM NAAQS Response to Comments—Section IV.5 (DRAFT)

16

inappropriate to use the results of the Phoenix study to inform a range of candidate 

protection levels for consideration by the Administrator in selecting a national level of 

visibility protection. Instead, the EPA believe the Phoenix results form an important data 

point that shed light on public preferences for visibility in western areas with urban 

scenes that include distant mountains. 

(3) Comment: Similar to comment (2) above, Colorado DPHE expressed concern with the 

EPA’s use of the Denver study results in a national context. In particular the commenter 

states:

The proposed secondary visibility standard relies in part on the Denver urban 

visibility standard as supporting justification for establishing a new secondary 

standard. However, the Denver visibility standard was originally developed in 

1989 in response to a bipartisan directive from the legislature to meet a statutory 

deadline of January 1990 for a standard based solely on aesthetics. The Denver 

visibility standard, set at about 20 deciviews (32 miles or 0.076/km), was based 

on a survey of 214 Denver citizens that evaluated 25 slides of the ‘Brown Cloud’, 

a publically understood name for the strong inversions that occur in the Denver 

metropolitan area. These slides were specifically referenced to data obtained 

using a transmissometer. Thus, this standard is based on a specific view of Denver 

and specific transmissometer location and path and is applicable only to this 

location. This constitutes a limitation with its applicability to a national secondary 

standard, particularly a proposed standard that does not use a direct measurement 

method. ... Visibility from year to year is highly variable.... [A]dditional studies 

are necessary to establish a secondary NAAQS standard, or what might ultimately 

become multiple standards, across the United States” (Colorado DPHE, p. 2-3).

Response: The Colorado comment raises some of the same concerns as the Arizona 

comment and the EPA refers the reader to its response to comment (2) above. In addition, 

this commenter states that the visibility standard is “based solely on aesthetics”. This is 

not a significant difference with the proposed secondary visibility PM NAAQS. While 

qualitative information was also available regarding personal comfort and well being, the 

proposed range of levels considered by the Administrator was derived from the 

quantitative information regarding public preferences for the aesthetic quality of the 

urban view. This commenter also points out that the slides presented to the study 

participants included images of the Brown Cloud that occurs under conditions of a strong 

inversion. The EPA notes that while this condition does not happen everywhere, there are 

other urban areas that experience it and therefore preference information that takes such a 

phenomenon into account provides useful information for the Administrator to consider. 

The commenter further notes that “this standard is based on a specific view of Denver 

and specific transmissometer location and path and is applicable only to this location.” In 

spite of this, however, the EPA observes that Denver still went ahead and established a 

standard that applies to the entire metropolitan area, even though there are many other 

views and paths that could be evaluated. Likewise, the EPA believes it is possible to 

select a national level of visibility protection based on a variety of disparate data points 

for urban visibility preferences, because when taken together, they provide useful 
Interagency Review Draft –Do Note Quote or Cite December 12, 2012

PM NAAQS Response to Comments—Section IV.5 (DRAFT)

17

information regarding the range of scene and site characteristics that can influence public 

perception of acceptable VAQ. While the EPA agrees that additional studies would be 

helpful and provide more useful data points to consider, it does not agree that they are 

necessary to establish a secondary standard.