POLICY ISSUE

NOTATION VOTE

April 28, 2004 SECY-04-0073

FOR: The Commissioners

FROM: William D. Travers 

Executive Director for Operations

SUBJECT: SURVEY TO MEASURE OPENNESS

PURPOSE:

This paper discusses the need to measure the effectiveness of NRC’s proposed strategic goal

of enhancing openness in our regulatory process, provides the Commission with a survey

method that staff believes would assess our effectiveness in this area, and seeks to obtain

approval on the conduct of such a survey.

BACKGROUND:

Based on stakeholder comments associated with the development of the FY 2000-2005

Strategic Plan, the staff recommended the use of a survey instrument to baseline public

confidence and document the public’s specific concerns. At that time, “enhancing public

confidence” was a goal delineated in the agency’s Strategic Plan. The staff proposed

conducting a public confidence survey in a staff paper to the Commission (SECY-00-0035,

dated February 11, 2000). The Commission disapproved the staff’s recommendation (SRM

dated March 27, 2000) and instead approved an augmented outreach program to include

enhancing the process of obtaining feedback from attendees at public meetings. However, the

Commission noted in the SRM that “The staff should not interpret the Commission’s disapproval

[of Option 2] as indicating that a public confidence survey may not be appropriate in the future.” 

To comply with the SRM, the staff developed a feedback form which has been distributed at all

our public meetings. We have been analyzing the responses to the forms and providing that

information to the staff. A database is being developed to more accurately and efficiently track

the responses to these forms. The most recent information entered into the database indicated

that 90% of respondents felt NRC’s public meetings achieved their stated purpose, and 95% of

respondents felt they were given a sufficient opportunity to ask questions or express their views

at such meetings. However, while public meeting feedback forms have been valuable in terms

of gauging satisfaction with our public meetings, they are by no means a comprehensive

indicator of the quality of “openness” in the NRC.

Mindy S. Landau, OEDO, (301) 415-8703
2

While public confidence is still an important theme underlying our regulatory activities, the 

proposed FY 2004-2009 Strategic Plan revised this goal to reflect our desire to “ensure

openness in our regulatory process.” However, discrete measures that would assess the

public’s perception of our effectiveness in attaining this goal have not yet been established. 

The proposed Strategic Plan states that we will “Identify areas that require improved messages

and more public engagement and dialogue. This may be achieved with independent surveys or

other measurement instruments.” In addition, the Merrifield Task Force Report issued on July

18, 2003, suggested that the Commission “measure the extent to which the public views the

agency as an independent, competent and a credible regulator; the extent to which the public

understands the NRC mission; and the extent to which the public has opportunities to express

its views effectively.” The Task Force suggested a baseline survey of targeted stakeholders to

assess the effectiveness of our communication efforts.

Efforts by the staff to improve openness have been ongoing for several years. They include

public meetings with stakeholders utilizing an enhanced public participation policy, workshops

and local meetings with members of the public and/or licensees, increased use of focus groups,

annual two-day media workshops, development of plain language brochures on a variety of

topics, continued publication of press releases and an extensively revised external web page. 

We also conduct outreach or coordination meetings to inform stakeholders of various activities

and to better understand their general interests and concerns. These actions are related to

regulatory initiatives such as the early site permit reviews, siting of materials facilities and the

proposed HLW repository, and many other areas. While these activities have provided

opportunities for improved communication with external stakeholders, the staff has neither

attempted to formally measure the effect of these interactions, nor their impact on our

openness goal. 

Public confidence and openness measures have also been addressed in stakeholders’

comments on the draft Strategic Plan. Comments included questioning reasons for the change

from public confidence to openness, and suggesting the agency publicize actions that resulted

from public comments, so the public can easily see that their suggestions are worthwhile and

considered in the agency’s decisions.

To measure the extent to which NRC has achieved its goal of ensuring openness, the staff

recommends that the Commission approve the utilization of a survey instrument. This paper

provides the staff’s assessment and recommendations related to conducting surveys to

measure openness in NRC’s regulatory activities.

DISCUSSION

The proposed FY 2004-2009 Strategic Plan includes a goal and strategic outcomes related to

enhancing openness. These strategic outcomes state that we will provide accurate and timely

information to the public about the uses of and risks associated with radioactive materials,

enhance the awareness of our role, provide information about the performance of our

licensees, and obtain early public involvement in issues likely to generate substantial interest.

In order to measure the achievement of these goals, we must first define a baseline measure of

openness. The results of these measures would be used to assess the effectiveness of staff

activities in this area. The process of measuring openness can be complicated by how we
3

define the term “public” in context of the NRC’s regulatory activities. The term “public” is 

defined in the proposed strategic plan as “the community at large.” “Stakeholders” are defined

as “a subsection of the general public that comprises a targeted population that has a specific

interest in a given topic.” For instance, residents living near nuclear facilities (including power

reactors or fuel facilities), members of Congress, licensees, and non-governmental

organizations would be considered groups of stakeholders. The definition is central to the

development and implementation of measurement tools to ensure that the proper populations

and issues are included, and to set expectations for the survey as well as uses for its results.

In general, certain stakeholders have a better understanding of who we are and what we do, as

opposed to the general public who may have little or no knowledge of the NRC. This distinction

is important when considering the types of activities which could be implemented by the staff to

increase and subsequently measure the effectiveness of their communications. For example, if

our efforts were intended to address the general public, the activities would be more general,

such as outreach efforts to inform this group of NRC’s mission, responsibilities and activities. 

The external NRC Web site is an example of an outreach effort to the general public. Different

types of efforts would be considered to address the concerns of specific stakeholders.

To obtain feedback on openness, survey questions should be designed to focus on the amount

or type of information stakeholders receive, including the clarity, timeliness and quality of such

information. In addition, the survey should inquire about the accessibility and scope of the

information we provide. To go even further, the survey could be designed to explore how

stakeholders view “open” avenues for participation in the decision making process. 

Identifying the types of survey questions and participants from the general public that would

provide meaningful data can be challenging. If the segment to be measured does not

understand the mission of the NRC, the information is more likely to reflect this lack of

information or understanding. Therefore, for the purposes of an initial survey, the staff

recommends targeting one or more specific stakeholder groups. It is more likely that

measurement of a targeted stakeholder group will yield more meaningful information about

specific NRC activities. For this reason, the staff will need the support of a contractor that has

in-depth experience in developing appropriate measurement techniques.

For example, a survey could include members of the public living in the emergency planning

zone of one or more commercial power reactors, or opinion leaders living near those facilities

who may be more knowledgeable about our activities. This survey could provide information

and insights that would describe this group’s understanding of NRC and its regulatory

programs. The results would be useful to determine the awareness of ongoing NRC activities

and assess the methods currently used to communicate with the group and determine what

changes could be made. 

Based upon the experience and knowledge gained by conducting this initial survey, the staff will

evaluate the usefulness of the information and make a recommendation to the Commission

regarding follow-up actions, depending on the survey results. Actions could include increasing

or improving our communication with the targeted groups using a variety of vehicles. 

Subsequent surveys may need to be performed with commercial contractors to better target

improvement areas. The Commission would be consulted before subsequent surveys are

initiated. 
4

The staff will continue to distribute and evaluate public meeting feedback forms to assess the

effectiveness of our public meetings.

Measurement Techniques

A primary need for any given technique is to determine which external stakeholder perceptions

or views should be measured, and the willingness of the participants to take part in the

measurement activity. After determining the population and issues of interest, the next decision

is to choose a measurement technique or method. Surveys and interviews offer the

advantages of direct measurement. An initial survey could be conducted which would define or

create an initial baseline measure. Follow up surveys could be conducted to determine if

perceptions of openness were changed as a result of staff activities. The questions would be

tailored to provide insight into how well we meet our openness goal. 

The staff evaluated two options for measuring openness -- to develop and conduct a survey

with support provided by a commercial contractor, or to participate in the American Customer

Satisfaction Index (ACSI). In considering these alternatives, the staff discussed experiences

with several other federal agencies, contacted a sample of commercial contractors that were on

GSA’s Federal Supply Schedule, and met with a representative of the American Customer

Service Index. 

Commercial Contractor

In considering the use of a survey performed by a commercial contractor, there are advantages

and disadvantages. The advantage would be that NRC would have more flexibility in designing

the questions and in obtaining more detailed information from the responses, perhaps even

verbatim. However, the cost is more uncertain. One large contractor, The Gallup Organization,

offered two options and a recommendation. If NRC contracted for four facilities at once, the

cost would be $30,000 per facility and would target 300 households or residents per facility. 

They also recommended surveying opinion leaders in communities near regulated facilities. 

They believe opinion leaders are more likely to be knowledgeable about our mission and more

likely to have had an opportunity to interact with NRC. These individuals could include mayors

or city managers, city council members, energy editors of newspapers or other media, Chamber

of Commerce leaders, civic or business leaders or state energy or environmental officials. 

Interviews of 25 opinion leaders at each of eight different facilities would cost approximately

$60,000. Another smaller contractor offered to interview local residents living near a facility for

less than the ACSI quoted cost of $35K. Completion of the survey and analysis of the data at

commercial firms would take approximately four to five months.

American Customer Satisfaction Index

Another option is to use the American Customer Satisfaction Index (ACSI). Established in 1994,

the ACSI is produced through a partnership of the University of Michigan Business School, the

American Society for Quality, and the international consulting firm, CFI Group. The Treasury

Department currently oversees the contract that allows ACSI to measure customer satisfaction

for certain agencies. The index currently measures satisfaction and the drivers and outcomes

of satisfaction, for more than 200 private sector companies and federal or local government
5

agencies. Among the agencies surveyed in the past are those with regulatory functions such

as FAA, DOT, FEMA, OSHA, IRS, and FDA. 

An ACSI survey would have a standard sample size of 250 individuals per facility. Agencies

work with ACSI to identify the stakeholder segment to be surveyed, develop a list of possible

participants in the segment, and tailor the questions about agency activities. Data are usually

obtained from telephone interviews with customers. The index provides a database of 

questions that can be tailored to customer satisfaction activities of concern to the agency. The

index includes questions which solicit information on activities such as processes (for obtaining

services), accessibility of information, and customer service from agency personnel. However,

questions can also be customized to target specific agency concerns. 

As discussed in the ACSI’s December 2003 report to the General Services Administration,

ACSI uses a model that ties customers’ evaluations of quality to satisfaction, and then explains

the effects of satisfaction on customer complaints and on an objective of importance to a

company or government agency. “For most government agencies the objective is some form of

customer/user trust, typically measured in terms of confidence in the agency, and either future

reliance on agency services or future compliance with agency regulations.” This report also

discussed the difference in survey results between regulatory agencies and agencies that

provide direct benefits to members of the public. The report stated that “Not unexpectedly...

satisfaction is highest among customers that receive a direct benefit from an agency and lowest

for customers subject to regulation by agencies such as the Federal Aviation Administration and

the Occupational Safety and Health Administration.” Overall, citizen satisfaction with

government service delivery grew between 2002 and 2003. The aggregate federal government

ACSI score rose by 1%, from 70.2 to 70.9. This score reflects citizen satisfaction with 64

federal government agency users, including 35 federal government web sites. 

The cost is similar to a commercial contract, approximately $35K per survey. There would be

some time and effort saved in developing the survey questions. The ACSI would be an

interagency agreement, not a commercial contract. 

The staff contacted several other federal agencies - FAA, IRS and FDA - who use ACSI and

other types of surveys to gauge customer and stakeholder satisfaction. Their general

assessment is that the ACSI survey is valuable for high level “trending” of stakeholder

satisfaction. It also provides a comparison to other federal and private industries that may have

similar goals and provides a long-term “track record” they found helpful. Some of the agencies

found it beneficial to use private contractors in addition to ACSI so they could design specific

questions and better target their improvements. Although none of the agencies reported

applying any sweeping changes as a result, they used both types of surveys to improve

incrementally in a variety of areas. 

The visibility of the results would be another aspect of the ACSI that would differ somewhat

from conventional surveys. For the agencies that participate in the ACSI surveys, agencies are

expected to list the results and follow up actions on an agency-sponsored web page. The NRC

would likewise be expected to publish the results of the survey on our web site and develop and

implement an action plan to respond to deficiencies in customer service noted in the index

results.
6

The primary advantages of participating in this index are 1) the ACSI survey is an established

and accepted measurement tool, 2) the survey can be conducted for specific segments in a

timely manner (approximately 4 to 6 months) at reasonable cost, 3) it allows the identification of

specific actions in response to stakeholder concerns and comments, 4) we may contract

through an interagency agreement, requiring less resources than a commercial contract, and 

5) OMB has issued Paperwork Reduction Act clearances for ACSI surveys conducted for

Federal agencies.

Following Commission approval, the staff would work with ACSI to identify the group of

stakeholders to be included in the survey and prepare the questions. The results of this

approach could be used to determine what changes to consider related to interactions with and

information provided to those stakeholder groups. The survey would not include NRC

licensees, industry representatives or individuals associated with the nuclear industry because

there are numerous methods already in place for these groups to communicate with the NRC. 

RECOMMENDATION

In order to provide a baseline measure of openness, the staff recommends that a survey be

conducted. Additionally, based upon the experience other agencies have had with the ACSI,

and the relatively low cost of this survey, the staff recommends that the ACSI technique be

used to establish that measure. In coordinating this paper with other offices, it should be noted

that OPA prefers the commercial contractor option because they believe it gives the agency

better control over the questions and results. 

RESOURCES:

If the Commission adopts the staff recommendation of using the ACSI technique, costs would

be directly proportional to the number of segments surveyed. We anticipate no more than two

segments in one fiscal year. The resources required to conduct each segment are

approximately $35K for a total of $70K. These resources have not been budgeted for and will

be requested as part of the upcoming FY 2006 Planning, Budgeting and Performance

Management process.

COORDINATION:

This plan has been coordinated with the Office of Public Affairs (OPA). The Office of the Chief

Information Officer (OCIO) has no objections and the Office of the General Counsel (OGC) has

no legal objection to this paper. The Office of the Chief Financial Officer (OCFO) has reviewed

this paper for resource implications and notes the potential for unbudgeted requirements. 

Offices will consider unbudgeted resources through the PBPM process.

/RA/

William D. Travers

Executive Director 

 for Operations
6

The primary advantages of participating in this index are 1) the ACSI survey is an established

and accepted measurement tool, 2) the survey can be conducted for specific segments in a

timely manner (approximately 4 to 6 months) at reasonable cost, 3) it allows the identification of

specific actions in response to stakeholder concerns and comments, 4) we may contract

through an interagency agreement, requiring less resources than a commercial contract, and 

5) OMB has issued Paperwork Reduction Act clearances for ACSI surveys conducted for

Federal agencies.

Following Commission approval, the staff would work with ACSI to identify the group of

stakeholders to be included in the survey and prepare the questions. The results of this

approach could be used to determine what changes to consider related to interactions with and

information provided to those stakeholder groups. The survey would not include NRC

licensees, industry representatives or individuals associated with the nuclear industry because

there are numerous methods already in place for these groups to communicate with the NRC. 

RECOMMENDATION

In order to provide a baseline measure of openness, the staff recommends that a survey be

conducted. Additionally, based upon the experience other agencies have had with the ACSI,

and the relatively low cost of this survey, the staff recommends that the ACSI technique be

used to establish that measure. In coordinating this paper with other offices, it should be noted

that OPA prefers the commercial contractor option because they believe it gives the agency

better control over the questions and results. 

RESOURCES:

If the Commission adopts the staff recommendation of using the ACSI technique, costs would

be directly proportional to the number of segments surveyed. We anticipate no more than two

segments in one fiscal year. The resources required to conduct each segment are

approximately $35K for a total of $70K. These resources have not been budgeted for and will

be requested as part of the upcoming FY 2006 Planning, Budgeting and Performance

Management process.

COORDINATION:

This plan has been coordinated with the Office of Public Affairs (OPA). The Office of the Chief

Information Officer (OCIO) has no objections and the Office of the General Counsel (OGC) has

no legal objection to this paper. The Office of the Chief Financial Officer (OCFO) has reviewed

this paper for resource implications and notes the potential for unbudgeted requirements. 

Offices will consider unbudgeted resources through the PBPM process.

/RA/

William D. Travers

Executive Director 

 for Operations

ADAMS ACCESSION NO: ML040560584

 *SEE PREVIOUS CONCURRENCE

OFFICE OEDO OPA CFO CIO OGC 

NAME MSLandau* EHayden* DDrucker* LCrawford* CCameron*

DATE 03/30/04 03/24/04 e-mail 03/22/04 e-mail 03/24/04 3-mail 03/24/04 3-mail

OFFICE DEDM EDO 

NAME PGNorry* WDTravers

DATE 03/30/04 04/28/04

OFFICIAL RECORD COPY

P:\Mindy\Survey Secy Paper.wpd