United States General Accounting Office

GAO Report to Committee on Governmental

Affairs, U.S. Senate, and Committee on

Government Reform and Oversight, U.S.

House of Representatives

June 1997 MANAGING FOR

RESULTS

Regulatory Agencies

Identified Significant

Barriers to Focusing on

Results

GAO/GGD-97-83

GAO United States

General Accounting Office

Washington, D.C. 20548

General Government Division

B-275122 

June 24, 1997

The Honorable Fred D. Thompson

Chairman

The Honorable John Glenn

Ranking Minority Member

Committee on Governmental Affairs

United States Senate

The Honorable Dan Burton

Chairman

The Honorable Henry A. Waxman

Ranking Minority Member

Committee on Government Reform and Oversight

House of Representatives

The Government Performance and Results Act of 1993 was enacted to

improve the efficiency and effectiveness of federal programs by

establishing a system to set goals for program performance and to

measure results. In recognition of the potential difficulties inherent in

establishing results-oriented goals and performance measures, the Act,

which is referred to as “GPRA” or “the Results Act,” provided for a phased

implementation period. Beginning in fiscal year 1994 and extending over

several years, agencies are to develop strategic goals tied to agency

missions, identify performance measures associated with those goals, and

by fiscal year 1999 implement annual results-oriented performance reports

linked to budget requests. In addition, in a March 4, 1995, memorandum,

the President specifically directed the heads of regulatory agencies to

change the way they measure both their agencies’ performance and the

performance of their regulatory staff so as to focus on results.

The Results Act required us to report to Congress on the prospects for the

Act’s implementation governmentwide, and our recent report—The

Government Performance and Results Act: 1997 Governmentwide

Implementation Will Be Uneven (GAO/GGD-97-109, June 2, 1997)—was

developed in response to that requirement. In support of that broader

assessment, we initiated an examination of the early progress toward

results-oriented goal setting and performance measurement in selected

federal regulatory agencies as they prepared to meet the first statutorily

required reporting date of September 30, 1997. We considered “regulatory

agencies” to include any agency (as defined in 5 U.S.C. 551[1]) that issues

rules (as defined in 5 U.S.C. 551[4]). We examined regulatory agencies in

part because of the administration’s initiative to better focus the

Page 1 GAO/GGD-97-83 Barriers to Managing for Results
B-275122 

performance of regulatory agencies on achieving results through more

results-oriented identification of organizational objectives and individual

performance standards. The administration’s initiative differs from the

Results Act by adding the requirement that agencies change the way they

measure the performance of frontline regulators to focus on results. In

addition, because the President required regulatory agencies to start

focusing on results in March 1995, these agencies were expected to adopt

a results orientation sooner than the rest of the federal government.

The regulatory agencies we focused on were the Occupational Safety and

Health Administration (OSHA), the Federal Aviation Administration (FAA),

the Food and Drug Administration (FDA), the Internal Revenue Service

(IRS), and the Environmental Protection Agency (EPA).

We developed this report to describe the status of the five agencies’

attempts as of January 1, 1997, to focus on results and to ascertain what

had assisted or impeded these attempts. Our specific objectives were to

describe (1) the five agencies’ strategic goals and related program

performance measures as well as selected employee performance

standards as of January 1, 1997; (2) the extent to which agency officials

and we believed that these goals, program performance measures, and

employee performance standards focused on results; and (3) the aids and

barriers that officials in the agencies said they faced in attempting to focus

on results.

Focusing on results represents a new approach for both federal agencies

and Congress. Both agency management and congressional oversight have

traditionally focused on inputs (e.g., the number of dollars or staff allotted

to an agency); compliance with legislative or agency-developed rules; and

the accomplishment of outputs (e.g., the number of inspections

completed). Although a results orientation does not ignore these other

factors, it focuses attention on what outcomes agencies—including

regulatory agencies—are trying to accomplish.

Because a focus on results is new for federal agencies, and criteria for

definitively characterizing goals, measures, and standards as “results

oriented” were not yet developed, we adopted an approach that was

intended to stimulate discussion of the characteristics of results-oriented

goals, measures, and standards. We hypothesized that, if there were

differences between our characterizations and those of the agencies, these

differences and the explanations for them might be useful to the agencies

by illustrating how, for example, a goal that appears to be clearly results

Page 2 GAO/GGD-97-83 Barriers to Managing for Results
B-275122 

oriented and linked to an agency’s mission in the agency’s view may not

appear the same way to someone from outside the agency. By comparing

our characterizations with those of agency officials, we intended to foster

discussion and contribute to agencies’ efforts to develop results-oriented

goals, measures, and standards.

Results in Brief As would be expected in the early stages of implementing new and

difficult initiatives, and consistent with our broader assessment of the

prospects for implementing the Results Act, we observed that some of the

five regulatory agencies were further along in the development of strategic

goals, program performance measures, and employee performance

standards than others. The agencies also varied in the degree to which

their goals, associated sets of program performance measures, and

employee performance standards that were in use as of January 1, 1997,

focused on results as judged by both agency officials and by us. In this

regard, it is important to note that although the Results Act was intended

to encourage agencies to focus their goals and measures on results, the

Act does not require that all of an agency’s goals or performance measures

be explicitly results oriented. Moreover, agencies were not required by the

Act to actually have strategic goals and related performance measures at

that point in time.1

 Similarly, the President’s directive to orient frontline

regulators’ performance standards toward results does not explicitly

require that every standard be results oriented.

There were differences in the extent to which agency officials

characterized their goals, program performance measures, and employee

performance standards as “results oriented,” and the extent to which we

did. In general, agencies frequently concluded that their goals, measures,

and standards were more results oriented than we did. At the broader and

more conceptual level of strategic goals, there were relatively few

differences between agency officials’ assessments of the extent of results

orientation and ours. As we and they considered the more specific

program performance measures, differences between our views and the

agencies were greater, although both we and agency officials

characterized the agencies’ performance measures as less results oriented

than the strategic goals. The differences between the agencies’

assessments of the extent of results orientation and ours were greatest in

considering the more detailed performance standards for regulatory

personnel. (See apps. I through V for a detailed description of the extent to

1

Under GPRA, agencies are to submit strategic plans identifying strategic goals tied to agency missions

to Congress by September 30, 1997, and annual performance plans related to the strategic plans are to

be submitted to Congress after the President’s budget is submitted in February 1998.

Page 3 GAO/GGD-97-83 Barriers to Managing for Results
B-275122 

which we and agency officials characterized the results orientation of

agencies’ goals, measures, and standards.) Since January 1997, the

agencies have continued their efforts to develop strategic goals and

performance measures to meet the requirements of the Results Act. In

commenting on a draft of this report, Office of Management and Budget

(OMB) officials noted that most of the agencies’ goals had changed since

our review and some of the revised goals and measures were more results

oriented. As we stated previously, it was our intent for this work to

contribute to such an outcome.

In enacting the Results Act, Congress realized that the transition to

results-oriented management would not be easy. For that reason, the Act

provided for a phased approach to implementation, during which time

agencies have been able to identify obstacles that need to be overcome

and some factors they found helpful. The factor that agency officials most

commonly said aided the establishment of a results orientation in their

agencies was the enactment of the Results Act. Specifically, they said that

GPRA-related requirements for outcome-oriented strategic goals and

performance measures by September 1997 forced them to focus their

attention on results more than they had prior to the passage of the Act.

Other factors that agency officials said aided their focus on results

included issue-specific legislation that cited results-oriented goals or

measures, feedback from stakeholders including Congress and other

interested parties, and top management leadership focused on achieving

results in agency performance.

While agency officials identified some aids to focusing their agencies on

results, they cited numerous barriers to their efforts to establish

results-oriented goals and measures. These barriers included significant

problems in identifying and collecting the data they needed to

demonstrate their agencies’ results. For example, IRS officials said that

there are significant difficulties involved in measuring the fair amount of

total taxes due, so it is hard to determine how much IRS should be

collecting. Other data-related barriers included the difficulty of

determining the causes of illnesses with long latency periods and

problems with data collection efforts, such as under- or inconsistent

reporting. Agencies also cited as a barrier the fact that diverse and

complex factors affect agencies’ results (e.g., business cycles,

technological innovations, or the need to deliver federal program

initiatives and thus achieve results through third parties), and their lack of

control or influence over those factors. For example, although FDA

regulations attempt to ensure that food is safe, FDA cannot control whether

Page 4 GAO/GGD-97-83 Barriers to Managing for Results
B-275122 

consumers ultimately ensure that food is stored and prepared properly.

Finally, agency officials observed that the long time period needed to see

results in some areas of federal regulation was a barrier to identifying and

managing toward those results in the framework of annual performance

plans and budgets. The impact of some agencies’ regulatory actions, such

as limiting exposure to a hazardous chemical, may not be evident for

years. These barriers suggest that the implementation of the Results Act in

a regulatory environment may prove more difficult in some cases than in

others.

Although we did not determine the validity of their claims or the extent to

which the relative lack of results orientation in the performance measures

was directly traceable to these barriers, the issues the agency officials

raised should not be summarily discounted. Yet, while solutions may not

be readily apparent, the potential long-term benefits merit efforts to

develop better measures. For example, although long-term results may be

difficult to use in an annual process, interim measures indicating the

degree of progress toward the achievement of those results can sometimes

be used to determine whether current policies are having the desired

effects. Likewise, output measures such as the number of inspections

conducted can be used when studies exist to demonstrate their

relationship to the results that agencies are attempting to accomplish.

Finally, agencies may be able to develop analytic techniques to enable

them to control for extraneous factors and isolate the impact the agencies

are having on such outcomes as injury rates or clean water.2

For GPRA to work in a regulatory environment, and for regulatory agencies

to meet their September 1997 deadlines, it is important that those agencies

begin to develop solutions to these measurement problems. Some agencies

have already begun to do so. For example, IRS has been working for some

time on improving its estimates of taxes owed, which might ultimately be

used to determine the effectiveness of the agency’s compliance programs.

Agencies will also need to learn from the experiences of other agencies

that have previously confronted and addressed similar issues. Our

Executive Guide, for example, identified key steps that agencies need to

take to implement the Results Act and provided case illustrations where

federal agencies had put these practices to use.3

2

Although Congress will probably require agencies to provide information on their unique

contributions to outcomes as part of the oversight process, GPRA does not technically require

agencies’ performance measures to provide this kind of information.

3

Executive Guide: Effectively Implementing the Government Performance and Results Act

(GAO/GGD-96-118, June 1996).

Page 5 GAO/GGD-97-83 Barriers to Managing for Results
B-275122 

However, while we conceptually agree with the Act’s emphasis on

managing for results, we also recognize that the development and practical

application of results-oriented program performance measures in

regulatory agencies will not always be done quickly or easily. For example,

in order for an agency to isolate and measure its unique contribution to a

result that is affected by a variety of factors outside of the agency, the

agency must control for the effects of all of those nonagency factors. It is

yet unclear whether that degree of analytical precision will always be

possible.

Background Federal regulations affect the lives of virtually every American, and have

requirements that are designed to ensure, among other things, that

workplaces, air travel, foods, and drugs are safe; that the nation’s air,

water, and land are not polluted; and that the appropriate amount of taxes

are collected. They are the means by which statutory public policy goals

are put into effect to achieve desired economic and social benefits.

Regulations also have billions of dollars of impact on businesses, state and

local governments, and the economy as a whole. Therefore, it is important

that decisionmakers and taxpayers understand what specific results

federal regulatory agencies are attempting to achieve and how those

agencies are measuring their progress toward achieving those results.

Two efforts—one by Congress and the other by the Clinton

administration—have required regulatory agencies to become more results

oriented. Congress passed the Results Act in July 1993 with the intent of

improving the efficiency and effectiveness of federal programs. Under the

Act, federal agencies are to develop a strategic plan by September 1997,

including general goals for the major functions and operations of the

agency and a description of how the goals are to be achieved. Agencies are

to submit annual performance plans to Congress after the President’s

budget submission in early 1998. Beginning in fiscal year 1999, the Act

requires that each agency report annually to the President and to Congress

on its performance and its progress toward meeting its goals. Agencies’

plans are to be used to support the President’s budget, and agencies’

annual performance reports are intended to provide important information

to agency managers, policymakers, and the public on what each agency

accomplished with the resources it was given.

Agencies’ strategic plans are to cover a period of at least 5 years and, as

shown in figure 1, should contain the agency’s mission statement, a set of

strategic goals (including outcome-related goals), a description of how the

Page 6 GAO/GGD-97-83 Barriers to Managing for Results
B-275122 

agency intends to achieve these goals, and a description of how the

agency’s performance goals will be related to its strategic goals. Strategic

plans are intended to be the starting point for each agency’s performance

measurement efforts. The annual performance plans are to use program

performance measurement to reinforce the connection between the

long-term strategic goals outlined in their strategic plans and the

day-to-day activities of their managers and staff. The annual plans are to

include performance goals for an agency’s program activities as listed in

the budget, a summary of the necessary resources to achieve these goals,

and the performance indicators that will be used to measure performance.

Although these requirements are intended to encourage agencies to move

toward results-oriented management, the Act recognizes that not all goals

and measures will necessarily be results oriented.

Figure 1: Linking Results Act

Requirements to Day-to-Day Activities

Strategic plans

Mission statement

Strategic goals

How to achieve goals

Annual performance

plans

Performance goals

Resource needs

Performance

 indicators

Day-to-day activities

of managers and

staff

Source: GPRA.

Getting regulatory agencies to focus on achieving desired results was also

one aspect of the administration’s regulatory reinvention initiative. In

Executive Order 12866, which was issued on September 30, 1993, the

President set forth his philosophy for reforming the regulatory process.

One of the “principles of regulation” in that order is that each regulatory

agency should, to the extent feasible, “specify performance objectives,

rather than the behavior or manner of compliance that regulated entities

must adopt.” In a March 4, 1995, memorandum, the President followed up

on this executive order by directing the heads of all federal regulatory

agencies to, among other things, “change the way you measure the

performance of both your agency and your frontline regulators so as to

focus on results, not process and punishment.”4

 Although the President’s

requirement that agencies measure their performance in terms of results

achieved is consistent with GPRA’s emphasis, measuring employees’

4

The President’s memorandum did not define “frontline regulators.” We defined “frontline regulators”

as regulatory staff who work directly with the regulated community by enforcing regulations.

Page 7 GAO/GGD-97-83 Barriers to Managing for Results
B-275122 

performance in that manner is unique to the administration’s regulatory

reinvention initiative. As previously noted, the President’s initiative does

not explicitly require that all performance standards be results oriented.

The five regulatory agencies have each been attempting to develop a

results orientation for some time. EPA launched a National Goals Project in

1992 to develop measurable long-range environmental goals and, more

recently, attempted to increase the agency’s focus on results in response

to an April 1995 National Academy of Public Administration report.5 OSHA

and IRS began agencywide GPRA pilot projects in 1993.6 Although FAA and

FDA started their agencywide efforts more recently, they had GPRA pilot

efforts under way in some of their units. All five of the agencies said they

are currently consulting with Congress and preparing for agencywide

compliance as part of the Act’s governmentwide implementation

beginning in September 1997. In addition, these five agencies are subject

to the President’s March 4, 1995, memorandum directing them to focus on

results.

These agencies also have implemented various reforms in response to the

administration’s regulatory reinvention initiatives. In June 1995, each

agency reported on its actions regarding four tasks the President assigned

to regulatory agencies in March 1995: eliminating or improving regulations,

rewarding results, creating grassroots partnerships, and negotiating rules.

The “rewarding results” sections of the reports focused on the status of

agency program performance measures and steps taken to ensure that

frontline regulators were evaluated on results rather than process or

punishment measures. For example, EPA noted in its report that it had

shifted its employee performance standards from specific process-driven

numeric goals to definitions of performance in terms of the quality of the

output.

Scope and

Methodology

We selected the five regulatory agencies in this review—OSHA, FAA, FDA, IRS,

and EPA—on the basis of their size (i.e., budget and personnel); the number

of significant regulations they planned to finalize or complete from

October 1, 1995, to October 1, 1996; and the opinions and

5

Setting Priorities, Getting Results: A New Direction for the Environmental Protection Agency, a

National Academy of Public Administration Report to Congress, April 1995.

6

GPRA Performance Reports (GAO/GGD-96-66R, Feb. 14, 1996) summarized our initial observations on

selected agencies’ first attempts to report on their performance in the pilot phase of GPRA.

Page 8 GAO/GGD-97-83 Barriers to Managing for Results
B-275122 

recommendations of experts in the field of federal regulation.7 The

agencies selected were those that appeared most frequently across all of

these factors, although each agency selected may not have been at the top

of any particular factor. For example, OSHA is a relatively small agency that

typically issues few rules each year, but it was listed by most of the

experts as a major regulatory agency.

To describe the agencies’ strategic goals and program performance

measures, we first obtained copies of the goals and measures the agencies

had developed as of January 1, 1997. To describe the employee

performance standards for selected regulator positions at these agencies,

we first asked officials in each agency to identify frontline regulator

positions that they felt would fairly represent the actions of their agencies’

regulators. Using our definition of “frontline regulator”—regulatory staff

who work directly with the regulated community by enforcing

regulations—the agencies designated which positions they considered

“frontline regulator” positions. We then obtained copies of the

performance standards being used as of January 1, 1997, for the selected

positions. Because the Results Act does not cover employee performance

standards, we judged each set of performance standards according to the

expectation set by the President in his March 4, 1995, memorandum that

these standards should “focus on results, not process and punishment.”

To describe the extent to which we and agency officials believed that the

goals, measures, and performance standards were results oriented, we

developed and provided to each agency guidelines for agency officials to

use if they chose in characterizing the agencies’ goals, program

performance measures, and employee performance standards as results

oriented to a great extent, to some extent, or to little or no extent. We used

a similar framework to characterize the agencies’ goals, performance

measures, and employee performance standards. (See app. VI for a more

extensive discussion of the methodology we used to determine the extent

to which their goals, measures, and standards were results oriented.)

To respond to our third objective, we asked officials in each agency to

describe the aids and barriers that their agencies faced in attempting to

focus on results. We also reviewed related documents that described how

agencies developed results-oriented goals and program performance

7

We contacted experts in the field of federal regulation from the following organizations and asked

them to identify which agencies they considered to be the major federal regulatory agencies: the

American Enterprise Institute, the U.S. Chamber of Commerce, the National Federation of

Independent Businesses, the Center for the Study of American Business, Citizens for a Sound

Economy, the National Performance Review, and the Small Business Administration.

Page 9 GAO/GGD-97-83 Barriers to Managing for Results
B-275122 

measures and any problems or successes agencies had encountered. We

did not attempt to determine the validity of the factors that agency

officials said aided or hindered their efforts to becoming results oriented.

We did our work between April 1996 and March 1997 in Washington, D.C.,

in accordance with generally accepted government auditing standards. We

sent a draft of this report to the heads of each of the five agencies included

in this review and to the Director of OMB. Officials from each of these

agencies provided oral comments, which are summarized at the end of this

report and discussed in more detail in the appendix relating to that agency.

In addition, two of the agencies (OSHA and IRS) commented in writing.

These comments are reprinted in their entirety in appendix VII.

Views on the Results

Orientation of

Strategic Goals,

Performance

Measures, and

Performance

Standards

There were differences in the extent to which agency officials

characterized their goals, program performance measures, and employee

performance standards as “results oriented,” and the extent to which we

did. At the broader and more conceptual level of strategic goals, there

were relatively few differences between agency officials’ assessments of

the extent of results orientation and ours. However, as we and they

considered the more specific program performance measures, greater

differences existed between our views and those of the agencies. The

differences between the agencies’ assessments of the extent of results

orientation and ours were greatest in considering the more detailed

performance standards for selected frontline regulator positions. (App. VI

contains a detailed description of our scope and the methods that we used.

Apps. I through V contain detailed information regarding each of the five

regulatory agencies’ strategic goals, program performance measures, and

employee performance standards.)

We and agency officials most frequently agreed in our characterizations of

strategic goals. We both said that most of the agencies’ goals were at least

somewhat results oriented. Agency officials considered almost 70 percent

of the strategic goals to be greatly results oriented, and considered the

remaining goals to be results oriented to some extent. We characterized

about one-half of the agencies’ goals as greatly results oriented and about

one-third as somewhat results oriented. We considered the remaining

goals to be results oriented to little or no extent. An example of one of IRS’

strategic goals was as follows:

“Increase compliance: Our objective is to encourage and assist taxpayers to voluntarily file

timely and accurate returns and to pay on time, and, if taxpayers do not comply, to take

Page 10 GAO/GGD-97-83 Barriers to Managing for Results
B-275122 

appropriate compliance actions. We do this to ensure we collect the proper amount of tax

at the least cost.”

Both we and IRS characterized this goal as results oriented to a great

extent because it was directly related to IRS’ mission of collecting tax

revenue, provided a clear definition of the results expected, and permitted

assessment of the agency’s programs.

As we and agency officials considered the more specific program

performance measures, differences between our views and the agencies’

were greater, although both we and agency officials characterized the

agencies’ performance measures as less results oriented than the strategic

goals. However, our characterizations of the performance measures were

not as positive as those of the agencies. Agency officials said they believed

that most of their sets of program performance measures were somewhat

results oriented, and characterized almost 40 percent of the measures as

greatly results oriented. However, we did not consider any of the sets of

program performance measures to be greatly results oriented. We agreed

with agency officials that most of the sets of measures were somewhat

results oriented, but we viewed over 40 percent of the measures as results

oriented to little or no extent. GPRA requires agencies to establish

performance indicators to be used in measuring or assessing the relevant

outputs, service levels, and outcomes of each program activity as part of

agencies’ performance plans.

For example, OSHA’s sole performance measure for its goal to “implement a

common sense strategy developed in partnership with stakeholders, for

rulemaking and alternative approaches to emerging and priority safety and

health issues” was “the percentage of rulemaking activity on the regulatory

agenda undertaken as negotiated rulemaking.” OSHA officials considered

this measure somewhat results oriented because they believed that using

the negotiated rulemaking process would improve the regulated entities’

acceptance of rules, which would lead to improved compliance and fewer

hazards, thereby leading to safer workplaces. However, we said this

measure was results oriented to little or no extent because (1) it was not

clear how use of the negotiated rulemaking process would make

workplaces safer; (2) the measure simply tracked the percentage of cases

in which this rulemaking process was used and, therefore, OSHA did not

have a set of measures that provided relevant information to gauge

progress toward the goal/mission; and (3) this measure alone did not

address other aspects of the goal, such as the use of alternatives to direct

regulation.

Page 11 GAO/GGD-97-83 Barriers to Managing for Results
B-275122 

We and agency officials did not often agree about how results oriented the

agencies’ employee performance standards were. Agency officials

characterized over 80 percent of their sets of performance standards as

greatly results oriented, and considered the rest somewhat results

oriented. In contrast, we viewed 40 percent of the sets of standards as

results oriented to little or no extent. We characterized over one-third of

the standards as somewhat results oriented, and thought that less than

one-fourth of the performance standards were greatly results oriented.

For example, we generally disagreed with EPA officials’ characterization of

the agency’s performance standards for the criminal investigator and

environmental scientist positions. They said that all but two of the nine

sets of performance standards for these positions were greatly results

oriented and that those two were somewhat results oriented. However, we

characterized eight of these sets of performance standards as results

oriented to little or no extent because they were not linked with any

expected results. For example, EPA officials said that the “professional

attributes” standards for EPA’s criminal investigator position were results

oriented to a great extent because “presenting a positive well-balanced

presence when conducting investigations” resulted in good internal and

external working relationships. EPA officials said these professional

attributes were especially important in risky situations where the

investigator must be tactful in enforcing the law so that the situation does

not get out of hand. However, we characterized this set of standards as

results oriented to little or no extent because it was unclear how

maintaining “a good level of physical conditioning” or displaying “an

appearance and demeanor which is consistent with the position” were

linked to any of EPA’s strategic goals or its overall mission of

environmental protection.

Agency Officials

Identified Some Aids

but Numerous

Barriers to Focusing

on Results

Agency officials identified relatively few aids to focusing their agencies on

results, but said they faced significant barriers in these efforts. Although a

few of the officials’ comments described aids to establishing

results-oriented goals and employee performance standards, most of their

comments focused on barriers to developing results-oriented program

performance measures.

Aids to Focusing on

Results

The factor that agency officials most commonly said aided the

establishment of a results orientation in their agencies was the enactment

of GPRA. Specifically, they said that GPRA-related requirements for

Page 12 GAO/GGD-97-83 Barriers to Managing for Results
B-275122 

outcome-oriented strategic goals and performance measures by

September 1997 forced their agencies to pay more attention to results than

they had prior to passage of the Act. The officials also said that

congressional interest in GPRA’s implementation was an impetus for a

stronger focus on results.

Some agency officials cited other issue-specific legislation that they said

had also encouraged their agencies to increase their results orientation.

For example, agency officials said that the Aviation Security Act of 1990

and the Prescription Drug User Fee Act of 1992 contained specific

results-oriented goals and performance measures for FAA and FDA,

respectively.

Other aids that some agency officials also mentioned to focusing agencies

on results included: training provided to staff on how to develop program

performance measures; feedback from stakeholders (including Congress,

the public, industry, other federal agencies, state and local governments,

and interest groups); and top agency management leadership. For

example, both OSHA and FDA officials said that the heads of their agencies

strongly emphasized using results-oriented performance measures to

focus on results.

Barriers to Focusing on

Results

Agency officials cited numerous barriers that their agencies faced in

becoming results oriented, some of which reportedly hindered them from

establishing results-oriented strategic goals or employee performance

standards. For example, some agency officials said that the conflicting

views of diverse stakeholders and inconsistent statutory requirements

were barriers to reaching agreement on agencies’ goals. In addition,

officials from IRS said that the agency is prevented by federal law from

using performance standards that evaluate its staff on the basis of

enforcement results.8

Most of the barriers that the officials mentioned focused on difficulties

they experienced in developing results-oriented program performance

measures. The barriers that they mentioned generally fell into three broad

categories: (1) problems in collecting program performance data, (2) the

diverse and complex factors that affect agencies’ results and the lack of

control agencies have over some of those factors, and (3) the long time

frames sometimes needed to see the results of agencies’ actions.

8

The Technical and Miscellaneous Revenue Act of 1988 contains the Omnibus Taxpayer Bill of Rights,

which prohibits IRS from evaluating employees on the basis of tax enforcement results.

Page 13 GAO/GGD-97-83 Barriers to Managing for Results
B-275122 

Agency Officials Identified

Problems in Collecting

Program Performance Data

Officials from all five agencies said that their agencies found it difficult to

establish program performance measures because of various problems

they experienced in collecting necessary data. Some of those problems

involved identifying the data they needed to collect in order to

demonstrate their agencies’ results. For example, officials at several

agencies said that they sometimes were put in the difficult position of

trying to gather data to “prove a negative,” such as the number of aircraft

accidents that were avoided by FAA’s efforts to improve airport runways or

the number of people who did not get sick because FDA approved a new

drug or kept an unsafe product off the market. FAA officials said that they

may improve runways after a runway condition was identified as

contributing to an accident. However, because each aircraft accident

results from a unique combination of factors, it is difficult to determine the

results from the improvements made. EPA officials said that in some cases

they did not know how to measure environmental outcomes. For example,

EPA officials said that scientific uncertainty about the effects of exposures

to various levels of pollution and pesticides made it difficult to establish

appropriate measures for aquatic environments. IRS officials said that there

are significant difficulties in attempting to measure the fair amount of tax

due, so it is hard to determine how much IRS should be collecting.9 IRS

officials also said it was difficult to determine the impact of customer

services on tax compliance levels.

Officials from FDA and OSHA pointed out that uncertainties regarding the

causes of illnesses with long latency periods made it more difficult for

their agencies to collect accurate data on mortality and morbidity. OSHA’s

Deputy Director of Policy said it was particularly difficult to determine

whether an illness resulted from conditions in the workplace or from other

factors. For example, he said it is difficult to determine whether a worker

developed cancer after inhaling asbestos fibers at the workplace for 20

years, or from the cigarettes the worker smoked during that period. FDA

officials said that an ongoing study attempting to monitor and define the

sources of foodborne diseases is expected to take years to complete.10

Agency officials noted other problems that made obtaining

results-oriented performance data difficult. For example, OSHA officials

said that it was difficult to collect data and track trends in the number of

9

IRS has been working to improve estimates of taxes owed by conducting research projects and

building databases to allow researchers access to current data. Taxpayer Compliance: Analyzing the

Nature of the Income Tax Gap (GAO/T-GGD-97-35, Jan. 9, 1997).

10This study, known as the Foodborne Diseases Active Surveillance Network, or “FoodNet,” is a

collaborative project among the Centers for Disease Control and Prevention, seven state health

departments, the Department of Agriculture’s Food Safety Inspection Service, and FDA.

Page 14 GAO/GGD-97-83 Barriers to Managing for Results
B-275122 

accidents and injuries that occurred in the construction industry because

construction site locations frequently change. Other agency officials said

that some outcome-oriented performance data must be collected through

voluntary self-reporting methods, which are often susceptible to

underreporting. For example, FDA officials said that people who become ill

from food poisoning may not recognize it as such, or even if they do, they

may not report it to a health official. Consequently, data on the number of

food poisoning incidents may severely undercount such occurrences.11

Similarly, one FAA official said that the data the agency collected on flight

“incidents” were self-reported, and so were likely to be an understatement

of the true number of such incidents.

Agency officials cited still other data-related problems. EPA officials said

that some states were not measuring certain environmental conditions

and, where data were being gathered, the states sometimes differed in the

measures they used. As a result, they said, EPA lacked nationwide

measures for some environmental conditions. FAA officials said that the

small number of aircraft accidents each year made analysis of the data

statistically unreliable. Therefore, they said that it was difficult to identify

trends in the factors contributing to aviation accidents.

Complexity and Lack of

Control Increased the

Difficulty of Measuring

Results

Officials in four of the five agencies we reviewed said it was often difficult

to sort out the interaction of diverse factors that affected the results that

the agencies were trying to achieve. They said that this complexity,

combined with their agencies’ lack of control over many of these factors,

made it difficult to establish meaningful performance measures that

focused solely on their agencies’ contributions toward achieving their

goals. For example, one of EPA’s strategic goals was to achieve clean and

safe water, which is affected by numerous factors that are outside of EPA’s

control (e.g., natural causes including weather conditions). In addition, EPA

shares governmental responsibility to regulate water cleanliness and

safety with other federal agencies and with state and local governments.

As a result, EPA officials said it is difficult to establish performance

measures that gauge only the agency’s contribution to “clean and safe

water.”

Similarly, OSHA officials said that business cycles, the development of new

and safer technologies, changes in laws or regulations, and other factors

over which OSHA has no control affect the number of workplace accidents

11See Food Safety: Information on Foodborne Illnesses (GAO/RCED-96-96, May 8, 1996), where we

found that most cases of foodborne illness go unreported and that existing data therefore understate

the extent of the problem.

Page 15 GAO/GGD-97-83 Barriers to Managing for Results
B-275122 

and injuries occurring each year more than OSHA’s actions. For example,

OSHA’s Deputy Director of Policy said that recent declines in accident and

injury rates in Maine were a function of at least three factors: (1) the

partial closure of a major industrial facility, (2) a tightening of the state’s

worker compensation laws, and (3) OSHA’s programs in that state. He also

said that worker safety data are sometimes influenced by catastrophic

events, such as bombings or aircraft crashes, over which OSHA has no

control. Finally, OSHA officials pointed out that the 2,000 OSHA and state

inspectors can only visit a small fraction of the 6 million workplaces in the

nation. Therefore, they said that they did not think it would be appropriate

to hold OSHA accountable for increases in nationwide workplace injury or

fatality rates.

FAA officials said that the diversity of factors in the aviation system that

contribute to aircraft accidents presented a strong barrier to establishing

results-based performance measures. Although they and academic

researchers have studied aircraft safety issues for years, no techniques

have been developed that can isolate any one factor or group of factors

that can predict an accident. Because a combination of unique factors

generally causes aircraft accidents, an FAA official said that it is “virtually

impossible” to scientifically measure FAA’s effect on aircraft safety. FAA

officials also said that they could not control many of the factors that

contribute to aircraft accidents, such as how alert aircraft pilots feel

during a flight or the weather conditions. Therefore, they said that FAA can

only “influence” aviation safety.

Similarly, FDA officials said that many factors affect whether food and

drugs are safe, some of which are outside of the agency’s control. For

example, although FDA enforces regulations at various stages of the

production process to ensure that food is safe and drugs are safe and

effective, FDA officials said that consumers must ultimately ensure that

food is stored properly and drugs are taken only as prescribed. Although

FDA also regulates the provision of consumer information to ensure the

public is aware of such considerations, FDA officials said that these and

other public health issues are primarily influenced by individual behaviors

that FDA cannot control.

Agency Officials Said Long

Time Frames Are Needed

to Judge Results

Officials from several of the agencies said that it was difficult to develop

annual performance measures and track their agencies’ yearly progress

when the impact of some regulatory actions cannot be seen for years. For

example, OSHA’s Deputy Director of Policy said that the latency period

Page 16 GAO/GGD-97-83 Barriers to Managing for Results
B-275122 

between exposure to a hazardous substance, such as asbestos, and a

resulting illness can be 20 years or more. Therefore, the results of any

actions OSHA takes to reduce exposures to those substances may be

equally long in coming. EPA also said that a significant lag usually exists

between the elimination of a chemical hazard and any corresponding

change in illness rates. Similarly, FAA officials said that it takes several

years to determine the effect that new equipment and facilities have on the

air system’s safety. They also told us that FAA’s impact on safety is only

evident when examining data over a number of years. The officials said

that an increase in the number of accidents in 1 year does not necessarily

mean that FAA’s performance was worse, because the small number of

accidents is affected by a variety of factors that the agency does not

control, such as weather. Likewise, an IRS official said that focusing on

yearly progress may result in solving immediate problems but may divert

attention and reduce resources available for accomplishing longer term

goals. For example, he said that a focus on current compliance levels and

enforcement efforts could lessen the agency’s long-term efforts to increase

taxpayer compliance through education and outreach efforts.

Agency Officials Identified

Various Other Barriers to

Results Orientation

Agency officials cited a variety of other barriers that they said hampered

their agencies’ efforts to focus on results. For example, some agency

officials said that other agency priorities, some of which are also

presidential initiatives, were inconsistent with a results orientation

because they focused on processes rather than results. For example, an

EPA official said that the President’s call to reduce reporting burdens on

the public appeared to be somewhat contrary to the emphasis on gathering

data to measure progress toward results. OSHA officials said that their

agency measured the number of times the negotiated rulemaking process

was used because the OSHA Administrator wanted to emphasize the

importance of responding to a presidential initiative to use this process.

In addition, some agency officials said resource constraints were a barrier

to a results orientation in their agencies. For example, an OSHA official said

that they could only do one or two retrospective studies each year to

determine the effects of prior regulatory initiatives on workplace safety

and health. OSHA officials planned to use the retrospective studies to

provide baseline information on the results of prior regulatory actions.

(See app. I for an example of one of OSHA’s retrospective studies.) FDA

officials said that resource constraints made it difficult to divert resources

from high priority risks to establish a statistically valid baseline for

measuring industry compliance.

Page 17 GAO/GGD-97-83 Barriers to Managing for Results
B-275122 

Also, some agency officials said that their agencies found it difficult to

focus on results when they did not perceive that Congress shared that

orientation. They said some congressional attention continued to focus on

regulatory activities unrelated to results or that the congressional

committees they work with had not yet shown any interest in GPRA.

Agency Comments

and Our Evaluation

We obtained comments from each of the five regulatory agencies in this

review as well as from OMB. The agencies generally agreed with the draft

report’s overall message that regulatory agencies face difficulties in

focusing on results. Officials from FDA, for example, said that the report’s

discussion of the aids and barriers was fair, essentially complete, and

consistent with their perceptions. However, officials from three of the

agencies (OSHA, FDA, and IRS) commented that the “snapshot” approach we

used did not adequately capture the dynamic nature of the process of

establishing results-oriented goals, measures, and standards or the

progress the agencies had made in improving their results orientation.

We recognized in the draft report that our approach provided an interim

assessment at an early stage of implementing new and difficult initiatives

(i.e., GPRA and regulatory reinvention) and that the agencies’ goals,

program performance measures, and some of their employee performance

standards were changing. We did not have historical information that

would have allowed us to determine the extent to which goals and

performance measures that were in effect as of January 1, 1997,

represented advances over past goals and measures. However, we have

expanded the discussion of our methodology to further clarify our purpose

and to describe the related constraints in our scope.

The following section summarizes the comments that we received from

OMB. The comments from each of the agencies are summarized, and our

response included, at the end of each of the relevant appendixes (apps. I

through V).

OMB’s Comments and Our

Evaluation

We met with OMB officials, including the Deputy Administrator of the

Office of Information and Regulatory Affairs, on May 5, 1997, to obtain

their comments on the draft report. In general, the OMB officials thought

that our characterizations of the extent of results orientation of agencies’

strategic goals, program performance measures, and employee

performance standards were fair and reasonable. OMB officials also said

that they agreed with the draft report’s overall message that these

Page 18 GAO/GGD-97-83 Barriers to Managing for Results
B-275122 

regulatory agencies face significant barriers in focusing on results. They

said that the discussion of barriers was important and noted that meeting

GPRA requirements will probably take a number of iterations and

considerable time. The officials said that given the difficulties agencies

face, it was not surprising that there were gaps in agencies’ results

orientation.

The officials said that most of the agencies’ goals had changed since our

review. For example, they said that all of OSHA’s goals have changed and

that OSHA’s new goals are more results oriented than those we examined.

They said that OSHA continues to grapple with the question that OSHA

officials identified in our report about how to measure OSHA’s effects on

national illness, injury, and fatality rates. As stated in our draft report, we

described the agencies’ goals, measures, and standards as of January 1,

1997, to provide an early indication of the extent to which these agencies

were implementing new and difficult initiatives. The draft report

recognized that many of the agencies were making changes to their goals

and measures and some of their performance standards at the time of our

review.

OMB officials recommended that our draft report state more clearly that the

expectation that employee performance standards for frontline regulators

focus on results was directed by the President’s regulatory reinvention

initiative and not by the Results Act. The officials pointed out that there

are many factors interspersed between the two end points we reviewed

(i.e., between the agencies’ goals and measures and the standards that

frontline regulators are held accountable for as they implement the

agencies’ regulatory programs). These factors include statutory provisions,

agencies’ cultures, and the level of individual performance. We agree that

it is important to clearly communicate that the administration’s regulatory

reinvention initiative, not GPRA, set the expectation for results-oriented

employee performance standards. We placed greater emphasis on this

difference throughout the final report by clearly stating in each section on

performance standards that the administration’s regulatory reinvention

initiative set the expectation for results-oriented performance standards.

We are sending copies of this report to the heads of OSHA, FAA, FDA, IRS, and

EPA as well as to the Director of OMB, the Chairmen and Ranking Minority

Members of the House and Senate Appropriations and other interested

congressional committees, and others who may have an interest in this

matter. Copies will also be made available to others upon request.

Page 19 GAO/GGD-97-83 Barriers to Managing for Results
B-275122 

The major contributors to this report are listed in appendix VIII. If you

have any questions, please contact me on (202) 512-8676.

L. Nye Stevens

Director

Federal Management and Workforce Issues

Page 20 GAO/GGD-97-83 Barriers to Managing for Results
Page 21 GAO/GGD-97-83 Barriers to Managing for Results
Contents

Letter 1

Appendix I 

OSHA’s Mission,

Strategic Goals,

Performance

Measures, and

Performance

Standards for

Selected Frontline

Regulator Positions

26

Views on the Results Orientation of OSHA’s Strategic Goals 27

Views on the Results Orientation of OSHA’s Program

Performance Measures

29

Views on the Results Orientation of OSHA’s Performance

Standards for Selected Frontline Regulator Positions

33

OSHA’s Comments and Our Evaluation 34

Appendix II 

FAA’s Mission,

Strategic Goals,

Performance

Measures, and

Performance

Standards for

Selected Frontline

Regulator Positions

36

Views on the Results Orientation of FAA’s Strategic Goals 37

Views on the Results Orientation of FAA’s Program Performance

Measures

40

Views on the Results Orientation of FAA’s Performance

Standards for Selected Frontline Regulator Positions

45

FAA’s Comments and Our Evaluation 46

Appendix III 

FDA’s Mission,

Strategic and

Performance Goals,

and Its Performance

Evaluation Plan

49

Views on the Results Orientation of FDA’s Strategic Goals 50

Views on the Results Orientation of FDA’s Performance Goals 52

FDA Replaced Performance Standards With a Customized

Performance Evaluation Plan

58

FDA’s Comments and Our Evaluation 59

Page 22 GAO/GGD-97-83 Barriers to Managing for Results
Contents

Appendix IV 

IRS’ Mission, Strategic

Objectives, Key

Performance

Indicators, and

Performance

Standards for

Selected Frontline

Regulator Positions

60

Views on the Results Orientation of IRS’ Strategic Objectives 61

Views on the Results Orientation of IRS’ Key Performance

Indicators

62

Views on the Results Orientation of IRS’ Performance Standards

for Selected Frontline Regulator Positions

65

IRS’ Comments and Our Evaluation 68

Appendix V 

EPA’s Mission,

Strategic Goals, and

Performance

Standards for

Selected Frontline

Regulator Positions

70

Views on the Results Orientation of EPA’s Strategic Goals 71

EPA Had No Program Performance Measures in Place at the

Time of Our Review

73

Views on the Results Orientation of EPA’s Performance

Standards for Selected Frontline Regulator Positions

74

EPA’s Comments and Our Evaluation 77

Appendix VI 

Scope and

Methodology

80

Appendix VII 

Comments From

OSHA and IRS

85

Appendix VIII 

Major Contributors to

This Report

89

Tables Table I.1: Differences Between Our and OSHA Officials’

Characterizations

27

Page 23 GAO/GGD-97-83 Barriers to Managing for Results
Contents

Table II.1: Differences Between Our and FAA Officials’

Characterizations

37

Table III.1: Differences Between Our and FDA Officials’

Characterizations

50

Table IV.1: Differences Between Our and IRS Officials’

Characterizations

61

Table V.1: Differences Between Our and EPA Officials’

Characterizations

71

Table VI.1: Initial Framework Provided to Agency Officials 81

Table VI.2: Frontline Regulator Positions and Number of Job

Elements as of January 1, 1997

84

Figures Figure 1: Linking Results Act Requirements to Day-to-Day

Activities

7

Figure I.1: Assumptions Linked Regulatory Approach to Results 28

Figure III.1: Sets of FDA Performance Goals Varied By Strategic

Goal and FDA Program

52

Abbreviations

CID Criminal Investigation Division

CSHO Compliance Safety and Health Officer

CSR customer service representative

ELA establishment licensing agreement

EPA Environmental Protection Agency

FAA Federal Aviation Administration

FDA Food and Drug Administration

FTD federal tax deposits

FTE full time equivalent

GAO General Accounting Office

GPRA Government Performance and Results Act

HACCP Hazard Assessment Critical Control Points

IDRS Integrated Data Retrieval System

IRC Internal Revenue Code

IRS Internal Revenue Service

MQSA Mammography Quality Standards Act

NDA new drug approval

OMB Office of Management and Budget

OSHA Occupational Safety and Health Administration

PDUFA Prescription Drug User Fee Act

PEP performance evaluation plan

PLA product licensing agreement

Page 24 GAO/GGD-97-83 Barriers to Managing for Results
Page 25 GAO/GGD-97-83 Barriers to Managing for Results
Appendix I 

OSHA’s Mission, Strategic Goals,

Performance Measures, and Performance

Standards for Selected Frontline Regulator

Positions

OSHA’s mission is “to assure so far as possible every working man and

woman in the nation safe and healthful working conditions.” OSHA

administers portions of the Occupational Safety and Health Act of 1970,

and is responsible for promulgating federal occupational safety and health

regulations, enforcing occupational safety and health standards, and for

approving plans for the development and enforcement of standards by

state governments.

We assessed the results orientation of OSHA’s strategic goals, performance

measures, and employee performance standards according to the criteria

presented in the scope and methodology appendix of this report. Because

we characterized the extent to which each agency goal and set of

performance measures was results oriented, the application of these

criteria represented a higher standard of results orientation than required

by the Government Performance and Results Act (referred to as “GPRA” or

“the Results Act”). GPRA does not require that all of an agency’s goals or

measures be results oriented. Also, the Results Act does not cover

employee performance standards. We judged each set of performance

standards according to the expectation set by the President in his March 4,

1995, memorandum that these standards should “focus on results, not

process and punishment.” Similar to the way in which GPRA views agencies’

goals and performance measures, we do not believe that each such

standard must be results oriented to achieve the President’s objective.

We applied our criteria to the agencies’ written strategic goals, program

performance measures, and employee performance standards and did not

consider any supplementary or contextual information. In some cases, an

agency’s goals, measures, and standards appeared results oriented on the

surface, but closer examination and strict application of our criteria

resulted in a lower characterization. We recognize that other information

outside of the written statements and outside of our review may have

existed that could have resulted in our characterizing certain goals,

measures, or standards differently.

Both we and OSHA officials characterized the extent of results orientation

of OSHA’s strategic goals, program performance measures, and employee

performance standards for selected frontline regulator positions. As

shown in table I.1, there were differences in how we and OSHA officials

viewed the agency’s goals, measures, and standards. We did not agree with

agency officials that most of OSHA’s goals were somewhat results oriented.

There was more disagreement between us and OSHA regarding the extent

of results orientation of OSHA’s performance measures, in that OSHA

Page 26 GAO/GGD-97-83 Barriers to Managing for Results
Appendix I 

OSHA’s Mission, Strategic Goals,

Performance Measures, and Performance

Standards for Selected Frontline Regulator

Positions

characterized them all as somewhat results oriented, but we characterized

them all as results oriented to little or no extent. We agreed with OSHA

officials, however, regarding the results orientation of the performance

standards for Compliance Safety and Health Officers (CSHOs).

Table I.1: Differences Between Our and OSHA Officials’ Characterizations

Number that OSHA officials considered

results oriented to

Number we considered results oriented

to

Total

A great

extent Some extent

Little or no

extent

A great

extent Some extent

Little or no

extent

Strategic goals 3 0 3 0 0 1 2

Performance measures 3 0 3 0 0 0 3

Performance standards 3 2 1 0 2 1 0

Source: Agency and GAO analysis.

Views on the Results

Orientation of OSHA’s

Strategic Goals

As of January 1, 1997, OSHA had three strategic goals:1

1. “Eliminate hazards through offering partnerships or traditional

enforcement;

2. Implement a common sense strategy, developed in partnership with

stakeholders, for rulemaking and alternative approaches to emerging and

priority safety and health issues; and

3. Focus OSHA programs and service delivery systems using internal and

external partnerships to achieve results.”

Although OSHA characterized each of these goals as results oriented to

some extent, we believed that only the first goal met that standard. We

considered the other two goals to be results oriented to little or no extent.

OSHA’s Deputy Director for Policy said that the agency’s first goal was

somewhat results oriented because hazard elimination should lead to a

reduction in workplace fatalities, injuries, and illnesses—the outcomes or

results OSHA was trying to achieve. He described this goal as a surrogate

for these results, and said that OSHA did not focus directly on reducing

workplace fatalities, injuries, and illnesses because of the difficulty

1

These goals were actually developed for fiscal year 1996. OSHA officials said that they were revising

the agency’s goals and performance measures and that we should use these goals to reflect their

position as of January 1, 1997.

Page 27 GAO/GGD-97-83 Barriers to Managing for Results
Appendix I 

OSHA’s Mission, Strategic Goals,

Performance Measures, and Performance

Standards for Selected Frontline Regulator

Positions

created by other factors such as business cycles and technological

advances which also affected these outcomes.2

 We agreed that this goal

was somewhat results oriented because the goal, although interim in

nature, was plausibly related to the results OSHA was trying to achieve.

However, it was not clear what results OSHA was expecting (i.e., whether

the elimination of a few hazards was sufficient) or how progress would be

assessed.

OSHA officials said that the agency’s second goal regarding implementation

of a “common sense strategy” and “alternative approaches” was somewhat

results oriented because, as figure I.1 shows, they believed that building

partnerships into the rulemaking process would increase the acceptability

of regulations, which would in turn improve regulatory compliance by

both industry and employee groups, and ultimately make workplaces

safer.3

Figure I.1: Assumptions Linked Regulatory Approach to Results

Improved rule

acceptance by

regulated 

entities

Improved

compliance

Fewer

hazards

Fewer

injuries,

illnesses, and

fatalities

Partnership

techniques

Source: Information from agency officials.

However, we considered this goal to be results oriented to little or no

extent because the statement was a strategy rather than an outcome that

the agency was attempting to achieve.4 OSHA has not clearly established

this goal’s link to OSHA’s mission and the goal did not define what results

2

Similarly, OSHA’s fiscal year 1995 report on performance measurement said that increased hazard

identification and abatement was an “intermediate outcome,” capturing the short-term results of

activities that reflect desirable changes in the workplace.

3

OSHA’s Deputy Director for Policy said that this goal may also help OSHA change its regulatory

framework. For example, he said that in some instances regulations might not be the best way to

achieve a desired result, so OSHA might issue guidelines instead of regulations.

4

Similarly, in Managing For Results: Achieving GPRA’s Objectives Requires Strong Congressional Role

(GAO/T-GGD-96-79, Mar. 6, 1996), we said that OSHA’s goal to “focus resources on achieving

workplace hazard abatement through strong enforcement and innovative incentive programs”

represented a strategy rather than a measurable goal.

Page 28 GAO/GGD-97-83 Barriers to Managing for Results
Appendix I 

OSHA’s Mission, Strategic Goals,

Performance Measures, and Performance

Standards for Selected Frontline Regulator

Positions

OSHA hoped to achieve, and therefore did not permit assessment of the

agency’s progress. OSHA could not provide any evidence that these

approaches actually result in increased compliance or contribute to the

agency’s desired results of reducing fatalities, injuries, and illnesses in the

workplace.

OSHA officials said that their third goal—to “focus OSHA programs and

service delivery systems using internal and external partnerships to

achieve results”—was established in reaction to the administration’s

desire for OSHA to improve its customer service by becoming more

responsive to its customers. The officials said that they considered the

goal somewhat results oriented because they believed that hazards would

be addressed more quickly through partnerships and hazard reduction

would ultimately result in fewer workplace fatalities, injuries, and

illnesses. However, we believed that this goal was results oriented to little

or no extent because it was unclear (“focus OSHA programs and ...

systems”) and it utilized a strategy (“partnerships”) to achieve unspecified

“results.” Therefore, it was also impossible to assess OSHA’s progress

toward achieving any results.

Views on the Results

Orientation of OSHA’s

Program Performance

Measures

OSHA officials said that the sets of program performance measures OSHA

had adopted for the three strategic goals were each results oriented to

some extent. However, we believed that each of these sets of performance

measures was results oriented to little or no extent.

Views on the Results

Orientation of Measures

Associated With OSHA’s

Strategic Goal to

“Eliminate Hazards”

As of January 1, 1997, the performance measures that OSHA used to assess

progress toward its first strategic goal, “to eliminate hazards through

offering partnerships or traditional enforcement,” were:

• “Percentage of programmed inspections that result in the identification of

significant hazards;

• Percentage of initial consultation visits that result in the identification of

significant hazards;

• Median time for hazard abatement for inspections;

• Median time for hazard abatement for consultation visits;

• Median time to respond to requests from smaller employers in high hazard

industries and operations;

• Average cost per person trained through the targeted training grant

program; and

Page 29 GAO/GGD-97-83 Barriers to Managing for Results
Appendix I 

OSHA’s Mission, Strategic Goals,

Performance Measures, and Performance

Standards for Selected Frontline Regulator

Positions

• Average evaluation score on courses offered by the OSHA Training Institute

and Education Centers.”

OSHA officials characterized this set of program performance measures as

somewhat results oriented because they believed that all of the measures

in the set were related to the goal of eliminating hazards. However, OSHA’s

Deputy Director for Policy also said that OSHA is working to make the

measures even more results oriented, and has taken other actions to

measure the agency’s effects and focus on achieving particular outcomes.

For example, he said that OSHA has done some retrospective analyses to

identify the results of their efforts. In one such study OSHA found that

within 5 years of the promulgation of OSHA’s general industry lead standard

in 1978, the number of workers in the lead smelting and battery

manufacturing industries who had high-lead concentrations in their blood

dropped by 66 percent, from about 19,000 to about 6,500. The Deputy

Director also told us that OSHA is in the process of redesigning its field

offices, and that each redesigned office has a local problem-solving

component with outcome measures. One of these redesigned field offices

is using data from the state registry on lead poisoning to measure OSHA’s

impact on the health and safety of construction workers who paint bridges

(which involves the removal of lead paint). The field office is conducting a

training program, and next year they will determine whether there are

significantly fewer reports of lead poisoning in the state registry.

We believed that the set of program performance measures associated

with OSHA’s goal of eliminating hazards was results oriented to little or no

extent because, although they initially appeared to be results oriented, the

measures were not clearly and unambiguously related either to the goal or

to the agency’s desired results of reducing workplace fatalities, injuries,

and illnesses. The set of measures did not address all aspects of the goal

and also did not provide relevant information for assessing progress

toward the goal or the agency’s mission.

For example, the first two measures—the percentage of inspections or

consultations that resulted in the identification of significant hazards—did

not reflect the actual number of significant hazards found, only how often

inspections or consultations revealed at least one such hazard. One

inspection that identified 50 significant hazards would be recorded the

same as another inspection that identified only 1 significant hazard. The

measures did not reflect the number of hazards that were eliminated in

those workplaces or whether the OSHA inspections and consultations

resulted in any reduction in fatalities, injuries, or illnesses. Furthermore, it

Page 30 GAO/GGD-97-83 Barriers to Managing for Results
Appendix I 

OSHA’s Mission, Strategic Goals,

Performance Measures, and Performance

Standards for Selected Frontline Regulator

Positions

was unclear whether changes in these measures would indicate that

workplaces are becoming more or less hazardous. For example, if

30 percent of the inspections conducted in 1995 resulted in the

identification of a significant hazard, and that figure rose to 50 percent in

1996, it would not be clear whether the 20 percentage point increase

reflected increased diligence and effectiveness on the part of OSHA

inspectors, better targeting of OSHA inspections, or that workplaces had

become more hazardous.

The third and fourth measures—“median time for hazard abatement” for

inspections and consultation visits—were also not as results oriented as

they appeared. These measures reflected the time OSHA gave companies to

abate a hazard, not the actual amount of time it took until the hazard was

abated or even whether the hazard ever was abated.5

 Similarly, the fifth

measure gauged the time it took OSHA to respond to requests from smaller

employers in high hazard industries and operations and did not measure

whether hazards were eliminated or whether workplace fatalities, injuries,

or illnesses were reduced as a result of their response. Also, OSHA’s 1995

performance report showed that smaller employers in the manufacturing

sector had the lowest percentage of inspections with significant hazards of

any size group. Therefore, this measure appears to have focused on the

group of employers with the fewest significant hazards.

We also did not believe that the training-related measures that OSHA

associated with this goal were clearly related to hazard abatement. For

example, although the average cost per person trained through the

targeted training grant program may be a useful measure in other respects

(e.g., a cost-effectiveness analysis), the resultant dollar figure may bear

little relationship to the elimination of hazards or reduction of workplace

fatalities, injuries, or illnesses. Similarly, average evaluation scores on

OSHA training may have had relevance in terms of selecting one course or

instructor over another, but those scores may be totally unrelated to the

agency’s goal or desired results.6

5

OSHA’s 1995 performance report indicated that OSHA might move to measure actual time to abate

hazards rather than the amount of time assigned for abatement.

6

However, the provision of relevant training (not the cost per person trained or average evaluation

scores) can be a legitimate and results-oriented measure for hazard abatement or even the reduction of

fatalities, injuries, and illnesses when the linkage between training and the result is demonstrated. For

example, OSHA’s fiscal year 1995 Performance Measurement Report cited a study examining the

effects of OSHA-funded training on logging safety provided to 250 companies in Idaho and Washington

during fiscal year 1993 and the first quarter of fiscal year 1994 that discovered that accident claims and

lost work days per year declined after the safety training was provided.

Page 31 GAO/GGD-97-83 Barriers to Managing for Results
Appendix I 

OSHA’s Mission, Strategic Goals,

Performance Measures, and Performance

Standards for Selected Frontline Regulator

Positions

Views on the Results

Orientation of the Measure

for OSHA’s Second Goal on

Rulemaking and

Alternative Approaches

OSHA used one performance measure, “the percentage of rulemaking

activity on the regulatory agenda undertaken as negotiated rulemaking,” to

assess the agency’s progress in meeting its second strategic goal, to

“implement a common sense strategy, developed in partnership with

stakeholders, for rulemaking and alternative approaches to emerging and

priority safety and health issues.”

A negotiated rulemaking brings together interested parties who draft a

proposed rule that then goes through notice and comment procedures.7

OSHA officials characterized this measure as results oriented to some

extent because, as noted previously and depicted in figure I.1, they

believed that using partnership techniques such as negotiated rulemaking

would ultimately make workplaces safer. However, OSHA officials could

provide no evidence to support this assumption. Therefore, we considered

this measure to be results oriented to little or no extent because it was not

clearly and unambiguously related to OSHA’s mission and would not

provide relevant information to permit assessment of the agency’s

programs. Also, according to OSHA’s fiscal year 1995 Performance

Measurement Report, the negotiated rulemaking process was applicable to

only a portion of proposed rules, and therefore this measure could not be

used for all of OSHA’s regulatory activities. Finally, the measure did not

address all aspects of the goal in that it did not gauge any “alternative

approaches” to regulation.

Views on the Results

Orientation of the Measure

for OSHA’s Third Goal to

Focus OSHA Programs and

Service Delivery Systems

OSHA used one performance measure, “the median time to respond to

complaints,” to measure progress toward its third goal, to “focus OSHA

programs and service delivery systems using internal and external

partnerships to achieve results.” OSHA officials characterized this measure

as results oriented to some extent because they said it reflected one of

OSHA’s responses to the administration’s reinvention initiatives to improve

its customer service.8

 We characterized the measure as results oriented to

little or no extent because it did not clearly relate to OSHA’s goal or mission

and did not provide relevant information regarding progress toward the

goal. The measure focused on how long it took OSHA to respond to a

7

See the Negotiated Rulemaking Act of 1990, 5 U.S.C. 561-570.

8

OSHA has changed its policy so that when it receives a complaint, it determines whether to

immediately inspect or investigate the alleged hazards. If OSHA determines an investigation is

appropriate, the agency will telephone the employer, describe the alleged hazards, and follow up with

a fax or a letter. The employer can respond in kind, identifying any problems found and noting

corrective actions taken or planned. An adequate response generally negates the need for an

inspection. According to OSHA officials, lower priority complaints can be resolved through the

phone/fax procedure, therefore allowing a more effective use of OSHA inspection resources.

Page 32 GAO/GGD-97-83 Barriers to Managing for Results
Appendix I 

OSHA’s Mission, Strategic Goals,

Performance Measures, and Performance

Standards for Selected Frontline Regulator

Positions

complaint, not how long it took to abate the hazard that originated the

complaint or even whether the hazard was eliminated. Also, there were no

other measures regarding the contributions of internal or external

partnerships to any results achieved, so the goal was incompletely

addressed.

Views on the Results

Orientation of OSHA’s

Performance

Standards for

Selected Frontline

Regulator Positions

In contrast to strategic goals and program performance measures, the

administration’s regulatory reinvention initiative, not GPRA, set the

expectation for agencies to focus on results when measuring the

performance of their frontline regulators. Two of OSHA’s frontline regulator

positions were safety inspectors and health inspectors.9

 Although both

positions were covered by the same set CSHO standards, the standards

varied somewhat by OSHA regional office. OSHA officials said we could

select the standards used in any of its regions, so we used the San

Francisco regional office’s CSHO standards in this study.

As of January 1, 1997, OSHA had three CSHO job elements, with one

descriptive performance standard for each element. They were:

1. “Building Partnerships and Improving the Agency’s Effectiveness.

Participates in planning and executing safety and health program activity

that serves our customer and improves OSHA’s mission of assuring worker

protection.

2. On Site Inspection. Performs inspections in accordance with agency

policy and organizational goals. The scope and focus of the inspection is

based on the type of inspection, the hazards found and the effectiveness of

the employer’s safety and health program.

3. Case Management. Plans, prepares, organizes and documents

enforcement and non-enforcement cases in accordance with agency policy

and organizational goals.”

Each standard also had a set of “performance measurement

considerations” that elaborated the standard. For example, the

considerations for the first standard included “promotes voluntary

compliance efforts” and “maintains and improves safety/health knowledge

and skills.”

9

Although the President’s March 4, 1995, memorandum on his regulatory reinvention initiative did not

define “frontline regulators,” we defined them as regulatory staff who work directly with the regulated

community by enforcing regulations. OSHA officials identified these two frontline regulator positions

for this review.

Page 33 GAO/GGD-97-83 Barriers to Managing for Results
Appendix I 

OSHA’s Mission, Strategic Goals,

Performance Measures, and Performance

Standards for Selected Frontline Regulator

Positions

OSHA officials characterized the performance standard for the “building

partnerships” and “on-site inspection” job elements as results oriented to a

great extent because they believed they were directly linked with OSHA’s

mission to assure worker protection. OSHA officials characterized the

performance standard for the “case management” element as somewhat

results oriented because it was less directly linked to the agency’s mission.

We agreed with OSHA’s characterizations of these three performance

standards. We viewed the first two performance standards as greatly

results oriented because they explicitly referred to OSHA’s mission and

goals, and a majority of the associated performance measurement

considerations addressed results. For example, in “building partnerships”

the CSHOs were expected to identify and design intervention tools “that will

effect the greatest reduction of illness, injury, and death.” We agreed with

OSHA officials that the third performance standard regarding case

management was somewhat results oriented because the standard and the

associated performance measurement considerations were only indirectly

related to results. For example, both the standard and the considerations

were primarily focused on internal procedures such as documenting cases,

demonstrating knowledge of OSHA standards, and applying professional

judgment—actions that may or may not lead to safer workplaces.

OSHA’s Comments

and Our Evaluation

We met with OSHA officials, including the Acting Director of Policy, on

April 25, 1997, to receive their comments. OSHA officials said that they

generally agreed with the draft report’s overall message regarding the

difficulties regulatory agencies have experienced in becoming results

oriented. They provided one technical correction, which was included in

the final report.

Both in our meeting with OSHA officials and in OSHA’s written comments

(see app. VII), OSHA emphasized that its approach to focusing on results

began with the development of processes and data systems, not the

development of results-oriented goals. They said that agencies can

legitimately be in different stages of results orientation because of

differences in the approaches agencies took to focusing on results.

Therefore, the officials said they believed that any comparison of the

information presented about the agencies would be misleading and

inappropriate. They also recommended that the final report more

prominently reflect that not all goals need to be results oriented.

Page 34 GAO/GGD-97-83 Barriers to Managing for Results
Appendix I 

OSHA’s Mission, Strategic Goals,

Performance Measures, and Performance

Standards for Selected Frontline Regulator

Positions

We agree that agencies can use different approaches to focus on results

and that they can legitimately vary in the degree to which they are results

oriented at a particular point in time. As reflected in the draft report, we

expected such variance in the early stages of implementing GPRA and the

regulatory reinvention initiative and explicitly recognized the interim

nature of the characterizations we made as agencies continued to change

their goals, measures, and standards. We removed tables in the draft

report that were intended only to summarize information across agencies

because OSHA’s comments led us to realize that the tables could have the

unintended effect of leading a reader to compare the agencies with one

another and to assume that all goals, measures, and standards are required

to be results oriented.

OSHA officials also said that the discussion in the draft report about the

barriers agencies face in focusing on results should be highlighted more

prominently in the results-in-brief section. In response, we included more

information in the results-in-brief section of the final report describing the

barriers agencies identified and also summarized the information about

the agencies’ goals, measures, and standards in a more general manner.

In its comment letter, OSHA said that its fiscal year 1995 Performance

Measurement System Framework had long-term outcome measures of

injuries, illnesses, and fatalities prevented that were not recognized in the

draft report. However, the letter also said that “these long term outcomes

may not be immediately evident and therefore were not included in OSHA’s

fiscal year 1996 goals and objectives.” Therefore, OSHA apologized “for any

misunderstanding or miscommunication of information related to our

measures.” The scope of this review included only OSHA’s fiscal year 1996

goals and measures, not those for fiscal year 1995. When asked to confirm

that our information on OSHA’s fiscal year 1996 goals and program

performance measures was accurate and complete, both before doing our

analysis and again in January 1997, OSHA officials did not identify any

additional measures for our review.

Page 35 GAO/GGD-97-83 Barriers to Managing for Results
Appendix II 

FAA’s Mission, Strategic Goals, Performance

Measures, and Performance Standards for

Selected Frontline Regulator Positions

FAA’s mission is to provide “a safe, secure, and efficient global aerospace

system that contributes to the national security and the efficiency of U.S.

aerospace” and be “responsive to the dynamic nature of customer needs,

economic conditions, and environmental concerns.”

We assessed the results orientation of FAA’s strategic goals, performance

measures, and employee performance standards according to the criteria

presented in the scope and methodology appendix of this report. Because

we characterized the extent to which each agency goal and set of

performance measures was results oriented, the application of these

criteria represented a higher standard of results orientation than required

by the Results Act. The Act does not require that all of an agency’s goals or

measures be results oriented. Also, GPRA does not cover employee

performance standards. We judged each set of performance standards

according to the expectation set by the President in his March 4, 1995,

memorandum that these standards should “focus on results, not process

and punishment.” Similar to the way in which the Act views agencies’

goals and performance measures, we do not believe that each such

standard must be results oriented to achieve the President’s objective.

We applied our criteria to the agencies’ written strategic goals, program

performance measures, and employee performance standards and did not

consider any supplementary or contextual information. In some cases an

agency’s goals, measures, and standards appeared results oriented on the

surface, but closer examination and strict application of our criteria

resulted in a lower characterization. We recognize that other information

outside of the written statements and outside of our review may have

existed that could have resulted in our characterizing certain goals or sets

of measures and standards differently.

Both we and FAA officials characterized the extent of results orientation of

FAA’s strategic goals, program performance measures, and employee

performance standards for selected frontline regulator positions. As

shown in table II.1, there were differences in how we and FAA officials

viewed the agency’s goals, measures, and standards. Generally, we agreed

that most of FAA’s strategic goals were results oriented. There was some

disagreement between us and FAA regarding the extent of results

orientation of FAA’s performance measures, although we both

characterized the measures as less results oriented than the goals. There

was a sharp disagreement in ours and the officials’ characterizations of

FAA’s performance standards. FAA officials said that five of the six sets of

performance standards were greatly results oriented. However, we

Page 36 GAO/GGD-97-83 Barriers to Managing for Results
Appendix II 

FAA’s Mission, Strategic Goals,

Performance Measures, and Performance

Standards for Selected Frontline Regulator

Positions

characterized one set as greatly results oriented, one set as somewhat

results oriented, and the remaining four sets as results oriented to little or

no extent.

Table II.1: Differences Between Our and FAA Officials’ Characterizations

Number that FAA officials considered

results oriented to

Number we considered results oriented

to

Total

A great

extent Some extent

Little or no

extent

A great

extent Some extent

Little or no

extent

Strategic goals 9 3 6 0 3 5 1

Performance measuresa 6 4 2 0 0 4 2

Performance standards 6 5 1 0 1 1 4

aThree of FAA’s strategic goals had no program performance measures.

Source: Agency and GAO analysis.

Views on the Results

Orientation of FAA’s

Strategic Goals

We and FAA officials reviewed 9 of FAA’s 10 strategic goals in place as of

January 1, 1997:1

1. “System safety—zero accidents. Eliminate accidents2 and incidents3 in

aviation and protect public safety and property in space transportation

systems by targeting the most critical areas.

2. Security—zero incidents. Eliminate security incidents in the aviation

system.

3. Human factors safety—zero accidents. Eliminate human factors as a

causal factor in accidents and incidents.

1

FAA officials said that one of the agency’s strategic goals—“promote active, vital, and efficient

aviation and space transportation industries”—would soon be eliminated because of a change in the

agency’s mission contained in the Federal Aviation Reauthorization Act of 1996. This act changed

FAA’s mission by eliminating the dual mandate of promoting both the “development” and “safety” of

U.S. air commerce. Therefore, we decided to eliminate it from our review.

2

An aircraft “accident” is defined by regulation as “an occurrence associated with the operation of an

aircraft which takes place between the time any person boards the aircraft with the intention of flight

until all such persons have disembarked, and in which any person suffers death or serious injury as a

result of being in or upon the aircraft or by direct contact with the aircraft or anything attached

thereto, or in which the aircraft receives substantial damage.”

3

An aviation “incident” is defined by regulation as “an occurrence other than an accident associated

with the operation of an aircraft which affects or could affect the safety of operations.”

Page 37 GAO/GGD-97-83 Barriers to Managing for Results
Appendix II 

FAA’s Mission, Strategic Goals,

Performance Measures, and Performance

Standards for Selected Frontline Regulator

Positions

4. System capacity. Meet the system capacity needs for air and space

transportation safely and efficiently through near-term actions targeted at

specific problems and a long-term comprehensive program of research,

planning, and investment matching user needs.

5. Achieve safe, secure, and efficient global air and space transportation

systems by promoting international cooperation and harmonization.

6. Provide strong leadership regarding the environmental impacts of

aviation and commercial space transportation.

7. Manage resources and investments to control costs and increase

productivity, efficiency, and effectiveness, while promoting safety and

customer service.

8. Transform FAA into the model Federal workplace.

9. Establish and maintain mutual lines of communications with the public

and with employees to promote understanding, awareness, and

cooperation, and to serve the interests of the traveling public.”

FAA officials characterized the first three goals as results oriented to a

great extent and the six remaining goals as somewhat results oriented. We

considered the first three goals to be results oriented to a great extent, five

goals somewhat results oriented, and one goal results oriented to little or

no extent.

According to FAA officials, the agency’s first strategic goal was results

oriented to a great extent because it established a clear definition of the

results expected (“zero accidents”). An FAA official said the agency

recognized that, although it was possible to achieve zero accidents for

commercial aviation during 1 year, in the long term an accident would

likely occur. However, the official said that this goal was created in

response to the administration’s desire to ensure aviation safety and

express no tolerance for aviation accidents. He also said that FAA

management did not want to imply that having accidents would be

acceptable or that anything other than achieving zero accidents was

desirable. Finally, the official said that the goal reflected a quality

management perspective of setting high expectations and striving for the

best results that could be achieved. We agreed that the goal was greatly

results oriented because it was directly linked to FAA’s mission of

providing a safe and secure aerospace system, contained a clear definition

Page 38 GAO/GGD-97-83 Barriers to Managing for Results
Appendix II 

FAA’s Mission, Strategic Goals,

Performance Measures, and Performance

Standards for Selected Frontline Regulator

Positions

of expected results, and permitted assessment of the agency’s progress

toward the goal.

FAA officials said the agency’s second strategic goal of eliminating security

incidents (i.e., no hijackings or explosions aboard aircraft) was also

results oriented to a great extent because, like the first goal, FAA

deliberately established a clear outcome-related goal of eliminating

security incidents in the aviation system. We agreed that this goal was

greatly results oriented because it provided a clear definition of what

outcomes were expected, was measurable, and was unambiguously linked

to FAA’s mission.

FAA’s third strategic goal, “human factors safety—zero accidents,” was a

subset of the first goal, but FAA established it as a separate goal to enable

the agency to isolate human factors from other causes of accidents. FAA

officials characterized this goal as results oriented to a great extent. We

agreed with that assessment because the goal was directly linked to FAA’s

mission, was clear in the results expected, and permitted assessment of

the agency’s progress toward the result.

FAA officials characterized the fourth strategic goal (“to meet system

capacity”) as somewhat results oriented because of the difficulties they

experienced in defining aviation system “capacity.” An FAA official said

that defining system capacity solely in terms of the increased number of

aircraft traveling through the nation’s airspace or the number of additional

airport runways would not be sufficiently inclusive. He said that any goal

relating to system capacity would need to reflect other considerations as

well, such as delays, capital expenditures for aviation facilities, and the

needs of the aviation community. We agreed with FAA that the fourth

strategic goal was somewhat results oriented because, although it was

clearly linked to FAA’s mission, it did not clearly identify the results FAA

was trying to achieve regarding system capacity, nor did it provide a

means of assessing the agency’s progress in achieving the results. We also

agreed with FAA officials’ characterization that the fifth strategic goal, to

“achieve safe, secure, and efficient global air and space transportation

systems,” was somewhat results oriented. Although the goal was linked to

the agency’s mission, it did not clearly define the results expected and did

not provide a means of assessing those results.

FAA officials characterized FAA’s sixth strategic goal, to “provide strong

leadership regarding the environmental impacts of aviation and

commercial space transportation,” as results oriented to some extent.

Page 39 GAO/GGD-97-83 Barriers to Managing for Results
Appendix II 

FAA’s Mission, Strategic Goals,

Performance Measures, and Performance

Standards for Selected Frontline Regulator

Positions

However, they noted that the term “strong leadership” was subject to a

variety of interpretations by interest groups and other stakeholders, such

as aircraft manufacturers, airline operators, and the general public. We

characterized this goal as somewhat results oriented because, although it

was linked to FAA’s mission, it did not provide any description of what

would result from the “strong leadership” that FAA was to provide or how

to assess the effects of this leadership.

FAA characterized the seventh (“managing resources”), eighth

(“transforming FAA into a model Federal workplace”), and ninth

(“establishing and maintaining lines of communications”) strategic goals

as somewhat results oriented because they were written in broad terms

and not easily measurable. We agreed with FAA officials that all three

strategic goals were not measurable, although two of the goals (“managing

resources” and “establishing and maintaining lines of communications”)

were linked to the agency’s mission. As such, we agreed with FAA officials’

characterization that these two goals were somewhat results oriented. We

considered the goal of transforming FAA into a model federal workplace as

results oriented to little or no extent because it was not linked to FAA’s

mission and did not provide a clear definition of what results were

expected or permit assessment of the agency’s progress.

Views on the Results

Orientation of FAA’s

Program Performance

Measures

FAA officials said that their agency’s program performance measures were

evolving and they planned to modify these measures to better focus on

results before submitting them to the Office of Management and Budget

(OMB) for review in September 1997 as part of GPRA implementation.

However, as of January 1, 1997, FAA had sets of program performance

measures for six of its nine strategic goals. FAA officials said that FAA was

still developing measures for the agency’s seventh and ninth goals. FAA

officials did not identify any performance measures for the agency’s fifth

goal.

FAA officials said that the sets of program performance measures for the

first four strategic goals were results oriented to a great extent and the

sets of measures for the sixth and eighth strategic goals were somewhat

results oriented. We considered three of the first four sets of measures as

somewhat results oriented and the sets of measures for the third and

eighth strategic goals to be results oriented to little or no extent. We

agreed with FAA officials that the set of measures for the sixth strategic

goal was somewhat results oriented.

Page 40 GAO/GGD-97-83 Barriers to Managing for Results
Appendix II 

FAA’s Mission, Strategic Goals,

Performance Measures, and Performance

Standards for Selected Frontline Regulator

Positions

Views on the Results

Orientation of the

Measures for FAA’s First

and Third Goals

The program performance measures FAA used for both its first (“system

safety—zero accidents”) and third (“human factors safety—zero

accidents”) strategic goals were:

• “percent reduction in accident rate;

• number of fatalities per million passenger miles;

• percent reduction in operational error4 rates;

• percent reduction of near mid-air collisions;5 and

• percent reduction in runway incursions.”6

FAA officials said they characterized this set of program performance

measures as greatly results oriented because the measures enabled the

agency to track accidents and fatalities and were directly related to the

FAA’s first and third strategic goals. We considered this set of measures to

be somewhat results oriented with regard to FAA’s first goal because it did

not address all aspects of the goal. For example, although part of the goal

was the elimination of all aviation incidents, the incident-related measures

focused on only certain types of incidents (operational errors, midair

collisions, and runway incursions) and not other types (e.g.,

weather-related incidents). Also, the accident rate measure (the number of

accidents per 100,000 flight hours) was not clearly related to the goal of

“zero accidents.” For example, the absolute number of accidents could

actually increase while the percentage of accidents per 100,000 flight

hours decreased because of a large increase of flight hours. We considered

this set of measures to be results oriented to little or no extent with regard

to FAA’s third goal—human factors safety. The measures were not clearly

linked to the goal of eliminating human factors in accidents and incidents

because none of them isolated human factors in the results they were

measuring. For example, the number of fatalities could be a function of

factors totally unrelated to human factors.

4

An operational error is an occurrence attributable to an element of the air traffic control system that

results in less than the applicable separation minima between two or more aircraft, or between an

aircraft and terrain or obstacles and obstructions. Obstacles include vehicles, equipment, and/or

personnel on runways.

5

A near midair collision is an incident associated with the operation of an aircraft in which a possibility

of collision occurs as a result of proximity of less than 500 feet to another aircraft, or a report is

received from a pilot or flight crew member stating that a collision hazard existed between two or

more aircraft.

6

A runway incursion is any occurrence at an airport involving an aircraft, vehicle, person, or object on

the ground that creates a collision hazard or results in loss of separation with an aircraft taking off,

intending to take off, landing, or intending to land.

Page 41 GAO/GGD-97-83 Barriers to Managing for Results
Appendix II 

FAA’s Mission, Strategic Goals,

Performance Measures, and Performance

Standards for Selected Frontline Regulator

Positions

Views on the Results

Orientation of the

Measures for FAA’s

Security Goal

FAA’s program performance measures for FAA’s second strategic goal

(“security—zero incidents”) were:

• “Percent improvement in detection of complex improvised explosive

devices through all threat vectors;7

• percent increase in the system’s ability to sustain 100 percent compliance

with contingency measures;8

 and

• percent decrease in vulnerability to existing or new threats.”9

FAA officials said they characterized this set of performance measures as

results oriented to a great extent because the measures were quantitative

and reflected improvements made to the security system. Although we

agree that this set of measures was related to the agency’s security system

goal, we considered it somewhat results oriented because it did not

measure all aspects of system security. For example, an FAA official said

that the set’s first measure will initially measure only the detection

performance of “threat vectors” with screening equipment (e.g., x-ray

machines) and may not include other threat sources. Also, the set’s first

measure refers only to explosive devices and not other weapons such as

handguns.

Views on the Results

Orientation of the

Measures for FAA’s System

Capacity Goal

The program performance measures for FAA’s fourth strategic goal

regarding system capacity were:

• “Percent increase in Instrument Flight Rule/Visual Flight Rule capacity

ratio;10

7

This measure refers to an air carrier or airport security system’s ability to detect explosive devices

attempting to reach a target through different approaches, such as within a checked bag, cargo, or with

a person.

8

“Contingency measures” refers to security countermeasures to be taken by air carriers and aviation

industry personnel as directed by FAA through its emergency authority. These countermeasures are

previously agreed-upon actions based on assessments made by law enforcement and intelligence

agencies. FAA assesses a security system’s implementation of a countermeasure through surveillance,

interviews, audits, and testing.

9

This measure refers to FAA’s airport-by-airport vulnerability assessments and use of risk assessment

methodologies and tools to determine the reduction of risk when mitigating or closing vulnerabilities

in airports’ security systems.

10The Instrument Flight Rule/Visual Flight Rule capacity ratio estimates an airport’s capacity to handle

air traffic during good and poor weather conditions.

Page 42 GAO/GGD-97-83 Barriers to Managing for Results
Appendix II 

FAA’s Mission, Strategic Goals,

Performance Measures, and Performance

Standards for Selected Frontline Regulator

Positions

• percent increase in number of Global Positioning System non-precision

and precision approaches;11

• percent reduction in ground movement time;12

• percent reduction in system induced delays per number of operations;

• percent reduction in equipment caused delays per number of operations;

and

• percent reduction in weather induced delays per number of operations.”

FAA officials characterized this set of program performance measures as

greatly results oriented because it focused on how the end user actually

experiences the aviation system. However, FAA officials also told us that

the agency recognized that this set of measures did not include indicators

for some capacity issues that were of concern to the aviation system’s

users, including the system’s flexibility, accessibility, and predictability.13

We concluded that the measures do not provide a means of assessing how

FAA will “meet the system capacity needs” in the near and long term as

described in the system capacity strategic goal. Therefore, we considered

this set of measures to be somewhat results oriented.

Views on the Results

Orientation of Measures

for FAA’s Environmental

Responsibility Goal

The program performance measures for FAA’s sixth strategic

goal—“provide strong leadership regarding environmental impacts of

aviation and commercial space transportation”—were as follows:

For industry operations:

• “Percent reduction in number of residents exposed to significant airport

noise levels.”

For FAA operations:

• “Percent of facilities which have received safety inspections,

• percent of facilities which have received environmental inspections,

• number of fuel tanks brought into compliance with EPA requirements, and

• number of facilities audited for energy conservation.”

11The Global Positioning System is a satellite-based navigation system which can provide aircraft with

extremely precise position information.

12The “percent reduction in ground movement time” measures the amount of time aircraft taxi in and

out of the airport, a means of measuring a delay in the aviation system.

13FAA has recently developed a new set of measures for system capacity. An FAA official said that this

new set of measures will be included in FAA’s Air Traffic Services’ Fiscal Year 1999 Business

Performance Plan.

Page 43 GAO/GGD-97-83 Barriers to Managing for Results
Appendix II 

FAA’s Mission, Strategic Goals,

Performance Measures, and Performance

Standards for Selected Frontline Regulator

Positions

FAA officials characterized this set of program performance measures as

somewhat results oriented because they believed that several of the

measures assessed FAA’s compliance with environmental laws rather than

being focused on results. However, because FAA’s environmental goal

lacked a clear definition of the outcomes the agency was trying to achieve,

it was difficult for us to determine whether this set of measures was

clearly linked to the goal. Although some of the measures may relate to the

goal (e.g., “number of fuel tanks brought into compliance with EPA

requirements”), we considered this set of measures somewhat results

oriented because many of the measures did not permit assessment of

progress toward the agency’s goal. For example, the percent of facilities

that have received safety or environmental inspections may bear no

relation to whether those facilities are in compliance with OSHA or EPA

requirements.

Views on the Results

Orientation of the

Measures Associated With

the Goal on the FAA

Workplace

The program performance measures for FAA’s eighth strategic

goal—“transform FAA into the model Federal workplace”—were:

• “Conduct an employee attitude survey of FAA employees and register a

70 percent positive response on overall job satisfaction;

• implement national actions to help manage diversity and transform the FAA

into a model Federal workplace; and

• through demonstrated actions and accomplishments, the FAA’s workforce

demographics will reflect positive trends or increases with regard to

workforce representation for minorities, women, and people with

disabilities.”

FAA officials characterized this set of measures as somewhat results

oriented because it did not provide a “measurable outcome” of what FAA

was trying to achieve. We agreed with FAA officials that the set did not

provide a measurable outcome. We also believe that the set did not

address all aspects of the strategic goal, such as the amount of support for

innovation or risk taking within the agency. Perhaps most importantly,

though, the set of performance measures and its related strategic goal

were not linked to the agency’s mission. Therefore, we considered this set

of measures to be results oriented to little or no extent.

Page 44 GAO/GGD-97-83 Barriers to Managing for Results
Appendix II 

FAA’s Mission, Strategic Goals,

Performance Measures, and Performance

Standards for Selected Frontline Regulator

Positions

Views on the Results

Orientation of FAA’s

Performance

Standards for

Selected Frontline

Regulator Positions

In contrast to strategic goals and program performance measures, the

administration’s regulatory reinvention initiative, not GPRA, set the

expectation for agencies to focus on results when measuring the

performance of their frontline regulators. Two of FAA’s frontline regulator

positions were air traffic control specialists and aviation safety

inspectors.14 Air traffic control specialists are responsible for issuing

control instructions and advice to pilots in the vicinity of airports as well

as guiding aircraft operations en route along the airways. Aviation safety

inspectors administer and enforce regulations concerning the

airworthiness of aircraft and aircraft systems and the competence of

pilots, mechanics, and other aviation personnel.

FAA used six sets of performance standards for these positions. FAA

officials characterized five sets as results oriented to a great extent and

one set as somewhat results oriented. They said the standards were linked

to outcomes specified in FAA’s strategic goals, but that the linkage was

through several steps and not always clearly apparent. We characterized

only one set of standards as results oriented to a great extent, one set as

somewhat results oriented, and four sets as results oriented to little or no

extent.

Views on the Results

Orientation of

Performance Standards for

Air Traffic Control

Specialists

As of January 1, 1997, the air traffic control specialist position in en route

centers and terminals had three job elements and associated sets of

performance standards: “operating methods and procedures,”

“communications,” and “training.” An FAA official characterized the first

two sets of performance standards as results oriented to a great extent

because following proper air traffic procedures and effective

communication with pilots were essential in providing safe aircraft

landings and departures. We agreed with FAA’s characterization of the first

set of standards because the standards were clearly related to the agency’s

mission, described the position’s purpose (provide for “the safe, orderly

and expeditious flow of air traffic”), and provided criteria for evaluating

performance (“in accordance with procedures, directives and letters of

agreement”). The second set of standards on “communications” required

incumbents to be “cooperative, courteous and tactful,” use “prescribed

phraseology,” and listen “effectively,” but did not clearly link these

standards to the agency’s safety goal or state what results would occur as

14Although the President’s March 4, 1995, memorandum on his regulatory reinvention initiative did not

define “frontline regulators,” we defined them as regulatory staff who work directly with the regulated

community by enforcing regulations. We selected these two positions for review from the frontline

regulator positions FAA officials identified.

Page 45 GAO/GGD-97-83 Barriers to Managing for Results
Appendix II 

FAA’s Mission, Strategic Goals,

Performance Measures, and Performance

Standards for Selected Frontline Regulator

Positions

a result of these behaviors. Therefore, we considered this set of standards

results oriented to little or no extent.

The “training” standards required employees taking instruction to

complete classes “in a timely manner,” and said that employees providing

instruction should do so “objectively.” FAA officials said that this set of

standards was somewhat results oriented because training was essential

to support the duties of the position. We considered the set of training

standards to be results oriented to little or no extent because it was not

clearly related to the agency’s goals or mission and did not provide a

description of the expected results from the training or criteria for

evaluating results.

Views on the Results

Orientation of the

Performance Standards for

Aviation Safety Inspectors

As of January 1, 1997, the aviation safety inspector position (for

operations/maintenance/avionics with responsibility for general aviation

aircraft) had three job elements and associated sets of performance

standards: “technical administration,” “certification,” and “surveillance.”

FAA officials characterized all three sets of performance standards as

results oriented to a great extent. However, we considered the first two

sets of standards results oriented to little or no extent because they were

not clearly linked to FAA’s mission or any of its strategic goals. For

example, the technical administration standards required inspectors to

conduct investigations “thoroughly” and “in a timely manner,” but did not

say what mission-related results would follow from those reviews. We

considered the set of performance standards for “surveillance” to be

somewhat results oriented because it described actions that were more

clearly linked to FAA’s mission. For example, inspectors were required to

identify and communicate safety issues that require special emphasis to

the appropriate management levels. However, the set of standards was

unclear as to what outcomes incumbents should achieve as a result of

those actions.

FAA’s Comments and

Our Evaluation

We met with FAA officials, including several program managers, on April

29, 1997, to receive their comments. FAA officials generally agreed with the

draft report’s overall message regarding the barriers regulatory agencies

face in focusing on results and particularly noted how difficult it was to

determine the unique effect that FAA had on aviation system safety. They

indicated that the agency’s goals and performance measures had changed

since the time of our review. They also said that differences between their

and our characterizations of the goals, measures, and standards in use as

Page 46 GAO/GGD-97-83 Barriers to Managing for Results
Appendix II 

FAA’s Mission, Strategic Goals,

Performance Measures, and Performance

Standards for Selected Frontline Regulator

Positions

of January 1, 1997, were partially due to the fact that we did not use the

same criteria to determine the extent of result orientation. FAA officials

also provided some technical corrections, which were included in the final

report.

The FAA officials said that FAA had a set of performance goals that

connected the agency’s fiscal year 1996 strategic goals to its fiscal year

1996 program performance measures. They contended that there was a

closer linkage between FAA’s program performance measures and its

performance goals than existed between the measures and FAA’s strategic

goals (the subject of our review). They also said that FAA’s strategic goals

were broad goals for the aviation system as a whole, whereas its

performance goals related more specifically to the agency’s performance.

We reviewed FAA’s strategic goals and the program performance measures

FAA officials identified that related to these goals. In order to be consistent

in reviewing the five regulatory agencies, we reviewed the strategic goals

of each of these agencies.15

FAA officials pointed out that the performance standards for the aviation

safety inspector frontline regulator position were written before the

President’s March 4, 1995, memorandum calling for performance

standards to be results oriented. They also believed that individual

regulators understood the relationship of their performance standards to

FAA’s mission. Finally, the FAA officials said they thought it was unrealistic

to expect the standards, by themselves, to reflect a results orientation

when there were other supporting documents that linked the activities of

the regulators to the agency’s mission.

The President’s memorandum directed the heads of all federal regulatory

agencies to change the way they measure the performance of their

frontline regulators so as to focus on results, not process and punishment.

Therefore, a group of performance standards for a frontline regulator

position that does not in some way focus on results would not comply

with the memorandum, regardless of when the standards were written.

Also, Office of Personnel Management guidance states that goal setting is

the first component of effective performance management.16 According to

this guidance, research has shown that specific, hard goals result in higher

performance than vague or abstract goals, or no goals at all. Therefore, the

15The only performance goals included in this report were those of FDA because these goals served as

the agency’s program performance measures.

16“Improving Customer Service Through Effective Performance Management,” Office of Personnel

Management Performance Management Practitioner Series PMD-04, May 1996.

Page 47 GAO/GGD-97-83 Barriers to Managing for Results
Appendix II 

FAA’s Mission, Strategic Goals,

Performance Measures, and Performance

Standards for Selected Frontline Regulator

Positions

President’s initiative focusing standards on results could improve

regulators’ performance in obtaining desired results. Although other

supporting documents may have linked regulators’ performance standards

to the agency’s mission, we focused solely on agencies’ performance

standards to ensure a fair and balanced approach across all of the

agencies.

Page 48 GAO/GGD-97-83 Barriers to Managing for Results
Appendix III 

FDA’s Mission, Strategic and Performance

Goals, and Its Performance Evaluation Plan

FDA’s mission is to “protect, promote and enhance the health of the

American people.” FDA seeks to achieve this mission by ensuring that:

• “Foods are safe, wholesome, and sanitary; human and veterinary drugs,

biological products, and medical devices are safe and effective; cosmetics

are safe; and electronic products that emit radiation are safe;

• regulated products are honestly, accurately, and informatively

represented; and

• products are in compliance with the law and FDA regulations;

noncompliance is identified and corrected; and any unsafe or unlawful

products are removed from the marketplace.”

According to the fiscal year 1996 FDA Almanac, the agency supports its

mission through the enforcement of 13 major laws enacted throughout the

century, from the Food and Drugs Act of 1906 to the Dietary Supplement

Health and Education Act of 1994.

We assessed the results orientation of FDA’s strategic and performance

goals according to the criteria presented in the scope and methodology

appendix of this report.1

 Because we characterized the extent to which

each strategic goal and related sets of performance goals were results

oriented, the application of these criteria represented a higher standard of

results orientation than required by GPRA. The Results Act does not require

that all of an agency’s goals or measures be results oriented.

We applied our criteria to FDA’s written strategic and performance goals

and did not consider any supplementary or contextual information. In

some cases, the agency’s goals appeared results oriented on the surface,

but closer examination and strict application of our criteria resulted in a

lower characterization. We recognize that other information outside of the

written statements and outside of our review may have existed that could

have resulted in our characterizing certain goals differently.

Both we and FDA officials characterized the extent of results orientation of

FDA’s strategic and performance goals. We did not assess any employee

performance standards because, as of January 1, 1997, FDA was using

generic performance standards which varied from employee to employee.

Therefore, no group of standards applicable to a frontline regulator

position existed. As shown in table III.1, we and FDA officials differed

1

Instead of using performance measures as the means of measuring their progress on the agency’s

strategic goals, FDA used sets of “performance goals.” These “performance goals,” as defined by

GPRA, are the target level of performance (either output or outcome) expressed as a tangible,

measurable objective, against which actual achievement will be compared.

Page 49 GAO/GGD-97-83 Barriers to Managing for Results
Appendix III 

FDA’s Mission, Strategic and Performance

Goals, and Its Performance Evaluation Plan

somewhat in how we characterized the results orientation of FDA’s

strategic and performance goals. Generally, we thought that FDA’s goals

were less results oriented than the FDA officials did. However, we agreed

with FDA officials that most of their performance goals were somewhat

results oriented.

Table III.1: Differences Between Our and FDA Officials’ Characterizations

Number that FDA officials considered

results oriented to

Number we considered results oriented

to

Total

A great

extent Some extent

Little or no

extent

A great

extent Some extent

Little or no

extent

Strategic goals 3 3 0 0 0 3 0

Performance goalsa 9 2 7 0 0 5 4

aFDA had nine sets of performance goals that it used to assess progress toward its three

strategic goals, with different sets of measures for different FDA programs. These performance

goals contained the program performance measure in the goal statement. In addition,

performance standards were not included because, as of January 1, 1997, the agency was using

performance standards that were customized for individual employees.

Source: Agency and GAO analysis.

Views on the Results

Orientation of FDA’s

Strategic Goals

As of January 1, 1997, FDA had three strategic goals:

1. Pre-market: “Make timely and cost effective pre-market review

decisions while assuring product safety and efficacy;”

2. Post-market: “Strengthen assurance that products on the market or

about to enter the market are safe;” and

3. Capacity: “Strengthen the nation’s capacity to manage health risks

associated with FDA-regulated products.”

Although FDA characterized all three strategic goals as results oriented to a

great extent, we viewed all three goals as somewhat results oriented.

FDA officials said that the agency’s first strategic goal regarding pre-market

review decisions was results oriented to a great extent. By establishing a

review process, they said that FDA can ensure the safety of FDA-regulated

products and, therefore, promote and protect the health of the American

public. Although we agreed that “assuring product safety and efficacy” is

related to the agency’s mission to promote and protect public health, we

Page 50 GAO/GGD-97-83 Barriers to Managing for Results
Appendix III 

FDA’s Mission, Strategic and Performance

Goals, and Its Performance Evaluation Plan

considered the goal to be somewhat results oriented because it did not

establish a clear definition of the results expected through its pre-market

review decisions (e.g., what a “timely” or “cost effective” decision was)

and, therefore, did not permit assessment of the agency’s progress toward

achieving these results.2

 Also, the major emphasis in the goal is on making

timely and cost-effective decisions, which is not part of the agency’s stated

mission.3

FDA officials said the agency’s second strategic goal regarding the safety of

products on the market or about to enter the market was greatly results

oriented because the agency’s inspection process provided a safeguard to

prevent the distribution of potentially harmful food, drugs, and other

FDA-regulated products. They also said that by preventing the distribution

of these harmful products, FDA protected the health of the American

public. We again characterized this goal as somewhat results oriented

because it did not provide a clear definition of its intended results or

permit assessment of the agency’s progress toward achieving the results.

For example, the goal did not identify how or the extent to which FDA

expects to “strengthen assurance” that products are safe, or even whose

“assurance” would be strengthened.

Finally, FDA officials said the agency’s third strategic goal—to “strengthen

the Nation’s capacity to manage health risks associated with FDA-regulated

products”—was results oriented to a great extent because it referred to

the agency’s internal processes to maintain the science capacity of its

staff. Through its science capacity, FDA officials said the agency can be

assured that it has valid and reliable science to support its review

processes and inspections—the activities by which FDA protects and

promotes the public’s health. However, we characterized this goal as

somewhat results oriented because, although it was tenuously linked to

FDA’s mission, it was unclear whose “capacity” would be strengthened,

what “capacity” means, or what “manage health risks” means. If, as FDA

officials said, the agency’s capacity was to be strengthened through

performance of this goal, it was unclear how and the degree to which FDA

would strengthen its capacity or the effect that strengthening would have

on the nation’s or the agency’s ability to manage health risks.

2

In October 1992, Congress passed the Prescription Drug User Fee Act (PDUFA) to provide FDA with

additional resources to expedite the review of human drug applications. PDUFA incorporated by

reference specific goals and time frames for FDA to review and act on drug applications set forth in

letters from the FDA Commissioner to the Chairmen of the House Committee on Energy and

Commerce and the Senate Committee on Labor and Human Resources. However, FDA did not link its

strategic goal with these time frames or any other criteria defining what would be considered timely.

3

PDUFA did not make any specific alterations to FDA’s mission.

Page 51 GAO/GGD-97-83 Barriers to Managing for Results
Appendix III 

FDA’s Mission, Strategic and Performance

Goals, and Its Performance Evaluation Plan

Views on the Results

Orientation of FDA’s

Performance Goals

As of January 1, 1997, FDA was using nine sets of performance goals for its

three strategic goals. As table III.2 shows, different sets of performance

goals were used not only for the different goals but also for many of the

different FDA programs. Post-market performance goals were not

applicable to two of the agency’s programs. However, the set of

performance goals for the capacity strategic goal was applicable to all of

the agency’s programs except for toxicological research. FDA officials

characterized two of these sets of performance goals as results oriented to

a great extent and seven sets as somewhat results oriented. However, we

considered five sets of FDA’s performance goals as somewhat results

oriented and four sets as results oriented to little or no extent.

Figure III.1: Sets of FDA Performance

Goals Varied by Strategic Goal and

FDA Program

All FDA programs

Biologics and drugs

Food safety and applied nutrition

Devices and radiological health

Veterinary medicine

Toxicological research

Pre-marketPost-market

Capacity

a

a na na

na na

na

na

na

na na

Strategic goals

Note 1: The use of "na" indicates that no performance goals were established specifically for that FDA program.

Note 2: A indicates that FDA established a set of performance goals applicable to the strategic goal

 and that FDA program or programs.

 

 Includes all FDA programs except for toxicological research.

Views on the Results

Orientation of the Sets of

Performance Goals for the

Pre-Market Strategic Goal

FDA officials characterized four sets of performance goals related to the

agency’s pre-market strategic goal as somewhat results oriented and one

set as results oriented to a great extent. We considered all five sets of

performance goals as somewhat results oriented.

FDA officials characterized three sets of pre-market performance goals

(biologics/drugs, food safety and nutrition, and devices and radiological

health) as somewhat results oriented. The officials classified these sets of

performance goals as such because, although the sets measured the

Page 52 GAO/GGD-97-83 Barriers to Managing for Results
Appendix III 

FDA’s Mission, Strategic and Performance

Goals, and Its Performance Evaluation Plan

number or percentage of applications FDA completed within a certain

period of time, these sets did not include measures for the length of time

needed to approve an application. For example, the performance goals

related to biologics and drugs were:

• “Review 90 percent of complete product licensing agreements (PLA),

establishment licensing agreements (ELA), and new drug approvals (NDA)

for priority applications within 6 months after the submission date;

• review 90 percent of complete PLAs, ELAs, and NDAs for standard

applications, within 12 months after the submission date;

• review 90 percent of priority supplements to complete PLAs, ELAs, and NDAs

within 6 months after the submission date;

• review 90 percent of standard supplements to complete PLAs, ELAs, and

NDAs that require review of clinical data (efficacy supplements) within 12

months after the submission date;

• review 90 percent of standard supplements to PLAs, ELAs, and NDAs that do

not require review of clinical data (efficacy supplements) within 6 months

after the submission date;

• review 90 percent of complete PLA, ELA, and NDA applications resubmitted

following receipt of a non-approvable letter, within 6 months after the

resubmission date; and

• review and act upon 55 percent of fileable generic drug applications

received during Fiscal Year 1997 within 6 months after submission.”

According to an FDA official, completed applications include those

approved by FDA or those requiring additional information from the

applicant.4

 When FDA requests additional information, the applicant may

respond to FDA’s request or withdraw the application from consideration.

The FDA official said that because applicants are never denied the

opportunity to return to the application process, FDA rarely issues a final

negative response to applications. However, some FDA programs do have a

“refuse to file” policy that prevents some incomplete applications from

entering FDA’s review process.

An FDA official said that if the performance goals had measured the

timeliness of approvals they would be more results oriented because

timeliness would translate into new products being made available for

public use, potentially improving the health of the American public. We

also considered these sets of performance goals somewhat results

4

FDA notifies applicants as to the disposition of their application by using one of three “action letters.”

Applicants may receive either an “approved letter,” an “approvable letter” (an approval of the

application noting minor deficiencies that could be corrected by the applicant), or a “not approvable

letter” (which provides the applicant the option to resubmit a modified application).

Page 53 GAO/GGD-97-83 Barriers to Managing for Results
Appendix III 

FDA’s Mission, Strategic and Performance

Goals, and Its Performance Evaluation Plan

oriented because they did not cover all aspects of the strategic goal. None

of the goals addressed the cost-effectiveness of these reviews or whether

product safety and efficacy were being assured.

FDA officials said the set of performance goals for veterinary medicine was

results oriented to a great extent because, unlike the first three sets of

performance goals, it measured the length of time to approve an

application.5

 For example, one performance goal was to “reduce the

average number of months required for approval of a new animal drug

from 41 months in fiscal year 1996 to 40 months in fiscal year 1997.” FDA

officials said that approving applications for animal drugs and feeds

assisted in maintaining the health of livestock and limited the spread of

contaminated foods, and therefore protected public health. However, we

considered this set of performance goals somewhat results oriented

because, like the previous sets of goals, it did not address all aspects of the

strategic goal. None of the performance goals measured the

cost-effectiveness of the reviews or whether product safety and efficacy

were being assured.

FDA had three performance goals relating to toxicological research:

1. “Complete the inlife portions of the mechanistic and tumor studies on

fumonisin and chloral hydrate, which will provide information on specific

agent toxicity and a new methodology for future assessments;

2. develop and evaluate a prototype predictive screening process to

streamline the review of new compounds and to more accurately predict

human risk; and

3. construct a prototype ‘knowledge base’ (expert systems to evaluate and

predict risk associated with toxicity by estrogenic compounds) for

selected reviewer evaluation.”

FDA officials characterized this set of goals as somewhat results oriented

because it focused on the development of a good “knowledge base,” which

they said enabled FDA to make valid and reliable decisions during the

approval process.

We agreed that developing a knowledge base is important for FDA’s review

process and that the measures were generally related to the strategic goal

5

Similar to other FDA programs, completed applications for veterinary medicine could be either

approved by FDA or a request could be made to the applicant for additional information.

Page 54 GAO/GGD-97-83 Barriers to Managing for Results
Appendix III 

FDA’s Mission, Strategic and Performance

Goals, and Its Performance Evaluation Plan

of making “timely and cost-effective pre-market decisions.” However, the

set of goals focused on developing processes (i.e., developing “a new

methodology for future assessments,” “a prototype predictive screening

process,” or “prototype ‘knowledge base”) did not provide a clear means

to assess progress toward achieving the pre-market strategic goal. In

addition, the set of goals did not address the elements of timeliness or

cost-effectiveness of the review process, and did not assess whether

product safety or efficacy were being assured. Therefore, we considered

this set of performance goals to be somewhat results oriented.

Views on the Results

Orientation of the Sets of

Post-Market Performance

Goals

FDA had three sets of performance goals for its post-market strategic goal

to “strengthen assurance that products on the market or about to enter the

market are safe.” FDA officials characterized one set of performance goals

as results oriented to a great extent and the remaining two sets as

somewhat results oriented. We considered all three sets of performance

goals to be results oriented to little or no extent.

The first set of post-market performance goals, related to food safety and

nutrition, were:

• “By December 31, 1996, develop a training program on how to conduct

Hazard Assessment Critical Control Points (HACCP) inspections;

• by April 1, 1997, develop a model Federal/State partnership agreement for

seafood HACCP to implement HACCP-based inspections of seafood

processors by the states;

• by May 31, 1997, provide training to FDA inspectors who will conduct

verification inspections, and appropriate headquarters staff of core HACCP

curriculum;

• by June 30, 1997, revise the FDA Inspection Operations Manual for

HACCP-based inspections of seafood processors;

• by June 30, 1997, revise compliance program criteria for HACCP-based

inspections of seafood processors;

• by September 30, 1997, determine utility of establishing a

reference/baseline of industry preventive controls based on an assessment

of past data;

• by September 30, 1997, establish a Seafood HACCP database to record

inspection results and facilitate evaluation of the system; and

• increase Threshold of Regulation Policy Actions Completed from 50 in

1996 to 55 in 1997.”

Page 55 GAO/GGD-97-83 Barriers to Managing for Results
Appendix III 

FDA’s Mission, Strategic and Performance

Goals, and Its Performance Evaluation Plan

FDA officials characterized the set of performance goals related to food

safety and nutrition as results oriented to some extent. They said that the

HACCP inspections indicated a new strategic direction for FDA by

emphasizing the detection and prevention of microbiological

contamination of food products and extending the responsibility for food

safety to the food processing industry. An FDA official said that this set of

performance goals illustrated how FDA was implementing HACCP (e.g.,

developing partnership agreements, training inspectors, and revising an

inspection manual and compliance program criteria). However, we

characterized this set of performance goals as results oriented to little or

no extent because none of the goals clearly related to or measured the

strategic goal or the agency’s mission. It was not clear to us that any of the

actions contained in these measures (e.g., develop a training program,

revise a manual, or establish a database) would, by themselves, strengthen

assurance that products are safe.

The second set of post-market performance goals, related to devices and

radiological health, were:

• “Process 95 percent of the medical device reports received, and maintain a

zero backlog;6

• process 5,400 reports under the MedWatch Voluntary Reporting Program;7

• train and certify approximately 250 State and Federal inspectors to

conduct annual and follow-up inspections at the approximately 10,000 U.S.

mammography facilities;

• conduct 10,000 annual and 200 follow-up inspections of U.S.

mammography facilities under the Mammography Quality Standards Act

(MQSA); and

• conduct 3,000 U.S. mammography facility recertifications under MQSA.”

FDA officials characterized this set of performance goals as greatly results

oriented because three of the goals related to the MQSA, which authorized

FDA to accredit and inspect mammography facilities. An FDA official said

that FDA enhances the performance of equipment and radiologists through

6

Under the 1976 Medical Device Amendments to the Federal Food, Drug, and Cosmetic Act,

manufacturers of medical devices are required to report to FDA—and maintain records

on—device-related deaths, serious injuries, and malfunctions that, should they recur, would be likely

to result in death or serious injury.

7

The MedWatch Voluntary Reporting Program (MedWatch) was established by FDA to encourage

health professionals to report problems associated with medical devices to FDA. MedWatch and

medical device reports are intended to gather information to enable FDA and the medical device

industry to take corrective action on device problems and, when appropriate, to alert the public to

potentially hazardous devices.

Page 56 GAO/GGD-97-83 Barriers to Managing for Results
Appendix III 

FDA’s Mission, Strategic and Performance

Goals, and Its Performance Evaluation Plan

its MQSA program, thereby producing improved patient care through a

better screening process and diagnosis.

However, we characterized this set of performance goals as results

oriented to little or no extent because it did not clearly relate to or provide

relevant information for assessing progress toward the strategic goal of

strengthening assurance that products are safe. For example, the fourth

goal only measured the number of MQSA inspections conducted and did not

measure how many facilities were in compliance with mammography

standards or assess the impact or effect of these inspections.8

 Similarly,

the performance goals regarding medical device reports and the

MedWatch Voluntary Reporting Program only measured the processing of

reports, and did not gauge FDA’s impact on the safety of medical devices.9

The third set of post-market performance goals, related to veterinary

medicine, were (1) “conduct 700 establishment inspections” and

(2) “conduct, by States under FDA contract, 550 feed mill inspections.” FDA

officials said they considered this set of goals somewhat results oriented

because inspections can improve the quality of animal drugs and feeds on

the market. However, similar to the performance goals on MQSA

inspections, this set did not measure how many establishments or feed

mills were in compliance with FDA regulations. As such we considered this

set of performance goals as results oriented to little or no extent because

it did not clearly relate to or provide relevant information for assessing

progress toward the strategic goal of strengthening assurance that

products are safe.

Views on the Results

Orientation of the Set of

Capacity Performance

Goals

FDA had three performance goals related to its strategic goal of

strengthening the nation’s capacity to manage health risks:

1. “Complete design on improved version of full Operational and

Administrative System for Import Support;

8

In our report entitled FDA’s Mammography Inspections: While Some Problems Need Attention,

Facility Compliance Is Growing (GAO/HEHS-97-25, Jan. 27, 1997), we found that although “FDA has

made progress in bringing facilities into compliance with mammography standards, it lacks procedures

to enforce timely correction of all deficiencies found during inspections.”

9

In our report entitled Medical Device Reporting: Improvements Needed in FDA’s System for

Monitoring Problems With Approved Devices (GAO/HEHS-97-21, Jan. 29, 1997), we found that FDA

had not “routinely documented the final corrective actions taken by manufacturers and FDA to resolve

the problems identified in these reports.” The report concluded that “without adequate documentation,

FDA has little assurance that the many reports received are, in fact, useful and result in better

protection of the public health.”

Page 57 GAO/GGD-97-83 Barriers to Managing for Results
Appendix III 

FDA’s Mission, Strategic and Performance

Goals, and Its Performance Evaluation Plan

2. allow for a range of between 62 percent and 65 percent automated ‘may

proceeds’ on a monthly basis; and

3. maintain a foreign inspection program with a planned total of 1,020

foreign inspections in fiscal year 1997.”

FDA officials characterized this set of performance goals as somewhat

results oriented because they were all output measures of FDA’s internal

processes. An FDA official said that the first goal was a process

improvement goal to automate FDA’s system of handling imported foods

and other products. The second goal measured the percentage of “may

proceed” decisions being automated as part of FDA’s import entry

clearance process, indicating a more efficient process. The third goal

referred to the number of inspections FDA expected to complete of foreign

manufacturers of drugs and devices, which FDA officials said was another

example of how FDA protects public health. However, we characterized

this set of performance goals as results oriented to little or no extent

because the actions contained in these goals (designing systems,

automating processes, or maintaining an inspection program) were not

clearly and unambiguously related to FDA’s mission of promoting and

protecting the health of the American people. Also, the set did not provide

relevant information to assess progress toward that mission, such as the

number of foreign establishments in compliance with FDA regulations.

FDA Replaced

Performance

Standards With a

Customized

Performance

Evaluation Plan

On September 1, 1996, FDA adopted a new employee performance

evaluation program that replaced most of FDA’s employee performance

standards.10 FDA officials said that the new program includes a

“performance evaluation plan” (PEP)—an evaluation form that lists five

generic job elements on “individual work management,” “technical

competency,” “teamwork,” “innovation,” and “customer service.” PEP also

contains blank spaces for supervisors to write in new job elements

tailored to the employee’s position. FDA documentation stated that each

employee and his or her supervisor should discuss the employee’s official

duties and responsibilities and use PEP’s checklist format to customize a

set of performance standards for the employee. PEP requires that

nonsupervisors be evaluated on at least one job element, but supervisors

have the final decision on the content of the PEP form.

10FDA officials said that the majority of FDA staff were under the new performance evaluation system

with the exception of some union employees remaining under the prior system.

Page 58 GAO/GGD-97-83 Barriers to Managing for Results
Appendix III 

FDA’s Mission, Strategic and Performance

Goals, and Its Performance Evaluation Plan

FDA officials said that most of the positions they considered “frontline

regulators” were using PEP as of January 1, 1997, and, therefore had unique

sets of performance standards. As a result, we did not include any

performance standards for FDA in our review because FDA officials told us

that the performance standards for any of their frontline regulator

positions would vary from employee to employee.

FDA’s Comments and

Our Evaluation

We met with FDA officials, including the Director for Planning and

Management Communications Staff, on April 22, 1997. FDA officials said

they generally agreed with the report’s overall message regarding the

difficulties that regulatory agencies have experienced in becoming results

oriented. They also said that they would have reached conclusions similar

to ours regarding the results orientation of their strategic and performance

goals had they used the same criteria we used. The FDA officials also

provided several technical corrections, which were included in the final

report.

FDA officials said that (1) the progress made by FDA’s field offices was not

reflected in the report’s focus on FDA’s agencywide goals; (2) the

performance goals FDA is developing for the fiscal year 1999 budget cycle

will be more results oriented than the January 1, 1997, goals assessed in

this report; and (3) many of the interim goals that FDA had (e.g., developing

a training program for conducting HACCP verification inspections), while

not results oriented themselves, would ultimately achieve

outcome-oriented results.11

We agree that focusing on agencywide goals did not capture information

on progress made in field office pilot programs. In the draft report, we

recognized that the goals, measures, and some of the standards were

changing at the time of our review. FDA’s efforts to establish

results-oriented systems, such as HACCPs, may allow FDA to achieve

outcome-oriented results in the future. However, our objective was to

describe the agency’s degree of results orientation as of January 1, 1997.

11The HACCP system is designed to identify the critical points in food processing and establish

controls to prevent adulteration caused by microbiological, chemical, or physical hazards in order to

produce safer food products.

Page 59 GAO/GGD-97-83 Barriers to Managing for Results
Appendix IV 

IRS’ Mission, Strategic Objectives, Key

Performance Indicators, and Performance

Standards for Selected Frontline Regulator

Positions

IRS’ mission is to “collect the proper amount of tax revenue at the least

cost; serve the public by continually improving the quality of our products

and services; and perform in a manner warranting the highest degree of

public confidence in our integrity, efficiency, and fairness.” IRS also has

developed a business vision “to be a state-of-the-art financial services

company that eliminates most paper in the quest for quick, efficient

customer service. The new IRS will be handling less paper and collecting

taxes in new and more efficient ways.”

We assessed the results orientation of IRS’ strategic objectives,

performance indicators, and employee performance standards according

to the criteria presented in the scope and methodology appendix of this

report.1

 Because we characterized the extent to which each agency goal

and set of performance measures was results oriented, the application of

these criteria represented a higher standard of results orientation than

required by the Results Act. The Act does not require that all of an

agency’s goals or measures be results oriented. Also, GPRA does not cover

employee performance standards. We judged each set of performance

standards according to the expectation set by the President in his March 4,

1995, memorandum that these standards should “focus on results, not

process and punishment.” Similar to the way in which GPRA views agencies’

goals and performance measures, we do not believe that each such

standard must be results oriented to achieve the President’s objective.

We applied our criteria to the agencies’ written strategic goals, program

performance measures, and employee performance standards and did not

consider any supplementary or contextual information. In some cases an

agency’s goals, measures, and standards appeared results oriented on the

surface, but closer examination and strict application of our criteria

resulted in a lower characterization. We recognize that other information

outside of the written statements and outside of our review may have

existed that could have resulted in our characterizing certain goals or sets

of measures and standards differently.

Both we and IRS officials characterized the extent of results orientation of

IRS’ strategic objectives, key performance indicators, and employee

performance standards for selected frontline regulator positions. As

shown in table IV.1, there were differences in how we and IRS officials

viewed the agency’s objectives, indicators, and standards. We and IRS

officials agreed that IRS’ strategic objectives were greatly results oriented.

1

IRS used the term “strategic objective” instead of “strategic goal” and the term “key performance

indicators” instead of “program performance measures.”

Page 60 GAO/GGD-97-83 Barriers to Managing for Results
Appendix IV 

IRS’ Mission, Strategic Objectives, Key

Performance Indicators, and Performance

Standards for Selected Frontline Regulator

Positions

There was somewhat more disagreement regarding IRS’ performance

indicators in that we characterized them as somewhat less results oriented

than did the IRS officials. The differences in our characterizations were

greatest with regard to the employee performance standards. IRS officials

said that most of the standards were greatly results oriented, but we

characterized most as somewhat results oriented.

Table IV.1: Differences Between Our and IRS Officials’ Characterizations

Number that IRS officials considered

results oriented to

Number we considered results oriented

to

Total

A great

extent Some extent

Little or no

extent

A great

extent Some extent

Little or no

extent

Strategic objectivesa 3 3 0 0 3 0 0

Key performance indicatorsb 3 2 1 0 0 3 0

Performance standards 10 9 1 0 3 7 0

aIRS used the term “strategic objective” instead of “strategic goal.”

bIRS used the term “key performance indicators” instead of “program performance measures.”

Source: Agency and GAO analysis.

Views on the Results

Orientation of IRS’

Strategic Objectives

As of January 1, 1997, IRS had established three “strategic objectives”:

1. “Increase compliance: Our objective is to encourage and assist

taxpayers to voluntarily file timely and accurate returns and to pay on

time, and, if taxpayers do not comply, to take appropriate compliance

actions. We do this to ensure we collect the proper amount of tax at the

least cost.

2. Improve customer service: Our objective is to reduce the time and

expense experienced by taxpayers, tax professionals and others in

complying with the tax laws, while increasing their satisfaction with tax

administration. We want to ensure that filing materials are readily

available, and to reduce the data required to be reported by taxpayers and

captured by IRS to only the most critical. We do this to help collect the

proper amount of tax at the least cost, serve the public, and increase

public confidence in tax administration.

3. Increase productivity: Our objective is to continually improve our

operations and the quality of products and services we provide to our

customers, through strategic use of reengineering and systems

Page 61 GAO/GGD-97-83 Barriers to Managing for Results
Appendix IV 

IRS’ Mission, Strategic Objectives, Key

Performance Indicators, and Performance

Standards for Selected Frontline Regulator

Positions

management tools and the development of a highly skilled work force. We

do this to improve compliance, provide improved customer service and

reduce the cost of tax administration.”

IRS officials said that these three objectives were results oriented to a great

extent because they related directly to IRS’ mission. We agreed with that

assessment with regard to each of the strategic objectives. IRS’ first

objective, to increase taxpayer compliance, was clearly related to IRS’

mission to collect the proper amount of tax revenue, provided a clear

definition of the results expected, and permitted assessment of the

agency’s programs. Similarly, IRS’ second objective, to improve customer

service, directly related to IRS’ mission to “serve the public,” and IRS’ third

objective of increasing productivity was also linked to IRS’ mission to

collect revenue “at the least cost.”

Views on the Results

Orientation of IRS’

Key Performance

Indicators

As of January 1, 1997, IRS was using three sets of servicewide “key

performance indicators” for its strategic objectives. IRS officials said that

two of the three sets of indicators were results oriented to a great extent,

and one was results oriented to some extent. We characterized the three

sets of indicators as somewhat results oriented.

Views on the Results

Orientation of the

Indicators Associated With

IRS’ Strategic Objective to

“Increase Compliance”

The indicators that IRS used to assess progress toward its first strategic

objective, to “increase compliance,” were:

• “Total collection percentage”: A ratio of IRS’ total collections to the total

estimated true tax liability.

• “Total net revenue collected”: All revenue collected by the IRS.

• “Servicewide enforcement revenue collected/protected”: All revenue

collected through enforcement actions and all revenue prevented from

being released from the Treasury in response to fraud, disallowed claims,

and other issues.2

• “Servicewide audit coverage”: The percentage of returns audited divided

by the total number of returns filed during the prior calendar year.

IRS officials said that this set of indicators was results oriented to a great

extent because they believed that the first three indicators were clear

indications of IRS’ collection results. However, we characterized this set of

indicators as somewhat results oriented because, although the “total

collection percentage” was clearly and unambiguously related to the

2

According to IRS, enforcement revenue protected and enforcement revenue collected will be

expressed as separate indicators in fiscal year 1998.

Page 62 GAO/GGD-97-83 Barriers to Managing for Results
Appendix IV 

IRS’ Mission, Strategic Objectives, Key

Performance Indicators, and Performance

Standards for Selected Frontline Regulator

Positions

strategic objective of increasing compliance, the set did not address all

aspects of the strategic objective.3

 For example, the set did not contain

indicators for IRS efforts to encourage and assist taxpayers in filing

accurate and timely returns.

We did not view the remaining indicators in this set as clearly linked to the

strategic objective to “increase compliance.” The second indicator, “total

net revenue collected,” is already contained in the ratio of the “total

collection percentage” and is ambiguous in its relation to the strategic

objective. An increase or decrease in revenue collected may simply reflect

a change in economic activity or tax laws rather than any change in the

level of taxpayer compliance. Similarly, the amount collected/protected

through enforcement actions and the percentage of returns audited may

not provide a valid indication of the level of taxpayer compliance. For

example, enforcement revenues collected could increase while voluntary

revenues decreased, resulting in no change (or even a decrease) in the

overall level of taxpayer compliance.

Views on the Results

Orientation of the

Indicators Associated With

IRS’ Strategic Objective to

“Improve Customer

Service”

IRS used two indicators to assess the agency’s progress toward its second

strategic objective, to “improve customer service”:

1. “Taxpayer burden cost (in dollars) for IRS to collect $100”: A ratio that

measured taxpayers’ paperwork burden costs compared to the cost for IRS

to collect $100 in net tax revenue.4

2. “Initial contact resolution rate”: The rate at which taxpayer issues were

resolved during the first contact with IRS.

5

IRS officials characterized this set of indicators as results oriented to some

extent, and said they were working to improve IRS’ customer service

measures. We also characterized the set of indices as somewhat results

oriented because the indicators did not address all aspects of the customer

3

Although the “total collection percentage” information is relevant, it may not be accurate. Our reports

have shown that IRS’ estimate of the true tax liability lacks statistical precision at the national level.

Without such an estimate, IRS will not be able to accurately determine the “total collection

percentage” nor judge its progress toward increasing compliance. See Taxpayer Compliance:

Analyzing the Nature of the Income Tax Gap (GAO/T-GGD-97-35, Jan. 9, 1997).

4

Paperwork burden costs were based on the estimated time individual and business taxpayers spend

keeping tax records, learning about tax laws, preparing tax forms, and sending tax forms to the IRS.

Taxpayer paperwork burden was converted from time to dollars by multiplying total time by $20 per

hour.

5

As of January 1, 1997, IRS was revising the definition of this rate.

Page 63 GAO/GGD-97-83 Barriers to Managing for Results
Appendix IV 

IRS’ Mission, Strategic Objectives, Key

Performance Indicators, and Performance

Standards for Selected Frontline Regulator

Positions

service objective.6 For example, IRS did not measure “public confidence in

tax administration” even though the agency regularly conducts customer

service surveys. IRS also did not include any indicators of how they would

“ensure that filing materials are readily available.”

Views on the Results

Orientation of Indicators

Associated With IRS’

Strategic Objective to

“Increase Productivity”

IRS’ indicators for the strategic objective to “increase productivity” were:

• “Budget cost to collect $100 in revenue.”

• “Percent of returns filed electronically”: The number of electronically filed

tax returns, divided by the total returns filed.

• “Percent of dollars received electronically.”7

• “Percent of dollars received via third party processors”: The dollar amount

of federal tax deposits (FTD) processed by commercial banks under

contract to the Financial Management Service as a percentage of the total

FTD dollars received (paper and electronic).

IRS officials said that this set of indicators was results oriented to a great

extent mainly because the first indicator provided an overall indicator of

how efficiently IRS collects revenues. However, we characterized the set as

somewhat results oriented because, although the indicator “budget cost to

collect $100 in revenue” is a good overall productivity measure, the set of

indicators did not address all aspects of the objective. For example, none

of the indicators measured the “quality of the products and services” IRS

provides.

6

The paperwork burden data upon which the first indicator is based, although relevant to assessing

progress toward the strategic objective, is generally considered to be inaccurately measured. Our

reviews of tax compliance burden have shown that a reliable estimate of the overall costs of tax

compliance is not currently available, and would be costly and burdensome to obtain. See Tax System:

Issues in Tax Compliance Burden (GAO/T-GGD-96-100, Apr. 3, 1996); Tax System Burden: Tax

Compliance Burden Faced by Business Taxpayers (GAO/T-GGD-95-42, Dec. 9, 1994); and Tax

Administration: Opportunities to Reduce the Burden of Filing and Processing Tax Returns

(GAO/T-GGD-92-41, May 13, 1992).

7

As of January 1, 1997, IRS was revising this indicator.

Page 64 GAO/GGD-97-83 Barriers to Managing for Results
Appendix IV 

IRS’ Mission, Strategic Objectives, Key

Performance Indicators, and Performance

Standards for Selected Frontline Regulator

Positions

Views on the Results

Orientation of IRS’

Performance

Standards for

Selected Frontline

Regulator Positions

In contrast to strategic objectives and performance indicators, the

administration’s regulatory reinvention initiative, not GPRA, set the

expectation for agencies to focus on results when measuring the

performance of their frontline regulators. IRS identified two frontline

regulator positions for this review: customer service representatives (CSR)

and revenue officers.8

 Employees in the CSR position work as technicians

in a district office or service center to provide information and direction

on tax issues to the public. Revenue officers collect delinquent taxes and

secure delinquent returns. Revenue officers’ work requires application of a

knowledge of (a) general or specialized business practices; (b) pertinent

tax laws, regulations, procedures, and precedents; (c) judicial processes,

laws of evidence, and the interrelationship between federal and state laws

with respect to collection and assessment processes; and (d) investigative

techniques and methods.

As of January 1, 1997, the CSR and revenue officer positions had a total of

12 job elements and we and IRS officials reviewed 10 of the 12 sets of

associated performance standards.9 IRS officials characterized nine of the

sets of standards as results oriented to a great extent and one set as

somewhat results oriented. We considered three of the sets of standards to

be greatly results oriented and the remaining seven sets as somewhat

results oriented.

Views on the Results

Orientation of IRS’

Customer Service

Representatives’

Performance Standards

We and IRS officials reviewed the sets of performance standards related to

five job elements for the customer service representative position:

1. “Customer relations: Achieves quality customer service by

communicating professionally with internal and external customers in all

oral and written contacts.

2. Workload management: Plans and utilizes time based on priorities in

accordance with Service policy and managerial direction. Arranges

activities to minimize unproductive time and unnecessary delays in

servicing the customer.

8

Although the President’s March 4, 1995, memorandum on his regulatory reinvention initiative did not

define “frontline regulators,” we defined them as regulatory staff who work directly with the regulated

community by enforcing regulations.

9

We did not review the sets of standards for the CSR or revenue officer positions’ job elements for

“other duties and assignments” because they were not specific and, although they may have related to

results in specific instances, reviewing these standards would not have provided any consistent

perspective on the extent to which these standards focused on results.

Page 65 GAO/GGD-97-83 Barriers to Managing for Results
Appendix IV 

IRS’ Mission, Strategic Objectives, Key

Performance Indicators, and Performance

Standards for Selected Frontline Regulator

Positions

3. Procedures and notices: Applies procedural instructions issued by

National, Regional, District and local offices to maximize the efficient flow

of work in meeting program objectives. Correctly interprets notices to

resolve inquiries appropriately.

4. Integrated Data Retrieval System (IDRS) inquiries and on-line resolution:

Resolves notice inquiries, refund and other account related issues through

accessing IDRS. Researches and analyzes or performs on-line adjustment to

accounts by utilizing appropriate reference materials.

5. Tax law: Encourages voluntary compliance and improves the public’s

confidence in the Service by correctly interpreting and applying tax laws,

researching technical reference materials, and analyzing and evaluating

the customer’s needs; informs customer of options and consequences as

they relate to tax obligations.”

The performance standards for each element described what the CSRs

were expected to do for that element. For example, the performance

standards for the “tax law” job element were:

• “Interpretation and application: Generally interprets and applies tax laws

appropriately resulting in technically correct responses to customer

inquiries.

• Analysis and evaluation: Generally evaluates and analyzes the customer’s

circumstances and advises of tax computation and other benefits, i.e.,

Earned Income Credit, Child Care Credit, estimated tax payments,

reasonable cause, etc.

• Research: Generally researches in appropriate and mandatory reference

materials such as the Probe and Response Guide, IRS Publications, Tax

Services, Internal Revenue Code (IRC), etc., to provide correct and

complete response; generally does not seek assistance from others even

on complex and non-routine issues. Increases customer confidence by

citing specific references and offering tax publications.”

IRS officials characterized all five sets of performance standards as results

oriented to a great extent because they said each set was related to IRS’

strategic objectives and/or its mission. We agreed with IRS officials’

assessment for the sets of standards for the “workload management” and

“tax law” job elements. For example, the set of standards related to the

“tax law” job element included the expectation that CSRs would provide

“technically correct responses to customer inquiries” and “increase

customer confidence by citing specific references and offering tax

Page 66 GAO/GGD-97-83 Barriers to Managing for Results
Appendix IV 

IRS’ Mission, Strategic Objectives, Key

Performance Indicators, and Performance

Standards for Selected Frontline Regulator

Positions

publications.” Therefore, these standards appeared directly related to IRS’

mission of encouraging voluntary compliance and warranting public

confidence in the agency, described the results expected, and established

criteria to evaluate performance. However, we characterized the

remaining three sets of performance standards as somewhat results

oriented, because they did not clearly describe the results CSRs were

expected to work toward and/or did not establish criteria for evaluating

performance directly associated with an expected result. For example, the

set of standards for the “IDRS inquiries and on-line resolution” job element

called for CSRs to “perform adjustments correctly” and “recognize when

related account data is pertinent and handle appropriately,” but these

actions were not clearly linked to an expected result.

Views on the Results

Orientation of the

Performance Standards for

IRS’ Revenue Officer

Position

We and IRS officials reviewed the sets of employee performance standards

related to five job elements for the revenue officer position: “investigation

and analysis,” “case decisions,” “time and workload management,”

“customer relations,” and “accounts maintenance.” The standards

described the types of actions revenue officers were expected to take for

that element. For example, the performance standards for the “case

decisions” job element were:

• “Collectibility: Based on the available facts, determines the proper case

direction. Decisions to pursue full payment, immediate or deferred, or

suspend collection activity are based on full consideration of the overall

benefit to the government and taxpayers’ rights.

• Collection tools: Based on case history, determines when levy, seizure, or

summons action is appropriate. Considers other collection tools such as

transferee, alter-ego, nominee, and Offer-in-Compromise. May need

occasional assistance making final decision on use of transferee, alter-ego,

etc.

• Protection of the government’s interest: Based on case history, determines

when actions are necessary to protect the government’s interest, including

collection and assessment statutes. Decides on the proper action to take,

including but not limited to filing and/or refiling liens, recommending

suits, 100 percent penalty, prompts, quick, and IRC 6020(b) assessments,

fraud, or other appropriate referrals and waivers.”

IRS officials characterized four of the sets of performance standards as

results oriented to a great extent, and said the set of standards related to

“customer relations” was somewhat results oriented. The officials said that

all five of the sets of standards were primarily process measures, but said

Page 67 GAO/GGD-97-83 Barriers to Managing for Results
Appendix IV 

IRS’ Mission, Strategic Objectives, Key

Performance Indicators, and Performance

Standards for Selected Frontline Regulator

Positions

the standards were related to IRS’ goals. For example, they said that the

standards for the “investigation and analysis” job element allow IRS to

collect the proper amount of tax and increase productivity. In addition, the

IRS officials pointed out that the Technical and Miscellaneous Revenue Act

of 1988 prohibits the IRS from evaluating employees on the basis of tax

enforcement results. Section 6231 of this act states that IRS cannot use

records of tax enforcement results to evaluate employees directly involved

in collection activities and cannot impose production quotas or goals on

individual employees. We characterized one set of performance standards

as results oriented to a great extent and the remaining four sets as

somewhat results oriented.

We agreed with IRS that the standards for the “case decisions” job element

were greatly results oriented because they were directly related to IRS’

mission to collect the proper amount of tax, clearly described the results

the revenue officers were expected to work toward, and established

criteria for evaluating the officers’ performance directly associated with

the desired results. For example, the standards stated that “decisions to

pursue full payment, immediate or deferred, or suspend collection activity

are based on full consideration of the overall benefit to the government

and taxpayers’ rights.” We characterized the remaining four sets of

performance standards as somewhat results oriented because they were

indirectly linked with IRS’ mission or one of its three strategic objectives,

but did not clearly specify the results revenue officers were expected to

work toward or establish criteria for evaluating performance in relation to

those results. For example, the set of standards for the “time and workload

management” job element were unclear about the results expected and did

not establish criteria for evaluating performance in relation to those

results. Revenue officers were to take “timely” initial, follow-up, and

closing actions, but the standards did not define what “timely” meant nor

how it was related to results.

IRS’ Comments and

Our Evaluation

We met with IRS officials, including the National Director of the Strategic

Planning Division, on April 29, 1997. IRS also provided a comment letter,

which is reprinted in appendix VII. Generally, the IRS officials said they

were pleased that we characterized IRS’ goals and measures as results

oriented.

In both the meeting and in written comments, IRS officials said that we did

not give the agency sufficient credit for doing a major overhaul of its

measures. However, one of the objectives of our review was to describe

Page 68 GAO/GGD-97-83 Barriers to Managing for Results
Appendix IV 

IRS’ Mission, Strategic Objectives, Key

Performance Indicators, and Performance

Standards for Selected Frontline Regulator

Positions

each agency’s performance measures as of January 1, 1997, not to describe

the specific efforts each agency had taken in developing these measures.

Furthermore, any efforts IRS had made to overhaul its measures prior to

January 1, 1997, should have been reflected in the measures we reviewed.

IRS’ written comments also state that IRS was concerned that we created

our own set of criteria in lieu of using GPRA criteria to evaluate the

agencies’ goals and program performance measures. Other than the fact

that GPRA does not require all of an agency’s goals or program performance

measures to be results oriented, which we emphasize throughout the

report, we do not believe that our criteria differ substantially from GPRA

requirements. For example, the report of the Senate Committee on

Governmental Affairs on GPRA states that strategic or general goals should

elaborate on the mission statement, should constitute a specific set of

objectives for the agency’s programs and operations, and must be

expressed in a manner that allows a future assessment of whether the goal

is being achieved.10 This is very similar to our criteria that greatly

results-oriented strategic goals (1) were linked closely to the agency’s

mission, (2) provided a clear definition of what results were expected, and

(3) permitted assessment of progress in achieving these results.

IRS officials also did not believe that the draft report placed enough

emphasis on the fact that the Technical and Miscellaneous Revenue Act of

1988 prohibits IRS from evaluating employees on the basis of tax

enforcement results. We agree that the Technical and Miscellaneous

Revenue Act of 1988 prohibits IRS from evaluating employees on the basis

of tax enforcement results and included this information in this appendix.

10Senate Report No. 103-58, June 16, 1993, page 24.

Page 69 GAO/GGD-97-83 Barriers to Managing for Results
Appendix V 

EPA’s Mission, Strategic Goals, and

Performance Standards for Selected

Frontline Regulator Positions

EPA’s mission is “to protect public health and to safeguard and improve the

natural environment—air, water, and land—upon which human life

depends.” In doing so, EPA ensures that:

• “Federal environmental laws are implemented and enforced fairly,

effectively, and cost effectively;

• environmental protection is an integral consideration in U.S. policies

concerning economic growth, energy, transportation, agriculture, industry,

international trade, and natural resources;

• national efforts to reduce environmental risk are based on the best

available scientific information; and

• all parts of society—business, state and local governments, communities,

citizens—have full access to information so that they can become full

participants in preventing pollution and protecting human health and the

environment.”

We assessed the results orientation of EPA’s strategic goals and employee

performance standards according to the criteria presented in the scope

and methodology appendix of this report. Because we characterized the

extent to which each agency goal was results oriented, the application of

these criteria represented a higher standard of results orientation than

required by the Results Act. The Act does not require that all of an

agency’s goals be results oriented. Also, GPRA does not cover employee

performance standards. We judged each set of performance standards

according to the expectation set by the President in his March 4, 1995,

memorandum that these standards should “focus on results, not process

and punishment.” Similar to the way in which GPRA views agencies’ goals

and performance measures, we do not believe that each such standard

must be results oriented to achieve the President’s objective.

We applied our criteria to the agencies’ written strategic goals and

employee performance standards and did not consider any supplementary

or contextual information. In some cases an agency’s goals and standards

appeared results oriented on the surface, but closer examination and strict

application of our criteria resulted in a lower characterization. We

recognize that other information outside of the written statements and

outside of our review may have existed that could have resulted in our

characterizing certain goals or sets of standards differently.

Both we and EPA officials characterized the extent of results orientation of

EPA’s strategic goals and employee performance standards for selected

frontline regulator positions. As shown in table V.1, there were differences

Page 70 GAO/GGD-97-83 Barriers to Managing for Results
Appendix V 

EPA’s Mission, Strategic Goals, and

Performance Standards for Selected

Frontline Regulator Positions

in how we and EPA officials viewed the goals and performance standards.

Generally, we agreed that most of EPA’s strategic goals were greatly results

oriented. However, we sharply disagreed regarding the extent to which the

performance standards were results oriented. EPA officials characterized

most of the 11 sets of performance standards as greatly results oriented,

but we characterized most of them as results oriented to little or no extent.

Table V.1: Differences Between Our and EPA Officials’ Characterizations

Number that EPA officials considered

results oriented to

Number we considered results oriented

to

Total

A great

extent Some extent

Little or no

extent

A great

extent Some extent

Little or no

extent

Strategic goalsa 10 10 0 0 8 1 1

Performance standards 11 9 2 0 1 2 8

aEPA did not have program performance measures as of January 1, 1997.

Source: Agency and GAO analysis.

Views on the Results

Orientation of EPA’s

Strategic Goals

As of January 1, 1997, EPA had 10 strategic goals, which were:

1. “Clean air: Every American city and community will be free of air

pollutants at levels that cause cancer or respiratory and other health

problems. The air will be clearer in many areas and life in damaged forests

and polluted waters will rebound as acid rain is reduced.

2. Clean and safe water: America’s rivers, lakes and coastal waters will

support healthy communities of fish, plants and other aquatic life and will

support uses such as fishing and swimming. Wetlands will be protected

and rehabilitated to provide wildlife habitat, reduce floods and improve

water quality. Ground waters will be uncontaminated and every American

public water system will provide water that is safe to drink all the time.

3. Safe food: All foods Americans produce or consume will continue to be

safe for all people to eat.

4. Toxic free communities and workplaces: By relying on pollution

prevention in the way we produce, consume, reuse and recycle materials,

all Americans will live and work in safe and healthy communities.

Page 71 GAO/GGD-97-83 Barriers to Managing for Results
Appendix V 

EPA’s Mission, Strategic Goals, and

Performance Standards for Selected

Frontline Regulator Positions

5. Safe waste management: Wastes produced by every person and business

in America will be stored, treated and disposed of in ways that prevent

harm to people and other living things. Places in America currently

contaminated by hazardous or radioactive materials will not endanger

public health nor the natural environment and will be restored to uses

desired by surrounding communities.

6. Reducing global and transboundary environmental risks: The United

States and other nations will eliminate significant risks to human health

and ecosystems arising from climate change, stratospheric ozone

depletion and other environmental problems of global concern.

7. Empowering people with information and education and expanding

their right to know: Information on the state of the environment will be

user-friendly and widely accessible, increasing opportunities for

Americans to learn about the environment and natural resources and

fostering environmental stewardship through information exchange within

and among communities and across economic sectors.

8. Develop tools for the future: EPA will develop innovative tools and

approaches that will permit environmental protection programs to

transcend the simple additivity of individual media-based programs and

move the Agency towards truly integrated and continually improving

environmental protection programs. The EPA will enhance its science and

research program in order to better anticipate environmental issues

coming over the horizon and to find early solutions.

9. Provide a credible deterrent: The EPA will continue to implement

programs and monitor compliance to ensure that environmental gains that

have been achieved are not lost.

10. Effective management: EPA will establish a management infrastructure

that will set and implement the highest quality standards for effective

internal management and fiscal responsibility.”

EPA officials characterized each of these goals as results oriented to a great

extent because the goals described the results the agency was trying to

achieve. For example, EPA officials said that the goal regarding “clean and

safe water” was greatly results oriented because it addressed the

environmental result of improving or maintaining water quality so that

aquatic communities will be healthy, human uses will be supported,

drinking water will be safe, and ground water will be uncontaminated. EPA

Page 72 GAO/GGD-97-83 Barriers to Managing for Results
Appendix V 

EPA’s Mission, Strategic Goals, and

Performance Standards for Selected

Frontline Regulator Positions

officials said that the goal to “provide a credible deterrent” was greatly

results focused in two ways. First, they said that the enforcement program

targets industries and facilities for attention based in part on the expected

degree of environmental benefit and risk reduction. Second, they cited the

deterrent effect of ensuring that the regulated community perceives that

compliance will be strongly pursued by EPA, so that overall environmental

results are enhanced due to voluntary compliance. Finally, EPA officials

said that the goal for “effective management” was greatly results oriented

because it complied with GPRA’s requirement to measure performance in

all areas, and that the results of the management area were primarily

outputs focused on customer service.

We agreed with EPA officials that 8 of the 10 goals were results oriented to

a great extent. In addition to the reasons cited by EPA officials, we also

viewed the eight goals as relating to EPA’s mission and permitting

assessment of the agency’s progress in achieving desired results. However,

we characterized the goal “developing tools for the future” as results

oriented to some extent because it did not provide a clear definition of the

results expected and it was unclear how EPA would assess progress toward

achieving this goal. We characterized the remaining goal for “effective

management” as results oriented to little or no extent because it was

focused internally and therefore did not relate to the agency’s mission.

EPA Had No Program

Performance

Measures in Place at

the Time of Our

Review

As of January 1, 1997, EPA had not established sets of program

performance measures to assess progress in meeting the agency’s goals.

According to EPA officials, the agency was establishing quantitative

“objectives” that describe what EPA wants to achieve by an agreed-upon

target date. EPA plans to have a full set of program performance measures

relating to these objectives by July 1997.

Although EPA had no official program performance measures as of

January 1, 1997, the agency has been working since 1992 to develop

long-range national environmental goals with results-oriented

performance goals and measures. EPA officials said that some of the

performance measures for the national goals program may be used as

measures for the agency’s strategic goals.

Page 73 GAO/GGD-97-83 Barriers to Managing for Results
Appendix V 

EPA’s Mission, Strategic Goals, and

Performance Standards for Selected

Frontline Regulator Positions

Views on the Results

Orientation of EPA’s

Performance

Standards for

Selected Frontline

Regulator Positions

In contrast to strategic goals and program performance measures, the

administration’s regulatory reinvention initiative, not GPRA, set the

expectation for agencies to focus on results when measuring the

performance of their frontline regulators. EPA identified two frontline

regulator positions for this review, criminal investigators and

environmental scientists.1

 As of January 1, 1997, there were the equivalent

of about 225 full-time criminal investigators at EPA, and the equivalent of

about 1,225 full-time environmental scientists working out of EPA’s

regional offices.2 EPA criminal investigators investigate allegations or

suspected violations of criminal environmental laws, and environmental

scientists implement and enforce environmental regulations.

EPA officials characterized 9 of the 11 sets of performance standards

reviewed for these 2 positions as results oriented to a great extent, and 2

sets of standards as somewhat results oriented. They believed that most of

the sets of standards clearly described the results that should occur

through performance of the standards. However, we characterized eight of

the sets of employee performance standards we reviewed as results

oriented to little or no extent, two as somewhat results oriented, and one

as results oriented to a great extent.

Views on the Results

Orientation of EPA’s

Criminal Investigator

Performance Standards

We and agency officials reviewed five job elements for the criminal

investigator position:3

1. “Professional attributes: In the performance of duties, displays and

possesses professional attributes and conduct which are a benefit to the

agency and which foster good working relationships within and outside

the office. Such professional attributes include physical conditioning,

appearance and demeanor, proficiency with weapons and equipment,

work attitude, and self-improvement measures.

2. Communication (oral and written): Prepares and presents written and

oral communication and reports as needed to the public; Regional

1

Although the President’s March 4, 1995, memorandum on his regulatory reinvention initiative did not

define “frontline regulators,” we defined them as regulatory staff who work directly with the regulated

community by enforcing regulations.

2

EPA also has environmental scientists working in its headquarters offices, but EPA officials said that

these environmental scientists were not frontline regulators.

3

We did not include the “special projects” job element in our review because of the general nature of

these standards and their variability between criminal investigators.

Page 74 GAO/GGD-97-83 Barriers to Managing for Results
Appendix V 

EPA’s Mission, Strategic Goals, and

Performance Standards for Selected

Frontline Regulator Positions

officials; Criminal Investigation Division (CID) Headquarters; supervisors,

the Director, CID; prosecuting attorneys and other officials.

3. Knowledge of directives, laws, regulations, and procedures: Maintains

and demonstrates knowledge of programs, missions, and operations

within EPA; CID policy; and pertinent environmental laws and regulations.

4. Case investigation: Conducts complex investigations within an assigned

area of responsibility.

5. Liaison: Identifies, establishes and maintains contacts with other

federal, state, and local law enforcement and regulatory agencies, EPA

Regional officials, public institutions, the private sector, and other CID

offices.”

EPA officials classified all but one of the sets of performance standards

related to these elements as results oriented to a great extent. For

example, EPA officials said that the set of standards related to the

“communication” job element was greatly results oriented because

presentations that were thorough, easily understood, and appropriately

targeted would result in an accurate understanding of the message being

delivered. EPA officials said the set of standards for the “liaison” job

element was somewhat results oriented because regular communications

with a broad base of external contacts provided the intermediate result of

a highly efficient and timely resource for the agency.

However, we considered four of the sets of performance standards (for the

“professional attributes,” “communication,” “knowledge of directives,” and

“liaison” job elements) as results oriented to little or no extent because

these standards were not tied to any expected results and therefore had no

criteria for evaluating performance associated with expected results. For

example, the standards for the “professional attributes” job element

included maintaining “a good level of physical conditioning,” displaying

“an appearance and demeanor which is consistent with the position,” and

displaying “a positive attitude towards the job.” We did not believe that

these standards clearly described the results that criminal investigators

were expected to work toward or were linked to any environmental

results that EPA was trying to achieve. We viewed the remaining set of

performance standards for the “case investigation” job element as

somewhat results oriented because although it was linked to EPA’s goal to

“provide a credible deterrent,” the set of standards did not clearly describe

the results the investigator was expected to work toward. For example,

Page 75 GAO/GGD-97-83 Barriers to Managing for Results
Appendix V 

EPA’s Mission, Strategic Goals, and

Performance Standards for Selected

Frontline Regulator Positions

investigators were expected to “identify and preserve physical evidence in

accordance with currently acceptable standards,” but the result expected

from following the standards for evidence was not specified.

Views on the Results

Orientation of EPA’s

Performance Standards for

the Environmental

Scientist Position

The environmental scientist position had six job elements:

1. “Customer service: To meet customer expectations with respect to

products or services provided.

2. Communication: To communicate effectively with internal and external

customers in accomplishing the organization’s work.

3. Teamwork: To contribute individual talents and energies toward

achieving team goals and objectives.

4. Grants/cooperative agreements: To coordinate the negotiation and

award of grants, cooperative agreements and/or interagency agreements,

and/or monitor work plans, progress, and activities performed under these

arrangements for acceptability and achievement of objectives.

5. Enforcement/compliance: To ensure compliance with asbestos

regulations under the Clean Air Act, National Emissions Standards for

Hazardous Air Pollutants, Toxic Substances Control Act, and related

legislation and regulations through effective management of compliance

assistance and enforcement activities while meeting programmatic goals

and objectives.

6. Inspections: To conduct compliance inspections consistent with Agency

priorities and programmatic goals and objectives.”

EPA officials classified all but one of the sets of performance standards for

these elements as results oriented to a great extent. For example, they said

that the set of standards for the “customer service” job element covered all

the major indicators of providing highly efficient customer services,

including mutual agreement between the customer and service provider

on product delivery dates, frequent feedback to customers, and checking

to determine whether customer needs were met. EPA officials said the set

of standards related to the “communication” job element was somewhat

results oriented because the set of standards did not cover whether the

communications were helpful to customers or could have been improved,

Page 76 GAO/GGD-97-83 Barriers to Managing for Results
Appendix V 

EPA’s Mission, Strategic Goals, and

Performance Standards for Selected

Frontline Regulator Positions

leaving a somewhat incomplete set regarding whether effective two-way

communication had been achieved.

We agreed that the set of standards associated with the environmental

scientist “enforcement” job element was greatly results oriented because it

was directly related to EPA’s mission and strategic goals, clearly described

the results scientists were expected to work toward, and established

criteria for evaluating their performance directly associated with the

result. However, we characterized the sets of standards for four of the job

elements (“customer service,” “communication,” “teamwork,” and

“grants/cooperative agreements”) as results oriented to little or no extent

because they did not establish what results would be achieved through

performance of the standards and did not establish results-oriented

criteria for evaluating performance. For example, the standards for the

“communication” job element included “conducts internal and external

communications through appropriate channels;” “writes clearly and

concisely, in compliance with established formats, and neat in

appearance;” and “promptly handles and responds to phone, local area

network, or voice mail messages.” It was not evident what environmental

results were anticipated by performing these communications in

accordance with the standards. We classified the remaining set of

standards for the “inspections” job element as somewhat results oriented.

This set was somewhat linked to EPA’s mission of protecting public health,

but the results scientists were expected to work toward and the criteria

EPA would use to evaluate performance toward achieving these results

were unclear.

EPA’s Comments and

Our Evaluation

We met with EPA officials, including the Acting Deputy Director of the

Office of Planning, Accountability, and Analysis and the Acting Director of

the Strategic Planning and Policy Services Division, on April 28, 1997, to

obtain their comments. The EPA officials agreed with the draft report’s

overall message that regulatory agencies face significant barriers in

becoming results oriented and noted that EPA faces a challenge in trying to

determine what portions of changes in environmental conditions can

realistically be considered a result of EPA’s programs. In fact, the officials

said they believed the draft report should more strongly reflect the

barriers that agencies face in order to emphasize that focusing on results

will take time and, in some situations, may be difficult to do. We agree that

managing for results may be difficult in some cases and, therefore,

included more information in the results-in-brief section of the final report

describing the barriers agencies identified.

Page 77 GAO/GGD-97-83 Barriers to Managing for Results
Appendix V 

EPA’s Mission, Strategic Goals, and

Performance Standards for Selected

Frontline Regulator Positions

EPA officials said that some EPA programs were upgrading their ability to

report on environmental results. They said that other EPA programs have

long had in place the environmental monitoring and reporting systems

needed to measure results and that the prospect of meeting GPRA reporting

requirements would not be an insurmountable challenge for these

programs. For example, the officials said that the “criteria air pollutants

program” has consistent, quality-assured data on both environmental

quality and emissions from many of the major sources of air pollution.

They said that these emissions data link EPA’s regulatory actions to

environmental results. However, they said that some EPA programs (those

that provide services) will have more difficulty determining

outcome-related results, as opposed to output-related (i.e., activity)

results. The EPA officials also said that the agency has made significant

progress in revising its goals and developing performance measures since

the time of our review in early January 1997.

EPA officials said that although it is appropriate to hold frontline regulators

accountable for organizational results, we should not have expected

agencies to have results-oriented performance standards that met our

criteria. They said that the Office of Personnel Management and the chief

financial officers and human resource communities have not decided how

to translate the mission of an agency into measurable standards that are

results oriented at the individual level. They also said that we based our

characterizations on a definition that could not be found in any law,

regulation, or guidance.

Although there may not yet be a consensus among the relevant

stakeholders or a formal definition regarding how the measurement of

employee performance should be linked to agencies’ results, the

President’s memorandum nevertheless requires that agencies change the

way they measure both agency and frontline regulators’ performance “so

as to focus on results, not process and punishment.” In addition, in a

June 15, 1995, report to the President, EPA stated that it reviewed its

existing employee performance standards and said that it had made some

changes to those standards and had plans for further changes. Nowhere in

the June 1995 report did EPA assert, as it does now, that it is inappropriate

to expect the agency’s employee performance standards to focus on

results. To develop our criteria for employee performance standards, we

considered Office of Personnel Management guidance on establishing

employee performance standards in conjunction with GPRA criteria for

goals and performance measures as a model.

Page 78 GAO/GGD-97-83 Barriers to Managing for Results
Appendix V 

EPA’s Mission, Strategic Goals, and

Performance Standards for Selected

Frontline Regulator Positions

The officials also said that one of the reasons we may have differed with

agencies’ conclusions about the results orientation of their employee

performance standards was the fact that we did not consider additional

contextual information when we reviewed the standards. As stated in the

draft report, we applied our criteria to the agencies’ performance

standards without other contextual information. We believe that in order

to characterize each agency’s performance standards using a fair and

balanced approach, it was necessary to limit our scope to the standards

themselves. Otherwise, we could have had additional information about

one aspect of an agency’s performance standard but not another, which

would have resulted in an inconsistent application of our criteria. In

addition, Office of Personnel Management guidance states that research

has shown that specific, hard goals result in higher performance than

vague or abstract goals, or no goals at all. Therefore, the President’s

initiative focusing standards on results could improve regulators’

performance in obtaining desired results.

Page 79 GAO/GGD-97-83 Barriers to Managing for Results
Appendix VI 

Scope and Methodology

To describe the agencies’ strategic goals and program performance

measures, we first obtained copies of the goals and measures the agencies

were using as of January 1, 1997. To describe the employee performance

standards for selected regulator positions at these agencies, we first asked

officials in each agency to identify frontline regulator positions that they

felt would fairly represent the actions of their agencies’ regulators.1

 We

then obtained copies of the performance standards being used as of

January 1, 1997, for the selected positions.

To describe the extent to which agency officials and we believed that

these goals, program performance measures, and employee performance

standards focused on results, we developed a standardized data collection

framework. We provided the initial framework to agency officials and told

them that the criteria (as shown in table VI.1) were under development

and would be further refined and revised as we progressed through the

assignment. We left it up to the agency officials to decide whether they

wanted to use these criteria, because we were primarily interested in

obtaining their perspectives about the extent to which and, more

importantly, why they considered their agencies’ goals, measures, and

standards to be results oriented. At a minimum, however, we asked

agencies to characterize their goals, program performance measures, and

employee performance standards as results oriented to a great extent,

some extent, or little or no extent.

1

Using our definition of “frontline regulators”—regulatory staff who work directly with the regulated

community by enforcing regulations—the agencies designated which positions they considered

“frontline regulator” positions.

Page 80 GAO/GGD-97-83 Barriers to Managing for Results
Appendix VI 

Scope and Methodology

Table VI.1: Initial Framework Provided

to Agency Officials To be considered results oriented to a great extent:

Strategic goals Must provide a clear definition of what results, effects, or

impact (i.e., outcomes) are expected from the regulatory

program, and permit assessment of progress in meeting

the goal.

Program performance

measures

Should (1) assess the actual results, effects, or impact of

a program activity compared with its intended purpose;

(2) be linked to a goal; (3) permit comparison of

organizational performance over time; and (4) if

appropriate, include a range of measures such as quality,

quantity, cost, and timeliness. (A range of measures

would be appropriate when the goal is multipart and not

self-measuring, or when the agency has set forth several

measures and we can specify gaps in the measures

used.)

Employee performance

standards

Should describe the objective or overall results which the

frontline regulators are expected to accomplish, and set

forth tangible or demonstrable criteria for evaluating each

performance element directly associated with desired

results. (It should enable a “cold reader” to clearly

understand how an individual would have to perform to

meet the standard.)

In our analysis, we also characterized the agencies’ goals, program

performance measures, and employee performance standards as results

oriented to a great extent, some extent, or little or no extent. As we carried

out these judgments, we expanded our initial framework somewhat as we

learned more about how agencies’ goals or measures were structured. As a

result, the framework we used was in the end somewhat more detailed

than the initial framework we provided to agency officials. We did not

believe these differences materially affected the outcome of our analysis.

Specifically,

• We characterized an agency’s strategic goal as results oriented to a great

extent if it was clearly linked to the agency’s mission, provided a clear

definition of what results were expected, and permitted assessment of

progress in achieving these results. (For example, IRS’ goal to increase

taxpayer compliance was clearly related to IRS’ mission to collect the

proper amount of tax revenue. It provided a clear definition of the

expected results—“to encourage and assist taxpayers to voluntarily file

timely and accurate returns and to pay on time, and, if taxpayers do not

comply, to take appropriate compliance actions.” Finally, it is possible to

assess progress toward increasing compliance.) If a goal was not linked to

an agency’s mission, did not clearly define desired results, and did not

permit assessment of progress toward achieving these results, we

Page 81 GAO/GGD-97-83 Barriers to Managing for Results
Appendix VI 

Scope and Methodology

considered it results oriented to little or no extent. We classified goals that

fell between these two categories as results oriented to some extent.

• We characterized an agency’s set of program performance measures as

results oriented to a great extent if the measures (1) were clearly and

unambiguously linked to the associated goal (or the agency’s mission, if

the goal was not results oriented); (2) provided relevant information for

assessing progress toward achieving the goal; and (3) addressed all

aspects of the goal. We classified sets of program performance measures

that were not linked to a goal or mission, that would not provide relevant

data for an agency to assess progress toward a result, and did not address

all aspects of the goal as results oriented to little or no extent. We

considered sets of measures that fell between these categories as results

oriented to some extent.

• We considered a position’s performance standards to be greatly results

oriented if they clearly described the results that the frontline regulators

were expected to work toward, these results were clearly and

unambiguously linked to the agency’s mission or goals, and the standards

established criteria for evaluating performance directly associated with

the desired results. We classified performance standards as having little or

no results orientation if they were unclear about the results that the

frontline regulators were expected to accomplish and did not establish

criteria for evaluating performance related to expected results. We

considered performance standards that fell between these two categories

as results oriented to some extent. We did not include performance

standards associated with “other duties as assigned” or “special projects”

in this review because of the general nature of these standards.

There are several aspects of our work which need to be emphasized to

clarify the significance of our results. First, our review was not intended to

determine agencies’ compliance with GPRA; compliance was not required at

the time of our review. Furthermore, because we characterized the extent

to which each agency goal and set of performance measures was results

oriented, the application of these criteria represented a higher standard of

results orientation than will be required by GPRA. As noted previously, the

Act does not require that all of an agencies’ goals or measures be results

oriented and the President’s March 4, 1995, memorandum does not

explicitly require that each employee performance standard be results

oriented to achieve the President’s objective.

In applying our criteria for results orientation to the goals, performance

measures, and performance standards, we limited our analysis to the

Page 82 GAO/GGD-97-83 Barriers to Managing for Results
Appendix VI 

Scope and Methodology

documents describing those goals, measures, and standards as of

January 1, 1997, as provided by the agencies. As a result, we did not

consider additional contextual information that may have existed and

could have resulted in our reaching different conclusions about results

orientation if it had been a part of our review. For example, it is possible

that we might have considered the goals or measures for an agency’s

training or inspection program as more results oriented if we had obtained

information about how they ultimately produced mission-oriented results.

We did not judge whether the agencies’ goals and program performance

measures were appropriate or achievable, only whether they appeared to

be results oriented. We also did not assess whether better program

performance measures existed that the agencies could have used. Because

many of these agencies were making changes to those goals and measures

and some of the performance standards at the time of our review, our and

the agencies’ assessments should be viewed as a snapshot at a particular

point in time.

Because we reviewed the performance standards for only a few frontline

regulator positions at each agency, the information we obtained is not

representative of all such positions. Rather, the discussion of these

standards should be viewed as examples, or case studies, of performance

standards used by the agencies as of January 1, 1997. Each of the positions

that were assessed contained several job elements (e.g., oral and written

communication or customer service) that, in combination, constituted the

major responsibilities for the position. Each job element, in turn, had a set

of performance standards that described the level of performance that the

employee needed to meet to be considered successful in that position.

Table VI.2 shows the seven frontline regulator positions that we focused

on in assessing employee performance standards. The positions had a total

of 30 job elements, each accompanied by a set of performance standards

associated with that element.

Page 83 GAO/GGD-97-83 Barriers to Managing for Results
Appendix VI 

Scope and Methodology

Table VI.2: Frontline Regulator

Positions and Number of Job Elements

as of January 1, 1997

Agencya

Position (and number

of frontline regulators) Description of position

Number

of job

elementsb

OSHA Compliance safety and

health officer

(930)c

Conduct workplace safety and

health inspections.

3

FAA Air traffic control

specialist (12,900)

Regulate air traffic throughout

the nation. 3

Aviation safety inspector

(1,525)

Inspect and certify aircraft for

safe operations. 3

IRS Customer service

representative (5,750)

Provide information and

direction on tax issues to the

public.

5

Revenue officer (7,360) Collect delinquent taxes and

secure delinquent returns.

5

EPA Criminal investigator

(225)d

Investigate allegations or

suspected violations of

criminal laws.

5

Environmental scientist

(1,225)c

Implement and enforce

environmental regulations. 6

Total e 30

aFDA is not included because, as of January 1, 1997, the agency was using performance

standards that were customized for individual employees.

bWe reviewed the sets of performance standards that were associated with each of these job

elements.

cIncludes both safety and health inspector positions.

dThe number of EPA frontline regulators (for the criminal investigator and environmental scientist

positions) were full time equivalent (FTE) staffing levels, not number of frontline regulators. In

addition, the number of FTEs in the environmental scientist position includes only environmental

scientists in EPA’s regional offices. EPA officials said that they would not classify environmental

scientists who work in headquarters as frontline regulators.

eNot applicable because the numbers listed include both number of staff and FTEs and therefore

these amounts cannot be added to a meaningful total.

Source: Agency information.

To describe the aids and barriers that officials in the agencies said they

faced in attempting to focus on results, we asked officials in each agency

to describe the aids and barriers they had experienced. We also reviewed

related documents that described any problems or successes agencies had

encountered.

Page 84 GAO/GGD-97-83 Barriers to Managing for Results
Appendix VII 

Comments From OSHA and IRS

Page 85 GAO/GGD-97-83 Barriers to Managing for Results
Appendix VII 

Comments From OSHA and IRS

Page 86 GAO/GGD-97-83 Barriers to Managing for Results
Appendix VII 

Comments From OSHA and IRS

Page 87 GAO/GGD-97-83 Barriers to Managing for Results
Appendix VII 

Comments From OSHA and IRS

Page 88 GAO/GGD-97-83 Barriers to Managing for Results
Appendix VIII 

Major Contributors to This Report

General Government

Division

Curtis Copeland, Assistant Director, Federal Management and Workforce

 Issues

Susan Ragland, Evaluator-in-Charge

Steven Lozano, Senior Evaluator

Thomas Beall, Technical Analyst

Kiki Theodoropoulos, Communications Analyst

Joseph S. Wholey, Senior Advisor for Evaluation Methodology

(410043) Page 89 GAO/GGD-97-83 Barriers to Managing for Results
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the

following address, accompanied by a check or money order

made out to the Superintendent of Documents, when

necessary. VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address

are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 6015

Gaithersburg, MD 20884-6015

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000 

or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and

testimony. To receive facsimile copies of the daily list or any

list from the past 30 days, please call (202) 512-6000 using a

touchtone phone. A recorded menu will provide information on

how to obtain these lists.

For information on how to access GAO reports on the INTERNET,

send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov

PRINTED ON RECYCLED PAPER
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. G100