United States General Accounting Office

GAO General Government Division

April 1998

Version 1 The Results Act

An Evaluator’s Guide

to Assessing Agency

Annual Performance

Plans

GAO/GGD-10.1.20

Preface

As Congress seeks to reduce the cost and improve the performance of the

federal government, holding agencies accountable for results is believed to

be key to better management of programs. However, Congress found that

the lack of adequate information on federal agencies’ performance was

handicapping congressional policymaking, spending decisions, and

oversight and diminishing federal accountability for program results. In

response, Congress enacted the Government Performance and Results Act

of 1993 (the Results Act) to address these concerns and thereby improve

federal program effectiveness and public accountability.

The Results Act seeks to improve the management of federal programs by

shifting the focus of decisionmaking from staffing and activity levels to the

results of federal programs. Under the Results Act, executive agencies are

to prepare 5-year strategic plans that set the general direction for their

efforts. Agencies then are to prepare annual performance plans that

establish the connections between the long-term strategic goals outlined in

the strategic plans and the day-to-day activities of program managers and

staff. Finally, the Act requires that each agency report annually on the

extent to which it is meeting its annual performance goals and the actions

needed to achieve or modify those goals that have not been met.

This guide is intended to facilitate assessment of the strengths and

weaknesses of agencies’ annual performance plans in meeting both the

requirements of the Results Act and congressional expectations that the

plans inform Congress and the public about agencies’ performance goals,

including how they will accomplish them and how they will measure their

results. While the Results Act does not require a specific format for the

annual performance plan, it requires the plan to (1) identify annual goals

and measures covering each of its program activities, (2) discuss the

strategies and resources needed to achieve annual goals, and (3) describe

the means the agency will use to verify and validate its performance data.

This guide is organized around three core questions that can help in

determining how well the agency’s plan meets these general requirements.

For each core question, issues are identified that need to be addressed,

and criteria and guidance are provided to assist evaluators in answering

the questions raised by these issues. A companion assessment guide

intended to facilitate congressional use of agency performance plans

contains the same core questions and issues.

As agencies and Congress gain experience in developing and using annual

performance plans, additional issues and key questions will likely emerge.

Page 1 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Preface

Accordingly, this guide is subject to refinement to reflect agencies’ and

Congress’ experiences.

This guide was developed by the Advanced Studies and Evaluation

Methodology group in GAO’s General Government Division. If you have

questions or comments, please contact Susan Westin or Stephanie

Shipman, Advanced Studies and Evaluation Methodology, on

(202) 512-7997. Other contributors to this guide were Joyce D. Corry, Allen

C. Lomax, and Thomas M. Beall, all of the General Government Division,

and Stan Divorski, on special assignment to GAO from the Office of the

Auditor General of Canada. An electronic version of this guide is available

from GAO’s World-Wide Web server at the following address:

http://www.gao.gov.

Susan S. Westin

Associate Director, Advanced Studies and

 Evaluation Methodology

J. Christopher Mihm

Associate Director, Federal Management

 and Workforce Issues

Page 2 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Page 3 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Contents

Preface 1

Introduction 6

Results Act Requirements 6

How This Guide Was Developed 8

General Guidance 10

Assessing the Plan Against the Questions and Criteria 12

Section I 

Annual Performance

Goals and Measures

14

Core Question 1 14

Section II 

Strategies and

Resources

32

Core Question 2 32

Section III 

Verification and

Validation

39

Core Question 3 39

Figures Figure I.1: Summary of the Issues and Criteria for Core Question

1

14

Figure I.2: Excerpt from the Department of Energy’s 1999 Budget 24

Figure I.3: Excerpt from the Environmental Protection Agency’s

1999 Budget

25

Figure I.4: Excerpt from the General Accounting Office’s 1999

Budget

26

Figure I.5: Aligning Annual Performance Plans With the Budget

Account and Program Activity Structure

28

Figure II.1: Summary of the Issues and Criteria for Core Question

2

32

Figure III.1: Summary of the Issues and Criteria for Core

Question 3

39

Page 4 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Page 5 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Introduction

Results Act

Requirements

The Results Act seeks to improve the management of federal programs, as

well as their effectiveness and efficiency, by establishing a system under

which agencies set goals for program performance and measure their

results. Specifically, the Results Act requires executive agencies to prepare

multiyear strategic plans, annual performance plans, and annual

performance reports.

Under the Act, strategic plans are the starting point for setting goals and

measuring progress towards them. The Act requires virtually every

executive agency to develop a strategic plan, covering a period of at least 5

years forward from the fiscal year in which the plan is submitted. These

strategic plans are to include an agency’s mission statement, long-term

general goals, and the strategies that the agency will use to achieve these

goals. The plans should also explain the key external factors that could

significantly affect achievement of these goals, and describe how

long-term goals will be related to annual performance goals.1

Next, the Results Act requires executive agencies to prepare an annual

performance plan that shall

“(1) establish performance goals to define the level of performance to be

achieved by a program activity;

“(2) express such goals in an objective, quantifiable, and measurable

form . . . ;

“(3) briefly describe the operational processes, skills and technology, and

the human, capital, information, or other resources required to meet the

performance goals;

“(4) establish performance indicators to be used in measuring or assessing

the relevant outputs, service levels, and outcomes of each program

activity;

“(5) provide a basis for comparing actual program results with the

established performance goals; and

“(6) describe the means to be used to verify and validate measured

values.”2

1

Agencies were to submit their first strategic plans to the Office of Management and Budget (OMB)

and Congress by September 30, 1997.

2

P.L. 103-62, sec. 4(b).

Page 6 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Introduction

The annual performance plans should systematically provide

congressional decisionmakers with information on the results to be

achieved for a proposed level of resources. These plans also reinforce the

connections between the long-term strategic goals outlined in the strategic

plans and the day-to-day activities of program managers and staff.

Moreover, through its requirements for these annual plans, the Results Act

established the first statutory link between agencies’ budget requests and

their performance planning efforts. The Act requires that performance

goals and measures be linked to the program activities in agencies’ budget

requests. The term “program activity” refers to the listings of projects and

activities in the Appendix to the Budget of the United States Government.

Program activity structures are intended to provide a meaningful

representation of the operations financed by a specific budget account.

In addition, agency performance plans are to be the basis for the overall

federal government performance plan that the Office of Management and

Budget (OMB) must submit to Congress. OMB is to submit this

governmentwide plan each year to Congress with the president’s budget.

According to the Senate Committee on Governmental Affairs report on the

Act, the overall governmentwide plan is to present to Congress a single,

cohesive picture of the federal government’s annual performance goals for

the fiscal year.3

 The agency annual performance plans are to be sent to

Congress and made available to the public soon after transmittal of the

president’s budget.4

Finally, the Results Act requires each agency to prepare annual reports on

program performance for the previous fiscal year. In each report, the

agency is to compare its performance with the goals established in its

annual performance plan, summarize the findings of program evaluations

completed during the year, and describe the actions needed to address or

revise any unmet goals. The performance reports are to be issued by

March 31 each year, with the first (for fiscal year 1999) to be provided to

Congress and the president by March 31, 2000.

Translating the use of agency resources into concrete, measurable results

will be a continual challenge requiring time and effort. Agencies’ early

efforts to pilot test the Act’s requirements faced a variety of significant

3

S. Rep. No. 58, 103d Cong. 1st Sess. (1993).

4

The first agency performance plans are to cover fiscal year 1999.

Page 7 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Introduction

challenges, some of which will not be easily overcome.5 One set of

challenges arises from the complications of government structure. Others

involve methodological difficulties in identifying performance measures or

a lack of the data needed to establish goals and assess performance.6 GAO’s

review of agencies’ strategic plans produced in September 1997 found that

they provided a workable foundation for Congress to use in pursuing its

responsibilities.7

 However, GAO also found that agencies need to make

continued progress in setting a strategic direction, coordinating

crosscutting programs, and ensuring their capacity to gather and use

reliable performance and cost data. With sustained attention from

Congress and executive agencies, annual performance plans can be an

invaluable tool for informing policy decisions, improving program

management, enhancing accountability, and helping to increase American

citizens’ confidence in their government.

How This Guide Was

Developed

In May 1997, GAO developed a guide to facilitate congressional review of

agencies’ strategic plans at the request of the Chairmen of the Government

Reform and Oversight Committee, Committee on Appropriations, and

Committee on the Budget in the House of Representatives.8

 The guide GAO

developed contained key questions to help Congress determine how these

plans could be improved to better support congressional and agency

decisionmaking. Subsequently, GAO employed that guide, the Results Act,

and other guidance to assess whether agencies’ draft strategic plans met

the requirements of the law and congressional expectations.

Building on that guide, those Chairmen, the Speaker of the House, the

Majority Leader of the House, the Chairman of the House Committee on

Science, and the Chairmen of the Committee on the Budget and the

Committee on Governmental Affairs of the Senate subsequently asked GAO

to develop a guide to help congressional decisionmakers both elicit the

information that Congress needs from agencies’ annual performance plans

and assess the quality of those plans.9

 That guide provides questions for

assessing the strengths and weaknesses of agencies’ annual performance

5

The Government Performance and Results Act: 1997 Governmentwide Implementation Will Be

Uneven (GAO/GGD-97-109, June 2, 1997).

6

Managing for Results: Analytic Challenges in Measuring Performance (GAO/HEHS/GGD-97-138,

May 30, 1997).

7

Managing for Results: Agencies’ Annual Performance Plans Can Help Address Strategic Planning

Challenges (GAO/GGD-98-44, Jan. 30, 1998).

8

Agencies’ Strategic Plans Under GPRA: Key Questions to Facilitate Congressional Review

(GAO/GGD-10.1.16, May 1997).

9

Agencies’ Annual Performance Plans Under the Results Act: An Assessment Guide to Facilitate

Congressional Decisionmaking (GAO/GGD/AIMD-10.1.18, February 1998).

Page 8 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Introduction

plans in meeting both the requirements of the Results Act and

congressional expectations that the plans will inform Congress and the

public about agencies’ performance goals, including how they will meet

them and how they will measure their results. GAO developed this

companion evaluator’s guide to help GAO and other evaluators make

detailed assessments using the same core questions and issues as are

contained in the congressional guide. In particular, this evaluator’s guide

includes criteria, additional discussion and guidance, and illustrations of

what to look for in assessing agencies’ progress.

While the issues in this guide make up a complete assessment of an

agency’s plan when considered together, and address concerns likely to be

common across a variety of agencies and congressional users, the

significance of some issues may vary for specific agencies and for

appropriations, authorization, oversight, and budget committees. This

guide, as well as the assessments derived from it, should help identify

additional questions that individual committees may wish to pursue

subsequently in more depth. GAO hopes that this guide will also be useful

to agencies in preparing future annual performance plans.

This guide was developed from the Results Act requirements for agency

performance plans; guidelines contained in OMB Circular No. A-11, Part 2;

and other relevant documents. For the purposes of this guide, the six

requirements of the Results Act for the annual performance plans (stated

previously) were collapsed into three core questions. GAO also drew on its

work over the last several years examining agencies’ pilot efforts to

implement the Act, and in particular our work identifying promising

practices for the Act’s effective implementation.10 The core questions and

issues benefited from the insights of a wide range of congressional staff

representing oversight, authorization, budget, and appropriation

committees. GAO also received and incorporated valuable comments from

senior agency officials directly involved in the development of annual

performance plans, including officials from OMB and other agencies, as

well as outside experts. To help identify the information needed to apply

this guide, the key questions, criteria and guidance were critiqued by

selected evaluators across GAO with experience in reviewing the strategic

plans of a variety of different agencies.

10Executive Guide: Effectively Implementing the Government Performance and Results Act

(GAO/GGD-96-118, June 1996).

Page 9 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Introduction

General Guidance

Objectives and Scope of

the Intended Review

This document outlines the issues that evaluators should address, in

reviewing an agency’s annual performance plan prepared under the

Results Act, to assess the extent to which the plan meets the requirements

of the Act and related expectations. The focus is on expectations

concerning how well the plan communicates what an agency proposes to

accomplish on an annual basis, how it will accomplish those goals, and

how it will assess whether those results were achieved. In order to

encourage agencies to improve their performance, as well as to depict

what can be expected in the future, GAO developed criteria geared to

agencies with a mature performance measurement system. However, in

the early stages of implementing the law, performance plans are not

expected to completely meet these expectations, because overcoming the

challenges involved may take many years. Agencies’ early performance

plans should be judged, instead, on whether the agencies are making

reasonable progress towards meeting the expectations. Therefore, this

document is intended to guide assessment of agency progress towards

meeting those expectations and identify opportunities for agencies to

refine and clarify their plans.

While the Results Act does not require a specific format for the annual

performance plan, it does require the plan to (1) identify annual goals and

measures covering each of its program activities, (2) discuss the strategies

and resources needed to achieve annual goals, and (3) describe the means

the agency will use to verify and validate its performance data. This guide

is organized around three core questions that can help determine how well

the agency’s plan meets these general requirements:

1. Annual Performance Goals and Measures: To what extent does the

agency’s performance plan provide a clear picture of intended

performance across the agency?

2. Strategies and Resources: How well does the performance plan discuss

the strategies and resources the agency will use to achieve its performance

goals?

3. Validation and Verification: To what extent does the agency’s

performance plan provide confidence that its performance information

will be credible?

Page 10 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Introduction

To help evaluators answer each core question, the guide identifies key

issues under each core question that the plan should address. For each of

these issues, the guide provides criteria—reflecting expectations derived

from the Act and other guidance—for evaluators to use to assess how well

the plan addresses that issue. The guidance also identifies what evaluators

should look for in making that assessment, and provides some illustrations

of the concept and how the information might be presented in the agency’s

plan. It is expected that, as the agencies and Congress gain experience

with the Act, additional issues will emerge. Those issues will be

incorporated into subsequent versions of this document and the

congressional guide, as appropriate.

This guide was developed to direct a timely, general assessment of an

agency’s annual performance plan viewed as an informative document. It

is not intended to direct a thorough review of an agency’s proposed

performance reporting system, although it may identify issues that deserve

additional in-depth attention in a follow-up review. This intended review is

expected to be based on an agency’s strategic and annual performance

plans; budget justification materials; the capital asset and technology

plans; the Appendix to the Budget of the United States Government for the

relevant fiscal year; previous reviews, if any, of the agency’s strategic plan;

and the evaluators’ knowledge of, and available materials on, the agency’s

programs, operations, and databases. In addition, evaluators will want to

consult the Results Act; the Senate Report on S. 20,11 OMB Circular No.

A-11, part 2; and OMB’s checklist for reviewing agencies’ performance plans

(dated November 24, 1997). Interviews with agency and Inspector General

officials may also be required to clarify issues that are incompletely

described in the performance plan.

Assessing the Performance

Plan as a Whole

The review guide poses three core questions about how well the

performance plans meet both the requirements of the Act and

congressional expectations for the plans. Under each of these core

questions are issues and criteria spelling out the key dimensions of those

requirements and the expectations about how they should be addressed in

the performance plan. Evaluators are expected to develop answers to each

of the three core questions by reviewing the plan against the individual

criteria for each issue, to assess how well the issues were addressed.

Answers to the three core questions should then be combined to form an

overall judgment about what level of progress the plan as a whole

represents—for example, whether the plan “generally meets,” “partially

11S. Rep. No. 58, 103d Cong. 1st Sess. (1993).

Page 11 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Introduction

meets,” or “falls well short of meeting” these expectations. A plan

generally meets expectations if it addresses all the key issues and contains

no significant deficiencies. It is quite likely that the quality of an agency’s

plan will be uneven in the way it addresses distinct components of the

agency, because an agency plan may represent a compilation of plans

developed by components of the agency. If the parts of the plan that

address one or two components fall well short of expectations, but the

remainder of the plan meets them, evaluators should characterize the

agency’s plan as having generally met expectations, and then note those

exceptions.

Each of the three core questions should be answered by characterizing the

strengths and weaknesses of the performance plan and identifying any

areas needing improvement, in accordance with how well each of the

issues was addressed. In addition, evaluators are encouraged to provide

suggestions for how the plan could be improved. Since a basis for setting

expectations for a specific level of performance is not yet available,

evaluators must assess the plans against the ideal, while recognizing that

few initial plans are likely to reach that level. In assessing progress

towards the ideal, evaluators should also recognize that some deficiencies

are more important than others, and that an unsuccessful agency effort to

address a particularly challenging task shows greater progress towards

accomplishing the Act’s goals than does not having attempted the task at

all. Answers to the core questions should use the same three categories

recommended for assessing the plan overall: “generally meets,” “partially

meets,” or “falls well short of meeting” these expectations.

Assessing the Plan

Against the Questions

and Criteria

“Answers” to each of the three core questions should strive to capture

whether or not the plan generally addressed each issue, while also

recognizing where the plan handled an issue very well. In some cases,

while the criteria provide standards representing the Act’s requirements,

the accompanying textual guidance also describes what a particularly

good plan would contain.

Scoring the plans by tallying up the instances of compliance—for example,

by counting the number of performance measures that were clearly linked

to the strategic plan—is not recommended. Rather, in this example, the

concern should be that, if there are no performance goals specified for a

given strategic goal, or if there are a large number of performance goals

with no visible link to the agency’s strategic plan, then there should be an

explanation of those exceptions. In some cases, evaluators will need to

Page 12 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Introduction

make a judgment as to whether the exceptions are limited enough, or

whether the presentation appears to be a reasonable and good-faith effort,

to warrant a “generally meets expectations” rating. The criteria and

accompanying guidance for each core question and issue are provided to

help guide these judgments.

All of the issues outlined in the guide are not equally important, and some

issues may be more important for some agencies than for others. It is not

necessary, or even likely, for evaluators to draw a conclusion against every

criterion in the guide. However, in those cases where evaluators lack the

information to assess the plan on an issue that seems significant to

forming a conclusion on one of the core questions, they should so indicate

in their report.

Although criteria and guidance are provided to help evaluators judge

whether an issue was adequately addressed, the criteria are not

self-assessing. The issues involved in setting goals for, and measuring the

accomplishments of, public programs are complex and subject to

interpretation, and can also vary from one agency context to another.

Thus, evaluators must apply the criteria in consideration of the specific

circumstances of each agency.

Page 13 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section I 

Annual Performance Goals and Measures

Core Question 1 To what extent does the agency’s performance plan provide a clear

picture of intended performance across the agency?

Figure I.1: Summary of the Issues and Criteria for Core Question 1

Guidance. As the Results Act requires, the annual performance plan must

provide a basis for an agency to compare actual results with performance

goals (Results Act, sec. 4(b)). To do this, the agency needs to set goals,

develop measures, and show in its performance plan how it will use these

Page 14 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section I 

Annual Performance Goals and Measures

goals and measures to assess performance.12 By showing the relationship

between the annual performance goals and the agency’s strategic goals

and mission, an agency’s performance plan can demonstrate how the

agency intends to make progress toward achievement of its strategic

goals.

In addition, the Results Act requires that the performance plan contain one

or more performance goals covering the program activities in the agency’s

budget, which can be done by aggregating, disaggregating, or

consolidating the program activities. Thus, the plan should define

performance to be achieved throughout the agency. Performance plans do

not have to include the complete array of goals and measures used in

managing programs, but they should reflect a picture of intended

performance without any significant gaps. Agencies with programs that

have strategic or performance goals similar to those of other federal

programs should address the need for coordination among such

crosscutting efforts. Annual performance plans can be useful tools for

identifying the need for coordination and for discussing ways to address

that need. These plans can help ensure that the goals are congruent and

that crosscutting efforts are mutually reinforcing.

Issue 1: Defining Expected

Performance

To what extent do the annual performance goals and measures provide a

succinct and concrete statement of expected performance for subsequent

comparison with actual performance?

Guidance. The Results Act requires an agency’s annual performance plan

to contain both a set of annual goals that establishes its intended

performance and measures that can be used to assess progress towards

achieving those goals. Some goals are self-measuring—that is, they are

expressed objectively and quantifiably—and thus do not require the use of

additional measures. For example, a performance goal to staff 300 airport

control towers on a 24-hour basis in a given year would not require

additional measures. Other goals, which may be more abstract, require the

defining of specific performance measures in order to assess progress

12The Results Act defines a performance goal as a target level of performance expressed as a tangible,

measurable objective against which actual achievement can be compared, including a goal expressed

as a quantitative standard, value, or rate. It is likely that such performance goals are self-measuring,

requiring no separate measures. However, other performance goals, those that are more abstract and

not tangible and measurable on their own, should be expected to have one or more separate

performance measures or indicators as needed to help define the goal for measurement. In this guide,

the terms “measure” and “performance measure” also include the concept of what are frequently

called “performance indicators.” In general, performance measures are a tabulation, calculation,

recording of activity or effort, or assessment of results compared to intended purpose, that can be

expressed quantitatively or in another way that indicates a level or degree of performance.

Page 15 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section I 

Annual Performance Goals and Measures

towards the goals. For example, a goal for tribal reservations to meet

national standards for maternal and child health would require the use of

specific measures reflecting those standards, such as morbidity and

mortality rates for this population, percentage of low birth weight babies,

the percentage of children receiving their full immunization series,

frequency of pediatric checkups, etc. In either case, the goals and

measures should be objective and precise, and should allow for the

assessment of performance. The measures should also be clearly related

to the performance they will be used to evaluate. While the number of

measures for each goal at a given organizational level should not be

excessive, it is critical that they represent the important dimensions of

performance for that goal.

An agency can use output goals, outcome goals, or some combination of

the two to reflect the agency’s intended performance. However, the

Results Act is clearly outcome-oriented, and thus an agency’s performance

plan should include outcome goals whenever possible. Outputs can be

defined as the direct products and services delivered by a program.

Outcomes are the results of those products and services. For example,

for a job training program whose objective is to provide the target

population with the skills needed to find and keep a job, an output might

be the percentage of participants who have completed training, while an

outcome might be the percentage of participants employed 1 year after

completing the training. In addition, results can be either intermediate

outcomes representing immediate benefits for program clients, such as

obtaining employment in their area of training, or end outcomes

representing benefits to the public at large, such as increased

self-sufficiency.

Criterion 1.1 The performance measures contained in the annual performance plan

should adequately indicate progress towards the performance goals

(Senate Report 103-58, p. 29).

Guidance. When performance goals are not self-measuring, performance

measures should translate those goals into concrete, observable

conditions that determine what data to collect to learn whether progress

was made towards achieving goals. Such measures are meant to cover the

key aspects of performance that will enable agencies to assess

accomplishments, make decisions, realign processes, and assign

accountability. To do this, the measures must clearly represent or be

related to the performance they are trying to assess. For example, a

measure of the average processing time for tax returns does not represent,

Page 16 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section I 

Annual Performance Goals and Measures

nor is it clearly related to, the goal of increasing the accuracy of tax return

processing. To adequately indicate progress towards the performance

goals, the measures need to validly (or appropriately) represent the

performance goals and sufficiently cover the key aspects of the agency’s

performance.

To be valid representations of the performance goals, the measures should

have a clearly apparent or commonly accepted relationship to the

intended performance or have been shown to be reasonable predictors of

the desired behaviors or events. For example, because the effectiveness of

immunizations for childhood diseases is well acknowledged, the

percentage of children receiving their full immunization series is

considered a valid and appropriate measure of their protection from those

diseases, and thus a valid indicator (or less direct measure) of their health

status. Also, the measures should not represent too constrained a view of

performance. For example, because only some people choose to appeal

the results of an audit of their tax return, the rate at which those cases are

overturned on appeal would not be a valid measure of the accuracy of all

tax return processing. In addition, to ensure that they will permit reliable

measurement, the measures should be amenable to applying standard

procedures for collecting data or calculating results—so that they would

be likely to produce the same assessment of results if applied repeatedly

to the same situation.

Valid measures should also not be unduly influenced by factors outside the

program’s control—because that influence makes it difficult to attribute

changes in the measure to the effectiveness of the program. However,

evaluators should note that certain measures may be unavoidably

influenced by factors outside the program but still be appropriate to use.

For example, the health of the economy may influence the rate of defaults

on loans to students or homeowners, but those measures assess an

important component of loan programs’ performance and can also reflect

a program’s success in managing its loans. When no potential measure

seems to be reasonably free of such influences, a program may choose to

use multiple measures to avoid drawing mistaken conclusions about the

program’s effectiveness.

The measures sufficiently cover key aspects of performance when

individual measures or sets of measures represent the important

dimensions of their performance goals and when the measures reflect the

core functions of their related programs or activities. For example, if a

program ranks the crashworthiness of vehicles by measuring the extent of

Page 17 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section I 

Annual Performance Goals and Measures

injury to passengers and damage to vehicles in crash tests that use only

front-end collisions, the measure may not be valid for ranking the vehicles’

overall crashworthiness. However, if front-end collisions are most

frequently those collisions producing the most injury and damage, then

this measure of crashworthiness may be the most appropriate one for

assessing programs that aim to reduce injury and damage from vehicle

collisions. Further, the number of measures for each goal at a given

organizational level should not be so large that some measures are

redundant, or unnecessary for assessing progress toward the goal.

In addition, because public programs often have multiple objectives and

must balance competing interests, a given program or activity often should

contain multiple goals and measures responding to multiple priorities,

such as quality, timeliness, cost, and outcomes. When complex program

goals are broken down into a set of component quantifiable measures, it is

important to ensure that the overall measurement of performance does not

become biased by measures that assess some priorities but neglect others.

Also, in reviewing measures, evaluators should be sensitive to the

possibility that the use of or reliance on certain measures could produce

unintended consequences. For example, for a welfare program that aims

to encourage the economic self-sufficiency of mothers yet only proposes

to measure the percentage who found employment within 6 months, an

unintended consequence might be a tendency of the program to provide

temporary rather than permanent positions for its clients.

The following steps capture the main elements for assessing the annual

performance plan against this criterion of whether the performance

measures contained in the plan adequately indicate progress towards the

performance goals.

The evaluator should

• assess whether the measures appear to be valid representations of the

performance goals;

• assess whether the measures sufficiently cover key aspects of

performance and adequately capture important distinctions between

programs; and

• assess whether the agency’s use of its measures, either individually or as a

set, could result in promoting any unintended consequences.

Criterion 1.2 The annual performance goals or their measures should be objective,

measurable, and quantifiable (Results Act, sec. 4(b)).

Page 18 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section I 

Annual Performance Goals and Measures

Guidance. For goals and measures to be considered objective, they should

be reasonably free of any significant bias or manipulation that would

distort the accurate assessment of performance. To the greatest extent

possible, the goals and measures should not require subjective

considerations or judgments to dominate the measurement. The

performance goals and measures usually should include a quantifiable,

numerical target level or other measurable value. Examples of quantifiable

goals would be goals to have 50 percent of participants in a job training

program obtain a job related to their training within the next year and hold

the job for at least 6 months, or to increase by 5 percentage points the

percentage of children under age 2 who have been adequately immunized.

However, without adequate baseline data, goals like the latter one may not

permit subsequent comparison with actual performance. If a goal is not

quantifiable, it may still be susceptible to measurement if a determination

of its level of achievement can be demonstrated and independently

verified. For example, a goal that is not quantifiable but is a statement of a

tangible condition or event, such as landing a person on the moon or

discovering the cure for a disease, can be measurable.

If a goal cannot be expressed in an objective, specific, and measurable

form, the Results Act allows OMB to authorize agencies to develop

alternative forms of measurement. An alternative form may be either

(1) separate descriptive statements of a minimally effective program and a

successful program, expressed with sufficient precision and in such terms

that would allow for an accurate, independent determination of whether

the actual performance meets the criteria of the description; or (2) some

other alternative that allows an accurate, independent determination to be

made of how actual performance compares with the goal as stated.

If OMB authorizes an agency’s request to use an alternative form, OMB has

stated that the plan need not document the authorization. Thus, evaluators

are on their own to identify such use of alternative forms. If the annual

performance plan includes performance goals that do not meet the

characteristics of being objective, measurable, and quantifiable, evaluators

should assess those goals against the characteristics for alternative forms

of measurement allowed by the Results Act. If the agency determines that

it is infeasible or impractical to express any performance goals in any form

for a program activity, the performance plan should state why.

Page 19 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section I 

Annual Performance Goals and Measures

Key features to look for include

• performance goals and measures that indicate specifically what should be

observed, in which population or conditions, in what time frame (for

example, an increase in immunizations of children under age 2 by

10 percent over the previous year);

• the use of baseline or benchmark data to allow for the assessment of

progress towards goals;

• performance goals and measures that state a particular target level of

performance, either as an absolute value (for example, 20,000 served) or

as a targeted level of improvement (10 percent increase over the previous

year’s level); and

• the use of alternative forms of goals and measures.

The evaluator should

• assess whether the measures avoid limiting observations to certain types

of cases or situations that may be more or less likely to show progress

toward the goal, thus avoiding biasing the measurement;

• assess the extent to which the performance goals and measures allow for

the comparison of actual performance with intended performance;

• when needed, determine whether baseline or trend data are contained or

referred to in the plan so as to allow for the measurement of goals defined

in terms of improvement;

• determine whether any attempts to benchmark against other

organizations’ performance are made and assess whether such attempts

seem appropriate, based on previous and ongoing GAO work or other

known sources;

• identify alternative forms used, if any; and

• if applicable, assess the alternative forms to determine whether such

forms allow for the comparison of actual performance with intended

performance (i.e., to assess whether or not the goal has been achieved).

Criterion 1.3 Outcome goals should be included in the annual performance plan

whenever possible (Senate Report 103-58, p. 15).

Guidance. An agency’s intended performance can be represented in its

performance plan by output goals, outcome goals, or some combination of

the two. It is anticipated that agencies will include output goals in their

annual plans, and these output goals may predominate, especially at this

early stage. However, agencies should make every attempt to identify and

use annual outcome goals whenever possible to reflect the results of their

activities.

Page 20 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section I 

Annual Performance Goals and Measures

Key features to look for include

• goals expressed as the results of the services or goods produced by the

program, such as actions that occur in response to, or conditions resulting

from, program activities, and

• discussions of efforts to identify and use outcome goals, as well as

explanations of why outcome goals were not used (e.g.: outcomes will not

be achieved on an annual basis; outputs are being used as intermediate

outcomes; the program’s primary objective is to produce an output; or it

may not be possible to identify an outcome for the program).

The evaluator should

• assess whether outcome goals are included in the performance plan and if

so, the extent to which they are used;

• if outcome goals are not included, assess whether the plan provides a

reasonable and credible explanation or rationale for why not; and

• based on the evaluator’s understanding of program purposes, evaluate the

plan for missed opportunities to identify or use outcome goals.

Issue 2: Connecting

Mission, Goals, and

Activities

How are the agency’s annual performance goals linked to the agency’s

mission, strategic goals, and program activities in its budget request?

Guidance. A clear relationship should exist between an agency’s long-term

strategic goals and mission and the performance goals in the annual

performance plan. Successful organizations try to link performance goals

and measures to the organization’s strategic goals and, to the extent

possible, have performance goals that will show annual progress towards

achieving their long-term strategic goals. For example, an annual

performance goal to reduce the rate of transportation-related fatalities per

passenger-mile-traveled by 1 percent would show annual progress toward

the strategic goal to promote the public health and safety by working

toward the elimination of transportation-related deaths, injuries, and

property damage. An agency’s performance plan should show the

relationship between the annual performance goals and the strategic goals

and mission through a crosswalk, other illustration, or description. If the

relationship is minimal or nonexistent, the evaluator should point out that

there may be a need to correct this situation, perhaps by revising the

strategic plan.

Page 21 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section I 

Annual Performance Goals and Measures

However, not all annual goals have to link directly to the long-term

strategic goals. Because agencies are required to have one or more

performance goals that cover each program activity in the agency’s

budget, and because program activity structures may not be

mission-based, some performance goals may cover program activities that

have no direct link to the strategic goals or mission. However, in general,

at least one performance goal should be related to each strategic goal or,

alternatively, some explanation of missing relationships be given. An

amendment to the strategic plan, documented in the annual plan, can be

an acceptable way to establish such a relationship. Minor changes to the

strategic plan can be made directly in the annual performance plan. In

addition, the annual plan should include performance goals for efforts to

address major management-related problems that are mission-critical or

would prevent achievement of the agency’s strategic goals, whether these

problems are identified in the strategic plan or otherwise based on

evaluators’ knowledge. In general, the annual performance plan could be

an appropriate place for an agency to address shortcomings in its strategic

plan that GAO and others identified in reviews of that plan.

Criterion 2.1 The annual performance goals should reflect the strategic goals and

mission (OMB Circular A-11, sec. 220.5).

Key features to look for include

• the mission statement and long-term strategic goals from the agency’s

current strategic plan, or a summary of them; any indication of whether

they have been revised; and, if so, an explanation for the revision;

• a diagram, table, model, or description that clearly aligns the agency’s

annual performance goals and the current strategic goals and mission;

• descriptions of how annual performance goals contribute to achieving or

supporting strategic goals, unless obvious;

• explanations for annual goals that do not relate to one or more of the

strategic goals;

• explanations for strategic goals that do not have related annual goals; and

• amendments to the strategic plan, or explanations that the strategic plan

has been revised, to correct the lack of these relationships.

The evaluator should

• compare the mission statement and strategic goals (or summary of them)

in the annual plan with those in the strategic plan to determine (1) if there

Page 22 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section I 

Annual Performance Goals and Measures

are differences and (2) if a reasonable explanation is provided for any

substantive changes made;

• assess the alignment of the annual performance goals in the performance

plan with the strategic goals and mission and determine whether this

alignment is clear and reasonable; for example, one indicator of a

reasonable alignment would be if the annual performance goals can show

progress towards achieving the strategic goals;

• determine whether there are any strategic goals that are not related to at

least one performance goal and assess whether a reasonable explanation

is provided if such relationships are missing;

• assess whether the agency has responded in its annual performance plan

to previous reviews of that strategic plan; and

• determine whether performance goals are identified that cover

mission-critical management problems or issues.

Criterion 2.2 The annual performance plan should identify annual performance goals

that cover all of the program activities in the agency’s budget (Results Act,

sec. 4(b)).13

Guidance. The annual performance plan should show how specific

performance goals are related to the specific program activities contained

in the agency’s program and financing (P&F) schedules under “obligations

by program activity” in the Appendix to the Budget of the United States

Government (OMB Circular A-11, sec. 220.9). The Results Act requires each

federal agency to prepare an annual performance plan that covers these

program activities while allowing the agency to aggregate, disaggregate, or

consolidate those program activities for purposes of the plan. An agency

can have one or more budget accounts, and each budget account has its

own P&F schedule. The “obligations by program activity” section is the

first section in each P&F schedule. This section shows obligations for the

agencies’ specific program activities. The number of program activities in

agencies’ budget accounts varies from account to account, with many

accounts having no program activity breakdown other than “total

obligations” or another single activity. (See figures I.2, I.3, and I.4 for

examples of P&F schedules from the budgets for the Department of Energy,

the Environmental Protection Agency, and the General Accounting Office,

respectively.)

13The term “program activity” refers to the listings of projects and activities in the Appendix to the

Budget of the United States Government. Program activity structures are intended to provide a

meaningful representation of the operations financed by a specific budget account.

Page 23 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section I 

Annual Performance Goals and Measures

Figure I.2: Excerpt From the

Department of Energy’s 1999 Budget

Program activities and obligations that relate to the agency's specific performance goals.

Source: Budget of the United States Government, Fiscal Year 1999—Appendix (Washington,

D.C.: U.S. Government Printing Office, 1998).

Page 24 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section I 

Annual Performance Goals and Measures

Figure I.3: Excerpt From the

Environmental Protection Agency’s

1999 Budget

Program activities and obligations that relate to the agency's specific performance goals.

Source: Budget of the United States Government, Fiscal Year 1999—Appendix (Washington,

D.C.: U.S. Government Printing Office, 1998).

Page 25 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section I 

Annual Performance Goals and Measures

Figure I.4: Excerpt From the General

Accounting Office’s 1999 Budget

Program activities and obligations that relate to the agency's specific performance goals.

Source: Budget of the United States Government, Fiscal Year 1999—Appendix (Washington,

D.C.: U.S. Government Printing Office, 1998).

Program activity structures can vary from agency to agency and, within an

agency, from budget account to budget account. Program activities, like

budget accounts, may represent programmatic, process, organizational, or

other orientations. Therefore, some of the program activities that are

currently used in an agency’s budget request may be either too detailed or

too general, or they may lack a clear relationship to the programs that are

useful for measuring the agency’s performance. The Results Act allows

agencies, at their discretion, to aggregate, disaggregate, or consolidate

program activities so that they align with the performance goals in the

performance plan.

The Act assumes that agencies will use aggregation, disaggregation, and

consolidation to highlight important activities while avoiding voluminous

Page 26 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section I 

Annual Performance Goals and Measures

presentations that would overwhelm a reader and detract from the plan’s

informative value and usefulness. Aggregating program activities consists

of applying the same performance goals and measures to several program

activities within a budget account. Disaggregating consists of dividing

individual program activities in a P&F schedule into component parts and

applying performance goals and measures specifically to component parts.

Consolidating program activities consists of applying a single set of

performance goals and measures to program activities from two or more

budget accounts. (See figure I.5.) According to OMB, it is through

aggregation or consolidation that an annual plan can cover every program

activity in the P&F schedules while allowing an agency to omit goals for its

nonmajor activities. The significance of any program activity constituting a

major function or operation for an agency is less likely to be overlooked or

minimized if the agency aggregates or consolidates program activities.

Page 27 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section I 

Annual Performance Goals and Measures

Figure I.5: Aligning Annual Performance Plans With the Budget Account and Program Activity Structure

Because many P&F schedules are not mission- or goal-oriented, it may be

difficult for agencies to clearly link program activities to performance

goals. As a by-product of Results Act implementation efforts, an agency

may develop and propose changes to the program activity structure in its

budget request, but any proposed changes must be coordinated with OMB

and the cognizant appropriations subcommittees. Although the agency’s

budget may show such proposed changes, and changes may occur in the

future to better align with performance measurement, at this time, it is the

evaluator’s responsibility to ensure that the major functions embedded in

the agency’s program activity sections are covered by the performance

goals and measures.

Page 28 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section I 

Annual Performance Goals and Measures

For key programs that do not appear in the program activity structure of

the president’s budget (e.g., certain programs funded by Medicare and

Medicaid, such as the Rural Health Clinic program and the Medicare

Incentive Payment program), evaluators should determine whether these

programs constitute major functions or operations and have goals that are

unique and distinguishable from their encompassing program activity. If

so, the encompassing program activity should be disaggregated to show

the results these programs are expected to achieve. In addition, the

performance plan may include performance goals for functions or

operations that are staff or support-type activities, even if such goals do

not correspond to a particular P&F schedule or program activity.

Whether the program activities used in the performance plan directly

correspond to the budget or are aggregated, disaggregated, or

consolidated specifically for performance planning, they should indicate

the link between performance goals and the budgetary resources the

agency is requesting to pursue the goals. Performance goals should be set

based on the funding proposed to achieve them.

In addition, an agency may ask OMB to authorize it to have no performance

goals in its performance plans for one or more program activities. If it was

determined to be infeasible or impractical to express performance goals in

any form for a given program activity, the performance plan should state

why.

Key features to look for include

• a diagram, model, or description that clearly shows how each of the

program activities in each budget account in the agency’s budget is

covered by the annual performance plan; and

• if the annual plan has aggregated, disaggregated, or consolidated the

program activities from the agency’s budget, a crosswalk that shows how

the program activities in the agency’s budget relate to the aggregated,

disaggregated, or consolidated program activities.

The evaluator should

• compare the presentation of the agency’s program activities in its

performance plan to the presentation in the P&F schedules to assess the

annual plan’s coverage of the budget’s program activities;

Page 29 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section I 

Annual Performance Goals and Measures

• based on an evaluator’s knowledge of the agency’s programs, assess the

reasonableness of any aggregation, disaggregation, or consolidation of

program activities for purposes of the annual plan; and

• based on knowledge of the agency’s programs, assess whether any major

function or operation of the agency covered by the program activities has

not been adequately addressed by the annual performance plan. (If no

performance goals have been established for a program activity, determine

what explanation the agency has provided as to why it is infeasible or

impractical to have any performance goals for the program activity).

Issue 3: Recognizing

Crosscutting Efforts

How are agencies coordinating efforts with related strategic or

performance goals?

Criterion 3.1 If applicable, the annual performance plan should identify performance

goals that reflect activities being undertaken to support programs of a

crosscutting nature (OMB Circular A-11, sec. 220.8).

Guidance. In many areas of the federal government, programs share

common purposes, although they may operate differently or address

different aspects of a broader national concern. In their strategic plans,

agencies were encouraged to identify such programs and crosscutting

issues. The annual performance plan provides an opportunity, when

applicable, for an agency to clarify the relationship between programs that

have related strategic or performance goals, whether they are in the same

or different federal agencies. The focus of an agency’s performance plan

should be on the agency’s performance goals and how it intends to achieve

them. However, these performance goals should reflect the crosscutting

nature of programs when applicable. When an agency’s program addresses

a crosscutting issue of broad national concern, the agency should discuss

in its performance plan how achieving the program’s goals will contribute

to addressing that crosscutting issue.

Criterion 3.2 If applicable, the annual performance plan should evidence coordination

among crosscutting programs (congressional leadership letter to the OMB

Director, dated December 17, 1997).

Guidance. When the need for better coordination or integration of efforts

that crosscut programs or agencies has been noted in the agency’s

strategic plan or identified in hearings or in GAO or Congressional Research

Service reports, the annual performance plan should address this need.

Page 30 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section I 

Annual Performance Goals and Measures

The agency should discuss efforts to coordinate its programs with other

internal and external federal programs performing related activities and

provide evidence of such coordination (e.g., evidence of joint planning and

coordination, relationship of programs’ goals and measures, discussion of

the agency’s contribution to the crosscutting issue).

The evaluator should

• determine whether the agency’s performance plan identifies performance

goals for crosscutting issues raised in the agency’s strategic plan;

• determine whether the agency’s performance plan identifies performance

goals for other crosscutting issues related to the agency’s programs and

identified in hearings or in GAO or Congressional Research Service reports;

and

• determine whether the plan discusses how it has coordinated or will

coordinate crosscutting efforts with other related federal programs.

Page 31 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section II 

Strategies and Resources

Core Question 2 How well does the performance plan discuss the strategies and

resources the agency will use to achieve its performance goals?

Figure II.1: Summary of the Issues and

Criteria for Core Question 2

Guidance. The Results Act prescribes that the performance plan should

discuss the operational processes, skills, technology, and resources an

agency will employ to achieve its performance goals. GAO and OMB use the

term “strategies” when discussing the operational processes, skills, and

technology component of the Act’s requirements.

The performance plan should communicate how an agency expects to

achieve its performance goals and the level of resources that will be

required. The plan should describe both its ongoing or existing strategies

and resources and any significant changes to its strategies and/or

resources needed to accomplish the agency’s performance goals. A

Page 32 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section II 

Strategies and Resources

performance plan, however, should not contain a detailed list of an

agency’s strategies and resources.

An agency’s performance plan may include some performance measures

called “means-type measures” that relate to strategies and resources, such

as redistributing workload or replacing computer systems. However,

according to OMB, these measures do not substitute for a description of the

agency’s strategies and resources needed to achieve its performance goals.

Issue 4: Connecting

Strategies to Results

How clear and reasonable are the agency’s strategies with respect to its

intended performance goals?

Criterion 4.1 The performance plan should briefly describe the agency’s strategies to

accomplish its performance goals (Results Act, sec. 4(b); OMB Circular

A-11, sec. 220.12; and OMB checklist, pp. 6 and 8).

Guidance. An agency’s strategies to help achieve its performance goals

include activities such as grant making, rulemaking, inspections, and

research as well as administrative processes such as procurement, real

property management, and human resources management. Additionally,

strategies can include the application of technology, such as introducing

electronic funds transfers to streamline payment processing or ensuring

that all critical information technology systems can function past the year

1999.

The performance plan should briefly describe both existing and new

strategies to achieve a performance goal. The discussion of new strategies

may involve an agency’s efforts to improve efficiency and effectiveness

through such approaches as streamlining, contracting out, privatization,

partnering, business process reengineering, and franchising. An agency’s

performance plan should also discuss any regulation, tax expenditure,

and/or proposed legislation that affects the achievement of a specific

performance goal. Additionally, the strategies discussed in the

performance plan should be consistent with the strategies section of an

agency’s strategic plan.

Experience has shown that, while this level of specificity is not required by

the Results Act, the better performance plans not only describe the

strategies needed, but also provide a rationale as to how the strategies will

contribute to accomplishing the expected level of performance. For

Page 33 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section II 

Strategies and Resources

example, an agency’s simply stating that it will administer block grants to

states, without indicating what the states will do with the money, would

fail to link strategies to intended results.

Typically, the descriptions of strategies included in performance plans

should be brief. However, an agency is expected to provide more details

on its strategies when it is planning a significant change to its existing

operational processes, skills, or technology. Also, the level of detail will

vary depending on the requirements and needs of particular congressional

committees.

Key features to look for include

• linkages between an agency’s strategies and its performance goals;

• distinctions between existing and new strategies;

• a rationale as to how the strategies will contribute to improving

performance;

• if applicable, a description of the agency’s streamlining, contracting out,

privatization, business process reengineering, franchising, and other

approaches for achieving greater efficiency and effectiveness in agency

operations;

• if applicable, a discussion of the agency’s intended use of regulations or

reliance on tax expenditures to achieve performance goals;

• if applicable, a description of any proposed legislation needed to help

achieve performance goals; and

• references in the performance plan to other plans and documents (e.g.,

business plan, reorganization plans, training plans) that show how the

identified strategies will contribute to achieving the agency’s performance

goals.

The evaluator should

• based on an evaluator’s knowledge of an agency, assess the

reasonableness of the agency’s rationale as to how the strategies will

contribute to achieving the performance goals;

• identify changes to existing strategies or new initiatives and assess

whether the agency provided a reasonable explanation of why the changes

are necessary;

• review previous work by GAO, the agency’s inspector general, and others to

identify unresolved recommendations for improving the agency’s

processes, skills, and technology; and

Page 34 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section II 

Strategies and Resources

• identify whether the agency’s strategies sufficiently incorporate corrective

actions recommended by GAO and/or the agency’s inspector general.

Criterion 4.2 If applicable, an agency’s proposed waiver of administrative procedural

requirements and controls should be described in a manner that quantifies

expected performance improvements (Results Act, sec. 5(a)).

Guidance. Agencies are authorized to apply, in their annual performance

plans, to OMB for waivers to administrative procedural requirements and

controls to provide agencies with managerial flexibility. The waiver

request must describe and quantify any anticipated effects on an agency’s

performance. It also must be endorsed by the agency that imposed the

requirement or control, which usually involves one of the four central

management agencies—OMB, the General Services Administration, the

Office of Personnel Management, and the Financial Management Service

in the Department of the Treasury. Such waivers are intended to provide

federal managers with more flexibility to structure agency systems to

better support performance goals. They may involve administrative

requirements such as staffing and the internal allocation and use of

resources. An example of increased flexibility would be to allow an agency

to recapture operating funds that remain unspent because of increased

efficiencies and then use the funds to purchase new equipment or expand

employee training. The waivers, however, do not involve statutory

requirements and controls.

Key features to look for include

• a discussion by the agency of the waiver requested;

• a description of the managerial flexibility that would be achieved if the

waiver is approved;

• a discussion of the anticipated effects on performance that will result from

greater managerial flexibility;

• a description of the likelihood of the agency’s achieving a specific

performance goal if the waiver is not granted; and

• an endorsement of the waiver from the agency that imposed the

administrative requirement or control.

The evaluator should

• assess whether the agency provided reasonable descriptions of the

managerial flexibility proposed and the impact on intended performance if

the waiver is not granted;

Page 35 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section II 

Strategies and Resources

• assess whether the agency’s quantification of the anticipated effects on

performance appears reasonable; and

• verify that the endorsement is included in the description.

Criterion 4.3 The plan should discuss an agency’s actions to address external factors

that are likely to affect its performance.

Guidance. Whether an agency achieves its performance goals can be

influenced by a number of external factors, such as emerging economic,

social, and technological trends and the role of third parties (e.g., state and

local governments). Thus, the Results Act requires agencies to identify, in

their strategic plans, external factors that could significantly affect

performance. GAO believes that agencies’ performance plans, although not

required to by the Results Act, should also discuss external factors in

order to provide additional context for anticipated performance. OMB

prescribes that, if external factors are identified in an agency’s

performance plan, they should be consistent with the external factors

identified in the agency’s strategic plan.

Key features to look for include

• consistency between the external factors affecting performance described

in the performance plan and the external factors identified in the agency’s

strategic plan or previous reviews of that plan;

• an explanation of how the agency’s performance will be positively or

negatively affected by such external factors; and

• a discussion of partnerships or working relationships with other

organizations aimed at achieving performance goals.

The evaluator should

• determine whether the agency’s discussion of external factors is

consistent with its strategic plan and addresses any issues raised in

reviews of that plan; and

• assess whether the agency provided a reasonable discussion of how it

would mitigate or use the identified conditions to achieve its performance

goals.

Issue 5: Connecting

Resources to Strategies

What capital, human, financial, and other resources are being applied to

achieve the agency’s performance goals?

Page 36 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section II 

Strategies and Resources

Criterion 5.1 The performance plan should briefly describe the capital, human,

financial, and other resources being applied to achieve the performance

goals (Results Act, sec. 4(b); OMB Circular A-11, sec. 220.12; and OMB

checklist, p. 6).

Guidance. OMB Circular A-11 requires that the description be brief and

limited to resources (dollars and personnel) reflected in an agency’s

budget request. The level of detail, however, will vary depending on the

requirements and needs of congressional committees. Also, an agency may

be expected to provide more details on the resources it needs when the

agency is requesting a significant change. The performance plan should

also include prior year resources available for the agency’s current fiscal

year performance goals, as well as current year resources available for

performance goals to be achieved in a future year.

Experience has shown that, while such specificity is not required by the

Results Act, the better performance plans not only describe the resources

needed, but also provide a rationale for how the resources will contribute

to accomplishing the expected level of performance.

Identifying the specific resources associated with an individual

performance goal will pose a challenge for some agencies, as well as for

evaluators evaluating those agencies’ performance plans. For example,

funding from one budget activity may provide support for more than one

performance goal. Agencies do not need to account for every dollar

allocated to a specific performance goal.

To achieve a specific performance goal, agencies may need to invest and

acquire new capital assets. When this is the case, OMB Circular A-11

requires that performance plans make reference to the agency’s capital

assets plan and briefly describe new capital acquisitions that will

significantly affect the achievement of performance goals. Capital assets

are defined by OMB as land, structures, equipment (including information

technology), and intellectual property, including software, that are used by

the federal government and have an estimated useful life of 2 years or

more. Some agencies (such as the Department of Defense and the National

Aeronautics and Space Administration) rely more heavily on capital asset

investments than others to achieve their performance goals. In addition to

capital asset plans, performance plans should refer to the agency’s

information technology and financial management plans, where such

resources are identified as needed to achieve the performance goals.

Page 37 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section II 

Strategies and Resources

Key features to look for include

• an allocation of resources (dollars and personnel) to performance goals;

• a discussion of the effect increases or decreases in resources (dollars

and/or personnel) may have on expected performance;

• references in the annual performance plan to the agency’s capital assets,

information technology, and financial management plans;

• a description of proposed major capital investments that will bear

significantly on the achievement of performance goals;

• linkages between specific capital investments and individual performance

goals;

• consistency between the capital assets’ performance goal(s) and related

annual performance goals; and

• consistency between the resource needs discussed in the performance

plan and an agency’s budget justification documents.

The evaluator should

• assess the reasonableness of the performance plan’s discussion of the

proposed resources for achieving the performance goals and how the

resources were allocated to each goal;

• assess whether the performance plan provides a clear rationale for how

the resources will contribute to improving performance;

• verify that the performance plan references the proposals in the agency’s

capital assets and information technology plans when appropriate;

• determine whether the capital assets in the performance plan sufficiently

reflect the goals and costs of those assets as presented in the capital assets

plan; and

• assess whether the agency’s performance plan or capital assets plan

provides a plausible explanation of how a specific capital asset investment

will contribute to achieving a performance goal.

Page 38 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section III 

Verification and Validation

Core Question 3 To what extent does the agency’s performance plan provide

confidence that its performance information will be credible?

Figure III.1: Summary of the Issues and

Criteria for Core Question 3

Guidance. To be able to assess progress toward the achievement of

performance goals, the measures used must be valid and reliable.

Reliability refers to the precision with which performance is measured,

while validity is the extent to which the measure adequately represents

actual performance. The reliability and validity of the measures are

influenced by a number of factors, including the quality of the data on

which the measures are based. Performance measures may be derived

from data collected and maintained by an agency or obtained from

external sources. In order for measures to be valid and reliable, the data

on which they are based must be free from significant error, especially

bias. This section discusses the influence of data quality. Other

Page 39 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section III 

Verification and Validation

considerations in the choice of reliable and valid measures are discussed

under question 1.

To assess whether the goals and measures proposed in a performance plan

will provide a reasonable and appropriate basis for assessing performance,

the reader must be able to judge whether the proposed performance data

will be sufficiently free of error, in particular bias. To this end, the Results

Act calls for the plan to describe the procedures that will be used to verify

and validate the measured values of actual performance. In this guide,

verification means the checking or testing of performance data to reduce

the risk of using data that contain significant errors. Validation means the

testing of data to ensure that no error creates significant bias. Significant

error, including bias, would affect conclusions about the extent to which

performance goals have been achieved.

The reasonableness and appropriateness of proposed measures are also

influenced by the extent to which the needed data can be obtained at a

reasonable cost—i.e. the extent to which benefits obtained from

providing the data outweigh the costs of producing it.

The role of the evaluator is to

• assess whether the agency has provided sufficient information to permit

an informed judgment by the reader of whether the performance data will

be sufficiently free of bias and other significant error, and

• determine whether the verification and validation procedures and the data

proposed by the agency are credible.

The credibility of the verification and validation procedures and of the

data is to be determined by assessing whether there are any problems with

them that would influence conclusions about the achievement of results.

Issue 6: Verifying and

Validating Performance

How will the agency ensure that its performance information is sufficiently

complete, accurate, and consistent?

Criterion 6 The plan should describe credible procedures to verify and validate

performance information (Results Act, sec. 4(b); OMB A-11, sec. 220.13).

Guidance. The Results Act requires that performance plans include

descriptions of procedures for verifying and validating the measured

Page 40 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section III 

Verification and Validation

values of actual performance (Results Act, sec. 4(b)). OMB encourages

agencies to discuss the verification and validation procedures with their

OMB representatives (OMB A-11, sec. 220.13). Agencies should have in place

or propose credible procedures for ensuring that their performance data

are reasonably complete, accurate, and consistent.

The data required for performance measurement can be collected on an

ongoing basis by agency personnel during the performance of duties or by

means of periodic studies. For example, client data can be recorded on an

ongoing basis during the determination of eligibility for services, or data

can be collected in the processing of financial transactions. Client surveys

are an example of periodic studies, which may be required for some

measures of performance—for example, for measures of service quality.

Error can occur at various points in the collection, maintenance,

processing, and reporting of data. At data collection, information can be

recorded incorrectly. For example, program personnel can incorrectly

record information about clients. Inadequate procedures for maintaining

data, such as on computer information systems, can result in the loss or

alteration of data. For instance, data from client survey forms can be

incorrectly entered into computer information systems. Processing and

reporting of data can also add error through, for example, the incorrect

transfer of data from computerized databases into analyzable or

reportable forms.

Error that is caused by a systematic source can introduce bias and lead to

inaccurate estimates of program performance. For example, in programs

intended to reduce such inappropriate behaviors as alcohol use, speeding,

smoking, etc., clients entering these programs may be asked to estimate

the extent to which they engage in these behaviors, as a baseline against

which to judge program effectiveness. Clients may initially underestimate

these undesirable behaviors, ultimately leading to underestimates of

program success.

As another example, a program designed to support the development of

small businesses by providing financial assistance may do follow-up of

recipients to assess their success. Some recipients may become

unreachable for a variety of reasons, including name changes, relocation,

or failure. If many of those small enterprises that go out of business are

unreachable, they would not show up in the database, and the data would

then contain a greater proportion of successful businesses than is the

Page 41 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section III 

Verification and Validation

actual situation. Thus, the ability of the program to stimulate small

enterprises could be overestimated.

The agency should have in place procedures for ensuring that the data are

free of significant levels of error and that bias is not introduced. These

procedures can include internal controls over such matters as data

collection, maintenance, and entry. Particularly important are external

assessments such as audits, evaluations, and peer reviews. One such

external assessment, of particular importance when goals and measures

are financially oriented, is the audit of agency annual financial statements,

required under the Chief Financial Officers (CFO) and Government

Management Reform acts. Whichever verification and validation

procedures are used, their description needs to be sufficiently detailed to

permit an assessment of their adequacy.

Key features to look for include

• a clear description of the data verification and validation procedures to be

used to ensure that significant error, including bias, are not added during

collection, maintenance, or processing of data;

• general procedures to control data quality, such as general controls for

computerized information systems; and

• procedures specific to the data required for performance measures

proposed in the performance plan, including data gathered both on an

ongoing and a periodic basis.

The described procedures are credible if they

• provide for periodic review of data collection, maintenance, and

processing procedures by the agency to ensure that they are consistently

applied and continue to be adequate;

• provide for periodic sampling and review of data to ensure their

completeness, accuracy, and consistency;

• rely on independent audits or other established procedures for verifying

and validating financial information when performance measures require

the use of financial information; and

• address problems, in verification and validation procedures, known to GAO

or the agency.

Credibility is enhanced if the procedures call for formal evaluation or audit

of the information by parties external to the program in question.

Page 42 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section III 

Verification and Validation

The evaluator should

• review the performance plans to identify references to verification and

validation procedures;

• review previous GAO work, or other available sources, for instances where

procedures for verifying and validating agency performance information

have been found deficient;

• where performance measurement requires financial information, review

the annual audits of the agency’s annual financial reports to identify

instances where controls over related financial information were found to

be deficient;

• interview agency staff to determine whether they are taking or are

proposing to take actions to address previously identified deficiencies in

verification and validation procedures;

• interview agency staff to determine the extent to which OMB has been

consulted on proposed verification and validation procedures prior to

submission of the plan, and inquire as to concerns raised by OMB and

whether and how the agency addressed them; and

• assess whether GAO and OMB concerns were addressed in the validation

and verification procedures described in the plan and whether the

procedures are credible.

Issue 7: Recognizing Data

Limitations

To what extent does the performance plan identify significant data

limitations and their implications for assessing the achievement of

performance goals?

Guidance. The specific performance data required—and the means for

collecting, maintaining, and analyzing them—should be identified and

described in detail sufficient to allow an assessment of the extent to which

they can be relied upon (OMB checklist, p. 7).

Evaluators should rely on previously conducted work and on departmental

sources to determine whether there are any known problems with the data

sources or the data that would cast doubt on the credibility of the

information. Known limitations are those identified by GAO in its work or

in available documents such as program evaluations, independent audits,

inspector general investigations, information systems analyses, etc. For

example, external audits of agency annual financial reports, required

under the CFO and Government Management Reform acts, have identified

problems with data accuracy, completeness, and consistency.

Page 43 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section III 

Verification and Validation

No data collection and maintenance process is free of error. Significant

limitations are errors that would lead to inaccurate assessment of goal

achievement. Proposals for performance data recognize known

limitations to the extent that

• they do not propose to use data with known problems that would lead to

inaccurate assessment of the extent of achievement of performance goals,

or

• they describe means taken or proposed to address these weaknesses.

Potentially significant data limitations include

• inconsistencies in data collection from location to location, from one time

period to another, or from one data source to another, when data from

more than one source must be combined to create a measure

(Inconsistencies can arise when standardized procedures are not used or

followed. For example, this could occur when data are not clearly defined.

It could also occur when definitions change, resulting in noncomparable

data—for instance, if the number of people finding permanent

employment is used to measure an outcome and the definition of

“permanent employment” changes in a way that affects the number of

people fitting the category.);

• inaccuracies due to imprecise measurement and recording (for example,

program personnel incorrectly recording client information); and

• incomplete data (for example, not all cases can be identified and recorded,

data are backlogged and not up to date, data are lost when an information

system crashes).

The significance of these limitations increases if the error constitutes bias.

For example, incomplete data can result in bias if certain types of program

clients are less likely to be recorded in a database because they are harder

to reach. Certain types of data collection are more prone to bias. For

example, management or grant recipient subjective ratings of outcomes

may lead to inaccuracies that reflect their interests.

Resolving limitations to the data needed for performance measurement

can pose serious challenges to agencies and take time. Some agencies may

not have available all the needed data because of the challenges involved.

GAO’s previous review of the Department of Health and Human Service’s

April 1997 draft strategic plan observed that some federal data on the

Medicaid program are of questionable accuracy.14 Some of the problems

14HHS’ Draft Strategic Plan (GAO/HEHS-97-173R, July 11, 1997).

Page 44 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section III 

Verification and Validation

stem from data collected from different states, all of which do not use

identical definitions for data categories. It is not expected that agencies

will have resolved all significant limitations, but that they are at least

taking reasonable steps to address them. In some instances, however,

previous work may have revealed that the agency was not making

reasonable progress in resolving the challenges.

Criterion 7.1 Known significant limitations to data from agency sources should be

recognized in the performance plan.

Key features to look for include

• a description of limitations to the data, the significance of the limitations

for judging the extent to which goals have been achieved, and the steps

being taken or proposed to address the problems (OMB checklist, p. 7;

congressional leadership letter to the OMB Director, dated Dec. 17, 1997).

The evaluator should

• review previous work by GAO, or other sources, to determine whether

previously identified potentially significant limitations to the proposed

performance information have not been discussed in the performance

plan;

• where performance measurement requires financial information, review

the annual audits of the agency’s annual financial reports to identify

instances where significant limitations have been found in related financial

information;

• assess the means used to collect the data to determine whether the

performance plan has not discussed weaknesses inherent in the form of

data collection itself that could result in significant error (examples of

such forms include management or client ratings and estimates of

outcomes);

• assess whether actions described in the plan as being taken or planned to

address the known data limitations appear reasonable; and

• interview agency officials to determine whether reasonable action, other

than that described in the performance plans, is being taken or proposed

to address potentially significant limitations or, alternatively, whether

there is reason to believe that the limitations do not apply to the particular

use intended for the data.

Page 45 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section III 

Verification and Validation

Criterion 7.2 The performance plan should indicate when performance data will come

from sources external to the agency (OMB A-11, sec. 220.13) and should

recognize known significant limitations to external data.

OMB requires that agencies identify in their performance plans data

received from external sources. OMB specifies that agencies are not

required to develop an independent capacity for verifying or validating

performance data from external sources. However, agencies should collect

available information from outside sources on the consistency, accuracy,

and completeness of the data, although they are not required to include

such information in the plan. It is expected that proposals to rely on

external sources should not include sources with known significant

limitations, unless actions are being taken or proposed to address these

limitations. Better performance plans will also identify limitations to the

data, discuss their significance, and describe steps being taken or

proposed to address the problems (OMB checklist, p. 7; congressional

leadership letter to the OMB Director, dated Dec. 17, 1997).

The evaluator should

• ascertain, based on previous reviews, whether external sources (for

example, information produced by other agencies, state governments, or

independent organizations) are used but not identified;

• determine whether the external sources to be relied on are recognized and

generally accepted as reliable sources of this data (e.g., statistical offices,

research organizations, etc.);

• interview agency officials to ascertain whether an effort was made to

collect available information regarding the completeness, accuracy, and

consistency of the information as well as on whatever limitations may

have been identified;

• examine prior work to identify whether there are any significant problems

with the completeness, accuracy, and consistency of the proposed

external data; and

• assess whether known data limitations have been adequately taken into

account in the performance plan.

Criterion 7.3 The performance plan should discuss or reference any significant new

information systems or major changes to existing systems that have been

proposed to make more credible data available for performance measures.

When existing data contain significant limitations, providing credible data

may require new information systems or major changes to existing

Page 46 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Section III 

Verification and Validation

systems. In deciding whether to continue to rely on existing data or

embark on major development or modification efforts, the agency will

have to balance the incremental costs of different data collection

approaches against the need to ensure that the collected data are

complete, consistent, and accurate enough to document performance and

support decisionmaking.

Excessively costly performance data may reduce the feasibility of

performance measurement, needlessly direct resources away from

program or service delivery, or place an undue burden on program clients

or third parties. Conversely, costs may be offset by the potential benefits

of more efficient and effective programs and greater accountability. They

may also be offset when performance measurement is implemented as

part of managing for results, through the elimination or reduction of

outdated management practices. Agencies can also reduce costs by

building performance data collection into the processes that govern daily

operations, rather than creating entirely new and separate data systems.

Potential costs include the management and staff time required to collect,

maintain, analyze, and use performance data; the costs of any contracts for

data collection and analysis; related equipment costs; the burden imposed

on those required to report the data; political and bureaucratic costs; and

other important negative consequences of performance measurement.

Agencies are not required to report on the costs of collecting, maintaining,

and reporting performance information. However, better performance

plans discuss the costs and benefits of proposed information collection

approaches, and of alternatives when the proposed performance measures

are costly to implement. Although evaluators are not expected to assess

whether costs are reasonable, recognition should be given to agencies that

report the costs of collecting performance data, the costs of alternative

data sources, or other information that allows the reader to assess

whether the benefits of collecting better data outweigh the costs.

The evaluator should

• examine the agency’s capital plan to assess whether new information

systems, or significant modifications to existing systems, are proposed to

implement performance measures and whether these proposals are

appropriately referenced in the performance plan.

(966706) Page 47 GAO/GGD-10.1.20 Guide to Assessing Agency Annual Performance Plans
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the

following address, accompanied by a check or money order

made out to the Superintendent of Documents, when

necessary. VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address

are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 37050

Washington, DC 20013

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000 

or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and

testimony. To receive facsimile copies of the daily list or any

list from the past 30 days, please call (202) 512-6000 using a

touchtone phone. A recorded menu will provide information on

how to obtain these lists.

For information on how to access GAO reports on the INTERNET,

send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov

PRINTED ON RECYCLED PAPER
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. G100