a

GAO

United States Government Accountability Office

Report to the Committee on Armed 

Services, U.S. Senate

September 2004 DEFENSE 

MANAGEMENT

Tools for Measuring 

and Managing Defense 

Agency Performance 

Could Be 

Strengthened

GAO-04-919
What GAO Found

United States Government Accountability Office

Why GAO Did This Study

Highlights

Accountability Integrity Reliability

www.gao.gov/cgi-bin/getrpt?GAO-04-919.

To view the full product, click on the link 

above. For more information, contact Sharon 

Pickup at (202) 512-4300 or 

pickups@gao.gov or Patricia Dalton at 

(202) 512-6806 or daltonp@gao.gov. 

Highlights of GAO-04-919, a report to the 

Committee on Armed Services, U.S. 

Senate 

September 2004

DEFENSE MANAGEMENT 

Tools for Measuring and Managing 

Defense Agency Performance Could Be 

Strengthened 

Since fiscal year 1998, the Department of Defense (DOD) has implemented 

various tools to help manage and oversee the performance of defense 

agencies. Between fiscal year 1999 and 2003, DLA, DISA, and DODEA 

initially used “performance contracts”—internal management agreements— 

to bring specific problems to the attention of senior DOD and agency 

leadership. While the contracts produced some useful information for 

decision makers, this tool would have been more effective for assessing 

performance, making resource allocation decisions, and taking corrective 

actions if DOD had required the agencies to include certain attributes 

associated with results-oriented management. Such attributes include 

aligning agency performance goals and measures with agency strategic plans 

and departmentwide goals; identifying individuals accountable for achieving 

results; providing a comprehensive view of organizational performance; 

linking resource needs to performance; discussing data quality; and 

providing contextual information, including external factors that affect 

reported performance. 

Beginning in fiscal year 2003, DOD renamed the performance contracts as 

“performance plans” and gave the defense agencies the option to use a 

“balanced scorecard” approach, a tool used in the public and private sectors 

to assess organizational performance. Based on experiences using the initial 

contracts, DOD took steps to strengthen performance plans and scorecards 

by revising the oversight and review process, requiring performance 

measures to align with agency and departmentwide goals, and requiring 

measures to provide a more comprehensive view of agency performance. 

DLA’s scorecard, DODEA’s performance plan, and DISA’s plans for the 

agency’s scorecard incorporated these changes and other attributes to 

varying degrees. While these tools have the potential to provide information 

useful to decision makers, they would be strengthened if DOD had required 

the agencies to include additional attributes such as designating specific 

individuals responsible for achieving results; identifying the relationship 

between resource needs and performance; reporting on data quality; and 

providing contextual information to allow top leaders to understand the 

extent of progress made, take corrective actions to achieve goals, and 

establish realistic performance goals for future years. With these attributes, 

decision makers would potentially gain additional insights into agency 

performance and areas needing greater management attention. 

DOD has developed mechanisms, such as a performance management Web 

site and roundtables, to help agencies share lessons learned from 

implementing performance plans and scorecards. In response to GAO’s 

suggestions during this review, DOD recognized the need to continue to hold 

roundtables more frequently. DLA and DISA have also proactively shared 

their experiences with each other. 

GAO was mandated to assess the 

effectiveness of defense agency 

performance contracts as 

management tools. As agreed, 

GAO also reviewed other tools 

(performance plans and balanced 

scorecards) and focused on three 

defense agencies—the Defense 

Logistics Agency (DLA), the 

Defense Information Systems 

Agency (DISA), and the 

Department of Defense Education 

Activity (DODEA). GAO addressed 

(1) the extent that the defense 

agencies initially used performance 

contracts, including whether this 

tool addressed attributes 

associated with results-oriented 

management; (2) defense agencies’ 

efforts to implement performance 

plans using lessons learned from 

the initial contracts; and (3) the 

extent DOD established 

mechanisms to share lessons 

learned. GAO reviewed the content 

of these tools, but not the actual or 

reported performance. DISA has 

not yet finalized its scorecard, thus 

this report discusses only DISA’s 

plans for its scorecard. 

GAO is making recommendations 

to DOD aimed at improving 

guidance to make performance 

plans and scorecards more 

informative and useful and further 

strengthen the potential of these 

tools for measuring and managing 

agency performance. 

In its comments, DOD generally 

concurred with GAO’s 

recommendations. 
Page i GAO-04-919 Defense Management 

Letter 1

Results in Brief 4 

Background 7 

While Used to Varying Degrees, Performance Contracts That We 

Reviewed Could Have Been More Effective Had They Included 

Certain Attributes Associated with Results-Oriented 

Management 10 

The Plan and Scorecard We Reviewed Show Improvement, but 

Could Be Strengthened by Including Certain Key Attributes 19 

DOD Captures and Shares Lessons Learned 35 

Conclusions 35 

Recommendations for Executive Action 36 

Agency Comments and Our Evaluation 37 

Appendix I Scope and Methodology 39 

Appendix II Department of Defense’s Risk Management 

Framework 41 

Appendix III Fiscal Year When Defense Agencies Implemented 

Their First Performance Contract, Plan, or Scorecard 42 

Appendix IV Certain Key Attributes Associated with 

Results-Oriented Management 45 

Appendix V Comments from the Department of Defense 48 

Appendix VI Key Contact and Staff Acknowledgments 51 

Related GAO Products 52 

Contents 
Page ii GAO-04-919 Defense Management 

Tables 

Table 1: Time Frames for DLA’s, DISA’s, and DODEA’s Use of 

Performance Contracts, Performance Plans, or Balanced 

Scorecards 3 

Table 2: Extent To Which DLA’s, DISA’s, and DODEA’s 

Performance Contracts Incorporated Certain Key 

Attributes Associated with Results-Oriented Management 13 

Table 3: Extent To Which DLA’s Scorecard and DODEA’s Plan 

Incorporated Certain Key Attributes Associated with 

Results-Oriented Management 21 

Figures 

Figure 1: Depiction of How DLA’s May 2004 Scorecard Supports 

Departmentwide Goals and Cascades to Lower 

Organizational Levels 23 

Figure 2: Example of the Type of Information in DODEA’s 

Performance Plan That OSD Officials Monitor, as of May 

2004 25 

Figure 3: Detailed Data on DODEA’s Pupil to Teacher Ratio as of 

May 2004 26 

Figure 4: Format of a Sample Measure from DISA’s Draft 

September 2003 Balanced Scorecard 34 

Figure 5: DOD’s Risk Management Framework 41 

Figure 6: Defense Agencies 43 

Figure 7: Defense Field Activities 44 
Page iii GAO-04-919 Defense Management 

Abbreviations 

DISA Defense Information Systems Agency 

DLA Defense Logistics Agency 

DOD Department of Defense 

DODEA Department of Defense Education Activity 

OSD Office of the Secretary of Defense 

PA&E Program Analysis and Evaluation 

USD/P&R Under Secretary of Defense for Personnel and Readiness 

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. It may be reproduced and distributed in its entirety without further 

permission from GAO. However, because this work may contain copyrighted images or 

other material, permission from the copyright holder may be necessary if you wish to 

reproduce this material separately. 
Page 1 GAO-04-919 Defense Management 

September 13, 2004 

The Honorable John W. Warner 

Chairman 

The Honorable Carl Levin 

Ranking Minority Member 

Committee on Armed Services 

United States Senate 

Department of Defense (DOD) agencies and field activities, such as the 

Defense Logistics Agency, are a key part of DOD’s business operations and 

provide numerous support services, ranging from information technology 

and education of servicemembers’ dependents to logistics support for the 

department.1

 The agencies’ customers include the military services, other 

agencies, and military personnel and their families. During the past 

2 decades, service officials have expressed concern about inadequate 

oversight and performance of the defense agencies; specifically, the lack 

of customer responsiveness and inefficient business processes in light of 

growing defense agency budgets. 

As part of a departmentwide effort, known as the Defense Reform 

Initiative, aimed at improving DOD’s business operations, in 1998, the 

Deputy Secretary of Defense directed the defense agencies to develop 

agreements between the defense agencies and the Under and Assistant 

Secretaries to which they report. Known as “performance contracts” at the 

time, these agreements were intended to help improve the Office of the 

Secretary of Defense’s (OSD) oversight of these agencies as well as their 

performance. According to a DOD official, these performance contracts 

were not intended to be contracts in the legal sense, but rather were one 

of a number of management tools—such as strategic plans—used by the 

defense agencies to improve performance by setting performance goals, 

focusing the attention of OSD and agency leaders on customer concerns, 

and bringing management visibility over key areas of the agencies’ 

 1

 Defense agencies and DOD field activities perform similar support functions. However, 

the field activities are smaller and serve a more limited portion of DOD than do defense 

agencies. For purposes of this report, we use the term “defense agencies” to refer 

collectively to defense agencies and field activities. 

United States Government Accountability Office

Washington, DC 20548 
Page 2 GAO-04-919 Defense Management 

performance.2 In fiscal year 2003, DOD refined its approach to 

performance management and renamed the contracts as “performance 

plans” in order to more accurately reflect the intent of the agreements. 

According to DOD, these plans are intended to provide a more 

comprehensive view of agency goals and performance and align with 

agency strategic plans and departmentwide goals. DOD gave the defense 

agencies the option to use a “balanced scorecard”—a form of performance 

plan.3

 Balanced scorecards organize performance measures by 

organizational drivers—including financial, customer, and internal 

business processes as well as workforce learning and growth—to help 

measure performance, make improvements, and assess how well 

organizations are positioned to perform in the future. Although the plans 

and scorecards themselves are not intended to resolve or prevent 

problems, OSD and defense agency officials intend to use these tools to 

monitor agency performance, identify developing problems and corrective 

actions to improve business practices, meet customer needs, and set 

funding priorities. 

We have conducted an extensive body of work over the last decade 

identifying leading practices in results-oriented management, including 

performance measurement and reporting. In our prior work on 

government performance, we identified attributes associated with resultsoriented management that help inform management decision making.4

 For 

this report, we identified seven key attributes that we feel are the most 

important for accurately assessing the strengths and weaknesses of 

programs and making improvements. These attributes include aligning 

performance measures with strategic goals, assigning accountability for 

achieving results, developing measures that demonstrate results, 

developing measures that provide a comprehensive view of agency 

 2

 For purposes of this report, we use the term “performance contracts” to include both 

performance contracts as well as accompanying annual performance contract reports that 

discuss agencies’ progress toward meeting performance contract goals. 

3

 The balanced scorecard is a private-sector concept introduced by Robert Kaplan and 

David Norton in 1992 to assess organizational performance and is used by several 

government agencies. 

4

 Some of these products include GAO, Results-Oriented Government: GPRA Has 

Established a Solid Foundation for Achieving Greater Results, GAO-04-38 (Washington, 

D.C.: Mar. 10, 2004); The Results Act: An Evaluator’s Guide to Assessing Agency Annual 

Performance Plans, GAO/GGD-10.1.20 (Washington, D.C.: Apr. 1, 1998); and Executive 

Guide: Effectively Implementing the Government Performance and Results Act,

GAO/GGD-96-118 (Washington, D.C.: June 1, 1996). 
Page 3 GAO-04-919 Defense Management 

performance, linking resource needs to performance, discussing data 

quality, and including contextual information. 

The Senate Committee on Armed Services report accompanying the 

National Defense Authorization Act for Fiscal Year 2004 directed us to 

assess the effectiveness of defense agency performance contracts as 

management tools.5

 Because of the evolution from performance contracts 

to plans or balanced scorecards, this report evaluates all of these tools. Of 

the 14 defense agencies that presently use these tools, we focused our 

work on 3—the Defense Logistics Agency (DLA), the Defense Information 

Systems Agency (DISA), and the Department of Defense Education 

Activity (DODEA)—based on their levels of experience using performance 

contracts, different types of missions, and types of services and 

customers. 

Table 1 shows the time frames during which these agencies used 

performance contracts, performance plans, or balanced scorecards. 

Table 1: Time Frames for DLA’s, DISA’s, and DODEA’s Use of Performance 

Contracts, Performance Plans, or Balanced Scorecards 

Fiscal year 

Agency Performance contract Performance plan Balanced scorecard 

DLA 1999-2001 Not applicable 2003-2004

DISA 2000-2002 2003 2004

DODEA 2000-2003 2004 Not applicable

Sources: DLA, DISA, and DODEA. 

Note: Because DLA was transitioning to a balanced scorecard, with DOD’s approval, the agency did 

not develop a performance contract or plan during fiscal year 2002. DODEA officials decided to use a 

performance plan rather than exercise their option to develop a balanced scorecard for fiscal year 

2004. DISA submitted a sample draft balanced scorecard to the Office of the Secretary of Defense, 

Program Analysis and Evaluation in September 2003 in order to meet reporting requirements. 

Although we obtained DISA’s draft balanced scorecard dated 

September 2003, we did not evaluate it against certain key attributes 

associated with results-oriented management that we discuss in this report 

because the scorecard had not been finalized by the time we completed 

our review. However, we do discuss DISA’s plans for using a scorecard 

 5

 S. Rep. No. 108-46, at 347 (2003). 
Page 4 GAO-04-919 Defense Management 

and addressing the attributes, as applicable. Our objectives were to 

evaluate 

(1) the extent to which the defense agencies initially used performance 

contracts to manage and measure agency performance, including whether 

the contracts addressed certain key attributes associated with resultsoriented management, such as performance measurement and reporting; 

(2) the status of efforts by the defense agencies to implement performance 

plans, including the extent to which these plans reflect any lessons learned 

from DOD’s experience with the initial performance contracts and address 

certain key attributes associated with results-oriented management; and 

(3) the extent to which DOD has established mechanisms to capture and 

share lessons learned from the implementation of performance plans. 

To address these objectives, we collected data from and interviewed 

knowledgeable OSD, defense agency, and service officials about the use of 

performance contracts, plans, and balanced scorecards, as applicable, at 

DLA, DISA, and DODEA. We obtained these agencies’ performance 

contracts, plans, and scorecards, as applicable, and analyzed them against 

certain key attributes associated with results-oriented management that 

we identified in our prior reports. We did not review actual or reported 

agency performance, only the content of the tools and how they were 

used. We also did not validate procedures that the defense agencies have 

in place to ascertain the reliability of data used in their performance plans 

and scorecards. We conducted our review between July 2003 and 

May 2004 in accordance with generally accepted government auditing 

standards. Details about our scope and methodology appear in appendix I. 

Between fiscal year 1999 and 2003, DLA, DISA, and DODEA—the three 

agencies we reviewed—initially used performance contracts to varying 

degrees to identify performance issues, elevate management challenges 

and customer concerns to the attention of senior OSD and defense agency 

leaders, and take corrective action. The contracts were an important step 

toward more results-oriented management practices for the defense 

agencies. However, the contracts would have been more effective tools for 

informing decision making and managing and measuring agency 

performance had they included additional attributes associated with 

results-oriented management. The contracts showed progress made and 

encouraged discussion among top agency and OSD leaders about actions 

needed to achieve targeted results, particularly in terms of quality, 

Results in Brief 
Page 5 GAO-04-919 Defense Management 

quantity, cost, and timeliness of services. For example, DODEA officials 

told us that the agency’s fiscal year 2003 performance contract included a 

measure to achieve optimum pupil to teacher ratios that influenced DOD 

to fund the hiring of 200 full-time teachers. However, DOD guidance did 

not require performance contracts to include certain key attributes that we 

have identified in prior reports that are associated with results-oriented 

management. While other management documents may reflect some of 

these attributes, it is important that they also be reflected in the contracts 

to provide a comprehensive picture of goals and performance. 

Consequently, the performance contracts could have been strengthened by 

including the following attributes: 

• Aligns goals and measures with agency and departmentwide goals.

Performance goals and measures for some of the agencies we reviewed 

only partially aligned with agency and departmentwide goals and none 

of the contracts cascaded goals and measures to lower organizational 

levels. 

• Assigns accountability for achieving results. Individuals at lower 

organizational levels accountable for accomplishing specific goals and 

taking corrective actions were not identified. 

• Demonstrates results. Contracts were inconsistent in using a 

combination of clearly defined output and outcome goals accompanied 

by trend data. 

• Provides a comprehensive view of agency performance. With the 

exception of DODEA, the contracts we reviewed lacked some 

measures that could have provided officials with a comprehensive view 

of factors that drive organizational performance, such as financial, 

customer, and internal business processes. The contracts, to some 

extent, did reflect the different concerns of customers and 

stakeholders. 

• Links resource needs to performance. Contracts did not discuss the 

relationship between resource needs and performance outcomes. 

• Discusses data quality. Contracts did not discuss whether data were 

reliable, valid, or verifiable, or identify data sources. 

• Provides contextual information. Contracts lacked some contextual 

information to help officials assess the significance of factors that 

affected reported performance, such as how performance measures 
Page 6 GAO-04-919 Defense Management 

help achieve mission and departmentwide goals or the impact of 

contingency operations on achieving performance goals. 

Based on experiences using the initial performance contracts, DOD took 

steps to strengthen performance plans and scorecards by revising the 

oversight and review process as well as guidance to require that, beginning 

with fiscal year 2004, performance measures align with agency and 

departmentwide goals and provide a more comprehensive view of agency 

performance. However, these tools could have been strengthened had 

DOD required the defense agencies to include certain key attributes 

associated with results-oriented management. For example, while DLA’s 

scorecard included a combination of output and outcome measures 

accompanied by trend data, it did not include clearly defined measures. 

DODEA’s performance plan measures also were not consistently clearly 

defined and not always accompanied by trend data. The scorecard and 

plan we reviewed also lacked other attributes—not required by DOD—that 

could be useful to officials for discussing and understanding competing 

priorities and resource allocation constraints, assessing and understanding 

the extent of progress made against existing performance targets, making 

improvements, and establishing realistic performance goals for future 

years. Such attributes include 

• identification of specific individuals who are responsible for achieving 

results; 

• linkage of resource needs to performance; 

• discussion of data quality, including data sources, as well as whether 

data were reliable, valid, and verifiable; and 

• inclusion of contextual information such as external factors that could 

affect performance and relevant information about the agencies’ 

missions, business lines, and customers. 

DOD developed some mechanisms—such as a performance management 

Web site and roundtable discussions—to help the defense agencies 

capture and share experiences and lessons learned from developing 

performance plans and balanced scorecards, and some agencies are 

proactively sharing such information with each other. Based on our 

suggestions during this review, DOD recognized the need to continue to 

hold roundtables to discuss balanced scorecards. A DOD official also told 

us that DOD plans to revise and update its performance management 

Web site to include defense agencies’ performance plans and balanced 

scorecards as well as guidance. Some defense agencies, such as DLA and 

DISA, are sharing their experiences with each other to leverage lessons 

learned. 
Page 7 GAO-04-919 Defense Management 

This report contains recommendations to DOD aimed at making 

performance plans and scorecards more informative for decision making 

and further strengthening the potential of these tools for measuring and 

managing agency performance. In commenting on a draft of this report, 

DOD generally concurred with our recommendations. 

In 1997, the Secretary of Defense issued the Defense Reform Initiative 

report outlining a plan for reforming the department’s business operations. 

DOD established a management oversight structure to help sustain the 

direction and emphasis of these efforts. As part of its efforts to improve its 

business operations, DOD began requiring selected defense agencies to 

develop annual performance contracts for fiscal year 1999. These 

contracts were internal management agreements between the principal 

staff assistants—also known as Under and Assistant Secretaries—in the 

Office of the Secretary of Defense, the Deputy Secretary, and the defense 

agencies. DOD intended the performance contracts to improve oversight 

of the defense agencies by (1) identifying programming and budgeting 

issues, (2) providing agency directors with clear objectives and 

performance targets, (3) establishing open and direct communications 

with customers to, among other things, demonstrate how their actions 

affect agency performance, and (4) changing the way DOD does business. 

The contracts included improvement goals for the agencies in terms of 

cost, productivity, quality, and responsiveness to customers. DOD also 

required the defense agencies to assess their progress toward achieving 

performance contract goals in annual performance reports. A Defense 

Management Council—chaired by the Deputy Secretary and consisting of 

key civilian and military leaders—was created to oversee Defense Reform 

Initiative efforts, and a Defense Agency Task Force—chaired by the 

Deputy Director of Resource Analysis in the Office of the Secretary of 

Defense, Program Analysis and Evaluation (OSD/PA&E) and consisting of 

senior-level representatives from the Under Secretaries of Defense, the 

service secretaries, and the Joint Staff—was formed to oversee the 

development and review of performance contracts. 

DOD initially selected 4 defense agencies to implement performance 

contracts during fiscal year 1999, and gradually expanded the requirement 

to a total of 10 agencies by fiscal year 2002. At that time, DOD issued 

guidelines that established a standard format for performance contracts 

and recommended that certain attributes associated with results-oriented 

Background 

Evolution of Performance 

Contracts 
Page 8 GAO-04-919 Defense Management 

management be included. DOD initially excluded defense agencies with 

intelligence- or research-related missions from the requirement because 

officials believed it would be more appropriate to focus primarily on 

certain agencies with businesslike missions and develop lessons learned 

based on their experiences. The 4 agencies DOD initially selected were 

• the Defense Logistics Agency, a revolving fund activity; 

• the Defense Finance and Accounting Service, a businesslike agency; 

• the Defense Contract Audit Agency, a small and appropriated fund 

agency; and 

• the Defense Health Program. 

As part of the current administration’s focus on business transformation, 

DOD issued guidance in February and August 2003 that refined its 

approach to performance management and renamed the contracts as 

performance plans. These plans, first implemented in September 2003, 

build on the performance goals and measures established in the 

performance contracts and are intended to include a more comprehensive 

set of performance measures that align with agency strategic plans and 

departmentwide goals set forth in DOD’s risk management framework. 

DOD uses the risk management framework, established in the 2001 

Quadrennial Defense Review Report, to consider trade-offs among defense 

objectives and resource constraints. The framework consists of four 

dimensions of risk: 

• Force management—the ability to recruit, retain, train, and equip 

sufficient numbers of quality personnel and sustain the readiness of the 

force while accomplishing its many operational tasks; 

• Operational—the ability to achieve military objectives in a near-term 

conflict or other contingency; 

• Future challenges—the ability to invest in new capabilities and 

develop new operational concepts to dissuade or defeat mid- to 

long-term military challenges; and 

• Institutional—the ability to develop management practices and 

controls that use resources efficiently and promote the effective 

operation of the defense establishment. 

These risk areas form the basis for DOD’s annual performance goals and 

for tracking associated performance results. Appendix II depicts DOD’s 

risk management framework. 

DOD gave the defense agencies the option to use a balanced scorecard to 

develop performance goals and report on progress. Currently, according to 
Page 9 GAO-04-919 Defense Management 

an official in OSD/PA&E, DOD requires 10 of 27 defense agencies with 

businesslike missions to have either performance plans or balanced 

scorecards. Five defense agencies use performance plans, and the 

remainder have exercised the option to develop balanced scorecards. In 

addition, this official told us that 3 intelligence agencies have developed 

performance plans in order to demonstrate how they are coordinating 

with DOD on strategic and budgetary planning in the post-September 2001 

environment. Appendix III depicts the first fiscal year when each defense 

agency began using performance contracts, plans, or scorecards. 

The Director, PA&E, has responsibility for overseeing the development of 

defense agency performance plans, including reviewing the progress of 

each defense agency. Specifically, this official oversees the development 

of defense agency performance plans and balanced scorecards, provides 

guidance for interpreting and applying DOD’s risk management 

framework, and, in conjunction with the relevant Under or Assistant 

Secretary of Defense, reviews the progress of each agency in building its 

performance plan or scorecard and reporting on its overall performance. 

The Under and Assistant Secretaries of Defense are responsible for 

overseeing the defense agencies and implementing performance plans and 

balanced scorecards. 

Measuring performance allows organizations to track the progress they 

are making toward their goals and gives managers critical information on 

which to base decisions for improving their programs. Our prior work on 

results-oriented management in the federal government indicates that 

agencies that are successful in measuring performance and achieving 

business transformation goals strive to establish goals and measures at all 

levels of the agency that 

• align goals and measures with the agency’s strategic plan and 

departmentwide goals; 

• assign accountability for achieving results; 

• demonstrate results; 

• provide a comprehensive view of agency performance; 

• link resource needs to performance; 

DOD Organizations 

Responsible for 

Development of 

Performance Contracts 

and Plans 

Our Prior Work Identifies 

Attributes Associated with 

Results-Oriented 

Management 
Page 10 GAO-04-919 Defense Management 

• discuss data quality, including reliability and data sources; and 

• provide contextual information to help officials evaluate the 

significance of underlying factors that may affect reported 

performance. 

While these may not cover all attributes associated with results-oriented 

management, we feel they are some of the most important for accurately 

assessing the strengths and weaknesses of programs and making 

improvements. Without these attributes, decision makers may not have as 

complete information for measuring and managing an agency’s 

performance as needed. Each of these attributes is discussed in 

appendix IV. 

Between fiscal year 1999 and 2003, DLA, DISA, and DODEA—the three 

agencies we reviewed—used performance contracts to varying degrees to 

identify performance issues, demonstrate progress made, and encourage 

discussions to improve agency performance. The contracts were an 

important step toward more results-oriented management practices for the 

defense agencies. However, the contracts could have been more effective 

had DOD guidance required the defense agencies to include certain 

attributes associated with results-oriented management. For example, the 

contracts addressed key business lines, but did not consistently include 

information that could have helped demonstrate results or provide some 

contextual information that could help officials assess progress and 

understand the significance of underlying factors that may affect reported 

performance. 

Officials at the three defense agencies we reviewed used performance 

contracts to varying degrees as part of their strategic planning systems to 

demonstrate progress made and increase visibility over the quality, 

quantity, cost, and timeliness of products and services for their key 

business lines. In some instances, the contracts helped OSD and defense 

agency officials to identify developing problems and assess the 

effectiveness of corrective actions, such as justifying additional funding to 

achieve performance goals and implementing or revising internal 

processes to meet customer expectations. We did not verify the actual or 

reported performance data included in the performance contracts or in 

these examples. The agencies we reviewed provided the following 

examples showing how they used performance contracts. 

While Used to Varying 

Degrees, Performance 

Contracts That We 

Reviewed Could Have 

Been More Effective 

Had They Included 

Certain Attributes 

Associated with 

Results-Oriented 

Management 

Defense Agencies Used 

Performance Contracts to 

Varying Degrees 
Page 11 GAO-04-919 Defense Management 

DLA officials told us that they used their performance contracts to focus 

management attention and monitor efforts to improve the agency’s 

response time for providing logistics commodities, such as clothing and 

textile items, medical and pharmaceutical supplies, and weapon system 

spare parts. In its fiscal year 2001 performance contract, DLA established a 

performance measure for its supply centers to process a greater 

percentage of requisitions within shorter time frames. DLA reported that 

by increasing emphasis on logistics response time goals to officials in its 

Ohio-based Defense Supply Center and dedicating more trucks to its 

Defense Distribution Center, the agency had processed 96 percent of 

requisitions of commodities in 5.8 days during fiscal year 2001 rather than 

the targeted 9.3 days. 

A DISA official told us that the performance contracts helped DISA reduce 

the cost of providing services in response to customer concerns. For 

example, DISA developed cost-related performance measures for 

delivering computing services, such as “Provide mainframe information 

processing services while incurring no more than the unit costs listed.” As 

a result of increased management attention, DISA reported in its fiscal 

year 2002 annual performance report, which assessed progress against 

contract goals, that it had reduced the cost per unit for providing OS/390 

mainframe processing services by 24 percent, from $38.26 in fiscal year 

2001 to $29.04 in fiscal year 2002. DISA officials told us that the high-level 

attention made possible by the performance contracts helped ensure that 

cost reduction goals were established and pursued. 

DODEA officials told us that in order to meet OSD requirements, they 

developed and submitted performance contracts that included measures 

taken directly from their community strategic plan. However, they 

primarily used their community strategic plan—containing strategic goals, 

objectives, and performance measures—rather than the performance 

contract, which mirrored the strategic plan, to manage and measure 

systemwide school performance. We determined that DODEA’s contract 

did not show the relationship between resource needs and performance, 

but performance data helped bring attention to initiatives and programs 

needing additional funding and provided support for funding requests. For 

instance, DODEA officials noted that a measure in DODEA’s fiscal year 

2003 performance contract influenced DOD’s decision to provide 

$114 million to hire 200 full-time teachers so DODEA could achieve lower 

pupil to teacher ratios and, thereby, further improve student performance. 
Page 12 GAO-04-919 Defense Management 

Although useful in some respects, the performance contracts for the 

agencies we reviewed would have been stronger tools if they had fully 

incorporated certain key attributes that are associated with resultsoriented management and that contribute to effective government 

performance. In previous reports, we identified seven such attributes. 

However, the contracts we reviewed incorporated aspects of these 

attributes to varying degrees. We found that the contracts did not identify 

individuals accountable for achieving results or clearly demonstrate 

relationships between resource needs and performance. We also 

determined that the contracts, with the exception of DODEA, did not 

provide officials with a comprehensive view of agency performance, 

including factors that drive organizational performance, such as financial, 

customer, and internal business processes. Furthermore, defense agency 

contracts were inconsistent in demonstrating results by providing trend 

data to show agency progress in meeting established goals over time, and 

in discussing data quality issues and providing contextual information to 

help officials understand the measures used. We based our analysis on the 

most recent performance contracts developed by DLA, DISA, and DODEA 

because DOD officials told us that these would reflect lessons learned 

from prior iterations. Our analysis did not include other internal 

management documents, such as strategic plans, that may have addressed 

these attributes. We also did not evaluate the usefulness or 

appropriateness of the measures themselves. Table 2 summarizes the 

extent to which the performance contracts we reviewed included the key 

attributes we have identified. 

Contracts Did Not Fully 

Incorporate Certain Key 

Attributes 
Page 13 GAO-04-919 Defense Management 

Table 2: Extent To Which DLA’s, DISA’s, and DODEA’s Performance Contracts Incorporated Certain Key Attributes 

Associated with Results-Oriented Management 

Notes: “Fully incorporated” indicates that all measures exhibited that particular attribute. “Partially 

incorporated” indicates that some of the measures, but not all, exhibited that attribute, whereas “not 

incorporated” means that none of the measures did. 

a Performance contract measures did not cascade to lower organizational levels. However, agency 

officials told us that they shared strategic goals and objectives from their strategic plans with all 

organizational levels and made these available to customers and stakeholders. 

Attribute 

DLA DODEA 

FY 2001 FY 2002 FY 2003 

Aligns goals and measures with agency and departmentwide goals 

Each of the agency’s strategic plan goals is supported by performance measures 

Measures align with departmentwide goals 

Goals and measures cascade from the corporate level to the lowest level of the agency a

Assigns accountability for achieving results

Establishes a foundation to hold top leadership accountable for achieving results 

Performance appraisals for top agency leadership reflect performance goals and 

measures 

b

Identifies individuals accountable for achieving results 

Demonstrates results 

Includes a combination of output- and outcome-oriented measures

Measures are clearly defined 

Provides trend data to demonstrate progress against targeted performance 

Provides comprehensive view of agency performance 

Measures address factors that drive organizational performance (financial, customer, and 

internal business processes and workforce learning and growth) 

Measures address cost, quality, quantity, and timeliness

Measures address key business lines 

Incorporates customer and stakeholder priorities 

Links resource needs to performance 

Measures identify resources (e.g., human capital and information technology) needed to 

achieve performance goals 

Discusses data quality 

Discusses data verification and validation procedures, including reliability 

Identifies sources of data for each measure 

Provides contextual information 

Provides context or explanation for understanding measures and reported performance 

Source: GAO analysis of DLA’s, DISA’s, and DODEA’s performance contracts.

DISA

Legend: Fully incorporated; Partially incorporated; Not incorporated
Page 14 GAO-04-919 Defense Management 

b According to a DLA official, performance appraisals for top agency leadership reflected performance 

contract measures. However, DLA officials were not able to provide a standardized performance 

appraisal template to demonstrate this. 

Additional information about the extent to which the performance 

contracts for the agencies we reviewed included these attributes is 

discussed below. 

DOD did not require performance contract measures to align with 

agencies’ strategic plans or departmentwide goals until fiscal year 2004. As 

a result, some of the agencies’ performance contract measures did align, 

while other measures did not. Each agency’s performance contract 

included statements conceptually linking performance goals with strategic 

plans and departmentwide goals. For example, DISA’s fiscal year 2002 

performance contract stated that “The performance measures used in this 

contract directly support the goals and objectives in the strategic plan,” 

and DISA’s strategic plan directly related to DOD strategic goals and 

objectives. We found that all of DODEA’s performance measures aligned 

with the agency’s strategic plan goals and could be linked to 

departmentwide goals. With the exception of the following example, all of 

DLA’s fiscal year 2001 performance contract measures aligned with the 

agency’s strategic plan goals, but not all aligned with departmentwide 

goals. DLA’s contract lacked performance measures that addressed its 

strategic plan goal to ensure that the agency’s workforce is able to deliver 

and sustain world-class performance. Consequently, the performance 

contract did not provide DLA and DOD officials with information to help 

assess recruiting needs or workforce skills and training requirements that 

contribute to meeting customer needs. 

Furthermore, performance goals and measures used in the contracts we 

reviewed did not cascade from the highest to the lowest levels of the 

agencies to reinforce accountability throughout the organization. 

While defense agency performance contracts identified individuals 

accountable for achieving performance results at the most senior levels of 

DOD and the defense agencies, they did not identify individuals 

responsible for tracking performance or making improvements at lower 

organizational levels. 

Defense Agency Task Force officials were directed to review the defense 

agencies’ performance contracts and make recommendations to the 

Deputy Secretary of Defense on whether the performance contracts 

satisfied DOD guidance. The Defense Management Council was directed to 

Performance Contracts Did Not 

Always Align with Agency and 

Departmentwide Goals 

Contracts Did Not Assign 

Accountability for Achieving 

Results to Individuals at Lower 

Organizational Levels 
Page 15 GAO-04-919 Defense Management 

recommend ways to reduce costs incurred by the defense agencies while 

improving customer service. DOD officials told us that after the events of 

September 11, 2001, OSD’s attention on the performance contracts 

temporarily diminished, less strict oversight occurred, and PA&E officials 

did not hold agencies to meeting reporting requirements. However, 

beginning in April 2002, the Deputy Secretary of Defense began requiring 

the Under and Assistant Secretaries of Defense to provide quarterly 

updates concerning the defense agencies’ performance on key 

performance contract measures and the corrective actions to be taken if 

performance goals were not expected to be met. DOD officials were not 

able to provide historical information concerning whether these reviews 

occurred as frequently as required or whether corrective actions were 

taken. 

Although top leaders had oversight of the performance contracts, none of 

the performance contracts we reviewed identified individuals, such as 

business managers, responsible for achieving results at lower 

organizational levels as a way to reinforce accountability for achieving 

results and maintain focus on organizational priorities. However, in a 

related internal management report called the Chief Financial Executive 

4th Quarter Fiscal Year 2002 Performance Contract Report—which DISA 

used internally to report progress against performance contract goals— 

DISA identified business managers responsible for monitoring each 

performance measure. 

Defense agency performance contracts did not consistently provide 

information to help assess progress and demonstrate results. Although the 

agencies’ performance contracts included milestones or time frames for 

completing specific initiatives, they differed in the degree to which 

measures were clearly defined and outcome-oriented and the extent to 

which they included trend data. For instance, we found that some 

measures in DODEA’s performance contract were not clearly defined. One 

example is: 

“improve student achievement and education quality consistent with Presidential 

and national educational goals. Per pupil costs will not increase more than 7% 

over the previous year.” 

This measure is actually two measures. Furthermore, the measure does 

not define the specific presidential and national education goals—such as 

Contracts Did Not Consistently 

Provide Information to Help 

Assess Progress and 

Demonstrate Results 
Page 16 GAO-04-919 Defense Management 

the No Child Left Behind Act of 2001—against which the agency measured 

its progress.6

In addition, DODEA and DISA did not consistently report trend data for 

each measure in their performance contracts to show progress made over 

time, and DLA limited trend data to the prior year. For example, in its 

fiscal year 2003 performance contract results, DODEA reported that it had 

met goals for teacher certification and school accreditation, but the 

agency did not identify the baseline or percentage of teachers certified or 

schools accredited. 

Each agency’s performance contract included measures that addressed 

key business lines as well as cost, quality, quantity, and timeliness of 

services, but, with the exception of DODEA, did not provide as 

comprehensive a view of agency performance as possible because the 

measures did not address all drivers of organizational performance, 

including customer, financial, and internal business processes, and 

workforce learning and growth. Doing so could have helped officials 

assess how well the agencies were positioned to perform in the future. 

We found that each agency’s most recent performance contract included 

measures assessing aspects of each key business line. For example, DLA’s 

fiscal year 2001 contract included 26 measures that supported its 5 

business lines, including Defense Automated Printing Service, Defense 

Distribution, Defense National Stockpile Center, Defense Reutilization and 

Marketing Service, and Supply (Energy and Non-Energy). DISA’s fiscal 

year 2002 contract included 29 measures that addressed 4 business lines, 

including Computing Services, Enterprise Acquisition Services, Joint 

Warfighting and DOD-wide Enterprise Capabilities, and 

Telecommunication Services. DODEA’s fiscal year 2003 performance 

contract included 15 measures that addressed its 2 business lines—

domestic and overseas education. 

While measures addressed the agencies’ key business lines as well as cost, 

quality, and timeliness of services provided, they did not all address all 

drivers of organizational performance. For example, the contracts did not 

always reflect the different concerns of customers and stakeholders. 

According to defense agency officials, PA&E officials prescribed certain 

 6

 The No Child Left Behind Act of 2001, among other things, focused attention on closing 

the achievement gaps among various groups of students. 

Contracts Did Not Provide a 

Comprehensive View of 

Organizational Performance 
Page 17 GAO-04-919 Defense Management 

measures, although agency officials believed these were not value added 

and did not align with agency strategic goals. For instance, PA&E officials 

required DODEA officials to identify Management Headquarters Costs—

and specifically the “cost per school”—in DODEA’s performance contract. 

Agency officials told us that this measure had limited value because it 

provided aggregated data rather than specific information for individual 

school’s costs. Furthermore, according to a former senior official in the 

Office of the Secretary of Defense for Acquisition, Technology, and 

Logistics, the performance contracts did not reflect the needs of the chiefs 

of staff of the services and the Secretary of Defense, who were key 

customers and stakeholders. This official believed that the measures were 

too technical and “in the weeds” to help inform strategic management 

decisions and were more appropriate for lower-level managers to use 

when monitoring program performance. 

None of the contracts we reviewed identified the resources needed to 

meet performance goals or explained how additional resources could have 

contributed to achieving expected levels of performance. Such 

information could have helped officials to better understand how 

trade-offs among different investments—such as human capital, 

information technology, and workforce training—affected performance 

outcomes and drove costs. For instance, DLA established a fiscal year 

2001 goal of improving inventory accuracy by 4.2 percent over the 

previous year’s performance, but it did not provide additional data in its 

performance contract that would have helped officials to determine the 

resources needed to achieve this goal, if any. DOD officials stated that they 

began to establish links between resource needs and performance 

outcomes by requiring the defense agencies to submit their performance 

contracts at the same time as their Program Objectives Memorandum.7

None of the performance contracts we reviewed discussed the quality of 

performance data, including reliability, data sources, or data verification 

and validation procedures. Consequently, when reviewing the 

performance contracts, top officials in DLA, DISA, and DODEA may not 

have had information needed to determine whether reported data and 

performance results were accurate, reliable, or current enough to use in 

decision making and that corrective actions would produce the desired 

 7

 Program Objectives Memorandum is the final product of the programming process within 

DOD. The Program Objectives Memorandum displays the resource allocation decisions of 

the DOD components in response to, and in accordance with, Defense Planning Guidance. 

Linkage between Resource 

Needs and Performance 

Outcomes Was Not Clearly 

Established 

Quality of Data Was Not 

Discussed 
Page 18 GAO-04-919 Defense Management 

outcomes. For example, an October 2000 internal audit initiated by the 

DISA Inspector General concluded that data reported in DISA’s 

1st Quarter Fiscal Year 2000 Chief Financial Executive Report 

contained errors or bias and lacked sufficient supporting documentation.8

The Inspector General reported that these conditions occurred because 

DISA had not established a systematic approach for collecting and 

reporting performance data. The Inspector General concluded that, as a 

result, reported data provided limited assurance that DISA had valid and 

reliable information to manage agencywide performance. The Inspector 

General concluded that DISA should establish a data collection and 

reporting process and definitive procedures to ensure performance 

information was sufficiently complete, accurate, and consistent to 

document actual performance and support decision making. Senior DISA 

officials told us that the agency did not initially implement related 

recommendations because DISA discontinued use of performance 

contracts. However, according to these officials, DISA is currently taking 

steps to address these recommendations by setting the expectation that 

senior officials will be personally accountable for the quality of data 

included in DISA’s balanced scorecard and establishing internal focal 

points for data collection and reporting. In addition, DISA officials are 

reviewing commercial products to track data electronically. 

The performance contracts provided some contextual information about 

some external factors outside of the agencies’ control that may have 

impacted the achievement of performance goals and provided general 

information about the agencies’ mission, business lines, and customers. 

However, the performance contracts did not always provide some 

contextual information to help officials understand each agency’s 

accomplishments for that fiscal year; determine how measures related to 

agency and departmentwide goals; assess agency progress; understand the 

impact that external factors, such as contingency operations, could have 

on the agency’s ability to meet its performance targets; and determine how 

resource needs should be prioritized. For example, DISA reported in its 

fiscal year 2002 performance contract under the Joint Warfighting and 

DOD-wide Enterprise Capabilities business line that it planned to, 

“Successfully complete fielding of a DOD standard spectrum management 

information system (Spectrum XXI) software version 3.0 by December 

2001.” However, it is unclear how this linked specifically to DISA’s 

 8

Defense Information Systems Agency Inspector General. Audit of DISA’s Performance 

Contract. Final Report. Project No. 2000-H-601. (Washington, D.C.: October 24, 2000). 

Contextual Information Could 

Have Better Informed Decision 

Making 
Page 19 GAO-04-919 Defense Management 

strategic plan or DOD’s goals or what external factors could have 

impacted DISA’s intended performance. 

In fiscal year 2003, DOD renamed the performance contracts as 

performance plans and, based on the agencies’ experiences using 

performance contracts, revised the oversight and review process and 

guidance. DOD required performance measures, beginning in fiscal year 

2004, to align with agency and departmentwide goals and provide a more 

comprehensive view of agency performance. However, the plan and 

scorecard we reviewed could be stronger tools for informing officials 

about agency performance if they incorporated some additional key 

attributes associated with results-oriented management such as linking 

resource needs to performance, discussing data quality, and providing 

contextual information for understanding the measures and reported 

performance. 

Six years after implementation of the performance contracts, the original 

intent—to assist senior leaders at the defense agencies and within OSD to 

oversee agency performance and to encourage the agencies to become 

more responsive to customer needs—remains unchanged. However, 

beginning in fiscal year 2004, DOD renamed the performance contracts as 

performance plans and gave the defense agencies the option to use either 

performance plans or balanced scorecards. These tools were to build on 

the performance contracts, and to include additional attributes associated 

with results-oriented management that were previously lacking under the 

contracts. These changes centered on (1) revising the oversight and review 

process to strengthen accountability for achieving results, (2) requiring 

performance measures to align with agency strategic goals and DOD’s risk 

management framework, and (3) requiring the measures to be more 

comprehensive in scope to better demonstrate results. 

The defense agencies we reviewed are at different stages of developing 

and implementing performance plans and balanced scorecards, and DOD 

has required these tools to be in place for less than a year. Consequently, it 

is too soon to determine how useful they will be for managing and 

measuring agency performance. However, DLA’s balanced scorecard and 

DODEA’s performance plan showed progress in incorporating many of the 

attributes associated with results-oriented management that we have 

identified, but because DOD did not require the defense agencies to 

include all of the attributes discussed in this report, the plan and scorecard 

we reviewed continue to lack certain information—such as the 

The Plan and 

Scorecard We 

Reviewed Show 

Improvement, but 

Could Be 

Strengthened by 

Including Certain Key 

Attributes 

DOD Strengthens 

Oversight Process and 

Requires Inclusion of 

Additional Key Attributes, 

but Some Are Still Lacking 
Page 20 GAO-04-919 Defense Management 

relationship between resource needs and performance, discussions of data 

quality, and some contextual information—that could strengthen these 

tools for managing defense agency performance. Although we obtained 

DISA’s draft balanced scorecard dated September 2003, we did not 

evaluate it against the key attributes discussed in this report because it 

had not been finalized by the time we completed our review. However, we 

do discuss DISA’s plans for addressing several of the key attributes in the 

balanced scorecard that the agency is developing. 

Table 3 shows the extent to which DLA’s balanced scorecard and 

DODEA’s performance plan incorporated certain key attributes associated 

with results-oriented management. We based our analysis specifically on 

the performance plan and balanced scorecard, although other agency 

internal management documents may have addressed these attributes. We 

also did not evaluate the usefulness or appropriateness of the measures 

themselves. 
Page 21 GAO-04-919 Defense Management 

Table 3: Extent To Which DLA’s Scorecard and DODEA’s Plan Incorporated Certain Key Attributes Associated with ResultsOriented Management 

Notes: “Fully incorporated” indicates that all measures exhibited that particular attribute. “Partially 

incorporated” indicates that some of the measures, but not all, exhibited that attribute, whereas “not 

incorporated” means that none of the measures did. 

a Performance measures did not cascade to lower organizational levels. However, DODEA officials 

told us that they shared strategic goals and objectives from their strategic plans with all organizational 

levels and made these available to customers and stakeholders. 

Legend: Fully incorporated; Partially incorporated; Not incorporated

DLA 

Attribute FY 2004 FY 2004 

Aligns goals and measures with agency and departmentwide goals 

Each of the agency’s strategic plan goals is supported by performance measures 

Measures align with departmentwide goals 

Goals and measures cascade from the corporate level to the lowest level of the agency a

Assigns accountability for achieving results 

Establishes a foundation to hold top leadership accountable for achieving results

Performance appraisals for top agency leadership reflect performance goals and 

measures 

Identifies individuals accountable for achieving results 

Demonstrates results 

Includes a combination of output- and outcome-oriented measures

Measures are clearly defined 

Provides trend data to demonstrate progress against targeted performance 

Provides comprehensive view of agency performance 

Measures address factors that drive organizational performance (financial, customer, and 

internal business processes and workforce learning and growth) 

 

Measures address cost, quality, quantity, and timeliness 

Measures address key business lines 

Incorporates customer and stakeholder priorities 

Links resources needs to performance 

Measures identify resources (e.g., human capital and information technology) needed to 

achieve performance goals 

 

Discusses data quality 

Discusses data verification and validation procedures, including reliability 

Identifies sources of data for each measure 

Provides contextual information 

Provides context or explanation for understanding measures and reported performance 

 DODEA

Source: GAO analysis of DLA's balanced scorecard and DODEA's performance plan.
Page 22 GAO-04-919 Defense Management 

Additional information about the extent to which DLA’s balanced 

scorecard and DODEA’s performance plan include these attributes is 

discussed on the following pages. 

The balanced scorecard and performance plan we reviewed showed 

varying progress in meeting DOD’s requirement to align performance goals 

and measures with agency strategic plan goals and DOD’s risk 

management framework beginning in fiscal year 2004. However, DLA and 

DODEA fulfilled DOD’s requirement that every organization’s performance 

goals and measures be consistent with those of its next higher and lower 

organizations. DLA and DODEA have adopted different strategies for 

cascading their scorecard and plan, respectively. DLA’s corporate-level 

balanced scorecard aligns with its strategic plan goals to transform DLA’s 

operations over the next 2 to 5 years to better support its customers and 

departmentwide performance measures outlined in DOD’s risk 

management framework. DLA has also cascaded its corporate scorecard 

to each of its Defense Supply Centers located in Richmond, Virginia, 

Columbus, Ohio, and Philadelphia, Pennsylvania, as well as other field 

activities including the Defense Energy Support Center, Defense 

Distribution Center, and Defense Reutilization and Marketing Service. 

Each center’s or field activity’s scorecard identifies “how and to what 

extent” their strategies support DLA’s corporate-level balanced scorecard 

and is expected to reflect DLA’s overall vision of “Right Item, Right Time, 

Right Place, Right Price, Every Time...Best Value Solutions for America’s 

Warfighters.” According to DLA officials, in this way, DLA provides a 

consistent vision to its customers, stakeholders, and employees. They told 

us that this process also allows customers, stakeholders, and employees to 

understand how corporate-level strategies are supported by functional, 

day-to-day measures. 

Figure 1 depicts how DLA’s corporate-level scorecard aligns with 

departmentwide goals and measures and how the Defense Supply Center 

Richmond’s internal scorecard supports these goals. DLA officials told us 

that the goals within the institutional risk area of DOD’s risk management 

framework cascade down to the Under Secretary of Defense for 

Acquisition, Technology, and Logistics’ balanced scorecard. This 

scorecard has four strategies that cascade down and inform DLA’s 

scorecard. In turn, the four strategies in DLA’s corporate-level scorecard 

cascade down to its defense supply centers. Figure 1 depicts specifically 

how the three strategies within the customer portion of DLA’s scorecard 

cascade down to the Defense Supply Center Richmond. The supply center 

has developed four measures that it uses to assess its performance in 

terms of customer service on a monthly basis. 

Varying Progress Made in 

Aligning Performance Goals 

and Measures with Agency and 

Departmentwide Strategic 

Goals and Cascading These to 

Lower Organizational Levels 
Page 23 GAO-04-919 Defense Management 

Figure 1: Depiction of How DLA’s May 2004 Scorecard Supports Departmentwide Goals and Cascades to Lower 

Organizational Levels 

DODEA’s fiscal year 2004 performance plan does not include goals and 

measures that support all of its community strategic plan goals and 

objectives such as (1) developing, promoting, and maintaining a network 

 

Under Secretary of Defense (Acquisition, Technology, and Logistics) scorecard

DLA scorecard 

DLA scorecard customer strategies 

Defense Supply Center Richmond's internal scorecard customer measures 

Source: GAO presentation of DOD and DLA information.

Industrial plant equipment 

on-time completion

Goal: Percentage of

 industrial plant equipment

 orders fulfilled on time

will be 90 percent.

Product quality –

deficiency reports

Goal: Maximum number

of unresolved reports 

from customers 

indicating deficiency with 

requisitioned product 

will be 500.

Supply availability

Goal: Percentage of 

stocked item requisitions

for all military services 

that are filled immediately 

will be 87.7 percent. 

Backorder age 

Goal: Reduce 

backorders that are

 unfilled for at least 180 

days to 16,900. 

Internal processes

Warfighting

Financial

Resources

Customer Learning and growth

Innovation and

learning

Engage customer in a 

structured, collaborative, 

and partnering 

relationship

Translate customer needs 

into integrated logistics 

solutions to maximize 

readiness and combat power

Deliver promised support

consistently 

Logistics process
Page 24 GAO-04-919 Defense Management 

of meaningful partnerships and alliances to enhance social, emotional, and 

academic growth and (2) maximizing resources or developing safe, 

well-managed, and disciplined environments conducive to learning. 

However, DODEA has linked each performance measure in its 

performance plan with the risk management framework area that it 

supports. For example, DODEA linked the measure “In fiscal years 2004-

2009, no less than 95 percent of the Department of Defense Domestic 

Dependent Elementary and Secondary Schools teachers will be 

professionally certified in the subject area and grade level to which they 

are assigned within three years of employment” with force management 

risk, and specifically the departmentwide goal to maintain a quality 

workforce. DODEA officials told us that they continue to communicate 

these goals and performance measures to employees and customers 

systemwide through their community strategic plan, and school 

improvement plans reflect these goals, as relevant. 

DOD has strengthened the accountability of OSD and top agency 

leadership for achieving performance results through the development of 

OSD-level scorecards and semiannual reviews of defense agency 

performance, and the agencies we reviewed are taking steps to strengthen 

accountability at lower organizational levels. However, DODEA’s 

performance plan does not identify specific individuals who are 

responsible for making improvements. The Under and Assistant 

Secretaries of Defense continue to have responsibility for reviewing the 

defense agencies’ performance plans or scorecards and providing direct 

feedback. However, some have developed balanced scorecards—such as 

the Under Secretary of Defense for Personnel and Readiness 

(USD/P&R)—and have taken steps to incorporate data from quarterly 

reviews of agency performance measures in their balanced scorecards 

while continuing to provide feedback to agency officials on a regular basis. 

For example, prior to approving DODEA’s fiscal year 2004 performance 

plan, the Deputy Under Secretary of Defense for Military Community and 

Family Policy within USD/P&R asked the agency to incorporate updated 

information on its customer satisfaction survey and to explain annual 

variations in the cost-per-pupil index. OSD and agency officials also review 

performance data during periodic internal meetings. For example, figure 2 

shows the high-level DODEA performance measures that officials within 

USD/P&R monitor on a quarterly basis in their balanced scorecard called 

Monitoring the Status of the Force. Part A: Balanced Scorecard. 

USD/P&R officials are responsible for monitoring the force management 

risk area, and within this area, performance strategies related to “maintain 

workforce satisfaction.” Specifically, these officials monitor DODEA’s 

progress in meeting three performance contract measures. 

Oversight and Review Process 

Strengthened to Reinforce 

Accountability 
Page 25 GAO-04-919 Defense Management 

Figure 2: Example of the Type of Information in DODEA’s Performance Plan That OSD Officials Monitor, as of May 2004 

Additional details about each performance measure are also available to 

USD/P&R officials, as depicted in figure 3. 

Maintain

reasonable

costs

Maintain a

quality military

force 

Reading Language 

arts 

DODEA performance measures Math 

75% of all students perform at or

above the standard on a national test

in reading, language arts, and math

8% or fewer of all students will fall in

the below the standard category on 

a national test in reading, language

arts, and math

Pupil-teacher ratio not less

than 18.0:1 nor greater than 24.0:1

Student achievement (2)

Pupil-teacher ratio

Student achievement (1)

Department of Defense 

Force Management Risk

Performance target met

Source: GAO presentation of DOD data.

hape the force

f the future

Sh

of

Ensure sustainable 

military tempo

Maint

satisfact

Maintain workforce

sfaction sfacti

Maint

sfact
Page 26 GAO-04-919 Defense Management 

Figure 3: Detailed Data on DODEA’s Pupil to Teacher Ratio as of May 2004 

DOD officials told us that the Office of the Under Secretary of Defense for 

Acquisition, Technology, and Logistics that oversees DLA and the 

Assistant Secretary of Defense for Network Information and Integration 

that oversees DISA are developing balanced scorecards that are expected 

to incorporate certain performance measures from DLA’s and DISA’s 

balanced scorecards, respectively, in order to help maintain oversight and 

monitor the agencies’ progress toward achieving specific performance 

goals. 

DOD has also implemented semiannual performance reviews at which 

time the defense agencies brief PA&E officials, respective Under and 

Assistant Secretaries of Defense, and agency officials on progress to date 

against performance goals. DOD began holding midyear reviews during 

May 2004, and officials expect to complete these reviews by August. A 

DOD official told us that in the future, the first cycle reviewing the prior 

year’s performance will occur between September and December and the 

second cycle, the midyear review, will occur between May and August. 

DOD officials expect results from these meetings to be reflected in the 

agencies’ fiscal year 2005 performance plans and scorecards. During a 

17

18

19

19.51

21.01

20.41 20.41

20.61

19.51 19.51

18.51

20

21

22

23

24

25

26

 FY 2000

Number of pupils

Source: DOD.

Actual performance

 FY 2001 FY 2002 FY 2003

Department of Defense Elementary and Secondary Schools

Department of Defense Dependent Schools

Target

pupil to teacher

ratio range
Page 27 GAO-04-919 Defense Management 

May 2004 midyear review of DLA’s balanced scorecard, the Director, 

PA&E, recommended that the agency include more specific information 

on its customer surveys, including the population surveyed and the way 

customer feedback is obtained. 

By assigning responsibility for each performance measure to specific 

individuals, some defense agencies are reinforcing accountability for 

achieving performance goals at all levels of the organization. For example, 

in addition to implementing performance appraisals for Senior Executive 

Service staff that are linked to strategic goals and objectives against which 

ratings are assessed, DLA identified individuals in its balanced scorecard 

and charged them with monitoring, reporting on, and implementing 

corrective actions for each scorecard measure. A senior DISA official told 

us that the agency recently approved linking balanced scorecard outcomes 

with Senior Executive Service members’ and supervisory employees’ 

performance appraisals beginning in fiscal year 2005 in order to 

institutionalize its balanced scorecard and improve accountability for 

results. DISA also is reinforcing accountability by requiring its senior 

leaders (scorecard owners), who are accountable for achieving results, to 

provide status updates during corporate-board meetings of senior agency 

officials. In contrast, DODEA’s performance plan does not identify specific 

individuals accountable for monitoring, reporting on, and implementing 

corrective actions for each performance measure. However, DODEA 

officials told us that the agency includes performance plan objectives in 

each manager’s performance appraisal, and both monetary and 

nonmonetary awards are used as incentives to meet or exceed goals and 

measures. 

DLA’s balanced scorecard and DODEA’s performance plan showed 

varying progress in the extent to which they included outcome-oriented 

measures and trend data to help demonstrate results, and the clarity of 

performance measures continues to be an issue. We determined that 

DLA’s 2004 scorecard includes outcome-oriented measures and trend data 

that help to demonstrate progress made. For example, one of DLA’s 

outcome-oriented financial measures—the achievement of accuracy in 

non-energy sales forecasting—includes baseline and trend data for the 

3 preceding fiscal years. However, DLA’s 2004 scorecard measures include 

terms that are generally not clearly defined, as demonstrated by the 

measure “Percentage of reengineered functionality implemented in BSM.” 

In contrast, DODEA’s performance plan includes almost all of the same 

measures as did its fiscal year 2003 performance contract, and these 

measures are not always clearly defined or accompanied by trend data. As 

we previously mentioned, DODEA does not clearly define the specific 

Agencies’ Progress in 

Demonstrating Results Varies 
Page 28 GAO-04-919 Defense Management 

presidential and national education goals against which it measures 

progress. Furthermore, DODEA’s performance plan does not consistently 

include baseline information or trend data to help officials gauge progress. 

DODEA did not indicate the percentage of teachers who are certified in 

the subject area and grade level to which they are assigned within 3 years 

of employment, although the agency set a goal of no less than 95 percent. 

The plan also does not indicate the percentage of schools that have full 

high-speed connectivity to the Internet, although it establishes a goal of 

100 percent. 

Our analysis showed that DLA’s scorecard and DODEA’s performance 

plan included measures for key business lines that provide officials with 

information about agency performance and, to varying degrees, address 

the four drivers of organizational performance as well as cost, quality, and 

timeliness of services provided, and that all of the agencies we reviewed 

are taking steps to better reflect customers’ and stakeholders’ priorities. 

DLA organizes its corporate-level scorecard by drivers of performance—

including learning and growth and customer, financial, and internal 

business processes—and includes strategies and supporting measures that 

are cascaded to its defense supply centers and field activities. As would be 

expected, lower organizational level scorecards used by the defense 

supply centers include performance measures that address the key 

business lines for which they are responsible and provide more specific 

assessments of cost, quality, timeliness, and customer satisfaction. For 

example, the Defense Supply Center Richmond’s scorecard includes 

performance measures focusing on aviation weapons systems and 

environmental logistics support, whereas the Defense Supply Center 

Philadelphia’s scorecard focuses on the provision of clothing, textiles, and 

medical equipment, among other items. The learning and growth quadrant 

of DLA’s corporate scorecard contains a measure to identify gaps between 

workforce competencies and skills required to meet mission requirements 

for 100 percent of DLA positions by the end of fiscal year 2007. The 

Defense Supply Center Richmond’s scorecard includes several measures 

that address employee certification, including acquisition training 

certification to ensure that the supply center’s workforce is fully proficient 

in the acquisition process. This measure, in turn, aligns with the 

departmentwide policy of establishing education and training for each 

defense acquisition position. Similarly, DODEA’s plan includes measures 

that address its two key business lines—domestic and overseas 

education—and are cascaded to schools worldwide through its 

community strategic plan. 

Agencies’ Scorecard and Plan 

Show Progress in Providing a 

Comprehensive View of Agency 

Performance 
Page 29 GAO-04-919 Defense Management 

Unlike DLA’s balanced scorecard, DODEA’s performance plan does not 

specifically categorize performance measures by the factors that drive 

organizational performance, but we identified at least one measure or 

initiative that can be linked with each organizational driver. For example, 

DODEA’s performance plan includes a measure to limit increases in per 

pupil cost to no more than 7 percent over the prior fiscal year. This 

measure addresses both financial and internal business processes that 

affect organizational performance. DODEA addresses workforce learning 

and growth by establishing a goal to certify no less than 95 percent of 

teachers in the subject and grade levels to which they are assigned within 

3 years of employment. Finally, DODEA includes measures that address 

customer satisfaction such as students’ performance and parents’ 

satisfaction with local schools. 

Our analysis showed that the defense agencies we reviewed are also 

taking steps to better address specific customer and stakeholder concerns 

about performance. Both DODEA and DLA conduct periodic reviews of 

agency performance and make adjustments to meet customers’ and 

stakeholders’ priorities. For instance, DLA’s balanced scorecard Executive 

Board, comprised of DLA’s Vice Director, designated leaders for each area 

of organizational performance (quadrant leaders), the Chief Information 

Officer, and a Lead Center representative, conducts monthly reviews of 

enterprisewide operational measures. Each quarter, DLA’s Corporate 

Board, comprised of DLA’s Director, Vice Director, and the heads of DLA’s 

major organizational components, meets with commanders from the field 

to review the agency’s performance from the corporate level down to the 

supply centers. The defense supply centers conduct weekly reviews to 

monitor progress, identify developing problems, and take corrective 

actions. According to DLA officials, to make information more timely, 

accessible, and relevant for measuring and managing the agency, DLA 

posts the agency’s balanced scorecard on its internal Web site to allow all 

employees, including agency managers and leaders, to share performance 

data. This Web site is also available to all of DLA’s military customers and 

stakeholders. DLA also continues to obtain customer and stakeholder 

feedback through a variety of mechanisms, such as customer surveys, 

reports from customer service liaisons embedded with deployed military 

service units, and meetings between the agency’s Director and leaders of 

the military services. Moreover, DLA is taking steps to address the unique 

concerns of each military service through the development of 

performance-based agreements that include specific performance 

measures that are based on collaborative, direct negotiations between 

DLA and each military service. These agreements are intended to create 

and sustain end-to-end customer support by aligning DLA’s resources with 
Page 30 GAO-04-919 Defense Management 

the logistics solutions that will meet the services’ needs and maximize 

their effectiveness. 

In addition, we determined that the defense combat support agencies—

such as DLA and DISA—are taking steps to incorporate additional 

strategies and measures in their balanced scorecards that respond to 

customers’ changing needs during contingencies and war. For example, 

DLA’s scorecard assesses readiness by measuring the percentage of items 

that the military services identify as requiring surge and sustainment 

support (e.g., hardware items, clothing and textiles, and medical supplies) 

to ensure appropriate inventory levels to accommodate short production 

lead times. DLA officials told us that to reduce procurement lead time and 

decrease delivery delays to the agency’s primary customer, the warfighter, 

the agency has developed 22 long-term contracts, called strategic supplier 

alliances, with its major contractors. According to these officials, having 

these agreements in place enhances customer support through the 

inclusion of better performance metrics, such as reduced administrative 

lead times and increased stock availability. DISA officials told us that as a 

combat support agency, DISA’s mission is to support the warfighter and 

that the strategies and performance measures DISA is developing focus on 

wartime requirements but also are applicable to peacetime environments. 

For example, DISA is developing measures that will support its customer 

strategy to eliminate bandwidth constraints in response to customer 

complaints that limited bandwidth is a problem, especially during 

contingency operations. According to DISA officials, removing current 

bandwidth limitations can provide DISA’s customers with better and 

timelier information during combat and peacetime. 

A major goal of DOD is to strategically link major decisions to program 

and budget development, and the defense agencies we reviewed are in the 

initial stages of attempting to do this. DOD has directed that the 

Under Secretary of Defense, Comptroller, and the Director, PA&E, review 

program performance and assess the degree to which budget estimates 

sustain and improve programmatic results and, as appropriate, 

recommend alternate solutions or funding adjustments to correct resource 

needs if performance goals are not met. Senior DOD officials told us that 

this continues to be a challenging undertaking. The defense agencies we 

reviewed are using their performance plans and scorecards to bring 

attention to priority management projects. Although these tools still do not 

discuss resource implications, DODEA included goals and measures from 

its fiscal year 2004 performance plan in its fiscal year 2004-2005 operation 

and maintenance, defensewide, biennial budget estimates. We believe that 

Funding Not Yet Linked with 

Performance 
Page 31 GAO-04-919 Defense Management 

inclusion of the agency’s priorities can help officials better consider 

appropriate funding levels and trade-offs. 

Although DISA is still developing its balanced scorecard, senior agency 

officials told us that the agency plans to use the scorecard to bring 

visibility and management attention to priority projects in order to fund 

them appropriately and to improve collaboration internally. These officials 

told us that DISA is in the process of aligning its investment and scorecard 

strategies and expects initiatives in its scorecard to be assigned funding 

priority. To this end, DISA is developing a Program Objectives 

Memorandum that aligns investments with its scorecard. During 

corporate-board meetings when senior agency officials review the 

agency’s performance, DISA requires each measure owner to discuss the 

context and status of initiatives, including the need for additional 

resources. According to DISA officials, the agency has already 

experienced some benefits when testing this approach. DISA officials 

identified a disconnect between customer expectations, performance 

levels, and funding for an initiative to develop a Web-based portal to share 

information about DISA’s business activities internally. The goal of the 

portal is to help employees perform critical operations faster and more 

effectively. Officials told us that although internal customers identified this 

initiative as a priority, the portal did not receive dedicated funding and, as 

a result, was neither as robust as planned nor on target to meet 

development time frames. DISA officials told us that inclusion of the 

project in its balanced scorecard is expected to raise these challenges to 

the attention of senior leaders to encourage corrective action and 

appropriate funding levels. 

The balanced scorecard and performance plan we reviewed do not 

provide decision makers with confidence that performance data are 

credible because they do not include specific information on the 

procedures to verify and validate performance information. In addition, no 

information on the sources of data is included. Thus, decision makers may 

not have complete information needed to determine whether reported 

data and performance results are accurate. 

The scorecard and plan we reviewed did not include some contextual 

information that could have helped to inform decision makers because 

DOD guidance did not require the defense agencies to do so. However, a 

sample measure from DISA’s draft September 2003 balanced scorecard 

depicts the type and amount of contextual information that agency 

officials have considered incorporating. 

Quality of Data Is Not 

Discussed 

Some Contextual Information 

Is Lacking 
Page 32 GAO-04-919 Defense Management 

DLA’s balanced scorecard lacks information about the agency’s mission 

and customers that had been included in its performance contract. In 

contrast, DODEA’s performance plan provides general information about 

the agency’s mission, business lines, and customers. However, neither 

agency clearly articulates how its performance strategies and measures 

help achieve mission or departmentwide goals or improve customer 

responsiveness. Also, information about external factors, such as recent 

contingency operations, that can affect achievement of performance goals 

is limited or absent. For example, DODEA’s performance plan includes a 

measure to maintain pupil to teacher ratios for kindergarten through grade 

12 between 18:1 and 24:1 during fiscal years 2004-2009. However, DODEA’s 

performance plan does not explain that studies have demonstrated that 

lower pupil to teacher ratios contribute to improved student performance 

or how this measure supports presidential and national goals against 

which the agency evaluates itself. Providing this contextual information 

could assist officials with realizing how human capital-related strategies, 

such as staffing, affect the agency’s goals to continue to improve student 

performance in reading, math, and languages, and help establish links 

between resource needs and performance outcomes. 

In September 2003, DISA developed a draft balanced scorecard with a 

sample measure that depicts the type and amount of contextual 

information that agency officials have considered incorporating. The 

sample measure, as currently structured, incorporates most of the 

attributes associated with results-oriented management that we have 

discussed in this report, including contextual information. However, the 

sample measure does not include some aspects of data quality such as 

reliability, validity, and verification, and it is unclear whether DISA plans 

to discuss external factors that could affect the achievement of 

performance goals or the link between resource needs and performance 

beyond funding problems. For example, as currently structured, the draft 

scorecard would 

(1) align measures with agency and departmentwide goals. The 

scorecard is expected to identify the quadrant of DISA’s scorecard 

that the measure supports (e.g., customer) and the Assistant 

Secretary of Defense’s balanced scorecard goal and risk 

management framework area with which it aligns. 

(2) assign accountability for achieving results. The scorecard is 

anticipated to identify individuals responsible for monitoring and 

reporting on performance and making corrections, as needed for 
Page 33 GAO-04-919 Defense Management 

each measure. 

(3) demonstrate results. The scorecard is expected to define the 

measure and present trend data showing performance 

improvements over 4 fiscal years. 

(4) discuss data quality. The scorecard is expected to identify data 

sources but does not include information about the way data are 

collected or the verification and validation procedures. 

(5) provide contextual information. The scorecard is expected to 

include historical data, and an assessment of the measure’s status 

compared to prior years, and identify any challenges that need 

attention (e.g., hot issues) such as funding problems. 

Figure 4 shows the content that DISA officials told us that they expect to 

include and the way this information addresses the attributes we have 

identified in this report. 
Page 34 GAO-04-919 Defense Management 

Figure 4: Format of a Sample Measure from DISA’s Draft September 2003 Balanced Scorecard 

Notes: PSA=Principal Staff Assistant. TBD=To Be Determined. 

0

10

20

30

40

50

60

70

1999

Fiscal year

Dollars

Source: GAO analysis of DISA data.

2000 2001 2002

Measure 

is 

on track

Operating Rate

MVS Central Processing Unit Billing Rate

Sample Metric Title: Multiple Virtual System Operating System (MVS OS)/390 Billing Rate

Alignment 

Accountability

Demonstrates

results

Demonstrates

results

Contextual

information

Data quality 

Perspective: Customer

Risk Management Framework: Institutional

PSA Goal: Make information available on a

network that people depend on and trust.

DISA Strategy: Improve affordability of 

net-centric products, services and solutions

Champion: TBD

Owner: TBD

Description: A measure of the cost per unit 

of workload for providing OS/390 mainframe 

processing services.

Historical Data: Indicated in the table for fiscal years 1999-2002. As indicated, the actual operating rate has 

decreased by 53.7 percent during this period.

Target: Fiscal year 2003: $28.93; Fiscal year 2004: $24.17

Overall Assessment: The operating rate is a measure of the actual cost per unit of workload based on execution 

during the fiscal year. In fiscal year 2002, for the fourth consecutive year, the operating rate is less than the billing rate.

The billing rate is established two years in advance based on projected workload data provided by DISA's customers. 

DISA must continue to work closely with its customers to assist them in developing more accurate workload estimates 

that take into account historical trends as well as new requirements. Workload has continued to grow and rates decline 

since fiscal year 1999. Rates will further decline with implementation of transformation initiatives. Based on benchmarking 

studies conducted in 2000 and 2001, rates in fiscal year 2004 will be fully competitive with private sector providers, while 

DISA continues to provide the best in class service shown by the 2002 customer survey.

Hot Issues: None

Data Source: TBD

Algorithm: TBD
Page 35 GAO-04-919 Defense Management 

OSD/PA&E officials have developed some mechanisms to help the defense 

agencies capture and share experiences and lessons learned from 

developing performance plans and balanced scorecards. Some defense 

agencies, such as DLA and DISA, are sharing their experiences with each 

other to leverage lessons learned. 

OSD/PA&E has taken some steps to help the defense agencies share their 

experiences and lessons learned with developing performance plans and 

balanced scorecards, and DOD plans additional initiatives. In response to 

our suggestions during this review, an OSD/PA&E official decided to 

continue to hold semiannual roundtables during which defense agency 

officials could voluntarily share their experiences developing and using 

performance plans or balanced scorecards. OSD/PA&E officials also told 

us that they plan to revise and update a departmentwide Web site 

originally used to disseminate agencies’ performance contracts and related 

guidance and to incorporate this information, as well as defense agencies’ 

performance plans and scorecards, into DOD’s performance management 

Web site. The original site was not updated as frequently as it could have 

been, and firewalls used to protect information from unauthorized access 

precluded defense agencies with domain names that did not end in “.mil” 

from gaining access. For example, DODEA officials were not able to 

access the original Web site because the agency’s World Wide Web domain 

address ends in “.edu.” Consequently, DODEA could not leverage other 

agencies’ contracts and performance measures when developing its own. 

DODEA officials told us that they were not even aware that the Web site 

existed, but that it would have been helpful for identifying some 

performance measures. 

In addition, some defense agencies that we reviewed are proactively 

sharing lessons learned from developing and implementing balanced 

scorecards. For example, DISA officials told us that they contacted DLA to 

learn how it had developed and implemented its balanced scorecard. In 

addition, officials at both agencies told us that they have or are developing 

internal Web sites to facilitate performance measurement. DISA’s internal 

Web site serves as a collaborative forum and repository for balanced 

scorecard material, and provides related guidance on developing balanced 

scorecards. 

DOD has improved its process for measuring and managing defense 

agency performance by taking steps to include certain attributes 

associated with results-oriented management in defense agencies’ 

performance plans and scorecards that were not included in the 

DOD Captures and 

Shares Lessons 

Learned 

Conclusions 
Page 36 GAO-04-919 Defense Management 

performance contracts. Specifically, changes made to DOD’s process for 

overseeing and reviewing defense agencies’ balanced scorecards and 

performance plans have the potential to further strengthen the 

accountability of top leadership for overseeing the defense agencies and 

achieving business transformation goals. While it is too early to determine 

the extent to which these plans and scorecards will be fully useful, these 

tools still do not incorporate some specific attributes that would 

strengthen them. Until all performance plans and scorecards identify 

individuals responsible for monitoring progress and taking corrective 

actions, DOD’s ability to ensure accountability and continuous assessment 

at lower organizational levels is hindered. Furthermore, without 

performance measures that are clearly defined and accompanied by trend 

data and absent a discussion of data quality and contextual information—

such as information about the agency’s missions, business lines, and 

customers as well as external limitations that could affect achievement of 

performance goals—senior OSD and agency leaders will not have the 

necessary information for fully assessing the extent to which progress has 

been made against existing performance targets and determining realistic 

performance goals for future years. Finally, without demonstrating the 

relationship between resource needs and performance outcomes, DOD 

officials do not have all of the information needed to make improvements 

and consider trade-offs among competing priorities. 

To make performance plans and scorecards more informative and useful 

for decision making, and further strengthen these tools’ potential for 

measuring and managing defense agency performance, we recommend 

that the Secretary of Defense direct the Director, Program Analysis and 

Evaluation, to revise guidance to require all defense agencies to address 

the following five recommendations in their plans and scorecards: 

• identify individuals accountable for achieving results at lower 

organizational levels; 

• include measures that are clearly defined and include trend data for at 

least the past fiscal year’s performance to help assess progress; 

• identify resources needed to achieve performance goals and inform 

budget decisions; 

• discuss data quality, including the reliability, validity, and limitations of 

performance measures as well as data sources; and 

• provide contextual information to better understand how performance 

measures support the agency’s mission. 

Recommendations for 

Executive Action 
Page 37 GAO-04-919 Defense Management 

In written comments on a draft of this report, DOD concurred or partially 

concurred with all of our recommendations. DOD's comments are 

reprinted in their entirety in appendix V. DOD also provided technical 

comments that we incorporated as appropriate. 

Specifically, DOD concurred with our recommendations to revise 

guidance to require defense agencies to include information on individuals 

accountable for achieving results at lower organizational levels; include 

measures that are clearly defined and include trend data for at least the 

past fiscal year’s performance; discuss data quality, including the 

reliability, validity, and limitations of performance measures as well as 

data sources; and provide contextual information to better understand 

how performance measures support the agency’s mission. DOD stated that 

it would revise guidance to either implement or clarify these 

recommendations. 

DOD partially concurred with our recommendation to revise guidance to 

require all defense agencies to identify resources needed to achieve 

performance goals and inform budget decisions in their performance plans 

or balanced scorecards. DOD stated that it may not be possible to fully 

cost performance goals given the complex nature of defense operations 

and that the appropriate process for proposing resource alternatives is the 

Planning, Programming, Budgeting, and Execution System. However, DOD 

did acknowledge that the defense agencies' performance plans should be 

synchronized with this process by demonstrating how Planning, 

Programming, Budgeting, and Execution System decisions are being 

implemented and by providing performance information and metrics that 

can be used to inform the department's annual budget and execution 

review and the allocation of resources. We believe DOD is moving in the 

right direction by trying to synchronize defense agency performance 

management with the budget process. However, we continue to believe 

that the defense agencies’ performance plans and balanced scorecards 

would be an appropriate vehicle to discuss resource implications. Full 

cost information may not be available, but the performance plans and 

balanced scorecards should discuss resource implications and include 

available cost information. Inclusion of this information would help 

decision makers understand the connection between performance goals 

and funding levels and evaluate competing needs at all phases of the 

planning and budgeting process. 

Agency Comments 

and Our Evaluation 
Page 38 GAO-04-919 Defense Management 

We are sending copies of this report to other interested congressional 

committees; the Secretary of Defense; the Secretaries of the Army, the 

Navy, and the Air Force; the Commandant of the Marine Corps; and the 

Director, Office of Management and Budget. We will make copies available 

to others upon request. In addition, the report will be available at no 

charge on the GAO Web site at http://www.gao.gov. 

If you or your staff have any questions regarding this report, please call 

Sharon Pickup at (202) 512-4300 or Patricia Dalton at (202) 512-6806. 

Additional contacts and staff acknowledgments are provided in appendix 

VI. 

Sharon L. Pickup 

Director, Defense Capabilities and Management 

Patricia A. Dalton 

Director, Strategic Issues 
Appendix I: Scope and Methodology 

Page 39 GAO-04-919 Defense Management 

To meet our first two reporting objectives, we interviewed knowledgeable 

Department of Defense (DOD), defense agency, and service officials. 

Specifically, we interviewed officials in the offices of the Under and 

Assistant Secretaries of Defense who act as principal staff assistants for 

the three defense agencies reviewed. We also collected data from and 

interviewed officials in the Office of the Secretary of Defense, Program 

Analysis and Evaluation and the Office of the Comptroller. We also 

reviewed our extensive prior work on results-oriented management and 

identified seven attributes that contribute to effective government 

performance. While these do not cover all attributes associated with 

results-oriented management, we believe they are the most important for 

accurately assessing the strengths and weaknesses of programs and 

making improvements. These include (1) aligning goals and measures with 

agency and departmentwide goals; (2) assigning accountability for 

achieving results; (3) demonstrating results; (4) providing a 

comprehensive view of agency performance by addressing factors that 

drive organizational performance; (5) linking resource needs to 

performance; (6) discussing data quality; and (7) providing contextual 

information. We systematically analyzed the measures contained in 

selected performance contracts, performance plans, and balanced 

scorecards against these attributes. We focused our analysis on the 

Defense Logistics Agency (DLA), the Defense Information Systems Agency 

(DISA), and the Department of Defense Education Activity (DODEA). We 

judgmentally selected these three defense agencies to review based on 

(1) their experience developing and implementing performance contracts, 

plans, and balanced scorecards; (2) different types of missions; and 

(3) types of services and types of customers. We analyzed the most recent 

performance contracts used by DLA, DISA, and DODEA, and we limited 

our analysis of balanced scorecards and performance plans to DLA’s fiscal 

year 2004 balanced scorecard and DODEA’s fiscal year 2004 performance 

plan. Although we obtained DISA’s draft balanced scorecard dated 

September 2003, we did not evaluate it against the attributes we discuss in 

this report because DISA had not yet finalized its scorecard by the time we 

completed our review. However, we did discuss DISA officials’ plans for 

developing a balanced scorecard. We did not evaluate the usefulness or 

appropriateness of the measures contained in the contracts, plans, and 

balanced scorecards or verify actual or reported performance. In addition 

to analyzing the performance contracts, performance plans, and balanced 

scorecards, we also reviewed the defense agencies’ strategic plans and 

annual performance contract reports. Further, we reviewed the 

cascaded-balanced scorecards that are used at lower organizational levels, 

specifically, the fiscal year 2004 scorecards for DLA’s supply centers 

(Richmond, Virginia, Philadelphia, Pennsylvania, and Columbus, Ohio) and 

Appendix I: Scope and Methodology 
Appendix I: Scope and Methodology 

Page 40 GAO-04-919 Defense Management 

for its Defense Energy Support Center, Defense Distribution Center, and 

Defense Reutilization and Marketing Service. We also reviewed school 

improvement plans for three elementary and middle schools located at 

Fort Bragg, North Carolina, to determine whether they aligned with 

DODEA’s community strategic plan and performance contract goals and 

measures. 

To assess the extent to which DOD has established mechanisms to capture 

and share lessons learned from the implementation of performance plans 

and balanced scorecards, we interviewed knowledgeable DOD, defense 

agency, and service officials and reviewed pertinent documents. 

We performed our work between July 2003 and May 2004 in accordance 

with generally accepted government auditing standards. We obtained 

comments on a draft of this report from DOD and incorporated comments 

where appropriate. 
Appendix II: Department of Defense’s Risk 

Management Framework 

Page 41 GAO-04-919 Defense Management 

In 2001, the Department of Defense (DOD) established a risk management 

framework to use in considering trade-offs among defense objectives and 

resource constraints. The framework consists of four dimensions of risk 

with supporting performance measures that are being cascaded to the 

defense agencies and activities as well as departmentwide. Although not 

depicted, financial considerations underlay each of DOD’s risk areas. 

Figure 5: DOD’s Risk Management Framework 

Appendix II: Department of Defense’s Risk 

Management Framework 

ch

r

Institutional goals Future challenges goals

Streamline the

decision process,

drive financial 

management 

and acquisition 

excellence

Improve the 

readiness and 

quality of key 

facilities

Realign support 

to the warfighter

(including defense

 agencies) 

Drive joint operations

(CONOPS) experiments

Define and develop

transformation

capabilities

Define future human 

skills and competencies

Manage overhead 

and indirect 

costs

Source: DOD.

Develop more effective

organizations

Institutional

 risk

perational

 risk

Force

managemen

risk

Future

challenges

risk

ch

r

recard

Oper

or

Force management goals

Ensure sustainable 

military tempo

Maintain quality

workforce

Maintain reasonable 

force costs

Maintain workforce

satisfaction

Shape the force 

of the future

Do we have the forces 

available?

Operational goals

Are they employed 

consistent with

the strategy?

Are they postured 

to succeed?

Are forces currently 

ready?

ce

ent 

Balance

rce

men
Appendix III: Fiscal Year When Defense 

Agencies Implemented Their First 

Performance Contract, Plan, or Scorecard 

Page 42 GAO-04-919 Defense Management 

This appendix shows the fiscal year during which each defense agency and 

activity implemented its first performance contract or performance plan. 

The defense agencies and activities that implemented performance 

contracts later converted these to performance plans or scorecards. We 

use the generic term “performance plan” in figures 6 and 7 to include both 

performance plans and balanced scorecards. 

Appendix III: Fiscal Year When Defense 

Agencies Implemented Their First 

Performance Contract, Plan, or Scorecard 
Appendix III: Fiscal Year When Defense 

Agencies Implemented Their First 

Performance Contract, Plan, or Scorecard 

Page 43 GAO-04-919 Defense Management 

Figure 6: Defense Agencies 

Defense 

Finance & 

Accounting 

Service

USD

(Comptroller)

PDUSD 

USD 

(Personnel & 

Readiness)

(Personnel & 

Readiness)

Defense 

Commissary

Agency

Director, 

Administration 

& Management

Pentagon

Force 

Protection

Agency

Defense 

Contract 

Management 

Agency

DUSD

(Logistics & 

Material 

Readiness)

Defense

Logistics

Agency

DUSD 

(Acquisition & 

Technology)

Defense 

Threat Reduction 

Agency 

Director, Defense 

Research & 

Engineering

Defense 

Advanced 

Research Projects 

Agency

ATSD (Nuclear & 

Chemical & 

Biological Defense 

Programs)

National 

Security Agency

Defense 

Intelligence 

Agency

Defense 

Security 

Service

National National 

Geospatial 

Intelligence 

Agency 

Missile Defense 

Agency 

USD 

(Policy)

ASD

(International

Security

Affairs)

Defense

Security 

Cooperation

Agency

USD 

(Acquisition,

Technology, & 

Logistics)

General 

Counsel 

Defense

Legal 

Services 

Agency

Deputy Secretary of Defense

Secretary of Defense

Defense 

Information 

Systems 

Agency

ASD (Networks 

& Information 

Integration)

USD 

(Intelligence)

Defense 

Contract 

Audit Agency

Fiscal Year 1999

Fiscal Year 2000

Fiscal Year 2001

Fiscal Year 2002

Fiscal Year 2004

DOD agencies without performance

plans as of fiscal year 2004

Source: GAO analysis and presentation of DOD data.

First Performance Contract/Plan Implemented ASD Assistant Secretary of Defense

ATSD Assistant to the Secretary of Defense

DUSD Deputy Under Secretary of Defense

PDUSD Principal Deputy Under Secretary of Defense

USD Under Secretary of Defense
Appendix III: Fiscal Year When Defense 

Agencies Implemented Their First 

Performance Contract, Plan, or Scorecard 

Page 44 GAO-04-919 Defense Management 

Figure 7: Defense Field Activities 

Department of 

Defense 

Education 

Activity

American 

Forces 

Information 

Service

ASD 

(Public Affairs)

ASD 

(Health Affairs)

Washington 

Headquarters 

Services

Director,

Administration 

& Management

USD 

(Intelligence)

Department of 

Defense 

Counterintelligence

Field Activity

ASD 

(International 

Security Policy)

ASD 

(International 

Security Affairs)

Defense 

Prisoner 

of War/Missing 

Personnel Office

Defense 

Technology 

Security 

Administration

USD (Acquisition, 

 Technology, & 

Logistics) 

DUSD 

(Installations &

 Environment) 

Office of 

Economic 

Adjustment

USD 

(Policy)

TRICARE 

Management 

Activity

PDUSD

(Personnel & 

Readiness) 

USD

(Personnel & 

Readiness) 

DOD Human 

Resources

Activity 

DOD Test 

Resource 

Management

Director,

Defense 

Research & 

Engineering

Defense 

 Technical

Information 

Center

Center

Deputy Secretary of Defense

Secretary of Defense

Source: GAO analysis and presentation of DOD data.

Fiscal Year 1999

Fiscal Year 2000

DOD field activities without performance 

plans as of fiscal year 2004

First Performance Contract/Plan Implemented ASD Assistant Secretary of Defense

ATSD Assistant to the Secretary of Defense

DUSD Deputy Under Secretary of Defense

PDUSD Principal Deputy Under Secretary of Defense

USD Under Secretary of Defense
Appendix IV: Certain Key Attributes 

Associated with Results-Oriented 

Management 

Page 45 GAO-04-919 Defense Management 

Establishing goals and measuring performance are essential to successful 

results-oriented management practices. Measuring performance allows 

organizations to track the progress they are making toward their goals and 

gives managers critical information on which to base decisions for 

improving their programs. Our body of work on results-oriented 

management practices has identified key attributes of success. This work 

indicates that agencies that are successful in achieving business 

transformation goals, such as those being focused on by the Department of 

Defense in its performance contracts and plans, strive to establish 

practices and performance systems at all levels of the agency that include 

the seven key attributes described in this appendix. 

• Aligns goals and measures with agency and departmentwide goals.

Performance goals and measures should align with an agency’s 

long-term strategic goals and mission as well as with higher-level 

departmentwide priorities, with the relationship clearly articulated. 

Such linkage is important in ensuring that agency efforts are properly 

aligned with goals (and thus contribute to their accomplishment) and 

in assessing progress toward achieving these goals. Goals and 

measures also should cascade from the corporate level of the agency to 

the operational level to provide managers and staff with a road map 

that shows how their day-to-day activities contribute to achieving 

agency and departmentwide performance goals. In addition, measures 

used at the lowest levels of the agency to manage specific programs 

should directly relate to unit results and upwards to the corporate level 

of the agency. 

• Assigns accountability for achieving results. We have previously 

reported that the single most important element of successful 

management improvement initiatives is the demonstrated commitment 

of top leaders in developing and directing reform efforts. Top 

leadership must play a critical role in setting results-oriented goals and 

quantifiable measures that are cascaded to lower organizational levels 

and used to develop and reinforce accountability for achieving results, 

maintain focus on the most pressing issues confronting the 

organization, and sustain improvement programs and performance, 

especially during times of leadership transition. One way to reinforce 

accountability is through the use of employee performance appraisals 

that reflect an organization’s goals. 

• Demonstrates results. A key element of any major program is its ability 

to establish clearly defined goals and performance measures to 

monitor and report progress to management, identify corrective 

actions, set priorities, and allocate resources. Performance measures 

Appendix IV: Certain Key Attributes 

Associated with Results-Oriented 

Management 
Appendix IV: Certain Key Attributes 

Associated with Results-Oriented 

Management 

Page 46 GAO-04-919 Defense Management 

should show an organization’s progress toward achieving an intended 

level of performance or results and include quantifiable targets or other 

measurable values to help assess progress and trend data toward 

accomplishing goals. Using a combination of output measures that 

assess the actual level of work accomplished or services provided and 

outcome measures that assess the actual results, effects, or impacts of 

a program or service compared to the intended purpose can also help 

determine progress made. 

• Provides a comprehensive view of agency performance. For each key 

business line, performance goals and measures should provide a 

comprehensive view of performance, including customers’ and 

stakeholders’ priorities. Goals and measures should address key 

performance dimensions such as (1) factors that drive organizational 

performance—including financial, customer, and internal business 

processes, and workforce learning and growth—and (2) aspects of 

customer satisfaction, including timeliness, quality, quantity, and cost 

of services provided. Doing so can allow managers and other 

stakeholders to assess accomplishments, make decisions, realign 

processes, and assign accountability without having an excess of data 

that could obscure rather than clarify performance issues. 

• Links resource needs to performance. One of the ways that 

performance management can be promoted is if this information 

becomes relevant for (1) identifying resources (e.g., human capital, 

information technology, and funding) needed to achieve performance 

goals; (2) measuring cost; and (3) informing budget decisions. When 

resource allocation decisions are linked to performance, decision 

makers can gain a better understanding of the potential effect of 

budget increases and decreases on results. 

• Discusses data quality. A key factor that contributes to the usefulness 

of performance data is the degree to which officials are confident that 

the performance information is credible. Useful practices for helping 

decision makers to assess the quality and value of performance data 

include discussion of data sources and standards and methods—

including data collection and reliability, verification and validation 

procedures, and proposals to review these procedures—to assess the 

quality of performance data. Reliability refers to whether standard 

procedures for collecting and calculating results can be applied to 

measures so that they produce the same results if applied repeatedly to 

the same situation. Validity is the extent to which data adequately 

represent actual performance. Verification is the assessment of data 
Appendix IV: Certain Key Attributes 

Associated with Results-Oriented 

Management 

Page 47 GAO-04-919 Defense Management 

completeness, accuracy, consistency, timeliness, and related quality 

control practices. 

• Provides contextual information. Performance reporting systems 

should include information to help clarify aspects of performance that 

are difficult to quantify or to provide explanatory information such as 

factors that were within or outside the control of the agency. This 

information is critical to identifying and understanding the factors that 

contributed to a particular result and can help officials measure, 

assess, and evaluate the significance of underlying factors that may 

affect reported performance. In addition, this information can provide 

context for decision makers to establish funding priorities and adjust 

performance targets and assess means and strategies for accomplishing 

an organization’s goals and objectives. 
Appendix V: Comments from the Department 

of Defense 

Page 48 GAO-04-919 Defense Management 

Appendix V: Comments from the Department 

of Defense 
Appendix V: Comments from the Department 

of Defense 

Page 49 GAO-04-919 Defense Management 

Now on p. 36. 

Now on p. 36. 

Now on p. 36. 
Appendix V: Comments from the Department 

of Defense 

Page 50 GAO-04-919 Defense Management 

Now on p. 36. 

Now on p. 36. 
Appendix VI: Key Contact and Staff 

Acknowledgments 

Page 51 GAO-04-919 Defense Management 

Marilyn K. Wasleski (202) 512-8436 

Gabrielle M. Anderson, Krislin M. Bolling, Elizabeth H. Curda, 

Benjamin T. Licht, William A. McKelligott, Elizabeth G. Mead, 

Marjorie L. Pratt, and Robert K. Wild also made significant contributions 

to the report. 

Appendix VI: Key Contact and Staff 

Acknowledgments 

GAO Contact 

Acknowledgments 
Related GAO Products 

Page 52 GAO-04-919 Defense Management 

Future Years Defense Program: Actions Needed to Improve 

Transparency of DOD’s Projected Resource Needs. GAO-04-514. 

Washington, D.C.: May 7, 2004. 

Department of Defense: Further Actions Needed to Establish and 

Implement a Framework for Successful Business Transformation. 

GAO-04-626T. Washington, D.C.: March 31, 2004.

Results-Oriented Government: GPRA Has Established a Solid 

Foundation for Achieving Greater Results. GAO-04-38. Washington, D.C.: 

March 10, 2004. 

Satellite Communications: Strategic Approach Needed for DOD’s 

Procurement of Commercial Satellite Bandwidth. GAO-04-206. 

Washington, D.C.: December 10, 2003.

Defense Inventory: Several Actions Needed to Further DLA’s Efforts to 

Mitigate Shortages of Critical Spare Parts. GAO-03-709. Washington, 

D.C.: August 1, 2003. 

Results-Oriented Cultures: Creating a Clear Linkage between Individual 

Performance and Organizational Success. GAO-03-488. Washington, D.C.: 

March 14, 2003. 

Defense Management: New Management Reform Program Still Evolving. 

GAO-03-58. Washington, D.C.: December 12, 2002. 

Tax Administration: IRS Needs to Further Refine Its Tax Filing Season 

Performance Measures. GAO-03-143. Washington, D.C.: 

November 22, 2002. 

Performance Budgeting: Opportunities and Challenges. GAO-02-1106T. 

Washington, D.C.: September 19, 2002. 

Information Technology: Defense Information Systems Agency Can 

Improve Investment Planning and Management Controls. GAO-02-50. 

Washington, D.C.: March 15, 2002. 

Defense Management: Actions Needed to Sustain Reform Initiatives and 

Achieve Greater Results. GAO/NSIAD-00-72. Washington, D.C.: 

July 25, 2000. 

Related GAO Products 
Related GAO Products 

Page 53 GAO-04-919 Defense Management 

Management Reform: Elements of Successful Improvement Initiatives

GAO/T-GGD-00-26. Washington, D.C.: October 15, 1999. 

Performance Plans: Selected Approaches for Verification and Validation 

of Agency Performance Information. GAO/GGD-99-139. Washington, D.C.: 

July 30, 1999. 

The Results Act: An Evaluator’s Guide to Assessing Agency Annual 

Performance Plans. GAO/GGD-10.1.20. Washington, D.C.: April 1, 1998. 

Executive Guide: Effectively Implementing the Government Performance 

and Results Act. GAO/GGD-96-118. Washington, D.C.: June 1, 1996. 

(350410) 
The Government Accountability Office, the audit, evaluation and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and policies; 

and provides analyses, recommendations, and other assistance to help 

Congress make informed oversight, policy, and funding decisions. GAO’s 

commitment to good government is reflected in its core values of 

accountability, integrity, and reliability. 

The fastest and easiest way to obtain copies of GAO documents at no cost 

is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts 

newly released reports, testimony, and correspondence on its Web site. To 

have GAO e-mail you a list of newly posted products every afternoon, go 

to www.gao.gov and select “Subscribe to Updates.” 

The first copy of each printed report is free. Additional copies are $2 each. 

A check or money order should be made out to the Superintendent of 

Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 

more copies mailed to a single address are discounted 25 percent. Orders 

should be sent to: 

U.S. Government Accountability Office 

441 G Street NW, Room LM 

Washington, D.C. 20548 

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm 

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470 

Gloria Jarmon, Managing Director, JarmonG@gao.gov(202) 512-4400 

U.S. Government Accountability Office, 441 G Street NW, Room 7125 

Washington, D.C. 20548 

Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800 

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, D.C. 20548 

GAO’s Mission 

Obtaining Copies of 

GAO Reports and 

Testimony 

Order by Mail or Phone 

To Report Fraud, 

Waste, and Abuse in 

Federal Programs 

Congressional 

Relations 

Public Affairs 

PRINTED ON RECYCLED PAPER
United States

Government Accountability Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Service Requested

Presorted Standard

Postage & Fees Paid

GAO

Permit No. GI00