GAO 

United States Government Accountability Office

Report to the Subcommittee on 

Readiness and Management Support, 

Committee on Armed Services, U.S. 

Senate 

MILITARY 

READINESS 

DOD Needs to 

Strengthen 

Management and 

Oversight of the 

Defense Readiness 

Reporting System 

September 2009 

GAO-09-518 
What GAO Found

United States Government Accountability Office

Why GAO Did This Study

Highlights

Accountability Integrity Reliability

September 2009

MILITARY READINESS

DOD Needs to Strengthen Management and Oversight 

of the Defense Readiness Reporting System 

Highlights of GAO-09-518, a report to the 

Subcommittee on Readiness and 

Management Support, Committee on 

Armed Services, U.S. Senate 

The Department of Defense (DOD) 

reports data about the operational 

readiness of its forces. In 1999, 

Congress directed DOD to create a 

comprehensive readiness system 

with timely, objective, and accurate 

data. In response, DOD started to 

develop the Defense Readiness 

Reporting System (DRRS). After 7 

years, DOD has incrementally 

fielded some capabilities, and, 

through fiscal year 2008, reported 

obligating about $96.5 million. 

GAO was asked to review the 

program including the extent that 

DOD has (1) effectively managed 

and overseen DRRS acquisition and 

deployment and (2) implemented 

features of DRRS consistent with 

legislative requirements and DOD 

guidance. GAO compared DRRS 

acquisition disciplines, such as 

requirements development, test 

management, and DRRS oversight 

activities, to DOD and related 

guidance, and reviewed the 

system’s current and intended 

capabilities relative to legislative 

requirements and DOD guidance. 

We did not evaluate DOD’s overall 

ability to assess force readiness or 

the extent that readiness data 

reflects capabilities, vulnerabilities, 

or performance issues. 

What GAO Recommends

GAO is making recommendations 

to address the risks facing DOD in 

acquiring and developing DRRS 

and increase the chance of success. 

DOD agreed or partially agreed 

with three of our eight 

recommendations, and disagreed 

with the remaining five because it 

stated that it was already taking 

actions in these areas. 

DOD has not effectively managed and overseen the DRRS acquisition and 

deployment, in large part because of the absence of rigorous and disciplined 

acquisition management controls and an effective governance and 

accountability structure for the program. In particular, system requirements 

have not been effectively developed and managed. For example, user 

participation and input in the requirements development process was, until 

recently, limited, and requirements have been experiencing considerable 

change, are not yet stable, and have not been effectively controlled. In 

addition, system testing has not been adequately performed and managed. For 

example, test events for already acquired system increments, as well as 

currently deployed and operating increments, were not based on well-defined 

plans or structures, and test events have not been executed in accordance 

with plans or in a verifiable manner. Moreover, DRRS has not been guided by 

a reliable schedule of work to be performed and key activities to occur. These 

program management weaknesses can, in part, be attributed to long-standing 

limitations in program office staffing and program oversight and 

accountability. Despite being a DOD-wide program, until April, 2009 DRRS 

was not accountable to a DOD-wide oversight body, and it was not subject to 

DOD’s established mechanisms and processes for overseeing business 

systems. Collectively, these acquisition management weaknesses have 

contributed to a program that has fallen well short of expectations, and is 

unlikely to meet future expectations.

DOD has implemented DRRS features that allow users to report certain 

mission capabilities that were not reported under the legacy system, but these 

features are not fully consistent with legislative requirements and DOD 

guidance; and DOD has not yet implemented other features. The geographic 

combatant commands are currently reporting their capabilities to execute 

most of their operations and major war plans in DRRS, and DOD is reporting 

this additional information to Congress. However, because DRRS does not yet 

fully interface with legacy systems to allow single reporting of readiness data, 

the military services have not consistently used DRRS’s enhanced capability 

reporting features. For example, as of May 2009, the Army and Navy had 

developed interfaces for reporting in DRRS, while the Marine Corps required 

units to only report in their legacy system. Recently, the Marine Corps also 

began developing an interface and has done limited reporting in DRRS. In 

addition, DRRS has not fully addressed the challenges with metrics that led 

Congress to require a new readiness reporting system. DRRS metrics are less 

objective and precise, and no more timely than the legacy system metrics. 

Users have also noted that DRRS lacks some of the current and historical data 

and connectivity with DOD’s planning systems necessary to manage and 

deploy forces. Until these limitations are fully addressed, DRRS will not have 

the full complement of features necessary to meet legislative and DOD 

requirements, and users will need to rely on legacy reporting systems to 

support mission-critical decisions. View GAO-09-518 or key components.

For more information, contact Sharon Pickup 

at (202) 512-9619 or pickups@gao.gov, or 

Randolph C. Hite at (202) 512-3439 or 

hiter@gao.gov . 
Page i GAO-09-518 

Contents 

Letter 1

Background 4 

DOD Disagreed with GAO’s Prior Recommendation to Develop an 

Implementation Plan to Guide DRRS Development 10 

DOD Has Not Effectively Managed and Overseen the Acquisition 

and Deployment of DRRS 10 

Some DRRS Features Are Operational but Are Not Fully Consistent 

with Legislative Requirements and DOD Guidance 20 

Conclusions 30 

Recommendations for Executive Action 31 

Agency Comments and Our Evaluation 33 

Appendix I Objectives, Scope, and Methodology 42 

Appendix II Detailed Analysis of DRRS Acquisition and 

Deployment Management and Oversight 47 

 

Appendix III Comments from the Department of Defense 75 

Appendix IV GAO Contacts and Staff Acknowledgments 80 

 

Tables 

Table 1: DRRS Capability Modules 15 

Table 2: Organizations Interviewed during Our Review 46 

Table 3: DRRS Satisfaction of Nine Schedule-Estimating Key 

Practices 66 

Figures 

Figure 1: Air Force and Marine Corps Dual Reporting Requirements 

to Meet Readiness Reporting Guidelines 24 

Figure 2: Schedule for Developing and Implementing DRRS 

Capabilities 71 

Figure 3: Changes in Estimated Dates for DRRS Capabilities 72 

 Military Readiness 
Abbreviations 

CJCSI Chairman of the Joint Chiefs of Staff Instruction 

DIO DRRS Implementation Office 

DOD Department of Defense 

DRRS Defense Readiness Reporting System 

ESORTS Enhanced Status of Resources and Training System 

GSORTS Global Status of Resources and Training System 

JITC Joint Interoperability Test Command 

MAIS Major Automated Information System 

OSD Office of the Secretary of Defense 

SIPRNet Secure Internet Protocol Router Network 

TEMP Test and Evaluation Master Plan 

USD (P&R) Under Secretary of Defense for Personnel and 

Readiness 

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately. 

Page ii GAO-09-518 Military Readiness
Page 1 GAO-09-518 

 

United States Government Accountability Office

Washington, DC 20548 

 

September 25, 2009 

The Honorable Evan Bayh 

Chairman 

The Honorable Richard Burr 

Ranking Member 

Subcommittee on Readiness and Management Support 

Committee on Armed Services 

United States Senate 

To assess the ability of U.S. forces to execute the wartime missions for 

which they were designed, as well as other assigned missions, and to make 

decisions about deploying forces, the Department of Defense (DOD) relies 

heavily on readiness information derived from multiple information 

systems. Over the years, we and others have identified shortcomings in 

DOD’s readiness assessment and reporting, such as limitations in the 

completeness and precision of readiness data and a tendency to focus on 

examining the status of personnel, equipment, and other resources rather 

than broader capabilities. Congress addressed DOD’s readiness reporting 

in the Strom Thurmond National Defense Authorization Act for Fiscal Year 

19991

 by adding section 117 to Title 10 of the U.S. Code, directing the 

Secretary of Defense to establish a comprehensive readiness reporting 

system to measure, in an “objective, accurate, and timely manner,” the 

capability of the armed forces to carry out the National Security Strategy 

prescribed by the President, the defense planning guidance provided by 

the Secretary of Defense, and the National Military Strategy prescribed by 

the Chairman of the Joint Chiefs of Staff. 

In June 2002, the Deputy Secretary of Defense directed2

 the Under 

Secretary of Defense for Personnel and Readiness (USD P&R) to develop, 

field, maintain, and fund the Enhanced Status of Resources and Training 

System (ESORTS), which is the automated readiness reporting system 

1

Pub. L. No. 105-261, §373 (1998). Codified at 10 U.S.C. §117. 

2

Department of Defense Directive 7730.65, Department of Defense Readiness Reporting 

System (DRRS) (June 3, 2002). 

 Military Readiness 
 

within the Defense Readiness Reporting System (DRRS).3 He also directed 

that DRRS build upon existing processes and readiness assessment tools 

to establish a capabilities-based, adaptive, near-real-time readiness 

reporting system. In addition, in June 2004, the Secretary of Defense 

directed USD (P&R) to develop DRRS in a manner that would support the 

data requirements of various users of readiness information, such as the 

Chairman of the Joint Chiefs of Staff, the combatant commanders, the 

Secretaries of the military departments, and the Chief of the National 

Guard Bureau, including their requirements for data on the availability, 

readiness, deployment, and redeployment of forces.4

USD (P&R) established a DRRS Implementation Office (DIO) to manage 

the system’s acquisition, including managing system development and 

engaging the user community. The DIO has used support contractors to 

develop the system and reported obligating about $96.5 million for DRRS 

from fiscal year 2001 through fiscal year 2008. Some fielded system 

capabilities are currently being used to varying degrees by the user 

community. The DIO originally estimated that DRRS would achieve full 

operational capability in fiscal year 2007, but currently expects DRRS to 

reach full capability in 2014. In September 2008, the DIO projected that it 

would spend about $135 million through fiscal year 2014. 

Recognizing that DRRS was not yet fully deployed or operational and in 

light of our prior work on readiness-related issues, you asked us to review 

DOD’s efforts to develop and implement DRRS, including the program’s 

status, and the extent that DRRS addresses the challenges that led 

Congress to require a new system, such as the availability of information 

 

3

ESORTS is an automated readiness reporting tool designed to collect capability and 

resource data, while DRRS is the broader system that, according to the 2002 DRRS 

Directive, measures and reports on the readiness of military forces and the supporting 

infrastructure to meet missions and goals assigned by the Secretary of Defense. However, 

ESORTS is now viewed as a tool within DRRS. 

4

Secretary of Defense Memorandum, Policy Implementation to Establish Commander, 

USJFCOM (CDRUSJFCOM), as the Primary Joint Force Provider (JFP) (June 25, 2004). 

The U.S. military organizes its global presence into a series of geographic and functional 

combatant commands. The geographic combatant commands—U.S. Central Command, 

U.S. European Command, U.S. Northern Command, U.S. Pacific Command, U.S. Southern 

Command, and U.S. Africa Command —have authority over all U.S. military forces 

operating within a specified area of operation and are directly responsible for the 

performance of missions assigned to the command. The functional combatant 

commands—U.S. Joint Forces Command, U.S. Special Operations Command, U.S. Strategic 

Command, and U.S. Transportation Command—possess worldwide functional 

responsibilities, such as joint training, force provision, and global command and control. 

Page 2 GAO-09-518 Military Readiness
 

on capabilities, and the precision, timeliness, reliability, and objectivity of 

readiness metrics. In addressing these issues, we assessed the extent to 

which (1) DOD has effectively managed and overseen DRRS acquisition 

and deployment, and (2) features of DRRS have been implemented and are 

consistent with legislative requirements and DOD guidance. 

To determine the extent that DOD has effectively managed and overseen 

DRRS acquisition and deployment, we analyzed a range of program 

documentation and interviewed cognizant program and contractor 

officials relative to the following acquisition management disciplines: 

requirements development and management, test management, schedule 

reliability, and human capital. For each discipline, we compared key 

program documentation, such as a requirements management plan, test 

and evaluation master plans, and test reports to relevant DOD, federal, and 

related guidance, and we analyzed a statistical sample of individual 

requirements and test cases to determine consistency among them. In 

addition, we attended meetings of organizations established to monitor or 

govern DRRS development and reviewed information from meetings that 

we did not attend and interviewed officials associated with these 

meetings. 

To determine the extent to which the features of DRRS have been 

implemented and are consistent with legislative requirements and DOD 

guidance, we reviewed criteria such as the legislation that mandated a 

comprehensive DOD readiness reporting system, the DOD directive that 

established DRRS, program documentation and USD (P&R) guidance 

memorandums, DIO briefings to the readiness community, other 

departmental instructions, and directives and memorandums related to 

DRRS requirements and implementation. From these documents, we 

identified desired features of DRRS and compared them to documentary 

and testimonial evidence concerning system performance during meetings 

with a full range of officials responsible for developing and using the 

system. To obtain the developer’s perspective, on numerous occasions 

throughout our review we met with officials from USD (P&R), the DIO, 

and the three current DRRS contractors. To obtain user perspectives, we 

met with and surveyed by questionnaire officials from the Joint Staff, the 

geographic and functional combatant commands, and the services, and 

also met with officials from USD (P&R). We also attended meetings of 

organizations established to monitor or govern DRRS development and 

analyzed information from meetings that we did not attend. We also 

directly observed the system’s capabilities through our own limited use of 

the system and by observing others using the system. 

Page 3 GAO-09-518 Military Readiness
 

We did not evaluate the department’s overall ability to assess the readiness 

of its forces or the extent that data contained in any of its readiness 

reporting systems, including DRRS and GSORTS, reflect capabilities, 

deficiencies, vulnerabilities, or performance issues. Our review focused on 

acquisition and program management issues, such as requirements 

management, schedule and human capital requirements, the current usage 

of DRRS, and the extent to which DRRS’ features address legislative 

requirements and DOD guidance. 

We conducted this performance audit from April 2008 through August 

2009 in accordance with generally accepted government auditing 

standards. Those standards require that we plan and perform the audit to 

obtain sufficient, appropriate evidence to provide a reasonable basis for 

our findings and conclusions based on our audit objectives. We believe 

that the evidence obtained provides a reasonable basis for our findings 

and conclusions based on our audit objectives. Additional details on our 

scope and methodology are in appendix I. 

Historically, DOD has used its readiness assessment system to assess the 

ability of units and joint forces to fight and meet the demands of the 

national security strategy. DOD’s readiness assessment and reporting 

system is designed to assess and report on military readiness at three 

levels—(1) the unit level; (2) the joint force level; and (3) the aggregate, or 

strategic, level. Using information from its readiness assessment system, 

DOD prepares and sends legislatively mandated Quarterly Readiness 

Reports to Congress. DRRS is DOD’s new readiness reporting system that 

is intended to capture information from the previous system, as well as 

information about organizational capabilities to perform a wider variety of 

missions and mission essential tasks. DRRS is also intended to capture 

readiness information from defense agencies and installations, which were 

not required to report under the previous system. Some DRRS features are 

currently fielded and being used to varying degrees by the user 

community. 

Background 

DOD Collects a Wide 

Range of Readiness 

Information to Support 

Decision Makers Within 

and Outside DOD 

Laws, directives, and guidance, including a DOD directive, Chairman of 

the Joint Chiefs of Staff Instruction (CJCSI), Secretary of Defense and 

USD (P&R) memorandums, and service regulations and messages, show 

that readiness information and data are needed to support a wide range of 

decision makers. These users of readiness data include Congress, the 

Secretary of Defense, the Chairman of the Joint Chiefs of Staff, the 

Page 4 GAO-09-518 Military Readiness
 

combatant commanders, the Secretaries of the military departments, and 

the Chief of the National Guard Bureau. 

The directives and guidance also list roles and responsibilities for 

collecting and reporting various types of readiness data. For example, 

CJCSI 3401.02A5

 assigns the service chiefs responsibility for ensuring 

required global status of resources and training system (GSORTS) reports 

are submitted. GSORTS is DOD’s legacy, resource-based readiness 

reporting system that provides a broad assessment of unit statuses based 

on units’ abilities to execute the missions for which they were organized 

or designed as well as the current missions for which they may be 

employed. The information in the required GSORTS reports includes units’ 

abilities to execute the missions for which they were organized or 

designed, as well as the status of their training, personnel, and equipment.6

In addition, DOD directive 7730.65, which established DRRS as DOD’s new 

readiness reporting system, assigns the Secretaries of the military 

departments and the commanders of the combatant commands 

responsibilities for developing mission essential tasks for all of their 

assigned missions.7

Requirements and 

Intended Characteristics of 

DOD’s New Readiness 

Reporting System 

Prior to 1999, we identified challenges with DOD’s existing readiness 

reporting system, GSORTS, and in 1999, Congress directed the Secretary 

of Defense to establish a comprehensive readiness reporting system.8

 The 

legislation requires the system to measure in an objective, accurate, and 

timely manner the capability of the armed forces to carry out (1) the 

National Security Strategy prescribed by the President, (2) the defense 

 

5

Chairman of the Joint Chiefs of Staff Instruction 3401.02A, Global Status of Resources and 

Training System (GSORTS) (Feb. 27, 2004). 

6

All combat, combat support, and combat service support units that have the potential to 

support an: operations, contingency, or single integrated operational plan; or a contingency 

operation are required to report. 

7

A mission is a task or a series of tasks with a purpose. Mission essential tasks are those 

tasks that are absolutely necessary, indispensable, or critical to mission success. DRRS’ 

capability data are based on commanders’ assessments of their organizations’ capabilities 

to carry out their missions and mission essential tasks. 

8

The Strom Thurmond National Defense Authorization Act for Fiscal Year 1999, Pub. L. No. 

105-261 §373 (1998), codified at 10 U.S.C. §117. Section 373 initially directed the Secretary 

to establish and implement the readiness reporting system no later than January 15, 2000. 

An amendment in the National Defense Authorization Act for Fiscal Year 2000, Pub. L. No. 

106-65, §361 changed the date from January 15, 2000, to April 1, 2000. 

Page 5 GAO-09-518 Military Readiness
 

planning guidance provided by the Secretary of Defense, and (3) the 

National Military Strategy prescribed by the Chairman of the Joint Chiefs 

of Staff. 

To address the requirements established by Congress, the Office of the 

Deputy Under Secretary of Defense (Readiness) began in 2001 to build 

consensus among DOD’s senior readiness leaders for an improved 

readiness assessment system. For example, the Deputy’s office distributed 

a list of key characteristics of the improved readiness assessment system 

to the leaders in advance of scheduled meetings. The system’s key desired 

characteristics included allowing near-real-time access to readiness data 

and trends, enabling rapid, low-cost development using classified Internet 

technology, and reducing the reporting burdens on people. Since then 

various directives and memorandums have been issued regarding DRRS 

responsibilities, requirements, and related issues. For example: 

• On June 3, 2002, the Deputy Secretary of Defense established DOD’s 

new readiness reporting system, as directed by Congress, by signing 

DOD Directive 7730.65. According to this directive, DRRS is intended 

to build upon DOD’s existing processes and readiness assessment tools 

to establish a capabilities-based, near-real-time readiness reporting 

system. The DRRS directive assigned USD (P&R) responsibilities for 

developing, fielding, maintaining, and funding ESORTS (the tool to 

collect capability, resource, and training information) and overseeing 

DRRS to ensure accuracy, completeness, and timeliness of its 

information and data, its responsiveness, and its effective and efficient 

use of modern practices and technologies. In addition, the USD P&R is 

responsible for ensuring that ESORTS information, where appropriate, 

is integrated into DOD’s planning systems and processes. The directive 

also states that until ESORTS becomes fully operational, the Chairman 

of the Joint Chiefs of Staff shall maintain the GSORTS database. 

• On June 25, 2004, the Secretary of Defense issued a memorandum, 

which directed USD (P&R) to develop DRRS to support data 

requirements identified by the Chairman of the Joint Chiefs of Staff, the 

combatant commanders, the Secretaries of the Military Departments, 

and the Chief, National Guard Bureau to include availability, readiness, 

deployment, and redeployment data.9

 

9

Secretary of Defense Memorandum, Policy Implementation to Establish Commander, 

USJFCOM (CDRUSJFCOM), as the Primary Joint Force Provider (JFP), (June 25, 2004). 

The memorandum’s primary purpose was to direct the Commander of the U.S. Joint Forces 

Command to assume the duties of DOD’s primary joint force provider. 

Page 6 GAO-09-518 Military Readiness
 

• On November 2, 2004, USD (P&R) issued a DRRS interim 

implementation guidance memorandum.10 In this memorandum, the 

undersecretary noted that he had established a DIO to provide 

reporting assistance for units. The memorandum also stated that 

combatant commanders would begin reporting readiness by mission 

essential tasks by November 30, 2004. The memorandum also directed 

the services to develop detailed implementing guidance for reporting 

and assessing mission essential task readiness in ESORTS within their 

respective services, and set a goal for the services to implement the 

mission essential task reporting process by September 30, 2005. To 

meet these mission essential task reporting requirements, USD (P&R) 

directed commanders to rate their organizational capabilities as (1) yes 

or “Y”, (2) qualified yes or “Q”, or (3) no or “N.” A “Y” indicates that an 

organization can accomplish the rated tasks or missions to prescribed 

standards and conditions in a specified environment. It should reflect 

demonstrated performance in training or operations. A “Q” indicates 

that performance has not been demonstrated, and, although data may 

not readily support a “Y,” the commander believes the organization can 

accomplish the rated task or mission to standard under most 

conditions. An “N” indicates that an organization cannot accomplish 

the rated task or mission to prescribed standards in the specified 

environment at the time of the assessment. 

• The November 2004 memorandum also stated that the expected 

transition from GSORTS to ESORTS was scheduled to begin in fiscal 

year 2005. According to the 2004 memorandum, the ESORTS module of 

DRRS would provide, among other things, visibility of the latest 

GSORTS information reported by units, and detailed resource 

information from authoritative data sources with the capability to 

aggregate or separate the data. This memorandum signaled a change in 

program direction. Although the 2002 DOD directive stated that DRRS 

is intended to build upon DOD’s existing processes and readiness 

assessment tools, the 2004 memorandum indicated that DRRS was to 

replace GSORTS, as the ESORTS module of DRRS captured both 

capabilities and resource data. 

 

10Under Secretary of Defense for Personnel and Readiness Memorandum, Department of 

Defense Readiness Reporting System (DRRS) Interim Implementation Guidance (Nov. 2, 

2004). USD (P&R) issued three subsequent memorandums between August 10, 2005, and 

August 23, 2006, which expanded and clarified reporting requirements. 

Page 7 GAO-09-518 Military Readiness
 

Since its establishment, the DIO has operated within the Office of USD 

(P&R) and has relied on multiple contractors. To provide governance of 

DRRS, and enhance communication between the development community, 

represented by the DIO and contractors, and the user community, which 

includes the Joint Staff, military services, and combatant commands, USD 

(P&R) established various bodies with representatives from the user 

community, including military services, combatant commands, and the 

defense agencies. Representatives from the Office of USD (P&R) and the 

Joint Staff currently serve as cochairs of the various bodies. DRRS Battle 

Staffs comprise colonels, Navy captains, and similar-graded civilians. They 

track DRRS development and identify issues with the system. At the onestar level, the DRRS General and Flag Officer Steering Committee 

discusses issues raised by the Battle Staff. In December 2007, USD (P&R) 

created a committee at the three-star level, referred to as the DRRS 

Executive Committee. Its charter, finalized about a year later in January 

2009, calls for the committee to review and approve proposals and plans to 

establish policy, processes, and system requirements for DRRS, including 

approving software development milestones required to reach objectives. 

Overview of DRRS 

Program Management and 

Oversight Structure 

To ensure that leadership is provided for the direction, oversight, and 

execution of DOD’s business transformation efforts, including business 

systems modernization efforts such as DRRS, DOD relies on several 

entities. These entities include the Defense Business Systems Management 

Committee, which is chaired by the Deputy Secretary of Defense and 

serves as the department’s highest-ranking governance body and the 

approval authority for business systems modernization activities; the 

Investment Review Boards, which are chartered by the Principal Staff 

Assistants—senior leaders from various offices within DOD—and serve as 

the review and certification bodies for business system investments in 

their respective areas of responsibility;11 and the Business Transformation 

Agency, which is responsible for supporting the Investment Review 

Boards and for leading and coordinating business transformation efforts 

across the department. Among other things, the Business Transformation 

Agency supports the Office of the Under Secretary of Defense, Acquisition, 

 

11The four Investment Review Boards are for (1) Financial Management, established by the 

Deputy Under Secretary of Defense for Financial Management; (2) Weapon Systems 

Lifecycle Management and Materiel Supply and Services Management; (3) Real Property 

and Installations Lifecycle Management, both established by the Under Secretary of 

Defense for Acquisition, Technology and Logistics; and (4) Human Resources Management, 

established by USD (P&R). 

Page 8 GAO-09-518 Military Readiness
 

Technology and Logistics in conducting system acquisition risk 

assessments.12

Disciplined System 

Acquisition and Oversight 

Are Keys to Program 

Success 

Our research and evaluations of information technology programs, 

including business systems modernization efforts within DOD, have shown 

that delivering promised system capabilities and benefits on time and 

within budget largely depends on the extent to which key program 

management disciplines are employed by an adequately staffed program 

management office. Among other things, these disciplines include a 

number of practices associated with effectively developing and managing 

system requirements, adequately testing system capabilities, and reliably 

scheduling the work to be performed. They also include proactively 

managing the program office’s human capital needs, and promoting 

program office accountability through executive-level program oversight. 

DOD acquisition policies and guidance, along with other relevant 

guidance, recognize the importance of these management and oversight 

disciplines.13 As we have previously reported, not employing these and 

other program management disciplines increases the risk that system 

acquisitions will not perform as intended and require expensive and timeconsuming rework. 

 

12The purpose of a risk assessment is to reduce systemic risk and support informed 

decision making by focusing on delivering business capabilities rapidly and at a reduced 

cost, and identifying program vulnerabilities and assisting in developing mitigation 

solutions. The assessment is generally performed prior to major acquisition decisions, 

although assessments may be requested for other reasons. 

13See for example, Department of Defense Instruction 5000.02, Operation of the Defense 

Acquisition System (Dec. 8, 2008); Defense Acquisition University, Test and Evaluation 

Management Guide, 5th ed. (Fort Belvoir, Va.: January 2005); Institute of Electrical and 

Electronics Engineers, Standard 1012-2004 for Software Verification and Validation 

(New York, N.Y.: June 8, 2005); GAO, GAO Cost Estimating and Assessment Guide: Best 

Practices for Developing and Managing Capital Program Costs, GAO-09-3SP 

(Washington, D.C.: March 2009); and GAO, A Model of Strategic Human Capital 

Management, GAO-02-373SP (Washington, D.C.: Mar. 15, 2002). 

Page 9 GAO-09-518 Military Readiness
 

In 2003, we reported that, according to USD (P&R) officials, DRRS was a 

large endeavor, and that development would be challenging and require 

buy-in from many users.14 We also reported that the program was only a 

concept without detailed plans to guide its development and 

implementation. Based on the status of the program at that time and 

DOD’s past record on readiness reporting initiatives, we recommended 

that the Secretary of Defense direct the Office of USD (P&R) to develop an 

implementation plan that identified 

• performance goals that are objective, quantifiable, and measurable; 

• performance indicators to measure outcomes; 

• an evaluation plan to compare program results with established goals; 

and 

• milestones to guide DRRS development to the planned 2007 full 

capability date. 

DOD did not agree with our recommendation, stating that it had 

established milestones, cost estimates, functional responsibilities, 

expected outcomes, and detailed plans for specific information technology 

requirements and progress indicators. In evaluating the DOD comments, 

we noted that DOD had established only two milestones—initial capability 

in 2004 and full capability in 2007—and did not have a road map explaining 

the steps needed to achieve full capability by 2007.15

DOD has not effectively managed and overseen the acquisition and 

deployment of DRRS in accordance with a number of key program 

management disciplines that are recognized in DOD acquisition policies 

and guidance, along with other relevant guidance, and are fundamental to 

delivering a system that performs as intended on time and within budget. 

In particular, DRRS requirements have not been effectively developed and 

managed, and DRRS testing has not been adequately performed and 

managed. Further, DRRS has not been guided by a reliable schedule of the 

DOD Disagreed with 

GAO’s Prior 

Recommendation to 

Develop an 

Implementation Plan 

to Guide DRRS 

Development 

DOD Has Not 

Effectively Managed 

and Overseen the 

Acquisition and 

Deployment of DRRS 

 

14The users of readiness data include the Secretary of Defense, the military services, the 

Chairman of the Joint Chiefs of Staff, the combatant commands, defense agencies, and 

field activities. 

15GAO, Military Readiness: New Reporting System Is Intended to Address Long-Standing 

Problems, but Better Planning Is Needed, GAO-03-456 (Washington, D.C.: Mar. 28, 2003). 

Page 10 GAO-09-518 Military Readiness
 

work needed to be performed and the key activities and events that need 

to occur. These program management weaknesses can be attributed in 

part to long-standing limitations in program office staffing and oversight. 

As a result, the program has not lived up to the requirements set for it by 

Congress, and the department has not received value from the program 

that is commensurate with the time and money invested—about 7 years 

and $96.5 million. Each of these weaknesses are summarized below and 

discussed in detail in appendix II. 

DRRS Requirements Have 

Not Been Effectively 

Developed and Managed 

According to DOD and other relevant guidance, effective requirements 

development and management includes, among other things, (1) 

effectively eliciting user needs early and continuously in the system lifecycle process, (2) establishing a stable baseline set of requirements and 

placing the baseline under configuration management, (3) ensuring that 

system requirements are traceable backward to higher level business or 

operational requirements (e.g., concept of operations) and forward to 

system design documents (e.g., software requirements specification) and 

test plans, and (4) controlling changes to baseline requirements. However, 

none of these conditions have been met on DRRS. Specifically, key users 

have only recently become fully engaged in developing requirements, and 

requirements have been experiencing considerable change and are not yet 

stable. Further, different levels of requirements and related test cases have 

not been aligned with one another, and changes to requirements have not 

been effectively controlled. As a result, efforts to develop and deliver 

initial DRRS capabilities have taken longer than envisioned and these 

capabilities have not lived up to user expectations. These failures increase 

the risk of future DRRS capabilities not meeting expectations and increase 

the likelihood that expensive and time-consuming system rework will be 

necessary. 

Until recently, key users were not fully or effectively engaged in DRRS 

requirements development and management. One of the leading practices 

associated with effective requirements development is engaging system 

users early and continuously in the process of defining requirements. 

However, DIO officials and representatives from the military services and 

the Joint Staff agree that until recently, key users were not effectively 

engaged in DRRS requirements development and management, although 

they disagree at to why user involvement has suffered. Regardless, DRRS 

Executive Committee direction has improved the situation. Specifically, in 

January 2008, the committee directed the Joint Staff to conduct an analysis 

of DRRS capabilities, referred to as the “capabilities gap analysis,” which 

involved the entire readiness community and resulted in 530 additional 

Recent Actions Have Been 

Taken to Address Limited User 

Involvement in Developing and 

Managing Requirements 

Page 11 GAO-09-518 Military Readiness
 

user requirements. In our view, this analysis is a positive step in 

addressing long-standing limited involvement by key DRRS users in 

defining requirements that has contributed to significant delays in the 

program, as discussed later in the report. 

As of April 2009, DRRS requirements continued to be in a state of flux. 

Establishing an authoritative set of baseline requirements prior to system 

design and development provides a stable basis for designing, developing, 

and delivering a system that meets its users’ operational needs. However, 

the fact that these 530 user requirements have recently been identified 

means that the suite of requirements documentation associated with the 

system, such as the detailed system requirements, will need to change and 

thus is not stable. To illustrate, these 530 requirements have not been fully 

evaluated by the DIO and the DRRS governance boards and according to 

program officials, have not yet been approved, and thus their impact on 

the program is not clear. Compounding this instability in the DRRS 

requirements is the fact that additional changes are envisioned. According 

to program officials, the changes resulting from the gap analysis and 

reflected in the latest version of the DRRS Concept of Operations, which 

was approved by the DRRS Executive Committee in January 2009, have 

yet to be reflected in other requirements documents, such as the detailed 

system requirements. Although defining and developing requirements is 

inherently an iterative process, having a baseline set of requirements that 

are stable is a prerequisite to effective and efficient system design and 

development. Without them, the DIO has not been able to deliver a system 

that meets user needs on time, and it is unlikely that future development 

and deployment efforts will produce better results. 

DRRS Requirements Are Not 

Stable 

During our review, DIO officials could not demonstrate that requirements 

and related system design and testing artifacts are properly aligned. One of 

the leading practices associated with developing and managing 

requirements is maintaining bidirectional traceability from high-level 

operational requirements through detailed lower-level requirements and 

design documents to test cases. We attempted on three separate occasions 

to verify the traceability of system requirements backwards to higher-level 

requirements and forward to lower-level software specifications and test 

cases, and each time we found that traceability did not exist. DIO and 

contractor officials attributed the absence of adequate requirements 

traceability to the ongoing instability in requirements and efforts to update 

program documentation. Without traceable requirements, the DIO does 

not have a sufficient basis for knowing that the scope of the design, 

development, and testing efforts will produce a system solution on time 

and on budget and that will meet users’ operational needs and perform as 

Alignment among 

Requirements and Related 

System Design and Testing 

Artifacts Has Not Been 

Demonstrated 

Page 12 GAO-09-518 Military Readiness
 

intended. As a result, the risk is significant that expensive and timeconsuming system rework will be required. 

Since the inception of the program in 2002, DRRS has been developed and 

managed without a formally documented and approved process for 

managing changes to system requirements. Adopting a disciplined process 

for reviewing and accepting changes to an approved baseline set of 

requirements in light of the estimated costs, benefits, and risk of each 

proposed change is a recognized best practice. However, requirements 

management and change-control plans developed in 2006 by the DRRS 

software development contractor, according to DIO officials, were not 

adequate. To address this, the Joint Staff developed what it referred to as a 

conceptual requirements change-control process in February 2008, as a 

basis for the DIO to develop more detailed plans that could be 

implemented. In January 2009, the DIO drafted more detailed requirements 

management and configuration management plans, the latter of which the 

DIO updated in March 2009. However, the plans have yet to be approved 

and implemented. Until the DIO effectively controls requirements changes, 

it increases the risk of needed DRRS capabilities taking longer and costing 

more to deliver than necessary. 

Changes to Requirements Are 

Not Being Effectively 

Controlled 

DRRS Testing Has Not 

Been Adequately 

Performed and Key Test 

Management Structures 

and Controls Have Not 

Been Established 

According to DOD and other relevant guidance, system testing should be 

progressive, meaning that it should consist of a series of test events that 

first focus on the performance of individual system components, then on 

the performance of integrated system components, followed by systemlevel tests that focus on whether the system (or major system increments) 

are acceptable, interoperable with related systems, and operationally 

suitable to users. For this series of related test events to be conducted 

effectively, each test event needs to be executed in accordance with welldefined test plans, the results of each test event need to be captured and 

used to ensure that problems discovered are disclosed and corrected, and 

all test events need to be governed by a well-defined test management 

structure. 

However, the DIO cannot demonstrate that it has adequately tested any of 

the DRRS increments, referred to as system releases and subreleases, even 

though it has already acquired and partially deployed a subset of these 

increments. Moreover, the DIO has yet to establish the test management 

structures and controls needed to effectively execute DRRS testing going 

forward. More specifically, the test events for already acquired, as well as 

currently deployed and operating, DRRS releases and subreleases were 

not based on well-defined plans. For example, the test plan did not include 

Page 13 GAO-09-518 Military Readiness
 

a schedule of activities to be performed or defined roles and 

responsibilities for performing them. Also, the test plan did not 

consistently include test entrance and exit criteria, a test defect 

management process, and metrics for measuring progress. Further, test 

events have not been fully executed in accordance with plans, or executed 

in a verifiable manner, or both. For example, although increments of 

DRRS functionality have been put into production, the DIO has not 

performed system integration testing, system acceptance testing, or 

operational testing on any DRRS release or subrelease. Moreover, the 

results of all executed test events have not been captured and used to 

ensure that problems discovered were disclosed to decision makers, and 

ultimately corrected. For example, the DIO has not captured the test 

results for at least 20 out of 63 DRRS subreleases. Test results that were 

captured did not include key elements, such as entrance/exit criteria 

status and unresolved defects and applicable resolution plans. 

The DIO has also not established an effective test management structure 

to include, for example, a clear assignment of test management roles and 

responsibilities, or a reliable schedule of planned test events. 

Compounding this absence of test management structures and controls is 

the fact that the DIO has yet to define how the development and testing to 

date of a series of system increments (system releases and subreleases) 

relate to the planned development and testing of the 10 system modules 

established in January 2009. (See table 1 for a list and description of these 

modules.) Collectively, this means that it is unlikely, that already 

developed and deployed DRRS increments can perform as intended and 

meet user operational needs. Equally doubtful are the chances that the 

DIO can adequately ensure that yet-to-be developed DRRS increments will 

meet expectations. 

Page 14 GAO-09-518 Military Readiness
 

Table 1: DRRS Capability Modules 

Module Definition 

1. Joint Mission Readiness Enables selected users to see and assess the highest 

strategic-level “joint” readiness data. 

2. Unit Mission Readiness Captures unit-level readiness data, as well as 

authoritative resources data (e.g., personnel, 

equipment) fed from external systems owned by military 

services. 

3. Asset Visibility Allows authoritative resources data from military 

services to be provided in near-real-time, and certifies 

them. 

4. Business Intelligence Provides the ability to query and analyze readiness and 

asset visibility data, based on the desires and needs of 

the user. 

5. Installation Readiness Allows readiness reporting by installations, 

training/firing ranges and other infrastructure facilities. 

6. Readiness Reviews Enables forcewide readiness reviews to be conducted, 

such as the Joint Combat Capability Assessment 

review process. 

7. Planning/Execution 

Support 

Allows DRRS to interact with other planning systems 

and processes, such as the Global Force Management 

and Adaptive Planning and Execution communities. 

8. Historical Data Focuses on the historical collection and presentation of 

information. Also includes the Continuity of Operations 

(COOP) capability. 

9. System Architecture Meets the underlying information technology system 

specifications, such as standards for information 

assurance, interoperability, bandwidth, and other 

issues. 

10. End-User Usability Fulfills end-user support of the DRRS application to 

include training, customer support, and documentation. 

Source: DOD. 

DRRS Has Not Been 

Guided by a Reliable 

Schedule 

The success of any program depends in part on having a reliable schedule 

that defines, among other things, when work activities will occur, how 

long they will take, and how they are related to one another. From its 

inception in 2002 until November 2008, the DIO did not have an integrated 

master schedule, and thus has long been allowed to proceed without a 

basis for executing the program and measuring its progress. In fact, the 

only milestone that we could identify for the program prior to November 

2008 was the date that DRRS was to achieve full operational capability, 

which was originally estimated to occur in fiscal year 2007, but later 

slipped to fiscal year 2008 and then fiscal year 2011, and is now fiscal year 

2014--a 7-year delay. 

Page 15 GAO-09-518 Military Readiness
 

Moreover, the DRRS integrated master schedule that was first developed 

in November 2008, and was updated in January 2009 and again in April 

2009 to address limitations that we identified and shared with the program 

office, is still not reliable. Specifically, our research has identified nine 

practices associated with developing and maintaining a reliable schedule.16

These practices are (1) capturing all key activities, (2) sequencing all key 

activities, (3) assigning resources to all key activities, (4) integrating all 

key activities horizontally and vertically, (5) establishing the duration of all 

key activities, (6) establishing the critical path for all key activities, (7) 

identifying float between key activities, (8) conducting a schedule risk 

analysis, and (9) updating the schedule using logic and durations to 

determine the dates for all key activities.17 The program’s latest integrated 

master schedule does not address three of the practices and only partially 

addresses the remaining six. For example, the schedule does not establish 

a critical path for all key activities, nor does it include a schedule risk 

analysis, and it is not being updated using logic and durations to determine 

the dates for all key activities. In addition, the schedule introduces 

considerable concurrency across key activities and events for several 

modules, which introduces increased risk. These limitations in the 

program’s latest integrated master schedule, coupled with the program’s 7-

year slippage to date and continued requirements instability, make it likely 

that DRRS will incur further delays. 

DIO Lacks Adequate 

Staffing and an Effective 

Approach to Meeting Its 

Human Capital Needs 

The DIO does not currently have adequate staff to fulfill its system 

acquisition and deployment responsibilities, and it has not managed its 

staffing needs in an effective manner. Effective human capital 

management should include an assessment of the core competencies and 

essential knowledge, skills, and abilities needed to perform key program 

management functions, an inventory of the program’s existing workforce 

capabilities, an analysis of the gap between the assessed needs and the 

existing capabilities, and plans for filling identified gaps. DIO performs a 

number of fundamental DRRS program management functions, such as 

acquisition planning, performance management, requirements 

development and management, test management, contractor tracking and 

oversight, quality management, and configuration management. To 

effectively perform such functions, program offices, such as the DIO, need 

 

16GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and 

Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: March 2009). 

17Float is the amount of time an activity can slip before affecting the critical path. 

Page 16 GAO-09-518 Military Readiness
 

to have not only well-defined policies and procedures and support tools 

for each of these functions, but also sufficient human capital to implement 

the processes and use the tools throughout the program’s life cycle. 

However, the DIO is staffed with only a single full-time government 

employee—the DIO Director. All other key program office functions are 

staffed by either contractor staff or staff temporarily detailed, on an asneeded basis, from other DOD organizations. In addition, key positions, 

such as performance manager and test manager, have either not been 

established or are vacant. According to DIO and contractor officials, they 

recognize that additional program management staffing is needed but 

stated that requests for additional staff had not been approved by USD 

(P&R) due to competing demands for staffing. Further, they stated that the 

requests were not based on an assessment of the program’s human capital 

needs and the gap between these needs and its onboard workforce 

capabilities. Until DIO adopts a strategic, proactive approach to managing 

its human capital needs, it is unlikely that it will have an adequate basis for 

obtaining the people it needs to effectively and efficiently manage DRRS. 

DOD Executive Oversight 

of DRRS Has Been Limited 

A key principle for acquiring and deploying system investments is to 

establish a senior-level governance body to oversee the investment and 

hold program management accountable for meeting cost, schedule, and 

performance commitments. Moreover, for investments that are 

organization wide in scope and introduce new ways of doing business, like 

DRRS, the membership of this oversight body should represent all 

stakeholders and have sufficient organizational seniority to commit their 

respective organizations to any decisions reached. For significant system 

investments, the department’s acquisition process provides for such 

executive governance bodies. For example, Major Automated Information 

Systems, which are investments over certain dollar thresholds or that are 

designated as special interest because of, among other things, their 

mission importance, are reviewed at major milestones by a designated 

milestone decision authority.18 These authorities are supported by a senior 

advisory group, known as the Information Technology Acquisition Board, 

which comprises senior officials from the Joint Staff, the military 

departments, and staff offices within the Office of the Secretary of 

 

18A Major Automated Information System is a program or initiative that is so designated by 

the Assistant Secretary of Defense (Networks and Information Integration) / Chief 

Information Officer or that is estimated to require program costs in any single year in 

excess of $32 million, total program costs in excess of $126 million, or total life-cycle costs 

in excess of $378 million in fiscal year 2000 constant dollars. 

Page 17 GAO-09-518 Military Readiness
 

Defense. In addition, all business system investments in DOD that involve 

more than $1 million in obligations are subject to review and approval by a 

hierarchy of DOD investment review boards that comprise senior DOD 

leaders, including the Defense Business Systems Management Committee, 

which is chaired by the Deputy Secretary of Defense. Through these 

executive oversight bodies and their associated processes, programs are to 

be, among other things, governed according to corporate needs and 

priorities, and program offices are to be held accountable for meeting cost, 

schedule, and performance expectations. 

Until April 2009, DRRS was not subject to any of DOD’s established 

mechanisms and processes for overseeing information technology 

systems. As previously discussed, USD (P&R) established the DRRS Battle 

Staff, which is a group of midlevel military officers and civilians from 

DRRS stakeholder organizations, and it established a higher-ranked 

General and Flag Officer Steering Committee, consisting of stakeholder 

representatives. However, neither of these entities had specific oversight 

responsibilities or decision-making authority for DRRS. Moreover, neither 

was responsible for holding the program office accountable for results. 

According to meeting minutes and knowledgeable officials, these entities 

met on an irregular basis over the last several years, with as much as a 1-

year gap in meeting time for one of them, to discuss DRRS status and 

related issues. 

In December 2007, USD (P&R) recognized the need for a more senior-level 

and formal governance body, and established the DRRS Executive 

Committee. Since January 2008, this committee, which consists of toplevel representatives from stakeholder organizations, has met at least 

seven times. In January 2009, the DRRS Executive Committee’s charter 

was approved by the Deputy Under Secretary of Defense (Readiness) and 

the three-star Director of the Joint Staff. According to the charter, the 

committee is to review and approve proposals and plans to establish 

policy, processes, and system requirements for DRRS, including approving 

software development milestones required to reach objectives. Consistent 

with its charter, the committee has thus far made various program-related 

decisions, including approving a DRRS concept of operations to better 

inform requirements development, and directing the Joint Staff to conduct 

an analysis to identify any gaps between DRRS requirements and user 

needs. However, the committee has not addressed the full range of 

acquisition management weaknesses previously discussed in this report, 

and it has not taken steps to ensure that the program office is accountable 

for well-defined program baseline requirements. 

Page 18 GAO-09-518 Military Readiness
 

More recently, the DOD Human Resources Management Investment 

Review Board and the Defense Business Systems Management Committee 

reviewed DRRS and certified and approved, respectively, the program to 

invest $24.625 million in fiscal years 2009 and 2010. These entities 

comprise senior leadership from across the department, including the 

Deputy Secretary of Defense as the Defense Business Systems 

Management Committee Chair, military service secretaries, the defense 

agency heads, principal staff assistants, and representatives from the Joint 

Staff and combatant commands. However, neither the Investment Review 

Board’s certification nor the Defense Business Systems Management 

Committee’s approval was based on complete and accurate information 

from USD (P&R). Specifically, the certification package submitted to both 

oversight bodies by the USD (P&R) precertification authority (Office of 

Readiness Programming and Assessment) stated that DRRS was on track 

for meeting its cost, schedule, and performance measures and highlighted 

no program risks despite the weaknesses discussed in this report. 

According to the chairwoman of the Investment Review Board, the board 

does not have a process or the resources to validate the information 

received from the programs that it reviews.19 Moreover, the chairwoman 

stated that program officials did not make the board aware of the results 

of our review that we shared with the DIO prior to either the Investment 

Review Board or Defense Business Systems Management Committee 

reviews. Since we briefed the chairwoman, the Investment Review Board 

has requested that the DIO provide it with additional information 

documenting DRRS compliance with applicable DOD regulations and 

statutes. 

According to USD (P&R) and DIO officials, DRRS was not subject to 

department executive-level oversight for almost 6 years because, among 

other things, they did not consider DRRS to be a more complex 

information technology system. Furthermore, because of the nature of the 

authority provided to the USD (P&R) in the DRRS charter, they did not 

believe it was necessary to apply the same type of oversight to DRRS as 

other information systems within DOD. This absence of effective oversight 

has contributed to a void in program accountability and limited prospects 

for program success. 

 

19As previously noted, the Investment Review Board is responsible for all business system 

investments in DOD that involve more than $1 million in obligations. 

Page 19 GAO-09-518 Military Readiness
 

DOD has implemented DRRS features that allow users to report certain 

mission capabilities that were not reported under the legacy system, but 

these features are not fully consistent with legislative requirements and 

DOD guidance; and DOD has not yet implemented other envisioned 

features of the system. While some users are consistently reporting 

enhanced capability information, reporting from other users has been 

inconsistent. In addition, DRRS has not fully addressed the challenges 

with metrics that were identified prior to 1999 when Congress required 

DOD to establish a new readiness reporting system. Users have also noted 

that DRRS lacks some of the current and historical data and connectivity 

with DOD’s planning systems necessary to manage and deploy forces. 

Some DRRS Features 

Are Operational but 

Are Not Fully 

Consistent with 

Legislative 

Requirements and 

DOD Guidance 

DOD Is Providing 

Congress with DRRS 

Capability Data from the 

Combatant Commands, 

but Services Have Not 

Consistently Reported 

Capability Data 

The geographic combatant commands are capturing enhanced capability 

data in DRRS, and DOD’s quarterly readiness reports to Congress 

currently contain this information, as well as information that is drawn 

from DOD’s legacy readiness reporting system, GSORTS. However, the 

military services have not consistently used the enhanced capability 

reporting features of DRRS. Because DRRS does not yet fully interface 

with legacy systems to allow single reporting of readiness data, the Army 

and Navy developed additional system interfaces and are reporting in 

DRRS. Until May 2009, the Marine Corps directed its units to report only in 

the legacy system to avoid the burden of dual reporting. The Air Force 

chose not to develop an interface and instructed its units to report in both 

DRRS and the legacy system. 

DRRS and GSORTS both contain capabilities information and resource 

(numbers of personnel, equipment availability, and equipment condition) 

and training data. However, DRRS currently provides more capabilities 

data than GSORTS. When Congress directed DOD to establish a new 

readiness reporting system, GSORTS was already providing capability 

information concerning unit capabilities to perform the missions for which 

they were organized or designed.20 More recently, some of the military 

services began reporting limited capability information on unit capabilities 

to perform missions other than those that they were organized or designed 

 

20The primary GSORTS metric—the “C” rating—is a mission-focused metric that is based 

not only on a unit’s resources but also on the unit’s capabilities to execute training to 

standards, in a specified environment. 

Page 20 GAO-09-518 Military Readiness
 

to perform into GSORTS.21 However, DRRS is designed to capture 

capabilities on a wider variety of missions and mission essential tasks. For 

example, organizations can report their capabilities to conduct missions 

associated with major war plans and operations such as Operation Iraqi 

Freedom into DRRS, as well as their capabilities to perform the missions 

for which they were organized or designed. DRRS also captures capability 

information from a wider range of organizations than GSORTS. Although 

the primary (monthly) focus is on operational units and commands, DRRS 

collects and displays readiness information from defense agencies and 

installations. 

Geographic combatant commands—such as U.S. Central Command, U.S. 

Pacific Command, and U.S. Northern Command—are currently reporting 

their commands’ capabilities to execute most of their operations and 

major war plans in DRRS. DOD reports this enhanced capability 

information from the geographic combatant commands in its Quarterly 

Readiness Report to Congress. The geographic combatant commands are 

also using DRRS to report their capabilities to perform headquarters-level, 

joint mission essential tasks, and some of these commands utilize DRRS as 

their primary readiness reporting tool. For example, U.S. Northern 

Command uses DRRS to assess risk and analyze capability gaps, and U.S. 

Pacific Command identifies critical shortfalls by evaluating mission 

essential task assessments that are captured in DRRS. 

While DRRS currently has the necessary software to collect and display 

these enhanced capability data from organizations at all levels throughout 

DOD, a variety of technical and other factors have hindered service 

reporting of capability data.22 As a result, the services have either 

developed their own systems to report required readiness data or have 

delayed issuing implementing guidance that would require their units to 

report standardized mission essential task data in DRRS. By 2005, DRRS 

was able to collect and display mission essential task information from 

any organizations that had access to a Secure Internet Protocol Router 

 

21The GSORTS metric that captures information concerning a unit’s capability to execute 

these other assigned or directed missions is the percent effective metric. This percent 

effective rating is a subjective assessment that is based on a commander’s professional 

military judgment. It is based on a number of considerations but not strictly tied to 

resources or training performance. 

22The software to collect and display this enhanced capability data has been in place for 

several years and the DIO has stated that DRRS reached its initial operating capability 

when the system was able to collect and display this information. 

Page 21 GAO-09-518 Military Readiness
 

Network (SIPRNet) workstation.23 In August 2005, USD (P&R) issued a 

memorandum that directed the services to ensure that all of their 

GSORTS-reporting units were reporting mission essential task capabilities 

in DRRS by September 30, 2005.24 The memorandum stated that, for 

tactical units, mission essential tasks were to be drawn from the Service 

Universal Task List and standardized across like-type entities, such as tank 

battalions, destroyers, or F-16 squadrons.25 However, two factors that have 

hindered compliance with the memorandum’s direction to report mission 

essential task capabilities in DRRS are discussed below. 

While DRRS has been able to collect and display mission essential task 

data since 2005, some Army and Navy users did not have the means to 

directly access DRRS and update mission essential task assessments. For 

example, some ships lacked hardware necessary to be able to transmit 

their mission essential task data directly into DRRS while at sea. In 

addition, many National Guard units lacked, and still lack, direct access to 

the SIPRNet workstations that are necessary to directly input mission 

essential task data directly into DRRS. However, the Army and the Navy 

have developed systems, respectively designated DRRS-A and DRRS-N 

that interface with DRRS and thus allow all of their units to report mission 

essential task data. After Army and Navy units report mission essential 

task data in their respective DRRS-A and DRRS-N service systems, the 

services transmit these data to DRRS. As a result, Army and Navy officials 

told us that they are currently complying with the requirement to ensure 

that all their GSORTS-reporting units report mission essential task data in 

DRRS. 

Lack of Direct Access to DRRS 

Unlike the Army and the Navy, the Marine Corps and the Air Force have 

not developed their own systems to allow their units to use a single tool to 

Delays in Developing Software 

Tools to Input Data 

 

23The SIPRNet is operated by the Defense Information Systems Agency and is DOD’s 

largest interoperable command and control data network. In addition to supporting the 

input of data to DRRS, the SIPRNet supports the Global Command and Control System, the 

Defense Message System, collaborative planning, and numerous other classified warfighter 

applications. 

24The memorandum—Under Secretary of Defense for Personnel and Readiness 

Memorandum, Department of Defense Readiness Reporting System (DRRS) Interim 

Implementation Guidance, (Aug. 10, 2005)—directed units report mission essential task 

capabilities in DRRS through the automated ESORTS reporting tool. 

25When missions or mission essential tasks are not standardized, capability searches may 

not yield desired results because similar units are measuring their capabilities against 

different task conditions and standards. 

Page 22 GAO-09-518 Military Readiness
 

enter readiness data to meet Office of the Secretary of Defense, Chairman 

of the Joint Chiefs of Staff, and service readiness reporting requirements. 

While the DIO has developed the software for users to enter mission 

essential task data into DRRS, the DIO has been unsuccessful in attempts 

to develop a tool that would allow Air Force and Marine Corps users to 

enter readiness data to meet all of their readiness reporting requirements 

through DRRS. As a result, rather than reducing the burden on reporting 

units, DRRS has actually increased the burden on Air Force and Marine 

Corps units because they are now required to report readiness information 

in both DRRS and GSORTS. On September 29, 2005, USD (P&R) issued a 

memorandum stating that DRRS is the single readiness reporting system 

for the entire Department of Defense and that legacy systems, such as 

GSORTS and associated service readiness systems, should be phased 

out.26 Since that time, officials have discussed whether to phase out 

GSORTS and tentative dates for this action have slipped several times. 

 by 

 

In 2001, the Office of the Deputy Undersecretary of Defense (Readiness) 

listed reducing reporting burdens as a key characteristic of its envisioned 

improved readiness assessment system. In an effort to eliminate this 

burden of dual reporting, the DIO began to develop a “current unit status” 

tool as a means for users to manage unit-specific readiness data and 

submit required reports in support of all current reporting guidelines.27

The tool was to minimize the burden associated with dual reporting

collecting, displaying, and integrating resource data from service 

authoritative data sources with GSORTS and DRRS. However, in 

December 2007, the DIO reported that it was unable to deliver the 

intended functionality of the “current unit status” tool. Instead, the DIO 

decided to develop an interim reporting tool, known as the SORTSREP 

tool, which would not provide the type of new capabilities envisioned for 

the “current unit status” tool, but would simply replicate the functionality 

of the input tool that the Air Force and Marines already used to input data 

into GSORTS. After delays, and 10 months of effort, the DIO delivered the 

SORTSREP tool to the Marine Corps for review. Based on this review, in 

December, 2008, the Marine Corps provided the developers and the DIO 

with 163 pages of detailed descriptions and graphics to explain the 

SORTSREP tool’s deficiencies. It then informed the DIO that it would no 

26Under Secretary of Defense for Personnel and Readiness Memorandum, Status of Defense 

Readiness Reporting System (DRRS) Implementation (Sept. 29, 2005). 

27Current reporting guidelines require the services to report both in the Chairman of the 

Joint Chiefs of Staff’s GSORTS and in OSD’s DRRS. 

Page 23 GAO-09-518 Military Readiness
 

longer expend energy and resources to review future versions of the 

SORTSREP tool and would instead look at leveraging the Army’s or Navy’s 

DRRS-A or DRRS-N systems. The Air Force also informed the DIO that it 

was no longer interested in the SORSTSREP tool, and said efforts should 

be focused on the “current unit status” tool instead. As a result, the Air 

Force and Marine Corps are currently faced with dual reporting 

requirements, as illustrated in figure 1. 

Figure 1: Air Force and Marine Corps Dual Reporting Requirements to Meet 

Readiness Reporting Guidelines 

GSORTS

(Designed mission 

capabilities, resources, and 

training)

DRRS

(Missions and mission 

essential tasks)

Marine Corps

units

Air Force

units RAS-IT

RAS-IT

RAS-IT: Readiness Assessment System Input Tool

Source: GAO based on Air Force and Marine Corps information.

On March 3, 2009, the Air Force Deputy Chief of Staff (Operations, Plans 

and Requirements) issued a memorandum that updated the Air Force’s 

previous implementing guidance and directed all GSORTS-reporting units 

to begin assessing readiness in DRRS based on standardized core task lists 

within 90 days.28 As a result, Air Force units will report readiness in both 

DRRS and GSORTS until the DIO is able to deliver the intended 

functionality of the “current unit status” tool. 

While some Marine Corps units are reporting their capabilities in DRRS, 

the Marine Corps had not yet directed its units to report in the system as 

of May 2009. The Commandant of the Marine Corps had stated that he 

 

28Department of the Air Force Memorandum, Defense Readiness Reporting System 

(DRRS) Core Unit Mission Essential Task List (METL) Implementation Guidance (Mar. 

3, 2009). 

Page 24 GAO-09-518 Military Readiness
 

supported the development and implementation of DRRS, but that he 

would not direct units to assess mission essential tasks in DRRS until the 

system met its stated requirements and was accepted as the single 

readiness reporting system of record. Marine Corps officials said that they 

did not want to place a burden on operational units, which were fighting 

or preparing to fight a war, by requiring that they report readiness in two 

different systems. After we completed our audit work, on May 12, 2009, the 

Marine Corps issued an administrative message that required that units 

assess their mission essential tasks and missions in DRRS. The message 

stated that doing so would improve familiarity with DRRS, which will lead 

to an easier transition when the Marine Corps fields DRRS-Marine Corps 

(DRRS-MC).29

Without a viable tool for inputting data, DRRS is not fully integrated with 

GSORTS or with the service readiness reporting systems and it is not 

capable of replacing those systems since it does not capture the required 

data that are contained in those systems. 

DRRS Enhanced 

Capability Data Are Not 

Likely to Fully Address the 

Challenges with Metrics 

That Existed Prior to the 

Establishment of the 

System 

While DRRS is being used to provide Congress with enhanced capability 

information, the quality of DRRS metrics still faces the same challenges, 

including limitations in timeliness, precision, and objectivity that existed 

prior to 1999 when Congress directed DOD to establish a new readiness 

reporting system. Section 117 of Title 10 of the U.S. Code directed the 

Secretary of Defense to establish a comprehensive readiness reporting 

system to measure the capability of the armed forces in an “objective, 

accurate, and timely manner.” However, the enhanced capability data that 

are captured in DRRS and reported to Congress are no more timely than 

the readiness data that were being provided to Congress in 1999 using 

GSORTS. Furthermore, the metrics that are being used to capture the 

enhanced capability information are less objective and precise than the 

metrics that were used to report readiness in 1999. 

The statute directing the development of a new readiness reporting system 

requires that the reporting system measure in a timely manner the 

capability of the armed forces to carry out the National Security Strategy, 

the Secretary of Defense’s defense planning guidance, and the National 

Timeliness 

 

29Marine Corps Administrative Message 0307//09, Update to Interim Defense Readiness 

Reporting System (DRRS) Policy and Procedures for Marine Corps Units and 

Installations (May 12, 2009). 

Page 25 GAO-09-518 Military Readiness
 

Military Strategy. The legislation also lists a number of specific 

requirements related to frequency of measurements and updates. For 

example, the law requires that the capability of units to conduct their 

assigned wartime missions be measured monthly, and that units report any 

changes in their overall readiness status within 24 hours of an event that 

necessitated the change in readiness status.30 In its DRRS directive, DOD 

assigned USD (P&R) responsibility for ensuring the timeliness of DRRS 

information and data, and it specified that DRRS was to be a near-real-time 

readiness reporting system. 

While DOD is reporting readiness information to Congress on a quarterly 

basis as required, and units are measuring readiness on a monthly basis, 

DRRS is not a near-real-time reporting system. Specifically, in DRRS, as in 

GSORTS, operational commanders assess the readiness of their 

organizations on a monthly basis or when an event occurs that changes 

the units’ overall reported readiness. Thus, DRRS has not improved the 

timeliness of the key readiness data that are reported to Congress. 

According to USD (P&R) officials, DRRS data will be more timely than 

GSORTS data because DRRS will update underlying data from 

authoritative data sources between the monthly updates.31 However, DRRS 

is not yet capturing all the data from the authoritative data sources, and 

according to service officials, the service systems that support GSORTS 

also draw information from their service authoritative data sources 

between the monthly updates. Furthermore, the source and currency of 

some of the authoritative data that are currently in DRRS are not clearly 

identified. As a result, some users told us that they are reluctant to use 

DRRS data to support their decisions. 

We previously reported that the readiness information that DOD provided 

to Congress lacked precision, noting that GSORTS readiness measures 

that differed by 10 percentage points or more could result in identical 

ratings, with DOD often not reporting the detailed information behind the 

ratings outside of the department.32 For example, units that were at 90 and 

Precision and Objectivity

 

30The law also lists annual requirements for training establishments, defense installations, 

and facilities to report their capabilities, and a number of other monthly, quarterly, and 

annual requirements. 

31While certain data, such as personnel data, are captured in multiple data systems, the 

services or OSD designate a single data system as the authoritative database for each type 

of information. 

32GAO-03-456. 

Page 26 GAO-09-518 Military Readiness
 

100 percent of their authorized personnel strengths both were reported as 

P-1 in DOD’s reports to Congress. In 2003, USD (P&R) recognized the 

imprecision of the reported metrics from GSORTS and noted that its 

efforts to develop DRRS would allow enhancements to reported readiness 

data. 

As previously noted, the DRRS capability information that DOD is 

reporting to Congress covers a broader range of missions than the 

GSORTS information that was provided in the past. However, when 

comparing the DRRS and GSORTS assessments of units’ capabilities to 

perform the missions for which the units were organized or designed, 

DRRS assessments are actually less precise than the GSORTS 

assessments. Specifically, within GSORTS, overall capability assessments 

are grouped into four categories based on four percentage ranges for the 

underlying data. For example, commanders compare on-hand and required 

levels of personnel and equipment. Within DRRS, mission essential task 

assessments are reported on a scale that includes only three ratings—

”yes”, “no”, and “qualified yes,” which can include any assessments that 

fall between the two extremes. 

The law directing DOD to establish a new readiness reporting system also 

requires that the system measure readiness in an objective manner. 

GSORTS assessments of units’ capabilities to execute the missions for 

which they were organized or designed are based on objective personnel 

and equipment data and training information that may include both 

objective and subjective measures. Furthermore, the overall capability 

assessment in GSORTS is based on an objective rule that calls for the 

overall assessment to be the same level as the lowest underlying resource 

or training data level. For example, if a unit reported the highest personnel 

level (P-1) and the lowest training level (T-4), the rules in the GSORTS 

guidance instruct the commander to rate the unit’s overall capability at the 

C-4 level. Because GSORTS contains these objective measures and rules, it 

is easy to evaluate reported readiness to see if it aligns with established 

reporting criteria.33

 

33Chairman of the Joint Chiefs of Staff Instruction 3401.02A, Global Status of Resources 

and Training System (GSORTS) (Feb. 27, 2004), permits commanders to subjectively 

upgrade or downgrade their units’ overall assessment, although not their individual 

resource levels. However, these changes are easy to identify and commanders are required 

to provide justifications for any changes. A recent GAO review of Army data found that 

commanders’ subjective changes constituted less than 10 percent of the total assessments. 

Page 27 GAO-09-518 Military Readiness
 

Within DRRS, organizations rate their capabilities based on mission 

essential tasks. These mission essentials tasks have conditions and 

standards associated with them. The conditions specify the types of 

environments that units are likely to face as they execute the tasks, such 

as weather conditions and political or cultural factors. Standards describe 

what it means for the unit to successfully execute the task under specified 

conditions. For example, a unit may have to achieve a 90 percent success 

rate for measures associated with the task being assessed. In spite of these 

conditions and standards, DRRS mission assessments are often subjective 

rather than objective. In DRRS program guidance, DOD has defined 

mission essential tasks as tasks that are approved by the commander and 

that, based on mission analysis, are “absolutely necessary, indispensable, 

or critical to mission success.” In prior briefings and reports to Congress, 

we have noted examples that highlight the subjective nature of DRRS 

mission assessments. For example, we noted that one commander used 

his professional judgment to decide that his command was “qualified” to 

execute a mission even though the preponderance of the “indispensable” 

tasks that supported that mission were rated as “no.” In contrast, other 

commanders used their professional judgments to rate missions as 

“qualified” based on one or more “qualified” tasks among many “yes” tasks. 

DRRS Lacks the Complete 

Resource and Training 

Data and System 

Connectivity Some Users 

Need to Manage and 

Deploy Forces 

DRRS does not have all of the resource, training, readiness data, and 

connectivity with the department’s operations planning and execution 

system that the services, Joint Staff, and certain combatant commands 

need to manage and deploy forces. As a result, DRRS is not yet able to 

operate as the department’s single readiness reporting system, as 

intended. The Secretary of Defense’s and the Under Secretary of Defense’s 

guidance documents recognize that DRRS needs to support the data 

requirements of multiple users. For example, the Secretary of Defense’s 

June 25, 2004, memorandum directed USD (P&R) to develop DRRS to 

support the data requirements identified by the Chairman of the Joint 

Chiefs of Staff, the combatant commanders, the Secretaries of the military 

departments, and the Chief of the National Guard Bureau.34 Furthermore, 

the 2002 DRRS directive noted that DRRS was to build upon DOD’s 

existing processes and readiness assessment tools and that ESORTS 

information (capability, resource, and training), where appropriate, is 

 

34The memorandum’s primary purpose was to provide policy implementation establishing 

the Commander of the U.S. Joint Forces Command to assume the duties as DOD’s primary 

joint force provider. 

Page 28 GAO-09-518 Military Readiness
 

integrated into DOD’s planning systems and processes. It also directed the 

Chairman of the Joint Chiefs of Staff to maintain the GSORTS database 

until key capabilities of DRRS become fully operational. 

Officials with U.S. Joint Forces Command and U.S. Special Operations 

Command reported that historical data are needed to manage forces and 

provide users the ability to analyze readiness trends.35 Similarly, service 

officials stated a need for historical data so they can manage their forces 

and take action to address readiness issues. In 2005, USD (P&R) reported 

that unit resource data, including detailed inventory and authorization 

data on personnel, equipment, supply, and ordnance were available in 

DRRS. However, in response to a survey we conducted in December 2008, 

the services and certain combatant commands stated that necessary 

current and historical resource and training data were not available in 

DRRS. For example, officials from all four services responded that DRRS, 

at that time, contained less than half of their GSORTS resources and 

training data. In addition, officials from U.S. Joint Forces Command, U.S. 

Special Operations Command, the U.S. Strategic Command, and the U.S. 

Transportation Command all responded that historical resource data were 

not available in DRRS. We confirmed that this information was still not 

available when we concluded our review, and in April, 2009, the DIO said 

it was still working on this data availability issue. 

Complete and Accurate Current 

and Historical Data 

Furthermore, user organizations have reported inaccuracies in the data 

that are available in DRRS. Marine Corps and U.S. Special Operations 

Command officials stated that inconsistencies between DRRS data and the 

data in other readiness systems have caused them to adjudicate the 

inconsistencies by contacting their subordinate units directly. Army 

officials noted that searches of DRRS data can produce different results 

than searches in the Army’s data systems. For example, they noted that a 

DRRS search for available personnel with a particular occupational 

specialty produced erroneously high results because DRRS did not employ 

the appropriate business rules when conducting the search. Specifically, 

DRRS did not apply a business rule to account for the fact that an 

individual person can possess multiple occupational specialty codes but 

can generally fill only one position at a time. DIO officials informed us that 

they intend to correct issues with the accuracy of data drawn from 

 

35These historical data would include training data, personnel data, and equipment supply 

and condition data as well as data concerning unit capabilities to perform the missions they 

were organized and designed to perform. 

Page 29 GAO-09-518 Military Readiness
 

external databases. However, the current version of the DRRS Integrated 

Master Schedule indicates that the ability of DRRS to provide the 

capability to correctly search, manipulate, and display current and 

historical GSORTS and mission essential task data will not be complete 

until June 2010. As a result, the reliability of the DRRS data is likely to 

remain questionable and a number of DOD organizations will likely 

continue to rely on GSORTS and other sources of readiness data to 

support their decision making. 

One important DRRS function is integration with DOD’s planning systems. 

Specifically, the 2002 DRRS directive requires USD (P&R) to ensure that, 

where appropriate, ESORTS information (capability, resource, and 

training) is compatible and integrated into DOD’s planning systems and 

processes. Global force management is one of the DOD planning 

processes that is to be integrated with DRRS. Global Force Management is 

a process to manage, assess, and display the worldwide disposition of U.S. 

forces, providing DOD with a global view of requirements and availability 

of forces to meet those requirements. The integration of DRRS with global 

force management planning processes is supposed to allow DOD to link 

force structure, resources, and capabilities data to support analyses, and 

thus help global force managers fill requests for forces or capabilities. 

Connections to Planning 

Systems 

Officials from the four organizations with primary responsibilities for 

providing forces (U.S. Joint Forces Command, U.S. Special Operations 

Command, U.S. Strategic Command, and U.S. Transportation Command) 

all stated that they are unable to effectively use DRRS to search for units 

that will meet requested capabilities. These commands also reported that 

DRRS does not currently contain the information and tools necessary to 

support global force management. For example, officials from U.S. 

Northern Command told us that when they used DRRS to search for 

available helicopters of a certain type, they found thousands, but when 

U.S. Joint Forces Command did the same DRRS search they found 

hundreds. The current version of the DRRS Integrated Master Schedule 

indicates that DRRS will not be able to fully support global force 

management until March 2011. As a result, these commands continue to 

rely on GSORTS rather than DRRS to support their planning and sourcing 

decisions. 

DRRS is not currently and consistently providing timely, objective, and 

accurate information, and it is not exactly clear where the department 

stands in its efforts to meet this expectation because system requirements 

remain in a state of flux, and the program office lacks disciplined program 

Conclusions 

Page 30 GAO-09-518 Military Readiness
 

management and results information due to a long-standing lack of rigor in 

its approach to acquiring and deploying system capabilities. This situation 

can be attributed, in part, to long-standing limitations in the program 

office’s focus on acquiring human capital skills needed to manage such a 

complex initiative. It can also be linked to many of years of limited 

program office oversight and accountability. Although program oversight 

has recently increased, oversight bodies have not had sufficient visibility 

into the program’s many management weaknesses. 

DRRS is providing Congress and readiness users with additional mission 

and mission essential task capability data that were not available in 

GSORTS. However, after investing about 7 years and about $96.5 million in 

developing and implementing DRRS, the system’s schedule has been 

extended, requirements are not stable, and the system still does not meet 

congressional and DOD requirements for a comprehensive readiness 

reporting system to assess readiness and help decision makers manage 

forces needed to conduct combat and contingency operations around the 

world. Given DRRS performance and management weaknesses, it is 

critical that immediate action be taken to put the program on track and 

position it for success. Without this action, it is likely that DRRS will cost 

more to develop and deploy than necessary and that DOD will not have a 

comprehensive reporting system that meets the needs of all the decision 

makers who rely on accurate, timely, and complete readiness information. 

To address the risks facing DOD in its acquisition and deployment of 

DRRS, and to increase the chances of DRRS meeting the needs of the DOD 

readiness community and Congress, we recommend that the Secretary of 

Defense direct the Deputy Secretary of Defense, as the Chair of the 

Defense Business Systems Management Committee, to reconsider the 

committee’s recent approval of DRRS planned investment for fiscal years 

2009 and 2010, and convene the Defense Business Systems Management 

Committee to review the program’s past performance and the DIO’s 

capability to manage and deliver DRRS going forward. 

Recommendations for 

Executive Action 

To fully inform this Defense Business Systems Management Committee 

review, we also recommend that the Secretary direct the Deputy Secretary 

to have the Director of the Business Transformation Agency, using the 

appropriate team of functional and technical experts and the established 

risk assessment methodology, conduct a program risk assessment of 

DRRS, and to use the findings in our report and the risk assessment to 

decide how to redirect the program’s structure, approach, funding, 

management, and oversight. In this regard, we recommend that the 

Page 31 GAO-09-518 Military Readiness
 

Secretary direct the Deputy Secretary to solicit the advice and 

recommendations of the DRRS Executive Committee. 

We also recommend that the Secretary, through the appropriate chain of 

command, take steps to ensure that the following occur: 

1. DRRS requirements are effectively developed and managed with 

appropriate input from the services, Joint Staff, and combatant 

commanders, including (1) establishing an authoritative set of baseline 

requirements prior to further system design and development; (2) 

ensuring that the different levels of requirements and their associated 

design specifications and test cases are aligned with one another; and 

(3) developing and instituting a disciplined process for reviewing and 

accepting changes to the baseline requirements in light of estimated 

costs, benefits, and risk. 

2. DRRS testing is effectively managed, including (1) developing test 

plans and procedures for each system increment test event that 

include a schedule of planned test activities, defined roles and 

responsibilities, test entrance and exit criteria, test defect management 

processes, and metrics for measuring test progress; (2) ensuring that 

all key test events are conducted on all DRRS increments; (3) 

capturing, analyzing, reporting, and resolving all test results and test 

defects of all developed and tested DRRS increments; and (4) 

establishing an effective test management structure that includes 

assigned test management roles and responsibilities, a designated test 

management lead and a supporting working group, and a reliable 

schedule of test events. 

3. DRRS integrated master schedule is reliable, including ensuring that 

the schedule (1) captures all activities from the work breakdown 

structure, including the work to be performed and the resources to be 

used; (2) identifies the logical sequencing of all activities, including 

defining predecessor and successor activities; (3) reflects whether all 

required resources will be available when needed and their cost; (4) 

ensures that all activities and their duration are not summarized at a 

level that could mask critical elements; (5) achieves horizontal 

integration in the schedule by ensuring that all external interfaces 

(hand-offs) are established and interdependencies among activities are 

defined; (6) identifies float between activities by ensuring that the 

linkages among all activities are defined; (7) defines a critical path that 

runs continuously to the program’s finish date; (8) incorporates the 

results of a schedule risk analysis to determine the level of confidence 

in meeting the program’s activities and completion date; and (9) 

Page 32 GAO-09-518 Military Readiness
 

includes the actual start and completion dates of work activities 

performed so that the impact of deviations on downstream work can 

be proactively addressed. 

4. The DRRS program office is staffed on the basis of a human capital 

strategy that is grounded in an assessment of the core competencies 

and essential knowledge, skills, and abilities needed to perform key 

DRRS program management functions, an inventory of the program 

office’s existing workforce capabilities, and an analysis of the gap 

between the assessed needs and the existing capabilities. 

5. DRRS is developed and implemented in a manner that does not 

increase the reporting burden on units and addresses the timeliness, 

precision, and objectivity of metrics that are reported to Congress. 

To ensure that these and other DRRS program management improvements 

and activities are effectively implemented and that any additional funds for 

DRRS implementation are used effectively and efficiently, we further 

recommend that the Secretary direct the Deputy Secretary to ensure that 

both the Human Resources Management Investment Review Board and 

the DRRS Executive Committee conduct frequent oversight activities of 

the DRRS program, and report any significant issues to the Deputy 

Secretary. 

In written comments on a draft of this report, signed by the Deputy Under 

Secretary of Defense (Military Personnel Policy) performing the duties of 

the Under Secretary of Defense (Personnel and Readiness), DOD stated 

that the report is flawed in its assessment of DRRS, noting that DRRS is a 

net-centric application that provides broad and detailed visibility on 

readiness issues, and that achieving data sharing across the DOD 

enterprise was groundbreaking work fraught with barriers and obstacles, 

many of which have now been overcome. In addition, DOD stated that it 

was disappointed that the report did not address cultural impediments 

that it considers to be the root cause of many of the issues cited in the 

report and of many previous congressional concerns on readiness 

reporting. DOD further stated that the report instead focuses on past 

acquisition process and software development problems that it believes 

have now been remedied According to the department, this focus, coupled 

with inaccurate and misleading factual information included in the report, 

led us to develop an incomplete picture of the program. Notwithstanding 

these comments, DOD agreed with two of our recommendations and 

partially agreed with a third. However, it disagreed with the remaining five 

Agency Comments 

and Our Evaluation 

Page 33 GAO-09-518 Military Readiness
 

recommendations, and provided comments relative to each 

recommendation. DOD’s comments are reprinted in their entirety in 

appendix III. 

In summary, we do not agree with DOD’s overall characterization of our 

report or the positions it has taken in disagreeing with five of our 

recommendations, finding them to be inconsistent with existing guidance 

and recognized best practices on system acquisition management, 

unsupported by verifiable evidence, and in conflict with the facts detailed 

in our report. Further, we recognize that developing DRRS is a significant 

and challenging undertaking that involves cultural impediments. As a 

result, our report explicitly focuses on the kind of program management 

rigor and disciplines needed to address such impediments and 

successfully acquire complex systems, including effective requirements 

development and management and executive oversight. We also disagree 

that our report focuses on past issues and problems. Rather, it provides 

evidence that demonstrates a long-standing and current pattern of system 

acquisition and program oversight weaknesses that existed when we 

concluded our audit work and that DOD has not provided any evidence to 

demonstrate has been corrected. 

In addition, we would emphasize that we defined our objectives, scope, 

and methodology, and executed our audit work in accordance with 

generally accepted government auditing standards, which require us to 

subject our approach as well as the results of our audit work to proven 

quality assurance checks and evidence standards that require us to seek 

documentation rather than relying solely on testimonial evidence. While 

we support any departmental efforts, whether completed or ongoing, that 

would address the significant problems cited in our report, we note that 

DOD, in its comments, did not specifically cite what these efforts are or 

provide documentation to support that they have either been completed or 

are ongoing. Therefore, we stand by our findings and recommendations. 

Moreover, we are concerned that in light of the program’s significant and 

long-standing management weaknesses, the department’s decision not to 

pursue recommendations aimed at corrective actions for five of our eight 

recommendations will further increase risk to achieving program success, 

and is not in the best interests of the military readiness community or the 

U.S. taxpayer. Accordingly, we encourage the department to reconsider its 

position when it submits its written statement of the actions taken on our 

recommendations to the Senate Committee on Homeland Security and 

Governmental Affairs and the House Committee on Oversight and 

Government Reform , as well has the House and Senate Committees on 

Appropriations, as required under 31 U.S.C. 720. 

Page 34 GAO-09-518 Military Readiness
 

DOD’s specific comments on each recommendation, along with our 

responses to its comments follow. 

• The department did not agree with our recommendation for the Deputy 

Secretary of Defense, as the Chair of the Defense Business Systems 

Management Committee, to reconsider the committee’s recent 

approval of DRRS planned investment for fiscal years 2009 and 2010, 

and to convene the Defense Business Systems Management Committee 

to review the program’s past performance and the DIO’s capability to 

manage and deliver DRRS going forward in deciding how best to 

proceed. In this regard, DOD stated that the Investment Review Board 

certification and Defense Business Systems Management Committee 

approval were granted in compliance with the established processes. It 

also added that oversight of the specific issues identified in this report 

are the responsibility of the DRRS Executive Committee, which it 

stated has and will continue to provide appropriate governance for this 

effort. It also stated that USD (P&R) will ensure that the program is 

compliant with all acquisition requirements prior to submission for 

further certifications. 

We do not question whether the Investment Review Board certification 

and Defense Business Systems Management Committee approval were 

provided in accordance with established processes, as this is not 

relevant to our recommendation. Rather, our point is that the 

Investment Review Board and Defense Business Systems Management 

Committee were provided, and thus based their respective decisions, 

on erroneous and incomplete information about DRRS progress, 

management weaknesses, and risks. Moreover, neither the Investment 

Review Board nor the Defense Business Systems Management 

Committee were informed about the findings in our report, even 

though we shared each of them with the DRRS program director and 

other DIO officials prior to both the Investment Review Board and the 

Defense Business Systems Management Committee deliberations. 

Therefore, while the Investment Review Board certification and the 

Defense Business Systems Management Committee approval were 

granted in accordance with established processes, they were not based 

on a full disclosure of facts. Moreover, while we support DOD’s 

comment that it will ensure that the program is in compliance with all 

acquisition requirements prior to further certifications, nothing 

precludes the board or the committee from reconsidering their 

respective decisions in light of our report. With respect to DOD’s 

comment that the DRRS Executive Committee has and will continue to 

provide appropriate governance for this effort, we do not disagree that 

the DRRS Executive Committee has an oversight role. However, the 

DRRS Executive Committee should not be solely responsible for 

Page 35 GAO-09-518 Military Readiness
 

oversight of the specific issues in our report. Both the Investment 

Review Board and the Defense Business Systems Management 

Committee provide additional layers of oversight pursuant to law36 and 

DOD policy.37 Accordingly, we stand by our recommendation as it 

appropriately seeks to have the Investment Review Board and Defense 

Business Systems Management Committee, in collaboration with the 

DRRS Executive Committee, act in a manner that is consistent with 

their respective roles as defined in law. In doing so, our intent is to 

promote accountability for DRRS progress and performance, and 

prompt action to address the many risks facing the program. 

• The department agreed with our recommendation for the Deputy 

Secretary of Defense, as the chair of the Defense Business Systems 

Management Committee, to have the Business Transformation Agency 

conduct a risk assessment of DRRS, and with the advice and 

recommendation of the DRRS Executive Committee, to use the results 

of this assessment and the findings in our report to decide how to 

redirect the program. In this regard, the department stated that this 

assessment will be complete by the middle of fiscal year 2010. 

• The department did not agree with our recommendation for ensuring 

that DRRS requirements are effectively developed and managed. In this 

regard, it stated that the program has an authoritative set of baseline 

requirements established with an effective governance process for 

overseeing the requirements management process, to include biweekly 

reviews as part of the DRRS configuration control process. 

We do not agree. At the time we concluded our work, DRRS 

requirements were not stable, as evidenced by the fact that an 

additional 530 requirements had been identified that the DIO was still 

in the process of reviewing and had yet to reach a position on their 

inclusion, or process them through the DRRS change control 

governance process. Moreover, when we concluded our work, this 

change control process had yet to be approved by the DRRS 

governance structure. As we state in our report, the introduction of 

such a large number of requirements provided a compelling basis for 

concluding that requirements had yet to progress to the point that they 

could be considered sufficiently complete and correct to provide a 

 

36Ronald W. Reagan National Defense Authorization Act for Fiscal Year 2005, Pub. L. No. 

108-375, § 332 (2004) (codified at 10 U.S.C. §§ 186 and 2222). 

37Department of Defense Instruction Number 5000.02, Operation of the Defense 

Acquisition System, Enclosure 11 (December 8, 2008). 

Page 36 GAO-09-518 Military Readiness
 

stable baseline. Our recommendation also noted that the Secretary 

should take steps to ensure that the different levels of requirements be 

aligned with one another. DOD’s comments did not address this aspect 

of our recommendation. 

• The department did not agree with our recommendation for ensuring 

that DRRS testing is effectively managed. In this regard, it stated that 

DRRS testing is already in place and performing effectively, and stated, 

among other things, that (1) the DIO goes through a rigorous testing 

regimen that includes documenting test plans with user test cases for 

each incremental release to include utilizing system integration, 

acceptance, interoperability, and operational testing; (2) user test cases 

and functionality are validated by designated testers independent of the 

developers prior to a deployment; and (3) for interoperability testing 

the DIO has a designated test director and the Joint Interoperability 

Test Command (JITC) is the designated interoperability and 

operational test activity. 

We do not agree. As our report concludes, DRRS testing has not been 

effectively managed because it has not followed a rigorous testing 

regimen that includes documented test plans, cases, and procedures. 

To support this conclusion, our report cites numerous examples of test 

planning and execution weaknesses, as well as the DIO’s repeated 

inability to demonstrate through requisite documentation that the 

testing performed on DRRS has been adequate. Our report shows that 

test events for already acquired, as well as currently deployed and 

operating, DRRS releases and subreleases were not based on welldefined plans and DOD had not filled its testing director vacancy. 

Further, our report shows that test events were not fully executed in 

accordance with plans that did exist, or executed in a verifiable 

manner, or both. For example, although increments of DRRS 

functionality had been put into production, the program had no 

documentation (e.g., test procedures, test cases, test results) to show 

that the program office had performed system integration testing, 

system acceptance testing, or operational testing on any DRRS release 

or subrelease, even though the DIO’s test strategy stated that such tests 

were to be performed before system capabilities became operational. 

Moreover, evidence showed that the results of all executed test events 

had not been captured and used to ensure that problems discovered 

were disclosed to decision makers, and ultimately corrected. With 

respect to DOD’s comments that JITC is the designated lead for 

interoperability and operational testing, our report recognizes that JITC 

is to conduct both interoperability and operational testing before the 

system is deployed and put into production (i.e., used operationally). 

Page 37 GAO-09-518 Military Readiness
 

However, during the course of our audit, the DIO could not produce 

any evidence to show that interoperability and operational testing of all 

operating system increments had been conducted. 

• The department did not agree with our recommendation for ensuring 

that the DRRS integrated master schedule is reliable. In this regard, it 

stated that a process is already in place to ensure that the schedule is 

current, reliable, and meets all the criteria outlined in the 

recommendation. 

We do not agree. As our report states, an integrated master schedule 

for DRRS did not exist until November 2008, which was 2 months after 

we first requested one. Moreover, following our feedback to the DIO on 

limitations in this initial version, a revised integrated master schedule 

was developed in January 2009, which was also not reliable. 

Subsequently, a revised integrated master schedule was developed in 

April 2009. However, as we detail in our report, that version still 

contained significant weaknesses. For example, it did not establish a 

critical path for all key activities or include a schedule risk analysis, 

and was not being updated using logic and durations to determine the 

dates for all key activities. These practices are fundamental to 

producing a sufficiently reliable schedule baseline that can be used to 

measure progress and forecast slippages. In addition, the schedule 

introduced considerable concurrency across key activities and events 

for several modules, which introduces increased risk. Therefore, we 

stand by our recommendation. 

• The department partially agreed with our recommendation for ensuring 

that it has an effective human capital strategy. In this regard, it stated 

that actions are underway to add more full-time civilian support to the 

DIO, and that plans exist to convert some contractor to civilian billets 

during the 2010/2011 time frame. 

We support the department’s actions and plans described in its 

comments to address the DIO human capital management limitations 

discussed in our report, but would note that they do not go far enough 

to systematically ensure that the program has the right people with the 

right skills to manage the program in both the near term and the long 

term. To accomplish this, the department needs to adopt the kind of 

strategic and proactive approach to DRRS workforce management that 

our report describes and our recommendation embodies. As our 

evaluations and research show, failure to do so increases the risk that 

the program office will not have the people it needs to effectively and 

Page 38 GAO-09-518 Military Readiness
 

efficiently manage DRRS. Therefore, we believe that the department 

needs to fully implement our recommendation. 

• The department did not agree with our recommendation to take steps 

to ensure that DRRS is developed and implemented in a manner that 

does not increase the reporting burden on units and addresses the 

timeliness, precision, and objectivity of metrics that are reported to 

Congress. In this regard, it stated that one of the primary tenets of 

DRRS has been to reduce reporting requirements on the war fighter. It 

also stated that DRRS is already using state-of-the-art technology to 

ensure that near-real-time data are available for the war fighters. 

Finally it stated that the DRRS governance structure that is currently in 

place ensures that DRRS development does not deviate from these core 

principles. 

While we recognize that a goal of DRRS is to reduce a reporting burden 

on the war fighter, we disagree with the department’s position because 

the system has not yet achieved this goal. As our report states, while 

the DIO has developed the software for users to enter mission essential 

task data into DRRS, the DIO has been unsuccessful in attempts to 

develop a tool that would allow Air Force and Marine Corps users to 

enter readiness data to meet all of their readiness reporting 

requirements through DRRS. As a result, rather than reducing the 

burden on reporting units, DRRS actually increased the burden on Air 

Force and Marine Corps units because they were required to report 

readiness information in both DRRS and GSORTS. Without a viable 

tool for inputting data, DRRS is not fully integrated with GSORTS or 

with the service readiness reporting systems and it is not capable of 

replacing those systems since it does not capture the required data that 

are contained in those systems. In addition, the DRRS readiness data 

that are currently reported to Congress are not near-real-time data. 

Specifically, the periodicity for DRRS capability assessments is the 

same as the legacy GSORTS system’s readiness reports—monthly or 

when an event occurs that changes a unit’s overall readiness. 

Furthermore, our report shows that DRRS mission assessments are 

often subjective and imprecise because they are reported on a scale 

that includes only three ratings—”yes,” “no,” and “qualified yes,” which 

can include any assessments that fall between the two extremes. 

Therefore, because additional actions are still needed to reduce 

reporting burdens and improve the timeliness, precision, and 

objectivity of the DRRS data that are reported to Congress, we stand by 

our recommendation. 

Page 39 GAO-09-518 Military Readiness
 

• The department agreed with our recommendation for ensuring that 

both the Human Resources Management Investment Review Board and 

the DRRS Executive Committee conduct frequent oversight activities 

of the DRRS program and report any significant issues to the Deputy 

Secretary of Defense. In this regard, the department stated that the 

USD (P&R) component acquisition executive is working with the 

program to ensure that it becomes fully compliant with all acquisition 

requirements. In addition, it stated that the acquisition executive will 

certify to the Human Resources Investment Review Board and the 

Deputy Chief Management Officer of compliance prior to submission of 

future certification requests. Further, it stated that the current DRRS 

governance process will provide sustained functional oversight of the 

program and that issues that arise in any of these areas will be elevated 

for review, as appropriate. We believe these are positive steps. 

We are sending copies of this report to the appropriate congressional 

committees; the Secretary of Defense; and the Director, Office of 

Management and Budget. The report will also be available at no charge on 

the GAO Web site at http://www.gao.gov. 

If you or your staffs have questions about this report, please contact us at 

pickups@gao.gov or hiter@gao.gov or at our respective phone numbers, 

(202) 512-9619 and (202) 512- 3439. Contact points for our Offices of 

Page 40 GAO-09-518 Military Readiness
 

Congressional Relations and Public Affairs may be found on the last page 

of this report. GAO staff who made key contributions to this report are 

Sharon L. Pickup 

listed in appendix IV. 

Director, Information 

ecture and Systems Issues 

Director, Defense Capabilities and Management 

Randolph C. Hite 

Technology Archit

Page 41 GAO-09-518 Military Readiness
Appendix I: Objectives, Scope, and 

Methodology 

Page 42 GAO-09-518 

Appendix I: Objectives, Scope, and 

Methodology 

Our objectives were to assess the extent to which (1) the Department of 

Defense (DOD) has effectively managed and overseen DRRS acquisition 

and deployment, and (2) features of the Defense Readiness Reporting 

System (DRRS) have been implemented and are consistent with legislative 

requirements and DOD guidance. 

We did not evaluate the department’s overall ability to assess the readiness 

of its forces or the extent that data contained in any of its readiness 

reporting systems, including DRRS and the Global Status of Resources and 

Training System (GSORTS), reflect capabilities, deficiencies, 

vulnerabilities, or performance issues. Our review focused on acquisition 

and program management issues, such as requirements management, 

schedule, and human capital requirements; the current usage of DRRS; and 

the extent to which DRRS’ features address legislative requirements and 

DOD guidance. 

To determine the extent to which the DRRS acquisition and deployment 

has been effectively managed and overseen, we focused on the following 

acquisition management areas: (1) requirements development and 

management, (2) test planning and execution, (3) DRRS schedule 

reliability, and (4) human capital planning. In doing so, we analyzed a 

range of program documentation, such as high-level and detailed-level 

requirements documentation, test plans and reports, the current DRRS 

schedule, and program management documentation and interviewed 

cognizant program and contractor officials. 

• To determine the extent to which the program had effectively 

implemented requirements development and management, we 

reviewed relevant program documentation, such as the concept of 

operations document, capability requirements document, software 

requirements document, requirements traceability matrix, 

configuration management plan, and the program management plan, as 

well as minutes of change control board meetings, and evaluated them 

against relevant guidance.1

 Moreover, we reviewed briefing slides from 

meetings of DRRS oversight bodies in order to identify concerns about 

 

1

The Capability Maturity Model® Integration for Development, developed by the Software 

Engineering Institute of Carnegie Mellon University, defines key practices that are 

recognized hallmarks for successful organizations that, if effectively implemented, can 

greatly increase the chances of successfully developing and acquiring software and 

systems. Carnegie Mellon Software Engineering Institute, Capability Maturity Model® 

Integration for Development, version 1.2 (Pittsburgh, Pa.: August 2006). 

 Military Readiness 
Appendix I: Objectives, Scope, and 

Methodology 

DRRS expressed by representatives from the DRRS community of 

users, as well as the efforts by the Joint Staff (at the direction of DRRS 

Executive Committee) to identify and address any gaps identified by 

users in the development of DRRS requirements. To determine the 

extent to which the program has maintained traceability backward to 

high-level business operation requirements and system requirements, 

and forward to system design specifications and test plans, we 

randomly selected 60 program requirements and traced them both 

backward and forward. This sample was designed with a 5 percent 

tolerable error rate at the 95 percent level of confidence so that, if we 

found zero problems in our sample, we could conclude statistically that 

the error rate was less than 5 percent. In addition, we interviewed 

program and development contractor officials to discuss the 

requirements development and management process. 

• To determine if the DRRS Implementation Office (DIO) is effectively 

managing DRRS testing, we reviewed relevant documentation, such as 

the DRRS Test and Evaluation Master Plans and test reports and 

compared them to DOD and other relevant guidance.2

 Further, we 

reviewed developmental test plans and procedures for each 

release/subrelease that to date has either been developed or fielded and 

compared them with best practices to determine whether test activities 

had been adequately documented. We also examined test results and 

reports for the already acquired, as well as currently deployed and 

operating, DRRS releases and subreleases and compared them against 

plans to determine whether they had been executed in accordance with 

plans. Moreover, we reviewed key test documentation, such as the 

Software Version Descriptions, and compared them against relevant 

guidance to determine whether defect data were being captured, 

analyzed, prioritized, and reported. We also interviewed program and 

contractor officials to gain clarity beyond what was included in the 

program documentation, including the Defense Information Systems 

Agency’s Joint Interoperability Test Center in order to determine the 

results of their efforts to independently test DRRS interoperability. In 

addition, to determine the extent to which the program had effectively 

tested its system requirements, we observed the DIO’s efforts to 

 

2

See for example, Department of Defense Instruction 5000.02, Operation of the Defense 

Acquisition System (Dec. 8, 2008); Defense Acquisition University, Test and Evaluation 

Management Guide, 5th ed. (Fort Belvoir, Va.: January 2005); Institute of Electrical and 

Electronic Engineers, Standard 1012-2004 for Software Verification and Validation 

(New York, N.Y.: June 8, 2005); and Software Engineering Institute, Capability Maturity 

Model Integration for Acquisition version 1.2 (Pittsburgh, Pa.: November 2007). 

Page 43 GAO-09-518 Military Readiness
Appendix I: Objectives, Scope, and 

Methodology 

demonstrate the traceability of 60 program requirements to test cases 

and results. This sample was designed with a 5 percent tolerable error 

rate at the 95 percent level of confidence so that, if we found zero 

problems in our sample, we could conclude statistically that the error 

rate was less than 5 percent. 

• To determine the extent to which the program’s schedule reflects key 

estimating practices that are fundamental to having a reliable schedule, 

we reviewed the DRRS integrated master schedules and schedule 

estimates and compared them with relevant guidance.3

 We also used 

schedule analysis software tools to determine whether the latest 

schedule included key information, such as the activities critical to ontime completion of DRRS, a logical sequence of activities, and evidence 

that the schedule was periodically updated. We also reviewed the 

schedule to determine the time frames for completing key program 

activities and to determine any changes to key milestones. In addition, 

we shared the results of our findings with program and contractor 

officials and asked for clarifications. We then reviewed the revised 

schedule, prepared in response to the weaknesses we found, and 

compared it with relevant guidance. 

• To evaluate whether DOD is adequately providing for the DRRS 

program’s human capital needs, we compared the program’s efforts 

against relevant criteria and guidance, including our own framework 

for strategic human capital management.4

 In doing so, we reviewed key 

program documentation, such as the program management plan and 

the DIO organizational structure to determine whether it reflected key 

acquisition functions and identified whether these functions were being 

performed by government or contractor officials. We interviewed key 

officials to discuss workforce analysis and human capital planning 

efforts. 

• To determine the level of oversight and governance available to the 

DRRS community of users, we attended relevant meetings, met with 

officials responsible for program certification, and reviewed relevant 

guidance and program documentation. Specifically, we attended Battle 

Staff meetings and analyzed briefing slides and meeting minutes from 

 

3

GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and 

Managing Capital Program Costs, GAO-09-3SP (Washington D.C.: March 2009). 

4

GAO, A Model of Strategic Human Capital Management, GAO-02-373SP (Washington, 

D.C.: Mar. 15, 2002). 

Page 44 GAO-09-518 Military Readiness
Appendix I: Objectives, Scope, and 

Methodology 

the DRRS Executive Committee, General and Flag Officer’s Steering 

Committee, and Battle Staff meetings—the main DRRS governance 

bodies. In addition, we reviewed key DRRS certification and approval 

documentation provided by the Human Resources Management 

Investment Review Board, such as economic viability analyses and the 

business system certification dashboard and met with Investment 

Review Board officials to determine the basis for certifying and 

approving DRRS. 

To determine the extent to which the features of DRRS have been 

implemented and are consistent with legislative requirements and DOD 

guidance, we first examined the language of Section 117 of Title 10 of the 

United States Code, which directs the Secretary of Defense to establish a 

comprehensive readiness reporting system. We identified criteria for this 

system in DOD’s directive formally establishing the system.5

 We evaluated 

the system by conducting interviews—see table 2 below for a list of these 

organizations—and receiving system demonstrations from members of the 

readiness community to determine how they used DRRS and how their 

usage compared with the criteria established for the system. We also 

conducted content and data analysis of system documents and briefing 

packages provided by the DIO and Joint Staff. In order to capture the 

broadest amount of data about the system we conducted a survey of 

readiness offices at all of the service headquarters, combatant commands, 

and the National Guard Bureau regarding how DRRS was currently being 

used and the types and amount of data available in the system.6

 In 

addition, to track the development of DRRS capabilities, we attended 

Battle Staff meetings and analyzed documentation from meetings of all the 

DRRS governance bodies. We also searched for and extracted information 

from DRRS in order to support other GAO ongoing readiness reviews. 

While our usage of the system was not intended as a formal test of the 

system, our general observations concerning system functionality and the 

range of available data were consistent with the observations of most 

other users, which were noted in our survey. 

 

5

Department of Defense Directive 7730.65, Department of Defense Readiness Reporting 

System (DRRS) (June 3, 2002). 

6

The reporting officer for U.S. Pacific Command indicated that he felt that the GAO’s 

survey did not “accurately and fully assess the usefulness and functionality of DRRS” and 

provided additional written information to support his survey answers. After consultation 

with GAO’s office of Applied Research and Methods we determined the survey did assess 

DRRS functionality. In order to capture U.S. Pacific Command’s concerns, we incorporated 

the information it provided into the report sections describing current DRRS usage. 

Page 45 GAO-09-518 Military Readiness
Appendix I: Objectives, Scope, and 

Methodology 

Table 2: Organizations Interviewed during Our Review 

Office of the Under Secretary of Defense (Personnel & Readiness), Arlington, 

Virginia 

U.S Army 

Office of the Deputy Chief of Staff (Operations), Arlington, Virginia 

U. S. Army Forces Command, Fort McPherson, Georgia 

U.S. Army Pacific, Fort Shafter, Hawaii 

Installation Management Command, Pacific Region Office, Fort Shafter, Hawaii 

U.S. Navy 

Headquarters Navy, Integrated Fleet Readiness, Arlington, Virginia 

Fleet Forces Command, Norfolk, Virginia 

U.S. Pacific Fleet, Pearl Harbor, Hawaii 

U.S. Air Force 

Headquarters, U.S. Air Force (Readiness), Arlington, Virginia 

Air Combat Command, Langley Air Force Base, Virginia 

U.S. Pacific Air Forces, Readiness Branch, Hickam Air Force Base, Hawaii 

13th Air Force, Hickam Air Force Base, Hawaii 

U.S. Marine Corps 

Headquarters Marine Corps (Readiness), Arlington, Virginia 

Marine Forces Command, Norfolk, Virginia 

Marine Forces Pacific, Camp Smith, Hawaii 

Combat Logistics Battalion 3, Kaneohe Bay, Hawaii 

U.S. Central Command, MacDill Air Force Base, Florida 

U.S. Joint Forces Command, Norfolk, Virginia 

U.S. Northern Command, Colorado Springs, Colorado 

U.S. Pacific Command, Camp Smith, Hawaii 

U.S. Special Operations Command, MacDill Air Force Base, Florida 

Source: GAO. 

We conducted our work from April 2008 through August 2009 in 

accordance with generally accepted government auditing standards. Those 

standards require that we plan and perform the audit to obtain sufficient, 

appropriate evidence to provide a reasonable basis for our findings and 

conclusions based on our audit objectives. We believe that the evidence 

obtained provides a reasonable basis for our findings and conclusions 

based on our audit objectives. 

Page 46 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and Appendix II: Detailed Analysis of DRRS Oversight 

Acquisition and Deployment Management 

and Oversight 

Our research and evaluations of information technology programs have 

shown that the ability to deliver promised system capabilities and benefits 

on time and within budget largely depends on the extent to which key 

program management disciplines are employed by an adequately staffed 

program management office. Among other things, these disciplines include 

a number of practices associated with effectively developing and 

managing system requirements, adequately testing system capabilities, and 

reliably scheduling the work to be performed. They also include 

proactively managing the program office’s human capital needs, and 

promoting program office accountability through effective program 

oversight. Department of Defense (DOD) acquisition policies and 

guidance, along with other relevant guidance, recognize the importance of 

these management and oversight disciplines.1

 As we have previously 

reported, not employing these and other program management disciplines 

increases the risk that system acquisitions will not perform as intended 

and require expensive and time-consuming rework. Defense Readiness 

Reporting System (DRRS) acquisition and deployment has for years not 

been effectively managed in accordance with these key program 

management disciplines that are recognized in DOD and other relevant 

guidance, and are fundamental to delivering a system that performs as 

intended on time and within budget. 

 

1

See for example, Department of Defense Instruction 5000.02, Operation of the Defense 

Acquisition System (Dec. 8, 2008); Defense Acquisition University, Test and Evaluation 

Management Guide, 5th ed. (Fort Belvoir, Va.: January 2005), Institute of Electrical and 

Electronic Engineers, Standard 1012-2004 for Software Verification and Validation 

(New York, N.Y.: June 8, 2005); GAO, GAO Cost Estimating and Assessment Guide: Best 

Practices for Developing and Managing Capital Program Costs, GAO-09-3SP (Washington 

D.C.: March 2009); and GAO, A Model of Strategic Human Capital Management, 

GAO-02-373SP (Washington, D.C.: Mar. 15, 2002). 

Page 47 GAO-09-518 Military Readiness 
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

Well-defined and well-managed requirements are a cornerstone of 

effective system development and acquisition. According to recognized 

guidance, documenting and implementing a disciplined process for 

developing and managing requirements can help to reduce the risks of 

producing a system that is not adequately tested, does not meet user 

needs, and does not perform as intended.2

 Effective requirements 

development and management includes, among other things, (1) 

effectively eliciting user needs early and continuously in the system lifecycle process, (2) establishing a stable baseline set of requirements and 

placing this baseline under configuration management, (3) ensuring that 

system requirements are traceable backward to higher level business or 

operational requirements (e.g., concept of operations) and forward to 

system design documents (e.g., software requirements specification) and 

test plans, and (4) controlling changes to baseline requirements. 

DRRS Requirements 

Have Not Been 

Effectively Developed 

and Managed 

DRRS requirements have not been effectively developed and managed. 

Specifically, (1) key users have only recently become engaged in 

developing requirements, (2) requirements have been experiencing 

considerable change and are not yet stable, (3) different levels of 

requirements and related test cases have not been aligned with one 

another, and (4) changes to requirements have not been effectively 

controlled. As a result, efforts to develop and deliver initial DRRS 

capabilities have taken longer than envisioned and these capabilities have 

not lived up to the readiness communities’ expectations. These failures 

increase the risk of future DRRS capabilities not meeting expectations and 

ensure that expensive and time-consuming system rework will be 

necessary. 

 

2

The Capability Maturity Model® Integration for Development, developed by the 

Software Engineering Institute of Carnegie Mellon University, defines key practices that 

are recognized hallmarks for successful organizations that, if effectively implemented, can 

greatly increase the chances of successfully developing and acquiring software and 

systems. Carnegie Mellon Software Engineering Institute, Capability Maturity Model® 

Integration for Development, version 1.2 (Pittsburgh, Pa.: August 2006). 

Page 48 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

One of the leading practices associated with effective requirements 

development is engaging system users early and continuously in the 

process of defining requirements. As we have previously reported, 

assessing user needs early in the process increases the probability of 

success in defining, designing, and delivering a system that meets user 

needs and performs as intended.3

Recent Actions Have Been 

Taken to Address Limited 

User Involvement in 

Developing and Managing 

Requirements 

To the DRRS Implementation Office’s (DIO) credit, the October 2008 

DRRS Risk Management Plan recognizes this by stating that the success of 

DRRS depends on participation and support from the broad readiness 

community, which includes combatant commands, Joint Staff, and the 

military services. However, until recently, key users were not effectively 

engaged in DRRS requirements development and management, although 

reasons vary why they have not. Specifically, DIO officials told us that 

beginning in 2002, they reached out to all user groups—combatant 

commands, Joint Staff, and the military services—in defining 

requirements. For example, they cited a July 2002 memorandum issued by 

the Office of the Under Secretary of Defense for Personnel and Readiness 

(USD P&R) that encouraged the Director of the Joint Chiefs of Staff, 

Deputy Commanders of the Combat Commands, Service Operations 

Deputies, and Directors of Defense Agencies to actively support the DRRS 

effort by ensuring that their organizations are represented at Battle Staff 

meetings.4

 However, these officials told us that the military services and 

Joint Staff chose not to participate. 

In contrast, officials from these user groups told us their involvement had 

been limited by what they characterized as difficulties in submitting 

requirements through the DRRS governance boards that were in place at 

that time. For example, an official from the Joint Forces Command said 

that the Forces Battle Staff governance board did not meet for about a 

year between 2005 and 2006. Further, the official said that the meetings 

that were held did not offer users the opportunity to discuss their 

concerns or influence the requirements process. Similarly, an official from 

the Marine Corps cited a lack of clear and transparent communication 

from the DIO as a significant impediment. 

 

3

GAO, Secure Border Initiative: DHS Needs to Address Significant Risks in Delivering 

Key Technology Investment, GAO-08-1086 (Washington, D.C.: Sept. 22, 2008). 

4

Office of the Under Secretary of Defense for Personnel and Readiness Memorandum, 

Implementing the Defense Readiness Reporting System (July 1, 2002). 

Page 49 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

Notwithstanding this lack of stakeholder involvement in setting 

requirements, the Office of USD (P&R) developed and issued a DRRS 

concept of operations in 2004, which DIO officials told us was based on 

input from the combatant commands, relevant departmental directives,5

and DRRS governance boards (e.g., Battle Staff). In our view, this 

document provided a high-level overview of proposed DRRS capabilities 

from which more detailed requirements could be derived. However, the 

concept of operations was not approved by all key players in the readiness 

community. Specifically, DIO officials stated that the document had not 

been approved by the military services and the Joint Staff. According to 

these officials, the reason for not seeking all stakeholders’ approval, and 

the decision to begin developing more detailed requirements in the 

absence of an approved concept of operations, was that the 2002 DRRS 

DOD directive provided a sufficient basis to begin developing and 

deploying what they anticipated being the initial versions of DRRS.6

In 2008, after 6 years of effort to define DRRS requirements and develop 

and deploy system capabilities, the Joint Staff, at the direction of the 

DRRS Executive Committee, conducted an analysis of DRRS 

capabilities—referred to as the “capabilities gap analysis.” To the Joint 

Staff’s credit, this analysis has appropriately focused on soliciting 

comments from the entire readiness community and on identifying any 

gaps between the DRRS requirements and the needs of this community. As 

will be discussed in the next section, this analysis resulted in 530 

additional user requirements. 

The extended period of limited involvement by key DRRS users in defining 

a concept of operations and related capabilities and requirements has 

impeded efforts to develop a clear understanding of DRRS expectations, 

constraints, and limitations, which, in turn, has contributed to significant 

delays in providing the readiness community with needed system support. 

While the recent Joint Staff action to engage the entire DRRS user 

community is a positive step towards overcoming this long-standing 

problem, it remains to be seen whether this engagement will produce 

 

5

Department of Defense Directive 7730.65, Department of Defense Readiness Reporting 

System (DRRS) (June 3, 2002). Additional requirements for DRRS have been established in 

DOD directives and instructions that govern information systems including information 

assurance, net-centric architecture, net-centric data strategy, and information system 

interoperability and supportability. 

6

Department of Defense Directive 7730.65, Department of Defense Readiness Reporting 

System (DRRS) (June 3, 2002). 

Page 50 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

agreement and commitment across the entire readiness user community 

around DRRS requirements. 

DRRS Requirements Are 

Not Stable 

As previously noted, establishing an authoritative set of baseline 

requirements prior to system design and development is necessary to 

design, develop, and deliver a system that performs as intended and meets 

users’ operational needs.7

 In general, a baselined set of requirements are 

those that are defined to the point that extensive changes are not 

expected, placed under configuration management, and formally 

controlled.8

DRRS requirements are currently in a state of flux. Specifically, the fact 

that 530 new user requirements have recently been identified means that 

the suite of requirements documentation associated with the system will 

need to be changed and thus are not stable. To illustrate, program officials 

told us that, as of late February 2009, these 530 new requirements had not 

been fully evaluated by the DIO and DRRS governance boards and thus 

not yet approved. As a result, their impact on the program is not clear. 

Compounding this instability in the DRRS requirements is the fact that 

additional changes are envisioned. According to program officials, the 

changes resulting from the gap analysis and reflected in the latest version 

of the DRRS concept of operations, which was approved by the DRRS 

Executive Committee in January 2009, have yet to be reflected in other 

requirements documents, such as the detailed system requirements. 

Although defining and developing requirements is inherently an iterative 

process, having a baseline set of requirements that are stable is a 

prerequisite to effective and efficient development of an operationally 

capable and suitable system. Without them, the DIO will not be able to 

deliver a system that meets user needs on time, and it is unlikely that 

future development and deployment efforts will produce better results. 

 

7

GAO-08-1086. 

8

The Capability Maturity Model® Integration for Development, developed by the 

Software Engineering Institute of Carnegie Mellon University, defines key practices that 

are recognized hallmarks for successful organizations that, if effectively implemented, can 

greatly increase the chances of successfully developing and acquiring software and 

systems. Carnegie Mellon Software Engineering Institute, Capability Maturity Model® 

Integration for Development, Version 1.2 (Pittsburgh, Pa.: August 2006). 

Page 51 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

One of the leading practices associated with developing and managing 

requirements is maintaining bidirectional traceability from high-level 

operational requirements (e.g., concept of operations and functional 

requirements) through detailed lower-level requirements and design 

documents (e.g., software requirements specification) to test cases. Such 

traceability is often accomplished through the use of a requirements 

traceability matrix, which serves as a crosswalk between different levels 

of related requirements, design, and testing documentation. The DRRS 

program management plan recognizes the importance of traceability, 

stating that requirements are to be documented and linked to acceptance 

tests, scripts, and criteria. 

Alignment among 

Requirements and Related 

System Design and Testing 

Artifacts Has Not Been 

Demonstrated 

Despite the importance of traceability, DIO officials could not demonstrate 

that requirements and related system design and testing artifacts are 

properly aligned. Specifically, we attempted on three separate occasions 

to verify the traceability of system requirements backward to higher-level 

requirements and forward to lower-level software specifications and test 

cases, and each time we found that traceability did not exist. Each attempt 

is discussed here: 

• In November 2008, our analysis of the requirements traceability matrix 

and the software requirements specification showed significant 

inconsistencies. For example, the traceability matrix did not include 29 

requirements that were included in the software requirements 

specification. As a result, we did not have an authoritative set of 

requirements to use to generate a random sample of requirements to 

trace. Program officials attributed the inconsistencies to delays in 

updating all the documents to reflect the aforementioned capability gap 

analysis. They also stated that these documents would be updated by 

December 2008. 

• In December 2008, we used an updated requirements traceability 

matrix to generate a randomized sample of 60 software requirements 

specifications and observed a DIO demonstration of the traceability of 

this sample. However, DIO officials were unable to demonstrate for us 

that these specifications could be traced backward to higher-level 

requirements and forward to test cases. Specifically, attempts to trace 

the first 21 requirements forward to test cases failed, and DIO officials 

stated that they could not trace the 60 requirements backward because 

the associated requirements documents were still being updated. 

According to the officials, 11 of the 21 could not be traced forward 

because these were implemented prior to 2006 and the related test 

information was not maintained by the program office but rather was 

at the development contractor’s site. They added that the remaining 10 

Page 52 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

either lacked test case information or test results. 

• In February 2009, we used an updated DIO-provided requirements 

traceability matrix, a capabilities requirement document, and software 

requirements specification to generate another randomized sample of 

60 detailed specifications. We then observed the development 

contractor’s demonstration of traceability using the contractor’s 

requirements management tool. Because of time constraints, this 

demonstration focused on 46 of the 60 requirements, and it showed 

that adequate traceability still did not exist. Specifically, 38 of the 46 

could not be traced backward to higher-level requirements or forward 

to test cases. This means that about 83 percent of the DRRS 

specifications (95 percent degree of confidence of being between 72 

and 91 percent) were not traceable. Of the 38, 14 did not trace because 

of incomplete traceability documentation; 5 due to inconsistent 

traceability documentation; 3 due to requirements not being resident in 

the tracking tool; and 16 due to no actual development work being 

started. 

In addition, none of the 46 requirements were traceable to the January 

2009 concept of operations. According to contractor officials, this is 

because the newly developed capability requirements document is 

considered to be a superset of the concept of operations, and thus 

traceability to this new document is their focus. However, they were 

unable to demonstrate traceability to the requirements in either the 

capability requirements document or the concept of operations. Further, 

we also found numerous inconsistencies among the capabilities 

requirements document, software requirements specification, and the 

requirements traceability matrix. For example, 15 capabilities 

requirements listed on the traceability matrix were not listed in the 

capabilities requirements document, but were listed in the updated 

software requirements specification, dated February 2009. Further, one 

requirement listed in the traceability matrix was not listed in either of 

these documents. One possible reason for these inconsistencies is that the 

traceability matrix was prepared manually, rather than being automatically 

generated from the tool, which would increase the probability of these and 

other discrepancies caused by human error. Another reason cited by 

program officials is that test results that occurred prior to October 2006 

had yet to be fully recorded in the contractor’s tracking tool. 

DIO and contractor officials attributed the absence of adequate 

requirements traceability to the ongoing instability in requirements and 

magnitude of the effort to update the chain of preexisting and new 

requirements documentation. They added that they expect traceability to 

Page 53 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

improve as requirements become more stable and the documentation is 

updated. Regardless, the DIO has and continues to invest in the 

development of DRRS in the absence of requirements traceability. Without 

traceable requirements, the DIO does not have a sufficient basis for 

knowing that the scope of the design, development, and testing efforts will 

produce a system solution on time and on budget and that will meet users’ 

operational needs and perform as intended. As a result, the risk is 

significant that expensive and time-consuming system rework will be 

required. 

Changes to Requirements 

Are Not Being Effectively 

Controlled 

Adopting a disciplined process for reviewing and accepting changes to an 

approved and authoritative baseline set of requirements in light of the 

estimated costs, benefits, and risk of each proposed change is a 

recognized best practice. Elements of a disciplined process include: (1) 

formally documenting a requirements change process; (2) adopting 

objective criteria for considering proposed changes, such as estimated 

cost or schedule impact; and (3) rigorously adhering to the documented 

change control process. 

Since the inception of the program in 2002, DRRS has been developed and 

managed without a formally documented and approved process for 

managing changes to system requirements. Further, while requirements 

management and change control plans were developed in 2006 by the 

DRRS software development contractor, according to DIO officials, the 

plans were not adequate. For example, the plans did not detail how DRRS 

user requirements were collected or how objective factors, such as cost, 

impacted development decisions. 

To address these problems, the Joint Staff developed what it referred to as 

a conceptual requirements change control process in February 2008, 

which was to serve as a basis for the DIO to develop more detailed plans 

that could be implemented. Eleven months later, in January 2009, the DIO 

drafted more detailed plans—a DRRS requirements management plan and 

a DRRS requirements configuration management plan, the latter of which 

the DIO updated in March 2009. Specifically, the draft plans call for new 

DRRS requirements to be collected using an online tool and reviewed by 

the DIO to determine whether the requirement constitutes a major change 

to DRRS. Once approved, the DIO and the contractor are to provide the 

Battle Staff with a formatted report specifying the anticipated benefit of 

each new requirement and an initial analysis of the cost and performance 

impact. The Battle Staff then is to prioritize the requirement based on the 

DIO’s impact analysis. If the issue cannot be resolved by the Battle Staff, it 

Page 54 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

is to be elevated to the senior oversight bodies (i.e., the General Officer’s 

Steering Committee and the DRRS Executive Committee). After a 

requirement has been approved, the software developer may prepare a 

more detailed “customer acceptance document” that analyzes the potential 

cost, schedule, and quality impact to DRRS objectives, which is then to be 

reviewed by the DIO at subsequent Change Control Board meetings. 

However, according to the user community and the DIO Director, the 

revised plans have not been submitted for review and approval to the 

DRRS community. Specifically, they stated that only a proposed process 

flow diagram was briefed at the Battle Staff, and according to them, the 

change control process was still being evaluated. Moreover, the DIO has 

yet to implement key aspects of its draft plans. For example, the DRRS 

Chief Engineer stated that until recently, the DIO had continued to accept 

changes to DRRS requirements that were submitted outside of the 

designated online tool. In addition, the reports that the Battle Staff are to 

use in making their requirement change determination do not include the 

anticipated benefit and estimated cost or schedule impact of new 

requirements. Rather, these reports only include the estimated number of 

hours necessary to complete work on a proposed requirement. Moreover, 

contractor officials and users from the Special Operations Command told 

us that cost or schedule impacts have rarely been discussed at the Battle 

Staff or Change Control Board meetings. Our analysis of minutes from 

change control meetings confirmed this. Furthermore, the DRRS Chief 

Engineer stated that the customer acceptance documents have only 

recently been used. 

Until the DIO effectively controls requirements changes, it increases the 

risk of needed DRRS capabilities taking longer and costing more to deliver 

than necessary. 

Page 55 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

Effective system testing is essential to successfully developing and 

deploying systems like DRRS. According to DOD and other relevant 

guidance, system testing should be progressive, meaning that it should 

consist of a series of test events that first focus on the performance of 

individual system components, then on the performance of integrated 

system components, followed by system-level tests that focus on whether 

the system (or major system increments) are acceptable, interoperable 

with related systems, and operationally suitable to users. 9

 For this series 

of related test events to be conducted effectively, (1) each test event needs 

to be executed in accordance with well-defined test plans, (2) the results 

of each test event need to be captured and used to ensure that problems 

discovered are disclosed and corrected, and (3) all test events need to be 

governed by a well-defined test management structure. 

DRRS Testing Has Not 

Been Adequately 

Performed and Key 

Test Management 

Structures and 

Controls Have Not 

Been Established 

Despite acquiring and partially deploying a subset of DRRS increments, 

the DIO cannot demonstrate that it has adequately tested any of these 

system increments, referred to as system releases and subreleases. 

Specifically, (1) the test events for already acquired, as well as currently 

deployed and operating, DRRS releases and subreleases were not based on 

well-defined plans, and test events have not been fully executed in 

accordance with plans or executed in a verifiable manner, or both; (2) the 

results of executed test events have not been captured and used to ensure 

that problems discovered were disclosed to decision makers and 

ultimately corrected; and (3) the DIO has not established an effective test 

management structure to include, for example, a clear assignment of test 

management roles and responsibilities, or a reliable schedule of planned 

test events. Compounding this absence of test management structures and 

controls is the fact that the DIO has yet to define how the series of system 

releases and subreleases relate to its recent restructuring of DRRS 

increments into a series of 10 modules. Collectively, this means that it is 

unlikely that already developed and deployed DRRS capabilities can 

perform as intended and meet user operational needs. Equally doubtful are 

the chances that the DIO can adequately ensure that yet-to-be developed 

DRRS capabilities will meet expectations. 

 

9

See for example, Department of Defense Instruction 5000.02, Operation of the Defense 

Acquisition System; Defense Acquisition University, Test and Evaluation Management 

Guide; Institute of Electrical and Electronics Engineers, Standard 1012-2004 for Software 

Verification and Validation; and Software Engineering Institute, Capability Maturity 

Model Integration for Acquisition version 1.2 (Pittsburgh, Pa.: November 2007). 

Page 56 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

Key tests required for already developed and partially fielded DRRS 

increments either did not have well-defined test plans, or these tests have 

yet to be conducted. According to program documentation, system 

releases and subreleases have been subjected to what are described as 30-

day test cycles, during which: (1) a Software Test Plan is updated if 

applicable, (2) test procedures are developed and incorporated in the 

Software Test Description, (3) a series of developmental tests on each 

release/subrelease is performed, (4) weekly meetings are held to review 

software defects identified during testing, (5) final test results are 

summarized within the Software Test Report and Software Version 

Description, and (6) the release/subrelease is made available to users. 

DIO Has Not Adequately 

Planned and Executed Key 

Test Events 

However, the program office has yet to provide us with the developmental 

test plans and procedures for each release/subrelease that to-date has 

either been developed or fielded. Instead, it provided us with a Software 

Test Plan and two Software Test Descriptions that it said applied to two 

subreleases within release 4.0. However, available information indicates 

that DRRS subreleases total at least 63, which means that we have yet to 

receive the test plans and procedures for 61. Further, the test plan that we 

were provided is generic in nature, meaning that it was not customized to 

apply specifically to the two subreleases within Release 4.0. Moreover, the 

plan and procedures lack important elements specified in industry 

guidance.10 For example, the test plan does not include a schedule of 

activities to be performed or defined roles and responsibilities for 

performing them. Also, the test plan does not consistently include test 

entrance and exit criteria, a test defect management process, and metrics 

for measuring progress. 

Moreover, the DIO has yet to demonstrate that it has performed other key 

developmental and operational test events that are required before the 

software is fielded for operational use. According to DIO officials, 

developmental testing concludes only after system integration testing and 

system acceptance testing, respectively, are performed. Further, following 

developmental testing, the Joint Interoperability Test Command (JITC), 

which is a DOD independent test organization, is to conduct both 

interoperability and operational testing before the system is deployed and 

put into production (i.e., used operationally). Although increments of 

 

10See for example, Institute of Electrical and Electronics Engineers, Standard 829-2008 

Standard for Software and System Test Documentation (New York, N.Y.: July 18, 2008) 

and Software Engineering Institute, Capability Maturity Model Integration for 

Acquisition, version 1.2. 

Page 57 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

DRRS functionality have been put into production, the DIO has not 

performed system integration testing, system acceptance testing, or 

operational testing on any DRRS release or subrelease. Further, JITC 

documentation shows that while an interoperability test of an increment 

of DRRS functionality known as ESORTS was conducted, this test did not 

result in an interoperability certification.11 According to JITC and Joint 

Staff officials, this was because the DIO did not address JITC’s identified 

limitations to the program’s Information Support Plan, which identifies 

essential information-exchange sharing strategies between interdependent 

systems that are needed for interoperability certification. Without 

interoperability certification, the ability of the DRRS to exchange accurate 

and timely readiness data with other critical systems, such as the Joint 

Operation Planning and Execution System, cannot be ensured. 

Similarly, while DIO officials stated that acceptance testing has occurred 

for one increment of DRRS functionality known as SORTSREP, the DIO 

does not have either a finalized acceptance test plan or documented test 

results. 12 Furthermore, the integrated master schedule (last updated in 

April 2009) shows that acceptance testing is not to occur until the 

July/August 2009 time frame, which is about 15 months later than 

originally envisioned. Moreover, this delay in acceptance testing has in 

turn delayed interoperability and operational testing by 16 months 

(May/June 2008 to September/November 2009), according to the latest 

schedule. Program officials attributed the delays to Marine Corps and Air 

Force concerns about the quality of SORTSREP. 

Until the DIO has effectively planned and executed the series of tests 

needed to demonstrate the readiness of DRRS increments to operate in a 

production environment, the risk of fielded system increments not 

performing as intended and requiring expensive rework to correct will be 

increased, and DOD will continue to experience delays in delivering 

mission-critical system capabilities to its readiness community. 

 

11According to the Concept of Operations, ESORTS is to provide current readiness status 

for operational forces and defense support organizations in terms of their ability to perform 

their mission essential tasks. 

12According to the draft SORTSREP Test and Evaluation Master Plan, SORTSREP is a tool 

to capture and input military departments’ readiness data requirements into a readiness 

reporting database. 

Page 58 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

Available results of tests performed on already developed and at least 

partially deployed DRRS releases/subreleases show that the test results 

have not been effectively captured and analyzed, and have not been fully 

reported. Moreover, test results for other releases/subreleases do not 

exist, thus minimizing the value of any testing that has been performed. 

According to relevant guidance, effective system testing includes 

recording the results of executing each test procedure and test case as 

well as capturing, analyzing, correcting, and disclosing to decision makers 

problems found during testing (test defects). 13 It also includes ensuring 

that test entry and exit criteria are met before beginning and ending, 

respectively, a given test event. 

DIO Has Not Adequately 

Documented and Reported 

Test Results 

The DIO does not have test results of all developed and tested DRRS 

releases and subreleases. Specifically, program officials provided us with 

the Software Test Reports and Software Version Descriptions that, based 

on program documentation, represent the full set of test results for three 

subreleases and a partial set of test results for 40 subreleases within 

releases 1.0, 3.0, and 4.0. However, as noted earlier, DRRS subreleases 

total at least 63, which means that test reports and results for at least 20 

subreleases do not exist. Moreover, the test reports and version 

descriptions that we received do not consistently include key elements 

provided for in industry guidance, such as a documented assessment of 

system capabilities and limitations, entrance/exit criteria status, an 

assessment as to whether the applicable requirements/thresholds were 

met, and unresolved defects and applicable resolution plans. 14 This 

information is important because it assists in determining and disclosing 

to decision makers current system performance and efforts needed to 

resolve known problems, and provides program officials with a needed 

basis for ensuring that a system increment is ready to move forward and 

be used. Without this information, the quality and readiness of a system is 

not clear. 

Furthermore, the DIO does not have detailed test defect documentation 

associated with all executed DRRS test events. According to relevant 

 

13See for example, Defense Acquisition University, Defense Acquisition Guidebook 

(Arlington, Va.: November 2006); Institute of Electrical and Electronics Engineers, 

Standard 1012-2004 for Software Verification and Validation; Software Engineering 

Institute, Capability Maturity Model Integration for Acquisition, version 1.2. 

14See for example, Institute of Electrical and Electronics Engineers, Standard 829-2008 

Standard for Software and System Test Documentation and Software Engineering 

Institute, Capability Maturity Model Integration for Acquisition version 1.2. 

Page 59 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

guidance, defect documentation should, among other things, identify each 

issue discovered, assign each a priority/criticality level, and provide for 

each a strategy for resolution or mitigation. 15 In lieu of detailed test defect 

documentation, program officials referred us to the above-mentioned 

Software Version Descriptions, and stated that additional information is 

available in an automated tool, known as the ISI BugTracker, that it uses 

to capture, among other things, defect data. However, these documents do 

not include the above-cited defect information, and defect data for each 

test event do not exist in the ISI BugTracker. 

Compounding the absence and limitations of test results are weaknesses 

in the program office’s process for collecting such results during test 

execution. According to relevant guidance, test results are to be collected 

and stored according to defined procedures and placed under appropriate 

levels of control. 16 Furthermore, these test results are to be reviewed 

against the source data to ensure that they are complete, accurate, and 

current. For DRRS, the program office is following a partially 

undocumented, manual process for collecting and storing test results and 

defects that involves a database and layers of documentation. As 

explained by program officials and documentation, the DIO initially 

documents defects and completed test case results manually on paper 

forms, and once the defect is approved by the test lead, it is input into a 

database. However, it does not have written procedures governing the 

entire process, and thus key controls, such as assigned levels of authority 

for database read/write access, are not clearly defined. Moreover, once the 

test results and defects are input into the database, traceability back to the 

original test data for data integrity checks cannot be established because 

the program office does not retain these original data sets. Program 

officials acknowledged these internal control weaknesses and stated that 

they intend to adopt a new test management tool that will allow them to 

capture in a single database test cases, test results, and test defects. 

Furthermore, the DIO’s process for analyzing and resolving test defects 

has limitations. According to relevant guidance and the draft SORTSREP 

Test and Evaluation Master Plan (TEMP), defects should be analyzed and 

 

15See for example, Institute for Electrical and Electronics Engineers, Standard 829-2008 

Standard for Software and System Documentation. 

16See for example, Software Engineering Institute, Capability Maturity Model Integration 

for Acquisition, version 1.2. 

Page 60 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

prioritized. 17 However, the program office has not established a standard 

definition for defect priority levels identified during testing. For example, 

the various release/subrelease test reports (dated through January 2009) 

prioritize defects on a scale of 1-3, where a priority 2 means critical but 

with a viable workaround. In contrast, the SORTSREP TEMP (dated 

January 2009) prioritizes defects on a scale of 1-5, where a priority 2 

means an error that adversely affects the accomplishment of an 

operational or mission essential function in accordance with official 

requirements so as to degrade performance and for which no alternative 

work around solution exists. By not using standard priority definitions for 

categorizing defects, the program office cannot ensure that it has an 

accurate and useful understanding of the scope and magnitude of the 

problems it is facing at any given time, and it will not know if it is 

addressing the highest priority issues first. 

In addition, the DIO has not ensured that critical defects are corrected 

prior to concluding a given test event. According to relevant guidance and 

the draft SORTSREP TEMP, all critical and high defects should be 

resolved prior to the conclusion of a test event, and all test results should 

be reviewed for validity and completeness.18 However, the DRRS 

release/subrelease test reports show that the DIO concluded five test 

events even though each had at least 11 open critical defects (priority 1 

defects with no workaround). Moreover, these numbers of open critical 

defects are potentially higher because they do not include defects for 

which a solution was identified but the solution failed during regression 

testing and do not include defects that were dismissed because the 

program official was unable to recreate it. 

Until the DIO adequately documents and reports the test results, and 

ensures that severe problems discovered are corrected prior to concluding 

a given test event, the probability of incomplete test coverage, and 

 

17See for example, GAO, Office of Personnel Management: Improvements Needed to 

Ensure Successful Retirement Systems Modernization, GAO-08-345 (Washington, D.C.: 

January 2008); Institute of Electrical and Electronics Engineers, Standard 1012-2004 for 

Software Verification and Validation; and Institute of Electrical and Electronics 

Engineers, Standard 1012-1986 for Software Verification and Validation Plans (New 

York, N.Y.: Nov. 14, 1986). 

18See for example, GAO-08-345; Institute of Electrical and Electronics Engineers, Standard 

1012-2004 for Software Verification and Validation; and Institute of Electrical and 

Electronics Engineers, Standard 1012-1986 for Software Verification and Validation 

Plans. 

Page 61 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

insufficient and invalid test results, is increased, thus unnecessarily 

increasing the risk of DRRS not meeting mission needs or otherwise not 

performing as intended. 

DIO Has Not Established 

an Effective Test 

Management Structure 

The DIO does not have an effective test management structure, to include 

a well-defined overall test management plan that clearly assigns test 

management roles and responsibilities, a designated test management lead 

and a supporting working group, and a reliable schedule of planned test 

events. According to relevant guidance, these aspects of test management 

are essential to adequately planning, executing, and reporting a program’s 

series of test events. 19

Although the program has been underway for 8 years, it did not have an 

overarching DRRS TEMP until very recently (February 2009), and this plan 

is still in draft and has yet to be approved. Further, this draft TEMP does 

not clearly define DRRS test management roles and responsibilities, such 

as those of the test manager, and it does not include a reliable schedule of 

test events that reflect the program’s recent restructuring of its software 

releases/subreleases into 10 modules. According to DIO officials, they 

recently decided not to approve this overarching TEMP. Instead, they said 

that they now intend to have individual TEMPs for each of the recently 

defined 10 modules, and to have supporting test plans for each module’s 

respective developmental and operational test events. According to 

program documentation, three individual TEMPs are under development 

(i.e., SORTSREP tool and the Mission Readiness and Readiness Review 

modules). However, drafts of these TEMPs also do not clearly define test 

entrance and exit criteria, test funding requirements, an integrated test 

program schedule, and the respective test management roles and 

responsibilities. For example, while the draft SORTSREP TEMP identifies 

the roles and responsibilities of some players, such as the test manager, 

the personnel or organization that is to be responsible is not always 

identified. In addition, while the various players in the user community are 

identified (i.e., military services, combatant commands), their associated 

roles or responsibilities are not. 

 

19See for example, Department of Defense Instruction 5000.02, Operation of the Defense 

Acquisition System; Defense Acquisition University, Defense Acquisition Guidebook; 

Software Engineering Institute, Capability Maturity Model Integration for Acquisition, 

version 1.2; Institute for Electrical and Electronics Engineers, Standard 829-2008 

Software and System Test Documentation (New York, N.Y.: July 18, 2008). 

Page 62 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

Furthermore, the DIO has yet to designate a test management lead and 

establish an effective test management working group. According to 

relevant guidance, test management responsibility and authority should be 

assigned to an individual, and this individual should be supported by a 

working integrated product team that includes program office and 

operational testing representatives.20 Among other things, the working 

integrated product team is to develop an overall system test strategy. 

However, DIO officials told us that the test manager position has been 

vacant, and this position is now being temporarily filled by the program’s 

chief engineer, who is a contractor. Furthermore, although DRRS system 

development began prior to 2004, a charter for a test and evaluation 

working integrated product team was not issued until February 2009. 

According to DIO officials, the delay in establishing the team has not had 

any impact because of corresponding delays in finalizing the program’s 

overall test strategy. However, this statement is not consistent with the 

Defense Acquisition Guidebook, which states that two of the key products 

of the working integrated product team are the program’s test strategy and 

TEMP. Further, JITC officials stated that the lack of a test manager and an 

active test and evaluation working integrated product team have reduced 

the effectiveness of DRRS testing activities. As a result, they stated that 

they have had to compensate by conducting individual meetings with the 

user community to discuss and receive documentation to support their 

operational and interoperability test planning efforts. 

Moreover, the DIO has yet to establish a reliable schedule of planned test 

events. For example, the schedule in the TEMPs is not consistent with 

either the integrated master schedule or the developmental test plans. 

Specifically, the draft SORTSREP TEMP (last updated in January 2009) 

identifies SORTSREP developmental testing occurring through January 

2009 and ending in early February 2009, while the integrated master 

schedule (last updated in April 2009) shows SORTSREP development 

testing as occurring in the July/August 2009 time frame. In addition, while 

program officials said that development testing for SORTSREP has 

occurred, the associated development test plans (e.g., system integration 

and system acceptance test plans) had no established dates for test 

execution, and are still in draft. As another example, a module referred to 

as “Mission Readiness” had no established dates for test execution in its 

 

20See for example, Defense Acquisition University, Defense Acquisition Guidebook; 

Software Engineering Institute, Capability Maturity Model Integration for Acquisition, 

version 1.2; Institute for Electrical and Electronics Engineers, Standard 829-2008 

Software and System Test Documentation. 

Page 63 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

TEMP, and while program documentation indicates that this module 

completed development testing in December 2008, the associated 

development test plans (e.g., system integration and system acceptance 

test plans) do not exist.21

In addition, the DIO has yet to define in its draft TEMPs how the 

development and testing to date of at least 63 subreleases relate to the 

development and testing of the recently established 10 system modules. 

According to Joint Staff and JITC officials, they do not know how the 

releases/subreleases relate to the modules, and attributed this to a lack of 

an approved description for each module that includes what functionality 

each is intended to provide. Furthermore, the high-level schedule in the 

TEMP does not describe what test events for the DRRS 

releases/subreleases that have already been developed and deployed relate 

to the development test efforts planned for the respective modules. These 

problems in linking release/subrelease test events to module test events 

limit the DIO and JITC in leveraging the testing already completed, which 

in turn will impact the program’s ability to meet cost, schedule, and 

performance expectations. 

Collectively, the weaknesses in this program’s test management structure 

increase the chances that the deployed system will not meet certification 

and operational requirements, and will not perform as intended. 

The success of any program depends in part on having a reliable schedule 

that defines, among other things, when work activities will occur, how 

long they will take, and how they are related to one another. As such, the 

schedule not only provides a road map for the systematic execution of a 

program, but also provides the means by which to gauge progress, identify 

and address potential problems, and promote accountability. From its 

inception in 2002 until November 2008, the DIO did not have an integrated 

master schedule. Moreover, the only milestone that we could identify for 

the program prior to November 2008 was the date that DRRS was to 

achieve full operational capability, which was originally estimated to 

occur in fiscal year 2007, but later slipped to fiscal year 2008 and then 

fiscal year 2011, and is now fiscal year 2014—a 7-year delay. 

DRRS Has Not Been 

Guided by a Reliable 

Schedule 

 

21According to the Mission Readiness TEMP, Mission Readiness is to create a common 

standard metric for assessing readiness by capability that can be uniformly applied 

throughout the department. 

Page 64 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

In addition, the DRRS integrated master schedule that was developed in 

November 2008, and was updated in January 2009 and again in April 2009 

to address limitations that we identified and shared with the program 

office, is still not reliable. Specifically, our research has identified nine 

practices associated with developing and maintaining a reliable schedule.22

These practices are (1) capturing all key activities, (2) sequencing all key 

activities, (3) assigning resources to all key activities, (4) integrating all 

key activities horizontally and vertically, (5) establishing the duration of all 

key activities, (6) establishing the critical path for all key activities, (7) 

identifying float between key activities, (8) conducting a schedule risk 

analysis, and (9) updating the schedule using logic and durations to 

determine the dates for all key activities.23 However, the program’s latest 

integrated master schedule does not address three of the practices and 

only partially addresses the remaining six. For example, the schedule does 

not establish a critical path for all key activities, include a schedule risk 

analysis, and it is not being updated using logic and durations to determine 

the dates for all key activities. Further, it does not fully capture, sequence, 

and establish the duration of all key work activities; fully assign resources 

to all key work activities; fully integrate all of these activities horizontally 

and vertically; and fully identify the amount of float—the time that a 

predecessor activity can slip before the delay affects successor activities—

between these activities. These practices are fundamental to producing a 

sufficiently reliable schedule baseline that can be used to measure 

progress and forecast slippages. (See table 3 for the results of our analyses 

relative to each of the nine practices.) 

 

22GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and 

Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: March 2009). 

23Float is the amount of time an activity can slip before affecting the critical path. 

Page 65 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

Table 3: DRRS Satisfaction of Nine Schedule-Estimating Key Practices 

Practice Explanation Practice met? GAO analysis 

Capturing all activities The schedule should reflect all key activities 

(e.g., steps, events, outcomes, etc.) as 

defined in the program’s work breakdown 

structure, to include activities to be performed 

by both the government and its contractors 

Partially The schedule reflects many key activities 

as defined in the program’s work 

breakdown structure. However, key 

activities are identified only as milestones 

rather than in terms of work to be 

performed and accomplished to achieve 

the milestones. Thus, the schedule does 

not account for all work to be performed. 

As a result, the reliability of the milestones 

is questionable. 

Sequencing all activities The schedule’s activities need to be logically 

sequenced in the order that they are to be 

carried out. In particular, activities that must 

finish prior to the start of other activities (i.e., 

predecessor activities) as well as activities 

that cannot begin until other activities are 

completed (i.e., successor activities) should 

be identified. By doing so, interdependencies 

among activities that collectively lead to the 

accomplishment of key events or milestones 

can be established and used as a basis for 

guiding work and measuring progress. 

Partially The schedule identifies some predecessor 

and successor activities, but not all. 

Specifically, out of 290 key activities, 139 

do not identify predecessor activities and 

121 do not identify successor activities. 

DIO officials stated that this is because 

interdependencies are discussed and 

addressed at various meetings, and thus 

do not need to be in the schedule. 

However, recognition of such 

interdependencies in the schedule is 

necessary to logically link tasks and 

events and thereby calculate dates and 

predict changes in the future. Without 

proper linkages, tasks that slip early in the 

schedule do not transmit delays to tasks 

that should depend on them. By not 

logically linking all key activities, the 

schedule does not provide a sufficient 

basis for understanding the program as a 

whole, and confidence that the right dates 

or critical paths are represented is low. 

This means that the DIO cannot use its 

schedule to identify disconnects as well as 

hidden opportunities, and cannot 

otherwise promote efficiency and 

accuracy, and control the program by 

comparing actual to planned progress. 

Page 66 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

Practice Explanation Practice met? GAO analysis 

Assigning resources to all 

activities 

The schedule should realistically reflect what 

resources (i.e., labor, material, and 

overhead) are needed to do the work, 

whether all required resources will be 

available when they are needed, and whether 

any funding or time constraints exist. 

Partially The schedule identifies 15 resources that 

are assigned to various activities. 

However, important information is not 

included, which hampers its usefulness. 

For example, one benefit of loading 

resource information is to ensure that 

resources in short supply will not be 

overscheduled in any time period. 

However, the schedule does not define the 

amount of each resource that is available, 

but instead assumes only one unit of each 

resource is available. As a result, 10 of the 

15 types of resources (e.g., information 

assurance and test and evaluation) in the 

schedule are overscheduled and thus 

estimated completion dates are not 

realistic. Further, the schedule does not 

identify the costs of the resources. 

Knowing the cost of resources is 

important, because some resources, such 

as labor, can cost more if the program 

takes longer. 

Establishing the duration 

of all activities 

The schedule should realistically reflect how 

long each activity will take to execute. In 

determining the duration of each activity, the 

same rationale, historical data, and 

assumptions used for cost estimating should 

be used for schedule estimating. Further, 

these durations should be as short as 

possible, and they should have specific start 

and end dates. Excessively long periods 

needed to execute an activity (i.e., greater 

than 2-3 months) should prompt further 

decomposition of the activity so that shorter 

execution durations will result. 

Partially The schedule establishes the duration of 

key activities and includes specific start 

and end dates. However, the activities are 

not always of short duration. For example, 

23 activities have remaining durations that 

exceed 100 days (108 to 551 days). By 

having a schedule summarized at too high 

a level, the program runs the risk of 

masking critical work elements within the 

key activities associated with executing the 

program and fails to show the risk 

management approaches being used. 

Further, it risks masking the schedule’s 

true critical path. 

Integrating schedule 

activities horizontally and 

vertically 

The schedule should be horizontally 

integrated, meaning that it should link the 

products and outcomes associated with 

already sequenced activities. These links are 

commonly referred to as “hand offs” and 

serve to verify that activities are arranged in 

the right order to achieve aggregated 

products or outcomes. The schedule should 

also be vertically integrated, meaning that 

traceability exists among varying levels of 

activities and supporting tasks and subtasks. 

Such mapping or alignment among levels 

enables different groups to work to the same 

master schedule. 

Partially The schedule is not horizontally integrated, 

meaning that the activities are not 

arranged in the right order to achieve 

aggregated products or outcomes. This is 

because, as previously noted, many 

activities do not identify interdependencies 

(predecessor and successor activities). As 

a result, management lacks information on 

how a slippage in completing one activity 

will affect others. In contrast, the schedule 

is vertically integrated, meaning that for 

those key activities that are identified, 

traceability exists among varying levels of 

activities and that lower-level activities are 

within the constraints of higher-level 

activities. 

Page 67 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

Practice Explanation Practice met? GAO analysis 

Establishing the critical 

path for all activities 

The critical path—the longest duration path 

through the sequenced list of key activities—

should be identified using scheduling 

software. The establishment of a program’s 

critical path is necessary for examining the 

effects of any activity slipping along this path. 

High-risk activities along or near the critical 

path should also be identified and associated 

time impacts of these risks should be 

reflected in the schedule. 

No The schedule does not identify a valid 

critical path because there is no logical link 

between the program’s key activities. 

Instead, the activities are presented as 

being independent from one another. 

Without a critical path, management 

cannot know the activities critical to the ontime completion of DRRS, or the impact of 

any changes to activities on the path. 

Identifying float between 

activities 

The schedule should identify float time so 

that schedule flexibility can be determined. 

As a general rule, activities along the critical 

path typically have the least amount of float 

time. 

Partially The schedule identifies the amount of float 

allocated to key activities. However, the 

amount of float associated with 56 of these 

activities is unusually large (100 – 461 

days). Such large amounts of float time 

can be attributed to the fact that many of 

the activities do not identify linkages to 

other activities (predecessor or successor 

activities). In addition, the amount of float 

between activities is of questionable 

accuracy because activity dependencies 

(predecessor or successor activities) are 

often not identified. Instead, the start and 

completion dates for many activities are 

imposed, rather than based on logic, such 

as an activity not starting until its 

predecessor is completed. Further, it is 

unclear whether activities along the critical 

path have the least amount of float 

because, as previously noted, the 

schedule does not have a valid critical 

path. 

Page 68 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

Practice Explanation Practice met? GAO analysis 

Conducting a schedule 

risk analysis 

A schedule risk analysis uses a good critical 

path method schedule and data about project 

schedule risks as well as Monte Carlo 

simulationa techniques to predict the level of 

confidence in meeting a program’s 

completion date, the amount of time 

contingency needed for a level of confidence, 

and the identification of high-priority risks. 

This analysis focuses not only on critical path 

activities but also on other schedule paths 

that may become critical. Schedule reserve 

for contingencies should be calculated by 

performing a schedule risk analysis. As a 

general rule, the reserve should be held by 

the project or program manager and applied 

as needed to those activities that take longer 

than scheduled because of the identified 

risks. Reserves of time should not be 

apportioned in advance to any specific 

activity since the risks that will actually occur 

and the magnitude of their impact is not 

known in advance. 

No The program office did not perform a 

schedule risk analysis. Without this 

analysis, the office cannot sufficiently 

understand the level of confidence in 

meeting the program’s completion date 

and identifying reserves for contingencies. 

Updating the schedule 

using logic and durations 

to determine the dates 

The schedule should use logic and durations 

in order to reflect realistic start and 

completion dates for program activities. The 

schedule should be continually monitored to 

determine when forecasted completion dates 

differ from the planned dates, which can be 

used to determine whether schedule 

variances will affect downstream work. 

Maintaining the integrity of the schedule logic 

is not only necessary to reflect true status, 

but is also required before conducting a 

schedule risk analysis. The schedule should 

avoid logic overrides and artificial constraint 

dates that are chosen to create a certain 

result on paper. To ensure that the schedule 

is properly updated, individuals trained in 

critical path method scheduling should be 

responsible for updating the schedule’s 

status. 

No The realism of start and completion dates 

for some activities is questionable 

because, as previously noted, some 

activities do not identify logical 

relationships (i.e., interdependencies with 

other activities) or are of unusually lengthy 

duration. In addition, there is no “status 

date” information in the schedule (i.e., 

evidence the overall schedule is updated 

on a regular basis to capture actual start 

and completion dates). Without this 

information, the impact of deviations on 

downstream work cannot be fully 

understood and proactively addressed. In 

addition, some dates in the schedule were 

updated erroneously. For example, the 

schedule shows 25 activities as having 

actual start dates in the future, including 6 

starting in 2010, even though the updated 

schedule was developed in April 2009. 

Likewise, there are 12 activities with actual 

100 percent complete finish dates that 

range from May 2009 to July 2010. 

Source: GAO analysis of DOD data. 

a

A Monte Carlo simulation allows the model’s parameters to vary simultaneously according to their 

associated probability distribution. The result is a set of estimated probabilities of achieving 

alternative outcomes, given the uncertainty in the underlying parameters. 

Page 69 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

The limitations in the program’s latest integrated master schedule, coupled 

with the program’s 7-year slippage to date, make it likely that DRRS will 

incur further delays. Compounding these limitations is the considerable 

concurrency in the key activities and events in the schedule associated 

with the 10 recently identified system modules (see fig. 2). For example, in 

2010 alone, the program office plans to complete development testing on 2 

modules and operational testing on 3 modules, while also reaching initial 

operational capability on 3 modules and full operational capability on 2 

modules. By way of comparison, the program office had almost no 

concurrency across a considerably more modest set of activities and 

events over the last 5 years, but nevertheless has fallen 7 years behind 

schedule. As previously reported, such significant overlap and 

concurrency among major program activities can create contention for 

limited resources and thus introduce considerable cost, schedule, and 

performance risks.24

 

24GAO, Information Technology: Improvements for Acquisition of Customs Trade 

Processing System Continue, but Further Efforts Needed to Avoid More Cost and 

Schedule Shortfalls, GAO-08-46 (Washington, D.C.: Oct. 25, 2007) and Secure Border 

Initiative: SBInet Planning and Management Improvements Needed to Control Risks, 

GAO-07-504T (Washington, D.C.: Feb. 27, 2007). 

Page 70 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

Figure 2: Schedule for Developing and Implementing DRRS Capabilities 

Begin planning, design, and development

Complete developmental testing and evaluation

Complete operational testing and evaluation

Complete initial operational capability

Complete final operational capability

Source: GAO analysis based on DOD data. 

Joint Mission

Readiness

Unit Mission

Readiness

Asset

Visibility

Business

Intelligence

Installation

Readiness

Readiness

Reviews

Planning/

Execution Support

Historical

Data

System

Architecture

End-User

Usability

Modules

J FMAMJ J A S OND J FMAMJ J A S OND J FMAMJ J A S OND J FMAMJ J A S OND J FMAMJ J A S ONDJ FMAMJ J A S ONDJ FMAMJ J A S OND J FMAMJ J A S OND

2004 2005 2006 2008 2009 2010 2011

J FMAMJ J A S OND J FMAMJ J A S OND

2007 2012 2013

April 1, 2009

In addition, the schedule remains unstable as evidenced by the degree of 

change it has experienced in just the past few months. For example, the 

January 2009 schedule had a full operational capability milestone of 

October 2011. By contrast, the April 2009 schedule has a December 2013 

milestone (see fig. 3 below). Moreover, some milestones are now to occur 

much earlier than they were a few months ago. For example, the January 

2009 schedule shows initial operational capability for “readiness reviews” 

to be June 2010. However, the April 2009 schedule shows that that this 

milestone was attained in August 2007. Overall, multiple milestones for 

four modules were extended by at least 1 year, including two milestones 

that were extended by more than 2 years. Such change in the schedule in 

Page 71 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

but a few months suggests a large degree of uncertainty, and illustrates the 

importance of e

ut a few months suggests a large degree of uncertainty, and illustrates the 

nsuring that the schedule is developed in accordance with 

best practices. 

Figure 3: Changes in Estimated Dates for DRRS Capabilities 

Initial date and direction of schedule change

Begin planning, design, and development

Complete developmental testing and evaluation

Complete operational testing and evaluation

Complete initial operational capability

Complete final operational capability

Source: GAO analysis based on DOD data. 

Joint Mission

Readiness

Unit Mission

Readiness

Asset

Visibility

Business

Intelligence

Installation

Readiness

Readiness

Reviews

Planning/

Execution Support

Historical

Data

System

Architecture

End-User

Usability

Modules

J FMAM J J A S ONDJ FMAM J J A S OND J FMAM J J A S OND

2009 2010 2011

J FMAM J J A S OND J FMAM J J A S OND

2012 2013

April 1, 2009

(12/08)

(8/07)

Page 72 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

As we have previously reported, effective human capital management is an 

essential ingredient to achieving successful program outcomes.25 Among 

other things, effective human capital management involves a number of 

actions to proactively understand and address any shortfalls in meeting a 

program’s current and future workforce needs. These include an 

assessment of the core competencies and essential knowledge, skills, and 

abilities needed to perform key program management functions, an 

inventory of the program’s existing workforce capabilities, and an analysis 

of the gap between the assessed needs and the existing capabilities. 

Moreover, they include explicitly defined strategies and actions for filling 

identified gaps, such as strategies for hiring new staff, training existing 

staff, and contracting for support services. 

DIO Lacks Adequate 

Staffing and an 

Effective Approach to 

Meeting its Human 

Capital Needs 

The DIO is responsible for performing a number of fundamental DRRS 

program management functions. For example, it is responsible for 

acquisition planning, performance management, requirements 

development and management, test management, contractor tracking and 

oversight, quality management, and configuration management. To 

effectively perform such functions, program offices, such as the DIO, need 

to have not only well-defined policies and procedures and support tools 

for each of these functions, but also sufficient human capital to implement 

the processes and use the tools throughout the program’s life cycle. 

Without sufficient human capital, it is unlikely that a program office can 

effectively perform its basic program management functions, which in turn 

increases the risk that the program will not deliver promised system 

capabilities and benefits on time and on budget. 

The DIO does not currently have adequate staff to fulfill its system 

acquisition and deployment responsibilities. In particular, the DIO is 

staffed with a single full-time government employee—the DIO Director. All 

other key program office functions are staffed by either contractor staff or 

staff temporarily detailed, on an as-needed basis, from other DOD 

organizations (referred to as “matrixed” staff). As a result, program 

management positions that the DIO itself has identified as critical to the 

program’s success, such as configuration manager and security manager, 

are being staffed by contractors. Moreover, these contractor staff report to 

program management positions also staffed by contractors. Other key 

positions, such as those for performing acquisition management, 

 

25GAO, A Model of Strategic Human Capital Management, GAO-02-373SP (Washington, 

D.C.: Mar. 15, 2002). 

Page 73 GAO-09-518 Military Readiness
Appendix II: Detailed Analysis of DRRS 

Acquisition and Deployment Management and 

Oversight 

requirements development and management, and performance 

management, have not even been established within the DIO. 

Furthermore, key positions, such as test manager, are vacant. These 

human capital limitations were acknowledged by the DRRS Executive 

Committee in November 2008. 

According to DIO and contractor officials, they recognize that additional 

program management staffing is needed. They also stated that while DRRS 

has been endorsed by USD (P&R) leadership and received funding 

support, past requests for additional staff have not been approved by USD 

(P&R) due to other competing demands for staffing. Further, DIO officials 

stated that the requests for staff were not based on a strategic gap analysis 

of its workforce needs and existing capabilities. Specifically, the program 

has not assessed its human capital needs and the gap between these needs 

and its onboard workforce capabilities. Until the program office adopts a 

strategic, proactive approach to managing its human capital needs, it is 

unlikely that it will have an adequate basis for obtaining the people it 

needs to effectively and efficiently manage DRRS. 

Page 74 GAO-09-518 Military Readiness
Appendix III: Comments from the Department Appendix III: Comments from the of Defense 

Department of Defense 

Page 75 GAO-09-518 Military Readiness 
Appendix III: Comments from the Department 

of Defense 

Page 76 GAO-09-518 Military Readiness
Appendix III: Comments from the Department 

of Defense 

Page 77 GAO-09-518 Military Readiness
Appendix III: Comments from the Department 

of Defense 

Page 78 GAO-09-518 Military Readiness
Appendix III: Comments from the Department 

of Defense 

Page 79 GAO-09-518 Military Readiness
Appendix IV: 

A

GAO Contacts and Staff 

cknowledgments 

Page 80 GAO-09-518 

Appendix IV: GAO Contacts and Staff 

Acknowledgments 

Sharon L. Pickup, (202) 512-9619 or pickups@gao.gov

Randolph C. Hite, (202) 512-6256 or hiter@gao.gov 

In addition to the contacts named above, key contributors to this report 

were Michael Ferren (Assistant Director), Neelaxi Lakhmani (Assistant 

Director), April Baugh, Mathew Butler, Richard J. Hagerman, Nicole 

Harms, James Houtz, John Lee, Stephen Pruitt, Terry Richardson, Karen 

Richey, Karl Seifert, and Kristy Williams. 

 Military Readiness 

GAO Contacts 

Staff 

Acknowledgments 

(351217) 
GAO’s Mission The Government Accountability Office, the audit, evaluation, and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and policies; 

and provides analyses, recommendations, and other assistance to help 

Congress make informed oversight, policy, and funding decisions. GAO’s 

commitment to good government is reflected in its core values of 

accountability, integrity, and reliability. 

The fastest and easiest way to obtain copies of GAO documents at no cost 

is through GAO’s Web site (www.gao.gov). Each weekday afternoon, GAO 

posts on its Web site newly released reports, testimony, and 

correspondence. To have GAO e-mail you a list of newly posted products, 

go to www.gao.gov and select “E-mail Updates.” 

Obtaining Copies of 

GAO Reports and 

Testimony 

Order by Phone The price of each GAO publication reflects GAO’s actual cost of 

production and distribution and depends on the number of pages in the 

publication and whether the publication is printed in color or black and 

white. Pricing and ordering information is posted on GAO’s Web site, 

http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 

MasterCard, Visa, check, or money order. Call for additional information. 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470 

Ralph Dawn, Managing Director, dawnr@gao.gov, (202) 512-4400 

U.S. Government Accountability Office, 441 G Street NW, Room 7125 

Washington, DC 20548 

To Report Fraud, 

Waste, and Abuse in 

Federal Programs 

Congressional 

Relations 

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548 

Public Affairs 

Please Print on Recycled Paper