Military Readiness: DOD Needs to Strengthen Management and Oversight of the Defense Readiness Reporting System
Published: 2009-09-25T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Defense (DOD) reports data about the operational readiness of its forces. In 1999, Congress directed DOD to create a comprehensive readiness system with timely, objective, and accurate data. In response, DOD started to develop the Defense Readiness Reporting System (DRRS). After 7 years, DOD has incrementally fielded some capabilities, and, through fiscal year 2008, reported obligating about $96.5 million. GAO was asked to review the program including the extent that DOD has (1) effectively managed and overseen DRRS acquisition and deployment and (2) implemented features of DRRS consistent with legislative requirements and DOD guidance. GAO compared DRRS acquisition disciplines, such as requirements development, test management, and DRRS oversight activities, to DOD and related guidance, and reviewed the system's current and intended capabilities relative to legislative requirements and DOD guidance. We did not evaluate DOD's overall ability to assess force readiness or the extent that readiness data reflects capabilities, vulnerabilities, or performance issues.
section_title: What GAO Found
paragraphs: DOD has not effectively managed and overseen the DRRS acquisition and deployment, in large part because of the absence of rigorous and disciplined acquisition management controls and an effective governance and accountability structure for the program. In particular, system requirements have not been effectively developed and managed. For example, user participation and input in the requirements development process was, until recently, limited, and requirements have been experiencing considerable change, are not yet stable, and have not been effectively controlled. In addition, system testing has not been adequately performed and managed. For example, test events for already acquired system increments, as well as currently deployed and operating increments, were not based on well-defined plans or structures, and test events have not been executed in accordance with plans or in a verifiable manner. Moreover, DRRS has not been guided by a reliable schedule of work to be performed and key activities to occur. These program management weaknesses can, in part, be attributed to long-standing limitations in program office staffing and program oversight and accountability. Despite being a DOD-wide program, until April, 2009 DRRS was not accountable to a DOD-wide oversight body, and it was not subject to DOD's established mechanisms and processes for overseeing business systems. Collectively, these acquisition management weaknesses have contributed to a program that has fallen well short of expectations, and is unlikely to meet future expectations. DOD has implemented DRRS features that allow users to report certain mission capabilities that were not reported under the legacy system, but these features are not fully consistent with legislative requirements and DOD guidance; and DOD has not yet implemented other features. The geographic combatant commands are currently reporting their capabilities to execute most of their operations and major war plans in DRRS, and DOD is reporting this additional information to Congress. However, because DRRS does not yet fully interface with legacy systems to allow single reporting of readiness data, the military services have not consistently used DRRS's enhanced capability reporting features. For example, as of May 2009, the Army and Navy had developed interfaces for reporting in DRRS, while the Marine Corps required units to only report in their legacy system. Recently, the Marine Corps also began developing an interface and has done limited reporting in DRRS. In addition, DRRS has not fully addressed the challenges with metrics that led Congress to require a new readiness reporting system. DRRS metrics are less objective and precise, and no more timely than the legacy system metrics. Users have also noted that DRRS lacks some of the current and historical data and connectivity with DOD's planning systems necessary to manage and deploy forces. Until these limitations are fully addressed, DRRS will not have the full complement of features necessary to meet legislative and DOD requirements, and users will need to rely on legacy reporting systems to support mission-critical decisions.

section_title: Letter
paragraphs: To assess the ability of U.S. forces to execute the wartime missions for  which they were designed, as well as other assigned missions, and to make  decisions about deploying forces, the Department of Defense (DOD) relies  heavily on readiness information derived from multiple information  systems. Over the years, we and others have identified shortcomings in  DOD’s readiness assessment and reporting, such as limitations in the  completeness and precision of readiness data and a tendency to focus on  examining the status of personnel, equipment, and other resources rather  than broader capabilities. Congress addressed DOD’s readiness reporting  in the Strom Thurmond National Defense Authorization Act for Fiscal Year  1999 by adding section 117 to Title 10 of the U.S. Code, directing the  Secretary of Defense to establish a comprehensive readiness reporting  system to measure, in an “objective, accurate, and timely manner,” the  capability of the armed forces to carry out the National Security Strategy  prescribed by the President, the defense planning guidance provided by  the Secretary of Defense, and the National Military Strategy prescribed by  the Chairman of the Joint Chiefs of Staff.
In June 2002, the Deputy Secretary of Defense directed the Under  Secretary of Defense for Personnel and Readiness (USD P&R) to develop,  field, maintain, and fund the Enhanced Status of Resources and Training  System (ESORTS), which is the automated readiness reporting system  within the Defense Readiness Reporting System (DRRS). He also directed  that DRRS build upon existing processes and readiness assessment tools  to establish a capabilities-based, adaptive, near-real-time readiness  reporting system. In addition, in June 2004, the Secretary of Defense  directed USD (P&R) to develop DRRS in a manner that would support the  data requirements of various users of readiness information, such as the  Chairman of the Joint Chiefs of Staff, the combatant commanders, the  Secretaries of the military departments, and the Chief of the National  Guard Bureau, including their requirements for data on the availability,  readiness, deployment, and redeployment of forces.
USD (P&R) established a DRRS Implementation Office (DIO) to manage  the system’s acquisition, including managing system development and  engaging the user community. The DIO has used support contractors to  develop the system and reported obligating about $96.5 million for DRRS  from fiscal year 2001 through fiscal year 2008. Some fielded system  capabilities are currently being used to varying degrees by the user  community. The DIO originally estimated that DRRS would achieve full  operational capability in fiscal year 2007, but currently expects DRRS to  reach full capability in 2014. In September 2008, the DIO projected that it  would spend about $135 million through fiscal year 2014.
Recognizing that DRRS was not yet fully deployed or operational and in  light of our prior work on readiness-related issues, you asked us to review  DOD’s efforts to develop and implement DRRS, including the program’s  status, and the extent that DRRS addresses the challenges that led  Congress to require a new system, such as the availability of information  on capabilities, and the precision, timeliness, reliability, and objectivity of  readiness metrics. In addressing these issues, we assessed the extent to  which (1) DOD has effectively managed and overseen DRRS acquisition  and deployment, and (2) features of DRRS have been implemented and are  consistent with legislative requirements and DOD guidance.
To determine the extent that DOD has effectively managed and overseen  DRRS acquisition and deployment, we analyzed a range of program  documentation and interviewed cognizant program and contractor  officials relative to the following acquisition management disciplines:  requirements development and management, test management, schedule  reliability, and human capital. For each discipline, we compared key  program documentation, such as a requirements management plan, test  and evaluation master plans, and test reports to relevant DOD, federal, and  related guidance, and we analyzed a statistical sample of individual  requirements and test cases to determine consistency among them. In  addition, we attended meetings of organizations established to monitor or  govern DRRS development and reviewed information from meetings that  we did not attend and interviewed officials associated with these  meetings.
To determine the extent to which the features of DRRS have been  implemented and are consistent with legislative requirements and DOD  guidance, we reviewed criteria such as the legislation that mandated a  comprehensive DOD readiness reporting system, the DOD directive that  established DRRS, program documentation and USD (P&R) guidance  memorandums, DIO briefings to the readiness community, other  departmental instructions, and directives and memorandums related to  DRRS requirements and implementation. From these documents, we  identified desired features of DRRS and compared them to documentary  and testimonial evidence concerning system performance during meetings  with a full range of officials responsible for developing and using the  system. To obtain the developer’s perspective, on numerous occasions  throughout our review we met with officials from USD (P&R), the DIO,  and the three current DRRS contractors. To obtain user perspectives, we  met with and surveyed by questionnaire officials from the Joint Staff, the  geographic and functional combatant commands, and the services, and  also met with officials from USD (P&R). We also attended meetings of  organizations established to monitor or govern DRRS development and  analyzed information from meetings that we did not attend. We also  directly observed the system’s capabilities through our own limited use of  the system and by observing others using the system.
We did not evaluate the department’s overall ability to assess the readiness  of its forces or the extent that data contained in any of its readiness  reporting systems, including DRRS and GSORTS, reflect capabilities,  deficiencies, vulnerabilities, or performance issues. Our review focused on  acquisition and program management issues, such as requirements  management, schedule and human capital requirements, the current usage  of DRRS, and the extent to which DRRS’ features address legislative  requirements and DOD guidance.
We conducted this performance audit from April 2008 through August  2009 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives. Additional details on our  scope and methodology are in appendix I.
subsections: section_title: Background
paragraphs: Historically, DOD has used its readiness assessment system to assess the  ability of units and joint forces to fight and meet the demands of the  national security strategy. DOD’s readiness assessment and reporting  system is designed to assess and report on military readiness at three  levels—(1) the unit level; (2) the joint force level; and (3) the aggregate, or  strategic, level. Using information from its readiness assessment system,  DOD prepares and sends legislatively mandated Quarterly Readiness  Reports to Congress. DRRS is DOD’s new readiness reporting system that  is intended to capture information from the previous system, as well as  information about organizational capabilities to perform a wider variety of  missions and mission essential tasks. DRRS is also intended to capture  readiness information from defense agencies and installations, which were  not required to report under the previous system. Some DRRS features are  currently fielded and being used to varying degrees by the user  community.
subsections: section_title: DOD Collects a Wide Range of Readiness Information to Support Decision Makers Within and Outside DOD
paragraphs: Laws, directives, and guidance, including a DOD directive, Chairman of  the Joint Chiefs of Staff Instruction (CJCSI), Secretary of Defense and  USD (P&R) memorandums, and service regulations and messages, show  that readiness information and data are needed to support a wide range of  decision makers. These users of readiness data include Congress, the  Secretary of Defense, the Chairman of the Joint Chiefs of Staff, the  combatant commanders, the Secretaries of the military departments, and  the Chief of the National Guard Bureau.
The directives and guidance also list roles and responsibilities for  collecting and reporting various types of readiness data. For example,  CJCSI 3401.02A assigns the service chiefs responsibility for ensuring  required global status of resources and training system (GSORTS) reports  are submitted. GSORTS is DOD’s legacy, resource-based readiness  reporting system that provides a broad assessment of unit statuses based  on units’ abilities to execute the missions for which they were organized  or designed as well as the current missions for which they may be  employed. The information in the required GSORTS reports includes units’  abilities to execute the missions for which they were organized or  designed, as well as the status of their training, personnel, and equipment.  In addition, DOD directive 7730.65, which established DRRS as DOD’s new  readiness reporting system, assigns the Secretaries of the military  departments and the commanders of the combatant commands  responsibilities for developing mission essential tasks for all of their  assigned missions.
section_title: Requirements and Intended Characteristics of DOD’s New Readiness Reporting System
paragraphs: Prior to 1999, we identified challenges with DOD’s existing readiness  reporting system, GSORTS, and in 1999, Congress directed the Secretary  of Defense to establish a comprehensive readiness reporting system. The  legislation requires the system to measure in an objective, accurate, and  timely manner the capability of the armed forces to carry out (1) the  National Security Strategy prescribed by the President, (2) the defense  planning guidance provided by the Secretary of Defense, and (3) the  National Military Strategy prescribed by the Chairman of the Joint Chiefs  of Staff.
To address the requirements established by Congress, the Office of the  Deputy Under Secretary of Defense (Readiness) began in 2001 to build  consensus among DOD’s senior readiness leaders for an improved  readiness assessment system. For example, the Deputy’s office distributed  a list of key characteristics of the improved readiness assessment system  to the leaders in advance of scheduled meetings. The system’s key desired  characteristics included allowing near-real-time access to readiness data  and trends, enabling rapid, low-cost development using classified Internet  technology, and reducing the reporting burdens on people. Since then  various directives and memorandums have been issued regarding DRRS  responsibilities, requirements, and related issues. For example:    On June 3, 2002, the Deputy Secretary of Defense established DOD’s  new readiness reporting system, as directed by Congress, by signing  DOD Directive 7730.65. According to this directive, DRRS is intended  to build upon DOD’s existing processes and readiness assessment tools  to establish a capabilities-based, near-real-time readiness reporting  system. The DRRS directive assigned USD (P&R) responsibilities for  developing, fielding, maintaining, and funding ESORTS (the tool to  collect capability, resource, and training information) and overseeing  DRRS to ensure accuracy, completeness, and timeliness of its  information and data, its responsiveness, and its effective and efficient  use of modern practices and technologies. In addition, the USD P&R is  responsible for ensuring that ESORTS information, where appropriate,  is integrated into DOD’s planning systems and processes. The directive  also states that until ESORTS becomes fully operational, the Chairman  of the Joint Chiefs of Staff shall maintain the GSORTS database.
On June 25, 2004, the Secretary of Defense issued a memorandum,  which directed USD (P&R) to develop DRRS to support data  requirements identified by the Chairman of the Joint Chiefs of Staff, the  combatant commanders, the Secretaries of the Military Departments,  and the Chief, National Guard Bureau to include availability, readiness,  deployment, and redeployment data.
On November 2, 2004, USD (P&R) issued a DRRS interim  implementation guidance memorandum. In this memorandum, the  undersecretary noted that he had established a DIO to provide  reporting assistance for units. The memorandum also stated that  combatant commanders would begin reporting readiness by mission  essential tasks by November 30, 2004. The memorandum also directed  the services to develop detailed implementing guidance for reporting  and assessing mission essential task readiness in ESORTS within their  respective services, and set a goal for the services to implement the  mission essential task reporting process by September 30, 2005. To  meet these mission essential task reporting requirements, USD (P&R)  directed commanders to rate their organizational capabilities as (1) yes  or “Y”, (2) qualified yes or “Q”, or (3) no or “N.” A “Y” indicates that an  organization can accomplish the rated tasks or missions to prescribed  standards and conditions in a specified environment. It should reflect  demonstrated performance in training or operations. A “Q” indicates  that performance has not been demonstrated, and, although data may  not readily support a “Y,” the commander believes the organization can  accomplish the rated task or mission to standard under most  conditions. An “N” indicates that an organization cannot accomplish  the rated task or mission to prescribed standards in the specified  environment at the time of the assessment.
The November 2004 memorandum also stated that the expected  transition from GSORTS to ESORTS was scheduled to begin in fiscal  year 2005. According to the 2004 memorandum, the ESORTS module of  DRRS would provide, among other things, visibility of the latest  GSORTS information reported by units, and detailed resource  information from authoritative data sources with the capability to  aggregate or separate the data. This memorandum signaled a change in  program direction. Although the 2002 DOD directive stated that DRRS  is intended to build upon DOD’s existing processes and readiness  assessment tools, the 2004 memorandum indicated that DRRS was to  replace GSORTS, as the ESORTS module of DRRS captured both  capabilities and resource data.
section_title: Overview of DRRS Program Management and Oversight Structure
paragraphs: Since its establishment, the DIO has operated within the Office of USD  (P&R) and has relied on multiple contractors. To provide governance of  DRRS, and enhance communication between the development community,  represented by the DIO and contractors, and the user community, which  includes the Joint Staff, military services, and combatant commands, USD  (P&R) established various bodies with representatives from the user  community, including military services, combatant commands, and the  defense agencies. Representatives from the Office of USD (P&R) and the  Joint Staff currently serve as cochairs of the various bodies. DRRS Battle  Staffs comprise colonels, Navy captains, and similar-graded civilians. They  track DRRS development and identify issues with the system. At the one- star level, the DRRS General and Flag Officer Steering Committee  discusses issues raised by the Battle Staff. In December 2007, USD (P&R)  created a committee at the three-star level, referred to as the DRRS  Executive Committee. Its charter, finalized about a year later in January  2009, calls for the committee to review and approve proposals and plans to  establish policy, processes, and system requirements for DRRS, including  approving software development milestones required to reach objectives.
To ensure that leadership is provided for the direction, oversight, and  execution of DOD’s business transformation efforts, including business  systems modernization efforts such as DRRS, DOD relies on several  entities. These entities include the Defense Business Systems Management  Committee, which is chaired by the Deputy Secretary of Defense and  serves as the department’s highest-ranking governance body and the  approval authority for business systems modernization activities; the  Investment Review Boards, which are chartered by the Principal Staff  Assistants—senior leaders from various offices within DOD—and serve as  the review and certification bodies for business system investments in  their respective areas of responsibility; and the Business Transformation  Agency, which is responsible for supporting the Investment Review  Boards and for leading and coordinating business transformation efforts  across the department. Among other things, the Business Transformation  Agency supports the Office of the Under Secretary of Defense, Acquisition,  Technology and Logistics in conducting system acquisition risk  assessments.
section_title: Disciplined System Acquisition and Oversight Are Keys to Program Success
paragraphs: Our research and evaluations of information technology programs,  including business systems modernization efforts within DOD, have shown  that delivering promised system capabilities and benefits on time and  within budget largely depends on the extent to which key program  management disciplines are employed by an adequately staffed program  management office. Among other things, these disciplines include a  number of practices associated with effectively developing and managing  system requirements, adequately testing system capabilities, and reliably  scheduling the work to be performed. They also include proactively  managing the program office’s human capital needs, and promoting  program office accountability through executive-level program oversight.  DOD acquisition policies and guidance, along with other relevant  guidance, recognize the importance of these management and oversight  disciplines. As we have previously reported, not employing these and  other program management disciplines increases the risk that system  acquisitions will not perform as intended and require expensive and time- consuming rework.
section_title: DOD Disagreed with GAO’s Prior Recommendation to Develop an Implementation Plan to Guide DRRS Development
paragraphs: In 2003, we reported that, according to USD (P&R) officials, DRRS was a  large endeavor, and that development would be challenging and require  buy-in from many users. We also reported that the program was only a  concept without detailed plans to guide its development and  implementation. Based on the status of the program at that time and  DOD’s past record on readiness reporting initiatives, we recommended  that the Secretary of Defense direct the Office of USD (P&R) to develop an  implementation plan that identified    performance goals that are objective, quantifiable, and measurable;    performance indicators to measure outcomes;    an evaluation plan to compare program results with established goals;    milestones to guide DRRS development to the planned 2007 full  capability date.
DOD did not agree with our recommendation, stating that it had  established milestones, cost estimates, functional responsibilities,  expected outcomes, and detailed plans for specific information technology  requirements and progress indicators. In evaluating the DOD comments,  we noted that DOD had established only two milestones—initial capability  in 2004 and full capability in 2007—and did not have a road map explaining  the steps needed to achieve full capability by 2007.
section_title: DOD Has Not Effectively Managed and Overseen the Acquisition and Deployment of DRRS
paragraphs: DOD has not effectively managed and overseen the acquisition and  deployment of DRRS in accordance with a number of key program  management disciplines that are recognized in DOD acquisition policies  and guidance, along with other relevant guidance, and are fundamental to  delivering a system that performs as intended on time and within budget.  In particular, DRRS requirements have not been effectively developed and  managed, and DRRS testing has not been adequately performed and  managed. Further, DRRS has not been guided by a reliable schedule of the  work needed to be performed and the key activities and events that need  to occur. These program management weaknesses can be attributed in  part to long-standing limitations in program office staffing and oversight.  As a result, the program has not lived up to the requirements set for it by  Congress, and the department has not received value from the program  that is commensurate with the time and money invested—about 7 years  and $96.5 million. Each of these weaknesses are summarized below and  discussed in detail in appendix II.
subsections: section_title: DRRS Requirements Have Not Been Effectively Developed and Managed
paragraphs: According to DOD and other relevant guidance, effective requirements  development and management includes, among other things, (1)  effectively eliciting user needs early and continuously in the system life- cycle process, (2) establishing a stable baseline set of requirements and  placing the baseline under configuration management, (3) ensuring that  system requirements are traceable backward to higher level business or  operational requirements (e.g., concept of operations) and forward to  system design documents (e.g., software requirements specification) and  test plans, and (4) controlling changes to baseline requirements. However,  none of these conditions have been met on DRRS. Specifically, key users  have only recently become fully engaged in developing requirements, and  requirements have been experiencing considerable change and are not yet  stable. Further, different levels of requirements and related test cases have  not been aligned with one another, and changes to requirements have not  been effectively controlled. As a result, efforts to develop and deliver  initial DRRS capabilities have taken longer than envisioned and these  capabilities have not lived up to user expectations. These failures increase  the risk of future DRRS capabilities not meeting expectations and increase  the likelihood that expensive and time-consuming system rework will be  necessary.
subsections: section_title: Recent Actions Have Been Taken to Address Limited User Involvement in Developing and Managing Requirements
paragraphs: Until recently, key users were not fully or effectively engaged in DRRS  requirements development and management. One of the leading practices  associated with effective requirements development is engaging system  users early and continuously in the process of defining requirements.  However, DIO officials and representatives from the military services and  the Joint Staff agree that until recently, key users were not effectively  engaged in DRRS requirements development and management, although  they disagree at to why user involvement has suffered. Regardless, DRRS  Executive Committee direction has improved the situation. Specifically, in  January 2008, the committee directed the Joint Staff to conduct an analysis  of DRRS capabilities, referred to as the “capabilities gap analysis,” which  involved the entire readiness community and resulted in 530 additional  user requirements. In our view, this analysis is a positive step in  addressing long-standing limited involvement by key DRRS users in  defining requirements that has contributed to significant delays in the  program, as discussed later in the report.
section_title: DRRS Requirements Are Not Stable
paragraphs: As of April 2009, DRRS requirements continued to be in a state of flux.  Establishing an authoritative set of baseline requirements prior to system  design and development provides a stable basis for designing, developing,  and delivering a system that meets its users’ operational needs. However,  the fact that these 530 user requirements have recently been identified  means that the suite of requirements documentation associated with the  system, such as the detailed system requirements, will need to change and  thus is not stable. To illustrate, these 530 requirements have not been fully  evaluated by the DIO and the DRRS governance boards and according to  program officials, have not yet been approved, and thus their impact on  the program is not clear. Compounding this instability in the DRRS  requirements is the fact that additional changes are envisioned. According  to program officials, the changes resulting from the gap analysis and  reflected in the latest version of the DRRS Concept of Operations, which  was approved by the DRRS Executive Committee in January 2009, have  yet to be reflected in other requirements documents, such as the detailed  system requirements. Although defining and developing requirements is  inherently an iterative process, having a baseline set of requirements that  are stable is a prerequisite to effective and efficient system design and  development. Without them, the DIO has not been able to deliver a system  that meets user needs on time, and it is unlikely that future development  and deployment efforts will produce better results.
section_title: Alignment among Requirements and Related System Design and Testing Artifacts Has Not Been Demonstrated
paragraphs: During our review, DIO officials could not demonstrate that requirements  and related system design and testing artifacts are properly aligned. One of  the leading practices associated with developing and managing  requirements is maintaining bidirectional traceability from high-level  operational requirements through detailed lower-level requirements and  design documents to test cases. We attempted on three separate occasions  to verify the traceability of system requirements backwards to higher-level  requirements and forward to lower-level software specifications and test  cases, and each time we found that traceability did not exist. DIO and  contractor officials attributed the absence of adequate requirements  traceability to the ongoing instability in requirements and efforts to update  program documentation. Without traceable requirements, the DIO does  not have a sufficient basis for knowing that the scope of the design,  development, and testing efforts will produce a system solution on time  and on budget and that will meet users’ operational needs and perform as  intended. As a result, the risk is significant that expensive and time- consuming system rework will be required.
section_title: Changes to Requirements Are Not Being Effectively Controlled
paragraphs: Since the inception of the program in 2002, DRRS has been developed and  managed without a formally documented and approved process for  managing changes to system requirements. Adopting a disciplined process  for reviewing and accepting changes to an approved baseline set of  requirements in light of the estimated costs, benefits, and risk of each  proposed change is a recognized best practice. However, requirements  management and change-control plans developed in 2006 by the DRRS  software development contractor, according to DIO officials, were not  adequate. To address this, the Joint Staff developed what it referred to as a  conceptual requirements change-control process in February 2008, as a  basis for the DIO to develop more detailed plans that could be  implemented. In January 2009, the DIO drafted more detailed requirements  management and configuration management plans, the latter of which the  DIO updated in March 2009. However, the plans have yet to be approved  and implemented. Until the DIO effectively controls requirements changes,  it increases the risk of needed DRRS capabilities taking longer and costing  more to deliver than necessary.
section_title: DRRS Testing Has Not Been Adequately Performed and Key Test Management Structures and Controls Have Not Been Established
paragraphs: According to DOD and other relevant guidance, system testing should be  progressive, meaning that it should consist of a series of test events that  first focus on the performance of individual system components, then on  the performance of integrated system components, followed by system- level tests that focus on whether the system (or major system increments)  are acceptable, interoperable with related systems, and operationally  suitable to users. For this series of related test events to be conducted  effectively, each test event needs to be executed in accordance with well- defined test plans, the results of each test event need to be captured and  used to ensure that problems discovered are disclosed and corrected, and  all test events need to be governed by a well-defined test management  structure.
However, the DIO cannot demonstrate that it has adequately tested any of  the DRRS increments, referred to as system releases and subreleases, even  though it has already acquired and partially deployed a subset of these  increments. Moreover, the DIO has yet to establish the test management  structures and controls needed to effectively execute DRRS testing going  forward. More specifically, the test events for already acquired, as well as  currently deployed and operating, DRRS releases and subreleases were  not based on well-defined plans. For example, the test plan did not include  a schedule of activities to be performed or defined roles and  responsibilities for performing them. Also, the test plan did not  consistently include test entrance and exit criteria, a test defect  management process, and metrics for measuring progress. Further, test  events have not been fully executed in accordance with plans, or executed  in a verifiable manner, or both. For example, although increments of  DRRS functionality have been put into production, the DIO has not  performed system integration testing, system acceptance testing, or  operational testing on any DRRS release or subrelease. Moreover, the  results of all executed test events have not been captured and used to  ensure that problems discovered were disclosed to decision makers, and  ultimately corrected. For example, the DIO has not captured the test  results for at least 20 out of 63 DRRS subreleases. Test results that were  captured did not include key elements, such as entrance/exit criteria  status and unresolved defects and applicable resolution plans.
The DIO has also not established an effective test management structure  to include, for example, a clear assignment of test management roles and  responsibilities, or a reliable schedule of planned test events.  Compounding this absence of test management structures and controls is  the fact that the DIO has yet to define how the development and testing to  date of a series of system increments (system releases and subreleases)  relate to the planned development and testing of the 10 system modules  established in January 2009. (See table 1 for a list and description of these  modules.) Collectively, this means that it is unlikely, that already  developed and deployed DRRS increments can perform as intended and  meet user operational needs. Equally doubtful are the chances that the  DIO can adequately ensure that yet-to-be developed DRRS increments will  meet expectations.
section_title: DRRS Has Not Been Guided by a Reliable Schedule
paragraphs: The success of any program depends in part on having a reliable schedule  that defines, among other things, when work activities will occur, how  long they will take, and how they are related to one another. From its  inception in 2002 until November 2008, the DIO did not have an integrated  master schedule, and thus has long been allowed to proceed without a  basis for executing the program and measuring its progress. In fact, the  only milestone that we could identify for the program prior to November  2008 was the date that DRRS was to achieve full operational capability,  which was originally estimated to occur in fiscal year 2007, but later  slipped to fiscal year 2008 and then fiscal year 2011, and is now fiscal year  2014--a 7-year delay.
Moreover, the DRRS integrated master schedule that was first developed  in November 2008, and was updated in January 2009 and again in April  2009 to address limitations that we identified and shared with the program  office, is still not reliable. Specifically, our research has identified nine  practices associated with developing and maintaining a reliable schedule.  These practices are (1) capturing all key activities, (2) sequencing all key  activities, (3) assigning resources to all key activities, (4) integrating all  key activities horizontally and vertically, (5) establishing the duration of all  key activities, (6) establishing the critical path for all key activities, (7)  identifying float between key activities, (8) conducting a schedule risk  analysis, and (9) updating the schedule using logic and durations to  determine the dates for all key activities. The program’s latest integrated  master schedule does not address three of the practices and only partially  addresses the remaining six. For example, the schedule does not establish  a critical path for all key activities, nor does it include a schedule risk  analysis, and it is not being updated using logic and durations to determine  the dates for all key activities. In addition, the schedule introduces  considerable concurrency across key activities and events for several  modules, which introduces increased risk. These limitations in the  program’s latest integrated master schedule, coupled with the program’s 7- year slippage to date and continued requirements instability, make it likely  that DRRS will incur further delays.
section_title: DIO Lacks Adequate Staffing and an Effective Approach to Meeting Its Human Capital Needs
paragraphs: The DIO does not currently have adequate staff to fulfill its system  acquisition and deployment responsibilities, and it has not managed its  staffing needs in an effective manner. Effective human capital  management should include an assessment of the core competencies and  essential knowledge, skills, and abilities needed to perform key program  management functions, an inventory of the program’s existing workforce  capabilities, an analysis of the gap between the assessed needs and the  existing capabilities, and plans for filling identified gaps. DIO performs a  number of fundamental DRRS program management functions, such as  acquisition planning, performance management, requirements  development and management, test management, contractor tracking and  oversight, quality management, and configuration management. To  effectively perform such functions, program offices, such as the DIO, need  to have not only well-defined policies and procedures and support tools  for each of these functions, but also sufficient human capital to implement  the processes and use the tools throughout the program’s life cycle.  However, the DIO is staffed with only a single full-time government  employee—the DIO Director. All other key program office functions are  staffed by either contractor staff or staff temporarily detailed, on an as- needed basis, from other DOD organizations. In addition, key positions,  such as performance manager and test manager, have either not been  established or are vacant. According to DIO and contractor officials, they  recognize that additional program management staffing is needed but  stated that requests for additional staff had not been approved by USD  (P&R) due to competing demands for staffing. Further, they stated that the  requests were not based on an assessment of the program’s human capital  needs and the gap between these needs and its onboard workforce  capabilities. Until DIO adopts a strategic, proactive approach to managing  its human capital needs, it is unlikely that it will have an adequate basis for  obtaining the people it needs to effectively and efficiently manage DRRS.
section_title: DOD Executive Oversight of DRRS Has Been Limited
paragraphs: A key principle for acquiring and deploying system investments is to  establish a senior-level governance body to oversee the investment and  hold program management accountable for meeting cost, schedule, and  performance commitments. Moreover, for investments that are  organization wide in scope and introduce new ways of doing business, like  DRRS, the membership of this oversight body should represent all  stakeholders and have sufficient organizational seniority to commit their  respective organizations to any decisions reached. For significant system  investments, the department’s acquisition process provides for such  executive governance bodies. For example, Major Automated Information  Systems, which are investments over certain dollar thresholds or that are  designated as special interest because of, among other things, their  mission importance, are reviewed at major milestones by a designated  milestone decision authority. These authorities are supported by a senior  advisory group, known as the Information Technology Acquisition Board,  which comprises senior officials from the Joint Staff, the military  departments, and staff offices within the Office of the Secretary of  Defense. In addition, all business system investments in DOD that involve  more than $1 million in obligations are subject to review and approval by a  hierarchy of DOD investment review boards that comprise senior DOD  leaders, including the Defense Business Systems Management Committee,  which is chaired by the Deputy Secretary of Defense. Through these  executive oversight bodies and their associated processes, programs are to  be, among other things, governed according to corporate needs and  priorities, and program offices are to be held accountable for meeting cost,  schedule, and performance expectations.
Until April 2009, DRRS was not subject to any of DOD’s established  mechanisms and processes for overseeing information technology  systems. As previously discussed, USD (P&R) established the DRRS Battle  Staff, which is a group of midlevel military officers and civilians from  DRRS stakeholder organizations, and it established a higher-ranked  General and Flag Officer Steering Committee, consisting of stakeholder  representatives. However, neither of these entities had specific oversight  responsibilities or decision-making authority for DRRS. Moreover, neither  was responsible for holding the program office accountable for results.  According to meeting minutes and knowledgeable officials, these entities  met on an irregular basis over the last several years, with as much as a 1- year gap in meeting time for one of them, to discuss DRRS status and  related issues.
In December 2007, USD (P&R) recognized the need for a more senior-level  and formal governance body, and established the DRRS Executive  Committee. Since January 2008, this committee, which consists of top- level representatives from stakeholder organizations, has met at least  seven times. In January 2009, the DRRS Executive Committee’s charter  was approved by the Deputy Under Secretary of Defense (Readiness) and  the three-star Director of the Joint Staff. According to the charter, the  committee is to review and approve proposals and plans to establish  policy, processes, and system requirements for DRRS, including approving  software development milestones required to reach objectives. Consistent  with its charter, the committee has thus far made various program-related  decisions, including approving a DRRS concept of operations to better  inform requirements development, and directing the Joint Staff to conduct  an analysis to identify any gaps between DRRS requirements and user  needs. However, the committee has not addressed the full range of  acquisition management weaknesses previously discussed in this report,  and it has not taken steps to ensure that the program office is accountable  for well-defined program baseline requirements.
More recently, the DOD Human Resources Management Investment  Review Board and the Defense Business Systems Management Committee  reviewed DRRS and certified and approved, respectively, the program to  invest $24.625 million in fiscal years 2009 and 2010. These entities  comprise senior leadership from across the department, including the  Deputy Secretary of Defense as the Defense Business Systems  Management Committee Chair, military service secretaries, the defense  agency heads, principal staff assistants, and representatives from the Joint  Staff and combatant commands. However, neither the Investment Review  Board’s certification nor the Defense Business Systems Management  Committee’s approval was based on complete and accurate information  from USD (P&R). Specifically, the certification package submitted to both  oversight bodies by the USD (P&R) precertification authority (Office of  Readiness Programming and Assessment) stated that DRRS was on track  for meeting its cost, schedule, and performance measures and highlighted  no program risks despite the weaknesses discussed in this report.  According to the chairwoman of the Investment Review Board, the board  does not have a process or the resources to validate the information  received from the programs that it reviews. Moreover, the chairwoman  stated that program officials did not make the board aware of the results  of our review that we shared with the DIO prior to either the Investment  Review Board or Defense Business Systems Management Committee  reviews. Since we briefed the chairwoman, the Investment Review Board  has requested that the DIO provide it with additional information  documenting DRRS compliance with applicable DOD regulations and  statutes.
According to USD (P&R) and DIO officials, DRRS was not subject to  department executive-level oversight for almost 6 years because, among  other things, they did not consider DRRS to be a more complex  information technology system. Furthermore, because of the nature of the  authority provided to the USD (P&R) in the DRRS charter, they did not  believe it was necessary to apply the same type of oversight to DRRS as  other information systems within DOD. This absence of effective oversight  has contributed to a void in program accountability and limited prospects  for program success.
section_title: Some DRRS Features Are Operational but Are Not Fully Consistent with Legislative Requirements and DOD Guidance
paragraphs: DOD has implemented DRRS features that allow users to report certain  mission capabilities that were not reported under the legacy system, but  these features are not fully consistent with legislative requirements and  DOD guidance; and DOD has not yet implemented other envisioned  features of the system. While some users are consistently reporting  enhanced capability information, reporting from other users has been  inconsistent. In addition, DRRS has not fully addressed the challenges  with metrics that were identified prior to 1999 when Congress required  DOD to establish a new readiness reporting system. Users have also noted  that DRRS lacks some of the current and historical data and connectivity  with DOD’s planning systems necessary to manage and deploy forces.
subsections: section_title: DOD Is Providing Congress with DRRS Capability Data from the Combatant Commands, but Services Have Not Consistently Reported Capability Data
paragraphs: The geographic combatant commands are capturing enhanced capability  data in DRRS, and DOD’s quarterly readiness reports to Congress  currently contain this information, as well as information that is drawn  from DOD’s legacy readiness reporting system, GSORTS. However, the  military services have not consistently used the enhanced capability  reporting features of DRRS. Because DRRS does not yet fully interface  with legacy systems to allow single reporting of readiness data, the Army  and Navy developed additional system interfaces and are reporting in  DRRS. Until May 2009, the Marine Corps directed its units to report only in  the legacy system to avoid the burden of dual reporting. The Air Force  chose not to develop an interface and instructed its units to report in both  DRRS and the legacy system.
DRRS and GSORTS both contain capabilities information and resource  (numbers of personnel, equipment availability, and equipment condition)  and training data. However, DRRS currently provides more capabilities  data than GSORTS. When Congress directed DOD to establish a new  readiness reporting system, GSORTS was already providing capability  information concerning unit capabilities to perform the missions for which  they were organized or designed. More recently, some of the military  services began reporting limited capability information on unit capabilities  to perform missions other than those that they were organized or designed  to perform into GSORTS. However, DRRS is designed to capture  capabilities on a wider variety of missions and mission essential tasks. For  example, organizations can report their capabilities to conduct missions  associated with major war plans and operations such as Operation Iraqi  Freedom into DRRS, as well as their capabilities to perform the missions  for which they were organized or designed. DRRS also captures capability  information from a wider range of organizations than GSORTS. Although  the primary (monthly) focus is on operational units and commands, DRRS  collects and displays readiness information from defense agencies and  installations.
Geographic combatant commands—such as U.S. Central Command, U.S.  Pacific Command, and U.S. Northern Command—are currently reporting  their commands’ capabilities to execute most of their operations and  major war plans in DRRS. DOD reports this enhanced capability  information from the geographic combatant commands in its Quarterly  Readiness Report to Congress. The geographic combatant commands are  also using DRRS to report their capabilities to perform headquarters-level,  joint mission essential tasks, and some of these commands utilize DRRS as  their primary readiness reporting tool. For example, U.S. Northern  Command uses DRRS to assess risk and analyze capability gaps, and U.S.  Pacific Command identifies critical shortfalls by evaluating mission  essential task assessments that are captured in DRRS.
While DRRS currently has the necessary software to collect and display  these enhanced capability data from organizations at all levels throughout  DOD, a variety of technical and other factors have hindered service  reporting of capability data. As a result, the services have either  developed their own systems to report required readiness data or have  delayed issuing implementing guidance that would require their units to  report standardized mission essential task data in DRRS. By 2005, DRRS  was able to collect and display mission essential task information from  any organizations that had access to a Secure Internet Protocol Router  Network (SIPRNet) workstation. In August 2005, USD (P&R) issued a  memorandum that directed the services to ensure that all of their  GSORTS-reporting units were reporting mission essential task capabilities  in DRRS by September 30, 2005. The memorandum stated that, for  tactical units, mission essential tasks were to be drawn from the Service  Universal Task List and standardized across like-type entities, such as tank  battalions, destroyers, or F-16 squadrons. However, two factors that have  hindered compliance with the memorandum’s direction to report mission  essential task capabilities in DRRS are discussed below.
subsections: section_title: Lack of Direct Access to DRRS
paragraphs: While DRRS has been able to collect and display mission essential task  data since 2005, some Army and Navy users did not have the means to  directly access DRRS and update mission essential task assessments. For  example, some ships lacked hardware necessary to be able to transmit  their mission essential task data directly into DRRS while at sea. In  addition, many National Guard units lacked, and still lack, direct access to  the SIPRNet workstations that are necessary to directly input mission  essential task data directly into DRRS. However, the Army and the Navy  have developed systems, respectively designated DRRS-A and DRRS-N  that interface with DRRS and thus allow all of their units to report mission  essential task data. After Army and Navy units report mission essential  task data in their respective DRRS-A and DRRS-N service systems, the  services transmit these data to DRRS. As a result, Army and Navy officials  told us that they are currently complying with the requirement to ensure  that all their GSORTS-reporting units report mission essential task data in  DRRS.
section_title: Delays in Developing Software Tools to Input Data
paragraphs: Unlike the Army and the Navy, the Marine Corps and the Air Force have  not developed their own systems to allow their units to use a single tool to  enter readiness data to meet Office of the Secretary of Defense, Chairman  of the Joint Chiefs of Staff, and service readiness reporting requirements.  While the DIO has developed the software for users to enter mission  essential task data into DRRS, the DIO has been unsuccessful in attempts  to develop a tool that would allow Air Force and Marine Corps users to  enter readiness data to meet all of their readiness reporting requirements  through DRRS. As a result, rather than reducing the burden on reporting  units, DRRS has actually increased the burden on Air Force and Marine  Corps units because they are now required to report readiness information  in both DRRS and GSORTS. On September 29, 2005, USD (P&R) issued a  memorandum stating that DRRS is the single readiness reporting system  for the entire Department of Defense and that legacy systems, such as  GSORTS and associated service readiness systems, should be phased  out. Since that time, officials have discussed whether to phase out  GSORTS and tentative dates for this action have slipped several times.
In 2001, the Office of the Deputy Undersecretary of Defense (Readiness)  listed reducing reporting burdens as a key characteristic of its envisioned  improved readiness assessment system. In an effort to eliminate this  burden of dual reporting, the DIO began to develop a “current unit status”  tool as a means for users to manage unit-specific readiness data and  submit required reports in support of all current reporting guidelines.  The tool was to minimize the burden associated with dual reporting  by  collecting, displaying, and integrating resource data from service  authoritative data sources with GSORTS and DRRS. However, in  December 2007, the DIO reported that it was unable to deliver the  intended functionality of the “current unit status” tool. Instead, the DIO  decided to develop an interim reporting tool, known as the SORTSREP  tool, which would not provide the type of new capabilities envisioned for  the “current unit status” tool, but would simply replicate the functionality  of the input tool that the Air Force and Marines already used to input data  into GSORTS. After delays, and 10 months of effort, the DIO delivered the  SORTSREP tool to the Marine Corps for review. Based on this review, in  December, 2008, the Marine Corps provided the developers and the DIO  with 163 pages of detailed descriptions and graphics to explain the  SORTSREP tool’s deficiencies. It then informed the DIO that it would no  longer expend energy and resources to review future versions of the  SORTSREP tool and would instead look at leveraging the Army’s or Navy’s  DRRS-A or DRRS-N systems. The Air Force also informed the DIO that it  was no longer interested in the SORSTSREP tool, and said efforts should  be focused on the “current unit status” tool instead. As a result, the Air  Force and Marine Corps are currently faced with dual reporting  requirements, as illustrated in figure 1.
On March 3, 2009, the Air Force Deputy Chief of Staff (Operations, Plans  and Requirements) issued a memorandum that updated the Air Force’s  previous implementing guidance and directed all GSORTS-reporting units  to begin assessing readiness in DRRS based on standardized core task lists  within 90 days. As a result, Air Force units will report readiness in both  DRRS and GSORTS until the DIO is able to deliver the intended  functionality of the “current unit status” tool.
While some Marine Corps units are reporting their capabilities in DRRS,  the Marine Corps had not yet directed its units to report in the system as  of May 2009. The Commandant of the Marine Corps had stated that he  supported the development and implementation of DRRS, but that he  would not direct units to assess mission essential tasks in DRRS until the  system met its stated requirements and was accepted as the single  readiness reporting system of record. Marine Corps officials said that they  did not want to place a burden on operational units, which were fighting  or preparing to fight a war, by requiring that they report readiness in two  different systems. After we completed our audit work, on May 12, 2009, the  Marine Corps issued an administrative message that required that units  assess their mission essential tasks and missions in DRRS. The message  stated that doing so would improve familiarity with DRRS, which will lead  to an easier transition when the Marine Corps fields DRRS-Marine Corps  (DRRS-MC).
Without a viable tool for inputting data, DRRS is not fully integrated with  GSORTS or with the service readiness reporting systems and it is not  capable of replacing those systems since it does not capture the required  data that are contained in those systems.
section_title: DRRS Enhanced Capability Data Are Not Likely to Fully Address the Challenges with Metrics That Existed Prior to the Establishment of the System
paragraphs: While DRRS is being used to provide Congress with enhanced capability  information, the quality of DRRS metrics still faces the same challenges,  including limitations in timeliness, precision, and objectivity that existed  prior to 1999 when Congress directed DOD to establish a new readiness  reporting system. Section 117 of Title 10 of the U.S. Code directed the  Secretary of Defense to establish a comprehensive readiness reporting  system to measure the capability of the armed forces in an “objective,  accurate, and timely manner.” However, the enhanced capability data that  are captured in DRRS and reported to Congress are no more timely than  the readiness data that were being provided to Congress in 1999 using  GSORTS. Furthermore, the metrics that are being used to capture the  enhanced capability information are less objective and precise than the  metrics that were used to report readiness in 1999.
subsections: section_title: Timeliness
paragraphs: The statute directing the development of a new readiness reporting system  requires that the reporting system measure in a timely manner the  capability of the armed forces to carry out the National Security Strategy,  the Secretary of Defense’s defense planning guidance, and the National  Military Strategy. The legislation also lists a number of specific  requirements related to frequency of measurements and updates. For  example, the law requires that the capability of units to conduct their  assigned wartime missions be measured monthly, and that units report any  changes in their overall readiness status within 24 hours of an event that  necessitated the change in readiness status. In its DRRS directive, DOD  assigned USD (P&R) responsibility for ensuring the timeliness of DRRS  information and data, and it specified that DRRS was to be a near-real-time  readiness reporting system.
While DOD is reporting readiness information to Congress on a quarterly  basis as required, and units are measuring readiness on a monthly basis,  DRRS is not a near-real-time reporting system. Specifically, in DRRS, as in  GSORTS, operational commanders assess the readiness of their  organizations on a monthly basis or when an event occurs that changes  the units’ overall reported readiness. Thus, DRRS has not improved the  timeliness of the key readiness data that are reported to Congress.  According to USD (P&R) officials, DRRS data will be more timely than  GSORTS data because DRRS will update underlying data from  authoritative data sources between the monthly updates. However, DRRS  is not yet capturing all the data from the authoritative data sources, and  according to service officials, the service systems that support GSORTS  also draw information from their service authoritative data sources  between the monthly updates. Furthermore, the source and currency of  some of the authoritative data that are currently in DRRS are not clearly  identified. As a result, some users told us that they are reluctant to use  DRRS data to support their decisions.
section_title: Precision and Objectivity
paragraphs: We previously reported that the readiness information that DOD provided  to Congress lacked precision, noting that GSORTS readiness measures  that differed by 10 percentage points or more could result in identical  ratings, with DOD often not reporting the detailed information behind the  ratings outside of the department. For example, units that were at 90 and  100 percent of their authorized personnel strengths both were reported as  P-1 in DOD’s reports to Congress. In 2003, USD (P&R) recognized the  imprecision of the reported metrics from GSORTS and noted that its  efforts to develop DRRS would allow enhancements to reported readiness  data.
As previously noted, the DRRS capability information that DOD is  reporting to Congress covers a broader range of missions than the  GSORTS information that was provided in the past. However, when  comparing the DRRS and GSORTS assessments of units’ capabilities to  perform the missions for which the units were organized or designed,  DRRS assessments are actually less precise than the GSORTS  assessments. Specifically, within GSORTS, overall capability assessments  are grouped into four categories based on four percentage ranges for the  underlying data. For example, commanders compare on-hand and required  levels of personnel and equipment. Within DRRS, mission essential task  assessments are reported on a scale that includes only three ratings— ”yes”, “no”, and “qualified yes,” which can include any assessments that  fall between the two extremes.
The law directing DOD to establish a new readiness reporting system also  requires that the system measure readiness in an objective manner.  GSORTS assessments of units’ capabilities to execute the missions for  which they were organized or designed are based on objective personnel  and equipment data and training information that may include both  objective and subjective measures. Furthermore, the overall capability  assessment in GSORTS is based on an objective rule that calls for the  overall assessment to be the same level as the lowest underlying resource  or training data level. For example, if a unit reported the highest personnel  level (P-1) and the lowest training level (T-4), the rules in the GSORTS  guidance instruct the commander to rate the unit’s overall capability at the  C-4 level. Because GSORTS contains these objective measures and rules, it  is easy to evaluate reported readiness to see if it aligns with established  reporting criteria.
Within DRRS, organizations rate their capabilities based on mission  essential tasks. These mission essentials tasks have conditions and  standards associated with them. The conditions specify the types of  environments that units are likely to face as they execute the tasks, such  as weather conditions and political or cultural factors. Standards describe  what it means for the unit to successfully execute the task under specified  conditions. For example, a unit may have to achieve a 90 percent success  rate for measures associated with the task being assessed. In spite of these  conditions and standards, DRRS mission assessments are often subjective  rather than objective. In DRRS program guidance, DOD has defined  mission essential tasks as tasks that are approved by the commander and  that, based on mission analysis, are “absolutely necessary, indispensable,  or critical to mission success.” In prior briefings and reports to Congress,  we have noted examples that highlight the subjective nature of DRRS  mission assessments. For example, we noted that one commander used  his professional judgment to decide that his command was “qualified” to  execute a mission even though the preponderance of the “indispensable”  tasks that supported that mission were rated as “no.” In contrast, other  commanders used their professional judgments to rate missions as  “qualified” based on one or more “qualified” tasks among many “yes” tasks.
section_title: DRRS Lacks the Complete Resource and Training Data and System Connectivity Some Users Need to Manage and Deploy Forces
paragraphs: DRRS does not have all of the resource, training, readiness data, and  connectivity with the department’s operations planning and execution  system that the services, Joint Staff, and certain combatant commands  need to manage and deploy forces. As a result, DRRS is not yet able to  operate as the department’s single readiness reporting system, as  intended. The Secretary of Defense’s and the Under Secretary of Defense’s  guidance documents recognize that DRRS needs to support the data  requirements of multiple users. For example, the Secretary of Defense’s  June 25, 2004, memorandum directed USD (P&R) to develop DRRS to  support the data requirements identified by the Chairman of the Joint  Chiefs of Staff, the combatant commanders, the Secretaries of the military  departments, and the Chief of the National Guard Bureau. Furthermore,  the 2002 DRRS directive noted that DRRS was to build upon DOD’s  existing processes and readiness assessment tools and that ESORTS  information (capability, resource, and training), where appropriate, is  integrated into DOD’s planning systems and processes. It also directed the  Chairman of the Joint Chiefs of Staff to maintain the GSORTS database  until key capabilities of DRRS become fully operational.
subsections: section_title: Complete and Accurate Current and Historical Data
paragraphs: Officials with U.S. Joint Forces Command and U.S. Special Operations  Command reported that historical data are needed to manage forces and  provide users the ability to analyze readiness trends. Similarly, service  officials stated a need for historical data so they can manage their forces  and take action to address readiness issues. In 2005, USD (P&R) reported  that unit resource data, including detailed inventory and authorization  data on personnel, equipment, supply, and ordnance were available in  DRRS. However, in response to a survey we conducted in December 2008,  the services and certain combatant commands stated that necessary  current and historical resource and training data were not available in  DRRS. For example, officials from all four services responded that DRRS,  at that time, contained less than half of their GSORTS resources and  training data. In addition, officials from U.S. Joint Forces Command, U.S.  Special Operations Command, the U.S. Strategic Command, and the U.S.  Transportation Command all responded that historical resource data were  not available in DRRS. We confirmed that this information was still not  available when we concluded our review, and in April, 2009, the DIO said  it was still working on this data availability issue.
Furthermore, user organizations have reported inaccuracies in the data  that are available in DRRS. Marine Corps and U.S. Special Operations  Command officials stated that inconsistencies between DRRS data and the  data in other readiness systems have caused them to adjudicate the  inconsistencies by contacting their subordinate units directly. Army  officials noted that searches of DRRS data can produce different results  than searches in the Army’s data systems. For example, they noted that a  DRRS search for available personnel with a particular occupational  specialty produced erroneously high results because DRRS did not employ  the appropriate business rules when conducting the search. Specifically,  DRRS did not apply a business rule to account for the fact that an  individual person can possess multiple occupational specialty codes but  can generally fill only one position at a time. DIO officials informed us that  they intend to correct issues with the accuracy of data drawn from  external databases. However, the current version of the DRRS Integrated  Master Schedule indicates that the ability of DRRS to provide the  capability to correctly search, manipulate, and display current and  historical GSORTS and mission essential task data will not be complete  until June 2010. As a result, the reliability of the DRRS data is likely to  remain questionable and a number of DOD organizations will likely  continue to rely on GSORTS and other sources of readiness data to  support their decision making.
section_title: Connections to Planning Systems
paragraphs: One important DRRS function is integration with DOD’s planning systems.  Specifically, the 2002 DRRS directive requires USD (P&R) to ensure that,  where appropriate, ESORTS information (capability, resource, and  training) is compatible and integrated into DOD’s planning systems and  processes. Global force management is one of the DOD planning  processes that is to be integrated with DRRS. Global Force Management is  a process to manage, assess, and display the worldwide disposition of U.S.  forces, providing DOD with a global view of requirements and availability  of forces to meet those requirements. The integration of DRRS with global  force management planning processes is supposed to allow DOD to link  force structure, resources, and capabilities data to support analyses, and  thus help global force managers fill requests for forces or capabilities.
Officials from the four organizations with primary responsibilities for  providing forces (U.S. Joint Forces Command, U.S. Special Operations  Command, U.S. Strategic Command, and U.S. Transportation Command)  all stated that they are unable to effectively use DRRS to search for units  that will meet requested capabilities. These commands also reported that  DRRS does not currently contain the information and tools necessary to  support global force management. For example, officials from U.S.  Northern Command told us that when they used DRRS to search for  available helicopters of a certain type, they found thousands, but when  U.S. Joint Forces Command did the same DRRS search they found  hundreds. The current version of the DRRS Integrated Master Schedule  indicates that DRRS will not be able to fully support global force  management until March 2011. As a result, these commands continue to  rely on GSORTS rather than DRRS to support their planning and sourcing  decisions.
section_title: Conclusions
paragraphs: DRRS is not currently and consistently providing timely, objective, and  accurate information, and it is not exactly clear where the department  stands in its efforts to meet this expectation because system requirements  remain in a state of flux, and the program office lacks disciplined program  management and results information due to a long-standing lack of rigor in  its approach to acquiring and deploying system capabilities. This situation  can be attributed, in part, to long-standing limitations in the program  office’s focus on acquiring human capital skills needed to manage such a  complex initiative. It can also be linked to many of years of limited  program office oversight and accountability. Although program oversight  has recently increased, oversight bodies have not had sufficient visibility  into the program’s many management weaknesses.
DRRS is providing Congress and readiness users with additional mission  and mission essential task capability data that were not available in  GSORTS. However, after investing about 7 years and about $96.5 million in  developing and implementing DRRS, the system’s schedule has been  extended, requirements are not stable, and the system still does not meet  congressional and DOD requirements for a comprehensive readiness  reporting system to assess readiness and help decision makers manage  forces needed to conduct combat and contingency operations around the  world. Given DRRS performance and management weaknesses, it is  critical that immediate action be taken to put the program on track and  position it for success. Without this action, it is likely that DRRS will cost  more to develop and deploy than necessary and that DOD will not have a  comprehensive reporting system that meets the needs of all the decision  makers who rely on accurate, timely, and complete readiness information.
section_title: Recommendations for Executive Action
paragraphs: To address the risks facing DOD in its acquisition and deployment of  DRRS, and to increase the chances of DRRS meeting the needs of the DOD  readiness community and Congress, we recommend that the Secretary of  Defense direct the Deputy Secretary of Defense, as the Chair of the  Defense Business Systems Management Committee, to reconsider the  committee’s recent approval of DRRS planned investment for fiscal years  2009 and 2010, and convene the Defense Business Systems Management  Committee to review the program’s past performance and the DIO’s  capability to manage and deliver DRRS going forward.
To fully inform this Defense Business Systems Management Committee  review, we also recommend that the Secretary direct the Deputy Secretary  to have the Director of the Business Transformation Agency, using the  appropriate team of functional and technical experts and the established  risk assessment methodology, conduct a program risk assessment of  DRRS, and to use the findings in our report and the risk assessment to  decide how to redirect the program’s structure, approach, funding,  management, and oversight. In this regard, we recommend that the  Secretary direct the Deputy Secretary to solicit the advice and  recommendations of the DRRS Executive Committee.
We also recommend that the Secretary, through the appropriate chain of  command, take steps to ensure that the following occur:  1.  DRRS requirements are effectively developed and managed with  appropriate input from the services, Joint Staff, and combatant  commanders, including (1) establishing an authoritative set of baseline  requirements prior to further system design and development; (2)  ensuring that the different levels of requirements and their associated  design specifications and test cases are aligned with one another; and  (3) developing and instituting a disciplined process for reviewing and  accepting changes to the baseline requirements in light of estimated  costs, benefits, and risk. 2.  DRRS testing is effectively managed, including (1) developing test  plans and procedures for each system increment test event that  include a schedule of planned test activities, defined roles and  responsibilities, test entrance and exit criteria, test defect management  processes, and metrics for measuring test progress; (2) ensuring that  all key test events are conducted on all DRRS increments; (3)  capturing, analyzing, reporting, and resolving all test results and test  defects of all developed and tested DRRS increments; and (4)  establishing an effective test management structure that includes  assigned test management roles and responsibilities, a designated test  management lead and a supporting working group, and a reliable  schedule of test events. 3.  DRRS integrated master schedule is reliable, including ensuring that  the schedule (1) captures all activities from the work breakdown  structure, including the work to be performed and the resources to be  used; (2) identifies the logical sequencing of all activities, including  defining predecessor and successor activities; (3) reflects whether all  required resources will be available when needed and their cost; (4)  ensures that all activities and their duration are not summarized at a  level that could mask critical elements; (5) achieves horizontal  integration in the schedule by ensuring that all external interfaces  (hand-offs) are established and interdependencies among activities are  defined; (6) identifies float between activities by ensuring that the  linkages among all activities are defined; (7) defines a critical path that  runs continuously to the program’s finish date; (8) incorporates the  results of a schedule risk analysis to determine the level of confidence  in meeting the program’s activities and completion date; and (9)  includes the actual start and completion dates of work activities  performed so that the impact of deviations on downstream work can  be proactively addressed. 4.  The DRRS program office is staffed on the basis of a human capital  strategy that is grounded in an assessment of the core competencies  and essential knowledge, skills, and abilities needed to perform key  DRRS program management functions, an inventory of the program  office’s existing workforce capabilities, and an analysis of the gap  between the assessed needs and the existing capabilities.  5.  DRRS is developed and implemented in a manner that does not  increase the reporting burden on units and addresses the timeliness,  precision, and objectivity of metrics that are reported to Congress.
To ensure that these and other DRRS program management improvements  and activities are effectively implemented and that any additional funds for  DRRS implementation are used effectively and efficiently, we further  recommend that the Secretary direct the Deputy Secretary to ensure that  both the Human Resources Management Investment Review Board and  the DRRS Executive Committee conduct frequent oversight activities of  the DRRS program, and report any significant issues to the Deputy  Secretary.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, signed by the Deputy Under  Secretary of Defense (Military Personnel Policy) performing the duties of  the Under Secretary of Defense (Personnel and Readiness), DOD stated  that the report is flawed in its assessment of DRRS, noting that DRRS is a  net-centric application that provides broad and detailed visibility on  readiness issues, and that achieving data sharing across the DOD  enterprise was groundbreaking work fraught with barriers and obstacles,  many of which have now been overcome. In addition, DOD stated that it  was disappointed that the report did not address cultural impediments  that it considers to be the root cause of many of the issues cited in the  report and of many previous congressional concerns on readiness  reporting. DOD further stated that the report instead focuses on past  acquisition process and software development problems that it believes  have now been remedied According to the department, this focus, coupled  with inaccurate and misleading factual information included in the report,  led us to develop an incomplete picture of the program. Notwithstanding  these comments, DOD agreed with two of our recommendations and  partially agreed with a third. However, it disagreed with the remaining five  recommendations, and provided comments relative to each  recommendation. DOD’s comments are reprinted in their entirety in  appendix III.
In summary, we do not agree with DOD’s overall characterization of our  report or the positions it has taken in disagreeing with five of our  recommendations, finding them to be inconsistent with existing guidance  and recognized best practices on system acquisition management,  unsupported by verifiable evidence, and in conflict with the facts detailed  in our report. Further, we recognize that developing DRRS is a significant  and challenging undertaking that involves cultural impediments. As a  result, our report explicitly focuses on the kind of program management  rigor and disciplines needed to address such impediments and  successfully acquire complex systems, including effective requirements  development and management and executive oversight. We also disagree  that our report focuses on past issues and problems. Rather, it provides  evidence that demonstrates a long-standing and current pattern of system  acquisition and program oversight weaknesses that existed when we  concluded our audit work and that DOD has not provided any evidence to  demonstrate has been corrected.
In addition, we would emphasize that we defined our objectives, scope,  and methodology, and executed our audit work in accordance with  generally accepted government auditing standards, which require us to  subject our approach as well as the results of our audit work to proven  quality assurance checks and evidence standards that require us to seek  documentation rather than relying solely on testimonial evidence. While  we support any departmental efforts, whether completed or ongoing, that  would address the significant problems cited in our report, we note that  DOD, in its comments, did not specifically cite what these efforts are or  provide documentation to support that they have either been completed or  are ongoing. Therefore, we stand by our findings and recommendations.  Moreover, we are concerned that in light of the program’s significant and  long-standing management weaknesses, the department’s decision not to  pursue recommendations aimed at corrective actions for five of our eight  recommendations will further increase risk to achieving program success,  and is not in the best interests of the military readiness community or the  U.S. taxpayer. Accordingly, we encourage the department to reconsider its  position when it submits its written statement of the actions taken on our  recommendations to the Senate Committee on Homeland Security and  Governmental Affairs and the House Committee on Oversight and  Government Reform , as well has the House and Senate Committees on  Appropriations, as required under 31 U.S.C. 720.
DOD’s specific comments on each recommendation, along with our  responses to its comments follow.
The department did not agree with our recommendation for the Deputy  Secretary of Defense, as the Chair of the Defense Business Systems  Management Committee, to reconsider the committee’s recent  approval of DRRS planned investment for fiscal years 2009 and 2010,  and to convene the Defense Business Systems Management Committee  to review the program’s past performance and the DIO’s capability to  manage and deliver DRRS going forward in deciding how best to  proceed. In this regard, DOD stated that the Investment Review Board  certification and Defense Business Systems Management Committee  approval were granted in compliance with the established processes. It  also added that oversight of the specific issues identified in this report  are the responsibility of the DRRS Executive Committee, which it  stated has and will continue to provide appropriate governance for this  effort. It also stated that USD (P&R) will ensure that the program is  compliant with all acquisition requirements prior to submission for  further certifications.
We do not question whether the Investment Review Board certification  and Defense Business Systems Management Committee approval were  provided in accordance with established processes, as this is not  relevant to our recommendation. Rather, our point is that the  Investment Review Board and Defense Business Systems Management  Committee were provided, and thus based their respective decisions,  on erroneous and incomplete information about DRRS progress,  management weaknesses, and risks. Moreover, neither the Investment  Review Board nor the Defense Business Systems Management  Committee were informed about the findings in our report, even  though we shared each of them with the DRRS program director and  other DIO officials prior to both the Investment Review Board and the  Defense Business Systems Management Committee deliberations.  Therefore, while the Investment Review Board certification and the  Defense Business Systems Management Committee approval were  granted in accordance with established processes, they were not based  on a full disclosure of facts. Moreover, while we support DOD’s  comment that it will ensure that the program is in compliance with all  acquisition requirements prior to further certifications, nothing  precludes the board or the committee from reconsidering their  respective decisions in light of our report. With respect to DOD’s  comment that the DRRS Executive Committee has and will continue to  provide appropriate governance for this effort, we do not disagree that  the DRRS Executive Committee has an oversight role. However, the  DRRS Executive Committee should not be solely responsible for  oversight of the specific issues in our report. Both the Investment  Review Board and the Defense Business Systems Management  Committee provide additional layers of oversight pursuant to law and  DOD policy. Accordingly, we stand by our recommendation as it  appropriately seeks to have the Investment Review Board and Defense  Business Systems Management Committee, in collaboration with the  DRRS Executive Committee, act in a manner that is consistent with  their respective roles as defined in law. In doing so, our intent is to  promote accountability for DRRS progress and performance, and  prompt action to address the many risks facing the program.
The department agreed with our recommendation for the Deputy  Secretary of Defense, as the chair of the Defense Business Systems  Management Committee, to have the Business Transformation Agency  conduct a risk assessment of DRRS, and with the advice and  recommendation of the DRRS Executive Committee, to use the results  of this assessment and the findings in our report to decide how to  redirect the program. In this regard, the department stated that this  assessment will be complete by the middle of fiscal year 2010.
The department did not agree with our recommendation for ensuring  that DRRS requirements are effectively developed and managed. In this  regard, it stated that the program has an authoritative set of baseline  requirements established with an effective governance process for  overseeing the requirements management process, to include biweekly  reviews as part of the DRRS configuration control process.
We do not agree. At the time we concluded our work, DRRS  requirements were not stable, as evidenced by the fact that an  additional 530 requirements had been identified that the DIO was still  in the process of reviewing and had yet to reach a position on their  inclusion, or process them through the DRRS change control  governance process. Moreover, when we concluded our work, this  change control process had yet to be approved by the DRRS  governance structure. As we state in our report, the introduction of  such a large number of requirements provided a compelling basis for  concluding that requirements had yet to progress to the point that they  could be considered sufficiently complete and correct to provide a  stable baseline. Our recommendation also noted that the Secretary  should take steps to ensure that the different levels of requirements be  aligned with one another. DOD’s comments did not address this aspect  of our recommendation.
The department did not agree with our recommendation for ensuring  that DRRS testing is effectively managed. In this regard, it stated that  DRRS testing is already in place and performing effectively, and stated,  among other things, that (1) the DIO goes through a rigorous testing  regimen that includes documenting test plans with user test cases for  each incremental release to include utilizing system integration,  acceptance, interoperability, and operational testing; (2) user test cases  and functionality are validated by designated testers independent of the  developers prior to a deployment; and (3) for interoperability testing  the DIO has a designated test director and the Joint Interoperability  Test Command (JITC) is the designated interoperability and  operational test activity.
We do not agree. As our report concludes, DRRS testing has not been  effectively managed because it has not followed a rigorous testing  regimen that includes documented test plans, cases, and procedures.  To support this conclusion, our report cites numerous examples of test  planning and execution weaknesses, as well as the DIO’s repeated  inability to demonstrate through requisite documentation that the  testing performed on DRRS has been adequate. Our report shows that  test events for already acquired, as well as currently deployed and  operating, DRRS releases and subreleases were not based on well- defined plans and DOD had not filled its testing director vacancy.  Further, our report shows that test events were not fully executed in  accordance with plans that did exist, or executed in a verifiable  manner, or both. For example, although increments of DRRS  functionality had been put into production, the program had no  documentation (e.g., test procedures, test cases, test results) to show  that the program office had performed system integration testing,  system acceptance testing, or operational testing on any DRRS release  or subrelease, even though the DIO’s test strategy stated that such tests  were to be performed before system capabilities became operational.  Moreover, evidence showed that the results of all executed test events  had not been captured and used to ensure that problems discovered  were disclosed to decision makers, and ultimately corrected. With  respect to DOD’s comments that JITC is the designated lead for  interoperability and operational testing, our report recognizes that JITC  is to conduct both interoperability and operational testing before the  system is deployed and put into production (i.e., used operationally).
However, during the course of our audit, the DIO could not produce  any evidence to show that interoperability and operational testing of all  operating system increments had been conducted.
The department did not agree with our recommendation for ensuring  that the DRRS integrated master schedule is reliable. In this regard, it  stated that a process is already in place to ensure that the schedule is  current, reliable, and meets all the criteria outlined in the  recommendation.
We do not agree. As our report states, an integrated master schedule  for DRRS did not exist until November 2008, which was 2 months after  we first requested one. Moreover, following our feedback to the DIO on  limitations in this initial version, a revised integrated master schedule  was developed in January 2009, which was also not reliable.  Subsequently, a revised integrated master schedule was developed in  April 2009. However, as we detail in our report, that version still  contained significant weaknesses. For example, it did not establish a  critical path for all key activities or include a schedule risk analysis,  and was not being updated using logic and durations to determine the  dates for all key activities. These practices are fundamental to  producing a sufficiently reliable schedule baseline that can be used to  measure progress and forecast slippages. In addition, the schedule  introduced considerable concurrency across key activities and events  for several modules, which introduces increased risk. Therefore, we  stand by our recommendation.
The department partially agreed with our recommendation for ensuring  that it has an effective human capital strategy. In this regard, it stated  that actions are underway to add more full-time civilian support to the  DIO, and that plans exist to convert some contractor to civilian billets  during the 2010/2011 time frame.
We support the department’s actions and plans described in its  comments to address the DIO human capital management limitations  discussed in our report, but would note that they do not go far enough  to systematically ensure that the program has the right people with the  right skills to manage the program in both the near term and the long  term. To accomplish this, the department needs to adopt the kind of  strategic and proactive approach to DRRS workforce management that  our report describes and our recommendation embodies. As our  evaluations and research show, failure to do so increases the risk that  the program office will not have the people it needs to effectively and  efficiently manage DRRS. Therefore, we believe that the department  needs to fully implement our recommendation.
The department did not agree with our recommendation to take steps  to ensure that DRRS is developed and implemented in a manner that  does not increase the reporting burden on units and addresses the  timeliness, precision, and objectivity of metrics that are reported to  Congress. In this regard, it stated that one of the primary tenets of  DRRS has been to reduce reporting requirements on the war fighter. It  also stated that DRRS is already using state-of-the-art technology to  ensure that near-real-time data are available for the war fighters.  Finally it stated that the DRRS governance structure that is currently in  place ensures that DRRS development does not deviate from these core  principles.
While we recognize that a goal of DRRS is to reduce a reporting burden  on the war fighter, we disagree with the department’s position because  the system has not yet achieved this goal. As our report states, while  the DIO has developed the software for users to enter mission essential  task data into DRRS, the DIO has been unsuccessful in attempts to  develop a tool that would allow Air Force and Marine Corps users to  enter readiness data to meet all of their readiness reporting  requirements through DRRS. As a result, rather than reducing the  burden on reporting units, DRRS actually increased the burden on Air  Force and Marine Corps units because they were required to report  readiness information in both DRRS and GSORTS. Without a viable  tool for inputting data, DRRS is not fully integrated with GSORTS or  with the service readiness reporting systems and it is not capable of  replacing those systems since it does not capture the required data that  are contained in those systems. In addition, the DRRS readiness data  that are currently reported to Congress are not near-real-time data.  Specifically, the periodicity for DRRS capability assessments is the  same as the legacy GSORTS system’s readiness reports—monthly or  when an event occurs that changes a unit’s overall readiness.  Furthermore, our report shows that DRRS mission assessments are  often subjective and imprecise because they are reported on a scale  that includes only three ratings—”yes,” “no,” and “qualified yes,” which  can include any assessments that fall between the two extremes.  Therefore, because additional actions are still needed to reduce  reporting burdens and improve the timeliness, precision, and  objectivity of the DRRS data that are reported to Congress, we stand by  our recommendation.
The department agreed with our recommendation for ensuring that  both the Human Resources Management Investment Review Board and  the DRRS Executive Committee conduct frequent oversight activities  of the DRRS program and report any significant issues to the Deputy  Secretary of Defense. In this regard, the department stated that the  USD (P&R) component acquisition executive is working with the  program to ensure that it becomes fully compliant with all acquisition  requirements. In addition, it stated that the acquisition executive will  certify to the Human Resources Investment Review Board and the  Deputy Chief Management Officer of compliance prior to submission of  future certification requests. Further, it stated that the current DRRS  governance process will provide sustained functional oversight of the  program and that issues that arise in any of these areas will be elevated  for review, as appropriate. We believe these are positive steps.
We are sending copies of this report to the appropriate congressional  committees; the Secretary of Defense; and the Director, Office of  Management and Budget. The report will also be available at no charge on  the GAO Web site at http://www.gao.gov.
If you or your staffs have questions about this report, please contact us at  pickups@gao.gov or hiter@gao.gov or at our respective phone numbers,  (202) 512-9619 and (202) 512- 3439. Contact points for our Offices of   Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made key contributions to this report are  listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to assess the extent to which (1) the Department of  Defense (DOD) has effectively managed and overseen DRRS acquisition  and deployment, and (2) features of the Defense Readiness Reporting  System (DRRS) have been implemented and are consistent with legislative  requirements and DOD guidance.
We did not evaluate the department’s overall ability to assess the readiness  of its forces or the extent that data contained in any of its readiness  reporting systems, including DRRS and the Global Status of Resources and  Training System (GSORTS), reflect capabilities, deficiencies,  vulnerabilities, or performance issues. Our review focused on acquisition  and program management issues, such as requirements management,  schedule, and human capital requirements; the current usage of DRRS; and  the extent to which DRRS’ features address legislative requirements and  DOD guidance.
To determine the extent to which the DRRS acquisition and deployment  has been effectively managed and overseen, we focused on the following  acquisition management areas: (1) requirements development and  management, (2) test planning and execution, (3) DRRS schedule  reliability, and (4) human capital planning. In doing so, we analyzed a  range of program documentation, such as high-level and detailed-level  requirements documentation, test plans and reports, the current DRRS  schedule, and program management documentation and interviewed  cognizant program and contractor officials.
To determine the extent to which the program had effectively  implemented requirements development and management, we  reviewed relevant program documentation, such as the concept of  operations document, capability requirements document, software  requirements document, requirements traceability matrix,  configuration management plan, and the program management plan, as  well as minutes of change control board meetings, and evaluated them  against relevant guidance. Moreover, we reviewed briefing slides from  meetings of DRRS oversight bodies in order to identify concerns about  DRRS expressed by representatives from the DRRS community of  users, as well as the efforts by the Joint Staff (at the direction of DRRS  Executive Committee) to identify and address any gaps identified by  users in the development of DRRS requirements. To determine the  extent to which the program has maintained traceability backward to  high-level business operation requirements and system requirements,  and forward to system design specifications and test plans, we  randomly selected 60 program requirements and traced them both  backward and forward. This sample was designed with a 5 percent  tolerable error rate at the 95 percent level of confidence so that, if we  found zero problems in our sample, we could conclude statistically that  the error rate was less than 5 percent. In addition, we interviewed  program and development contractor officials to discuss the  requirements development and management process.
To determine if the DRRS Implementation Office (DIO) is effectively  managing DRRS testing, we reviewed relevant documentation, such as  the DRRS Test and Evaluation Master Plans and test reports and  compared them to DOD and other relevant guidance. Further, we  reviewed developmental test plans and procedures for each  release/subrelease that to date has either been developed or fielded and  compared them with best practices to determine whether test activities  had been adequately documented. We also examined test results and  reports for the already acquired, as well as currently deployed and  operating, DRRS releases and subreleases and compared them against  plans to determine whether they had been executed in accordance with  plans. Moreover, we reviewed key test documentation, such as the  Software Version Descriptions, and compared them against relevant  guidance to determine whether defect data were being captured,  analyzed, prioritized, and reported. We also interviewed program and  contractor officials to gain clarity beyond what was included in the  program documentation, including the Defense Information Systems  Agency’s Joint Interoperability Test Center in order to determine the  results of their efforts to independently test DRRS interoperability. In  addition, to determine the extent to which the program had effectively  tested its system requirements, we observed the DIO’s efforts to  demonstrate the traceability of 60 program requirements to test cases  and results. This sample was designed with a 5 percent tolerable error  rate at the 95 percent level of confidence so that, if we found zero  problems in our sample, we could conclude statistically that the error  rate was less than 5 percent.
To determine the extent to which the program’s schedule reflects key  estimating practices that are fundamental to having a reliable schedule,  we reviewed the DRRS integrated master schedules and schedule  estimates and compared them with relevant guidance. We also used  schedule analysis software tools to determine whether the latest  schedule included key information, such as the activities critical to on- time completion of DRRS, a logical sequence of activities, and evidence  that the schedule was periodically updated. We also reviewed the  schedule to determine the time frames for completing key program  activities and to determine any changes to key milestones. In addition,  we shared the results of our findings with program and contractor  officials and asked for clarifications. We then reviewed the revised  schedule, prepared in response to the weaknesses we found, and  compared it with relevant guidance.
To evaluate whether DOD is adequately providing for the DRRS  program’s human capital needs, we compared the program’s efforts  against relevant criteria and guidance, including our own framework  for strategic human capital management. In doing so, we reviewed key  program documentation, such as the program management plan and  the DIO organizational structure to determine whether it reflected key  acquisition functions and identified whether these functions were being  performed by government or contractor officials. We interviewed key  officials to discuss workforce analysis and human capital planning  efforts.
To determine the level of oversight and governance available to the  DRRS community of users, we attended relevant meetings, met with  officials responsible for program certification, and reviewed relevant  guidance and program documentation. Specifically, we attended Battle  Staff meetings and analyzed briefing slides and meeting minutes from  the DRRS Executive Committee, General and Flag Officer’s Steering  Committee, and Battle Staff meetings—the main DRRS governance  bodies. In addition, we reviewed key DRRS certification and approval  documentation provided by the Human Resources Management  Investment Review Board, such as economic viability analyses and the  business system certification dashboard and met with Investment  Review Board officials to determine the basis for certifying and  approving DRRS.
To determine the extent to which the features of DRRS have been  implemented and are consistent with legislative requirements and DOD  guidance, we first examined the language of Section 117 of Title 10 of the  United States Code, which directs the Secretary of Defense to establish a  comprehensive readiness reporting system. We identified criteria for this  system in DOD’s directive formally establishing the system. We evaluated  the system by conducting interviews—see table 2 below for a list of these  organizations—and receiving system demonstrations from members of the  readiness community to determine how they used DRRS and how their  usage compared with the criteria established for the system. We also  conducted content and data analysis of system documents and briefing  packages provided by the DIO and Joint Staff. In order to capture the  broadest amount of data about the system we conducted a survey of  readiness offices at all of the service headquarters, combatant commands,  and the National Guard Bureau regarding how DRRS was currently being  used and the types and amount of data available in the system. In  addition, to track the development of DRRS capabilities, we attended  Battle Staff meetings and analyzed documentation from meetings of all the  DRRS governance bodies. We also searched for and extracted information  from DRRS in order to support other GAO ongoing readiness reviews.  While our usage of the system was not intended as a formal test of the  system, our general observations concerning system functionality and the  range of available data were consistent with the observations of most  other users, which were noted in our survey.
We conducted our work from April 2008 through August 2009 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
section_title: Appendix II: Detailed Analysis of DRRS Acquisition and Deployment Management and Oversight
paragraphs: Our research and evaluations of information technology programs have  shown that the ability to deliver promised system capabilities and benefits  on time and within budget largely depends on the extent to which key  program management disciplines are employed by an adequately staffed  program management office. Among other things, these disciplines include  a number of practices associated with effectively developing and  managing system requirements, adequately testing system capabilities, and  reliably scheduling the work to be performed. They also include  proactively managing the program office’s human capital needs, and  promoting program office accountability through effective program  oversight. Department of Defense (DOD) acquisition policies and  guidance, along with other relevant guidance, recognize the importance of  these management and oversight disciplines. As we have previously  reported, not employing these and other program management disciplines  increases the risk that system acquisitions will not perform as intended  and require expensive and time-consuming rework. Defense Readiness  Reporting System (DRRS) acquisition and deployment has for years not  been effectively managed in accordance with these key program  management disciplines that are recognized in DOD and other relevant  guidance, and are fundamental to delivering a system that performs as  intended on time and within budget.
subsections: section_title: DRRS Requirements Have Not Been Effectively Developed and Managed
paragraphs: Well-defined and well-managed requirements are a cornerstone of  effective system development and acquisition. According to recognized  guidance, documenting and implementing a disciplined process for  developing and managing requirements can help to reduce the risks of  producing a system that is not adequately tested, does not meet user  needs, and does not perform as intended. Effective requirements  development and management includes, among other things, (1)  effectively eliciting user needs early and continuously in the system life- cycle process, (2) establishing a stable baseline set of requirements and  placing this baseline under configuration management, (3) ensuring that  system requirements are traceable backward to higher level business or  operational requirements (e.g., concept of operations) and forward to  system design documents (e.g., software requirements specification) and  test plans, and (4) controlling changes to baseline requirements.
DRRS requirements have not been effectively developed and managed.  Specifically, (1) key users have only recently become engaged in  developing requirements, (2) requirements have been experiencing  considerable change and are not yet stable, (3) different levels of  requirements and related test cases have not been aligned with one  another, and (4) changes to requirements have not been effectively  controlled. As a result, efforts to develop and deliver initial DRRS  capabilities have taken longer than envisioned and these capabilities have  not lived up to the readiness communities’ expectations. These failures  increase the risk of future DRRS capabilities not meeting expectations and  ensure that expensive and time-consuming system rework will be  necessary.
subsections: section_title: Recent Actions Have Been Taken to Address Limited User Involvement in Developing and Managing Requirements
paragraphs: One of the leading practices associated with effective requirements  development is engaging system users early and continuously in the  process of defining requirements. As we have previously reported,  assessing user needs early in the process increases the probability of  success in defining, designing, and delivering a system that meets user  needs and performs as intended.
To the DRRS Implementation Office’s (DIO) credit, the October 2008  DRRS Risk Management Plan recognizes this by stating that the success of  DRRS depends on participation and support from the broad readiness  community, which includes combatant commands, Joint Staff, and the  military services. However, until recently, key users were not effectively  engaged in DRRS requirements development and management, although  reasons vary why they have not. Specifically, DIO officials told us that  beginning in 2002, they reached out to all user groups—combatant  commands, Joint Staff, and the military services—in defining  requirements. For example, they cited a July 2002 memorandum issued by  the Office of the Under Secretary of Defense for Personnel and Readiness  (USD P&R) that encouraged the Director of the Joint Chiefs of Staff,  Deputy Commanders of the Combat Commands, Service Operations  Deputies, and Directors of Defense Agencies to actively support the DRRS  effort by ensuring that their organizations are represented at Battle Staff  meetings. However, these officials told us that the military services and  Joint Staff chose not to participate.
In contrast, officials from these user groups told us their involvement had  been limited by what they characterized as difficulties in submitting  requirements through the DRRS governance boards that were in place at  that time. For example, an official from the Joint Forces Command said  that the Forces Battle Staff governance board did not meet for about a  year between 2005 and 2006. Further, the official said that the meetings  that were held did not offer users the opportunity to discuss their  concerns or influence the requirements process. Similarly, an official from  the Marine Corps cited a lack of clear and transparent communication  from the DIO as a significant impediment.
Notwithstanding this lack of stakeholder involvement in setting  requirements, the Office of USD (P&R) developed and issued a DRRS  concept of operations in 2004, which DIO officials told us was based on  input from the combatant commands, relevant departmental directives,  and DRRS governance boards (e.g., Battle Staff). In our view, this  document provided a high-level overview of proposed DRRS capabilities  from which more detailed requirements could be derived. However, the  concept of operations was not approved by all key players in the readiness  community. Specifically, DIO officials stated that the document had not  been approved by the military services and the Joint Staff. According to  these officials, the reason for not seeking all stakeholders’ approval, and  the decision to begin developing more detailed requirements in the  absence of an approved concept of operations, was that the 2002 DRRS  DOD directive provided a sufficient basis to begin developing and  deploying what they anticipated being the initial versions of DRRS.
In 2008, after 6 years of effort to define DRRS requirements and develop  and deploy system capabilities, the Joint Staff, at the direction of the  DRRS Executive Committee, conducted an analysis of DRRS  capabilities—referred to as the “capabilities gap analysis.” To the Joint  Staff’s credit, this analysis has appropriately focused on soliciting  comments from the entire readiness community and on identifying any  gaps between the DRRS requirements and the needs of this community. As  will be discussed in the next section, this analysis resulted in 530  additional user requirements.
The extended period of limited involvement by key DRRS users in defining  a concept of operations and related capabilities and requirements has  impeded efforts to develop a clear understanding of DRRS expectations,  constraints, and limitations, which, in turn, has contributed to significant  delays in providing the readiness community with needed system support.  While the recent Joint Staff action to engage the entire DRRS user  community is a positive step towards overcoming this long-standing  problem, it remains to be seen whether this engagement will produce  agreement and commitment across the entire readiness user community  around DRRS requirements.
section_title: DRRS Requirements Are Not Stable
paragraphs: As previously noted, establishing an authoritative set of baseline  requirements prior to system design and development is necessary to  design, develop, and deliver a system that performs as intended and meets  users’ operational needs. In general, a baselined set of requirements are  those that are defined to the point that extensive changes are not  expected, placed under configuration management, and formally  controlled.
DRRS requirements are currently in a state of flux. Specifically, the fact  that 530 new user requirements have recently been identified means that  the suite of requirements documentation associated with the system will  need to be changed and thus are not stable. To illustrate, program officials  told us that, as of late February 2009, these 530 new requirements had not  been fully evaluated by the DIO and DRRS governance boards and thus  not yet approved. As a result, their impact on the program is not clear.
Compounding this instability in the DRRS requirements is the fact that  additional changes are envisioned. According to program officials, the  changes resulting from the gap analysis and reflected in the latest version  of the DRRS concept of operations, which was approved by the DRRS  Executive Committee in January 2009, have yet to be reflected in other  requirements documents, such as the detailed system requirements.
Although defining and developing requirements is inherently an iterative  process, having a baseline set of requirements that are stable is a  prerequisite to effective and efficient development of an operationally  capable and suitable system. Without them, the DIO will not be able to  deliver a system that meets user needs on time, and it is unlikely that  future development and deployment efforts will produce better results.
section_title: Alignment among Requirements and Related System Design and Testing Artifacts Has Not Been Demonstrated
paragraphs: One of the leading practices associated with developing and managing  requirements is maintaining bidirectional traceability from high-level  operational requirements (e.g., concept of operations and functional  requirements) through detailed lower-level requirements and design  documents (e.g., software requirements specification) to test cases. Such  traceability is often accomplished through the use of a requirements  traceability matrix, which serves as a crosswalk between different levels  of related requirements, design, and testing documentation. The DRRS  program management plan recognizes the importance of traceability,  stating that requirements are to be documented and linked to acceptance  tests, scripts, and criteria.
Despite the importance of traceability, DIO officials could not demonstrate  that requirements and related system design and testing artifacts are  properly aligned. Specifically, we attempted on three separate occasions  to verify the traceability of system requirements backward to higher-level  requirements and forward to lower-level software specifications and test  cases, and each time we found that traceability did not exist. Each attempt  is discussed here:  In November 2008, our analysis of the requirements traceability matrix  and the software requirements specification showed significant  inconsistencies. For example, the traceability matrix did not include 29  requirements that were included in the software requirements  specification. As a result, we did not have an authoritative set of  requirements to use to generate a random sample of requirements to  trace. Program officials attributed the inconsistencies to delays in  updating all the documents to reflect the aforementioned capability gap  analysis. They also stated that these documents would be updated by  December 2008.
In December 2008, we used an updated requirements traceability  matrix to generate a randomized sample of 60 software requirements  specifications and observed a DIO demonstration of the traceability of  this sample. However, DIO officials were unable to demonstrate for us  that these specifications could be traced backward to higher-level  requirements and forward to test cases. Specifically, attempts to trace  the first 21 requirements forward to test cases failed, and DIO officials  stated that they could not trace the 60 requirements backward because  the associated requirements documents were still being updated.  According to the officials, 11 of the 21 could not be traced forward  because these were implemented prior to 2006 and the related test  information was not maintained by the program office but rather was  at the development contractor’s site. They added that the remaining 10  either lacked test case information or test results.
In February 2009, we used an updated DIO-provided requirements  traceability matrix, a capabilities requirement document, and software  requirements specification to generate another randomized sample of  60 detailed specifications. We then observed the development  contractor’s demonstration of traceability using the contractor’s  requirements management tool. Because of time constraints, this  demonstration focused on 46 of the 60 requirements, and it showed  that adequate traceability still did not exist. Specifically, 38 of the 46  could not be traced backward to higher-level requirements or forward  to test cases. This means that about 83 percent of the DRRS  specifications (95 percent degree of confidence of being between 72  and 91 percent) were not traceable. Of the 38, 14 did not trace because  of incomplete traceability documentation; 5 due to inconsistent  traceability documentation; 3 due to requirements not being resident in  the tracking tool; and 16 due to no actual development work being  started.
In addition, none of the 46 requirements were traceable to the January  2009 concept of operations. According to contractor officials, this is  because the newly developed capability requirements document is  considered to be a superset of the concept of operations, and thus  traceability to this new document is their focus. However, they were  unable to demonstrate traceability to the requirements in either the  capability requirements document or the concept of operations. Further,  we also found numerous inconsistencies among the capabilities  requirements document, software requirements specification, and the  requirements traceability matrix. For example, 15 capabilities  requirements listed on the traceability matrix were not listed in the  capabilities requirements document, but were listed in the updated  software requirements specification, dated February 2009. Further, one  requirement listed in the traceability matrix was not listed in either of  these documents. One possible reason for these inconsistencies is that the  traceability matrix was prepared manually, rather than being automatically  generated from the tool, which would increase the probability of these and  other discrepancies caused by human error. Another reason cited by  program officials is that test results that occurred prior to October 2006  had yet to be fully recorded in the contractor’s tracking tool.
DIO and contractor officials attributed the absence of adequate  requirements traceability to the ongoing instability in requirements and  magnitude of the effort to update the chain of preexisting and new  requirements documentation. They added that they expect traceability to  improve as requirements become more stable and the documentation is  updated. Regardless, the DIO has and continues to invest in the  development of DRRS in the absence of requirements traceability. Without  traceable requirements, the DIO does not have a sufficient basis for  knowing that the scope of the design, development, and testing efforts will  produce a system solution on time and on budget and that will meet users’  operational needs and perform as intended. As a result, the risk is  significant that expensive and time-consuming system rework will be  required.
section_title: Changes to Requirements Are Not Being Effectively Controlled
paragraphs: Adopting a disciplined process for reviewing and accepting changes to an  approved and authoritative baseline set of requirements in light of the  estimated costs, benefits, and risk of each proposed change is a  recognized best practice. Elements of a disciplined process include: (1)  formally documenting a requirements change process; (2) adopting  objective criteria for considering proposed changes, such as estimated  cost or schedule impact; and (3) rigorously adhering to the documented  change control process.
Since the inception of the program in 2002, DRRS has been developed and  managed without a formally documented and approved process for  managing changes to system requirements. Further, while requirements  management and change control plans were developed in 2006 by the  DRRS software development contractor, according to DIO officials, the  plans were not adequate. For example, the plans did not detail how DRRS  user requirements were collected or how objective factors, such as cost,  impacted development decisions.
To address these problems, the Joint Staff developed what it referred to as  a conceptual requirements change control process in February 2008,  which was to serve as a basis for the DIO to develop more detailed plans  that could be implemented. Eleven months later, in January 2009, the DIO  drafted more detailed plans—a DRRS requirements management plan and  a DRRS requirements configuration management plan, the latter of which  the DIO updated in March 2009. Specifically, the draft plans call for new  DRRS requirements to be collected using an online tool and reviewed by  the DIO to determine whether the requirement constitutes a major change  to DRRS. Once approved, the DIO and the contractor are to provide the  Battle Staff with a formatted report specifying the anticipated benefit of  each new requirement and an initial analysis of the cost and performance  impact. The Battle Staff then is to prioritize the requirement based on the  DIO’s impact analysis. If the issue cannot be resolved by the Battle Staff, it  is to be elevated to the senior oversight bodies (i.e., the General Officer’s  Steering Committee and the DRRS Executive Committee). After a  requirement has been approved, the software developer may prepare a  more detailed “customer acceptance document” that analyzes the potential  cost, schedule, and quality impact to DRRS objectives, which is then to be  reviewed by the DIO at subsequent Change Control Board meetings.
However, according to the user community and the DIO Director, the  revised plans have not been submitted for review and approval to the  DRRS community. Specifically, they stated that only a proposed process  flow diagram was briefed at the Battle Staff, and according to them, the  change control process was still being evaluated. Moreover, the DIO has  yet to implement key aspects of its draft plans. For example, the DRRS  Chief Engineer stated that until recently, the DIO had continued to accept  changes to DRRS requirements that were submitted outside of the  designated online tool. In addition, the reports that the Battle Staff are to  use in making their requirement change determination do not include the  anticipated benefit and estimated cost or schedule impact of new  requirements. Rather, these reports only include the estimated number of  hours necessary to complete work on a proposed requirement. Moreover,  contractor officials and users from the Special Operations Command told  us that cost or schedule impacts have rarely been discussed at the Battle  Staff or Change Control Board meetings. Our analysis of minutes from  change control meetings confirmed this. Furthermore, the DRRS Chief  Engineer stated that the customer acceptance documents have only  recently been used.
Until the DIO effectively controls requirements changes, it increases the  risk of needed DRRS capabilities taking longer and costing more to deliver  than necessary.
section_title: DRRS Testing Has Not Been Adequately Performed and Key Test Management Structures and Controls Have Not Been Established
paragraphs: Effective system testing is essential to successfully developing and  deploying systems like DRRS. According to DOD and other relevant  guidance, system testing should be progressive, meaning that it should  consist of a series of test events that first focus on the performance of  individual system components, then on the performance of integrated  system components, followed by system-level tests that focus on whether  the system (or major system increments) are acceptable, interoperable  with related systems, and operationally suitable to users.  For this series  of related test events to be conducted effectively, (1) each test event needs  to be executed in accordance with well-defined test plans, (2) the results  of each test event need to be captured and used to ensure that problems  discovered are disclosed and corrected, and (3) all test events need to be  governed by a well-defined test management structure.
Despite acquiring and partially deploying a subset of DRRS increments,  the DIO cannot demonstrate that it has adequately tested any of these  system increments, referred to as system releases and subreleases.  Specifically, (1) the test events for already acquired, as well as currently  deployed and operating, DRRS releases and subreleases were not based on  well-defined plans, and test events have not been fully executed in  accordance with plans or executed in a verifiable manner, or both; (2) the  results of executed test events have not been captured and used to ensure  that problems discovered were disclosed to decision makers and  ultimately corrected; and (3) the DIO has not established an effective test  management structure to include, for example, a clear assignment of test  management roles and responsibilities, or a reliable schedule of planned  test events. Compounding this absence of test management structures and  controls is the fact that the DIO has yet to define how the series of system  releases and subreleases relate to its recent restructuring of DRRS  increments into a series of 10 modules. Collectively, this means that it is  unlikely that already developed and deployed DRRS capabilities can  perform as intended and meet user operational needs. Equally doubtful are  the chances that the DIO can adequately ensure that yet-to-be developed  DRRS capabilities will meet expectations.
subsections: section_title: DIO Has Not Adequately Planned and Executed Key Test Events
paragraphs: Key tests required for already developed and partially fielded DRRS  increments either did not have well-defined test plans, or these tests have  yet to be conducted. According to program documentation, system  releases and subreleases have been subjected to what are described as 30- day test cycles, during which: (1) a Software Test Plan is updated if  applicable, (2) test procedures are developed and incorporated in the  Software Test Description, (3) a series of developmental tests on each  release/subrelease is performed, (4) weekly meetings are held to review  software defects identified during testing, (5) final test results are  summarized within the Software Test Report and Software Version  Description, and (6) the release/subrelease is made available to users.
However, the program office has yet to provide us with the developmental  test plans and procedures for each release/subrelease that to-date has  either been developed or fielded. Instead, it provided us with a Software  Test Plan and two Software Test Descriptions that it said applied to two  subreleases within release 4.0. However, available information indicates  that DRRS subreleases total at least 63, which means that we have yet to  receive the test plans and procedures for 61. Further, the test plan that we  were provided is generic in nature, meaning that it was not customized to  apply specifically to the two subreleases within Release 4.0. Moreover, the  plan and procedures lack important elements specified in industry  guidance. For example, the test plan does not include a schedule of  activities to be performed or defined roles and responsibilities for  performing them. Also, the test plan does not consistently include test  entrance and exit criteria, a test defect management process, and metrics  for measuring progress.
Moreover, the DIO has yet to demonstrate that it has performed other key  developmental and operational test events that are required before the  software is fielded for operational use. According to DIO officials,  developmental testing concludes only after system integration testing and  system acceptance testing, respectively, are performed. Further, following  developmental testing, the Joint Interoperability Test Command (JITC),  which is a DOD independent test organization, is to conduct both  interoperability and operational testing before the system is deployed and  put into production (i.e., used operationally). Although increments of  DRRS functionality have been put into production, the DIO has not  performed system integration testing, system acceptance testing, or  operational testing on any DRRS release or subrelease. Further, JITC  documentation shows that while an interoperability test of an increment  of DRRS functionality known as ESORTS was conducted, this test did not  result in an interoperability certification. According to JITC and Joint  Staff officials, this was because the DIO did not address JITC’s identified  limitations to the program’s Information Support Plan, which identifies  essential information-exchange sharing strategies between interdependent  systems that are needed for interoperability certification. Without  interoperability certification, the ability of the DRRS to exchange accurate  and timely readiness data with other critical systems, such as the Joint  Operation Planning and Execution System, cannot be ensured.
Similarly, while DIO officials stated that acceptance testing has occurred  for one increment of DRRS functionality known as SORTSREP, the DIO  does not have either a finalized acceptance test plan or documented test  results.  Furthermore, the integrated master schedule (last updated in  April 2009) shows that acceptance testing is not to occur until the  July/August 2009 time frame, which is about 15 months later than  originally envisioned. Moreover, this delay in acceptance testing has in  turn delayed interoperability and operational testing by 16 months  (May/June 2008 to September/November 2009), according to the latest  schedule. Program officials attributed the delays to Marine Corps and Air  Force concerns about the quality of SORTSREP.
Until the DIO has effectively planned and executed the series of tests  needed to demonstrate the readiness of DRRS increments to operate in a  production environment, the risk of fielded system increments not  performing as intended and requiring expensive rework to correct will be  increased, and DOD will continue to experience delays in delivering  mission-critical system capabilities to its readiness community.
section_title: DIO Has Not Adequately Documented and Reported Test Results
paragraphs: Available results of tests performed on already developed and at least  partially deployed DRRS releases/subreleases show that the test results  have not been effectively captured and analyzed, and have not been fully  reported. Moreover, test results for other releases/subreleases do not  exist, thus minimizing the value of any testing that has been performed.  According to relevant guidance, effective system testing includes  recording the results of executing each test procedure and test case as  well as capturing, analyzing, correcting, and disclosing to decision makers  problems found during testing (test defects).  It also includes ensuring  that test entry and exit criteria are met before beginning and ending,  respectively, a given test event.
The DIO does not have test results of all developed and tested DRRS  releases and subreleases. Specifically, program officials provided us with  the Software Test Reports and Software Version Descriptions that, based  on program documentation, represent the full set of test results for three  subreleases and a partial set of test results for 40 subreleases within  releases 1.0, 3.0, and 4.0. However, as noted earlier, DRRS subreleases  total at least 63, which means that test reports and results for at least 20  subreleases do not exist. Moreover, the test reports and version  descriptions that we received do not consistently include key elements  provided for in industry guidance, such as a documented assessment of  system capabilities and limitations, entrance/exit criteria status, an  assessment as to whether the applicable requirements/thresholds were  met, and unresolved defects and applicable resolution plans.  This  information is important because it assists in determining and disclosing  to decision makers current system performance and efforts needed to  resolve known problems, and provides program officials with a needed  basis for ensuring that a system increment is ready to move forward and  be used. Without this information, the quality and readiness of a system is  not clear.
Furthermore, the DIO does not have detailed test defect documentation  associated with all executed DRRS test events. According to relevant  guidance, defect documentation should, among other things, identify each  issue discovered, assign each a priority/criticality level, and provide for  each a strategy for resolution or mitigation.  In lieu of detailed test defect  documentation, program officials referred us to the above-mentioned  Software Version Descriptions, and stated that additional information is  available in an automated tool, known as the ISI BugTracker, that it uses  to capture, among other things, defect data. However, these documents do  not include the above-cited defect information, and defect data for each  test event do not exist in the ISI BugTracker.
Compounding the absence and limitations of test results are weaknesses  in the program office’s process for collecting such results during test  execution. According to relevant guidance, test results are to be collected  and stored according to defined procedures and placed under appropriate  levels of control.  Furthermore, these test results are to be reviewed  against the source data to ensure that they are complete, accurate, and  current. For DRRS, the program office is following a partially  undocumented, manual process for collecting and storing test results and  defects that involves a database and layers of documentation. As  explained by program officials and documentation, the DIO initially  documents defects and completed test case results manually on paper  forms, and once the defect is approved by the test lead, it is input into a  database. However, it does not have written procedures governing the  entire process, and thus key controls, such as assigned levels of authority  for database read/write access, are not clearly defined. Moreover, once the  test results and defects are input into the database, traceability back to the  original test data for data integrity checks cannot be established because  the program office does not retain these original data sets. Program  officials acknowledged these internal control weaknesses and stated that  they intend to adopt a new test management tool that will allow them to  capture in a single database test cases, test results, and test defects.
Furthermore, the DIO’s process for analyzing and resolving test defects  has limitations. According to relevant guidance and the draft SORTSREP  Test and Evaluation Master Plan (TEMP), defects should be analyzed and  prioritized.  However, the program office has not established a standard  definition for defect priority levels identified during testing. For example,  the various release/subrelease test reports (dated through January 2009)  prioritize defects on a scale of 1-3, where a priority 2 means critical but  with a viable workaround. In contrast, the SORTSREP TEMP (dated  January 2009) prioritizes defects on a scale of 1-5, where a priority 2  means an error that adversely affects the accomplishment of an  operational or mission essential function in accordance with official  requirements so as to degrade performance and for which no alternative  work around solution exists. By not using standard priority definitions for  categorizing defects, the program office cannot ensure that it has an  accurate and useful understanding of the scope and magnitude of the  problems it is facing at any given time, and it will not know if it is  addressing the highest priority issues first.
In addition, the DIO has not ensured that critical defects are corrected  prior to concluding a given test event. According to relevant guidance and  the draft SORTSREP TEMP, all critical and high defects should be  resolved prior to the conclusion of a test event, and all test results should  be reviewed for validity and completeness.  However, the DRRS  release/subrelease test reports show that the DIO concluded five test  events even though each had at least 11 open critical defects (priority 1  defects with no workaround). Moreover, these numbers of open critical  defects are potentially higher because they do not include defects for  which a solution was identified but the solution failed during regression  testing and do not include defects that were dismissed because the  program official was unable to recreate it.
Until the DIO adequately documents and reports the test results, and  ensures that severe problems discovered are corrected prior to concluding  a given test event, the probability of incomplete test coverage, and  insufficient and invalid test results, is increased, thus unnecessarily  increasing the risk of DRRS not meeting mission needs or otherwise not  performing as intended.
section_title: DIO Has Not Established an Effective Test Management Structure
paragraphs: The DIO does not have an effective test management structure, to include  a well-defined overall test management plan that clearly assigns test  management roles and responsibilities, a designated test management lead  and a supporting working group, and a reliable schedule of planned test  events. According to relevant guidance, these aspects of test management  are essential to adequately planning, executing, and reporting a program’s  series of test events.
Although the program has been underway for 8 years, it did not have an  overarching DRRS TEMP until very recently (February 2009), and this plan  is still in draft and has yet to be approved. Further, this draft TEMP does  not clearly define DRRS test management roles and responsibilities, such  as those of the test manager, and it does not include a reliable schedule of  test events that reflect the program’s recent restructuring of its software  releases/subreleases into 10 modules. According to DIO officials, they  recently decided not to approve this overarching TEMP. Instead, they said  that they now intend to have individual TEMPs for each of the recently  defined 10 modules, and to have supporting test plans for each module’s  respective developmental and operational test events. According to  program documentation, three individual TEMPs are under development  (i.e., SORTSREP tool and the Mission Readiness and Readiness Review  modules). However, drafts of these TEMPs also do not clearly define test  entrance and exit criteria, test funding requirements, an integrated test  program schedule, and the respective test management roles and  responsibilities. For example, while the draft SORTSREP TEMP identifies  the roles and responsibilities of some players, such as the test manager,  the personnel or organization that is to be responsible is not always  identified. In addition, while the various players in the user community are  identified (i.e., military services, combatant commands), their associated  roles or responsibilities are not.
Furthermore, the DIO has yet to designate a test management lead and  establish an effective test management working group. According to  relevant guidance, test management responsibility and authority should be  assigned to an individual, and this individual should be supported by a  working integrated product team that includes program office and  operational testing representatives. Among other things, the working  integrated product team is to develop an overall system test strategy.  However, DIO officials told us that the test manager position has been  vacant, and this position is now being temporarily filled by the program’s  chief engineer, who is a contractor. Furthermore, although DRRS system  development began prior to 2004, a charter for a test and evaluation  working integrated product team was not issued until February 2009.  According to DIO officials, the delay in establishing the team has not had  any impact because of corresponding delays in finalizing the program’s  overall test strategy. However, this statement is not consistent with the  Defense Acquisition Guidebook, which states that two of the key products  of the working integrated product team are the program’s test strategy and  TEMP. Further, JITC officials stated that the lack of a test manager and an  active test and evaluation working integrated product team have reduced  the effectiveness of DRRS testing activities. As a result, they stated that  they have had to compensate by conducting individual meetings with the  user community to discuss and receive documentation to support their  operational and interoperability test planning efforts.
Moreover, the DIO has yet to establish a reliable schedule of planned test  events. For example, the schedule in the TEMPs is not consistent with  either the integrated master schedule or the developmental test plans.  Specifically, the draft SORTSREP TEMP (last updated in January 2009)  identifies SORTSREP developmental testing occurring through January  2009 and ending in early February 2009, while the integrated master  schedule (last updated in April 2009) shows SORTSREP development  testing as occurring in the July/August 2009 time frame. In addition, while  program officials said that development testing for SORTSREP has  occurred, the associated development test plans (e.g., system integration  and system acceptance test plans) had no established dates for test  execution, and are still in draft. As another example, a module referred to  as “Mission Readiness” had no established dates for test execution in its  TEMP, and while program documentation indicates that this module  completed development testing in December 2008, the associated  development test plans (e.g., system integration and system acceptance  test plans) do not exist.
In addition, the DIO has yet to define in its draft TEMPs how the  development and testing to date of at least 63 subreleases relate to the  development and testing of the recently established 10 system modules.  According to Joint Staff and JITC officials, they do not know how the  releases/subreleases relate to the modules, and attributed this to a lack of  an approved description for each module that includes what functionality  each is intended to provide. Furthermore, the high-level schedule in the  TEMP does not describe what test events for the DRRS  releases/subreleases that have already been developed and deployed relate  to the development test efforts planned for the respective modules. These  problems in linking release/subrelease test events to module test events  limit the DIO and JITC in leveraging the testing already completed, which  in turn will impact the program’s ability to meet cost, schedule, and  performance expectations.
Collectively, the weaknesses in this program’s test management structure  increase the chances that the deployed system will not meet certification  and operational requirements, and will not perform as intended.
section_title: DRRS Has Not Been Guided by a Reliable Schedule
paragraphs: The success of any program depends in part on having a reliable schedule  that defines, among other things, when work activities will occur, how  long they will take, and how they are related to one another. As such, the  schedule not only provides a road map for the systematic execution of a  program, but also provides the means by which to gauge progress, identify  and address potential problems, and promote accountability. From its  inception in 2002 until November 2008, the DIO did not have an integrated  master schedule. Moreover, the only milestone that we could identify for  the program prior to November 2008 was the date that DRRS was to  achieve full operational capability, which was originally estimated to  occur in fiscal year 2007, but later slipped to fiscal year 2008 and then  fiscal year 2011, and is now fiscal year 2014—a 7-year delay.
In addition, the DRRS integrated master schedule that was developed in  November 2008, and was updated in January 2009 and again in April 2009  to address limitations that we identified and shared with the program  office, is still not reliable. Specifically, our research has identified nine  practices associated with developing and maintaining a reliable schedule.  These practices are (1) capturing all key activities, (2) sequencing all key  activities, (3) assigning resources to all key activities, (4) integrating all  key activities horizontally and vertically, (5) establishing the duration of all  key activities, (6) establishing the critical path for all key activities, (7)  identifying float between key activities, (8) conducting a schedule risk  analysis, and (9) updating the schedule using logic and durations to  determine the dates for all key activities. However, the program’s latest  integrated master schedule does not address three of the practices and  only partially addresses the remaining six. For example, the schedule does  not establish a critical path for all key activities, include a schedule risk  analysis, and it is not being updated using logic and durations to determine  the dates for all key activities. Further, it does not fully capture, sequence,  and establish the duration of all key work activities; fully assign resources  to all key work activities; fully integrate all of these activities horizontally  and vertically; and fully identify the amount of float—the time that a  predecessor activity can slip before the delay affects successor activities— between these activities. These practices are fundamental to producing a  sufficiently reliable schedule baseline that can be used to measure  progress and forecast slippages. (See table 3 for the results of our analyses  relative to each of the nine practices.)
The limitations in the program’s latest integrated master schedule, coupled  with the program’s 7-year slippage to date, make it likely that DRRS will  incur further delays. Compounding these limitations is the considerable  concurrency in the key activities and events in the schedule associated  with the 10 recently identified system modules (see fig. 2). For example, in  2010 alone, the program office plans to complete development testing on 2  modules and operational testing on 3 modules, while also reaching initial  operational capability on 3 modules and full operational capability on 2  modules. By way of comparison, the program office had almost no  concurrency across a considerably more modest set of activities and  events over the last 5 years, but nevertheless has fallen 7 years behind  schedule. As previously reported, such significant overlap and  concurrency among major program activities can create contention for  limited resources and thus introduce considerable cost, schedule, and  performance risks.
In addition, the schedule remains unstable as evidenced by the degree of  change it has experienced in just the past few months. For example, the  January 2009 schedule had a full operational capability milestone of  October 2011. By contrast, the April 2009 schedule has a December 2013  milestone (see fig. 3 below). Moreover, some milestones are now to occur  much earlier than they were a few months ago. For example, the January  2009 schedule shows initial operational capability for “readiness reviews”  to be June 2010. However, the April 2009 schedule shows that that this  milestone was attained in August 2007. Overall, multiple milestones for  four modules were extended by at least 1 year, including two milestones  that were extended by more than 2 years. Such change in the schedule in  but a few months suggests a large degree of uncertainty, and illustrates the  ut a few months suggests a large degree of uncertainty, and illustrates the  importance of ensuring that the schedule is developed in accordance with  importance of e nsuring that the schedule is developed in accordance with  best practices.  best practices.
section_title: DIO Lacks Adequate Staffing and an Effective Approach to Meeting its Human Capital Needs
paragraphs: As we have previously reported, effective human capital management is an  essential ingredient to achieving successful program outcomes. Among  other things, effective human capital management involves a number of  actions to proactively understand and address any shortfalls in meeting a  program’s current and future workforce needs. These include an  assessment of the core competencies and essential knowledge, skills, and  abilities needed to perform key program management functions, an  inventory of the program’s existing workforce capabilities, and an analysis  of the gap between the assessed needs and the existing capabilities.  Moreover, they include explicitly defined strategies and actions for filling  identified gaps, such as strategies for hiring new staff, training existing  staff, and contracting for support services.
The DIO is responsible for performing a number of fundamental DRRS  program management functions. For example, it is responsible for  acquisition planning, performance management, requirements  development and management, test management, contractor tracking and  oversight, quality management, and configuration management. To  effectively perform such functions, program offices, such as the DIO, need  to have not only well-defined policies and procedures and support tools  for each of these functions, but also sufficient human capital to implement  the processes and use the tools throughout the program’s life cycle.  Without sufficient human capital, it is unlikely that a program office can  effectively perform its basic program management functions, which in turn  increases the risk that the program will not deliver promised system  capabilities and benefits on time and on budget.
The DIO does not currently have adequate staff to fulfill its system  acquisition and deployment responsibilities. In particular, the DIO is  staffed with a single full-time government employee—the DIO Director. All  other key program office functions are staffed by either contractor staff or  staff temporarily detailed, on an as-needed basis, from other DOD  organizations (referred to as “matrixed” staff). As a result, program  management positions that the DIO itself has identified as critical to the  program’s success, such as configuration manager and security manager,  are being staffed by contractors. Moreover, these contractor staff report to  program management positions also staffed by contractors. Other key  positions, such as those for performing acquisition management,  requirements development and management, and performance  management, have not even been established within the DIO.  Furthermore, key positions, such as test manager, are vacant. These  human capital limitations were acknowledged by the DRRS Executive  Committee in November 2008.
According to DIO and contractor officials, they recognize that additional  program management staffing is needed. They also stated that while DRRS  has been endorsed by USD (P&R) leadership and received funding  support, past requests for additional staff have not been approved by USD  (P&R) due to other competing demands for staffing. Further, DIO officials  stated that the requests for staff were not based on a strategic gap analysis  of its workforce needs and existing capabilities. Specifically, the program  has not assessed its human capital needs and the gap between these needs  and its onboard workforce capabilities. Until the program office adopts a  strategic, proactive approach to managing its human capital needs, it is  unlikely that it will have an adequate basis for obtaining the people it  needs to effectively and efficiently manage DRRS.
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix IV: GAO Contacts and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the contacts named above, key contributors to this report  were Michael Ferren (Assistant Director), Neelaxi Lakhmani (Assistant  Director), April Baugh, Mathew Butler, Richard J. Hagerman, Nicole  Harms, James Houtz, John Lee, Stephen Pruitt, Terry Richardson, Karen  Richey, Karl Seifert, and Kristy Williams.