DOD Systems Modernization: Planned Investment in the Naval Tactical Command Support System Needs to be Reassessed
Published: 2005-12-05T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Because it is important that the Department of Defense (DOD) adheres to disciplined information technology (IT) acquisition processes to successfully modernize its business systems, GAO was asked to determine whether the Naval Tactical Command Support System (NTCSS) is being managed according to important aspects of DOD's acquisition policies and guidance, as well as other relevant acquisition management best practices. NTCSS was started in 1995 to help Navy personnel effectively manage ship, submarine, and aircraft support activities. To date, about $1 billion has been spent to partially deploy NTCSS to about one-half its intended ashore and afloat sites.
section_title: What GAO Found
paragraphs: The Department of the Navy has not managed its NTCSS program in accordance with key aspects of the department's policies and related guidance, including federal and recognized best practice guidance. Collectively, these policies and guidance are intended to reasonably ensure that investment in a given IT system represents the right solution to fill a mission need and, if it is, that acquisition and deployment of the system are handled in a manner that maximizes the chances of delivering defined system capabilities on time and within budget. In the case of NTCSS, neither of these outcomes is being realized. The Navy has not economically justified its ongoing and planned investment in NTCSS. Specifically, it (1) has not reliably estimated future costs and benefits and (2) has not ensured that independent reviews of its economic justification were performed to determine its reliability. The Navy has not invested in NTCSS within the context of a well-defined DOD or Navy enterprise architecture, which is necessary to guide and constrain NTCSS in a way that promotes interoperability and reduces redundancy with related and dependent systems. The Navy has not effectively performed key measurement, reporting, budgeting, and oversight activities. In particular, earned value management, which is a means for determining and disclosing actual performance against budget and schedule estimates, has not been implemented effectively, and oversight entities have not had the visibility into the program needed to affect its direction. The Navy has not adequately conducted requirements management and testing activities. For example, requirements were neither prioritized nor traced to related documentation to ensure that the system delivers capabilities that meet user needs. This contributed to failures in developmental testing that have prevented the latest component of NTCSS from passing operational testing twice over the last 4 years. Reasons the Navy cited for not following policies and guidance ranged from their not being applicable to the NTCSS program, to lack of time available to apply them, to plans for strengthening system practices not being applied retroactively. Nevertheless, the Navy has begun taking steps and is considering other steps intended to address some of the above problems. Until program management improves, NTCSS will remain a risky program.

section_title: Letter
paragraphs: Because it is so important that the Department of Defense (DOD) adhere to  disciplined information technology (IT) acquisition processes in order to  successfully modernize its business systems, you requested that we  determine whether the department is following its own revised policies and  guidance for acquiring systems, which it issued in May 2003. As part of our  response to your request, we agreed to review the Naval Tactical Command  Support System (NTCSS) program. NTCSS was started in 1995 and is  intended to help Navy personnel effectively manage ships, submarines, and  aircraft support activities. The Navy expects to spend $348 million on  NTCSS between fiscal years 2006 and 2009, for a total of approximately  $1.45 billion since program inception.
As agreed, our objective was to determine whether NTCSS is being  managed according to important aspects of DOD’s acquisition policies and  guidance, as well as other relevant acquisition management best practices.  We focused on the program’s (1) economic justification; (2) architectural  alignment; (3) project management, including progress measurement,  progress reporting, funding disclosure, and oversight activities; and  (4) system development, including requirements management and testing.  For requirements management and testing, we focused on the NTCSS  application that is currently being developed, known as the Optimized  Organizational Maintenance Activity (OOMA).
We conducted our review from September 2004 through November 2005 in  accordance with generally accepted government auditing standards. For  details on our objective, scope, and methodology, see appendix I.
subsections: section_title: Background
paragraphs: The Navy’s primary mission is to organize, train, maintain, and equip  combat-ready naval forces capable of winning the global war on terror and  any other armed conflict, deterring aggression by would-be foes, preserving  freedom of the seas, and promoting peace and security. To support this  mission, the Navy performs a variety of interrelated and interdependent  business functions such as logistics and financial management. The Navy  requested, for fiscal year 2005, about $3.5 billion to operate, maintain, and  modernize its business systems and related IT infrastructure that support  these business functions. This request represents about 27 percent of the  $13 billion that DOD requested for all of its business systems for fiscal year  2005. Of the 4,150 business systems that DOD reports in its current  inventory, the Navy accounts for 2,353, or about 57 percent, of the total.
In 1995, we designated DOD’s business systems modernization efforts as a  high-risk program and continue to designate it as such today for several  reasons, including the department’s challenges in implementing effective IT  investment management structures and processes, developing and  implementing an enterprise architecture, and implementing effective IT  system acquisition and development processes.
subsections: section_title: NTCSS Genesis and Status Overview
paragraphs: In the early 1990s, the Navy employed a variety of IT systems to support the  management of information, personnel, materials, and funds required to  maintain and operate ships, submarines, and aircraft. Three core systems— each managed by a separate program office—consisting of nine major  applications, provided this support: (1) the Shipboard Non-Tactical  Automated Data Processing Program (SNAP), managed by the Space and  Naval Warfare Systems Command; (2) the Naval Aviation Logistics  Command Management Information System (NALCOMIS), managed by the  Naval Air Systems Command; and (3) the Maintenance Resource  Management System (MRMS), managed by the Naval Sea Systems  Command. See table 1 for a description of these three legacy systems and a  list of their respective applications.
In 1992, we recommended that the Navy merge the management of all  shipboard nontactical programs under a single command that would have  authority and control over funding and development. In 1993, the Navy  developed a strategy to do so. In 1994, the Navy also identified a number of  problems with the three legacy systems. Specifically, the Navy determined  that (1) the individual systems did not consistently handle increasing  workloads and provide the flexibility to meet changing operational  demands; (2) the systems’ software architectures were ineffective and  inefficient; (3) the hardware was outdated, slow, expensive to maintain,  and nonstandard; and (4) the systems could not support modernized  applications.
To address these concerns, the Navy initiated the NTCSS program in 1995  to enhance the combat readiness of ships, submarines, and aircraft. To  accomplish this, NTCSS was to provide unit commanding officers and  crews with information about, for example, maintenance activities, parts  inventories, finances, technical manuals and drawings, and personnel.  According to the Navy, it spent approximately $1.1 billion for NTCSS from  its inception through fiscal year 2005 and expects to spend another $348  million between fiscal years 2006 and 2009, for a total of approximately  $1.45 billion.
The Navy defined a three-stage acquisition process for NTCSS.
Stage 1: Purpose was to replace hardware in order to establish a common  infrastructure across all force-level ships, unit-level ships, aviation  squadrons, Naval air stations, marine aviation logistics squadrons, and  other maintenance activities—both at sea and ashore. During this stage,  software and business processes were not to be changed. This phase was  begun in 1994 under the legacy SNAP and NALCOMIS programs and,  according to program officials, it is fundamentally complete—although  technology refresh or replacement activities are still occurring.
Stage 2: Purpose was to provide the functionality of the legacy systems  software with more efficient, more easily maintained software and to  eliminate functional overlap among the systems. This stage was to involve  software technology modernization but no changes in software  functionality or business processes. Existing legacy systems used flat files  and hierarchical databases, which were to be converted to relational  databases, and the existing application code was to be rewritten using  modern software languages. A common hardware and systems software  environment was also to be implemented, and functionality found in eight  of the nine legacy applications was to be consolidated and rewritten as four  new NTCSS applications. Development of these four applications began in  1995 and was reportedly completed in 2000. This stage was known as  NTCSS Optimization. See table 2 for a description of the functionality of  these new applications.
Stage 3: Purpose was to improve NTCSS’s functionality by implementing  business process improvements. According to Navy officials, this stage is  known as NTCSS Modernization and, to date, includes two efforts:  (1) replace the last legacy application and (2) create a Web-enabled version  of the three unit-level Optimized NTCSS applications that were developed  under Stage 2. See table 3 for a description of the functionality of these  business process improvements.
As of April 2005, legacy applications were still in use at 51 percent of the  Navy’s 659 sites. These 659 sites either have legacy, Optimized, or  modernized applications. Table 4 shows the distribution of the legacy,  Optimized, and modernized applications.
According to Navy officials, about $1.1 billion was spent on NTCSS  between 1995 and 2005. This includes about $1 billion on NTCSS Optimized  applications and $91 million on OOMA and eNTCSS. Table 5 shows  NTCSS’s budget totals from the time the program began in fiscal year 1995  through fiscal year 2005.
section_title: NTCSS Oversight and Management Roles and Responsibilities
paragraphs: A number of Navy and DOD organizations are involved in overseeing and  managing the NTCSS program. Table 6 lists the organizations involved in  NTCSS oversight and their respective roles and responsibilities.
There have been three milestone decision authorities for NTCSS since the  program was begun. Initially, the milestone decision authority was in the  Office of the Assistant Secretary of Defense for Networks and Information  Integration/Chief Information Officer. In July 1999, this authority was  delegated to the Assistant Secretary of the Navy for Research,  Development, and Acquisition, who then delegated oversight authority to  Deputy Assistant Secretary of the Navy for Command, Control,  Communication, Computers and Intelligence, and Space in March 2000.
Table 7 lists the organizations involved in NTCSS management and their  respective roles and responsibilities.
section_title: NTCSS Participation in DOD’s Rapid Improvement Team Pilot
paragraphs: In 2001, the DOD Chief Information Officer and the Undersecretary of  Defense for Acquisition, Technology, and Logistics chartered a pilot project  aimed at saving time by significantly reducing the reporting and oversight  requirements. The ultimate goal was to enable the acquisition process to  deliver mission-effective IT systems within 18 months. Known as the Rapid  Improvement Team (RIT) for IT Acquisition Management Transformation,  the pilot was to cover a 2-year period from January 1, 2002, through  December 31, 2003. Nine programs from the military services participated  in the pilot. NTCSS was selected to participate in the pilot by its milestone  decision authority due to its longevity and because of its perceived low  risk, stability, and compliance with IT management best practices. It was  also believed that little system development remained to be done. NTCSS  began participating in the RIT pilot in October 2002.
The RIT pilot relieved the program office of the normal acquisition process  activities, such as preplanned, formal milestone decision reviews or  briefings, and it granted the program office the authority to pass key  milestones once it determined that established requirements had been met.  This streamlined approach was considered possible because all  information related to these requirements was to be continually updated  and available to oversight organizations and stakeholders via a RIT Web  site. More specifically, the program office was to update the Web site  monthly via a set of electronic forms with the kind of data that were  traditionally found in DOD oversight documents. The program office was  also to use the Web site to input key acquisition documents (e.g.,  acquisition plans, economic analyses, requirements documents and test  plans) in an electronic library. In turn, the milestone decision authority and  other oversight organizations were to review these data on at least a  monthly basis and directly retrieve any acquisition documents to be  reviewed from the library. No response from the milestone decision  authority would indicate implicit approval of the program data. Although  the formal RIT pilot ended in December 2003, program officials told us that  they continued to operate using the RIT pilot’s procedures and continued to  update program information on the Web site through December 2004.
According to a memorandum issued by the Office of the Assistant  Secretary of Defense for Networks and Information Integration/Chief  Information Officer and the Undersecretary of Defense for Acquisition,  Technology, and Logistics, the principal output of the pilot would be a  blueprint for IT acquisition that is transferable to other systems. A report  summarizing the results of the entire RIT pilot program was published in April 2005. This report concluded that (1) by instituting risk-based  governance, the milestone decision authority can be assigned to an  organization subordinate to the Office of the Secretary of Defense without  adding unacceptable risk to the investment process and (2) the success of  risk-based governance and cycle time reduction is predicated on the  adoption of net-centricity by the business community.
section_title: Prior Review Identified Strengths and Weaknesses in DOD’s Acquisition Policies and Guidance
paragraphs: In July 2004, we reported that DOD’s revised systems acquisition policies  and guidance incorporated many best practices for acquiring business  systems, such as (1) justifying system investments economically, on the  basis of costs, benefits, and risks, and (2) continually measuring an  acquisition’s performance, cost, and schedule against approved baselines.  However, the revised policies and guidance did not incorporate a number  of other best practices, particularly those associated with acquiring  commercial component-based business systems, and DOD did not have  documented plans for incorporating these additional best practices into its  policies. We also reported that the department’s revised acquisition policies  did not include sufficient controls to ensure that military services and  defense agencies would appropriately follow these practices. We  concluded that, until these additional best practices were incorporated into  DOD’s acquisition policies and guidance, there was increased risk that  system acquisitions would not deliver planned capabilities and benefits on  time and within budget and increased risk that an organization will not  adopt and use best practices that were defined. Accordingly, we made 14  recommendations to the Secretary of Defense that were aimed at  strengthening DOD’s acquisition policy and guidance by including  additional IT systems acquisition best practices and controls for ensuring  that these best practices were followed. DOD agreed with most of our  recommendations and has since issued additional system acquisition  guidance.
section_title: NTCSS Has Not Been Managed in Accordance with DOD and Other Relevant System Acquisition and Development Guidance
paragraphs: DOD system acquisition and development policies and guidance, along  with other federal and best practices guidance, provide an effective  framework within which to manage IT business system programs and  investments, like NTCSS. Proper implementation of this framework can  minimize program risks and better ensure that system investments deliver  promised capabilities and benefits on time and within budget. The Navy  has not managed NTCSS in accordance with many key aspects of these  policies and guidance. For example, the Navy has not economically  justified its investment in NTCSS on the basis of cost and benefits. It has  not invested in NTCSS within the context of a well-defined enterprise  architecture. Further, the Navy has not effectively performed key  measurement, reporting, and oversight activities, and has not adequately  conducted requirements management and testing activities. Reasons the  Navy cited for not following policies and guidance included questioning  their applicability to the NTCSS program, having insufficient time in which  to apply them, and believing that plans to adopt them were not meant to be  applied retroactively. In some cases, the Navy did not acknowledge that  any deviations from policies and guidance had occurred but, in these cases,  it has yet to provide us with documentation demonstrating that it did  adhere to them. As a result, the Navy does not currently have a sufficient  basis for determining whether NTCSS is the right system solution for its  tactical command support needs, and it has not pursued the proposed  solution in a way that increases the likelihood of delivering defined  capabilities on time and within budget.
subsections: section_title: The Navy Has Not Economically Justified Investment in NTCSS on the Basis of Costs and Benefits
paragraphs: The decision to invest in any system should be based on reliable analyses of  estimated system costs and expected benefits over the life of the program.  DOD policy requires such analyses, and other relevant acquisition  management practices provide guidance on how these analyses should be  prepared. However, the current economic analysis for the NTCSS program  does not meet this guidance. Additionally, the analysis was not  independently reviewed in accordance with DOD guidance. Finally,  contrary to DOD policy and relevant acquisition management practices, the  Navy has not demonstrated that NTCSS Optimized applications are  producing expected benefits. Without such reliable analyses, an  organization cannot adequately know that a given system investment is  justified.
subsections: section_title: The Latest NTCSS Cost Estimate Was Not Derived Reliably
paragraphs: According to DOD guidance, the cost estimates used to economically  justify an investment should be reasonable, traceable, and based on  realistic assumptions. Our research shows that a reliable cost estimate  should meet nine specific criteria developed by Carnegie Mellon  University’s Software Engineering Institute (SEI), such as appropriately  sizing the task being estimated and identifying and explaining estimate  assumptions.
In March 2004, the NTCSS program office prepared its fifth NTCSS  economic analysis. This analysis examined the costs associated with three  alternative NTCSS hardware, software, operating system and data base  management configurations, and was to be used to inform decisions about  system development and implementation. The analysis did include  estimated costs for each alternative. However, it did not include  measurable, quantifiable benefits for each alternative. Rather, it included  only qualitative benefits. Further, the cost estimates used in this analysis  did not meet six of the nine criteria associated with reliable cost estimates.  For example, while the estimate’s purpose was stated in writing, the system  life cycle used was 6 years rather than the 10 years recommended. Also,  documentation showing that the costs were based on data from the  program’s demonstrated accomplishments has yet to be provided to us, and  the assumptions used to create the cost estimate were not identified and  explained. See table 8 for the results of our analyses relative to each of the  nine criteria.
Program officials told us that they did not develop the 2004 cost estimate in  accordance with all of the SEI cost estimating criteria because they had  only a month to complete the economic analysis. By not following  practices associated with reliable estimates, the Navy has decided on a  course of action that is not based on one of the key ingredients to sound  and prudent decision making— a reliable estimate of system life cycle  costs. Among other things, this means that the investment decision made  by the Navy has not been adequately justified and, that to the extent that  program budgets were based on cost estimates, the likelihood of funding  shortfalls and inadequate funding reserves is increased.
section_title: The Latest NTCSS Economic Analysis Did Not Meet Key Federal Guidance
paragraphs: According to Office of Management and Budget (OMB) guidance,  economic analyses should meet certain criteria to be considered  reasonable, such as comparing alternatives on the basis of net present  value and conducting an uncertainty analysis of costs and benefits.
The latest NTCSS economic analysis, prepared in March 2004, identified  potential costs and benefits from three alternative NTCSS hardware,  software, operating system, and data base management configurations.  However, the analysis provided only monetized costs for each alternative.  It did not provide monetized benefits. Further, the analysis did not meet  five of eight OMB criteria. For example, the alternatives were not  compared on the basis of their net present values, an appropriate interest  rate was not used to discount the net present values, and the uncertainty  associated with the cost estimates was not disclosed and used in the  analysis. See table 9 for the results of our analyses relative to each of the  eight criteria.
Program officials told us that they did not adhere to the OMB criteria  because they had only a month to complete the economic analysis and,  therefore, did not have the time necessary to comply with it. By not  following established OMB guidance, the reliability of the latest NTCSS  economic analysis is questionable. This further increases the risk that the  Navy is following a course of action that will not produce the expected  return on investment.
subsections: section_title: The Latest NTCSS Economic Analysis Was Not Independently Reviewed
paragraphs: DOD guidance states that economic analyses and cost estimates should  be independently reviewed and assessed. In this regard, the Office of  Program Analysis and Evaluation, located in the Office of the Secretary of  Defense, is responsible for verifying and validating the reliability of  economic analyses and providing the results to the milestone decision  authority; the Naval Cost Analysis Division is responsible for preparing  independent cost estimates.
However, neither of these offices reviewed the most recent economic  analysis for NTCSS. An official from the Office of Program Analysis and  Evaluation told us that this office did not review the 2004 economic  analysis because, once NTCSS entered the RIT Pilot, the program office no  longer provided documentation needed to review the analysis. Officials  from the Naval Cost Analysis Division also stated that they did not review  the estimates in this economic analysis. According to officials from this  office, they are only required to review cost estimates that are prepared for  milestone reviews, and staffing limitations do not permit them to review all  cost estimates.
By not having the economic analysis reviewed by independent parties, the  Navy has no independent verification that the estimates of life cycle costs  and benefits are reasonable and traceable, that the cost estimates are built  on realistic program and schedule assumptions, or that the return on  investment calculation is valid. This casts further doubt on the reliability of  the economic analysis the Navy has used to justify its ongoing investment  in NTCSS.
section_title: The Navy Has Yet to Measure Whether Actual Benefits Have Accrued from Deployed NTCSS Capabilities
paragraphs: The Clinger-Cohen Act of 1996 and OMB guidance emphasize the need to  develop information to ensure that IT projects are actually contributing to  tangible, observable improvements in mission performance. DOD  guidance also requires that analyses be conducted to validate estimated  benefits and measure the extent to which desired outcomes have been  achieved. To this end, agencies should define and collect metrics to  determine whether expected benefits are being achieved and modify  subsequent applications and investments to reflect the lessons learned.
However, the Navy has yet to measure whether NTCSS Optimized  applications are actually producing expected benefits commensurate with  actual costs. For example, in 1999 the Navy projected that deploying the  NTCSS Optimized applications would result in reduced costs associated  with NTCSS maintenance, training, and other support activities. However,  the Navy does not know the extent to which NTCSS Optimized applications  are meeting these expectations—even though these applications have been  deployed to 229 user sites since 1998—because metrics to demonstrate that  these expectations have been met have not been defined and collected.
Program officials and officials representing the milestone decision  authority stated that the Navy is not required to measure actual accrual of  benefits because DOD guidance to do so was not yet in effect when the  NTCSS Optimized applications were deployed, and there was no explicit  requirement to apply this guidance retroactively. Program officials also  stated that it will not be possible to measure actual return-on-investment  for the already deployed NTCSS Optimized applications until the entire  NTCSS system is deployed and operational. Similarly, an official with the  milestone decision authority stated that actual NTCSS return-on-  investment has not yet been measured.
Because it is not measuring whether cost and benefit projections are being  met, the Navy lacks important information that it will need to inform future  economic analyses and investment decisions.
section_title: The Navy Recently Decided to Prepare a Benefits Assessment
paragraphs: In February 2005, officials from the Office of the Chief of Naval Operations  for Material Readiness and Logistics Operations and representatives from  key user organizations questioned whether NTCSS can cost effectively  meet users’ future needs. Initially this office tasked the program office to  develop a new economic analysis to determine whether to continue  investing in NTCSS or in some other system solution, such as the Navy  enterprise resource planning (ERP) program. In November 2005, officials  from the Office of the Chief of Naval Operations for Material Readiness and  Logistics Operations stated that they were no longer planning to develop a  new economic analysis but planning to conduct a benefits assessment to  evaluate changing NTCSS to some solution to enable the system to perform  future ashore activities. These officials acknowledged that this assessment  will be less than the initially planned economic analysis in that it will  exclude any analysis of costs and alternative solutions. However, they also  acknowledged that DOD policy and guidance does not address benefits  assessments as a recognized acquisition program document. They stated  that this assessment will be prepared for inclusion in the 2006 budget  submission.
Without knowing the extent to which NTCSS Optimized applications are  meeting cost and benefit expectations, the Navy is not in a position to make  informed, and thus justified, decisions on whether and how to proceed with  the program. Such a situation introduces a serious risk that the Navy will  not be able to demonstrate whether NTCSS is cost-effective until it has  already spent hundreds of millions of dollars more on the NTCSS  Optimized applications and OOMA.
section_title: The Navy Has Not Defined and Developed NTCSS within the Context of an Enterprise Architecture
paragraphs: DOD policy and guidance, as well as federal and best practice guidance,  recognize the importance of investing in IT business systems within the  context of an enterprise architecture. Our research and experience in  reviewing federal agencies shows that not doing so often results in systems  that are duplicative, not well integrated, unnecessarily costly to interface  and maintain, and do not optimally support mission outcomes. NTCSS has  not been defined and developed in the context of a DOD or Navy enterprise  architecture because a well-defined version of either has not existed to  guide and constrain the program, and meaningful analysis showing how  NTCSS aligns to evolving DOD and Navy architecture efforts was not  produced. This means that the Navy does not have a sufficient basis for  knowing if NTCSS, as defined, properly fits within the context of future  DOD and Navy business operational and technological environments.
More specifically, a well-defined enterprise architecture provides a clear  and comprehensive picture of an entity, whether it is an organization (e.g.,  a federal department) or a functional or mission area that cuts across more  than one organization (e.g., personnel management). This picture consists  of snapshots of both the enterprise’s current or “As Is” environment and its  target or “To Be” environment, as well as a capital investment road map for  transitioning from the current to the target environment. These snapshots  consist of integrated “views,” which are one or more architecture products  that describe, for example, the enterprise’s business processes and rules;  information needs and flows among functions; supporting systems,  services, and applications; and data and technical standards and structures.  GAO has promoted the use of architectures to guide and constrain systems  modernization, recognizing them as a crucial means to a challenging goal:  agency operational structures that are optimally defined in both the  business and technological environments.
DOD has long operated without a well-defined enterprise architecture for  its business environment. In 2001, we first reported that DOD did not have  such an architecture and recommended that it develop one to guide and  constrain IT business systems, like NTCSS. Over the next 4 years, we  reported that DOD’s architecture development efforts were not resulting in  the kind of business enterprise architecture that could effectively guide and  constrain business system investments, largely because the department  did not have in place the architecture management structures and  processes described in federal guidance. In particular, we most recently  reported in July 2005 that despite spending about $318 million producing  eight versions of its architecture, DOD’s latest version still did not have, for  example, a clearly defined purpose that could be linked to the department’s  goals and objectives and a description of the “As Is” environment and a  transition plan. Further, we reported that the description of the “To Be”  environment was still missing important content (depth and scope) relative  to, for example, the actual systems to be developed or acquired to support  future business operations and the physical infrastructure (e.g., hardware  and software) that would be needed to support the business systems. Over  the last several years, we have also reported that DOD’s efforts for  determining whether ongoing investments were aligned to its evolving  architecture were not documented and independently verifiable. On  September 28, 2005, DOD issued the next version of its business enterprise  architecture, which we are required to review, along with other things  such as the department’s efforts to review certain investments’ alignment  with the architecture, pursuant to the Fiscal Year 2005 National Defense  Authorization Act.
The Navy has also not had an enterprise architecture to guide and constrain  its IT system investments. For example, in February 2002 and November  2003, we reported that while the Navy was developing an enterprise  architecture, the architecture products were not complete and they were  not, for example, under configuration management. Since that time, the  Navy has yet to develop an enterprise architecture. In response to our  request for the latest version of its architecture, the Assistant Secretary of  the Navy, Research Development and Acquisition, Chief Engineer, provided  us documentation that describes high-level principles or goals that the  Navy wants to achieve, such as systems interoperability. However, most of  the critical products that an enterprise architecture should include were  not provided, such as (1) a data dictionary, which is a repository of  standard data definitions for applications; (2) a logical database model that  provides the data structures that support information flows and that  provides the basis for developing the schemas for designing, building, and  maintaining the existing physical databases; and (3) an analysis of the gaps  between the baseline and target architecture for business processes,  information/data, and services/application systems to define missing and  needed capabilities. According to the Deputy Assistant Secretary of the  Navy for Command, Control, Communication, Computers and Intelligence,  and Space, the Navy does not have an enterprise architecture. However,  these officials stated that the Navy recognizes the importance of  developing and using one and is taking steps to do so. They did not have a  time frame as to when this would be accomplished, however.
In addition, NTCSS program officials told us that the system has been  assessed against DOD’s business enterprise architecture, and based on this  assessment, the system is aligned. However, our analysis of the alignment  documentation showed while NTCSS could be mapped to several  enterprise architecture elements (e.g., strategic goals and organizational  roles), it was not mapped to other important elements (e.g., technical  standards and data model). Moreover, as previously discussed, the version  of the enterprise architecture used to assess alignment lacked utility and  did not provide a sufficient basis for making informed investment  decisions.
These officials stated that they have not yet assessed the system against the  Navy’s architecture because (1) the architecture has yet to be sufficiently  developed and (2) compliance with this architecture may not be required.
Without having a well-defined architecture to set the institutional context  within which a given investment like NTCSS must fit and taking proactive  and verifiable steps to understand the extent to which the system as it is  defined fits within this context, misalignments can occur that can  introduce redundancies and incompatibilities and that can produce  inefficiencies and require costly and time consuming rework to fix. In the  case of NTCSS, this could be a problem because of the Navy’s ongoing  investment in its ERP program. As we recently reported, this program is  intended to provide functionality in such areas as supply and workforce  management for ashore activities, which is functionality similar to that of  NTCSS for afloat activities. However, both programs have proceeded  without a common, institutional frame of reference (i.e., enterprise  architecture) that can be used to effectively manage their relationships and  dependencies. Our research and experience in reviewing federal agencies  shows that managing such relationships on a program to program basis is  untenable and has proven unsuccessful. This is why the inherent risks  associated with investing in systems in the absence of a well-defined  architecture need to be explicitly disclosed and deliberately evaluated in  order to make a well-informed investment decision.
section_title: Key Program Management and Oversight Activities Have Not Been Effectively Performed
paragraphs: Key aspects of effective program management include reliable progress  measurement and reporting, appropriate budgeting, and meaningful  oversight. DOD policy requires such activities, and DOD and other industry  best practices provide guidance on how these activities should be  conducted. However, these activities have not been effectively performed  on the NTCSS program. Specifically, the Navy has not adequately measured  progress against planned cost and scheduled work commitments, fulfilled  defined reporting requirements, properly budgeted for expenditures, and  conducted meaningful program oversight. As a result, opportunities for  proactive program intervention and actions to address risks and problems  were missed, allowing the program to proceed largely unchecked.
subsections: section_title: The Navy is Not Adequately Measuring Progress Against Planned Cost and Scheduled Work Commitments
paragraphs: Measuring and reporting progress against cost and schedule commitments  is a vital element of effective program management. DOD policy and  guidance recognize this by requiring the use of earned value management,  and describing how it is to be performed. The NTCSS program has elected  to use earned value management; however, it is not doing so effectively. As  a result, the program, as well as Navy and DOD oversight authorities, have  not had access to the kind of reliable and timely information they need to  make informed decisions.
DOD Has Adopted Industry Standards for Earned Value  Management According to DOD policy and guidance, program offices should obtain  data from contractors and central design agencies on work progress, and  these data should relate cost, schedule, and technical accomplishments.
Moreover, the guidance states that these data should be valid, timely, and  auditable. The tool that many DOD entities, including the NTCSS’s program  office and its central design agency, use to obtain and report these data is  known as earned value management (EVM). Through EVM, program  offices and others can determine a contractor’s or central design agency’s  ability to perform work within cost and schedule estimates. It does so by  examining variances between the actual cost and time to perform work  tasks and the budgeted/estimated cost and time to perform the tasks.
In 1996, DOD adopted industry guidance that identifies 32 criteria that a  reliable EVM system should meet. The 32 criteria are organized into five  categories: organization, planning and budgeting, accounting, analysis and  management reports, and revisions and data maintenance (see app. III for  the 32 criteria). As we previously reported, EVM offers many benefits  when done properly. In particular, it is a means to measure performance  and serves as an early warning system for deviations from plans. It  therefore enables a program office to mitigate the risk of cost and schedule  overruns.
NTCSS Has Not Effectively Implemented EVM The EVM system that NTCSS has implemented to measure program  performance does not provide the kind of reliable and timely data needed  to effectively identify and mitigate risks. According to the NTCSS central  design agency’s self-assessment of its earned value management system, 17  of the 32 industry best practice criteria are not being satisfied by the EVM  system it has implemented. For example, the central design agency  reported that the system cannot (1) establish and maintain a budget  baseline against which program performance can be measured over time,  (2) identify management reserves in case of contingencies, (3) record all  indirect costs that will be allocated to the work, (4) summarize data  elements and associated variances through the work breakdown structure  to support management needs, and (5) develop revised estimates of cost at  completion based on performance to date.
Beyond this self-assessment, our review showed that 29 of the 32 criteria  were not satisfied. For example, the system does not (1) provide for the  integration of planning, scheduling, budgeting, work authorization, and  cost accumulation management process; (2) identify physical products,  milestones, technical performance goals, or other indicators used to  measure progress; (3) reconcile current budgets to prior budgets in terms  of changes to the authorized work and internal replanning; and (4) control  retroactive changes to records. See appendix III for the Navy’s complete  self-assessment and our full analysis of the extent to which the 32 criteria  are satisfied.
Officials with the program office and the central design agency stated that  although they chose to use EVM, they are not required by DOD policy to do  so and, therefore, do not have to comply with the 32 criteria. These officials  stated that one reason they are not required to use it is because the  program office and the central design agency are part of the same  organization (the Space and Naval Warfare Systems Command) and thus a  formal contract or written agreement between them does not exist. They  also stated that although the program as a whole exceeds dollar thresholds  for which EVM is required, they have chosen to break the program into  smaller projects managed on a fiscal year basis, and none of these projects  individually exceeds either the new or old DOD policy thresholds that  would require the use of EVM.
We do not agree that the absence of a contractual relationship or the  decomposition of the program into small, fiscal year-based projects is a  valid reason for not effectively implementing EVM. DOD and OMB  guidance require that the Navy base programmatic decisions on reliable  analyses of estimated system’s costs and expected benefits over the life of  the program. The program office chose to use EVM as a means to satisfy  these requirements and to measure progress and identify potential  problems early, so that they could be effectively addressed. To accomplish  this, EVM must be performed correctly. By not implementing it correctly on  NTCSS, the Navy is losing an opportunity to gain the kind of visibility into  program progress needed to identify problems and risks early and better  ensure program success. Moreover, by tracking individual projects on a  yearly basis the program office cannot adequately understand the status of  the NTCSS program as a whole, which hinders its ability to accurately  forecast program costs at completion and provide realistic schedule  projections. In short, without reliable, timely, and auditable EVM data, the  program office cannot adequately manage technical, cost, and schedule  risks and problems.
Two NTCSS Projects Illustrate How EVM Has Been Poorly  Implemented Two of the individual NTCSS projects for which EVM activities were  reportedly being performed are (1) 2004 OOMA software development and  (2) 2004 NTCSS hardware installation and integration (for both OOMA and  Optimized NTCSS). For the OOMA software project, EVM was performed  by the central design agency and for the NTCSS hardware project it was  performed by the Space and Naval Warfare Systems Command Systems  Center, Charleston. On both projects, we found several examples of  ineffective EVM implementation, including the following:   An integrated baseline review was not conducted for either of the  projects. According to DOD guidance and best practices, an integrated  baseline review should be conducted as needed throughout the life of a  program to ensure that the baseline for tracking cost, technical, and  schedule status reflects (1) all tasks in the statement of work, (2)  adequate resources in terms of staff and materials to complete the tasks,  and (3) integration of the tasks into a well-defined schedule. Further,  program managers are to use cost performance reports that have been  validated by an integrated baseline review. Without verifying the  baseline, monthly cost performance reporting, which is to track against  a set budget and schedule, does not have sufficient meaning or validity.
The estimate at completion for the 2004 OOMA software project, which  is a forecast value expressed in dollars representing the final projected  costs of the project when all work is completed, showed a negative cost  for a 6-month period (November 2003 to April 2004). When EVM is  properly implemented, this amount should include all work completed  and always be a positive number. The negative estimate at completion  for this project would mean that the central design agency had incurred  a savings rather than spending money, even though during that time  more than $1.7 million had been spent.
The schedule performance index for the OOMA software project, which  is to reflect the critical relationship between the actual work performed  versus the costs expended to accomplish the work, showed program  performance during a time when the program office stated no work was  being performed. Specifically, the reports showed the schedule  performance fluctuating between $0.21 worth of work performed for  every dollar spent to more than $3.75 worth of work performed for every  dollar spent during a time that the program office claims all work was  halted. Perfect performance would indicate schedule indices equal to  1.0 at best (i.e., for every dollar spent there was 100 percent of the  schedule achieved).
The estimate at completion for the OOMA hardware installation project  showed that almost $1 million in installation costs had been removed  from the total sunk costs, but no reason for doing so was provided in the  cost performance report.
The cost and schedule indices for the OOMA hardware installation  project showed improbably high program performance during a time  when the installation schedules and installation budget had been  drastically cut because OOMA software failed operational testing.  Specifically, the reports between March 2004 and July 2004 showed the  current cost performance fluctuating between $0.07 worth of work  performed for every dollar spent to $8.48 worth of work performed for  every dollar spent.
Navy officials cited several reasons for these shortcomings. For the  software project, program officials stated that prior to the operational  testing of OOMA in 2003, the central design agency’s implementation of  EVM was primitive at best and that the resulting data were not usable. They  also stated that after the project failed operational testing, they did not see  the value in rebaselining the project and thus all EVM analysis was halted.  They did, however, continue to invest in OOMA. For the hardware  installation project, a Charleston Center official responsible for developing  the installation reports stated that there were problems with collecting  actual costs because the individuals responsible for doing the work were  covered by other contracts, and there was no way to ensure that the costs  were being reported consistently. Regarding the approximately $1 million  in installation costs that were removed from the total sunk costs, this  official stated that these costs were erroneously charged to this project and  were thus removed because they were not part of the original plan.
Ineffective implementation of EVM, as occurred on these two projects,  precludes NTCSS program officials from having reliable and timely  information about actual program status and does not provide these  officials with a sound basis for making informed program decisions.
section_title: The Navy Has Not Adequately Reported NTCSS’s Progress and Problems
paragraphs: One essential aspect of effective program management is complete and  current reporting by the program office to oversight organizations  responsible for making decisions regarding the program’s future. DOD  policy recognizes this, stating that the program office is accountable for  providing credible schedule, performance, and cost reporting information  to the milestone decision authority. Officials from the NTCSS milestone  decision authority told us that they relied on the program office to fully  disclose progress against, and deviations from, program cost, schedule, and  performance goals. However, the program office has not reported  consistently or reliably on the program’s progress and, as a result, has not  fully disclosed program status to Navy and DOD oversight authorities who  are responsible for making proper investment decisions.
Navy Reporting Requirements for NTCSS Have Changed over the  Last Several Years Since program inception, NTCSS requirements for reporting cost, schedule,  and performance information have changed. Prior to October 2002, the  program office was required to comply with applicable DOD acquisition  policies and guidance. This guidance generally required the program  office to provide oversight organizations with the following three key  reports:  The Acquisition Program Baseline, which describes the program’s cost,  schedule, and performance goals. This baseline document is to be  developed when the program is initiated, and it is to be updated for each  milestone review. Within 90 days of a program breach, unless the  program is back within its baseline goals, a new Acquisition Program  Baseline is to be prepared by the program office and approved by the  milestone decision authority.
The Program Deviation Report, which is to be prepared when the  program office identifies deviations from the approved Acquisition  Program Baseline goals. More specifically, when the program office has  reason to believe that a program breach will occur, it is to immediately  notify the milestone decision authority. Within 30 days, the program  office is to inform the milestone decision authority of the reason for the  deviation and the actions it considers necessary to bring the program  back within baseline goals.
The Defense Acquisition Executive Summary, which is prepared to  inform the milestone decision authority on the program’s progress  against cost, schedule, and performance goals reflected in the  Acquisition Program Baseline. Prepared quarterly, the summary is  designed to provide an early warning to the DOD Chief Information  Officer (CIO) and the milestone decision authority by identifying  existing and potential program problems and describing mitigating  actions that have been taken.
Between October 2002 and December 2004, the reporting requirements for  the program changed. As previously discussed, NTCSS was selected by its  milestone decision authority to participate in the RIT pilot, which was  aimed at saving time in the acquisition management process by reducing  traditional DOD reporting and oversight requirements, while still adhering  to DOD acquisition guidance. Under the RIT pilot, the program office was  required to prepare the following two monthly electronic reports:   The Monthly Acquisition Program Review, which was to assess the  current health of the program on a monthly basis in such areas as cost  and schedule performance, testing, funding, and contracting. This report  was broken into eight parts. According to the program office, the main  part for NTCSS was the Program Manager Assessment.
The Smart Chart, which was to address risks for different projects  within the program, including a description of the risk, actions taken to  address the risk, and recommendations for further actions. The Smart  Chart was also to contain any updates to the Acquisition Program  Baseline.
In short, the RIT reporting was to provide the same information reported  via the traditional acquisition baseline and the summary report, but it was  to be more frequent (monthly versus quarterly) and use a different format  (electronic versus paper). In addition, under the RIT pilot, certain  acquisition documents, such as acquisition plans, economic analyses,  requirements documents, and test plans, were to be posted to the RIT Web  site’s electronic library rather than sent in hard copy to the program’s  stakeholders.
In December 2004, the program office and the milestone decision authority  agreed to discontinue use of the RIT pilot procedures. In January 2005, the  reporting requirements reverted to the acquisition policies and procedures  as prescribed in the updated DOD 5000 series. Currently, the program  office is required to prepare the summary report quarterly and the  acquisition baseline as needed. Also, in January 2005, the Navy required the  program office to begin making entries into the Dashboard. The  Dashboard, like the summary report, is prepared by the program office on a  quarterly basis for the milestone decision authority and is to provide an  assessment of the program in such areas as cost, schedule, and  performance characteristics.
The Navy Has Not Satisfied All NTCSS Reporting Requirements The program office did not comply with the reporting requirements that  were in effect during the 27 months of the RIT pilot. Specifically:  The Smart Chart was not updated for 19 of the 27 months. Specifically,  the data were updated eight times between October 2002 and November  2003; the data were not updated after November 2003.
The Program Manager Assessment was not updated for 11 of the 27  months. In addition, the updates were not always made in a timely  manner. For the 16 months that were updated, 7 were done after the  month had ended, and most of these updates were a month late.
Of the 15 essential acquisition documents that the program office  committed to entering in the RIT electronic library, 10 were not entered.  For example, the most recent economic analysis and the test and  evaluation master plan for OOMA were not entered.
The Program Deviation Report and Acquisition Program Baseline were  not prepared in a timely manner. Specifically, in April 2004, the  acquisition of eNTCSS was cancelled and, in May 2004, OOMA did not  pass operational testing—two events that caused the related cost and  schedule thresholds in the Acquisition Program Baseline to be breached.  While program officials had notified the milestone decision authority of  these events via (1) e-mail, (2) entries into the Program Manager  Assessment on the RIT Web site, and (3) briefings, the program office  did not prepare a Program Deviation Report until about 15 months later.  Moreover, this deviation report addressed only the OOMA failure, not  the cancellation of eNTCSS and reprogramming of unexpended eNTCSS  funding. In addition, program officials have yet to provide us with a new  Acquisition Program Baseline to reflect the program breach or  documentation showing that this revised baseline has been approved by  the milestone decision authority.
For the DOD and Navy reporting requirements in effect since January 2005,  the Navy has satisfied some, but not all, of the reporting requirements. For  example, the program office has prepared the Dashboard reports quarterly  as required. However, it has not prepared the Defense Acquisition  Executive Summary quarterly as required; the first report was not prepared  until June 2005—6 months after the requirement resumed and the report  was due.
Program officials provided various reasons why the required program  reporting has not occurred. In the case of the Smart Charts and the  Program Manager Assessment reports, a contractor supporting the  Assistant Program Manager stated that the data may have been entered into  the Web site but not properly saved. Regarding the posting of documents  into the electronic library, an official from the milestone decision authority  stated that there was no documentation from the Office of the Assistant  Secretary of Defense for Networks and Information Integration/Chief  Information Officer that directed which, if any, acquisition documents were  to be entered into the RIT Web site. Similarly, a contractor supporting the  Assistant Program Manager stated that the folders in the electronic library  were established by the Army and thus the Navy was not required to use  them. However, our review of documentation provided by the program  office shows that it clearly states which specific documents should be  included in the electronic library. Regarding the delay in preparation of the  Program Deviation Report and subsequent Acquisition Program Baseline  revision, a contractor supporting the Assistant Program Manager stated  that a new baseline should have been prepared sooner, but that this  reporting was delayed due to the uncertainty of which reporting methods  to use after the end of the formal RIT pilot.
Officials representing the milestone decision authority stated that they  relied on program office reporting on program status and progress, and  that they expected the program office to inform them if the program  exceeded its cost, schedule, and performance thresholds. Without adequate  reporting, oversight officials were not positioned to effectively execute  their roles and responsibilities.
section_title: The Navy Has Not Properly Budgeted for NTCSS
paragraphs: In September 1999, the Navy Comptroller issued guidance directing  program offices to review their budgets and identify efforts that were being  improperly funded and to take the steps necessary to realign these funds to  “Research, Development, Test and Evaluation” as quickly as possible.  Further, DOD Financial Management Regulation requires that IT  development, test, and evaluation requirements generally be funded in the  “Research, Development, Test and Evaluation” appropriations. More  specifically it states that, “The Research, Development, Test and Evaluation  funds should be used to develop major upgrades increasing the  performance envelope of existing systems, purchase test articles, and  conduct developmental testing and/or initial operational test and  evaluation prior to system acceptance.” Similarly, Navy financial  management policy states that, “All costs associated with software  development/modification efforts that provide a new capability or expand  the capability of the current software program (i.e., expand the  performance envelope) are funded in the Research, Development, Test and  Evaluation appropriation.”
However, this has not occurred. Since 1997, the program office has not  identified “Research, Development, Test and Evaluation” funds in five of its  seven Acquisition Program Baseline documents, three of which were  prepared after the guidance was issued by the Comptroller of the Navy.  Instead, the Navy funded these activities primarily out of the “Operations  and Maintenance,” “Other Procurement,” and “Ship Construction”  appropriations. (See table 10.)
Program officials agreed that they have funded NTCSS development  activities, such as those associated with OOMA, out of the “Operation and  Maintenance” appropriation rather than the “Research, Development, Test  and Evaluation” appropriation. A contractor supporting the Assistant  Program Manager stated that, although they were aware of the Comptroller  of the Navy’s budget guidance, the program office chose not to comply  because program officials believed in 1999 that the OOMA application,  which had been under development for 3 years, would pass developmental  testing and operational testing by 2001. As a result, program officials  determined that the effort required to reprogram funding from the  “Operation and Maintenance” appropriation into the “Research,  Development, Test and Evaluation” appropriation was not warranted.  Further, the official stated that although OOMA did not pass operational  testing in 2001, the program office did not fund OOMA with “Research,  Development, Test and Evaluation” funds until 2004 because it continued to  consider OOMA as being close to becoming operational.
The lack of proper budgeting for “Research, Development, Test and  Evaluation” funding has given oversight authorities the misleading  impression that NTCSS development activities were completed and that  the system was fully operational. Specifically, officials from the Office of  the Assistant Secretary of Defense for Networks and Information  Integration/Chief Information Officer, which was the original NTCSS  milestone decision authority, stated that since most of the “Research,  Development, Test and Evaluation” funding appeared to have been spent,  they concluded that the development portion of NTCSS was essentially  complete. As a result, these officials stated that they had considered taking  NTCSS off of the list of programs subject to oversight reviews. However,  after 9 years and over $79 million in expenditures, the OOMA application  still has not passed operational testing and thus is still in development.
section_title: Navy Oversight of NTCSS Has Not Been Adequate
paragraphs: DOD and Navy policies task a number of organizations with oversight of IT  system acquisition and development programs. For example, DOD policy  states that a milestone decision authority has overall program  responsibility. In addition, the Navy Chief Information Officer is  responsible for reviewing programs at certain points in the acquisition  cycle. Finally, the NTCSS Executive Steering Committee is responsible for  monitoring the near-term development and evolution of the NTCSS  program. However, effective oversight by these entities has not occurred.  As a result, opportunities to address long-standing program weaknesses  have been missed, and the program has been allowed to proceed virtually  unchecked.
The Milestone Decision Authority Has Not Adequately Overseen  the Program DOD acquisition policy states that a milestone decision authority is the  designated individual with overall responsibility for a program and is to  ensure accountability and maximize credibility in program cost, schedule,  and performance reporting. In this role, the milestone decision authority is  responsible for reviewing the program throughout its acquisition life cycle,  including: (1) whenever the program reaches a milestone decision point;  (2) whenever cost, schedule, or performance goals are baselined or must  be changed; and (3) periodically through review of management  information such as that found in the Defense Acquisition Executive  Summary reports.
However, the Navy milestone decision authority has not conducted such  reviews. Specifically:  The NTCSS program has not reached a milestone decision point in over  5 years. The last such milestone was in April 2000 when the final two  NTCSS Optimized applications became operational. The next scheduled  milestone was to be in 2001, but because OOMA operational testing was  stopped and has yet to be successfully completed, a milestone decision  point has yet to occur. As a result, there has not been a triggering event  that would cause the milestone decision authority to formally review the  program or any of its projects. We discussed the state of NTCSS in  March 2005 with the milestone decision authority’s representatives. In  July 2005, the authority was briefed by the program office. According to  program officials, this was the first formal program review to occur  since termination of the RIT pilot in December 2003. These officials also  stated that quarterly acquisition team meetings have since resumed— with the first meeting having occurred in September 2005 and the next  scheduled for December 2005—to prepare for the next milestone review  of OOMA.
The program office notified the milestone decision authority in April  and June 2004 that OOMA failed operational testing and that eNTCSS  was cancelled via e-mail, entries into the Program Manager Assessment  on the RIT Web site, and briefings. According to officials with the  milestone decision authority, they followed up with the program office  and provided guidance; however, these events did not trigger a formal  program review.
The milestone decision authority did not contact the program office to  inquire as to the reason why monthly reports were not being prepared as  agreed to after the formal RIT pilot had ended. For example, Smart  Charts were not prepared after November 2003. However, according to  milestone decision authority officials, they did not seek an explanation  from the program office as to why. Milestone decision authority officials  told us that they were relying on the Dashboard report in order to stay  informed on the program’s progress. However, they did not require the  program office to begin preparing the Dashboard report until January  2005.
According to DOD and Navy officials, including officials from the Office of  the Assistant Secretary of Defense for Networks and Information  Integration/Chief Information Officer, the Navy milestone decision  authority, and the program office, NTCSS participation in the RIT pilot  resulted in disruption of normal oversight activities, which have yet to be  fully restored. They added that compounding this is the fact that the Navy’s  milestone decision authority’s staffing has been reduced in recent years.  According to these officials, approximately 2 years ago the number of full  time staff in the Office of the Deputy Assistant Secretary of the Navy for  Command, Control, Communication, Computers and Intelligence, and  Space was reduced from 16 to 6 people, and these 6 are responsible for  reviewing approximately 60 acquisition programs. The officials stated that,  given the large number of programs and limited staffing, they are unable to  fully perform oversight activities so they have increasingly relied on the  program executive office’s assistance to perform detailed oversight of this  program. Without adequate oversight by the milestone decision authority,  the NTCSS program has been allowed to proceed despite the program  weaknesses discussed in this report.
While the milestone decision authority is the main program oversight  entity, two other Navy organizations have oversight responsibilities.  However, these entities also have not performed effective oversight of the  program. Specifically,  Department of Navy CIO is responsible for reviewing programs at  certain points in the acquisition cycle to ensure, among other things,  that program goals are achievable and executable and that the program  is providing value (i.e., producing a positive return-on-investment). Navy  CIO officials stated that they have overseen NTCSS primarily by  reviewing the Capital Investment Reports prepared by the program  office. They stated that they have not performed any proactive activities  to verify and validate the program’s status and progress. Instead, they  rely on information in the Capital Investment Reports, such as economic  justification; budget information by appropriation type; and cost,  schedule, progress, and status. However, as was discussed previously,  the program office does not have or has not reported reliable  information on these topics.
The NTCSS Executive Steering Committee is responsible for  establishing priorities for NTCSS development and implementation and  determining the strategic direction of the program. Among other things,  it is to meet immediately following each major NTCSS program meeting.  However, it has not met since December 2002, even though the program  office convened both a Requirements Integrated Product Team meeting  and a Forum meeting in February 2005. Further, during this period,  major setbacks occurred on the program, including the failure of OOMA  to pass operational testing and the cancellation of eNTCSS, which were  issues that affected the direction of the program and its priorities and  thus were consistent with the committee’s charter. Program officials  agreed that the Executive Steering Committee has not formally  convened during this time frame. However, program officials stated that  members of the committee informally met to discuss and provide advice  regarding OOMA concerns, and Navy officials higher than the Executive  Steering Committee made the decision to cancel eNTCSS. Therefore,  these officials stated there was no need to formally convene an  Executive Steering Committee meeting. Program officials stated that the  Executive Steering Committee will be meeting in January 2006.
section_title: NTCSS Requirements and Test Management Weaknesses Have Contributed to Deployment Delays and System Quality Problems
paragraphs: As we have previously reported, the effectiveness of the processes used  to develop a system is a reliable predictor of the quality of the system  products produced. Two key system development processes are  requirements development and management and test management. For the  NTCSS application currently under development, we found weaknesses  with both of these process areas. While improvements are planned, until  they are implemented effectively, the risk of continued NTCSS cost,  schedule, and performance shortfalls persists.
subsections: section_title: The Navy Has Not Adequately Managed Requirements for the NTCSS Application Currently Under Development
paragraphs: Well-defined requirements can be viewed as a cornerstone of effective  system development and implementation. Accordingly, DOD guidance and  industry best practices recognize effective requirements development and  management as an essential system development and acquisition  management process. For the NTCSS application that is currently under  development—OOMA—the Navy has not adequately managed its 732  requirements, as evidenced by a lack of requirements traceability and  prioritization. NTCSS program officials told us that NTCSS requirements  development practices have historically been poor, but that improvements  are under way. Without effective requirements management, it is likely that  the Navy’s challenges to date in developing NTCSS applications that meet  user needs on time and on schedule will continue.
Requirements for OOMA Release 4.10 Were Not Traced DOD guidance and industry best practices also recognize the importance of  requirements traceability. The purpose of requirements traceability is to  ensure that the finished product is compliant with the requirements. To do  this, the system documentation should be consistent and thus complete,  allowing for requirements traceability. Requirements traceability involves  both the alignment and consistency backward to system documentation  and forward to system design and test documentation.
OOMA release 4.10 requirements were not traced to an Operational  Requirements Document. According to DOD guidance, an Operational  Requirements Document translates nonsystem-specific statements of a  needed operational capability into a set of validated and prioritized user  requirements. However, the Navy did not develop an Operational  Requirements Document for NTCSS. As a result, the Navy did not take a  basic validation step to ensure that the requirements to which it designed  and built the application were complete and correct. In addition, release  4.10 requirements were not always traceable to associated system  specifications. Specifically, we were unable to trace 215 requirements  found in the system segment specification to the requirements listed in the  requirements checklist. Requirements should also be traced to test cases,  but the program office has yet to provide us with the developmental test  cases used to test the OOMA release 4.10 so that we could verify this  traceability.
Program officials acknowledged that release 4.10 requirements were not  traceable but that improvements are planned for the next OOMA release.  We found that 97 percent of the OOMA release 5.0 requirements found in  the system segment specification were traceable to the requirements listed  in the requirements checklist. However, these documents have yet to be  approved. Requirements should also be traced to test cases, but the  program office has yet to provide us with the developmental test cases  used to test the OOMA release 5.0 so that we could verify this traceability.  Without this traceability, the Navy has not had a sufficient basis for  knowing that the scope of its development efforts, including testing,  provides adequate assurance that applications will perform as intended.
Requirements for OOMA Release 4.10 Were Not Prioritized According to published best practices guidance, any project with  resource limitations should establish the relative priorities of the requested  features or requirements. Prioritization helps the project office resolve  conflicts, make trade-off decisions among competing requirements, and  helps to ensure that the delivered system will be operationally suitable.
However, OOMA’s approximately 732 requirements have never been  prioritized, and a program official told us that they are all considered to be  equally important. This means, for example, that a requirement that  dictates basic application functionality (e.g., if text can be entered on a  particular screen) is as important as a requirement addressing safety issues  that, if not met, could result in the loss of an aircraft or even a life.
This lack of requirements prioritization contributed to release 4.10 passing  developmental testing but failing operational testing. (See later section of  this report for a detailed discussion of OOMA testing.) A developmental  testing threshold that the Navy set for release 4.10 was that each  requirement was to be tested, and 95 percent of the requirements had to  pass in order for the application to proceed to operational testing. For  developmental testing of the OOMA release 4.10, 97 percent of the  requirements passed. Of the 3 percent of the requirements that failed this  test, some of these deficiencies seriously impacted squadron level  operations. Further, for operational testing of OOMA release 4.10, 96  percent of the requirements passed. However, the remaining 4 percent  contained significant defects. Specifically, the release provided  inconsistent and inaccurate flight and usage hours, as well as incorrect  aircraft usage records. According to the Navy’s independent operational  test organization, these deficiencies impacted aircraft and component time- based inspection cycles and thus were the basis for the system failing  operational testing. The Navy has yet to provide evidence that the  requirements have been prioritized for the OOMA release 5.0.
section_title: The Navy’s Developmental Testing for OOMA Has Not Been Effective, but Improvements Planned
paragraphs: Both DOD policy and relevant guidance recognize that effective testing is  an essential component of system development or acquisition programs.  Generally, testing can be viewed as consisting of two major phases—a  developmental phase in which tests are performed to ensure that defined  system requirements and specifications are met and an operational phase  that includes tests to determine if the system meets user needs and is  suitable in an operational environment. The OOMA application has failed  operational testing twice over the last 4 years reportedly because of  deficiencies in developmental testing. Program officials attributed  developmental testing deficiencies to poor software development  practices, such as the earlier discussed requirements development  problems. These testing deficiencies can also be attributed to incomplete  testing documentation. Without effective developmental testing, there is an  increased risk that application problems will be detected later in the  system life cycle when they are more expensive and difficult to fix.
Navy Operational Testing Organization Reported That  Developmental Testing Has Failed to Identify Problems According to DOD guidance and recognized best practices, the purpose of  developmental testing is to provide objective evidence that the product  (e.g., software module, application, system) satisfies defined requirements  and performs as intended. Successful completion of developmental testing  provides the basis for proceeding into operational testing to determine  whether the integrated product (e.g., application, system, system of  systems) performs as intended in an operational or real-world setting.
OOMA operational testing results over the last 4 years show that the  program office’s developmental testing efforts have not been effective in  identifying critical product problems. In particular, the application has  failed operational testing twice during this time frame and, according to an  official in the office of the Director of Navy Test and Evaluation and  Technology Requirements, the failures occurred in operational testing  because they were not identified during developmental testing. More  specifically,  In March 2001, the program office certified that OOMA release 3.25 had  passed developmental testing and was ready for operational testing.  However, 1 month into a scheduled 3-month operational test, the  decision was made to cease further testing because of significant  problems with system reliability, data transfer between the application  and the database, and user training on the application. As a result, the  program office decertified this release, and the Navy’s independent test  organization recommended discontinuing OOMA deployment.
Using results from the failed operational test, the central design agency  developed release 4.0. In February and March 2002, developmental  testing of this release was conducted. Test results showed that the  application was not ready for operational testing because it did not  satisfy key functional requirements. Subsequently, the central design  agency incorporated software fixes in release 4.10. In August and  September 2002, developmental testing was conducted on this release  and, while a number of deficiencies were verified as fixed, additional  corrections were needed. From January to June 2003, developmental  testing was again conducted on OOMA release 4.10.
From August 2002 to April 2003, the Naval Audit Service reviewed  OOMA and reported several problems that would affect the application’s  readiness for operational testing. For example, it reported that controls  to prevent unauthorized access were not in place, Privacy Act  information was not adequately protected, and backup and recovery  procedures were not in place. It also reported that the program had not  adopted and implemented a risk-based system life cycle management  approach. According to the report, these weaknesses could compromise  safety, affect planning, and distort readiness reporting if OOMA was  implemented throughout the Navy.
In June 2003, the program office certified OOMA release 4.10 as having  passed developmental testing and being ready for operational testing.  The Navy’s independent operational test organization subsequently  conducted testing from August to December 2003 and, in May 2004,  this organization concluded that OOMA was not operationally effective  or suitable and thus it again failed operational testing. In particular, the  operational testing results showed that the application was incorrectly  calculating flight and component usage hours—defects, which  according to an official in the office of the Director of Navy Test and  Evaluation and Technology Requirements, could have resulted in the  loss of aircraft or life. The Assistant Program Manager also told us that  release 4.10 did not address all of the deficiencies reported by the Naval  Audit Service.
For about a year, the central design agency has been developing and testing  OOMA release 5.0 to fix the problems found in the prior version. The  program office expects that this release will be certified as ready for  operational testing sometime between April and June 2006. In preparation  for operational testing, the Navy’s independent operational test  organization has been observing OOMA 5.0 developmental testing. A memo  from this organization states that this release is an improvement over the  previous releases.
According to Navy officials, including the NTCSS Assistant Program  Manager and the official responsible for OOMA developmental testing,  previous application development practices were poor, which led to testing  problems. Specifically, they cited poor requirements definitions, poor  documentation, and concurrent development of application releases as  examples. Further, Navy officials stated that the central design agency has  not had a developmental testing lab to facilitate effective testing of  application components and their integration. To address the poor  development practices, program officials told us that they are in the  process of implementing a new system life cycle management process that  they said incorporates industry best practices, including those related to  testing. However, the program office has yet to provide us with information  defining how the practices in this plan will be implemented. To address the  need for a developmental testing lab, the Naval Air Systems Command  organization representing NTCSS users recently created a lab to strengthen  the program’s developmental testing capability. According to officials  associated with the lab, they are finding defects that the central design  agency should have found.
It is important that the NTCSS program improve its developmental testing.  Without effective developmental testing, there is an increased risk that  system application problems will be detected late in the system life cycle,  such as during operational testing. Generally, problems discovered late in  the cycle are more expensive and difficult to fix than those discovered  early.
Developmental Test Documentation Has Not Been Adequate, but  Improvements Planned To be effective, testing should be approached in a rigorous and disciplined  fashion. One aspect of such testing is developing and using various testing  documentation. DOD policy, guidance, and related best practices state  that such documentation includes a test and evaluation master plan for the  program, as well as documentation that is system product (e.g., module,  application, system) and test type (e.g., integration, stress, regression,  developmental) specific. This documentation includes approved test plans,  test procedures and cases, and test results. According to DOD and other  guidance, test plans should include, among other things, objectives,  responsibilities, resources (tools, people, and facilities), schedules, and  performance and exit criteria; test procedures should include detailed test  scenarios, test events, steps, inputs, and expected outputs that are traced  back to requirements. Test results include the test scenarios that passed  and failed, assessments of deviations from test plans, and the extent to  which requirements have been met.
The NTCSS test and evaluation master plan identified, among other things,  three phases of developmental testing for OOMA release 4.10. However,  key test documentation for each of these phases was not produced.  Specifically,  For the first phase, a test report was produced that contained detailed  information on test results, but the program office has yet to provide us  with a test plan or test procedures.
For the second phase, a test report was produced but it only contained  the number of defects found (organized by severity) and did not include  any other information on test results. Moreover, the program office has  yet to provide us with a test plan or test procedures.
For the third phase, both a test plan and test report were produced, and  the plan included the test purpose and objectives, schedule,  responsibilities, and people resources, while the test report described  test issues and contained detailed test results. However, the program  office has yet to provide us with test procedures.
According to Navy officials, including the Assistant Program Manager and  officials responsible for developmental testing, the previously mentioned  poor application development practices contributed to the absence of  testing documentation. To address these poor practices, the program has  developed a system life cycle plan that they said incorporates industry best  practices, including those associated with testing documentation.  However, the program has yet to provide us with plans defining how these  practices will be implemented. Moreover, while the plan contains a  recommended list of testing documents (e.g., test plan, test procedures,  and test results report), our review of OOMA release 5.0 developmental  testing documentation shows that not all the documentation is being  prepared. Specifically, available documentation included an updated test  and evaluation master plan and two test reports. Documentation not yet  provided to us included test procedures, which would include  documentation tracing test cases to requirements.
The lack of a full set of developmental test documentation is a problem.  Without such documentation, the adequacy and reliability of  developmental testing cannot be substantiated, and thus the quality of the  associated system products is in doubt.
section_title: Central Design Agency Reports Management Improvements are Under Way
paragraphs: In an effort to improve its performance on NTCSS and other programs,  central design agency officials told us that they chose to undergo an SEI  Capability Maturity Model Software Capability Appraisal in July and August  2005. Carnegie Mellon University’s SEI, recognized for its expertise in  software and system processes, has developed the Capability Maturity  Model™ for Software (SW-CMM) to provide guidance on how to gain  control of their processes for developing and maintaining software and  how to evolve toward a culture of software engineering and management  excellence.
In brief, SW-CMM calls for assessing different process areas—clusters of  related activities such as project planning, requirements management, and  quality assurance—by determining whether key practices are implemented  and whether overarching goals are satisfied. Successful implementation of  these practices and satisfaction of these goals result in the achievement of  successive maturity levels. SW-CMM maturity levels range from 1 to 5, with  level 1 meaning that the process is either characterized as ad hoc and  occasionally even chaotic with few processes defined and success  depending on individual effort; level 2 meaning that the process is  repeatable; level 3 meaning that the process is defined; level 4 meaning that  the process is managed; and level 5 meaning that the process is optimized.
According to the central design agency they achieved a maturity rating of  level 3 against the SW-CMM based on 13 process areas, including  requirements management, software project planning, software project  tracking and oversight, subcontract management, software quality  assurance, software configuration management, organizational process  focus, organizational process definition, training program, integrated  software management, software product engineering, intergroup  coordination, and peer reviews. Further, we were told that NTCSS was one  of the programs included in the review. However, we have yet to receive the  appraisal report to determine the extent to which the appraisal addressed  the weaknesses discussed in this report. Nevertheless, our research has  shown that properly performing such appraisals can be a useful starting  point for making software and system related development improvements.
section_title: Conclusions
paragraphs: It is unclear whether the Navy’s planned investment in NTCSS is warranted.  Of particular concern is the absence of reliable analysis showing that  further investment will produce future mission benefits commensurate  with estimated costs, as well as the void in information concerning whether  the deployed and operational components of NTCSS are actually producing  expected value. Compounding this uncertainty is the inherent risk of  defining and developing NTCSS outside the context of either a well-defined  DOD or Navy enterprise architecture. Without this information, the Navy  cannot determine whether NTCSS as defined, and as being developed, is  the right solution to meet its strategic business and technological needs.
Even if these uncertainties were to be addressed, and the Navy had the data  needed to demonstrate that NTCSS plans are the right course of action,  then the manner in which NTCSS is being defined, developed, tested,  measured, and overseen would still be of concern. While any one of the  concerns that we found is troubling, their combination subjects the  program to an unacceptably high risk of failure. These effects are being  realized on NTCSS, as evidenced by the cancellation of one system  component and the repeated failure of another key component to pass  testing.
It is extremely important that Navy and DOD authorities responsible and  accountable for ensuring prudent use of limited resources reassess  whether allowing NTCSS to continue as planned is warranted. It is also  important that the decision on how to proceed be based on reliable data  about program cost, benefits, risk, and status.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Defense direct the Secretary of the  Navy to determine if continued investment in NTCSS, as planned,  represents a prudent use of the department’s limited resources. To  accomplish this, the Secretary of the Navy should direct the program office  to take the following three actions:  collaborate with the Office of the Assistant Secretary of Defense for  Networks and Information Integration/Chief Information Officer, the  Office of Program Analysis and Evaluation, and the Naval Cost Analysis  Division to prepare a reliable economic analysis that encompasses all  viable alternatives, including the Navy’s recent enterprise resource  planning program;  ensure that development of this economic analysis (1) complies with  cost estimating best practices, including recognition of costs to resolve  open trouble reports and change proposals, and relevant OMB cost  benefit guidance and (2) incorporates available data on whether  deployed NTCSS capabilities are actually producing benefits; and  collaborate with the Undersecretary of Defense for Acquisition,  Technology, and Logistics and the Under Secretary of Defense  (Comptroller) to ensure that NTCSS is adequately aligned with evolving  DOD and Navy enterprise architectures.
In addition, we recommend that the Secretary of Defense direct the  Secretary of the Navy to present the results of these analyses to the Deputy  Secretary of Defense, or his designee, and seek a departmental decision on  how best to proceed with the program. Until this is done, we recommend  that the Secretary of Defense direct the Secretary of the Navy to halt  further deployment of NTCSS and to limit future investment in already  deployed applications to essential operation and maintenance activities  and only developmental activities deemed essential to national security  needs.
If—based on reliable data—a decision is made to continue the NTCSS  program, we recommend that the Secretary of Defense direct the Secretary  of the Navy to ensure that the following two actions are taken:  the NTCSS program implements effective program management  activities, including earned value management, requirements  development and management, and test management; and  key stakeholders, such as the central design agency and the  developmental testing organization, have the people, processes, and  tools to effectively execute their respective roles and responsibilities.
Finally, we recommend that Secretary of Defense reestablish the Office of  the Assistant Secretary of Defense for Networks and Information  Integration/Chief Information Officer as the milestone decision authority  and direct the Secretary of the Navy to take steps to ensure that Navy  oversight entities fulfill their roles and responsibilities on NTCSS, including  ensuring that reliable program reporting occurs and is acted upon.
section_title: Agency Comments and Our Evaluation
paragraphs: In its written comments on our draft report, signed by the Deputy to the  Assistant Secretary of Defense for Networks and Information Integration  (Command, Control, Communications, Intelligence, Surveillance, and  Reconnaissance and Information Technology Acquisition) and reprinted in  appendix IV along with our detailed responses, DOD stated that some of  our findings are valid. For example, it acknowledged that NTCSS was  defined and implemented without a complete and formal enterprise  architecture. However, it also commented that our overall findings  significantly understated and misrepresented the program’s level of  discipline and conformance with applicable guidance and direction. The  department added that NTCSS “has proven to be the right solution to meet  the Navy’s strategic business and technological needs,” and that sound  program management practices are in place and improving.
Neither DOD’s comment about our overall findings nor its claims about  NTCSS being the right solution and being effectively managed are  adequately supported, as evidenced by the numerous factual instances that  we site in the report where the Navy did not comply with either DOD  acquisition policies and guidelines or industry best practices. Specifically,  the report shows that the program’s latest economic analysis did not  provide the Navy a reliable basis upon which to make investment  decisions. For example, the analysis did not include measurable,  quantifiable benefits for each alternative, and the cost estimates did not  meet six of the nine criteria associated with reliable cost estimates. The  analysis also was not independently reviewed in accordance with DOD  guidance and the Navy had yet to demonstrate that already deployed  NTCSS Optimized applications are producing expected benefits. We  appropriately concluded that the Navy does not know whether the program  as defined is the right solution to meet the Navy’s strategic business and  technological needs.
With respect to our recommendations, DOD fully concurred with two of the  recommendations and partially concurred with the remaining five  recommendations. The five areas of disagreement, DOD’s basis for its  disagreement, and our response to DOD’s position follow.
First, DOD stated that it does not see merit in conducting a formal  economic analysis for the NTCSS program that would address all viable  alternatives because, at this late stage, NTCSS  is  a “very mature program,”  and the final application (OOMA) is about to be fielded. Further, DOD said  it saw no merit in seeking Office of Program Analysis and Evaluation  (PA&E) review of the economic analysis. Instead, it said that it will  “coordinate” with PA&E in analyzing the relationship of NTCSS with other  programs that may provide similar functionality and “brief the results” to  selected stakeholders.
We do not agree that NTCSS is a “very mature program.” In particular, the  Navy still plans to spend in fiscal years 2006 through 2009 an additional  $348 million, which is approximately one-third of what has been spent on  the program to date. Further, there is no evidence to support the claim that  the OOMA application is about to be fielded. OOMA has failed operational  testing twice and is not yet fully developed or tested despite the Navy’s  initial plan to field it in 2001. In addition, the Navy’s stated intention to  develop an economic analysis for OOMA only and then, separately, prepare  an “analysis to determine the relationship” of NTCSS and other alternative  programs is not consistent with guidance and best practice, which  advocate basing such analyses on the full scope of the planned investment.  In addition, the proposal to limit key stakeholders’ involvement in  developing the economic justification to “coordinating” and “briefing  would be inappropriate.” These stakeholders have specific expertise and  roles relative to economically justifying system investments that should be  exploited. Until it conducts a complete and disciplined analysis of the  entire NTCSS program (reviewed and approved by PA&E and the Naval  Cost Analysis Division) and provides this analysis to all key stakeholders,  the Navy’s investment decisions will continue to be made without complete  and reliable data.
Second, the department stated that further deployment of NTCSS should  not be limited at this time. Nevertheless, it stated that it will use the results  of the analysis referred to above that depicts NTCSS’s relationship with  other programs to provide appropriate direction to the program. We do not  agree that development should not be limited and would note that the  department’s own comment acknowledges the need to decide on an  appropriate direction for the program. In our view, prudent use of taxpayer  resources warrant both a reliable economic analysis that can be used to  inform any decision on this direction and fiscal restraint to investing until  an informed decision can be made.
Third, DOD said that the Navy does not need to be directed to ensure that  effective program management activities are implemented because it is  continuously improving program management activities. Further, DOD  stated that, although it is not required to implement an earned value  management system because the individual projects do not meet the dollar  threshold and there are no formal contract deliverables, it is nevertheless  adhering to the 32 earned value management criteria set forth in applicable  standards. The department added that it intends to have the Navy Inspector  General conduct a separate study to further ensure that the program is  using the best program management activities.
We do not agree with these comments. In particular, neither during our  review nor in its comments did the Navy provide evidence that it has  implemented effective program management activities or has  improvements under way. As we state in our report, neither the  decomposition of the program into small, fiscal year-based projects nor the  absence of a contractual relationship is a valid reason for not effectively  implementing earned value management. Further, the Navy’s earned value  management self-assessment showed that it had not adhered to 17 of the 32  earned value management standards. Without reliable, timely, and  auditable earned value management data, the program office cannot  adequately manage technical, cost, and schedule risks and problems.
Fourth, the department stated that key stakeholders of the NTCSS program  have the necessary people, processes, and tools to effectively execute their  respective roles and responsibilities, noting in particular that the central  design agency has demonstrated its competency and capability and was  certified as SW-CMM maturity level 3. Nevertheless, the department agreed  to address this recommendation.
We support the Navy’s stated commitment to address this  recommendation. In addition, we would note that DOD’s comment that  stakeholders have the resources they need is not consistent with  statements from stakeholders during our review who said that there were  manpower and resource shortfalls that affected the oversight and  execution of program activities. Further, despite the Navy’s statement that  the central design agency achieved SW-CMM maturity level 3, no  documentation supporting this statement, such as appraisal reports, were  provided. Furthermore, Navy officials told us that the central design agency  did not have a development testing lab and was therefore unable to  effectively execute testing activities.
Fifth, DOD stated that it is “premature” to reestablish the DOD Chief  Information Officer as the milestone decision authority as NTCSS  development is over 95 percent complete. Instead, it stated that existing  oversight entities would ensure that effective program management and  reporting was occurring.
We do not agree that elevating the milestone decision authority at this time  is premature based on the statement that the program is 95 percent  complete. For programs that have not been developed using industry best  practices and technical and management discipline, which is the case for  NTCSS, such claims of being essentially complete have historically proven  inaccurate because they are not grounded in reliable performance data.  Moreover, the Navy still plans to spend $348 million on NTCSS over the  next three fiscal years. Finally, as stated in our report, the current  milestone decision authority has allowed the program to operate  unchecked although a major application has repeatedly failed operational  testing, and another application was cancelled.
We are sending copies of this report to interested congressional  committees; the Director, Office of Management and Budget; the Secretary  of Defense; the Deputy Secretary of Defense; the Under Secretary of  Defense for Acquisition, Technology and Logistics; the Under Secretary of  Defense (Comptroller); the Assistant Secretary of Defense (Networks and  Information Integration)/Chief Information Officer; the Deputy Assistant  Secretary of the Navy for Command, Control, Communication, Computers  and Intelligence, and Space; the Program Executive Office for Command,  Control, Communication, Computers and Intelligence, and Space within  the Space and Naval Warfare Systems Command; the Department of the  Navy Chief Information Officer; and the Office of the Chief of Naval  Operations for Material Readiness and Logistics Operations. This report  will also be available at no charge on our Web site at http://www.gao.gov.
If you or your staff have any questions on matters discussed in this report,  please contact me at (202) 512-3439 or hiter@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to this  report are listed in appendix V.
section_title: Objective, Scope, and Methodology
paragraphs: Our objective was to determine whether the Naval Tactical Command  Support System (NTCSS) is being managed according to important aspects  of the Department of Defense’s (DOD) acquisition policies and guidance, as  well as other relevant acquisition management best practices. To  accomplish our objective, we focused on the program’s (1) economic  justification; (2) architectural alignment; (3) program management, namely  progress measurement and reporting, funding disclosure, and oversight;  and (4) key system development activities, namely requirements  development and management, test management, and system maturity  indicators. For requirements and test management, we focused on the one  NTCSS application that is currently being acquired, known as the  Optimized Organizational Maintenance Activity (OOMA).
To determine whether the Navy has economically justified its investment in  NTCSS, we reviewed the latest economic analysis to determine the basis  for the cost and benefit estimates and net present value calculations. This  included evaluating the analysis against DOD and Office of Management  and Budget (OMB) guidance, as well as relevant best practices. It also  included interviewing program officials, including the Assistant Program  Manager; the office of the Deputy Assistant Secretary of the Navy for  Command, Control, Communication, Computers and Intelligence, and  Space; the Office of Program Analysis and Evaluation; and the Naval Cost  Analysis Division as to their respective roles, responsibilities, and actual  efforts in developing and/or reviewing the economic analysis. In addition,  we also interviewed the Assistant Program Manager and the office of the  Deputy Assistant Secretary of the Navy for Command, Control,  Communication, Computers and Intelligence, and Space about the purpose  and use of the analysis for managing the Navy’s investment in the NTCSS  program including the extent to which measures and metrics showed that  projected benefits in the economic analysis were actually being realized.
To determine whether the Navy has aligned NTCSS to either the DOD  business enterprise architecture or a Navy architecture, we relied on our  prior reports addressing DOD and Navy architecture development and  implementation efforts, a memo and analysis results on NTCSS’s  compliance with the business enterprise architecture, and documents on  the Navy’s architecture efforts. We also interviewed Navy officials from the  program office; the office of the Deputy Assistant Secretary of the Navy for  Command, Control, Communication, Computers and Intelligence, and  Space; the office of the Navy Research, Development, and Acquisition  Chief Engineer; and the Office of the Assistant Secretary of Defense for  Networks and Information Integration/Chief Information Officer about  DOD and Navy architecture efforts and NTCSS’s alignment to them.
To determine whether the Navy was effectively measuring, reporting, and  overseeing the program, we did the following:  We first asked the central design agency to self-assess their satisfaction  of 32 best practice criteria regarding their earned value management  system. Using the results of their self-assessment to target our analysis,  we then assessed those aspects of the earned value management system  the self-assessment reported as meeting the criteria, by comparing the  documentation with relevant DOD guidance and best practices. We  selected these two projects as case studies to determine the degree to  which earned value management was being implemented. The two  projects selected were (1) 2004 OOMA software project and (2) 2004  NTCSS hardware installation and integration (for both OOMA and  Optimized NTCSS). We selected these two because they were the  projects for which Navy provided us the most earned value management  related documentation. To understand the Navy’s reasons why they  were not performing certain elements of earned value management, we  interviewed officials including the Assistant Program Manager, and  officials at the central design agency in Norfolk and the in service  engineering agency in Charleston.
To assess reporting capabilities, we reviewed program documentation  such as Acquisition Program Baselines, program deviation reports, and  Defense Acquisition Executive Summary reports. We also reviewed  information and documentation on the Rapid Improvement Team pilot  Web site including a report that assesses the current health of the  program on a monthly basis and a report that address risks for different  projects within the program.
To assess compliance with budget policies and guidance, we compared  NTCSS budget documentation with DOD and Navy financial  management policies and guidance.
To assess oversight of the program, we interviewed the program  manager, milestone decision authority, functional sponsor, Navy Chief  Information Officer, and a representative of the program’s executive  steering committee.
To determine whether the Navy was effectively managing key system  development activities, namely requirements management, testing, and  system maturity indicators, we did the following:  To assess requirements development and management capabilities, we  reviewed program documentation such as the official list of  requirements and system specifications, and evaluated them against  relevant best practices for several characteristics including traceability  and prioritization. We attempted to trace requirements to both higher  level documents and lower level specifications. We also attended the  NTCSS Forum where requirements were gathered and discussed. We  interviewed Navy officials such as the Assistant Program Manager,  Commanding Officer and Executive Director of the central design  agency, and the OOMA Functional Manager to discuss their roles and  responsibilities for developing and managing requirements.
To assess test management, we reviewed program documentation such  as the test and evaluation master plan, test plans, test reports, and  guidance. We then compared these documents with DOD guidance and  best practices and focused on the effectiveness of developmental testing  and the adequacy of developmental testing documentation. Our review  also included an audit report prepared by the Naval Audit Service and a  test report prepared by Navy’s independent operational test  organization. We interviewed Navy officials such as the Assistant  Program Manager, Commanding Officer and Executive Director of the  central design agency, OOMA Functional Manager, and an official in the  office of the Director of Navy Test and Evaluation and Technology  Requirements to discuss their roles and responsibilities for test  management.
We did not independently validate information on the program’s cost and  budget or the number of trouble reports and change proposals.
We conducted our work at DOD headquarters in Arlington, Virginia; at  Space and Naval Warfare Center, San Diego, California; Space and Naval  Warfare Systems Center, Norfolk, Virginia; and Naval Air Systems  Command in Patuxent River, Maryland. We performed our work from  September 2004 through November 2005 in accordance with generally  accepted government auditing standards.
section_title: Trouble Reports and Change Proposals Assessment
paragraphs: One indicator of system quality, and thus the effectiveness of the  development activities used to produce system products, is the volume and  significance of system problems and change proposals. For the Naval  Tactical Command Support System (NTCSS), trouble reports are prepared  to document system defects, and change proposals are prepared to  introduce additional system functionality. Priority levels are assigned to  trouble reports and change proposals, with 1 being the most critical and 5  being the least critical. Table 11 defines the 5 priority levels.
Available data on the number and significance of open trouble reports and  change proposals over the last 2 years do not demonstrate that NTCSS  overall is a high-quality system that is delivering promised or expected  capabilities. Specifically, the data shows that hundreds of open (yet to be  resolved) trouble reports and change proposals have continued to affect  the system.
subsections: section_title: Trouble Reports
paragraphs: The total number of NTCSS priority 1, 2, and 3 trouble reports have stayed  about the same over the last 2 years—totaling about 700. Of this total,  NTCSS priority 1 and 2 trouble reports have decreased by 117, with priority  1 trouble reports being virtually eliminated. While this is movement in a  positive direction, about 300 priority 2 trouble reports still remain open and  these by definition are adversely affecting accomplishment of an  operational or mission-essential capability. (See figs. 1 and 2.)
Further, open priority 3 trouble reports have increased during this time to  about 250 and, given that priority 3s require work-arounds, they decrease  system capability and performance. Neither the number of priority 2  trouble reports, which continue to be in the hundreds, nor the upward  trend in priority 3 trouble reports are indicative of a maturing, high-quality  system. (See fig. 3.)
With respect to the OOMA application in particular, the trend in the volume  of significant trouble reports shows that this application is particularly  problematic. Specifically, while priority 1 OOMA open trouble reports have  been virtually eliminated, the number of open priority 2 OOMA trouble  reports has risen significantly from 12 to 90 in about the last 2 years. (See  fig. 4.)
Moreover, the number of open OOMA priority 3 trouble reports has not  significantly declined over the last 2 years, with these remaining at roughly  160. (See fig. 5.)
section_title: Change Proposals
paragraphs: The picture for NTCSS change proposals is similar to that for trouble  reports. Specifically, the total number of open NTCSS priority 1, 2, and 3  change proposals has increased over the last 2 years—going from about  325 to 425. Of this total, NTCSS priority 2 change proposals have increased  by 72, with 247 priority 2 proposals still being open. (See figs. 6 and 7.)
Further, NTCSS priority 3 change proposals have increased during this time  to about 81, and given that priority 3 change proposals require current  work-arounds, this is not a positive trend. (See fig. 8.)
With respect to OOMA specifically, the number of open priority 2 change  proposals has risen slightly from 7 to 12. (See fig. 9.) Similarly, the number  of open priority 3 change proposals has also increased somewhat from 78  to 97. (See fig. 10.) While the number of priority 2 change proposals is not  large, the trend in these, as well as the trend in the more significant number  of priority 3 change proposals, is not consistent with those of a maturing  system.
section_title: Earned Value Management Assessment
paragraphs: Earned value management (EVM) guidance was developed by the  American National Standards Institute/Electronic Industries Alliance. This  guidance identifies 32 criteria that reliable EVM systems should meet. The  32 criteria are organized into the following five categories:  Organization: Activities that define the scope of the effort and  assign responsibilities for the work;  Planning and budgeting: Activities for planning, scheduling,  budgeting, and authorizing the work;  Accounting: Activities to accumulate the costs of work and material  needed to complete the work;  Analysis: Activities to compare budgeted, performed, and actual  costs; analyze variances; and develop estimates of final costs; and  Revisions and data maintenance: Activities to incorporate internal  and external changes to the scheduled, budgeted, and authorized  work.
NTCSS central design agency (CDA) officials provided a self-assessment of  their compliance with each of the criteria, reporting that they met 15 of the  32 criteria (see table 12). Using the results of their self-assessment to target  our analysis, we then assessed those aspects of the EVM system the self- assessment reported as meeting the criteria, by comparing the  documentation with relevant Department of Defense (DOD) guidance and  best practices. Our assessment indicates that the NTCSS program satisfied  two, and partially satisfied one, of the 32 criteria (see table 12).
section_title: Comments from the Department of Defense
paragraphs: The following are GAO’s comments on the Department of Defense’s letter  dated November 23, 2005.
subsections: section_title: GAO Comments
paragraphs: 1. See the Agency Comments and Our Evaluation section of this report. 2. We disagree. Our report contains numerous instances where the Navy  did not comply with either DOD acquisition policies and guidelines or  industry best practices, in the areas of (1) economic justification; (2)  architectural alignment; (3) project management, including progress  measurement and reporting, funding disclosure, and oversight  activities; and (4) system development, including requirements  management and testing. Moreover, the Navy has not provided any  evidence to demonstrate that our report is incorrect with respect to the  level of program discipline and conformance with applicable policy and  guidance in the areas that we reviewed.  3. We disagree. Knowing that NTCSS is the right solution to meet the  Navy’s strategic business and technological needs would require that a  frame of reference articulating these needs be available as a point of  comparison. Such a frame of reference is an enterprise architecture.  However, the Navy stated the system was defined and implemented  without a complete and formal enterprise architecture. Our experience  with federal agencies has shown that investing in an information  technology solution without defining the solution in the context of an  architecture often results in systems that are duplicative, not well  integrated, and unnecessarily costly to maintain and interface. In  addition, in February 2005, key program stakeholders and  representatives from user organizations questioned whether NTCSS as  defined was the right solution to cost effectively meet users’ needs. At  that time, program officials stated their intent to develop a new  economic analysis to gather the information needed to determine  whether to continue investing in NTCSS. In November 2005, program  officials told us that they no longer planned to develop this economic  analysis. Without a well-defined architecture and a reliable economic  analysis, the Navy cannot be sure that NTCSS is the right solution.  4. See comment 2.  5. We acknowledge DOD’s comment but would note that it is contrary to  statements made to us during the audit. For example, officials with the  milestone decision authority stated that, due to staffing reductions, the  office was unable to fully perform oversight activities and has had to  delegate completion of these activities. Also, Naval Cost Analysis  Division officials stated that they only review cost estimates that are  prepared for milestone reviews because staffing limitations do not  permit them to review all cost estimates. Further, Navy officials stated  that the central design agency was unable to effectively execute testing  activities because it did not have a development testing lab.  6. We disagree with this approach because its scope is narrower than our  recommendation. Specifically, we recommended that the Navy develop  a reliable economic analysis of the NTCSS program that includes all  viable alternatives, including the Navy’s Enterprise Resource Planning  program. DOD acquisition policy and guidance provide detailed  instructions on how economic analyses should be performed to obtain  information that is critical for decisions regarding investments of  scarce resources. Without such information, Navy risks that its  continued investment in the system may not be justified.  7. We disagree. With respect to the statement that NTCSS is a “very  mature program,” NTCSS has been under development for about 10  years at a cost of about $1.1 billion, and the Navy plans to spend an  additional $348 million between fiscal years 2006 and 2009. Further, as  appendix II of our report shows, there are hundreds of open trouble  reports and change proposals that need to be addressed before the  system can deliver promised or expected capabilities. In addition,  should the OOMA application pass operational testing and be fielded,  there are over 200 sites where the necessary hardware must be  installed and related training must occur. These two efforts will require  a significant investment of time and resources, and it is therefore  critical that the Navy ensure that NTCSS is the proper system before  investing additional funds. With respect to the statement that “the final  application is about to fielded,” there is no evidence to support this.  Since its originally planned fielding date of 2001, OOMA has failed  operational testing twice, and the application is still under  development. Therefore, it is premature to assert that the application  will soon pass developmental and follow-on operational testing. 8. See comment 6. Further, we disagree with the proposal to limit key  stakeholders’ involvement in developing the economic justification to  “coordinating” and “briefing.” These stakeholders have specific  expertise and roles relative to economically justifying system  investments that should be exploited. Until it conducts a complete and  disciplined analysis of the entire NTCSS program (reviewed and  approved by the Office of Program Analysis and Evaluation and the  Naval Cost Analysis Division) and provides this analysis to all key  stakeholders, the Navy’s investment decisions will continue to be made  without complete and reliable data. 9. We disagree. As discussed in our report, the 2004 economic analysis did  not adhere to five of eight criteria elements contained in the Office of  Management and Budget Circulars A-94 and A-11.  10. We disagree. The 2004 economic analysis that the Navy provided us  focused on three fielding alternatives for the NTCSS program, not just  the OOMA application. The Navy did not provide a 2004 economic  analysis for just OOMA as the final NTCSS application.  11. We disagree. As stated in our report, officials from the Office of  Program Analysis and Evaluation and the Naval Cost Analysis Division  told us that they did not review the 2004 NTCSS economic analysis.  12. See comment 10. 13. We agree that the Navy ERP program did not exist when the original  NTCSS analysis of alternatives was conducted. However, the Navy ERP  program was initiated in 1998 and therefore did exist when the Navy  conducted subsequent analysis of alternatives. 14. See comment 9.  15. We do not question whether these annual reviews occurred and what  resulted from them. However, the point in our report is that NTCSS has  not been defined and developed in the context of a DOD or Navy  enterprise architecture because a well-defined version of either has not  existed to guide and constrain the program. As a result, meaningful  analysis showing how NTCSS aligns to evolving DOD and Navy  architecture efforts could not be produced. This means that the Navy  does not have a sufficient basis for knowing if NTCSS, as defined,  properly fits within the context of future DOD and Navy business  operational and technological environments. 16. We disagree. Our recommendation to limit further deployment of  NTCSS is a way of ensuring that the Navy takes a “strategic pause”  while it takes the time to ensure that decisions regarding future  investment are made using reliable information, which our report  shows has not historically been the case. As long as the Navy is not  appropriately limiting work on NTCSS, it is continuing to invest  resources without having justified doing so.  17. See comment 6.  18. See comment 2. 19. We disagree. As we state in our report, neither the decomposition of the  program into small, fiscal year-based projects nor the absence of a  contractual relationship is a valid reason for not effectively  implementing earned value management. Without reliable, timely, and  auditable earned value management data, the program office cannot  adequately manage technical, cost, and schedule risks and problems.  20. We disagree. The Navy’s own self-assessment of compliance with the 32  criteria, detailed in appendix III of our report, showed that 17 of these  criteria were not being satisfied. Further, our assessment showed that  the Navy did not satisfy 29 of the 32 criteria, and program officials did  not provide any evidence to refute the results of our assessment.  21. The Navy did not provide us with a copy of the CDA Software  Measurement Plan.  22. See comment 5. Further, the Navy’s position that “key stakeholders of  the NTCSS program do, in fact, have the people, processes and tools to  effectively execute their respective roles and responsibilities,” is not  consistent with its comment that this area will be part of a planned  review.  23. We disagree. Although the Navy states that the program is 95 percent  complete, it still plans to spend $348 million over the next three fiscal  years, which is approximately 32 percent of what has been spent on the  program to date. In addition, because the Navy lacks disciplined  acquisition management practices, as discussed in our report, including  earned value management, we question how it is able to reliably  determine what percentage of the work has been completed and the  percentage that remains to be done. As stated in our report, the current  milestone decision authority has allowed the program to proceed while  a major application repeatedly failed operational testing, and another  application was cancelled. In addition, the Navy stated its intent to  revisit the need to change milestone decision authority.
section_title: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Cynthia Jackson, Assistant  Director; Harold J. Brumm; Calvin L. H. Chang; Jennifer K. Echard; Joanne  Fiorino; Neelaxi Lakhmani; Freda Paintsil; Jamelyn Payan; Karen Richey;  Dr. Karl Seifert; Andrea Smith; and Dr. Rona B. Stillman made key  contributions to this report.