DOD Business Systems Modernization: Navy ERP Adherence to Best Business Practices Critical to Avoid Past Failures
Published: 2005-09-29T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Defense's (DOD) difficulty in implementing business systems that are efficient and effective continues despite the billions of dollars that it invests each year. For a decade now--since 1995--we have designated DOD's business systems modernization as "high-risk." GAO was asked to (1) provide a historical perspective on the planning and costs of the Navy's four Enterprise Resource Planning (ERP) pilot projects, and the decision to merge them into one program; (2) determine if the Navy has identified lessons from the pilots, how the lessons are being used, and challenges that remain; and (3) determine if there are additional best business practices that could be used to improve management oversight of the Navy ERP.
section_title: What GAO Found
paragraphs: The Navy invested approximately $1 billion in four ERP pilots without marked improvement in its day-to-day operations. The planning for the pilots started in 1998, with implementation beginning in fiscal year 2000. The four pilots were limited in scope and were not intended to be corporate solutions for any of the Navy's long-standing financial and business management problems. Furthermore, because of the various inconsistencies in the design and implementation of the pilots, they were not interoperable, even though they performed many of the same business functions. In short, the efforts were failures and $1 billion was largely wasted. Because the pilots would not meet its overall requirements, the Navy decided to start over and develop a new ERP system, under the leadership of a central program office. Using the lessons learned from the pilots, the current Navy ERP program office has so far been committed to the disciplined processes necessary to manage this effort. GAO found that, unlike other systems projects it has reviewed at DOD and other agencies, Navy ERP management is following an effective process for identifying and documenting requirements. The strong emphasis on requirements management, which was lacking in the previous efforts, is critical since requirements represent the essential blueprint that system developers and program managers use to design, develop, test, and implement a system and are key factors in projects that are considered successful. While the Navy ERP has the potential to address some of the Navy's financial management weaknesses, as currently planned, it will not provide an all-inclusive end-to-end corporate solution for the Navy. For example, the current scope of the ERP does not include the activities of the aviation and shipyard depots. Further, there are still significant challenges and risks ahead as the project moves forward, such as developing and implementing 44 system interfaces with other Navy and DOD systems and converting data from legacy systems into the ERP system. The project is in its early phases, with a current estimated completion date of 2011 at an estimated cost of $800 million. These estimates are subject to, and very likely will, change. Broader challenges, such as alignment with DOD's business enterprise architecture, which is not fully defined, also present a significant risk. Given DOD's past inability to implement business systems that provide the promised capability, continued close management oversight--by the Navy and DOD--will be critical. In this regard, the Navy does not have in place the structure to capture quantitative data that can be used to assess the effectiveness of the overall effort. Also, the Navy has not established an independent verification and validation (IV&V) function. Rather, the Navy is using in-house subject matter experts and others within the project. Industry best practices indicate that the IV&V activity should be independent of the project and report directly to agency management in order to provide added assurance that reported results on the project's status are unbiased.

section_title: Letter
paragraphs: The Department of Defense (DOD) has historically been unable to develop  and implement business systems on time, within budget, and with the  promised capability.  As noted in our recent report, this difficulty  continues despite the billions of dollars that DOD spends each year to  operate, maintain, and modernize its currently reported 4,150 business  systems.  For fiscal year 2005, the department requested $13 billion for its  existing business systems environment.  For a decade now—since 1995— we have designated DOD’s financial management and business systems  modernization as “high-risk.”  In fact, of the 25 areas on GAO’s  governmentwide “high-risk” list, 8 are DOD specific program areas, and the  department shares responsibility for 6 other high-risk areas that are  governmentwide in scope.
DOD has recognized the importance of transforming its business  operations and systems to make them more efficient and effective in  support of the department’s defense mission.  A critical aspect of the  department’s transformation effort will be its ability to effectively  implement business systems on time, within budget, and with the promised  capability.
This report responds to your request for information on DOD’s  management of selected facets of its business modernization efforts that  are intended to enhance the department’s reporting on its results of  operation. As agreed with your offices, we selected the Department of the  Navy’s (Navy) Enterprise Resource Planning (ERP) program, initially  created as four pilot projects and now being pursued as a consolidated  effort, to review and determine if it will help to resolve some of the Navy’s  long-standing financial and business management problems.  The Navy  expects the ERP to be fully operational by fiscal year 2011.  It is currently  estimated that for fiscal years 2004-2011, the program will cost  approximately $800 million.  When fully operational, the Navy reports that  the ERP will manage an estimated 80 percent of its appropriated funds.   Our objectives were to (1) provide a historical perspective on the planning  and costs of the Navy’s four ERP pilot projects, and the decision to merge  them into one program; (2) determine if the Navy has identified lessons  from the pilots, how the lessons are being used, and the challenges that  remain; and (3) determine if there are additional best business practices  that could be used to improve management oversight of the Navy ERP.
To obtain a historical perspective on the planning and costs of the Navy’s  four ERP pilot projects, and the decision to merge them into one program,  we reviewed DOD’s budget justification materials and other background  information on the four pilot projects and met with program management  and DOD Chief Information Officer (CIO) officials.  Additionally, we  reviewed project documentation provided by DOD and held discussions  with program management officials related to two key processes— requirements management and testing—to determine if these key aspects  of program management were being performed and if the system is being  designed to help address some of Navy’s long-standing management  problems.  Further, we reviewed relevant industry standards and best  practices, and interviewed and requested documentation from the Navy  ERP to determine whether there are additional best business practices that  could appropriately be used to improve management oversight of the ERP.   Given that this effort is still in the early stages of development, we did not  evaluate all best practices.  Rather, we concentrated on those that could  have an immediate impact in improving management’s oversight.
Our work was performed from August 2004 through June 2005 in  accordance with U.S. generally accepted government auditing standards.   Details on our scope and methodology are included in appendix I.  We  requested comments on a draft of this report from the Secretary of Defense  or his designee.  Written comments from the Deputy Under Secretary of  Defense (Financial Management) and the Deputy Under Secretary of  Defense (Business Transformation) are reprinted in appendix II.
subsections: section_title: Background
paragraphs: The Navy, with reported assets totaling $321 billion in fiscal year 2004,   would be ranked among the largest corporations in the world if it were a  private sector entity.  According to the Navy, based upon the reported value  of its assets, it would be ranked among the 15 largest corporations on the  Fortune 500 list.  Additionally, in fiscal year 2004 the Navy reported that its  inventory was valued at almost $73 billion and that it held property, plant,  and equipment with a reported value of almost $156 billion.  Furthermore,  the Navy reported for fiscal year 2004 that its operations involved total  liabilities of $38 billion, that its operations had a net cost of $130 billion,  and that it employed approximately 870,000 military and civilian  personnel—including reserve components.
The primary mission of the Navy is to control and maintain freedom of the  seas, performing an assortment of interrelated and interdependent  business functions to support its military mission with service members  and civilian personnel in geographically dispersed locations throughout the  world.  To support its military mission and perform its business functions,  the Navy requested for fiscal year 2005 almost $3.5 billion for the operation,  maintenance, and modernization of its business systems and related  infrastructure—the most of all the DOD components—or about 27 percent  of the total $13 billion DOD fiscal year 2005 business systems budget  request.  Of the 4,150 reported DOD business systems, the Navy holds the  largest inventory of business systems—with 2,353 reported systems or 57  percent of DOD’s reported inventory of business systems.
The Secretary of Defense recognized that the department’s business  operations and systems have not effectively worked together to provide  reliable information to make the most effective business decisions.  He  challenged each military service to transform its business operations to  support DOD’s warfighting capabilities and initiated the Business  Management Modernization Program (BMMP) in July 2001.  Further, the  Assistant Secretary of the Navy for Financial Management and Comptroller  (Navy Comptroller) testified that transforming the Navy’s business  processes, while concurrently supporting the Global War on Terrorism, is a  formidable but essential task.  He stated that the goal of the  transformation is to “establish a culture and sound business processes that  produce high-quality financial information for decision making.”  One of  the primary elements of the Navy’s business transformation strategy is the  Navy ERP.
subsections: section_title: Recently Reported Business and Financial Weaknesses at the Navy
paragraphs: The need for business processes and systems transformation to provide  management with timely information to make important business decisions  is clear.  However, none of the military services, including the Navy, have  passed the scrutiny of an independent financial audit.  Obtaining a clean  (unqualified) financial audit opinion is a basic prescription for any well- managed organization, as recognized by the President’s Management  Agenda.  For fiscal year 2004, the DOD Inspector General issued a  disclaimer on the Navy’s financial statements—Navy’s General Fund and  Working Capital Fund—citing eight material weaknesses and six material  weaknesses respectively, in internal control and noncompliance with the  Federal Financial Management Integrity Act of 1996 (FFMIA).  The  inability to obtain a clean financial audit opinion is the result of  weaknesses in the Navy’s financial management and related business  processes and systems.  Most importantly, the Navy’s pervasive  weaknesses have (1) resulted in a lack of reliable information to make  sound decisions and report on the status of activities, including  accountability of assets, through financial and other reports to the Navy  and DOD management and the Congress; (2) hindered its operational  efficiency; (3) adversely affected mission performance; and (4) left the  Navy and DOD vulnerable to fraud, waste, and abuse, as the following  examples illustrate.
The Navy’s lack of detailed cost information hinders its ability to  monitor programs and analyze the cost of its activities. We reported  that the Navy lacked the detailed cost and inventory data needed to  assess its needs, evaluate spending patterns, and leverage its  telecommunications buying power.  As a result, at the sites we  reviewed, the Navy paid for telecommunications services it no longer  required, paid too much for services it used, and paid for potentially  fraudulent or abusive long-distance charges.  In one instance, we found  that DOD paid over $5,000 in charges for one card that was used to place  189 calls in one 24-hour period from 12 different cities to 12 different  countries.
Ineffective controls over Navy foreign military sales using blanket  purchase orders placed classified and controlled spare parts at risk of  being shipped to foreign countries that may not be eligible to receive  them. For example, we identified instances in which Navy country  managers (1) overrode the system to release classified parts under  blanket purchase orders without filing required documentation  justifying the release; and (2) substituted classified parts for parts  ordered under blanket purchase orders, bypassing the control-edit  function of the system designed to check a country’s eligibility to  receive the parts.
The Naval Inventory Control Point and its repair contractors have not  followed DOD and Navy procedures intended to provide the  accountability for and visibility of inventory shipped to Navy repair  contractors. Specifically, Navy repair contractors are not routinely  acknowledging receipt of government-furnished material received from  the Navy. A DOD procedure requires repair contractors to acknowledge  receipt of government-furnished material that has been shipped to them  from the Navy’s supply system.  However, Naval Inventory Control Point  officials are not requiring repair contractors to acknowledge receipt of  these materials.  By not requiring repair contractors to acknowledge  receipt of government-furnished material, the Naval Inventory Control  Point has also departed from the procedure to follow up with the  contractor within 45 days when the contractor fails to confirm receipt  for an item. Without material receipt notification, the Naval Inventory  Control Point cannot be assured that its repair contractors have  received the shipped material. This failure to acknowledge receipt of  material shipped to repair contractors can potentially impair the Navy’s  ability to account for shipments leading to possible fraud, waste, or  abuse.
A limited Naval Audit Service audit revealed that 53 of 118 erroneous  payment transactions, valued at more than $990,000, occurred because  Navy certifying officials did not ensure accurate information was  submitted to the Defense Finance and Accounting Service (DFAS) prior  to authorizing payment. In addition, certifying officials submitted  invoices to DFAS authorizing payment more than once for the same  transaction.
Brief Overview of Navy ERP To address the need for business operations reform, in fiscal year 1998 the  Navy established an executive committee responsible for creating a  “Revolution in Business Affairs” to begin looking at transforming business  affairs and identifying areas of opportunity for change.  This committee, in  turn, set up a number of working groups, including one called the  Commercial Business Practices (CBP) Working Group, which consisted  of representatives from financial management organizations across the  Navy.  This working group recommended that the Navy should use ERP as  a foundation for change and identified various ERP initiatives that were  already being developed or under consideration within the Navy.   Ultimately, the Navy approved the continuation of four of these initiatives,  using funds from existing resources from each of the sponsors (i.e.,  commands) to test the feasibility of ERP solutions within the Navy.  From  1998 to 2003, four different Navy commands began planning, developing,  and implementing four separate ERP pilot programs to address specific  business areas.  A CBP Executive Steering Group was created in December  1998 to monitor the pilot activities.
As the pilots progressed in their development and implementation, the  Navy identified issues that had to be addressed at a higher level than the  individual pilots, such as the integration between the pilots as well as with  other DOD systems, and decided that one program would provide a more  enterprisewide solution for the Navy.  In August 2002, the Assistant  Secretary of the Navy for Research, Development, and Acquisition  established a Navy-wide ERP program to “converge” the four ongoing  pilots into a single program.  This Navy-wide program is expected to  replace all four pilots by fiscal year 2008 and to be “fully operational” by  fiscal year 2011.  The Navy estimates that the ERP will manage about 80  percent of the Navy’s estimated appropriated funds—after excluding  appropriated funds for the Marine Corps and military personnel and pay.   Based on the Navy’s fiscal years 2006 to 2011 defense planning budget, the  Navy ERP will manage approximately $74 billion annually.
According to a Navy ERP official, while the Navy ERP would account for  the total appropriated amount, once transactions occur at the depots, such  as when a work order is prepared for the repair of an airplane part, the  respective systems at the depots will execute and maintain the detailed  transactions.  This accounts for about 2 percent, or approximately $1.6  billion, being executed and maintained in detail by the respective systems  at the aviation and shipyard depots—not by the Navy ERP.  The remaining  20 percent that the ERP will not manage comprises funds for the Navy  Installations Command, field support activity, and others.
section_title: Navy’s Pilot Projects Lacked Coordinated Management Oversight
paragraphs: Each of the Navy’s four ERP pilot projects was managed and funded by  different major commands within the Navy.  The pilots, costing over $1  billion in total, were limited in scope and were not intended to provide  corporate solutions to any of the Navy’s long-standing financial and  business management problems.  The lack of centralized management  oversight and control over all four pilots allowed the pilots to be developed  independently.  This resulted in four more DOD stovepiped systems that  could not operate with each other, even though each carried out many of  the same functions and were based on the same ERP commercial-off-the- shelf (COTS) software.  Moreover, due to the lack of high-level  departmentwide oversight from the start, the pilots were not required to go  through the same review process as other acquisition projects of similar  magnitude.
subsections: section_title: Pilots Developed Independently of Each Other
paragraphs: Four separate Navy organizations began their ERP pilot programs  independently of each other, at different times, and with separate funding.   All of the pilots implemented the same ERP COTS software, and each pilot  was small in scale—relative to the entire Navy.  For example, one of the  pilots, SMART, was responsible for managing the inventory items and  repair work associated with one type of engine, although the organization  that implemented SMART—the Naval Supply Systems Command— managed the inventory for several types of engines.  As of September 2004,  the Navy estimated that the total investment in these four pilots was  approximately $1 billion.  Table 1 summarizes each of the pilots, the  cognizant Navy organization, the business areas they address, and their  reported costs through September 2004.
Even after the pilots came under the purview of the CBP Executive  Steering Group in December 1998, they continued to be funded and  controlled by their respective organizations.  We have previously reported  that allowing systems to be funded and controlled by component  organizations has led to the proliferation of DOD’s business systems.   These four pilots are prime examples.  While there was an attempt made to  coordinate the pilots, ultimately each organization designed its ERP pilot to  accommodate its specific business needs.  The Navy recognized the need  for a working group that would focus on integration issues among the  pilots, especially because of the desire to eventually extend the pilot  programs beyond the pilot organizations to the entire Navy.  In this regard,  the Navy established the Horizontal Integration Team in June 1999,  consisting of representatives from all of the pilots to address this matter.   However, one Navy official described this team as more of a “loose  confederation” that had limited authority.  As a result, significant resources  have been invested that have not and will not result in corporate solutions  to any of the Navy’s long-standing business and financial management  problems.  This is evident as noted in the DOD Inspector General’s audit  reports of the Navy’s financial statements discussed above.
In addition to the lack of centralized funding and control, each of the pilots  configured the software differently, which, according to Navy ERP program  officials, caused integration and interoperability problems.  While each  pilot used the same COTS software package, the software offers a high  degree of flexibility in how similar business functions can be processed by  providing numerous configuration points.  According to the Navy, over 2.4  million configuration points exist within the software.  The pilots  configured the software differently from each other to accommodate  differences in the way they wanted to manage their functional area focus.   These differences were allowed even though they perform many of the  same types of business functions, such as financial management.  These  configuration differences include the levels of complexity in workflow  activities and the establishment of the organizational structure.  For  example, the primary work order managed by the NEMAIS pilot is an  intricate ship repair job, with numerous tasks and workers at many levels.
Other pilots had much simpler work order definitions, such as preparing a  budget document or procuring a single part for an engine.
Because of the various inconsistencies in the design and implementation,  the pilots were stovepiped and could not operate with each other, even  though they performed many of the same business functions.  Table 2  illustrates the similar business functions that are performed by more than  one pilot.
By definition, an ERP solution should integrate the financial and business  operations of an organization.  However, the lack of a coordinated effort  among the pilots led to a duplication of efforts and problems in  implementing many business functions and resulted in ERP solutions that  carry out redundant functions in different ways from one another.
The end result of all of the differences was a “system” that could not  successfully process transactions associated with the normal Navy  practices of moving ships and aircraft between fleets.  Another  configuration problem occurred because the pilots generally developed  custom roles for systems users.  Problems arose after the systems began  operating.  Some roles did not have the correct transactions assigned to  enable the users with that role to do their entire job correctly.  Further,  other roles violated the segregation-of-duties principle due to the  inappropriateness of roles assigned to individual users.
The pilots experienced other difficulties with respect to controlling the  scope and performance schedules due to the lack of disciplined  processes, such as requirements management.  For example, the pilots  did not identify in a disciplined manner the amount of work necessary to  achieve the originally specified capabilities—even as the end of testing  approached.  There were repeated contract cost-growth adjustments,  delays in delivery of many planned capabilities, and initial periods of  systems’ instabilities after the systems began operating.  All of these  problems have been shown as typical of the adverse effects normally  associated with projects that have not effectively implemented disciplined  processes.
section_title: Pilots Lacked Departmentwide Oversight
paragraphs: The Navy circumvented departmentwide policy by not designating the  pilots as major automated information systems acquisition programs.  DOD  policy in effect at the time stipulated that a system acquisition should be  designated as a major program if the estimated cost of the system exceeds  $32 million in a single year, $126 million in total program costs, or $378  million in total life-cycle costs, or if deemed of special interest by the DOD  Chief Information Officer (CIO).  According to the Naval Audit Service, all  four of the pilots should have been designated as major programs based on  their costs—which were estimated to be about $2.5 billion at the time—and  their significance to Navy’s operations.  More specifically, at the time of its  review, SMART’s total estimated costs for development, implementation,  and sustainment was over $1.3 billion—far exceeding the $378 million life- cycle cost threshold.  However, because Navy management considered  each of its ERP programs to be “pilots,” it did not designate the efforts as  major automated information systems acquisitions, thereby limiting  departmental oversight.
Consistent with the Clinger-Cohen Act of 1996, DOD acquisition guidance  requires that certain documentation be prepared at each milestone within  the system life cycle. This documentation is intended to provide relevant  information for management oversight and in making decisions as to  whether the investment of resources is cost beneficial. The Naval Audit  Service reported that a key missing document that should have been  prepared for each of the pilots was a mission needs statement.  A mission  needs statement was required early on in the acquisition process to  describe the projected mission needs of the user in the context of the  business need to be met.  The mission needs statement should also address  interoperability needs.  As noted by the Naval Audit Service, the result of  not designating the four ERP pilots as major programs was that program  managers did not prepare and obtain approval of this required document  before proceeding into the next acquisition phase.  In addition, the pilots  did not undergo mandatory integrated reviews that assess where to spend  limited resources departmentwide.  The DOD CIO is responsible for  overseeing major automated information systems and a program executive  office is required to be dedicated to executive management and not have  other command responsibilities.  However, because the pilots were not  designated major programs, the oversight was at the organizational level  that funded the pilots (i.e., command level).  Navy ERP officials stated that  at the beginning of the pilots, investment authority was dispersed  throughout the Navy and there was no established overall requirement  within the Navy to address systems from a centralized Navy enterprise  level.  The Navy ERP is now designated a major program under the  oversight of the DOD CIO.
section_title: Experience Has Shown the Effects of Not Effectively Implementing Disciplined Processes
paragraphs: The problems identified in the failed implementation of the four pilots are  indicative of a system program that did not adhere to the disciplined  processes.  The successful development and implementation of systems is  dependent on an organization’s ability to effectively implement best  practices, commonly referred to as disciplined processes, which are  essential to reduce the risks associated with these projects to acceptable  levels.  However, the inability to effectively implement the disciplined  processes necessary to reduce risks to acceptable levels does not mean  that an entity cannot put in place a viable system that is capable of meeting  its needs.  Nevertheless, history shows that the failure to effectively  implement disciplined processes and the necessary metrics to understand  the effectiveness of processes implemented increases the risk that a given  system will not meet its cost, schedule, and performance objectives.
In past reports we have highlighted the impact of not effectively  implementing the disciplined processes.  These results are consistent with  those experienced by the private sector.  More specifically:    In April 2003, we reported that NASA had not implemented an effective  requirements management process and that these requirement  management problems adversely affected its testing activities. We also  noted that because of the testing inadequacies, significant defects later  surfaced in the production system.  In May 2004, we reported that  NASA’s new financial management system, which was fully deployed in  June 2003 as called for in the project schedule, still did not address  many of the agency’s most challenging external reporting issues, such as  external reporting problems related to property accounting and  budgetary accounting.  The system continues to be unable to produce  reliable financial statements.
In May 2004, we reported that the Army’s initial deployments for its  Logistics Modernization Program (LMP) did not operate as intended and  experienced significant operational difficulties.  In large part, these  operational problems were due to the Army not effectively  implementing the disciplined processes that are necessary to manage  the development and implementation of the systems in the areas of  requirements management and testing. The Army program officials have  acknowledged that the problems experienced in the initial deployment  of LMP could be attributed to requirements and testing. Subsequently, in  June 2005, we reported that the Army still had not put into place  effective management control and processes to help ensure that the  problems that have been identified since LMP became operational in  July 2003 are resolved in an efficient and effective manner.  The Army’s  inability to effectively implement the disciplined processes provides it  with little assurance that (1) system problems experienced during the  initial deployment that caused the delay of future deployments have  been corrected and (2) LMP is capable of providing the promised system  functionality. The failure to resolve these problems will continue to  impede operations at Tobyhanna Army Depot, and future deployment  locations can expect to experience similar significant disruptions in  their operations, as well as having a system that is unable to produce  reliable and accurate financial and logistics data.
We reported in February 2005 that DOD had not effectively managed  important aspects of the requirements for the Defense Integrated  Military Human Resources System, which is to be an integrated  personnel and pay system standardized across all military components.   For example, DOD had not obtained user acceptance of the detailed  requirements nor had it ensured that the detailed requirements were  complete and understandable.  Based on GAO’s review of a random  sample of the requirements documentation, about 77 percent of the  detailed requirements were difficult to understand.
The problems experienced by DOD and other agencies are illustrative of  the types of problems that can result when disciplined processes are not  properly implemented. The four Navy pilots provide yet another example.   As discussed previously, because the pilots were four stovepiped efforts,  lacking centralized management and oversight, the Navy had to start over  when it decided to proceed with the current ERP effort after investing  about $1 billion.  Figure 1 shows how organizations that do not effectively  implement disciplined processes lose the productive benefits of their  efforts as a project continues through its development and implementation  cycle.  Although undisciplined projects show a great deal of productive  work at the beginning of the project, the rework associated with defects  begins to consume more and more resources.  In response, processes are  adopted in the hopes of managing what later turns out to be, in reality,  unproductive work.  However, generally these processes are “too little, too  late,” and rework begins to consume more and more resources because the  adequate foundations for building the systems were not done or not done  adequately.  In essence, experience shows that projects that fail to  implement disciplined processes at the beginning are forced to implement  them later, when it takes more time and they are less effective.
As can be seen in figure 1, a major consumer of project resources in  undisciplined efforts is rework (also known as thrashing).
Rework occurs when the original work has defects or is no longer needed  because of changes in project direction.  Disciplined organizations focus  their efforts on reducing the amount of rework because it is expensive.   Studies have shown that fixing a defect during testing is anywhere from 10  to 100 times more expensive than fixing it during the design or  requirements phase.
section_title: Requirements Management Process Effective, but Implementation Challenges and Risks Remain
paragraphs: To date, Navy ERP management has followed a comprehensive and  disciplined requirements management process, as well as leveraged  lessons learned from the implementation of the four ERP pilot programs to  avoid repeating past mistakes.  Assuming that the project continues to  effectively implement the processes it has adopted, the planned  functionality of the Navy ERP has the potential to address at least some of  the weaknesses identified in the Navy’s financial improvement plan.   However, the project faces numerous challenges and risks.  Since the  program is still in a relatively early phase—it will not be fully operational  until fiscal year 2011, at a currently estimated cost of $800 million—the  project team must be continually vigilant and held accountable for ensuring  that the disciplined processes are followed in all phases to help achieve  overall success.  For example, the project management office will need to  ensure that it effectively oversees the challenges and risks associated with  developing interfaces with 44 Navy and DOD systems and data  conversion—areas that were troublesome in other DOD efforts we have  audited.  Considering that the project is in a relatively early phase and  DOD’s history of not implementing systems on time and within budget, the  projected schedule and costs estimates are subject to, and very likely will,  change.   Furthermore, a far broader challenge, which lies outside the  immediate control of the Navy ERP program office, is that the ERP is  proceeding without DOD having clearly defined its BEA.  As we have  recently reported, DOD’s BEA still lacks many of the key elements of a  well-defined architecture.  The real value of a BEA is that it provides the  necessary content for guiding and constraining system investments in a  way that promotes interoperability and minimizes overlap and duplication.   Without it, rework will likely be needed to achieve those outcomes.
subsections: section_title: Navy ERP Built on Lessons Learned from Pilot Projects
paragraphs: Although the four pilot projects were under the control of different entities  and had different functional focuses, a pattern of issues emerged that the  Navy recognized as being critical for effective development of future  projects.  The Navy determined that the pilots would not meet its overall  requirements and concluded that the best alternative was to develop a new  ERP system—under the leadership of a central program office—and use  efforts from the pilots as starting points by performing a review of their  functionality and lessons learned, eliminating redundancies, and  developing new functionalities that were not addressed by the pilots.  The  lessons learned from the pilots cover technical, organizational, and  managerial issues and reinforce the Navy’s belief that it must effectively  implement the processes that are necessary to effectively oversee and  manage the ERP efforts.  Navy ERP project management recognizes that  the failure to do so would, in all likelihood, result in this ERP effort  experiencing the same problems as those resulting in the failure of the four  earlier pilots.
One of the most important lessons learned from the earlier experiences by  the Navy ERP project management is the need for following disciplined  processes to identify and manage requirements.  As discussed later in this  report, the ERP program is following best practices in managing the  system’s requirements.  A key part of requirements identification is to have  system users involved in the process to ensure that the system will meet  their needs.  Additionally, the inclusion of system users in the detailed  requirement development process creates a sense of ownership in the  system, and prepares system users for upcoming changes to the way they  conduct their business.  Moreover, the experience from the pilots  demonstrated that the working-level reviews must be cross functional.  For  example, the end-to-end process walkthroughs, discussed later, reinforce  the overall business effect of a transaction throughout the enterprise, and  help to avoid a stovepiped view of an entity’s operations.
Another lesson learned is the need to adopt business processes to conform  with the types of business practices on which the standard COTS packages  are based, along with the associated transaction formats.  Just the  opposite approach was pursued for the pilots, during which the Navy  customized many portions of the COTS software to match the existing  business process environment.  However, the current Navy ERP  management is restraining customization to the core COTS software to  allow modifications only where legal or regulatory demands require.   Obviously, minimizing the amount of customization reduces the complexity  and costs of development.  Perhaps more importantly, holding  customization to a minimum helps an entity take advantage of two valuable  benefits of COTS software.  First, COTS software provides a mature,  industry-proven “best practices” approach to doing business.  The core  elements of work-flow management, logistics, financial management, and  other components have been optimized for efficiency and standardization  in private industry over many years.  According to program officials, the  Navy ERP will adhere to the fundamental concepts of using a COTS  package and thus take advantage of this efficiency benefit by modifying  their business practices to match the COTS software rather than vice versa  as was done in the four pilots.  Having the software dictate processes is a  difficult transition for users to accept, and Navy ERP officials recognize the  challenge in obtaining buy-in from system users.  To meet this challenge,  they are getting users involved early in requirements definition, planning  for extensive training, and ensuring that senior level leadership emphasize  the importance of process change, so the entire chain of command  understands and accepts its role in the new environment.  In effect, the  Navy is taking the adopted COTS process and then presenting it to the  users.  As a result, the Navy is attempting to limit the amount of  customization of the software package.  One important consideration in  doing this is that if the standard COTS components are adopted, the  maintenance burden of upgrades remains with the COTS vendor.
Finally, the Navy learned from the pilots that it needed to manage its  system integrators better.  The ERP officials also found that they could  significantly reduce their risk by using the implementation methodology of  the COTS vendor rather than the specific approach of a system integrator.   Each of the pilots had separate system integrators with their own particular  methodology for implementing the COTS software.  According to Navy  ERP officials, using the implementation methodology and tool set of the  COTS vendor maintains a closer link to the underlying software, and  provides more robust requirements management by easily linking  requirements from the highest level down to the COTS transaction level.   Navy ERP is focused on staying as close as possible to the delivered COTS  package, both in its avoidance of customization and its use of tools  provided by the COTS vendor.  In contrast, with the pilots, the Navy  allowed the system integrators more latitude in the development process,  relying on their expertise and experience with other ERP efforts to guide  the projects.  Navy ERP management realized they needed to maintain  much better control over the integrators’ work.  As a result, the Navy  established the Strategy, Architecture, and Standards Group to structure  and guide the effort across the Navy.
section_title: Requirements Management Process Following Best Practices
paragraphs: Our review found that the ERP development team has so far followed an  effective process for managing its requirements development.   Documentation was readily available for us to trace selected requirements  from the highest level to the lowest, detailed transaction level.  This  traceability allows the user to follow the life of the requirement both  forward and backward through the documentation, and from origin  through implementation.  Traceability is also critical to understanding the  parentage, interconnections, and dependencies among the individual  requirements.  This information in turn is critical to understanding the  impact when a requirement is changed or deleted.
Requirements represent the blueprint that system developers and program  managers use to design, develop, test, and implement a system. Improperly  defined or incomplete requirements have been commonly identified as a  cause of system failure and systems that do not meet their cost, schedule,  or performance goals. Without adequately defined requirements that have  been properly reviewed and tested, significant risk exists that the system  will need extensive and costly changes before it will achieve its intended  capability.
Because requirements provide the foundation for system testing,  specificity and traceability defects in system requirements preclude an  entity from implementing a disciplined testing process. That is,  requirements must be complete, clear, and well documented to design and  implement an effective testing program. Absent this, an organization is  taking a significant risk that its testing efforts will not detect significant  defects until after the system is placed into production. Industry  experience indicates that the sooner a defect is recognized and corrected,  the cheaper it is to fix. As shown in figure 2, there is a direct relationship  between requirements and testing.
Although the actual testing activities occur late in the development cycle,  test planning can help disciplined activities reduce requirements-related  defects. For example, developing conceptual test cases based on the  requirements derived from the concept of operations and functional  requirements stages can identify errors, omissions, and ambiguities long  before any code is written or a system is configured. Disciplined  organizations also recognize that planning testing activities in coordination  with the requirements development process has major benefits.  As we  have previously reported, failure to effectively manage requirements and  testing activities has posed operational problems for other system  development efforts.
The Navy ERP requirements identification process began with formal  agreement among the major stakeholders on the scope of the system,  followed by detailed, working-level business needs from user groups and  legacy systems.  The high-level business or functional requirements  identified initially are documented in the Operational Requirements  Document (ORD).  The ORD incorporates requirements from numerous  major DOD framework documents and defines the capabilities that the  system must support, including business operation needs such as  acquisition, finance, and logistics.  In addition, the ORD also identifies the  numerous policy directives to which the Navy ERP must conform, such as  numerous DOD infrastructure systems, initiatives, and policies.  The ORD  was distributed to over 150 Navy and DOD reviewers.  It went through  seven major revisions to incorporate the comments and suggestions  provided by the reviewers before being finalized in April 2004.  According  to Navy ERP program officials, any requested role for the Navy ERP to  perform that was not included in the ORD will not be supported.  This is a  critical decision that reduces the project’s risks since “requirements creep”  is another cause of projects that do not meet their cost, schedule, and  performance objectives.
We selected seven requirements from the ORD that related to specific  Navy problem areas, such as financial reporting and asset management,  and found that the requirements had the expected attributes, including the  necessary detail one would normally expect to find for the requirement  being reviewed.  For example, a requirement stated that the ERP will  provide reports of funds expended versus funds allocated.  We found this  requirement was described in a low-level requirement document called a  Customer Input Template, which included a series of questions that must  be addressed.  The documentation further detailed the standard reports  that were available based on the selection of configuration options.   Further, the documentation of the detailed requirements identified the  specific COTS screen number that would be used and described the screen  settings that would be used when a screen was “activated.”
While the ORD specifies the overall capabilities of the system at a high  level, more specific, working-level requirements also had to be developed  to achieve a usable blueprint for configuration and testing of the system.   To develop these lower-level requirements, the Navy ERP project held  detailed working sessions where requirements and design specifications  were discussed, refined, formalized, and documented.  Each high-level  requirement was broken down into its corresponding business processes,  which in turn drove the selection of transactions (COTS functions) to be  used for configuration of the software.  For each selected transaction,  comprehensive documentation was created to capture the source  information that defines how and why a transaction must be configured.   This documentation is critical for ensuring accurate configuration of the  software, as well as for testing the functionality of the software after  configuration.  Table 3 describes the kinds of documentation used to  maintain these lower-level detailed requirements.
Additionally, the Navy ERP program is using a requirements management  tool containing a database that links each requirement from the highest to  the lowest level and maintains the relationship between the requirements.   This tool helps to automate the linkage between requirements and helps  provide the project staff reasonable assurance that its stated processes  have been effectively implemented.  This linkage is critical to  understanding the scope of any potential change.  For example, the users  can utilize the tool to (1) determine the number of transactions affected by  a proposed change and (2) identify the detailed documentation necessary  for understanding how this change will affect each business process.  To  further ensure that the individual transactions ultimately support the  adopted business process, Navy ERP officials conducted master business  scenarios or end-to-end process walkthroughs.  This end-to-end view of  the business process ensures that the business functionality works across  the various subsystems of the COTS package.  For instance, the  requirements for a purchase order could be viewed simply from the  vantage point of a logistics person or the acquisition community.  However,  a purchase order also has financial ramifications, and therefore must be  posted to financial records, such as the general ledger.  The master  business scenarios provide a holistic review of the business process  surrounding each transaction.
section_title: ERP Has the Potential to Address Some of the Navy’s Reported Financial Management Weaknesses
paragraphs: The Navy expects the new ERP project to address a number of the  weaknesses cited in the Department of the Navy Financial Improvement  Plan—a course of action directed towards achieving better financial  management and an unqualified audit opinion for the Department of the  Navy annual financial statements.  According to ERP officials, the COTS  software used for the ERP program will improve the Navy’s current  financial controls in the areas of asset visibility, financial reporting, and full  cost accounting.  However, the currently planned ERP is not intended to  provide an all-inclusive end-to-end corporate solution for the Navy.
The COTS software offers the potential for real-time asset visibility for the  Navy, limited by two factors beyond the project’s scope.  First, items in  transit fall under the authority of the U.S. Transportation Command  (TRANSCOM).  Once the Navy hands off an item to TRANSCOM, it does  not retain visibility of that asset until it arrives at another Navy location.   The second factor is the limited ability for communication with ships at  sea.  Once the currently planned ERP is fully implemented, it will cover all  inventories, including inventory on ships.  However, the data for shipboard  inventory will be current only as of when the ship leaves port.  Those data  will typically not be updated until the ship docks in another port and can  transmit updated information to the ERP system.  This lag time for some  ships could be as much as 3 to 4 months.  While the ERP has the capability  to maintain real-time shipboard inventory, the Navy has yet to decide  whether to expand the scope of the ERP and build an interface with the  ships, which could be extensive and costly, or install the ERP on the ships.   Both options present additional challenges that necessitate thorough  analysis of all alternatives before a decision is made.  According to the  program office, a time frame for making this critical decision has not been  established.
The COTS software is also intended to provide standardized government  and proprietary financial reporting at any level within the defined  organization.  According to Navy ERP officials, full cost accounting will be  facilitated by a software component integrated with the ERP.  For example,  the Navy expects that this component will provide up-to-date cost  information—including labor, materials, and overhead—for its numerous,  and often complicated, maintenance jobs.  Full cost information is  necessary for effective management of production, maintenance, and other  activities.
According to Navy ERP program officials, when fully operational in fiscal  year 2011, the Navy ERP will be used by organizations comprising  approximately 80 percent of Navy’s estimated appropriated funds—after  excluding the Marine Corps and military pay and personnel.  Based on  fiscal years’ 2006 through 2011 defense planning budget, the Navy ERP will  manage approximately $74 billion annually.  The organizations that will use  Navy ERP include the Naval Air Systems, the Naval Sea Systems, the Naval  Supply Systems, the Space and Naval Warfare Systems, and the Navy  Facilities Engineering Commands, as well as the Office of Naval Research,  the Atlantic and Pacific Fleets, and the Strategic Systems Programs.    However, the Navy ERP will not manage in detail all of the 80 percent.   About 2 percent, or approximately $1.6 billion, will be executed and  maintained in detail by respective financial management systems at the  aviation and shipyard depots.  For example, when a work order for a repair  of an airplane part is prepared, the respective financial management  system at the depot will execute and maintain the detailed transactions.   The remaining 20 percent that the Navy ERP will not manage comprises the  Navy Installations Command, field support activities, and others.  Navy  ERP officials have indicated that it is the Navy’s intent to further expand  the system in the future to include the aviation and shipyard depots, but  definite plans have not yet been made.  According to Navy ERP officials,  the software has the capability to be used at the aviation and shipyard  depots, but additional work would be necessary.  For example, the desired  functionality and related requirements—which as discussed above, are  critical to the success of any project—would have to be defined for the  aviation and shipyard depots.
section_title: System Interfaces and Data Conversion Will Be Challenges
paragraphs: While the Navy’s requirements management process is following  disciplined processes and comprises one critical aspect of the overall  project development and implementation, by itself, it is not sufficient to  provide reasonable assurance of the ERP’s success.  Going forward, the  Navy faces very difficult challenges and risks in the areas of developing and  implementing 44 system interfaces with other Navy and DOD systems, and  accurately converting data from the existing legacy systems to the ERP.  As  previously noted, financial management is a high-risk area in the  department and has been designated as such since 1995.  One of the  contributing factors has been DOD’s inability to develop integrated  systems.  As a result, the Navy is dependent upon the numerous interfaces  to help improve the accuracy of its financial management data.  Navy ERP  program managers have recognized the issues of system interfaces and  data conversion in their current list of key risks.  They have identified some  actions that need to be taken to mitigate the risks; however, they have not  yet developed the memorandums of agreement with the owners of the  systems which the Navy ERP will interface.  According to the Navy ERP  program office, they plan to complete these memorandums of agreement  by October 2005.
subsections: section_title: Integrated Systems
paragraphs: One of the long-standing problems within DOD has been the lack of  integrated systems.  This is evident in the many duplicative, stovepiped  business systems among the 4,150 that DOD reported as belonging to its  systems environment.  Lacking integrated systems, DOD has a difficult time  obtaining accurate and reliable information on the results of its business  operations and continues to rely on either manual reentry of data into  multiple systems, convoluted system interfaces, or both.  These system  interfaces provide data that are critical to day-to-day operations, such as  obligations, disbursements, purchase orders, requisitions, and other  procurement activities.  Testing the system interfaces in an end-to-end  manner is necessary in order for the Navy to have reasonable assurance  that the ERP will be capable of providing the intended functionality.
The testing process begins with the initial requirements development  process.  Furthermore, test planning can help disciplined activities reduce  requirements-related defects. For example, developing conceptual test  cases based on the requirements can identify errors, omissions, and  ambiguities long before any code is written or a system is configured.  The  challenge now before Navy ERP is to be sure its testing scenarios  accurately reflect the activities of the “real users,” and the dependencies of  external systems.
We previously reported that Sears and Wal-Mart, recognized as leading- edge inventory management companies, have automated systems that  electronically receive and exchange standard data throughout the entire  inventory management process, thereby reducing the need for manual data  entry. As a result, information moves through the data systems with  automated ordering of inventory from suppliers; receiving and shipping at  distribution centers; and receiving, selling, and reordering at retail stores.  Unlike DOD, which has a proliferation of nonintegrated systems using  nonstandard data, Sears and Wal-Mart require all components and  subsidiaries to operate within a standard systems framework that results in  an integrated system and does not allow individual systems development.
For the first deployment, the Navy has to develop interfaces that permit the  ERP to communicate with 44 systems—27 that are Navy specific and 17  systems belonging to other DOD entities.  Figure 3 illustrates the numerous  required system interfaces.
Long-standing problems regarding the lack of integrated systems and use of  nonstandard data within DOD pose significant risks for the Navy ERP to  successfully interface with these systems.  Even if integration is successful,  if the information within the 44 systems is not accurate and reliable, the  overall information on Navy’s operation provided by the ERP to Navy  management and the Congress will not be useful in the decision-making  process.  While the Navy ERP project office is working to develop  agreements with system owners for the interfaces and has been developing  the functional specifications for each system, officials acknowledged that,  as of May 2005, they are behind schedule in completing the interface  agreements due to other tasks.  The Navy ERP is dependent on the system  owners to achieve their time frames for implementation.  For example, the  Defense Travel System (DTS) is one of the DOD systems with which the  Navy ERP is to interface and exchange data.  DTS is currently being  implemented, and any problems that result in a DTS schedule slippage will,  in turn, affect Navy ERP’s interface testing.
We have previously reported that the lack of system interface testing has  seriously impaired the operation of other system implementation efforts.   For example, in May 2004, we reported that because the system interfaces  for the Defense Logistics Agency’s Business Systems Modernization (BSM)  program and the Army’s LMP were not properly tested prior to deployment,  severe operational problems were experienced.  Such problems have led  BSM, LMP, and organizations with which they interface—such as DFAS—to  perform costly manual reentry of transactions, which can cause additional  data integrity problems. For example:  BSM’s functional capabilities were adversely affected because a  significant number of interfaces were still in development or were being  executed manually once the system became operational. Since the  design of system interfaces had not been fully developed and tested,  BSM experienced problems with receipts being rejected, customer  orders being canceled, and vendors not being paid in a timely manner.   At one point, DFAS suspended all vendor payments for about 2 months,  thereby increasing the risk of late payments to contractors and  violations of the Prompt Payment Act.
In January 2004, the Army reported that due to an interface failure, LMP  had been unable to communicate with the Work Ordering and Reporting  Communications System (WORCS) since September 2003. WORCS is  the means by which LMP communicates with customers on the status of  items that have been sent to the depot for repair and initiates  procurement actions for inventory items.  The Army has acknowledged  that the failure of WORCS has resulted in duplicative shipments and  billings and inventory items being delivered to the wrong locations.   Additionally, the LMP program office has stated that it has not yet  identified the specific cause of the interface failure. The Army is  currently entering the information manually, which, as noted above, can  cause additional data integrity errors.
Besides the challenge of developing the 44 interfaces, the Navy ERP must  also develop the means to be compliant with DOD’s efforts to standardize  the way that various systems exchange data with each other.  As discussed  in our July 2004 report, DOD is undertaking a huge and complex task  (commonly referred to as the Global Information Grid or GIG) that is  intended to integrate virtually all of DOD’s information systems, services,  applications, and data into one seamless, reliable, and secure network. The  GIG initiative is focused on promoting interoperability throughout DOD by  building an Internet-like network for DOD-related operations based on  common standards and protocols rather than on trying to establish  interoperability after individual systems become operational.  DOD  envisions that this type of network would help ensure systems can easily  and quickly exchange data and change how military operations are planned  and executed since much more information would be dynamically available  to users.
DOD’s plans for realizing the GIG involve building a new core network and  information capability and successfully integrating the majority of its  weapon systems; command, control, and communications systems; and  business systems with the new network. The effort to build the GIG will  require DOD to make a substantial investment in a new set of core  enterprise programs and initiatives.  To integrate systems such as the Navy  ERP into the GIG, DOD has developed (1) an initial blueprint or  architecture for the GIG and (2) new policies, guidance, and standards to  guide implementation.  According to project officials, the Navy ERP system  will be designed to support the GIG.  However, they face challenges that  can result in significant cost and schedule risks depending on the decisions  reached.  One challenge is the extent to which other DOD applications with  which the Navy ERP must exchange data are compliant with the GIG.   While traditional interfaces with systems that are not GIG compliant can be  developed, these interfaces may suboptimize the benefits expected from  the Navy ERP.  The following is one example of the difficulties faced by the  Navy ERP project.
As mentioned previously, one system that will need to exchange data with  the Navy ERP system is DTS.  However, the DTS program office and the  Navy ERP project office hold different views of how data should be  exchanged between the two systems.  The travel authorization process  exemplifies these differences.  DTS requires that funding information and  the associated funds be provided to DTS in advance of a travel  authorization being processed.  In effect, DTS requires that the financial  management systems set aside the funds necessary for DTS operations.   Once a travel authorization is approved, DTS notifies the appropriate  financial management system that an obligation has been incurred.  The  Navy ERP system, on the other hand, only envisions providing basic  funding information to DTS in advance, and would delay providing the  actual funds to DTS until they are needed in order to (1) maintain adequate  funds control, (2) ensure that the funds under its control are not tied up by  other systems, and (3) ensure that the proper accounting data are provided  when an entry is made into its system.
According to the Software Engineering Institute (SEI), a widely recognized  model evaluating a system of systems interoperability is the Levels of  Information System Interoperability.  This model focuses on the increasing  levels of sophistication of system interoperability.  According to Navy ERP  officials, the GIG and the ERP effort are expected to accomplish the  highest level of this model—enterprise-based interoperability.  In essence,  systems that achieve this level of interoperability can provide multiple  users access to complex data simultaneously, data and applications are  fully shared and distributed, and data have a common interpretation  regardless of format.  This is in contrast to traditional interface strategies,  such as the one used by DTS.  The traditional approach is more aligned  with the lowest level of the SEI model.  Data exchanged at this level rely on  electronic links that result in a simple electronic exchange of data.
section_title: Alignment with DOD’s BEA Is a Significant Risk Factor
paragraphs: A broader challenge and risk that is out of the Navy ERP project’s control,  but could significantly affect it, is DOD’s development of a BEA.  As we  recently reported, DOD’s BEA still lacks many of the key elements of a  well-defined architecture and no basis exists for evaluating whether the  Navy ERP will be aligned with the BEA and whether it would be a  corporate solution for DOD in its “To Be” or target environment.
An enterprise architecture consists of snapshots of the enterprise’s current  environment and its target environment, as well as a capital investment  road map for transitioning from the current to the target environment.  The  real value of an enterprise architecture is that it provides the necessary  content for guiding and constraining system investments in a way that  promotes interoperability and minimizes overlap and duplication.  At this  time, it is unknown what the target environment will be.  Therefore, it is  unknown what business processes, data standards, and technological  standards the Navy ERP must align to, as well as what legacy systems will  be transitioned into the target environment.
The Navy ERP project team is cognizant of the BEA development and has  attempted to align to prior versions of it.  The project team analyzed the  BEA requirements and architectural elements to assess Navy ERP’s  compliance.  The project team mapped the BEA requirements to the Navy  ERP functional areas and the BEA operational activities to the Navy ERP’s  business processes.   The Navy ERP project team recognizes that  architectures evolve over time, and analysis and assessments will continue  as requirements are further developed and refined.  The scope of the BEA  and the development approach are being revised.  As a result of the new  focus, DOD is determining which products from prior releases of the BEA  could be salvaged and used.
Since the Navy ERP is being developed absent the benefit of an enterprise  architecture, there is limited, if any, assurance that the Navy ERP will be  compliant with the architecture once it becomes more robust in the future.   Given this scenario, it is conceivable that the Navy ERP will be faced with  rework in order to be compliant with the architecture, once it is defined,  and as noted earlier, rework is expensive.  At the extreme, the project could  fail as the four pilots did.  If rework is needed, the overall cost of the Navy  ERP could exceed the Navy’s current estimate of $800 million.
section_title: Accuracy of Data Conversion Is Critical
paragraphs: The ability of the Navy to effectively address its data conversion challenges  will also be critical to the ultimate success of the ERP effort.  A Joint  Financial Management Improvement Program (JFMIP) white paper on  financial system data conversion noted that data conversion (that is,  converting data in a legacy system to a new system) was one of the critical  tasks necessary to successfully implement a new financial system.  The  paper further pointed out that data conversion is one of the most frequently  underestimated tasks.
If data conversion is done right, the new system has a much greater  opportunity for success. On the other hand, converting data incorrectly or  entering unreliable data from a legacy system can have lengthy and long- term repercussions.  The adage “garbage in, garbage out” best describes the  adverse impact.  Accurately converting data, such as account balances,  from the pilots, as well as other systems that the Navy ERP is to replace,  will be critical to the success of the Navy ERP.  While data conversion is  identified in the Navy ERP’s list of key risks, it is too early in the ERP  system life cycle for the development of specific testing plans.
However, our previous audits have shown that if data conversion is not  done properly, it can negatively impact system efficiency.  For example, the  Army’s LMP data conversion effort has proven to be troublesome and  continues to affect business operations.  As noted in our recent report,  when the Tobyhanna Army Depot converted ending balances from its  legacy finance and accounting system—the Standard Depot System  (SDS)—to LMP in July 2003, the June 30, 2003, ending account balances in  SDS did not reconcile to the beginning account balances in LMP.  Accurate  account balances are important for producing reliable financial reports.   Another example is LMP’s inability to transfer accurate unit-of-issue data— quantity of an item, such as each number, dozen, or gallon—from its legacy  system to LMP.  This resulted in excess amounts of material ordered.   Similar problems could occur with the Navy ERP if data conversion issues  are not adequately addressed.  The agreements between the Navy ERP and  the other systems owners, discussed previously, will be critical to  effectively support Navy’s ERP data conversion efforts.
section_title: Additional Actions Can be Taken to Improve Management Oversight of the Navy ERP Effort
paragraphs: Navy officials could take additional actions to improve management  oversight of the Navy ERP effort.  For example, we found that the Navy  does not have a mechanism in place to capture the data that can be used to  effectively assess the project management processes.  Best business  practices indicate that a key facet of project management and oversight is  the ability to effectively monitor and evaluate a project’s actual  performance, cost, and schedule against what was planned.  Performing  this critical task requires the accumulation of quantitative data or metrics  that can be used to evaluate a project’s performance.  This information is  necessary to understand the risk being assumed and whether the project  will provide the desired functionality.  Lacking such data, the ERP program  management team can only focus on the project schedule and whether  activities have occurred as planned, not whether the activities achieved  their objectives.
Additionally, although the Navy ERP program has a verification and  validation function, it relies on in-house subject matter experts and others  who are not independent to provide an assessment of the Navy ERP to  DOD and Navy management.  The use of an IV&V function is recognized as  a best business practice and can help provide reasonable assurance that  the system satisfies its intended use and user needs.  Further, an  independent assessment of the Navy ERP would provide information to  DOD and Navy management on the overall status of the project, including  the effectiveness of the management processes being utilized and  identification of any potential risks that could affect the project with  respect to cost, schedule, and performance.  Given DOD’s long-standing  inability to implement business systems that provide users with the  promised capabilities, an independent assessment of the ERP’s  performance is warranted.
subsections: section_title: Quantitative Data Necessary for Assessing Whether the System Will Provide the Needed Functionality
paragraphs: The Navy’s ability to understand the impact of the weaknesses in its  processes will be limited because it has not determined the quantitative  data or metrics that can be used to assess the effectiveness of its project  management processes.  This information is necessary to understand the  risk being assumed and whether the project will provide the desired  functionality.  The Navy has yet to establish the metrics that would allow it  to fully understand (1) its capability to manage the entire ERP effort; (2)  how its process problems will affect the ERP cost, schedule, and  performance objectives; and (3) the corrective actions needed to reduce  the risks associated with the problems identified.  Experience has shown  that such an approach leads to rework and thrashing instead of making real  progress on the project.
SEI has found that metrics identifying important events and trends are  invaluable in guiding software organizations to informed decisions. Key  SEI findings relating to metrics include the following.
The success of any software organization depends on its ability to make  predictions and commitments relative to the products it produces.
Effective measurement processes help software groups succeed by  enabling them to understand their capabilities so that they can develop  achievable plans for producing and delivering products and services.
Measurements enable people to detect trends and anticipate problems,  thus providing better control of costs, reducing risks, improving quality,  and ensuring that business objectives are achieved.
The lack of quantitative data to assess a project has been a key concern in  other projects we have reviewed.  Without such a process, management  can only focus on the project schedule and whether activities have  occurred as planned, not whether the activities achieved their objectives.   Further, such quantitative data can be used to hold the project team  accountable for providing the promised capability.
Defect-tracking systems are one means of capturing quantitative data that  can be used to evaluate project efforts.  Although HHS had a system that  captured the reported defects, we found that the system was not updated in  a timely manner with this critical information.  More specifically, one of  the users identified a process weakness related to grant accounting as a  problem that will affect the deployment of HHS’s system—commonly  referred to as a “showstopper.” However, this weakness did not appear in  the defect-tracking system until about 1 month later.  As a result, during  this interval the HHS defect-tracking system did not accurately reflect the  potential problems identified by users, and HHS management was unable  to determine (1) how well the system was working and (2) the amount of  work necessary to correct known defects.  Such information is critical  when assessing a project’s status.
We have also reported that while NASA had a system that captured the  defects that have been identified during testing, an analysis was not  performed to determine the root causes of reported defects.   A critical  element in helping to ensure that a project meets its cost, schedule, and  performance goals is to ensure that defects are minimized and corrected as  early in the process as possible.  Understanding the root cause of a defect is  critical to evaluating the effectiveness of a process. For example, if a  significant number of defects are caused by inadequate requirements  definition, then the organization knows that the requirements management  process it has adopted is not effectively reducing risks to acceptable levels.   Analysis of the root causes of identified defects allows an organization to  determine whether the requirements management approach it has adopted  sufficiently reduces the risks of the system not meeting cost, schedule, and  functionality goals to acceptable levels. Root-cause analysis would also  help to quantify the risks inherent in the testing process that has been  selected.
Further, the Navy has not yet implemented an earned value management  system, which is another metric that can be employed to better manage and  oversee a system project.  Both OMB and DOD require the use of an  earned value management system.  The earned value management system  attempts to compare the value of work accomplished during a given period  with the work scheduled for that period. By using the value of completed  work as a basis for estimating the cost and time needed to complete the  program, management can be alerted to potential problems early in the  program.  For example, if a task is expected to take 100 hours to complete  and it is 50 percent complete, the earned value management system would  compare the number of hours actually spent to complete the task to the  number of hours expected for the amount of work performed.  In this  example, if the actual hours spent equaled 50 percent of the hours  expected, then the earned value would show that the project’s resources  were consistent with the estimate.  Without an effective earned value  management system, the Navy and DOD management have little assurance  that they know the status of the various project deliverables in the context  of progress and the cost incurred in completing each of the deliverables.  In  other words, an effective earned value management system would be able  to provide quantitative data on the status of a given project deliverable,  such as a data conversion program.  Based on this information, Navy  management would be able to determine whether the progress of the data  conversion effort was within the expected parameters for completion.   Management could then use this information to determine actions to take  to mitigate risk and manage cost and schedule performance.  According to  Navy ERP officials, they intend to implement the earned value management  system as part of the contract for the next phase of the project.
section_title: Independent Assessment of Navy ERP Could Enhance DOD and Navy Management Oversight of the Project
paragraphs: The Navy has not established an IV&V function to provide an assessment of  the Navy ERP to DOD and Navy management.  Best business practices  indicate that use of an IV&V function is a viable means to provide  management reasonable assurance that the planned system satisfies its  planned use and users.  An effective IV&V review process would provide  independent information to DOD and Navy management on the overall  status of the project, including a discussion of any impacts or potential  impacts to the project with respect to cost, schedule, and performance.   These assessments involve reviewing project documentation, participating  in meetings at all levels within the project, and providing periodic reports  and recommendations, if deemed warranted, to senior management.  The  IV&V function should report on every facet of a system project such as:  Testing program adequacy.  Testing activities would be evaluated to  ensure they are properly defined and developed in accordance with  industry standard and best practices.
Critical-path analysis. A critical path defines the series of tasks that  must be finished in time for the entire project to finish on schedule.  Each task on the critical path is a critical task.  A critical-path analysis  helps to identify the impact of various project events, such as delays in  project deliverables, and ensures that the impact of such delays is  clearly understood by all parties involved with the project.
System strategy documents.  Numerous system strategy documents  that provide the foundation for the system development and operations  are critical aspects of an effective system project.  These documents are  used for guidance in developing documents for articulating the plans  and procedures used to implement a system.  Examples of such  documents include the Life-cycle Test Strategy, Interface Strategy, and  Conversion Strategy.
The IV&V reports should identify the project management weaknesses that  increase the risks associated with the project to senior management so that  they can be promptly addressed. The Navy ERP program’s approach to the  verification and validation of its project management activities relies on in- house subject matter experts and others who work for the project team’s  Quality Assurance leader.  The results of these efforts are reported to the  project manager.  While various approaches can be used to perform this  function, such as using the Navy’s approach or hiring a contractor to  perform these activities, independence is a key component to successful  verification and validation activities. The system developer and project  management office may have vested interests and may not be objective in  their self-assessments. Accordingly, performing verification and validation  activities independently of the development and management functions  helps to ensure that verification and validation activities are unbiased and  based on objective evidence.  The Navy’s adoption of verification and  validation processes is a key component of its efforts to implement the  disciplined processes necessary to manage this project.  However, Navy  and DOD management cannot obtain reasonable assurance that the  processes have been effectively implemented since the present verification  and validation efforts are not conducted by an independent party.
In response to the Ronald W. Reagan National Defense Authorization Act  for Fiscal Year 2005, DOD has established a hierarchy of investment  review boards from across the department to improve the control and  accountability over business system investments.   The boards are  responsible for reviewing and approving investments to develop, operate,  maintain, and modernize business systems for their respective business  areas.  The various boards are to report to the Defense Business Systems  Management Committee (DBSMC), which is ultimately responsible for the  review and approval of the department’s investments in its business  systems.  To help facilitate this oversight responsibility, the reports  prepared by the IV&V function should be provided to the appropriate  investment review board and the DBSMC to assist them in the decision- making process regarding the continued investment in the Navy ERP.  The  information in the reports should provide reasonable assurance that an  appropriate rate of return is received on the hundreds of millions of dollars  that will be invested over the next several years and the Navy ERP provides  the promised capabilities.
To help ensure that the Navy ERP achieves its cost, schedule, and  performance goals, the investment review should employ an early warning  system that enables it to take corrective action at the first sign of slippages.  Effective project oversight requires having regular reviews of the project’s  performance against stated expectations and ensuring that corrective  actions for each underperforming project are documented, agreed to,  implemented, and tracked until the desired outcome is achieved.
section_title: Conclusions
paragraphs: The lack of management control and oversight and a poorly conceived  concept resulted in the Navy largely wasting about $1 billion on four ERP  system projects that had only a limited positive impact on the Navy’s ability  to produce reliable, useful, and timely information to aid in its day-to-day  operations.  The Navy recognizes that it must have the appropriate  management controls and processes in place to have reasonable assurance  that the current effort will be successful.  While the current requirements  management effort is adhering to the disciplined processes, the overall  effort is still in the early stages and numerous challenges and significant  risks remain, such as validating data conversion efforts and developing  numerous systems interfaces.  Given that the current effort is not  scheduled to be complete until 2011 and is currently estimated by the Navy  to cost about $800 million, it is incumbent upon Navy and DOD  management to provide the vigilant oversight that was lacking in the four  pilots.  Absent this oversight, the Navy and DOD run a higher risk than  necessary of finding, as has been the case with many other DOD business  systems efforts, that the system may cost more than anticipated, take  longer to develop and implement, and does not provide the promised  capabilities.  In addition, attempting large-scale systems modernization  programs without a well-defined architecture to guide and constrain  business systems investments, which is the current DOD state, presents the  risk of costly rework or even system failure once the enterprise  architecture is fully defined.  Considering (1) the large investment of time  and money essentially wasted on the pilots and (2) the size, complexity,  and estimated costs of the current ERP effort, the Navy can ill afford  another business system failure.
section_title: Recommendations for Executive Action
paragraphs: To improve the Navy’s and DOD’s oversight of the Navy ERP effort, we  recommend that the Secretary of Defense direct the Secretary of the Navy  to require that the Navy ERP Program Management Office (1) develop and  implement the quantitative metrics needed to evaluate project performance  and risks and use the quantitative metrics to assess progress and  compliance with disciplined processes and (2) establish an IV&V function  and direct that all IV&V reports be provided to Navy management and to  the appropriate DOD investment review board, as well as the project  management.
Furthermore, given the uncertainty of the DOD business enterprise  architecture, we recommend that the Secretary of Defense direct the  DBSMC to institute semiannual reviews of the Navy ERP to ensure that the  project continues to follows the disciplined processes and meets its  intended costs, schedule, and performance goals.  Particular attention  should be directed towards system testing, data conversion, and  development of the numerous system interfaces with the other Navy and  DOD systems.
section_title: Agency Comments and Our Evaluation
paragraphs: We received written comments on a draft of this report from the Deputy  Under Secretary of Defense (Financial Management) and the Deputy Under  Secretary of Defense (Business Transformation), which are reprinted in  appendix II.  While DOD generally concurred with our recommendations, it  took exception to our characterization that the pilots were failures and a  waste of $1 billion.
Regarding the recommendations, DOD agreed that it should develop and  implement quantitative metrics that can be used to evaluate the Navy ERP  and noted that it intends to have such metrics developed by December  2005.  The department also agreed that the Navy ERP program  management office should establish an IV&V function and noted that the  IV&V team will report directly to the program manager.  We continue to  reiterate the need for the IV&V to be completely independent of the project.   As noted in the report, performing IV&V activities independently of the  development and management functions helps to ensure that the results  are unbiased and based on objective evidence.  Further, rather than having  the IV&V reports provided directly to the appropriate DOD investment  review boards as we recommended, DOD stated that the Navy management  and/or the project management office shall inform the Office of the Under  Secretary of Defense for Business Transformation of any significant IV&V  results.  We reiterate our support for the recommendation that the IV&V  reports be provided to the appropriate investment review board so that it  can determine whether any of the IV&V results are significant.  Again, by  providing the reports directly to the appropriate investment review board,  we believe there would be added assurances that the results were objective  and that the managers who will be responsible for authorizing future  investments in the Navy ERP will have the information needed to make the  most informed decision.
With regard to the reviews by the DBSMC, DOD partially agreed.  Rather  than semiannual reviews by the DBSMC as we recommended, the  department noted that the components (e.g., the Navy) would provide  briefings on their overall efforts, initiatives, and systems during meetings  with the DBSMC.  Given the significance of the Navy ERP, in terms of  dollars and its importance to the overall transformation of the department’s  business operations, and the failure of the four ERP pilots, we continue to  support more proactive semiannual reviews by the DBSMC.  As noted in  the report, the Navy’s initial estimate is that the ERP will cost at least $800  million, and given the department’s past difficulties in effectively  developing and implementing business systems, substantive reviews by  individuals outside of the program office that are focused just on the Navy  ERP by the highest levels of management within the department are  warranted.  Further, we are concerned that the briefings contemplated to  the DBSMC may not necessarily discuss the Navy ERP, nor provide the  necessary detailed discussions to offer the requisite level of confidence and  assurance that the project continues to follow disciplined processes with  particular attention to numerous challenges, such as system interfaces and  system testing.
In commenting on the report, the department depicted the pilots in a much  more positive light than we believe is merited.  DOD pointed out that it  viewed the pilots as successful, exceeding initial expectations, and forming  the foundation upon which to build a Navy enterprise solution, and took  exception to our characterization that the pilots were failures and largely a  waste of $1 billion.  As discussed in the report, the four pilots were narrow  in scope, and were never intended to be a corporate solution for resolving  any of the Navy’s long-standing financial and business management  problems.  We characterized the pilots as failures because the department  spent $1 billion on systems that did not result in marked improvement in  the Navy’s day-to-day operations.  While there may have been marginal  improvements, it is difficult to ascertain the sustained, long-term benefits  that will be derived by the American taxpayers for the $1 billion.
Additionally, the pilots present an excellent case study as to why the  centralization of the business systems funding would be an appropriate  course of action for the department, as we have previously recommended.   Each Navy command was allowed to develop an independent solution that  focused on its own parochial interest.  There was no consideration as to  how the separate efforts fit within an overall departmental framework, or,  for that matter, even a Navy framework. As noted in table 2, the pilots  performed many of the same functions and used the same software, but yet  were not interoperable because of the various inconsistencies in the design  and implementation.
Because the department followed the status quo, the pilots, at best,  provided the department with four more stovepiped systems that perform  duplicate functions.  Such investments are one reason why the department  reported in February 2005 that it had 4,150 business systems.   Further, in  its comments the department noted one of the benefits of the pilots was  that they “proved that the Navy could exploit commercial ERP tools  without significant customization.”  Based upon our review and during  discussions with the program office, just the opposite occurred in the  pilots.  Many portions of the pilots’ COTS software were customized to  accommodate the existing business processes, which negated the  advantages of procuring a COTS package.  Additionally, the department  noted that one of the pilots—SMART, on which, as noted in our report, the  Navy spent approximately $346 million through September 30, 2004—has  already been retired.  We continue to question the overall benefit that the  Navy and the department derived from these four pilots and the $1 billion it  spent.
As agreed with your offices, unless you announce the contents of this  report earlier, we will not distribute it until 30 days after its issuance date.   At that time, we will send copies to the Chairmen and Ranking Minority  Members, Senate Committee on Armed Services; Senate Committee on  Homeland Security and Governmental Affairs; Subcommittee on Defense,  Senate Committee on Appropriations; House Committee on Armed  Services; House Committee on Government Reform; and Subcommittee on  Defense, House Committee on Appropriations. We are also sending copies  to the Under Secretary of Defense (Comptroller); the Under Secretary of  Defense (Acquisition, Technology and Logistics); the Under Secretary of  Defense (Personnel and Readiness); the Assistant Secretary of Defense  (Networks and Information Integration); and the Director, Office of  Management and Budget. Copies of this report will be made available to  others upon request. In addition, the report will be available at no charge  on the GAO Web site at http://www.gao.gov. If you or your staff have any  questions on matters discussed in this report, please contact Gregory D.  Kutz at (202) 512-9505 or kutzg@gao.gov or Keith A. Rhodes at (202) 512- 6412 or rhodesk@gao.gov.  Key contributors to this report are listed in  appendix IV. Contact points for the Offices of Congressional Relations and  Public Affairs are shown on the last page of the report.
section_title: Scope and Methodology
paragraphs: To obtain a historical perspective on the planning and costs of the Navy’s  four Enterprise Resource Planning (ERP) pilot projects, and the decision to  merge them into one program, we reviewed the Department of Defense’s  (DOD) budget justification materials and other background information on  the four pilot projects.  We also reviewed Naval Audit Service reports on  the pilots.  In addition, we interviewed Navy ERP program management  and DOD Chief Information Officer (CIO) officials and obtained  informational briefings on the pilots.
To determine if the Navy has identified lessons learned from the pilots, how  they are being used, and the challenges that remain, we reviewed program  documentation and interviewed Navy ERP program officials.  Program  documentation that we reviewed included concept of operations  documentation, requirements documents, the testing strategy, and the test  plan.  In order to determine whether the stated requirements management  processes were effectively implemented, we performed an in-depth review  and analysis of seven requirements that relate to the Navy’s problem areas,  such as financial reporting and asset management, and traced them  through the various requirements documents.  These requirements were  selected in a manner that ensured that the requirements selected were  included in the Navy’s Financial Improvement Plan.
Our approach to validating the effectiveness of the requirements  management process relied on a selection of seven requirements from  different functional areas.  From the finance area, we selected the  requirement to provide reports of funds expended versus funds allocated.   From the intermediate-level maintenance management area, we selected  the requirement related to direct cost per job and forecasting accuracy.   From the procurement area, we selected the requirement to enable  monitoring and management of cost versus plan.  In the plant supply  functions area, we reviewed the requirement related to total material  visibility and access of material held by the activity and the enterprise.   From the wholesale supply functions area, we selected the requirements of  in-transit losses/in-transit write-offs and total material visibility and access  of material held by the activity and the enterprise.  Additionally, we  reviewed the requirement that the ERP be compliant with federal mandates  and requirements and the U.S. Standard General Ledger.
In order to provide reasonable assurance that our test results for the  selected requirements reflected the same processes used to document all  requirements, we did not notify the project office of the specific  requirements we had chosen until the tests were conducted.  Accordingly,  the project office had to be able to respond to a large number of potential  requests rather than prepare for the selected requirements in advance.   Additionally, we obtained the list of systems the Navy ERP will interface  with and interviewed selected officials responsible for these systems to  determine what activities the Navy ERP program office is working with  them on and what challenges remain.
To determine if there are additional business practices that could be used  to improve management oversight of the Navy ERP, we reviewed industry  standards and best practices from the Institute of Electrical and  Electronics Engineers, the Software Engineering Institute, the Joint  Financial Management Improvement Program, GAO executive guides, and  prior GAO reports.  Given that the Navy ERP effort is still in the early  stages of development, we did not evaluate all best practices.  Rather, we  concentrated on those that could have an immediate impact in improving  management’s oversight.  We interviewed Navy ERP program officials and  requested program documentation to determine if the Navy ERP had  addressed or had plans for addressing these industry standards and best  practices.
We did not verify the accuracy and completeness of the cost information  provided by DOD for the four pilots or the Navy ERP effort. We conducted  our work from August 2004 through June 2005 in accordance with U.S.  generally accepted government auditing standards.
We requested comments on a draft of this report from the Secretary of  Defense or his designee.  We received written comments on a draft of the  report from the Deputy Under Secretary of Defense (Financial  Management) and the Deputy Under Secretary of Defense (Business  Transformation), which are reprinted in appendix II.
section_title: Comments from the Department of Defense
section_title: Identification of the Navy and Defense Systems That Must Interface with the ERP
paragraphs: Configuration Data Manager’s Database – Open Architecture Common Rates Computation System/Common Allowance Development System Department of the Navy Industrial Budget Information System Integrated Technical Item Management & Procurement Maintenance and Ship Work Planning Naval Aviation Logistic Command Management Information System (2 different versions)
section_title: GAO Contacts and Acknowledgments
subsections: section_title: GAO Contacts
section_title: Acknowledgments
paragraphs: In addition to the contacts above, Darby Smith, Assistant Director; J.  Christopher Martin, Senior Level Technologist; Francine DelVecchio; Kristi  Karls; Jason Kelly; Mai Nguyen; and Philip Reiff made key contributions to  this report.