DOD Business Systems Modernization: Navy Implementing a Number of Key Management Controls on Enterprise Resource Planning System, but Improvements Still Needed
Published: 2009-09-15T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Defense (DOD) has long been challenged in effectively implementing key acquisition management controls on its thousands of business system investments. For this and other reasons, GAO has designated DOD's business systems modernization efforts as high-risk since 1995. One major business system investment is the Navy's Enterprise Resource Planning (ERP) system. Initiated in 2003, it is to standardize the Navy's business processes, such as acquisition and financial management. It is being delivered in increments, the first of which is to cost about $2.4 billion over its 20-year useful life and be fully deployed by fiscal year 2013. To date, the program has experienced about $570 million in cost overruns and a 2-year schedule delay. GAO was asked to determine whether (1) system testing is being effectively managed, (2) system changes are being effectively controlled, and (3) independent verification and validation (IV&V) activities are being effectively managed. To do this, GAO analyzed relevant program documentation, traced random samples of test defects and change requests, and interviewed cognizant officials.
section_title: What GAO Found
paragraphs: The Navy has largely implemented effective controls on Navy ERP associated with system testing and change control. For example, it has established a well-defined structure for managing tests, including providing for a logical sequence of test events, adequately planning key test events, and documenting and reporting test results. In addition, it has documented, and is largely following, its change request review and approval process, which reflects key aspects of relevant guidance, such as having defined roles and responsibilities and a hierarchy of control boards. However, important aspects of test management and change control have not been fully implemented. Specifically, the program's tool for auditing defect management did not always record key data about changes made to the status of identified defects. To its credit, the program office recently took steps to address this, thereby reducing the risk of defect status errors or unauthorized changes. Also, while the program office's change review and approval procedures include important steps, such as considering the impact of a change, and program officials told GAO that cost and schedule impacts of a change are discussed at control board meetings, GAO's analysis of 60 randomly selected change requests showed no evidence that cost and schedule impacts were in fact considered. Without such key information, decision-making authorities lack an adequate basis for making informed investment decisions, which could result in cost overruns and schedule delays. The Navy has not effectively managed its IV&V activities, which are designed to obtain an unbiased position on whether product and process standards are being met. In particular, the Navy has not ensured that the IV&V contractor is independent of the products and processes that it is reviewing. Specifically, the same contractor responsible for performing IV&V of Navy ERP products (e.g., system releases) is also responsible for ensuring that system releases are delivered within cost and schedule constraints. Because performance of this system development and management role makes the contractor potentially unable to render impartial assistance to the government in performing the IV&V function, there is an inherent conflict of interest. In addition, the IV&V agent reports directly and solely to the program manager and not to program oversight officials. As GAO has previously reported, the IV&V agent should report the findings and associated risks to program oversight officials, as well as program management, in order to better ensure that the IV&V results are objective and that the officials responsible for making program investment decisions are fully informed. Furthermore, the contractor has largely not produced the range of IV&V deliverables that were contractually required between 2006 and 2008. To its credit, the program office recently began requiring the contractor to provide assessment reports, as required under the contract, as well as formal quarterly reports; the contractor delivered the results of the first planned assessment in March 2009. Notwithstanding the recent steps that the program office has taken, it nevertheless lacks an independent perspective on the program's products and management processes

section_title: Letter
paragraphs: For decades, the Department of the Defense (DOD) has been challenged in  modernizing its timeworn business systems. In 1995, we designated  DOD’s business systems modernization program as high-risk, and continue  to do so today. Our reasons include the modernization’s large size,  complexity, and its critical role in addressing other high-risk areas, such as  overall business transformation and financial management. Moreover, we  continue to report on business system investments that fail to effectively  employ acquisition management controls and deliver promised benefits  and capabilities on time and within budget.
Nevertheless, DOD continues to invest billions of dollars in thousands of  these business systems, 11 of which account for about two-thirds of the  department’s annual spending on business programs. The Navy Enterprise  Resource Planning (ERP) program is one such program. Initiated in 2003,  Navy ERP is to standardize the Navy’s acquisition, financial, program  management, plant and wholesale supply, and workforce management  business processes across its dispersed organizational environment. As  envisioned, the program consists of a series of major increments, the first  of which includes three releases and is expected to cost approximately  $2.4 billion over its 20-year life cycle and to be fully operational in fiscal  year 2013. We recently reported that Navy ERP program management  weaknesses had contributed to a 2-year schedule delay and about $570  million in cost overruns.
As agreed, our objectives were to determine whether (1) system testing is  being effectively managed, (2) system changes are being effectively  controlled, and (3) independent verification and validation (IV&V)  activities are being effectively managed. To accomplish this, we analyzed  relevant program documentation, such as test management documents,  individual test plans and procedures and related test results and defect  reports; system change procedures and specific change requests and  decisions; change review board minutes; and verification and validation  plans and contract documents. We also observed the use of tools for  recording and tracking test defects and change requests, including tracing  a statistically valid sample of transactions through these tools.
We conducted this performance audit from August 2008 to September  2009, in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives. Additional details on our  objectives, scope, and methodology are in appendix I.
subsections: section_title: Background
paragraphs: The Department of the Navy’s (DON) primary mission is to organize, train,  maintain, and equip combat-ready naval forces capable of winning wars,  deterring aggression by would-be foes, preserving freedom of the seas, and  promoting peace and security. Its operating forces, known as the fleet, are  supported by four systems commands. Table 1 provides a brief description  of each command’s responsibilities.
To support the department’s mission, these commands perform a variety  of interrelated and interdependent business functions (e.g., acquisition and  financial management), relying heavily on business systems to do so. In  fiscal year 2009, DON’s budget for business systems and associated  infrastructure was about $2.7 billion, of which about $2.2 billion was  allocated to operations and maintenance of existing systems and about  $500 million to systems in development and modernization. Of the  approximately 2,480 business systems that DOD reports having, DON  accounts for 569, or about 23 percent, of the total. Navy ERP is one such  system investment.
subsections: section_title: Navy ERP: A Brief Description
paragraphs: In July 2003, the Assistant Secretary of the Navy for Research,  Development, and Acquisition established Navy ERP to converge the  functionality of four pilot systems that were under way at the four  commands into one system. According to DOD, Navy ERP is to address  the Navy’s long-standing problems related to financial transparency and  asset visibility. Specifically, the program is intended to standardize the  Navy’s acquisition, financial, program management, plant and wholesale  supply, and workforce management business processes across its  dispersed organizational components, and support about 86,000 users  when fully implemented.
Navy ERP is being developed in a series of increments using the Systems  Applications and Products (SAP) commercial software package,  augmented as needed by customized software. SAP consists of multiple,  integrated functional modules that perform a variety of business-related  tasks, such as finance and acquisition. The first increment, called  Template 1, is currently the only funded portion of the program and  consists of three releases (1.0, 1.1, and 1.2). Release1.0, Financial and  Acquisition, is the largest of the three releases in terms of Template 1  functional requirements. See table 2 for a description of these releases.
DON estimates the life-cycle cost for Template 1 to be about $2.4 billion,  including about $1 billion for acquisition and $1.4 billion for operations  and maintenance. The program office reported that approximately $600  million was spent from fiscal year 2004 through fiscal year 2008. For fiscal  year 2009, about $190 million is planned to be spent.
section_title: Program Oversight, Management, and Contractor Roles and Responsibilities
paragraphs: To acquire and deploy Navy ERP, DON established a program  management office within the Program Executive Office for Executive  Information Systems. The program office manages the program’s scope  and funding and is responsible for ensuring that the program meets its key  objectives. To accomplish this, the program office performs program  management functions, including testing, change control, and IV&V. In  addition, various DOD and DON organizations share program oversight  and review activities. A listing of key entities and their roles and  responsibilities is provided in table 3.
To deliver system and other program capabilities and to provide program  management support services, Navy ERP relies on multiple contractors, as  described in table 4.
section_title: Overview of Navy ERP’s Status
paragraphs: Template 1 of Navy ERP was originally planned to reach full operational  capability (FOC) in fiscal year 2011, and its original estimated life-cycle  cost was about $1.87 billion. The estimate was later baselined in August  2004 at about $2.0 billion. In December 2006 and again in September  2007, the program was rebaselined. FOC is now planned for fiscal year  2013, and the estimated life-cycle cost is about $2.4 billion (a 31 percent  increase over the original estimate).
The program is currently in the production and deployment phase of the  defense acquisition system, having completed the system development and  demonstration phase in September 2007. This was 17 months later than  the program’s original schedule set in August 2004, but on time according  to the revised schedule set in December 2006. Changes in the program’s  acquisition phase timeline are depicted in figure 1, and life-cycle cost  estimates are depicted in figure 2.
Release 1.0 was deployed at NAVAIR in October 2007, after passing  developmental testing and evaluation. Initial operational capability (IOC)  was achieved in May 2008, 22 months later than the baseline established in  August 2004, and 4 months later than the new baseline established in  September 2007. According to program documentation, these delays were  due, in part, to challenges experienced at NAVAIR in converting data from  legacy systems to run on the new system and implementing new business  procedures associated with the system. In light of the delays at NAVAIR in  achieving IOC, the deployment schedules for the other commands were  revised in 2008. Release 1.0 was deployed at NAVSUP in October 2008 as  scheduled, but deployment at SPAWAR was rescheduled for October 2009,  18 months later than planned, and at NAVSEA General Fund in October  2010, and at Navy Working Capital Fund in October 2011, each 12 months  later than planned.
Release 1.1 is currently being developed and tested, and is planned to be  deployed at NAVSUP in February 2010, 7 months later than planned, and  at the Navy’s Fleet and Industrial Supply Centers (FISC) starting in  February 2011. Changes in the deployment schedule are depicted in   figure 3.
section_title: Prior GAO Reviews of DOD Business System Investments Have Identified IT Management Weaknesses
paragraphs: We have previously reported that DOD has not effectively managed key  aspects of a number of business system investments, including Navy  ERP. Among other things, our reviews have identified weaknesses in such  areas as architectural alignment and informed investment decision  making, which are the focus of the Fiscal Year 2005 Defense Authorization  Act business system provisions. Our reviews have also identified  weaknesses in other system acquisition and investment management  areas, such as earned value management, economic justification, risk  management, requirements management, test management, and IV&V  practices.
In September 2008, we reported that DOD had implemented key  information technology (IT) management controls on Navy ERP to varying  degrees of effectiveness. For example, the control associated with  managing system requirements had been effectively implemented, and  important aspects of other controls had been at least partially  implemented, including those associated with economically justifying  investment in the program and proactively managing program risks.  However, other aspects of these controls, as well as the bulk of what was  needed to effectively implement earned value management, had not been  effectively implemented. As a result, the controls that were not effectively  implemented had, in part, contributed to sizable cost and schedule  shortfalls. Accordingly, we made recommendations aimed at improving  cost and schedule estimating, earned value management, and risk  management. DOD largely agreed with our recommendations.
In July 2008, we reported that DOD had not implemented key aspects of its  IT acquisition policies and related guidance on its Global Combat Support  System–Marine Corps (GCSS-MC) program. For example, we reported  that it had not economically justified its investment in GCSS-MC on the  basis of reliable estimates of both benefits and costs and had not  effectively implemented earned value management. Moreover, the  program office had not adequately managed all program risks and had not  used key system quality measures. We concluded that by not effectively  implementing these IT management controls, the program was at risk of  not delivering a system solution that optimally supports corporate mission  needs, maximizes capability mission performance, and is delivered on time  and within budget. Accordingly, we made recommendations aimed at  strengthening cost estimating, schedule estimating, risk management, and  system quality measurement. The department largely agreed with our  recommendations.
In July 2007, we reported that the Army’s approach for investing about $5  billion in three related programs—the General Fund Enterprise Business  System, Global Combat Support System-Army Field/Tactical, and Logistics  Modernization Program—did not include alignment with the Army  enterprise architecture or use of a portfolio-based business system  investment review process. Further, the Logistics Modernization  Program’s testing was not adequate and had contributed to the Army’s  inability to resolve operational problems. In addition, the Army had not  established an IV&V function for any of the three programs. Accordingly,  we recommended, among other things, use of an independent test team  and establishment of an IV&V function. DOD agreed with the  recommendations.
In December 2005, we reported that DON had not, among other things,  economically justified its ongoing and planned investment in the Naval  Tactical Command Support System (NTCSS) and had not adequately  conducted requirements management and testing activities. Specifically,  requirements were not traceable and developmental testing had not  identified problems that, subsequently, twice prevented the system from  passing operational testing. Moreover, DON had not effectively performed  key measurement, reporting, budgeting, and oversight activities. We  concluded that DON could not determine whether NTCSS, as defined and  as being developed, was the right solution to meet its strategic business  and technological needs. Accordingly, we recommended developing the  analytical basis necessary to know if continued investment in NTCSS  represented a prudent use of limited resources, and strengthening  program management, conditional upon a decision to proceed with further  investment in the program. The department largely agreed with our  recommendations.
In September 2005, we reported that while Navy ERP had the potential to  address some of DON’s financial management weaknesses, it faced  significant challenges and risks, including developing and implementing  system interfaces with other systems and converting data from legacy  systems. Also, we reported that the program was not capturing  quantitative data to assess effectiveness, and had not established an IV&V  function. We made recommendations to address these areas, including  having the IV&V agent report directly to program oversight bodies, as well  as the program manager. DOD generally agreed with our  recommendations, including that an IV&V function should be established.
However, it stated that the IV&V team would report directly to program  management who in turn would inform program oversight officials of any  significant IV&V results. In response, we reiterated the need for the IV&V  to be independent of the program and stated that performing IV&V  activities independently of the development and management functions  helps to ensure that the results are unbiased and based on objective  evidence. We also reiterated our support for the recommendation that the  IV&V reports be provided to the appropriate oversight body so that it can  determine whether any of the IV&V results are significant. We noted that  doing so would give added assurance that the results were objective and  that those responsible for authorizing future investments in Navy ERP  have the information needed to make informed decisions.
section_title: Key Aspects of Navy ERP Testing Have Been Effectively Managed
paragraphs: To be effectively managed, testing should be planned and conducted in a  structured and disciplined fashion. According to DOD and industry  guidance, system testing should be progressive, meaning that it should  consist of a series of test events that first focus on the performance of  individual system components, then on the performance of integrated  system components, followed by system-level tests that focus on whether  the entire system (or major system increments) is acceptable,  interoperable with related systems, and operationally suitable to users.  For this series of related test events to be conducted effectively, all test  events need to be, among other things, governed by a well-defined test  management structure and adequately planned. Further, the results of  each test event need to be captured and used to ensure that problems  discovered are disclosed and corrected.
Key aspects of Navy ERP testing have been effectively managed.  Specifically, the program has established an effective test management  structure, key development events were based on well-defined plans, the  results of all executed test events were documented, and problems found  during testing (i.e., test defects) were captured in a test management tool  and subsequently analyzed, resolved, and disclosed to decision makers.
Further, while we identified instances in which the tool did not contain  key data about defects that are needed to ensure that unauthorized  changes to the status of defects do not occur, the number of instances  found are not sufficient to conclude that the controls were not operating  effectively. Notwithstanding the missing data, this means that Navy ERP  testing has been performed in a manner that increases the chances that the  system will meet operational needs and perform as intended.
subsections: section_title: A Well-defined Test Management Structure Has Been Established
paragraphs: The program office has established a test management structure that  satisfies key elements of DOD and industry guidance. For example, the  program has developed a Test and Evaluation Master Plan (TEMP) that  defines the program’s test strategy. As provided for in the guidance, this  strategy consists of a sequence of tests in a simulated environment to  verify first that individual system parts meet specified requirements (i.e.,  development testing) and then verify that these combined parts perform as  intended in an operational environment (i.e., operational testing). As we  have previously reported, such a sequencing of test events is an effective  approach because it permits the source of defects to be isolated sooner,  before it is more difficult and expensive to address.
More specifically, the strategy includes a sequence of developmental tests  for each release consisting of three cycles of integrated system testing  (IST) followed by user acceptance testing (UAT). Following development  testing, the sequence of operational tests includes the Navy’s independent  operational test agency conducting initial operational test and evaluation  (IOT&E) and then follow-on operational test and evaluation (FOT&E), as  needed, to validate the resolution of deficiencies found during IOT&E. See  table 5 for a brief description of the purpose of each test activity, and  figure 4 for the schedule of Release 1.0 and 1.1 test activities.
section_title: Well-defined Plans for Developmental Test Events Were Developed
paragraphs: According to relevant guidance, test activities should be governed by  well-defined and approved plans. Among other things, such plans are to  include a defect triage process, metrics for measuring progress in  resolving defects, test entrance and exit criteria, and test readiness  reviews.
Each developmental test event for Release 1.0 (i.e., each cycle of  integrated systems testing and user acceptance testing) was based on a  well-defined test plan. For example, each plan provided for conducting  daily triage meetings to (1) assign new defe documented criteria, (2) defects in the test management tool, and (3) address other de testing issues. Further, each plan included defect metrics, such as t number of  plan specified that testing was not complete until all major defects found  during the cycle were resolved, and all unresolved defects’ impact on the  olding  next test event were understood. Further, the plans provided for h test readiness reviews to review test results as a condition for proceeding  to the next  activities in activities no increasing t and perform as inten cts a criticality level using   record new defects and update the status of old  defects found and corrected and their age. In addition, each  event. By ensuring that plans for key  clude these aspects of effective  test planning, the risk of test  t being effectively and efficiently p erformed is reduced, thus  he chances that the system will  ded.
section_title: Test Results Were Documented and Reported, but Key Information about Changes to the Status of Reported Defects Was Not Always Recorded
paragraphs: According to industry guidance, effective system testing includes  capturing, analyzing, resolving, and disclosing to decision makers the  status of problems found during testing (i.e., test defects). Further, this  guidance states that these results should be collected and stored accordin to defined procedures and placed under appropriate levels of control to  ensure that any changes to the results are fully documented.
To the program’s credit, the relevant testing organizations have  documented test defects in accordance with defined plans. For example ,  daily triage meetings involving the test team lead, testers, and functional  experts were held to review each new defect, assign it a criticality level,  and designate someone responsible for resolving it and for monitoring  and  updating its resolution in the test management tool. Further, test readines reviews were conducted at which entrance and exit criteria for each key  test event were evaluated before proceeding to the next event. As p art of  these reviews, the program office and oversight officials, command  representatives, and test officials reviewed the results of test events to  ensure, among other things, that significant defects were closed and that  there were no unresolved  defects that could affect execution of the next  test event.
However, the test management tool did not always contain key data recorded defects that are needed to ensure that unauthorized changes to  the status of defects do not occur. According to information systems  auditing guidelines, audit tools should be in place to monitor user access  to systems to detect possible errors or unauthorized changes. For Navy  ERP, this was not always the case. Specifically, while the tool capability to track changes to test defects in a history log, our analysis of  80 randomly selected defects in the tool disclosed two instances the tool did not record when a change in the defect’s status was made or  who made the change. In addition, our analysis of 12 additional defects that were potential anomalies disclosed two additional instances where the tool did not record when a change was made and who made it. While  our sample size and results do not support any conclusions as to the  overall effectiveness of the controls in place for recording and tracking  test defect status changes, they do show that it is possible that changes  es.  can be made without a complete audit trail surrounding those chang After we shared our results with program officials, they stated that they provided each instance for resolution to the vendor responsible for the  tracking tool. These officials attributed these instances to vendor updates to the tool t hat caused the history settings to default to “off.” To address  this weakness, they added that they are now ensuring that the history log are set correctly after any update to the tool. This addition is a positive  step because without an effective information system access audit tool,  the probability of test defect status errors or unauthorized changes is  increased.
section_title: System Changes Have Been Controlled, but Their Cost and Schedule Impacts Were Not Sufficiently Considered
paragraphs: Industry best practices and DOD guidance recognize the importance of  system change control when developing and maintaining a system. Once  the composition of a system is sufficiently defined, a baseline  configuration is normally established, and changes to that baseline are  placed under a disciplined change control process to ensure that  unjustified and unauthorized changes are not introduced. Elements of  disciplined change control include (1) formally documenting a change  control process, (2) rigorously adhering to the documented process, and  (3) adopting objective criteria for considering a proposed change,  including its estimated cost and schedule impact.
To its credit, the Navy ERP program has formally documented a change  control process. Specifically, it has a plan and related procedures that  include the  made to the system are properly identified, developed, and implemented in  a  defined and controlled environment. It also is using an automated tool to  capture and track the disposition of each change request. Further, it has  defined roles and responsibilities and a related decision-making structure  for reviewing and approving system changes. In this regard, the program  has established a hierarchy of review and approval boards, including a  Configuration Control Board to review all changes and a Configuration  Management Board to further review changes estimated to require more  than 100 hours or $25,000 to implement. Furthermore, a Navy ERP Senior  Integration Board was recently established to review and approve requests  to add, delete, or change the program’s requirements. In addition, the  change control process states that the decisions are to be based on, amon others, the system engineering and earned value management (i.e., co and schedule) impacts the change w number of work hours that will be required to effect the change. Table 7  purpose and scope of the process—to ensure that any changes  st  ill introduce, such as the estimated  provides a brief description of the decision-making authorities and boards  and their respective roles and responsibilities.
Navy ERP is largely adhering to its documented change control process.  Specifically, our review of a random sample of 60 change requests and  minutes of related board meetings held between May 2006 and April 20 showed that the change requests were captured and tracked using an  automated tool, and they were reviewed and approved by the designated decision-making authorities and boards, in accordance with the program documented process.
However, the program has not sufficiently or co cost and schedule impacts of proposed changes. Our analysis of the  random sample of 60 change requests, including our review of related  board meeting minutes, showed no evidence that cost and schedule  impacts were identified or that they were considered. Specifically, we did  not see evidence that the cost and schedule impacts of these change  requests were assessed. According to program officials, the cost and  schedule impacts of each change were discussed at control board  meetings. In addition, they provided two change requests to demonstrate  this. However, while these change requests did include schedule impact,  they did not include the anticipated cost impact of proposed changes.  Rather, these two, as well as those in our random sample, included the  estimated number of work hours required to implement the change.  nsistently considered the  Because the cost of any proposed change depends on other factors  besides work hours, such as labor rates, the estimated number of work  hours is not sufficient for considering the cost impact of a change. I absence of verifiable evidence that cost and schedule impacts were  consistently considered, approval authorities do not appear to have b provided key information needed to fully inform their decisions on  whether or not to approve a change. System changes that are approved  without a full understanding of their cost and schedule impacts could  result in unwarranted cost increases and schedule delays.
section_title: Navy ERP IV&V Function Is Not Independent and Has Not Been Fully Performed
paragraphs: The purpose of IV&V is to independently ensure that program processes  and products meet quality standards. The use of an IV&V function is  recognized as an effective practice for large and complex system  development and acquisition programs, like Navy ERP, as it provides  objective insight into the program’s processes and associated work  p To be effective, verification and validation activities should be roducts.performed by an entity that is managerially independent of the system  development and management processes and products that are being  reviewed. Among other things, such independence helps to ensure that  the results are unbiased and based on objective evidence.
The Navy has not effectively managed its IV&V function because it has not  ensured that the contra ctor performing this function is independent of the  products and processes that this contractor is reviewing and because it  has not ensured that the contractor is meeting contractual requirements.  In June 2006, DON awarded a professional support services contract  General Dynamics Information Technology (GDIT), to include  responsibilities for, among other things, IV&V, program management support, and delivery of releases according to cost and schedule  constraints. According to the program manager, the contractor’s IV&Vfunction is organizationally separate from, and thus independent of, th e  contractor’s Navy ERP system development function. However, the  subcontractor performing the IV&V function is also performing release  management. According to the GDIT contract, the release manager  is  responsible for developing and deploying a system release that meets  operational requirements within the program’s cost and schedule  constraints, but it also states that the IV&V function is resp supporting the government in its review, approval, and acceptance of Navy  RP products (e.g., releases). The contract also states that GDIT is eligible  E for an optional award fee payment based on its performance in meeting,  among other things, these cost and schedule constraints. Because  performance of the system development and management role ma contractor potentially unable to render impartial assistance to the  government in performing the IV&V function, the contractor has an  inherent conflict of interest relative to meeting cost and schedule  commitments and disclosing the results of verification and validation  reviews that may affect its ability to do so.
The IV&V function’s lack of independence is amplified by t reports directly and solely to the program manager. As we have previously  reported, the IV&V function should report the issues or weaknesses that  increase the risks associated with the project to program oversight  officials, as well as to program management, to better ensure that the  verification and validation results are objective and that the officials  responsible for making program investment decisions are fully informed .  Furthermore, these officials, once informed, can ensure that the issues or  weaknesses reported are promptly addressed.
Without ensuring sufficient managerial independence, valuable  information may not reach decision makers, potentially leading to the  release of a system that does not adequately meet users’ needs and o as intended.
Beyond the IV&V function’s lack of independence, the program office h not ensured that the subcontractor has produced the range of deliv that were contractually required and defined in the IV&V plan. For  example, the contract and plan call for weekly and monthly reports  identifying weaknesses in program processes and recommendations for  improvement, a work plan for accomplishing IV&V tasks, and associated  assessment reports that follow the System Engineering Plan and program  schedule. However, the IV&V contractor has largely not delivered these  products. Specifically, until recently, it did not produce a work plan and  erables  only monthly reports were delivered, and these reports only list meetings  that the IV&V contactor attended and documents that it reviewed. They d not, for example, identify program weaknesses or provide  recommendations for improvement. According to program officials, they  have relied on oral reports from the subcontractor at weekly meetings,  and these lessons learned have been incorporated into program guid According to the contractor, the Navy has expended about $1.8 million  between June 2006 and September 2008 for IV&V activities, with an  additional $249,000 planned to be spent in fiscal year 2009.  ance.
Following our inquiries about an IV&V work plan, the IV&V contractor  developed such a plan in October 2008, more than 2 years after the  contract was awarded, that lists program activities and processes to be  assessed, such as configuration management and testing. While this  does not include time frames for starting and completing these  assessments, meeting minutes show that the status of assessments ha been discussed with the program manager during IV&V review meetings.  The first planned assessment was delivered to the program in March 2009  he program’s configuration  and provides recommendations for improving t management process, such as using the automated tool to produce certain  reports and enhancing training to understand how the tool is use Further, program officials stated that they have also recently begun  requiring the contractor to provide formal quarterly reports, the first of  which was de of this quarterly report shows that it provides recommendations for  improving the program’s risk management process and organizational  change management strategy.  d.  livered to the program manager in January 2009. Our review  Notwithstanding the recent steps that the program office has taken,  nevertheless lacks an independent perspective on the program’s products and management processes.
section_title: Conclusions
paragraphs: DOD’s successes in delivering large-scale business systems, such as Navy  ERP, are in large part determined by the extent to which it employs the  kind of rigorous and disciplined IT management controls that are reflected  in department policies and related guidance. While implementing these  controls does not guarantee a successful program, it does minimize a  program’s exposure to risk and thus the likelihood that it will fall short of  expectations. In the case of Navy ERP, living up to expectations is  important because the program is large, complex, and critical to  addressing the department’s long-standing problems related to financial  transparency and asset visibility.
The Navy ERP program office has largely implemented a range of effective controls associated with system testing and change control, including  acting quickly to address issues with the audit log for its test manage tool, but more can be done to ensure that the cost and sched proposed changes are explicitly documented and considered when  decisions are reached. Moreover, while the program office has contract for IV&V activities, it has not ensured that the contractor is indepen the products and processes that it is to review and has not held the  contractor accountable for producing the full range of IV&V deliverables  required under the contract. Moreover, it has not ensured that it s IV&V  contractor is accountable to a level of management above the program  office, as we previously recommended. Notwithstanding the program  office’s considerable effectiveness in how it has managed both system  testing and change control, these weaknesses increase the risk of investing  in system changes that are not economically justified and unnecessarily  limit the value that an IV&V agent can bring to a program like Navy E By addressing these weaknesses, the department can better ensure t taxpayer dollars are wisely and prudently invested.
RP.
section_title: Recommendations for Executive Action
paragraphs: To strengthen the management of Navy ERP’s change control process,  recommend that the Secretary of Defense direct the Secretary of the Navy,  through the appropriate chain of command, to (1) revise the Navy ERP  procedures for controlling system changes to explicitly require that a  proposed change’s life-cycle cost impact be estimated and considered in  making change request decisions and (2) capture the cost and schedule  impacts of each proposed change in the Navy ERP automated change  c ontrol tracking tool.
To increase the value of Navy ERP IV&V, we recommend that the  Secretary of Defense direct the Secretary of the Navy, through the  appropriate chain of command, to (1) stop performance of the IV&V  function under the existing contract and (2) engage the services of a n IV&V agent that is independent of all Navy ERP management,  development, testing, and deployment activities that it may review. In  addition, we reiterate our prior recommendation relative to ensur the Navy ERP IV&V agent report directly to program oversight officials,  while concurrently sharing IV&V results with the program office.
section_title: Agency Comments
paragraphs: In written comments on a draft of this report, signed by the Assistan Deputy Chief Management Officer and reprinted in appendix II, the  department concurred with our recommendations, and stated that it will  take the appropriate corrective actions within the next 7 months.
We are sending copies of this report to interested congressional  committees; the Director, Office of Management and Budget; the  Congressional Budget Office; and the Secretary of Defense. The report  also is available at no charge on our Web site at http://www.gao.gov.
If you or your staffs have any questions on matters discussed in this  report, please contact me at (202) 512-3439 or hiter@gao.gov. Contact  points for our Offices of Congressional Relations and be found on the last page of this report. GAO staff who made major  c ontributions to this report are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to determine whether (1) system testing is being  effectively managed, (2) system changes are being effectively con and (3) independent verification and validation (IV&V) activities a effectively managed for the Navy Enterprise Resource Planning (ERP)  program.
To determine if Navy ERP testing is being effectively managed, we  reviewed relevant documentation, such as the Test and Evaluation Mas Plan and test reports and compared them with relevant federal and related  res  guidance. Further, we reviewed development test plans and procedu for each test event and compared them with best practices to determine  whether well-defined plans were developed. We also examined test results  and reports, including test readiness review documentation and compared  them against plans to determine whether they had been executed in  accordance with the plans. Moreover, to determine the extent to which  test defect data were being captured, analyzed, and reported, we inspected  80 randomly selected defects from a sample of 2,258 def program’s test management system. In addition, we re logs associated with each appropriate levels the results were fully documented. Th percent tolerable error rate at the 95 pe we found 0 problems in the error rate was less than 4 percent. In addition, we interviewed  cognizant officials, including the program’s test lead and the Navy’s  independent operational testers, about their roles and responsibilities for  test management.   of these 80 defects to determine whether   of control were in place to ensure that any changes to  is sample was designed with a 5  rcent level of confidence, so that, if   our sample, we could conclude statistically that  To determine if Navy ERP changes are being effectively controlled, we  reviewed relevant program documentation, such as the change control  policies, plans, and procedures, and compared them with relevant federal  and industry guidance. Further, to determine the extent to which the  program is reviewing and approving change requests according to its  documented plans and procedures, we inspected 60 randomly selected  change requests in the program’s configuration management system. In  addition, we reviewed the change request forms associated with these 60  change requests and related control board meeting minutes to determine  whether objective criteria for considering a proposed change, including  estimated cost or schedule impacts, were adopted. In addition, we  interviewed cognizant officials, including the program manager and  systems engineer, about their roles and responsibilities for reviewing,  approving, and tracking change requests.
To determine if IV&V activities are being effectively managed we revie Navy ERP’s IV&V contract, strategy, and plans and compared them with  relevant industry guidance. We also analyzed the contractual relationships  relative to legal standards that govern organizational conflict of interest . In  addition, w assessment report, and a quarterly report, to determine the extent to  which contract requirements were met. We interviewed contractor a nd  program officials about their roles and responsibilities for IV&V and to  determine the extent to which the program’s IV&V function is  independent.  e examined IV&V monthly status reports, work plans, an  We conducted this performance audit at Department of Defense offices in  the Washington, D.C., metropolitan area; Annapolis, Maryland; and  Norfolk, Virginia; from August 2008 to September 2009, in accordance with  generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on our  audit objectives.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, key contributors to this report  were Neelaxi Lakhmani, Assistant Director; Monica Anatalio; Carl Barden;  Neil Doherty; Cheryl Dottermusch; Lee McCracken; Karl Seifert; Adam  Vodraska; Shaunyce Wallace; and Jeffrey Woodward.