DOD Business Systems Modernization: Planned Investment in Navy Program to Create Cashless Shipboard Environment Needs to Be Justified and Better Managed
Published: 2008-09-08T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: GAO has designated the Department of Defense's (DOD) multi-billion dollar business systems modernization efforts as high risk, in part because key information technology (IT) management controls have not been implemented on key investments, such as the Navy Cash program. Initiated in 2001, Navy Cash is a joint Department of the Navy (DON) and Department of the Treasury Financial Management Service (FMS) program to create a cashless environment on ships using smart card technology, and is estimated to cost about $320 million to fully deploy. As requested, GAO analyzed whether DON is effectively implementing IT management controls on the program, including architectural alignment, economic justification, requirements development and management, risk management, security management, and system quality measurement against relevant guidance.
section_title: What GAO Found
paragraphs: Key IT management controls have not been effectively implemented on Navy Cash, to the point that further investment in this program, as it is currently defined, has not been shown to be a prudent and judicious use of scarce modernization resources. In particular, Navy Cash has not been (1) assessed and defined in a way to ensure that it is not duplicative of programs in the Air Force and the Army that use smart card technology for electronic retail transactions and (2) economically justified on the basis of reliable analyses of estimated costs and expected benefits over the program's life. As a result, DON cannot demonstrate that the investment alternative that it is pursuing is the most cost-effective solution to satisfying its mission needs. Moreover, other management controls, which are intended to maximize the chances of delivering defined and justified system capabilities and benefits on time and within budget, have not been effectively implemented. System requirements have not been effectively managed. For example, neither policies nor plans that define how system requirements are to be managed, nor an approved baseline set of requirements that are justified and needed to cost-effectively meet mission needs, exist. Instead, requirements are addressed reactively through requests for changes to the system based primarily on the availability of funding. Program risks have not been effectively managed. In particular, plans, processes, and procedures that provide for identifying, mitigating, and disclosing risks have not been defined, nor have risk-related roles and responsibilities for key stakeholders. System security has not been effectively managed, thus putting the confidentiality, integrity, and availability of deployed and operating shipboard devices, applications, and data at increased risk of being compromised. For example, the mitigation of system vulnerabilities by applying software patches has not been effectively implemented. Key aspects of system quality are not being effectively measured. For example, data for determining trends in unresolved system change requests, which is an indicator of system stability, as well as user feedback on system satisfaction, are not being collected and used. Program oversight and management officials acknowledged these weaknesses and cited turnover of staff in key positions and their primary focus on deploying Navy Cash as reasons for the state of some of these IT management controls. Collectively, this means that, after investing about 6 years and $132 million on Navy Cash and planning to invest an additional $60 million to further develop the program, the department has yet to demonstrate through verifiable analysis and evidence that the program, as currently defined, is justified. Moreover, even if further investment was to be demonstrated, the manner in which the delivery of program capabilities is being managed is not adequate. As a result, the program is at risk of delivering a system solution that falls short of cost, schedule, and performance expectations.

section_title: Letter
paragraphs: For decades, the Department of Defense (DOD) has been challenged in  For decades, the Department of Defense (DOD) has been challenged in  modernizing its business systems. In 1995, we designated the  modernizing its business systems. Among our reasons for doing so are the enormous size and  today.  cashless environment on ships through the use of smart card technology.
Business systems include financial and non-financial systems that support DOD’s business  operations, such as civilian personnel, finance, health, logistics, military personnel,  procurement, and transportation.
GAO, High-Risk Series: An Update, GAO-07-310 (Washington, D.C.: January 2007).
Smart cards are plastic devices that are about the size of a credit card and contain an  embedded integrated circuit chip capable of storing and processing data. The term “smart  card” may also be used to refer to cards with a computer chip, also referred to as an            e-purse, that store information to be processed by hardware such as point-of-sale terminals  or card access devices. million is being funded by DON and $100 million is being funded by FMS.  The system is to be fully deployed in fiscal year 2011.
As agreed, our objective was to determine whether DON is effectively  implementing IT management controls on Navy Cash. To accomplish this,  we analyzed a range of program documentation and interviewed cognizant  officials relative to the following IT management controls: architectural  alignment, economic justification, requirements development and  management, risk management, security management, and system quality  measurement. In doing so, we compared DON’s efforts in each control  area to relevant federal and industry requirements and guidance.
We conducted this performance audit between June 2007 and September  2008, in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objective. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objective. Additional details on our  objective, scope, and methodology are in appendix I.
subsections: section_title: Background
paragraphs: DON’s primary mission is to organize, train, maintain, and equip combat- ready naval forces capable of winning the global war on terror and any  other armed conflict, deterring aggression by would-be foes, preserving  freedom of the seas, and promoting peace and security. To support this  mission, DON performs a variety of interrelated and interdependent  business functions (e.g., acquisition and financial management), relying  heavily on IT systems. In fiscal year 2008, DON’s IT budget was about $2.7  billion, of which $2.2 billion was allocated to operations and maintenance  of existing systems and the remaining $500 million to systems in  development and modernization. Of the approximately 3,000 business  systems that DOD reports in its current inventory, DON accounts for 904,  or about 30 percent, of the total. The Navy Cash system is one such system  investment.
subsections: section_title: Navy Cash: A Brief Description
paragraphs: In 2001, DON initiated Navy Cash in partnership with Treasury’s FMS to  enable sailors and marines to use smart cards that store monetary value,  also known as stored value cards, to make retail purchases and conduct  banking transactions while on ships and ashore. The program builds upon  capabilities that have been incrementally introduced from previously  deployed systems. (Table 1 summarizes these systems and their  capabilities and limitations.)
According to DOD, Navy Cash’s key objectives include introducing  workload efficiencies and improving the quality of life for sailors and  marines by  reducing the amount of currency on ships, which lowers costs associated  with cash handling activities;    enabling sailors and marines to conduct ashore banking transactions from    enabling sailors and marines to conduct banking or retail transactions  while ashore (wherever these branded debit cards are accepted).
Navy Cash consists of various equipment and devices, including servers  that connect to the ship’s local area network as well as point-of-sale  terminals and ATMs that communicate with Navy Cash smart cards. These  cards contain an electronic chip that stores monetary value and interacts  with the various devices for conducting electronic retail purchases and  personal banking transactions on the ships. On shore, cardholders can  access their Navy Cash accounts via ATMs worldwide or conduct retail  purchases using the card’s magnetic stripe, which provides a debit card  feature. According to program officials, while ashore, sailors and marines  have access to over 1,000,000 ATMs and 23 million merchants worldwide.
Navy Cash uses a ship’s Automated Digital Network System to access  satellite communications systems, and then transmits transaction files off  the ship through fleet network operations centers to a financial agent (i.e.,  bank) ashore. To do so, it uses a store-and-forward process to batch  transactions together and transmit them off the ship typically during non- peak evening hours. These transactions are then processed in a manner  similar to personal check processing through the Automated Clearing  House. Figure 1 is a simplified illustration of the Navy Cash network used  to transmit these transactions.
Originally, the program was expected to be fully deployed and reach full  operational capability by December 2008 at an estimated cost of about  $100 million over a 6-year life cycle. The program office now expects the  program to reach full operational capability in fiscal year 2011, and it  estimates the program’s 14-year life cycle cost to be about $320 million, of  which about $100 million is to be funded by FMS. Of the $320 million,  about $136 million is for development and modernization, and about $184  million is for operations and maintenance. From fiscal year 2002 to 2007,  DON and FMS reported that approximately $132 million has been spent on  the program, of which $47 million is FMS’s cost. Of the $188 million  expected to be spent (fiscal years 2008-2015), about $57 million is for  development and modernization. (See fig. 2 for a breakdown of the actual  and planned costs.)
When fully deployed, the program office estimates that Navy Cash could  process over $350 million annually in transactions initiated by about  170,000 sailors and marines worldwide on approximately 160 ships. As of  April 2008, the program has been deployed to approximately 130 ships.
section_title: Navy Cash Oversight and Management Roles and Responsibilities
paragraphs: To manage the acquisition and deployment of Navy Cash, DON established  a program management office within the Naval Supply Systems Command  (NAVSUP). As authorized by statute and because of its experience in  developing stored value card programs for other military departments,  NAVSUP has partnered with FMS to develop Navy Cash. In February 2001,  NAVSUP and FMS signed a memorandum of agreement that, among other  things, delineated their respective program roles and responsibilities.  According to the agreement, NAVSUP, through the Navy Cash program  office, is responsible for managing the acquisition of the program,  including managing system requirements and developing program cost and  benefit estimates. According to DOD and other relevant guidance,  acquisition management includes, among other things, such key IT  management control areas as architectural alignment, economic  justification, requirements management, risk management, security  management, and system quality measurement.
Also according to the agreement, FMS, through a designated financial  agent, is to (1) provide for all financial services (i.e., manage the funds  distributed through Navy Cash) and (2) develop, test, operate, and  maintain the system’s software (e.g., terminal and accounting  applications) and hardware (e.g., accounting servers, smart cards). In  short, the financial agent acts as the depository bank, holding and  managing the pool of sailor and marine funds, including accounting for the  funds and settling transactions processed. FMS is also responsible for  tracking and overseeing the financial agent’s provision of services, as  defined in a financial agency agreement between FMS and the agent. (See  fig. 3 for DON and FMS roles and relationships for Navy Cash.)
In addition, various other organizations share program oversight and  review activities. A listing of key entities and their roles and  responsibilities can be found in table 2.
section_title: Use of IT Management Controls Maximizes Chances for Success
paragraphs: Effective IT management controls are grounded in tried and proven  methods, processes, techniques, and activities that organizations define  and use to minimize program risks and maximize the chances of a  program’s success. Using such best practices can result in better  outcomes, including cost savings, improved service and product quality,  and a better return on investment. For example, two software engineering  analyses of nearly 200 systems acquisitions projects indicate that teams  using systems acquisition best practices produced cost savings of at least  11 percent over similar projects conducted by teams that did not employ  the kind of rigor and discipline embedded in these practices. In addition,  our research shows that best practices are a significant factor in  successful acquisition outcomes, including increasing the likelihood that  programs and projects will be executed within cost and schedule  estimates.
We and others have identified and promoted the use of a number of best  practices associated with acquiring IT systems. See table 3 for a  description of several of these activities.
section_title: Prior GAO Reviews Have Identified IT Management Control Weaknesses on DOD Business System Investments
paragraphs: We have previously reported that DOD has not effectively managed a  number of business system investments. Among other things, our reviews  of individual system investments have identified weaknesses in such  things as architectural alignment and informed investment decision  making, which are also the focus areas of the Ronald W. Reagan National  Defense Authorization Act for Fiscal Year 2005 business system  provisions. Our reviews have also identified weaknesses in other system  acquisition and investment management areas—such as economic  justification, requirements management, and risk management.
Recently, for example, we reported that the Army’s approach for investing  about $5 billion over the next several years in its General Fund Enterprise  Business System, Global Combat Support System-Army Field/Tactical,  and Logistics Modernization Program did not include alignment with Army  enterprise architecture or use of a portfolio-based business system  investment review process. Moreover, we reported that the Army did not  have reliable processes, such as an independent verification and validation  function, or analyses, such as economic analyses, to support its  management of these programs. We concluded that until the Army adopts  a business system investment management approach that provides for  reviewing groups of systems and making enterprise decisions on how  these groups will collectively interoperate to provide a desired capability,  it runs the risk of investing significant resources in business systems that  do not provide the desired functionality and efficiency. Accordingly, we  made recommendations aimed at improving the department’s efforts to  achieve total asset visibility and enhancing its efforts to improve its  control and accountability over business system investments. The  department agreed with our recommendations.
We also reported that DON had not, among other things, economically  justified its ongoing and planned investment in the Naval Tactical  Command Support System (NTCSS) and had not invested in NTCSS  within the context of a well-defined DOD or DON enterprise architecture.  In addition, we reported that DON had not effectively performed key  measurement, reporting, budgeting, and oversight activities, and had not  adequately conducted requirements management and testing activities. We  concluded that without this information, DON could not determine  whether NTCSS as defined, and as being developed, is the right solution to  meet its strategic business and technological needs. Accordingly, we  recommended that the department develop the analytical basis to  determine if continued investment in NTCSS represents prudent use of  limited resources and to strengthen management of the program,  conditional upon a decision to proceed with further investment in the  program. The department largely agreed with these recommendations.
In addition, we reported that the Army had not defined and developed its  Transportation Coordinators’ Automated Information for Movements  System II—a joint services system with the goal of helping to manage the  movement of forces and equipment within the United States and abroad— in the context of a DOD enterprise architecture. We also reported that the  Army had not economically justified the program on the basis of reliable  estimates of life cycle costs and benefits and had not effectively  implemented risk management. As a result, we concluded that the Army  did not know if its investment in this program, as planned, is warranted or  represents a prudent use of limited DOD resources. Accordingly, we  recommended that DOD, among other things, develop the analytical basis  needed to determine if continued investment in this program, as planned,  represents prudent use of limited defense resources. In response, the  department largely agreed with our recommendations, and has since  reduced the program’s scope by canceling planned investments.
section_title: Key IT Management Controls Have Not Been Effectively Implemented on Navy Cash
paragraphs: DOD acquisition policies and related federal guidance provide a  framework within which to manage system investments, like Navy Cash.  Effective implementation of this framework can minimize program risks  and better ensure that system investments are defined in a way to  optimally support mission operations and performance, as well as deliver  promised system capabilities and benefits on time and within budget. Thus  far, key IT management controls associated with this framework have not  been implemented on Navy Cash. In particular, the program’s overlap with  and duplication of other DOD programs has not been assessed, and the  program has not been economically justified on the basis of reliable  estimates of life cycle costs and benefits. As a result, the program, as  defined, has not been shown to be the most cost-effective investment  option.
Even if investment in the proposed Navy Cash solution is shown to be a  wise and prudent course of action, the manner in which Navy Cash is  being acquired and deployed is not adequate because (1) requirements  have not been adequately developed and managed; (2) program risks have  not been effectively managed; (3) security has not been effectively  managed; and (4) system quality has not been adequately measured. As a  result, the system will likely experience performance shortfalls and cost  more and take longer to implement and maintain than necessary.
Program officials acknowledged these weaknesses and attributed them to,  among other things, turnover of staff in key positions and their focus on  deploying the system. Further, they stated that addressing these  weaknesses has not been a top program priority because Navy Cash has  been deployed to and is operating on about 80 percent of the ships.  Nevertheless, about $60 million in development and modernization  funding remains to be spent on this program. As a result, it is important  that all these weaknesses be addressed to reduce the risk of delivering a  system solution that falls short of expectations.
subsections: section_title: Key Controls for Justifying Planned Investment in Navy Cash Have Not Been Effectively Implemented
paragraphs: Investment in the proposed Navy Cash solution has not been adequately  justified. Specifically, the system solution has not been assessed relative to  other DOD programs that employ smart cards for electronic retail  transactions. Moreover, it has not been economically justified on the basis  of reliable estimates of cost and benefits over the system’s expected life.  As a result, planned investment in the system, as defined, may not be a  cost-effective course of action.
subsections: section_title: Navy Cash Duplication with Other DOD Programs Has Not Been Assessed
paragraphs: DOD’s acquisition policies and guidance, as well as federal and best  practice guidance, recognize the importance of investing in business  systems within the context of an enterprise architecture. Moreover, the  Ronald W. Reagan National Defense Authorization Act for Fiscal Year  2005 requires that defense business systems be compliant with the  federated BEA. Our research and experience in reviewing federal  agencies show that making investments without the context of a well- defined enterprise architecture often results in systems that are, among  other things, duplicative of other systems.
Navy Cash has not been assessed and defined in a way to ensure that it is  not duplicative of the Eagle Cash and EZpay programs, both of which  provide for the use of smart card technology for electronic retail  transactions in support of the Air Force and the Army. Within DOD, the  means for avoiding business system duplication and overlap is the  department’s process for assessing compliance with the DOD BEA and its  associated investment review and decision making processes. In 2005,  2006, and 2007, Navy Cash was evaluated for compliance with the BEA.  However, the BEA does not contain business activities that Navy Cash  supports. According to officials from DOD’s Business Transformation  Agency, which is responsible for DOD’s BEA, these business activities are  not included nor are they planned for inclusion in the BEA, because the  capabilities provided by Navy Cash relate strictly to personal banking,  which is outside of the current scope of the BEA. As a result, compliance  could not be assessed beyond concluding that Navy Cash was compliant  because it did not conflict with the BEA. Moreover, even if the BEA  included the business activities that Navy Cash supports, the program’s  ability to assess BEA compliance would have been limited because the  program office did not develop a complete set of system-level architecture  products needed to perform a meaningful compliance assessment. Thus,  Navy Cash’s potential overlap and duplication with similar programs is not  sufficiently understood.
According to program officials, Navy Cash is not duplicative of Eagle Cash  and EZpay because it is designed to operate on ships at sea, which do not  maintain constant network connectivity with on shore networks.  Therefore, they said that it requires different communications and  financial transaction capabilities than the other two stored value card  programs. We agree that there are important differences between the  programs. However, they all perform chip-based financial transactions,  and thus opportunities may exist for them to provide or reuse shared  system services, as well as to merge into a DOD-wide stored value card  program. According to program officials, overlap and duplication among  the programs was not assessed. This means that aspects of Navy Cash  could be potentially duplicative of these other programs, and thus DOD  may not be pursuing the most cost-effective solution to meet its mission  needs. In this regard, the program’s Milestone Decision Authority told us  that the differences between Navy Cash and other stored value card  programs are minimal and stated that officials with the three stored value  card programs have recently begun discussions with FMS on how to  collaborate and possibly move towards one system solution.
section_title: Navy Cash Has Not Been Economically Justified
paragraphs: Investment in Navy Cash has not been economically justified on the basis  of a reliable analysis of estimated system costs and expected benefits over  the life of the program. Specifically, according to the latest economic  analysis, the program is expected to produce estimated benefits of about  $133 million for an estimated cost of about $100 million. However, the cost  estimate is not reliable, because the program’s 2002 economic analysis is 6  years old and is based on a cost estimate of about $100 million that was  not derived in accordance with effective estimating practices, such as  including all costs over the system’s life cycle, and adjusting the estimate  to account for program risks and material program changes. Further, this  economic analysis did not comply with applicable federal guidance. For  example, it did not adequately consider all relevant alternatives, and it  erroneously counted $40 million as cost savings rather than transfers  (i.e., shift of control over spending of resources from one group to another  that do not result in an economic gain). Further, the economic analysis has  yet to be validated using actual data on the accrual of benefits. Without an  economic analysis that is reliable, DON’s ongoing and planned investment  in Navy Cash lacks justification as a cost-effective course of action.
Economic Analysis Used a Cost Estimate That Omits Relevant  Costs and Was Not Derived Using Key Estimating Practices  A reliable cost estimate is an essential element for informed investment  decision making, realistic budget formulation and program resourcing,  meaningful progress measurement, proactive course correction, and  accountability for results. According to the Office of Management and  Budget (OMB), programs must maintain current and well-documented  estimates of program costs, and these estimates must span the full  expected life of the program. Without reliable estimates, programs cannot  be adequately justified on the basis of reliable costs and benefits and they  are at increased risk of experiencing cost overruns, missed deadlines, and  performance shortfalls.
Our research has identified a number of best practices for effective  program cost estimating, and we have issued guidance that associates  these practices with four characteristics of a reliable cost estimate.  Specifically, estimates need to be:    Comprehensive: The cost estimates should include both government and  financial agent costs over the program’s full life cycle, from the inception  of the program through design, development, deployment, and operation  and maintenance to retirement. They should also provide a level of detail  appropriate to ensure that cost elements are neither omitted nor double  counted, and include documentation of all cost-influencing ground rules  and assumptions.
Well-documented: The cost estimates should have clearly-defined  purposes, and be supported by documented descriptions of key program  or system characteristics (e.g., relationships with other systems,  performance parameters). Additionally, they should capture in writing  such things as the source data used and their significance, the calculations  performed and their results, and the rationale for choosing a particular  estimating method or reference. Moreover, this information should be  captured in such a way that the data used to derive the estimate can be  traced back to, and verified against, their sources.
Accurate: The cost estimates should provide for results that are unbiased  and not be overly conservative or optimistic (i.e., should represent the  most likely costs). In addition, the estimates should be updated regularly  to reflect material changes in the program, and steps should be taken to  minimize mathematical mistakes and their significance. The estimates  should also be grounded in a historical record of cost estimating and  actual experiences on comparable programs.
Credible: The cost estimates should discuss any limitations in the analysis  performed that are due to uncertainty or biases surrounding data or  assumptions. Further, the estimates’ derivation should provide for varying  any major assumptions and recalculating outcomes based on sensitivity  analyses, and the estimates’ associated risks and inherent uncertainty  should be disclosed. Also, the estimates should be verified based on cross- checks using other estimating methods.
The $100 million life cycle cost estimate, as documented in the program’s  6-year old economic analysis, does not reflect many of the practices  associated with a reliable cost estimate, including several practices related  to being comprehensive and well documented, and all related to being  accurate and credible (see table 4).
The cost estimate of about $100 million, as documented in the program’s  2002 economic analysis, does not meet all of the practices related to being  comprehensive. Specifically, it only includes costs from fiscal years 2003  through 2008 (6-year period), and it does not include both the government  and financial agent costs associated with development, acquisition (non- development), implementation, and operations and support over the  system’s life cycle. Moreover, it does not include FMS’s portion of the  program’s cost, which is estimated to be about $100 million over a 14-year  period. In addition, the cost estimate does not clearly describe how the  various cost sub-elements are aggregated to produce the amounts  associated with the two documented cost categories, system installation  costs, and operations and maintenance costs. Therefore, it is not clear that  all pertinent costs are included and no costs are double counted. Lastly,  although some key assumptions have been identified, such as the ship  implementation schedule, other key assumptions, such as labor rates and  inflation rates, are not. As a result, the estimate cannot be considered  comprehensive.
The cost estimate used in the economic analysis also addresses some, but  not all, of the practices related to being well-documented. Specifically, the  purpose of the cost estimate was clearly defined and a technical baseline  has been documented that includes, among others things, the hardware  and software specifications and planned performance parameters.  However, the calculations used to derive the cost estimate, including  descriptions of the methodologies used and traceability back to source  data (e.g., vendor quotes, salary data), are not documented. In addition,  while program officials described the estimating approach used, such as  using market research and historical data to determine the costs  associated with hardware, software, and installations, they did not have  documentation of the methodology used to arrive at the total costs of each  of these elements and how they were combined to produce the overall  cost estimate. Therefore, the program’s cost estimate cannot be  considered well-documented.
In addition, the $100 million documented cost estimate lacks accuracy  because it does not reflect an assessment of the costs most likely to be  incurred. Specifically, this estimate covers only 6 years of costs (fiscal  years 2003 through 2008). In contrast, the program’s current cost estimate  is about $320 million over a 14-year life cycle, and according to program  officials, the program’s life cycle is being reexamined and will likely be  extended.
Lastly, the $100 million cost estimate is not credible because a complete  uncertainty analysis (i.e., both a sensitivity analysis and a Monte Carlo  simulation) was not performed on this estimate. A sensitivity analysis  reveals how the cost estimate is affected by a change in a single  assumption or cost driver, such as the ship installation schedule, while  holding all other parameters constant. A Monte Carlo simulation assesses  the aggregate variability of the cost estimate to determine a confidence  range around the estimate. Without such analyses of uncertainty, the  program office cannot have confidence that the program can be completed  within the cost estimate.
Program officials acknowledged the limitations in the estimate, and  attributed them to turnover of staff and their current focus on deploying  the system. Nevertheless, program officials stated that they intend to  develop a revised cost estimate when they update the program’s economic  analysis, but they had yet to establish a date for accomplishing this. Given  that a significant amount of development and modernization funding  remains to be invested on the program, it is important that the program  office economically justify such investment.
Economic Analysis Does Not Satisfy Other Relevant Guidance  According to OMB, economic analyses should meet certain criteria to be  considered reasonable, such as comparing alternatives on the basis of net  present value and conducting an uncertainty analysis of benefits.
The program’s December 2002 economic analysis meets one, does not  meet four, and partially meets two of the seven OMB criteria governing  how to perform such analyses. For example, while the analysis explained  why the investment is needed, it did not consider the costs and benefits  associated with at least three alternatives to the status quo, such as Eagle  Cash, EZpay, or some derivative that provided for reuse of shared services  among the programs. Moreover, at least three alternatives to the status  quo were not assessed on the basis of net present value, using the proper  discount rate to account for inflation. Instead, the analysis only  qualitatively evaluated Navy Cash against its predecessor systems. For  example, the analysis included evaluation of the capabilities and  limitations of the predecessor systems, but did not include evaluating the  relative cost and benefits of any alternatives to Navy Cash.
In addition, the program’s benefit projections erroneously counted about  $40 million in cost transfers as cost savings, thus overstating projected  benefits (i.e., projected benefits should only be $93 million). Transfers  represent shifts of control over the spending of resources from one group  to another and thus do not result in an economic gain. According to OMB  guidance, transfers do not produce economic gains because the benefits to  those government entities that receive such a transfer are the same as the  costs borne by those government entities that provide the transfer.  Moreover, no uncertainty analysis was performed on the benefit estimates.  (See table 5 for the results of our analyses relative to each of the seven  criteria.)
Program officials stated that they do not know why the economic analysis  was not developed in accordance with OMB guidance. They also stated  that they intend to update the economic analysis and, in doing so, intend  to address OMB guidance. However, they did not have a date for  accomplishing this because their priority is deploying the system.
Actual Accrual of Estimated Benefits Has Not Been Validated  The Clinger-Cohen Act of 1996 and OMB guidance emphasize the need to  develop information to ensure that IT investments are actually  contributing to tangible, observable improvements in mission  performance. DOD guidance also states that estimated benefits should be  validated to ensure that desired outcomes are being achieved. To this end,  agencies should define and collect metrics to determine whether expected  benefits from a given investment are being accrued, and they should  modify subsequent economic analyses to reflect the lessons learned.
Despite the fact that Navy Cash has been installed and is operating on  approximately 130 ships, DON has yet to determine whether the system is  actually producing expected benefits. For example, the 2002 economic  analysis stated that Navy Cash would reduce cash on ships, and contribute  to man-hour savings as a result of increased productivity. It also stated  that it would improve quality-of-life for sailors and marines. While DON  has measured the reduction in the cash onboard some ships where Navy  Cash is operating, this reduction represents a transfer and is not an actual  benefit. Moreover, the extent to which the system is achieving expected  man-hour savings, which would constitute a true benefit, has not been  measured. Lastly, customer (sailor and marine) satisfaction with the  system, which is a legitimate qualitative benefit, has not been determined  since a prototype of Navy Cash was installed on two ships in 2001.
Program officials stated that DON’s Manpower Analysis Center is  responsible for measuring man-hour savings. Further, they said that  customer satisfaction with the system was being measured through  informal feedback from the sailors and marines, and they recently began a  more formal customer satisfaction survey. They also stated that in  updating the economic analysis, they plan to assess and reflect the accrual  of actual benefits. However, they had not established a date for  accomplishing this.
section_title: Key Controls for Ensuring That Defined Navy Cash Capabilities Are Delivered on Time and Within Budget Have Not Been Effectively Implemented
paragraphs: DOD policy and related guidance recognizes the importance of  implementing a range of management controls associated with ensuring  that IT investments are defined, developed, deployed, and operated  efficiently and effectively. By implementing these controls, the chances  of delivering systems that perform as intended, and not costing more or  taking longer than necessary, are increased. These controls include  requirements development and management, risk management, security  management, and system quality measurement. For Navy Cash, none of  these controls have been effectively implemented. Specifically,    program requirements have not been adequately developed and managed;    program risks have not been effectively managed;    security has not been adequately managed; and    data needed to measure two aspects of system quality—trends in  unresolved change requests and evaluation of user satisfaction with the  system—have not been collected and used.
As a result, Navy Cash is unlikely to perform in a manner that meets user  and operational needs, and it is likely to cost more and take longer than  necessary.
subsections: section_title: Navy Cash Requirements Have Not Been Adequately Developed and Managed
paragraphs: Well-defined and managed requirements are recognized by DOD guidance  and relevant best practices as essential, and can be viewed as a  cornerstone of effective system acquisition. Effective requirements  development and management includes (1) developing detailed system  requirements; (2) establishing policies and plans for managing changes to  requirements, including defining roles and responsibilities, and identifying  how the integrity of a baseline set of requirements will be maintained; and  (3) maintaining bi-directional requirements traceability, meaning that  system-level requirements can be traced both backward to higher level  business or operational requirements, and forward to system design  specifications and test plans.
The program office has not satisfied these three aspects of effective  requirements development and management. Specifically,    The program office has not developed system-level requirements for Navy  Cash. System-level requirements are derived from higher-level operational  requirements and are specified at a level of detail needed for system  developers to design and build to. Without system requirements, the ability  of the program office to understand the impact of any system change  requests (i.e., cost, schedule, and performance) and thus make informed  decisions about such changes, is limited. For example, although the  program office identified a high-level requirement for the system to share  information with the Retail Operations Management system used in ships’  store operations, the associated system-level requirements were not  defined. As a result, the deployed version of the system was not designed  and developed to provide this interface. The requirement for this interface  was later realized after a number of system and operational problems  surfaced. Addressing these problems through a series of changes required  additional time and funding. Program officials acknowledged that more  effective requirements development and management practices could have  avoided these problems. As another example, a system requirement for  automatically deploying software patches to operational systems was not  defined. Had this requirement been defined, the system design could have  provided for developing a capability to minimize the level of effort  required to identify, distribute, and install patches. Instead, a less efficient  and labor-intensive manual process has been used.
The program office does not have a policy or plans for managing  requirements. Such policies and plans establish organizational roles and  responsibilities for managing requirements, including maintaining and  controlling modifications or changes to the baseline sets of requirements,  establishing priorities among competing requests for changes, and  assessing the impact on cost, schedule, and performance of each change.  In lieu of a policy or plans, the program office has established an ad hoc  change control process, whereby change proposals are approved or  disapproved by a joint DON and FMS change control board based on a  change management policy that was drafted in 2003. However, this policy  was never finalized or approved and does not define roles and  responsibilities or how requirements will be managed. Further, the board  has not been chartered. Moreover, program officials told us that the  board’s decisions are made primarily on the basis of consensus about the  need for the change and the availability of funds.
Other than security requirements, Navy Cash requirements cannot be  traced from the higher level business or operational requirements to  system design specifications and test plans. Specifically, we attempted to  trace a sample of Navy Cash system-level requirements backward to high- level requirements and forward to design documents and test plans and  results. However, as noted above, no system-level requirements exist.  Without this link in the requirements traceability chain, traceability could  not be demonstrated. Having requirements traceability is essential for  ensuring that developed and deployed system products satisfy operational  needs and user expectations. In the case of Navy Cash, where system  capabilities are reactive to change requests rather than proactively driven  by requirements, such traceability is also essential to understanding the  impact to the system of each change request and thus having an informed  basis for approving and prioritizing any changes.
Program officials acknowledged these weaknesses and recently stated that  they intend to address them. To accomplish this, they reported that they  have hired a new employee who is to be trained in requirements  development and management, and who is to develop a requirements  management plan.
Until the program office employs fundamental requirements development  and management practices, it cannot reliably estimate the program costs  and develop schedules needed to accomplish the work associated with  delivering predetermined and economically justified system capabilities.  The result is an inability to develop and measure performance against  meaningful cost, schedule, and capability baselines, and thereby  reasonably ensure that the program is meeting expectations and those  responsible for it are accountable for results.
section_title: Navy Cash’s Risks Have Not Been Effectively Managed
paragraphs: Proactively managing program risks is a key acquisition management  control that, if done properly, can increase the chances of programs  delivering promised capabilities and benefits on time and within budget.  For Navy Cash, program risks have not been effectively managed. Rather,  the program office has reacted to the realization of actual problems. In  particular, plans, processes, and procedures are not in place that provide  for identifying, controlling, and disclosing risks, and risk management  roles and responsibilities have not been assigned to key stakeholders. As a  result, the program office is not positioned to proactively avoid the  occurrence of cost, schedule, and performance problems.
DOD and related guidance recognize the importance of performing  effective risk management on programs like Navy Cash. Among other  things, effective risk management includes: (1) establishing and  implementing a written plan and defined process for risk identification,  analysis, and mitigation; (2) assigning responsibility for managing risks to  key stakeholders; (3) encouraging program-wide participation in risk  management; and (4) examining the status of identified risks during  program milestone reviews.
The program office has not fully satisfied any of the above cited risk  management practices. For example:    A written plan or defined process that provides for identifying, analyzing,  and mitigating risks has not been established. In the absence of a plan and  process, program officials stated that risks are informally addressed  during bi-monthly program management reviews that involve key  stakeholders, including the program office, FMS, and the financial agent.  However, our analysis of minutes of these reviews indicates that they are  more focused on reacting to the consequences of actual problems, rather  than proactively attempting to avoid the occurrence of potential problems.
While program officials stated that responsibility for managing risks rests  with the program manager, roles and responsibilities for managing and  identifying risks have not been documented for any key stakeholders,  including individuals in the program office, and with FMS and the financial  agent. Without clearly documenting their roles and responsibilities,  proactive identification, disclosure, and mitigation of all key risks is  unlikely to occur, and program approval and decision making authorities  will not be adequately informed.
While program officials stated that attending and participating in program  management reviews is encouraged, we have yet to receive any verifiable  evidence that risks are addressed in these reviews or that involvement in  risk management is encouraged.
Program officials have yet to provide any verifiable evidence that program  decision making and oversight authorities have been apprised of the status  of identified risks.
Program officials acknowledged the above weaknesses and attributed  them to staff turnover in key positions and their focus on deploying the  system rather than establishing management processes and procedures.  Nevertheless, program officials stated that they intend to develop a risk  plan and process, but said that this would not occur until December 2008.  Given that a significant amount of development and modernization  investment remains, it is important that mitigating existing risks, including  those discussed in this report, as well as future risks be treated as a  program priority.
section_title: Navy Cash Security Management Has Not Been Effectively Implemented
paragraphs: A number of Navy Cash security management weaknesses exist.  Specifically, the program office has not (1) fully implemented a  comprehensive patch management process; (2) followed an adequate  process for planning, implementing, evaluating, and documenting remedial  actions for known information security weaknesses; (3) obtained adequate  assurance that FMS has effective security controls in place to protect Navy  Cash applications and data; and (4) developed an adequate contingency  plan and conducted effective contingency plan testing. Program officials  acknowledged these weaknesses but have yet to provide us with plans for  addressing them. As a result, the confidentiality, integrity, and availability  of deployed and operating Navy Cash shipboard devices, applications, and  financial data are at increased risk of being compromised.
Patch Management Has Not Been Fully Implemented  DOD guidance states that component organizations should develop a  process for patching system vulnerabilities. Further, National Institute of  Standards and Technology (NIST) guidance recognizes the importance of  implementing comprehensive patch management that includes, among  other things, (1) having a complete inventory of system hardware and  software assets, (2) automatically deploying vulnerability patches, and (3)  measuring patch management performance.
Although the program office performs patch management for Navy Cash,  key practices have not been fully implemented. Specifically,    A complete inventory of system assets does not exist. According to NIST, a  system inventory enables organizations to monitor system hardware and  software assets for the presence of all threats, vulnerabilities, and patches.  While the financial agent maintains a Navy Cash asset database for the 128  ships on which the system is operating, this database is missing 3  hardware inventories and 19 software inventories. According to program  officials, the financial agent’s database is incomplete because it was  created from purchase orders after the system was in operation.  Furthermore, although the program office maintains hardware inventories  for each ship in a DON configuration management database, the office  does not maintain inventories of Navy Cash software. Until the program  office develops a complete inventory of Navy Cash system assets, it will  not be able to identify and patch all system threats and vulnerabilities.
Vulnerability patches are not deployed in an automated or timely manner.
According to NIST guidance, deploying patches automatically minimizes  the level of effort and time required to identify, distribute, and install  patches. However, patches are currently deployed manually for Navy Cash  when ships are in port for maintenance. As a result, the risk of  vulnerabilities being exploited before ships return to port is increased.  Although the program office plans to introduce the capability to  automatically deploy patches as part of the next software release in the  first quarter of fiscal year 2009, program officials said that it will take  between 18 to 24 months to rollout this capability to the entire fleet.  Program officials also stated that they do not know why this capability  was not part of the original system requirements and design. Until the  program office begins automatically deploying patches, Navy Cash assets  and data will be exposed to increased risk.
The performance of patch management is not being measured. NIST  guidance recommends consistent measurement of the effectiveness of  patch management through the use of metrics, such as susceptibility to  attack and mitigation response time. Although program officials stated  that they maintain patch management metrics, they have yet to provide us  with a description of the metrics or an explanation of how they are used.  Until the program office develops and uses performance metrics, it will  not be able to assess and improve the effectiveness of its patch  management effort.
To strengthen its patch management efforts, the program office has  developed a vulnerability management guide. However, this guide has not  been finalized and approved, and according to program officials, it does  not follow NIST patch management guidance. Without comprehensive  patch management, increased risk exists that system vulnerabilities could  be exploited.
Remedial Action Plans Have Not Been Documented  The Federal Information Security Management Act (FISMA) requires that  agencies’ information security programs must include a process for  planning, implementing, evaluating, and documenting remedial actions to  address any deficiencies in the information security policies, procedures,  and practices of the agency. OMB has outlined steps for documenting  remedial actions—referred to by OMB as a plan of action and  milestones—for systems where IT security weaknesses have been  identified. Additionally, NIST guidance states that a plan of action and  milestones should be included in a system’s accreditation package and  describe how the information system owner intends to address those  vulnerabilities by reducing, eliminating, or accepting the identified  vulnerabilities.
Since the system was accredited in November 2006, the program office has  not developed any plans of action and milestones, even though medium  and low information security risks were identified during security test and  evaluation efforts supporting the certification and accreditation.  According to program officials, the risks were accepted by the designated  approving authority, rather than corrected, because they involve features  that are necessary for the system to operate, such as having certain  hardware interfaces and access permissions. While accepting rather than  correcting such weaknesses is consistent with DON guidance for  developing plans of action and milestones, it is not consistent with NIST  guidance. Specifically, DON guidance states that these plans are only  required for accreditation decisions that are conditional upon corrective  actions being taken. However, NIST guidance specifies that the  development of a plan of action and milestones should include instances  where risk is being accepted.
The lack of plans of action and milestones means that the program office  has not adequately addressed information security risks. Moreover, the  limitations in DON guidance mean that other Navy programs may not have  done so as well. Until the program office fully implements a remedial  action process that meets the FISMA requirements and OMB and NIST  guidance, program management and oversight officials will not have  sufficient assurance that all security weaknesses are being reported and  tracked, and that options for addressing them are fully considered.
Information Security Requirements Have Not Been Fully Defined  FISMA requires each federal agency to develop, document, and implement  an agencywide information security program to provide information  security for the information and information systems that support the  operations and assets of the agency, including those provided or managed  by another agency, contractor, or other source. Among other things, this  includes testing system management, operational, and technical security  controls. Although the program office has partnered with FMS to develop  and support the operation of Navy Cash, it is ultimately responsible for  ensuring the security of Navy Cash systems and data.
The program office has not taken adequate steps to ensure that security  controls are tested. Specifically, the memorandum of agreement between  the program office and FMS does not establish requirements for FMS and  the financial agent relative to periodic information security control  reviews, including reviews of applicable management, operational, and  technical controls, and to provide DON with copies of information security  control reviews that are performed on the Navy Cash system and its  supporting infrastructure. This is important because FMS—through its  financial agent—provides services that support Navy Cash that must be  secure, such as holding and accounting for funds distributed throughout  the system and processing transactions. Although FMS has performed  some management and operational control tests, such as periodic  personnel and physical security assessments of selected commercial  facilities that provide services and support to Navy Cash, these  assessments were not designed to evaluate the technical controls of the  system’s computing environment because the memorandum of agreement  does not include such requirements.
Until the program office and FMS establish information security  requirements for overseeing the financial agent’s technical information  security controls, an increased risk exists that the confidentiality, integrity,  and availability of information stored, transmitted, and processed by the  financial agent can be compromised.
OMB guidance requires agencies to develop contingency plans and to test  those plans at least annually. NIST guidance states that contingency plans  should include a sequence of recovery activities, which describe system  priorities based on business impact and notification procedures, which  describe the methods used to notify personnel with recovery  responsibilities. In addition, according to NIST, contingency plan tests  should include explicit test objectives and success criteria for each  planned activity and related procedure and documentation of lessons  learned.
Although the program office has developed contingency plans for Navy  Cash, it did not identify the sequence of recovery activities and  notification procedures for recovery personnel in them. The sequence of  activities should prioritize the recovery of system components by  criticality and the notification procedures should describe the methods  used to notify recovery personnel during business and non-business hours.  Until the program office includes these areas in the contingency plans, it  cannot ensure that system components will restore in a logical manner  and that ship recovery personnel will be notified promptly when a system  disruption is detected. In addition, while the program office has largely  included explicit test objectives and success criteria in all the test  procedures, they did not document the lessons learned. According to  NIST, lessons learned can improve contingency plan effectiveness and this  should be incorporated into the plan. According to program officials, NIST  was not used for developing and conducting tests of the contingency plan.  Without lessons learned, the program office will not be able to properly  maintain and improve the contingency planning guide.
Until DON develops sufficient contingency plans and testing procedures,  increased risk exists that Navy Cash systems, data, and operations will not  be able to fully recover from a disruption or disaster.
section_title: Navy Cash Quality Measures Are Not Being Collected
paragraphs: Effective management of programs like Navy Cash depends in part on the  ability to measure the quality of the system being acquired and operated.  One measure of system quality is the trend in the number of unaddressed,  high-priority system change requests.
Sufficient data to measure trends in open (i.e., unresolved) system change  requests, which is a recognized indicator of a system’s stability and quality  are not being collected. To the program’s credit, it has formed a group  consisting of program office, FMS, and financial agent representatives to  review and decide whether to approve requests for changes to the system.  However, this group is not consistently collecting data as to when a  change request is opened or closed and what the priority level of each  change request is. Thus, it does not know at any given time, for example,  how many change requests are pending, the significance of pending  change requests, and the age of these change requests. Program officials  acknowledged these weaknesses but stated that their focus has been on  deploying the system. This means that the program office cannot know  and disclose to DOD decision makers whether the system’s stability and  maturity are moving in the right direction.
In addition, the program office has not consistently collected data on user  and operator satisfaction with the system. Specifically, the program office  conducted two surveys in the last 6 years—a user satisfaction survey and a  shipboard merchant satisfaction survey—but neither of these surveys is  meaningful. More specifically, the user satisfaction survey was done in  2002 and thus is dated; and it covered only two ships and a prototype  version of Navy Cash and thus its scope is limited. In addition, neither  survey produced a response rate that can be generalized and projected  (about 50 percent and 20 percent for the two ships in the user survey, and  about 30 percent for the merchant survey).
Program officials stated that they have relied on informal user feedback  from disbursing officers, who have indicated overall satisfaction with the  system. Nevertheless, they said that a survey of users and operators is  being planned and expected to be completed by the fall of 2008. Without  meaningful data about Navy Cash’s stability and the satisfaction of those  who use it, it is not clear Navy Cash is a quality system.
section_title: Conclusions
paragraphs: Navy Cash’s potential duplication of other DOD programs that perform  similar functions, combined with its lack of meaningful economic  justification, together mean that the department does not have an  adequate basis for knowing whether Navy Cash, as defined, is the most  cost-effective solution to meeting its strategic business and technological  needs. Because such a basis is absolutely fundamental to informed  investment decision making, a compelling case exists for the department  to reevaluate current plans for investing almost $60 million of additional  modernization funding to further develop the system.
Even if reevaluation supports current or modified investment plans, the  manner in which the program is being executed remains a source of  considerable cost, schedule, and performance risk. In particular, without  employing fundamental requirements development and management  practices, the department cannot reliably estimate program costs and  develop schedules needed to accomplish the work associated with  delivering predetermined and economically justified system capabilities.  In addition, without effective risk management, the department is not  positioned to proactively avoid the occurrence of cost, schedule, and  performance problems. Furthermore, the lack of adequate security  management puts the confidentiality, integrity, and availability of deployed  and operating Navy Cash shipboard devices, applications, and financial  data at increased risk of being compromised. Moreover, without  meaningful data about the Navy Cash’s stability and the satisfaction of  those who use it, it is not clear that Navy Cash is a quality system.
To overcome each of these weaknesses, it is important to not only  acknowledge them, which the program office has done, but to also treat  them as program priorities, including developing and implementing plans  for addressing them, which the program office has largely not done.
section_title: Recommendations
paragraphs: Because of the uncertainty surrounding whether Navy Cash, as defined,  represents a cost-effective solution, we recommend that the Secretary of  Defense direct the Secretary of the Navy to limit further investment of  modernization funding in the program to only (1) deployment to remaining  ships of already developed and tested capabilities; (2) correction of  information security vulnerabilities and weaknesses on ships where it is  deployed and operating; and (3) development of the basis for an informed  decision as to whether further development and modernization is  economically justified and in the department’s collective best interests.
To develop the basis for an informed decision about further Navy Cash  development, we further recommend that the Secretary of Defense, direct  the appropriate DOD organizations to (1) examine the relationships among  DOD’s programs for delivering military personnel with smart card  technology for electronic retail and banking transactions; (2) identify, in  coordination with the respective program offices, alternatives for  optimizing the relationships of these programs in a way that minimizes  areas of duplication, maximizes reuse of shared services across the  programs, and considers opportunities for a consolidated stored value  card program across the military services; and (3) share the results with  the appropriate organizations for use in making an informed decision  about planned investment in Navy Cash.
To further develop this basis for an informed decision about Navy Cash  development, we also recommend that the Secretary of Defense direct the  Secretary of the Navy to ensure that the appropriate Navy organizational  entities prepare a reliable economic analysis that encompasses the  program’s total life cycle costs, including those of FMS, and that (1)  addresses cost-estimating best practices and complies with relevant OMB  cost-benefit guidance and (2) incorporates data on whether deployed Navy  Cash capabilities are actually producing benefits.
To address Navy Cash information security management weaknesses and  improve the operational security of the system, we recommend that the  Secretary of Defense direct the Secretary of the Navy to ensure that the  Navy Cash program manager, in collaboration with the appropriate  organizations, take the following five actions:    Develop and implement a patch management approach based on NIST  guidance, which includes a complete Navy Cash systems inventory; an  automated patch deployment capability; and a patch management  performance vulnerability measurement capability, including metrics for  susceptibility to attack and mitigation response time.
Institute a process to plan, implement, evaluate, and document remedial  actions for deficiencies in Navy Cash information security policies,  procedures, and practices, and ensure that this process meets FISMA  requirements, as well as applicable OMB and NIST guidance.
Update the NAVSUP/FMS memorandum of agreement, in collaboration  with FMS, to establish specific security requirements for FMS and the  financial agent to periodically perform information security control  reviews, including applicable management, operational, and technical  controls, of the Navy Cash system, and to provide NAVSUP with copies of  the results of these reviews that pertain to the Navy Cash system and its  supporting infrastructure.
Develop a complete contingency plan to include a (1) sequence of  recovery activities and (2) procedures for notifying ship personnel with  contingency plan responsibilities to begin recovery activities; and to test  the contingency plan in accordance with NIST guidance, including  documenting lessons learned from testing.
To address DON information security guidance limitations, we also  recommend that the Secretary of Defense direct the Secretary of the Navy  to ensure that the Navy Operational Designated Approving Authority, as  part of the Naval Network Warfare Command, updates its certification and  accreditation guidance to require the development of plans of action and  milestones for all above identified security weaknesses.
If further investment in development of Navy Cash can be justified, we  then recommend that the Secretary of Defense direct the Secretary of the  Navy, through the appropriate chain of command, to ensure that the Navy  Cash program manager takes the following actions.
With respect to requirements development and management, (1) develop  detailed system requirements; (2) establish policies and plans for  managing changes to requirements, including defining roles and  responsibilities, and identifying how the integrity of a baseline set of  requirements will be maintained; and (3) maintain bi-directional  requirements traceability.
With respect to risk management, (1) establish and implement a written  plan and defined process for risk identification, analysis, and mitigation;  (2) assign responsibility for managing risk to key stakeholders; (3)  encourage program-wide participation in risk management; (4) include  and track the risks discussed in this report as part of a risk inventory; and  (5) apprise decision making and oversight authorities of the status of risks  identified during program reviews.
With respect to system quality measurement, collect and use sufficient  data for (1) determining trends in unresolved change requests and (2)  understanding users’ satisfaction with the system.
section_title: Agency Comments and Our Evaluation
paragraphs: Both DOD and FMS provided written comments on a draft of this report.  In DOD’s comments, signed by the Deputy Under Secretary of Defense  (Business Transformation) and reprinted in appendix II, the department  stated that it concurred with 9 of our 11 recommendations, partially  concurred with 1, and non-concurred with the remaining 1. In non- concurring with our recommendation for limiting further investment in the  program, the department actually concurred with two out of three aspects  of the recommendation. Nevertheless, for the aspect of our  recommendation aimed at limiting further investment in the program to  certain types of spending, it stated that it did not concur with limiting  investment to the exclusion of needed maintenance (e.g., technology  refresh) of operational systems. We agree with this comment, as it is  consistent with statements in our report, including the recommendation  summary on the report’s highlights page and the report’s conclusions, both  of which focus on limiting investment of modernization funding only, and  not operations and maintenance funding. To avoid any misunderstanding  as to our intent, we clarified our report.
With respect to our recommendation for optimizing the relationships  among DOD’s programs that provide smart card technology for electronic  retail and banking transactions, the department stated that, while it  concurs with the overall intent of the recommendation, it believes that the  Office of the Under Secretary of Defense (Comptroller) is the appropriate  organization to implement it. Since our intent was not to prescribe the  only DOD organization that should be responsible for implementing the  recommendation, we have slightly modified the recommendation to  provide the department flexibility in this regard.
Notwithstanding DOD’s considerable agreement with our  recommendations, the department provided additional comments on the  findings that underlie several of the recommendations, which it described  as needed to clarify and avoid confusion about the program. For various  reasons discussed below, we either do not agree with most of these  additional comments or do not find them germane to our findings and  recommendations.
First, the department stated that the report’s overall findings understate  the program’s discipline and conformance with applicable guidance and  best practices. We do not agree. Our review extended to six key  acquisition control areas, all of which are reflected in DOD’s own  acquisition policies as well as other federal guidance. Effective  implementation of these controls can minimize program risks and better  ensure that system investments are defined in a way to optimally support  mission operations and performance, as well as deliver promised system  capabilities and benefits on time and within budget. However, we found  that none of these key IT management controls were being effectively  implemented on Navy Cash, and the department agreed with our  recommendations aimed at correcting this.
Second, the department stated that the report’s findings do not accurately  capture the program’s maturity since the system has been deployed to  over 80 percent of its user base. While we do not question the extent to  which the system has been deployed to date, and in fact state in our report  that the system has been deployed to about 80 percent of the fleet, we do  not agree that the program is mature, as evidence by the numerous IT  management control weaknesses that we found and the fact that about $60  million in modernization funding remains to be spent on the system.
Third, the department stated that it recognizes that some security  management limitations exist, but added that these limitations do not pose  a serious risk to the confidentiality, integrity, or availability of the  deployed system, and that our report may cause cardholders to become  unnecessarily concerned. We do not agree that these limitations do not  pose a serious risk. Our report details a number of serious security  management weaknesses relative to both DOD and NIST guidance, such as  not following an adequate process for planning, implementing, evaluating  and documenting remedial actions for known information security  vulnerabilities, as well as not obtaining adequate assurance that FMS has  effective security controls in place to protect Navy Cash applications and  data. As a result, we appropriately conclude in our report that such  failures to effectively manage Navy Cash security places the  confidentiality, integrity, and availability of deployed and operating  shipboard devices, applications, and financial data at increased risk of  being compromised. Swift implementation of our recommendations is the  best solution to alleviating any cardholder concerns that may arise from  these weaknesses.
In FMS’s comments, signed by the Commissioner of FMS and reprinted in  appendix III, the service stated that our recommendations will help  strengthen Navy Cash and that it has begun addressing our findings and  recommendations. In addition, it stated that it will support DOD in  implementing the recommendations, and consistent with DOD,  commented that it did not agree with one part of one of our  recommendations, adding that limiting investment in Navy Cash beyond  fielding and maintaining already tested system capabilities would place  future operations at risk. As stated above, this recommendation is focused  on limiting further investment in modernization funding, not operations  and maintenance funding. To avoid any confusion about this, we have  added language to other parts of the report to emphasize this focus.
In addition to the above, and notwithstanding its overall agreement with  our recommendations, FMS provided other comments relative to several  of the findings that underlie our recommendations. As discussed below,  we either do not agree with these additional comments or do not find them  to be germane to our findings and recommendations.
First, FMS stated that our report does not identify a security breach, loss  of cardholder or government funds, unauthorized release of personal or  other sensitive information, or any other compromise of system integrity.  We agree that our report does not identify these things, as the scope of  work was not intended to identify them. Rather, our scope focused on the  program’s implementation of key security management controls outlined  in DOD and NIST guidance. In this regard, we found serious information  security management control weaknesses and concluded that these  weaknesses increased the risk to the confidentiality, integrity, and  availability of information stored, transmitted, and processed by the  financial agent.
Second, FMS stated that the issue of whether Navy Cash is duplicative of  other similar DOD smart card programs was addressed before Navy Cash  was initiated in 2001, when DON and FMS determined that for technical  and cost reasons it could not alter the other DOD programs to meet Navy  Cash requirements. We do not find this comment relevant to our  recommendation because our point is not that one of the other DOD  programs should be altered and used in place of Navy Cash. Rather, our  point is that these smart card programs need to be looked at collectively to  decide whether it is in the department’s best interest to continue investing  in separate smart card programs or to invest in a single department-wide  solution. This point is consistent with FMS’s stated goal of having a single  smart card for DOD.
Third, FMS stated that it disagreed with our finding that the Navy Cash  benefits projection erroneously counted $40 million as cost savings rather  than cost transfers, adding that this value represents not merely a transfer  between agencies but actual savings to the United States. While we do not  disagree that this interest savings represents a benefit to the United States  government, it also represents a cost—interest foregone—to holders of  Treasury debt. Therefore, the interest savings represents a transfer rather  than savings from one member or sector to another.
We are sending copies of this report to interested congressional  committees; the Director, Office of Management and Budget; the  Congressional Budget Office; the Secretary of Defense; the Secretary of  the Treasury; and the Department of Defense Office of the Inspector  General. We also will make copies available to others upon request. In  addition, the report will be available at no charge on the GAO Web site  http://www.gao.gov.
If you or your staffs have any questions on matters discussed in this  report, please contact Randolph C. Hite at (202) 512-3439 or  hiter@gao.gov, or Gregory C. Wilshusen at (202) 512-3789 or  wilshuseng@gao.gov. Contact points for our Offices of Congressional  Relations and Public Affairs may be found on the last page of this report.  GAO staff who made major contributions to this report are listed in  appendix IV.
section_title: Appendix I: Objective, Scope, and Methodology
paragraphs: Our objective was to determine whether the Department of the Navy  (DON) is effectively implementing information technology management  controls on Navy Cash. We selected Navy Cash primarily because the  Department of Defense’s (DOD) inventory of DON systems identified the  program as one of DON’s five largest development and modernization  investments. To address the objective, we focused on the following  management areas (1) architectural alignment; (2) economic justification;  (3) requirements development and management; (4) risk management; (5)  security management; and (6) system quality measurement. In doing so,  we analyzed a range of program documentation, such as the acquisition  strategy, business case, economic analysis, agreements between the  partnering organizations, and interviewed cognizant officials, such as the  Milestone Decision Authority, program manager, and Financial  Management Service (FMS) and financial agent officials responsible for  Navy Cash.
To address architectural alignment, we reviewed the program’s business  enterprise architecture (BEA) compliance assessments and system  architecture products as well as versions 4.0, 4.1, and 5.0 of the BEA and  compared them to the BEA compliance requirements described in the  Ronald W. Reagan National Defense Authorization Act for Fiscal Year  2005 and DOD’s BEA compliance guidance and evaluated the extent to  which the compliance assessments addressed all relevant BEA products.  We also reviewed DOD guidance for program architecture development,  such as DOD’s Business Transformation Guidance, and compared Navy  Cash’s program architecture development activities to this guidance. In  addition, we interviewed Navy Cash and FMS officials, as well as Navy  Cash’s Milestone Decision Authority, and requested related documentation  on the potential duplication between Navy Cash and other DOD programs  that involve the use of smart card functionality, such as the Air Force’s and  Army’s Eagle Cash and EZpay programs.
To address the program’s economic justification, we reviewed the latest  economic analysis to determine the basis for the cost and benefit  estimates. This included evaluating the analysis against Office of  Management and Budget guidance and GAO’s Cost Assessment Guide. In  addition, we interviewed cognizant program officials, including the Navy  Cash program manager and FMS, regarding their respective roles,  responsibilities, and actual efforts in developing and/or reviewing the  economic analysis and the extent to which measures and metrics showed  that projected benefits in the economic analysis were actually being  realized. We also interviewed cognizant officials such as the Milestone  Decision Authority about the purpose and use of the program’s economic  analysis for managing the investment in the Navy Cash program.
To address requirements development and management, we reviewed  relevant program documentation, such as the concept of operations  document, and interviewed relevant program officials and evaluated this  information against relevant best practices. We also reviewed interface  requirements documents, minutes of program management meetings, and  traceability of security requirements. In addition, we interviewed program  officials involved in the requirements management process to discuss the  change control process they use and their roles and responsibilities for  managing requirements.
To address risk management, we reviewed relevant risk management  documentation, such as program management review meeting minutes  and compared the program office’s activities with DOD’s risk management  guidance and related best practices. We analyzed the effectiveness of the  program’s management reviews in terms of managing risks. In doing so,  we interviewed cognizant program officials responsible, such as the  program manager, Milestone Decision Authority, and FMS officials to  discuss their roles and responsibilities and obtain clarification on the  program’s approach to managing risks associated with acquiring and  implementing Navy Cash.
To address security management, we reviewed relevant security  documentation, such as DOD and National Institute of Standards and  Technology information security guidance, and the Navy Cash afloat and  ashore system security authorization agreements. In addition, we observed  the system in operation aboard the USS Theodore Roosevelt and  discussed security issues with ship personnel, program office, FMS, and  financial agent officials. We also reviewed USS Harry S. Truman  contingency plan test results. Additionally, we reviewed a database used  to maintain the inventory of Navy Cash hardware and software assets as a  part of our analysis on the Navy Cash vulnerability management program.  Furthermore, we interviewed cognizant DON, FMS, and financial agent  officials to discuss their roles and responsibilities and obtain clarification  on the program’s approach to protecting the confidentiality, integrity, and  availability of Navy Cash systems and information.
To address system quality measurement, we reviewed program  documentation, such as change request logs, and a plan of action and  milestones for change requests. We also compared the program’s data  collection and analysis practices relative to these areas to program  guidance and best practices. We reviewed the plans for and results of  surveys that were performed on user and shipboard merchant satisfaction  with Navy Cash, and we interviewed program management and technical  officials.
We conducted our work at DOD offices and program office and ship  facilities in the Washington, D.C. metropolitan area, Norfolk, Virginia, and  Mechanicsburg, Pennsylvania, between June 2007 and September 2008, in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objective. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objective.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: Comments from the Department of the Treasury, Financial Management Service
section_title: Appendix IV: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: In addition to the contact persons named above, key contributors to this  report were Neelaxi Lakhmani (Assistant Director), Jenniffer Wilson  (Assistant Director), Ed Glagola (Assistant Director), Monica Anatalio,  Carolyn Boyce, Harold Brumm, West Coile, Neil Doherty, Cheryl  Dottermusch, Joshua Hammerstein, Mustafa Hassan, Michael Holland,  James Houtz, Ethan Iczkovitz, Rebecca LaPaze, Anh Le, Josh Leiling, Mary  Marshall, Karen Richey, Melissa Schermerhorn, Karl Seifert, Jonathan  Ticehurst, and Adam Vodraska.