DOD Business Systems Modernization: Planned Investment in Navy Program to Create Cashless Shipboard Environment Needs to Be Justified and Better Managed
Released: 2008-09-08T00:00:00

Letter

Background

DON’s primary mission is to organize, train, maintain, and equip combat- ready naval forces capable of winning the global war on terror and any other armed conflict, deterring aggression by would-be foes, preserving freedom of the seas, and promoting peace and security. To support this mission, DON performs a variety of interrelated and interdependent business functions (e.g., acquisition and financial management), relying heavily on IT systems. In fiscal year 2008, DON’s IT budget was about $2.7 billion, of which $2.2 billion was allocated to operations and maintenance of existing systems and the remaining $500 million to systems in development and modernization. Of the approximately 3,000 business systems that DOD reports in its current inventory, DON accounts for 904, or about 30 percent, of the total. The Navy Cash system is one such system investment.

Navy Cash: A Brief Description

In 2001, DON initiated Navy Cash in partnership with Treasury’s FMS to enable sailors and marines to use smart cards that store monetary value, also known as stored value cards, to make retail purchases and conduct banking transactions while on ships and ashore. The program builds upon capabilities that have been incrementally introduced from previously deployed systems. (Table 1 summarizes these systems and their capabilities and limitations.)

According to DOD, Navy Cash’s key objectives include introducing workload efficiencies and improving the quality of life for sailors and marines by reducing the amount of currency on ships, which lowers costs associated with cash handling activities; enabling sailors and marines to conduct ashore banking transactions from enabling sailors and marines to conduct banking or retail transactions while ashore (wherever these branded debit cards are accepted).

Navy Cash consists of various equipment and devices, including servers that connect to the ship’s local area network as well as point-of-sale terminals and ATMs that communicate with Navy Cash smart cards. These cards contain an electronic chip that stores monetary value and interacts with the various devices for conducting electronic retail purchases and personal banking transactions on the ships. On shore, cardholders can access their Navy Cash accounts via ATMs worldwide or conduct retail purchases using the card’s magnetic stripe, which provides a debit card feature. According to program officials, while ashore, sailors and marines have access to over 1,000,000 ATMs and 23 million merchants worldwide.

Navy Cash uses a ship’s Automated Digital Network System to access satellite communications systems, and then transmits transaction files off the ship through fleet network operations centers to a financial agent (i.e., bank) ashore. To do so, it uses a store-and-forward process to batch transactions together and transmit them off the ship typically during non- peak evening hours. These transactions are then processed in a manner similar to personal check processing through the Automated Clearing House. Figure 1 is a simplified illustration of the Navy Cash network used to transmit these transactions.

Originally, the program was expected to be fully deployed and reach full operational capability by December 2008 at an estimated cost of about $100 million over a 6-year life cycle. The program office now expects the program to reach full operational capability in fiscal year 2011, and it estimates the program’s 14-year life cycle cost to be about $320 million, of which about $100 million is to be funded by FMS. Of the $320 million, about $136 million is for development and modernization, and about $184 million is for operations and maintenance. From fiscal year 2002 to 2007, DON and FMS reported that approximately $132 million has been spent on the program, of which $47 million is FMS’s cost. Of the $188 million expected to be spent (fiscal years 2008-2015), about $57 million is for development and modernization. (See fig. 2 for a breakdown of the actual and planned costs.)

When fully deployed, the program office estimates that Navy Cash could process over $350 million annually in transactions initiated by about 170,000 sailors and marines worldwide on approximately 160 ships. As of April 2008, the program has been deployed to approximately 130 ships.

Navy Cash Oversight and Management Roles and Responsibilities

To manage the acquisition and deployment of Navy Cash, DON established a program management office within the Naval Supply Systems Command (NAVSUP). As authorized by statute and because of its experience in developing stored value card programs for other military departments, NAVSUP has partnered with FMS to develop Navy Cash. In February 2001, NAVSUP and FMS signed a memorandum of agreement that, among other things, delineated their respective program roles and responsibilities. According to the agreement, NAVSUP, through the Navy Cash program office, is responsible for managing the acquisition of the program, including managing system requirements and developing program cost and benefit estimates. According to DOD and other relevant guidance, acquisition management includes, among other things, such key IT management control areas as architectural alignment, economic justification, requirements management, risk management, security management, and system quality measurement.

Also according to the agreement, FMS, through a designated financial agent, is to (1) provide for all financial services (i.e., manage the funds distributed through Navy Cash) and (2) develop, test, operate, and maintain the system’s software (e.g., terminal and accounting applications) and hardware (e.g., accounting servers, smart cards). In short, the financial agent acts as the depository bank, holding and managing the pool of sailor and marine funds, including accounting for the funds and settling transactions processed. FMS is also responsible for tracking and overseeing the financial agent’s provision of services, as defined in a financial agency agreement between FMS and the agent. (See fig. 3 for DON and FMS roles and relationships for Navy Cash.)

In addition, various other organizations share program oversight and review activities. A listing of key entities and their roles and responsibilities can be found in table 2.

Use of IT Management Controls Maximizes Chances for Success

Effective IT management controls are grounded in tried and proven methods, processes, techniques, and activities that organizations define and use to minimize program risks and maximize the chances of a program’s success. Using such best practices can result in better outcomes, including cost savings, improved service and product quality, and a better return on investment. For example, two software engineering analyses of nearly 200 systems acquisitions projects indicate that teams using systems acquisition best practices produced cost savings of at least 11 percent over similar projects conducted by teams that did not employ the kind of rigor and discipline embedded in these practices. In addition, our research shows that best practices are a significant factor in successful acquisition outcomes, including increasing the likelihood that programs and projects will be executed within cost and schedule estimates.

We and others have identified and promoted the use of a number of best practices associated with acquiring IT systems. See table 3 for a description of several of these activities.

Prior GAO Reviews Have Identified IT Management Control Weaknesses on DOD Business System Investments

We have previously reported that DOD has not effectively managed a number of business system investments. Among other things, our reviews of individual system investments have identified weaknesses in such things as architectural alignment and informed investment decision making, which are also the focus areas of the Ronald W. Reagan National Defense Authorization Act for Fiscal Year 2005 business system provisions. Our reviews have also identified weaknesses in other system acquisition and investment management areas—such as economic justification, requirements management, and risk management.

Recently, for example, we reported that the Army’s approach for investing about $5 billion over the next several years in its General Fund Enterprise Business System, Global Combat Support System-Army Field/Tactical, and Logistics Modernization Program did not include alignment with Army enterprise architecture or use of a portfolio-based business system investment review process. Moreover, we reported that the Army did not have reliable processes, such as an independent verification and validation function, or analyses, such as economic analyses, to support its management of these programs. We concluded that until the Army adopts a business system investment management approach that provides for reviewing groups of systems and making enterprise decisions on how these groups will collectively interoperate to provide a desired capability, it runs the risk of investing significant resources in business systems that do not provide the desired functionality and efficiency. Accordingly, we made recommendations aimed at improving the department’s efforts to achieve total asset visibility and enhancing its efforts to improve its control and accountability over business system investments. The department agreed with our recommendations.

We also reported that DON had not, among other things, economically justified its ongoing and planned investment in the Naval Tactical Command Support System (NTCSS) and had not invested in NTCSS within the context of a well-defined DOD or DON enterprise architecture. In addition, we reported that DON had not effectively performed key measurement, reporting, budgeting, and oversight activities, and had not adequately conducted requirements management and testing activities. We concluded that without this information, DON could not determine whether NTCSS as defined, and as being developed, is the right solution to meet its strategic business and technological needs. Accordingly, we recommended that the department develop the analytical basis to determine if continued investment in NTCSS represents prudent use of limited resources and to strengthen management of the program, conditional upon a decision to proceed with further investment in the program. The department largely agreed with these recommendations.

In addition, we reported that the Army had not defined and developed its Transportation Coordinators’ Automated Information for Movements System II—a joint services system with the goal of helping to manage the movement of forces and equipment within the United States and abroad— in the context of a DOD enterprise architecture. We also reported that the Army had not economically justified the program on the basis of reliable estimates of life cycle costs and benefits and had not effectively implemented risk management. As a result, we concluded that the Army did not know if its investment in this program, as planned, is warranted or represents a prudent use of limited DOD resources. Accordingly, we recommended that DOD, among other things, develop the analytical basis needed to determine if continued investment in this program, as planned, represents prudent use of limited defense resources. In response, the department largely agreed with our recommendations, and has since reduced the program’s scope by canceling planned investments.

Key IT Management Controls Have Not Been Effectively Implemented on Navy Cash

DOD acquisition policies and related federal guidance provide a framework within which to manage system investments, like Navy Cash. Effective implementation of this framework can minimize program risks and better ensure that system investments are defined in a way to optimally support mission operations and performance, as well as deliver promised system capabilities and benefits on time and within budget. Thus far, key IT management controls associated with this framework have not been implemented on Navy Cash. In particular, the program’s overlap with and duplication of other DOD programs has not been assessed, and the program has not been economically justified on the basis of reliable estimates of life cycle costs and benefits. As a result, the program, as defined, has not been shown to be the most cost-effective investment option.

Even if investment in the proposed Navy Cash solution is shown to be a wise and prudent course of action, the manner in which Navy Cash is being acquired and deployed is not adequate because (1) requirements have not been adequately developed and managed; (2) program risks have not been effectively managed; (3) security has not been effectively managed; and (4) system quality has not been adequately measured. As a result, the system will likely experience performance shortfalls and cost more and take longer to implement and maintain than necessary.

Program officials acknowledged these weaknesses and attributed them to, among other things, turnover of staff in key positions and their focus on deploying the system. Further, they stated that addressing these weaknesses has not been a top program priority because Navy Cash has been deployed to and is operating on about 80 percent of the ships. Nevertheless, about $60 million in development and modernization funding remains to be spent on this program. As a result, it is important that all these weaknesses be addressed to reduce the risk of delivering a system solution that falls short of expectations.

Key Controls for Justifying Planned Investment in Navy Cash Have Not Been Effectively Implemented

Investment in the proposed Navy Cash solution has not been adequately justified. Specifically, the system solution has not been assessed relative to other DOD programs that employ smart cards for electronic retail transactions. Moreover, it has not been economically justified on the basis of reliable estimates of cost and benefits over the system’s expected life. As a result, planned investment in the system, as defined, may not be a cost-effective course of action.

Navy Cash Duplication with Other DOD Programs Has Not Been Assessed

DOD’s acquisition policies and guidance, as well as federal and best practice guidance, recognize the importance of investing in business systems within the context of an enterprise architecture. Moreover, the Ronald W. Reagan National Defense Authorization Act for Fiscal Year 2005 requires that defense business systems be compliant with the federated BEA. Our research and experience in reviewing federal agencies show that making investments without the context of a well- defined enterprise architecture often results in systems that are, among other things, duplicative of other systems.

Navy Cash has not been assessed and defined in a way to ensure that it is not duplicative of the Eagle Cash and EZpay programs, both of which provide for the use of smart card technology for electronic retail transactions in support of the Air Force and the Army. Within DOD, the means for avoiding business system duplication and overlap is the department’s process for assessing compliance with the DOD BEA and its associated investment review and decision making processes. In 2005, 2006, and 2007, Navy Cash was evaluated for compliance with the BEA. However, the BEA does not contain business activities that Navy Cash supports. According to officials from DOD’s Business Transformation Agency, which is responsible for DOD’s BEA, these business activities are not included nor are they planned for inclusion in the BEA, because the capabilities provided by Navy Cash relate strictly to personal banking, which is outside of the current scope of the BEA. As a result, compliance could not be assessed beyond concluding that Navy Cash was compliant because it did not conflict with the BEA. Moreover, even if the BEA included the business activities that Navy Cash supports, the program’s ability to assess BEA compliance would have been limited because the program office did not develop a complete set of system-level architecture products needed to perform a meaningful compliance assessment. Thus, Navy Cash’s potential overlap and duplication with similar programs is not sufficiently understood.

According to program officials, Navy Cash is not duplicative of Eagle Cash and EZpay because it is designed to operate on ships at sea, which do not maintain constant network connectivity with on shore networks. Therefore, they said that it requires different communications and financial transaction capabilities than the other two stored value card programs. We agree that there are important differences between the programs. However, they all perform chip-based financial transactions, and thus opportunities may exist for them to provide or reuse shared system services, as well as to merge into a DOD-wide stored value card program. According to program officials, overlap and duplication among the programs was not assessed. This means that aspects of Navy Cash could be potentially duplicative of these other programs, and thus DOD may not be pursuing the most cost-effective solution to meet its mission needs. In this regard, the program’s Milestone Decision Authority told us that the differences between Navy Cash and other stored value card programs are minimal and stated that officials with the three stored value card programs have recently begun discussions with FMS on how to collaborate and possibly move towards one system solution.

Navy Cash Has Not Been Economically Justified

Investment in Navy Cash has not been economically justified on the basis of a reliable analysis of estimated system costs and expected benefits over the life of the program. Specifically, according to the latest economic analysis, the program is expected to produce estimated benefits of about $133 million for an estimated cost of about $100 million. However, the cost estimate is not reliable, because the program’s 2002 economic analysis is 6 years old and is based on a cost estimate of about $100 million that was not derived in accordance with effective estimating practices, such as including all costs over the system’s life cycle, and adjusting the estimate to account for program risks and material program changes. Further, this economic analysis did not comply with applicable federal guidance. For example, it did not adequately consider all relevant alternatives, and it erroneously counted $40 million as cost savings rather than transfers (i.e., shift of control over spending of resources from one group to another that do not result in an economic gain). Further, the economic analysis has yet to be validated using actual data on the accrual of benefits. Without an economic analysis that is reliable, DON’s ongoing and planned investment in Navy Cash lacks justification as a cost-effective course of action.

Economic Analysis Used a Cost Estimate That Omits Relevant Costs and Was Not Derived Using Key Estimating Practices A reliable cost estimate is an essential element for informed investment decision making, realistic budget formulation and program resourcing, meaningful progress measurement, proactive course correction, and accountability for results. According to the Office of Management and Budget (OMB), programs must maintain current and well-documented estimates of program costs, and these estimates must span the full expected life of the program. Without reliable estimates, programs cannot be adequately justified on the basis of reliable costs and benefits and they are at increased risk of experiencing cost overruns, missed deadlines, and performance shortfalls.

Our research has identified a number of best practices for effective program cost estimating, and we have issued guidance that associates these practices with four characteristics of a reliable cost estimate. Specifically, estimates need to be: Comprehensive: The cost estimates should include both government and financial agent costs over the program’s full life cycle, from the inception of the program through design, development, deployment, and operation and maintenance to retirement. They should also provide a level of detail appropriate to ensure that cost elements are neither omitted nor double counted, and include documentation of all cost-influencing ground rules and assumptions.

Well-documented: The cost estimates should have clearly-defined purposes, and be supported by documented descriptions of key program or system characteristics (e.g., relationships with other systems, performance parameters). Additionally, they should capture in writing such things as the source data used and their significance, the calculations performed and their results, and the rationale for choosing a particular estimating method or reference. Moreover, this information should be captured in such a way that the data used to derive the estimate can be traced back to, and verified against, their sources.

Accurate: The cost estimates should provide for results that are unbiased and not be overly conservative or optimistic (i.e., should represent the most likely costs). In addition, the estimates should be updated regularly to reflect material changes in the program, and steps should be taken to minimize mathematical mistakes and their significance. The estimates should also be grounded in a historical record of cost estimating and actual experiences on comparable programs.

Credible: The cost estimates should discuss any limitations in the analysis performed that are due to uncertainty or biases surrounding data or assumptions. Further, the estimates’ derivation should provide for varying any major assumptions and recalculating outcomes based on sensitivity analyses, and the estimates’ associated risks and inherent uncertainty should be disclosed. Also, the estimates should be verified based on cross- checks using other estimating methods.

The $100 million life cycle cost estimate, as documented in the program’s 6-year old economic analysis, does not reflect many of the practices associated with a reliable cost estimate, including several practices related to being comprehensive and well documented, and all related to being accurate and credible (see table 4).

The cost estimate of about $100 million, as documented in the program’s 2002 economic analysis, does not meet all of the practices related to being comprehensive. Specifically, it only includes costs from fiscal years 2003 through 2008 (6-year period), and it does not include both the government and financial agent costs associated with development, acquisition (non- development), implementation, and operations and support over the system’s life cycle. Moreover, it does not include FMS’s portion of the program’s cost, which is estimated to be about $100 million over a 14-year period. In addition, the cost estimate does not clearly describe how the various cost sub-elements are aggregated to produce the amounts associated with the two documented cost categories, system installation costs, and operations and maintenance costs. Therefore, it is not clear that all pertinent costs are included and no costs are double counted. Lastly, although some key assumptions have been identified, such as the ship implementation schedule, other key assumptions, such as labor rates and inflation rates, are not. As a result, the estimate cannot be considered comprehensive.

The cost estimate used in the economic analysis also addresses some, but not all, of the practices related to being well-documented. Specifically, the purpose of the cost estimate was clearly defined and a technical baseline has been documented that includes, among others things, the hardware and software specifications and planned performance parameters. However, the calculations used to derive the cost estimate, including descriptions of the methodologies used and traceability back to source data (e.g., vendor quotes, salary data), are not documented. In addition, while program officials described the estimating approach used, such as using market research and historical data to determine the costs associated with hardware, software, and installations, they did not have documentation of the methodology used to arrive at the total costs of each of these elements and how they were combined to produce the overall cost estimate. Therefore, the program’s cost estimate cannot be considered well-documented.

In addition, the $100 million documented cost estimate lacks accuracy because it does not reflect an assessment of the costs most likely to be incurred. Specifically, this estimate covers only 6 years of costs (fiscal years 2003 through 2008). In contrast, the program’s current cost estimate is about $320 million over a 14-year life cycle, and according to program officials, the program’s life cycle is being reexamined and will likely be extended.

Lastly, the $100 million cost estimate is not credible because a complete uncertainty analysis (i.e., both a sensitivity analysis and a Monte Carlo simulation) was not performed on this estimate. A sensitivity analysis reveals how the cost estimate is affected by a change in a single assumption or cost driver, such as the ship installation schedule, while holding all other parameters constant. A Monte Carlo simulation assesses the aggregate variability of the cost estimate to determine a confidence range around the estimate. Without such analyses of uncertainty, the program office cannot have confidence that the program can be completed within the cost estimate.

Program officials acknowledged the limitations in the estimate, and attributed them to turnover of staff and their current focus on deploying the system. Nevertheless, program officials stated that they intend to develop a revised cost estimate when they update the program’s economic analysis, but they had yet to establish a date for accomplishing this. Given that a significant amount of development and modernization funding remains to be invested on the program, it is important that the program office economically justify such investment.

Economic Analysis Does Not Satisfy Other Relevant Guidance According to OMB, economic analyses should meet certain criteria to be considered reasonable, such as comparing alternatives on the basis of net present value and conducting an uncertainty analysis of benefits.

The program’s December 2002 economic analysis meets one, does not meet four, and partially meets two of the seven OMB criteria governing how to perform such analyses. For example, while the analysis explained why the investment is needed, it did not consider the costs and benefits associated with at least three alternatives to the status quo, such as Eagle Cash, EZpay, or some derivative that provided for reuse of shared services among the programs. Moreover, at least three alternatives to the status quo were not assessed on the basis of net present value, using the proper discount rate to account for inflation. Instead, the analysis only qualitatively evaluated Navy Cash against its predecessor systems. For example, the analysis included evaluation of the capabilities and limitations of the predecessor systems, but did not include evaluating the relative cost and benefits of any alternatives to Navy Cash.

In addition, the program’s benefit projections erroneously counted about $40 million in cost transfers as cost savings, thus overstating projected benefits (i.e., projected benefits should only be $93 million). Transfers represent shifts of control over the spending of resources from one group to another and thus do not result in an economic gain. According to OMB guidance, transfers do not produce economic gains because the benefits to those government entities that receive such a transfer are the same as the costs borne by those government entities that provide the transfer. Moreover, no uncertainty analysis was performed on the benefit estimates. (See table 5 for the results of our analyses relative to each of the seven criteria.)

Program officials stated that they do not know why the economic analysis was not developed in accordance with OMB guidance. They also stated that they intend to update the economic analysis and, in doing so, intend to address OMB guidance. However, they did not have a date for accomplishing this because their priority is deploying the system.

Actual Accrual of Estimated Benefits Has Not Been Validated The Clinger-Cohen Act of 1996 and OMB guidance emphasize the need to develop information to ensure that IT investments are actually contributing to tangible, observable improvements in mission performance. DOD guidance also states that estimated benefits should be validated to ensure that desired outcomes are being achieved. To this end, agencies should define and collect metrics to determine whether expected benefits from a given investment are being accrued, and they should modify subsequent economic analyses to reflect the lessons learned.

Despite the fact that Navy Cash has been installed and is operating on approximately 130 ships, DON has yet to determine whether the system is actually producing expected benefits. For example, the 2002 economic analysis stated that Navy Cash would reduce cash on ships, and contribute to man-hour savings as a result of increased productivity. It also stated that it would improve quality-of-life for sailors and marines. While DON has measured the reduction in the cash onboard some ships where Navy Cash is operating, this reduction represents a transfer and is not an actual benefit. Moreover, the extent to which the system is achieving expected man-hour savings, which would constitute a true benefit, has not been measured. Lastly, customer (sailor and marine) satisfaction with the system, which is a legitimate qualitative benefit, has not been determined since a prototype of Navy Cash was installed on two ships in 2001.

Program officials stated that DON’s Manpower Analysis Center is responsible for measuring man-hour savings. Further, they said that customer satisfaction with the system was being measured through informal feedback from the sailors and marines, and they recently began a more formal customer satisfaction survey. They also stated that in updating the economic analysis, they plan to assess and reflect the accrual of actual benefits. However, they had not established a date for accomplishing this.

Key Controls for Ensuring That Defined Navy Cash Capabilities Are Delivered on Time and Within Budget Have Not Been Effectively Implemented

DOD policy and related guidance recognizes the importance of implementing a range of management controls associated with ensuring that IT investments are defined, developed, deployed, and operated efficiently and effectively. By implementing these controls, the chances of delivering systems that perform as intended, and not costing more or taking longer than necessary, are increased. These controls include requirements development and management, risk management, security management, and system quality measurement. For Navy Cash, none of these controls have been effectively implemented. Specifically, program requirements have not been adequately developed and managed; program risks have not been effectively managed; security has not been adequately managed; and data needed to measure two aspects of system quality—trends in unresolved change requests and evaluation of user satisfaction with the system—have not been collected and used.

As a result, Navy Cash is unlikely to perform in a manner that meets user and operational needs, and it is likely to cost more and take longer than necessary.

Navy Cash Requirements Have Not Been Adequately Developed and Managed

Well-defined and managed requirements are recognized by DOD guidance and relevant best practices as essential, and can be viewed as a cornerstone of effective system acquisition. Effective requirements development and management includes (1) developing detailed system requirements; (2) establishing policies and plans for managing changes to requirements, including defining roles and responsibilities, and identifying how the integrity of a baseline set of requirements will be maintained; and (3) maintaining bi-directional requirements traceability, meaning that system-level requirements can be traced both backward to higher level business or operational requirements, and forward to system design specifications and test plans.

The program office has not satisfied these three aspects of effective requirements development and management. Specifically, The program office has not developed system-level requirements for Navy Cash. System-level requirements are derived from higher-level operational requirements and are specified at a level of detail needed for system developers to design and build to. Without system requirements, the ability of the program office to understand the impact of any system change requests (i.e., cost, schedule, and performance) and thus make informed decisions about such changes, is limited. For example, although the program office identified a high-level requirement for the system to share information with the Retail Operations Management system used in ships’ store operations, the associated system-level requirements were not defined. As a result, the deployed version of the system was not designed and developed to provide this interface. The requirement for this interface was later realized after a number of system and operational problems surfaced. Addressing these problems through a series of changes required additional time and funding. Program officials acknowledged that more effective requirements development and management practices could have avoided these problems. As another example, a system requirement for automatically deploying software patches to operational systems was not defined. Had this requirement been defined, the system design could have provided for developing a capability to minimize the level of effort required to identify, distribute, and install patches. Instead, a less efficient and labor-intensive manual process has been used.

The program office does not have a policy or plans for managing requirements. Such policies and plans establish organizational roles and responsibilities for managing requirements, including maintaining and controlling modifications or changes to the baseline sets of requirements, establishing priorities among competing requests for changes, and assessing the impact on cost, schedule, and performance of each change. In lieu of a policy or plans, the program office has established an ad hoc change control process, whereby change proposals are approved or disapproved by a joint DON and FMS change control board based on a change management policy that was drafted in 2003. However, this policy was never finalized or approved and does not define roles and responsibilities or how requirements will be managed. Further, the board has not been chartered. Moreover, program officials told us that the board’s decisions are made primarily on the basis of consensus about the need for the change and the availability of funds.

Other than security requirements, Navy Cash requirements cannot be traced from the higher level business or operational requirements to system design specifications and test plans. Specifically, we attempted to trace a sample of Navy Cash system-level requirements backward to high- level requirements and forward to design documents and test plans and results. However, as noted above, no system-level requirements exist. Without this link in the requirements traceability chain, traceability could not be demonstrated. Having requirements traceability is essential for ensuring that developed and deployed system products satisfy operational needs and user expectations. In the case of Navy Cash, where system capabilities are reactive to change requests rather than proactively driven by requirements, such traceability is also essential to understanding the impact to the system of each change request and thus having an informed basis for approving and prioritizing any changes.

Program officials acknowledged these weaknesses and recently stated that they intend to address them. To accomplish this, they reported that they have hired a new employee who is to be trained in requirements development and management, and who is to develop a requirements management plan.

Until the program office employs fundamental requirements development and management practices, it cannot reliably estimate the program costs and develop schedules needed to accomplish the work associated with delivering predetermined and economically justified system capabilities. The result is an inability to develop and measure performance against meaningful cost, schedule, and capability baselines, and thereby reasonably ensure that the program is meeting expectations and those responsible for it are accountable for results.

Navy Cash’s Risks Have Not Been Effectively Managed

Proactively managing program risks is a key acquisition management control that, if done properly, can increase the chances of programs delivering promised capabilities and benefits on time and within budget. For Navy Cash, program risks have not been effectively managed. Rather, the program office has reacted to the realization of actual problems. In particular, plans, processes, and procedures are not in place that provide for identifying, controlling, and disclosing risks, and risk management roles and responsibilities have not been assigned to key stakeholders. As a result, the program office is not positioned to proactively avoid the occurrence of cost, schedule, and performance problems.

DOD and related guidance recognize the importance of performing effective risk management on programs like Navy Cash. Among other things, effective risk management includes: (1) establishing and implementing a written plan and defined process for risk identification, analysis, and mitigation; (2) assigning responsibility for managing risks to key stakeholders; (3) encouraging program-wide participation in risk management; and (4) examining the status of identified risks during program milestone reviews.

The program office has not fully satisfied any of the above cited risk management practices. For example: A written plan or defined process that provides for identifying, analyzing, and mitigating risks has not been established. In the absence of a plan and process, program officials stated that risks are informally addressed during bi-monthly program management reviews that involve key stakeholders, including the program office, FMS, and the financial agent. However, our analysis of minutes of these reviews indicates that they are more focused on reacting to the consequences of actual problems, rather than proactively attempting to avoid the occurrence of potential problems.

While program officials stated that responsibility for managing risks rests with the program manager, roles and responsibilities for managing and identifying risks have not been documented for any key stakeholders, including individuals in the program office, and with FMS and the financial agent. Without clearly documenting their roles and responsibilities, proactive identification, disclosure, and mitigation of all key risks is unlikely to occur, and program approval and decision making authorities will not be adequately informed.

While program officials stated that attending and participating in program management reviews is encouraged, we have yet to receive any verifiable evidence that risks are addressed in these reviews or that involvement in risk management is encouraged.

Program officials have yet to provide any verifiable evidence that program decision making and oversight authorities have been apprised of the status of identified risks.

Program officials acknowledged the above weaknesses and attributed them to staff turnover in key positions and their focus on deploying the system rather than establishing management processes and procedures. Nevertheless, program officials stated that they intend to develop a risk plan and process, but said that this would not occur until December 2008. Given that a significant amount of development and modernization investment remains, it is important that mitigating existing risks, including those discussed in this report, as well as future risks be treated as a program priority.

Navy Cash Security Management Has Not Been Effectively Implemented

A number of Navy Cash security management weaknesses exist. Specifically, the program office has not (1) fully implemented a comprehensive patch management process; (2) followed an adequate process for planning, implementing, evaluating, and documenting remedial actions for known information security weaknesses; (3) obtained adequate assurance that FMS has effective security controls in place to protect Navy Cash applications and data; and (4) developed an adequate contingency plan and conducted effective contingency plan testing. Program officials acknowledged these weaknesses but have yet to provide us with plans for addressing them. As a result, the confidentiality, integrity, and availability of deployed and operating Navy Cash shipboard devices, applications, and financial data are at increased risk of being compromised.

Patch Management Has Not Been Fully Implemented DOD guidance states that component organizations should develop a process for patching system vulnerabilities. Further, National Institute of Standards and Technology (NIST) guidance recognizes the importance of implementing comprehensive patch management that includes, among other things, (1) having a complete inventory of system hardware and software assets, (2) automatically deploying vulnerability patches, and (3) measuring patch management performance.

Although the program office performs patch management for Navy Cash, key practices have not been fully implemented. Specifically, A complete inventory of system assets does not exist. According to NIST, a system inventory enables organizations to monitor system hardware and software assets for the presence of all threats, vulnerabilities, and patches. While the financial agent maintains a Navy Cash asset database for the 128 ships on which the system is operating, this database is missing 3 hardware inventories and 19 software inventories. According to program officials, the financial agent’s database is incomplete because it was created from purchase orders after the system was in operation. Furthermore, although the program office maintains hardware inventories for each ship in a DON configuration management database, the office does not maintain inventories of Navy Cash software. Until the program office develops a complete inventory of Navy Cash system assets, it will not be able to identify and patch all system threats and vulnerabilities.

Vulnerability patches are not deployed in an automated or timely manner.

According to NIST guidance, deploying patches automatically minimizes the level of effort and time required to identify, distribute, and install patches. However, patches are currently deployed manually for Navy Cash when ships are in port for maintenance. As a result, the risk of vulnerabilities being exploited before ships return to port is increased. Although the program office plans to introduce the capability to automatically deploy patches as part of the next software release in the first quarter of fiscal year 2009, program officials said that it will take between 18 to 24 months to rollout this capability to the entire fleet. Program officials also stated that they do not know why this capability was not part of the original system requirements and design. Until the program office begins automatically deploying patches, Navy Cash assets and data will be exposed to increased risk.

The performance of patch management is not being measured. NIST guidance recommends consistent measurement of the effectiveness of patch management through the use of metrics, such as susceptibility to attack and mitigation response time. Although program officials stated that they maintain patch management metrics, they have yet to provide us with a description of the metrics or an explanation of how they are used. Until the program office develops and uses performance metrics, it will not be able to assess and improve the effectiveness of its patch management effort.

To strengthen its patch management efforts, the program office has developed a vulnerability management guide. However, this guide has not been finalized and approved, and according to program officials, it does not follow NIST patch management guidance. Without comprehensive patch management, increased risk exists that system vulnerabilities could be exploited.

Remedial Action Plans Have Not Been Documented The Federal Information Security Management Act (FISMA) requires that agencies’ information security programs must include a process for planning, implementing, evaluating, and documenting remedial actions to address any deficiencies in the information security policies, procedures, and practices of the agency. OMB has outlined steps for documenting remedial actions—referred to by OMB as a plan of action and milestones—for systems where IT security weaknesses have been identified. Additionally, NIST guidance states that a plan of action and milestones should be included in a system’s accreditation package and describe how the information system owner intends to address those vulnerabilities by reducing, eliminating, or accepting the identified vulnerabilities.

Since the system was accredited in November 2006, the program office has not developed any plans of action and milestones, even though medium and low information security risks were identified during security test and evaluation efforts supporting the certification and accreditation. According to program officials, the risks were accepted by the designated approving authority, rather than corrected, because they involve features that are necessary for the system to operate, such as having certain hardware interfaces and access permissions. While accepting rather than correcting such weaknesses is consistent with DON guidance for developing plans of action and milestones, it is not consistent with NIST guidance. Specifically, DON guidance states that these plans are only required for accreditation decisions that are conditional upon corrective actions being taken. However, NIST guidance specifies that the development of a plan of action and milestones should include instances where risk is being accepted.

The lack of plans of action and milestones means that the program office has not adequately addressed information security risks. Moreover, the limitations in DON guidance mean that other Navy programs may not have done so as well. Until the program office fully implements a remedial action process that meets the FISMA requirements and OMB and NIST guidance, program management and oversight officials will not have sufficient assurance that all security weaknesses are being reported and tracked, and that options for addressing them are fully considered.

Information Security Requirements Have Not Been Fully Defined FISMA requires each federal agency to develop, document, and implement an agencywide information security program to provide information security for the information and information systems that support the operations and assets of the agency, including those provided or managed by another agency, contractor, or other source. Among other things, this includes testing system management, operational, and technical security controls. Although the program office has partnered with FMS to develop and support the operation of Navy Cash, it is ultimately responsible for ensuring the security of Navy Cash systems and data.

The program office has not taken adequate steps to ensure that security controls are tested. Specifically, the memorandum of agreement between the program office and FMS does not establish requirements for FMS and the financial agent relative to periodic information security control reviews, including reviews of applicable management, operational, and technical controls, and to provide DON with copies of information security control reviews that are performed on the Navy Cash system and its supporting infrastructure. This is important because FMS—through its financial agent—provides services that support Navy Cash that must be secure, such as holding and accounting for funds distributed throughout the system and processing transactions. Although FMS has performed some management and operational control tests, such as periodic personnel and physical security assessments of selected commercial facilities that provide services and support to Navy Cash, these assessments were not designed to evaluate the technical controls of the system’s computing environment because the memorandum of agreement does not include such requirements.

Until the program office and FMS establish information security requirements for overseeing the financial agent’s technical information security controls, an increased risk exists that the confidentiality, integrity, and availability of information stored, transmitted, and processed by the financial agent can be compromised.

OMB guidance requires agencies to develop contingency plans and to test those plans at least annually. NIST guidance states that contingency plans should include a sequence of recovery activities, which describe system priorities based on business impact and notification procedures, which describe the methods used to notify personnel with recovery responsibilities. In addition, according to NIST, contingency plan tests should include explicit test objectives and success criteria for each planned activity and related procedure and documentation of lessons learned.

Although the program office has developed contingency plans for Navy Cash, it did not identify the sequence of recovery activities and notification procedures for recovery personnel in them. The sequence of activities should prioritize the recovery of system components by criticality and the notification procedures should describe the methods used to notify recovery personnel during business and non-business hours. Until the program office includes these areas in the contingency plans, it cannot ensure that system components will restore in a logical manner and that ship recovery personnel will be notified promptly when a system disruption is detected. In addition, while the program office has largely included explicit test objectives and success criteria in all the test procedures, they did not document the lessons learned. According to NIST, lessons learned can improve contingency plan effectiveness and this should be incorporated into the plan. According to program officials, NIST was not used for developing and conducting tests of the contingency plan. Without lessons learned, the program office will not be able to properly maintain and improve the contingency planning guide.

Until DON develops sufficient contingency plans and testing procedures, increased risk exists that Navy Cash systems, data, and operations will not be able to fully recover from a disruption or disaster.

Navy Cash Quality Measures Are Not Being Collected

Effective management of programs like Navy Cash depends in part on the ability to measure the quality of the system being acquired and operated. One measure of system quality is the trend in the number of unaddressed, high-priority system change requests.

Sufficient data to measure trends in open (i.e., unresolved) system change requests, which is a recognized indicator of a system’s stability and quality are not being collected. To the program’s credit, it has formed a group consisting of program office, FMS, and financial agent representatives to review and decide whether to approve requests for changes to the system. However, this group is not consistently collecting data as to when a change request is opened or closed and what the priority level of each change request is. Thus, it does not know at any given time, for example, how many change requests are pending, the significance of pending change requests, and the age of these change requests. Program officials acknowledged these weaknesses but stated that their focus has been on deploying the system. This means that the program office cannot know and disclose to DOD decision makers whether the system’s stability and maturity are moving in the right direction.

In addition, the program office has not consistently collected data on user and operator satisfaction with the system. Specifically, the program office conducted two surveys in the last 6 years—a user satisfaction survey and a shipboard merchant satisfaction survey—but neither of these surveys is meaningful. More specifically, the user satisfaction survey was done in 2002 and thus is dated; and it covered only two ships and a prototype version of Navy Cash and thus its scope is limited. In addition, neither survey produced a response rate that can be generalized and projected (about 50 percent and 20 percent for the two ships in the user survey, and about 30 percent for the merchant survey).

Program officials stated that they have relied on informal user feedback from disbursing officers, who have indicated overall satisfaction with the system. Nevertheless, they said that a survey of users and operators is being planned and expected to be completed by the fall of 2008. Without meaningful data about Navy Cash’s stability and the satisfaction of those who use it, it is not clear Navy Cash is a quality system.

Conclusions

Navy Cash’s potential duplication of other DOD programs that perform similar functions, combined with its lack of meaningful economic justification, together mean that the department does not have an adequate basis for knowing whether Navy Cash, as defined, is the most cost-effective solution to meeting its strategic business and technological needs. Because such a basis is absolutely fundamental to informed investment decision making, a compelling case exists for the department to reevaluate current plans for investing almost $60 million of additional modernization funding to further develop the system.

Even if reevaluation supports current or modified investment plans, the manner in which the program is being executed remains a source of considerable cost, schedule, and performance risk. In particular, without employing fundamental requirements development and management practices, the department cannot reliably estimate program costs and develop schedules needed to accomplish the work associated with delivering predetermined and economically justified system capabilities. In addition, without effective risk management, the department is not positioned to proactively avoid the occurrence of cost, schedule, and performance problems. Furthermore, the lack of adequate security management puts the confidentiality, integrity, and availability of deployed and operating Navy Cash shipboard devices, applications, and financial data at increased risk of being compromised. Moreover, without meaningful data about the Navy Cash’s stability and the satisfaction of those who use it, it is not clear that Navy Cash is a quality system.

To overcome each of these weaknesses, it is important to not only acknowledge them, which the program office has done, but to also treat them as program priorities, including developing and implementing plans for addressing them, which the program office has largely not done.

Recommendations

Because of the uncertainty surrounding whether Navy Cash, as defined, represents a cost-effective solution, we recommend that the Secretary of Defense direct the Secretary of the Navy to limit further investment of modernization funding in the program to only (1) deployment to remaining ships of already developed and tested capabilities; (2) correction of information security vulnerabilities and weaknesses on ships where it is deployed and operating; and (3) development of the basis for an informed decision as to whether further development and modernization is economically justified and in the department’s collective best interests.

To develop the basis for an informed decision about further Navy Cash development, we further recommend that the Secretary of Defense, direct the appropriate DOD organizations to (1) examine the relationships among DOD’s programs for delivering military personnel with smart card technology for electronic retail and banking transactions; (2) identify, in coordination with the respective program offices, alternatives for optimizing the relationships of these programs in a way that minimizes areas of duplication, maximizes reuse of shared services across the programs, and considers opportunities for a consolidated stored value card program across the military services; and (3) share the results with the appropriate organizations for use in making an informed decision about planned investment in Navy Cash.

To further develop this basis for an informed decision about Navy Cash development, we also recommend that the Secretary of Defense direct the Secretary of the Navy to ensure that the appropriate Navy organizational entities prepare a reliable economic analysis that encompasses the program’s total life cycle costs, including those of FMS, and that (1) addresses cost-estimating best practices and complies with relevant OMB cost-benefit guidance and (2) incorporates data on whether deployed Navy Cash capabilities are actually producing benefits.

To address Navy Cash information security management weaknesses and improve the operational security of the system, we recommend that the Secretary of Defense direct the Secretary of the Navy to ensure that the Navy Cash program manager, in collaboration with the appropriate organizations, take the following five actions: Develop and implement a patch management approach based on NIST guidance, which includes a complete Navy Cash systems inventory; an automated patch deployment capability; and a patch management performance vulnerability measurement capability, including metrics for susceptibility to attack and mitigation response time.

Institute a process to plan, implement, evaluate, and document remedial actions for deficiencies in Navy Cash information security policies, procedures, and practices, and ensure that this process meets FISMA requirements, as well as applicable OMB and NIST guidance.

Update the NAVSUP/FMS memorandum of agreement, in collaboration with FMS, to establish specific security requirements for FMS and the financial agent to periodically perform information security control reviews, including applicable management, operational, and technical controls, of the Navy Cash system, and to provide NAVSUP with copies of the results of these reviews that pertain to the Navy Cash system and its supporting infrastructure.

Develop a complete contingency plan to include a (1) sequence of recovery activities and (2) procedures for notifying ship personnel with contingency plan responsibilities to begin recovery activities; and to test the contingency plan in accordance with NIST guidance, including documenting lessons learned from testing.

To address DON information security guidance limitations, we also recommend that the Secretary of Defense direct the Secretary of the Navy to ensure that the Navy Operational Designated Approving Authority, as part of the Naval Network Warfare Command, updates its certification and accreditation guidance to require the development of plans of action and milestones for all above identified security weaknesses.

If further investment in development of Navy Cash can be justified, we then recommend that the Secretary of Defense direct the Secretary of the Navy, through the appropriate chain of command, to ensure that the Navy Cash program manager takes the following actions.

With respect to requirements development and management, (1) develop detailed system requirements; (2) establish policies and plans for managing changes to requirements, including defining roles and responsibilities, and identifying how the integrity of a baseline set of requirements will be maintained; and (3) maintain bi-directional requirements traceability.

With respect to risk management, (1) establish and implement a written plan and defined process for risk identification, analysis, and mitigation; (2) assign responsibility for managing risk to key stakeholders; (3) encourage program-wide participation in risk management; (4) include and track the risks discussed in this report as part of a risk inventory; and (5) apprise decision making and oversight authorities of the status of risks identified during program reviews.

With respect to system quality measurement, collect and use sufficient data for (1) determining trends in unresolved change requests and (2) understanding users’ satisfaction with the system.

Agency Comments and Our Evaluation

Both DOD and FMS provided written comments on a draft of this report. In DOD’s comments, signed by the Deputy Under Secretary of Defense (Business Transformation) and reprinted in appendix II, the department stated that it concurred with 9 of our 11 recommendations, partially concurred with 1, and non-concurred with the remaining 1. In non- concurring with our recommendation for limiting further investment in the program, the department actually concurred with two out of three aspects of the recommendation. Nevertheless, for the aspect of our recommendation aimed at limiting further investment in the program to certain types of spending, it stated that it did not concur with limiting investment to the exclusion of needed maintenance (e.g., technology refresh) of operational systems. We agree with this comment, as it is consistent with statements in our report, including the recommendation summary on the report’s highlights page and the report’s conclusions, both of which focus on limiting investment of modernization funding only, and not operations and maintenance funding. To avoid any misunderstanding as to our intent, we clarified our report.

With respect to our recommendation for optimizing the relationships among DOD’s programs that provide smart card technology for electronic retail and banking transactions, the department stated that, while it concurs with the overall intent of the recommendation, it believes that the Office of the Under Secretary of Defense (Comptroller) is the appropriate organization to implement it. Since our intent was not to prescribe the only DOD organization that should be responsible for implementing the recommendation, we have slightly modified the recommendation to provide the department flexibility in this regard.

Notwithstanding DOD’s considerable agreement with our recommendations, the department provided additional comments on the findings that underlie several of the recommendations, which it described as needed to clarify and avoid confusion about the program. For various reasons discussed below, we either do not agree with most of these additional comments or do not find them germane to our findings and recommendations.

First, the department stated that the report’s overall findings understate the program’s discipline and conformance with applicable guidance and best practices. We do not agree. Our review extended to six key acquisition control areas, all of which are reflected in DOD’s own acquisition policies as well as other federal guidance. Effective implementation of these controls can minimize program risks and better ensure that system investments are defined in a way to optimally support mission operations and performance, as well as deliver promised system capabilities and benefits on time and within budget. However, we found that none of these key IT management controls were being effectively implemented on Navy Cash, and the department agreed with our recommendations aimed at correcting this.

Second, the department stated that the report’s findings do not accurately capture the program’s maturity since the system has been deployed to over 80 percent of its user base. While we do not question the extent to which the system has been deployed to date, and in fact state in our report that the system has been deployed to about 80 percent of the fleet, we do not agree that the program is mature, as evidence by the numerous IT management control weaknesses that we found and the fact that about $60 million in modernization funding remains to be spent on the system.

Third, the department stated that it recognizes that some security management limitations exist, but added that these limitations do not pose a serious risk to the confidentiality, integrity, or availability of the deployed system, and that our report may cause cardholders to become unnecessarily concerned. We do not agree that these limitations do not pose a serious risk. Our report details a number of serious security management weaknesses relative to both DOD and NIST guidance, such as not following an adequate process for planning, implementing, evaluating and documenting remedial actions for known information security vulnerabilities, as well as not obtaining adequate assurance that FMS has effective security controls in place to protect Navy Cash applications and data. As a result, we appropriately conclude in our report that such failures to effectively manage Navy Cash security places the confidentiality, integrity, and availability of deployed and operating shipboard devices, applications, and financial data at increased risk of being compromised. Swift implementation of our recommendations is the best solution to alleviating any cardholder concerns that may arise from these weaknesses.

In FMS’s comments, signed by the Commissioner of FMS and reprinted in appendix III, the service stated that our recommendations will help strengthen Navy Cash and that it has begun addressing our findings and recommendations. In addition, it stated that it will support DOD in implementing the recommendations, and consistent with DOD, commented that it did not agree with one part of one of our recommendations, adding that limiting investment in Navy Cash beyond fielding and maintaining already tested system capabilities would place future operations at risk. As stated above, this recommendation is focused on limiting further investment in modernization funding, not operations and maintenance funding. To avoid any confusion about this, we have added language to other parts of the report to emphasize this focus.

In addition to the above, and notwithstanding its overall agreement with our recommendations, FMS provided other comments relative to several of the findings that underlie our recommendations. As discussed below, we either do not agree with these additional comments or do not find them to be germane to our findings and recommendations.

First, FMS stated that our report does not identify a security breach, loss of cardholder or government funds, unauthorized release of personal or other sensitive information, or any other compromise of system integrity. We agree that our report does not identify these things, as the scope of work was not intended to identify them. Rather, our scope focused on the program’s implementation of key security management controls outlined in DOD and NIST guidance. In this regard, we found serious information security management control weaknesses and concluded that these weaknesses increased the risk to the confidentiality, integrity, and availability of information stored, transmitted, and processed by the financial agent.

Second, FMS stated that the issue of whether Navy Cash is duplicative of other similar DOD smart card programs was addressed before Navy Cash was initiated in 2001, when DON and FMS determined that for technical and cost reasons it could not alter the other DOD programs to meet Navy Cash requirements. We do not find this comment relevant to our recommendation because our point is not that one of the other DOD programs should be altered and used in place of Navy Cash. Rather, our point is that these smart card programs need to be looked at collectively to decide whether it is in the department’s best interest to continue investing in separate smart card programs or to invest in a single department-wide solution. This point is consistent with FMS’s stated goal of having a single smart card for DOD.

Third, FMS stated that it disagreed with our finding that the Navy Cash benefits projection erroneously counted $40 million as cost savings rather than cost transfers, adding that this value represents not merely a transfer between agencies but actual savings to the United States. While we do not disagree that this interest savings represents a benefit to the United States government, it also represents a cost—interest foregone—to holders of Treasury debt. Therefore, the interest savings represents a transfer rather than savings from one member or sector to another.

We are sending copies of this report to interested congressional committees; the Director, Office of Management and Budget; the Congressional Budget Office; the Secretary of Defense; the Secretary of the Treasury; and the Department of Defense Office of the Inspector General. We also will make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site http://www.gao.gov.

If you or your staffs have any questions on matters discussed in this report, please contact Randolph C. Hite at (202) 512-3439 or hiter@gao.gov, or Gregory C. Wilshusen at (202) 512-3789 or wilshuseng@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Objective, Scope, and Methodology

Our objective was to determine whether the Department of the Navy (DON) is effectively implementing information technology management controls on Navy Cash. We selected Navy Cash primarily because the Department of Defense’s (DOD) inventory of DON systems identified the program as one of DON’s five largest development and modernization investments. To address the objective, we focused on the following management areas (1) architectural alignment; (2) economic justification; (3) requirements development and management; (4) risk management; (5) security management; and (6) system quality measurement. In doing so, we analyzed a range of program documentation, such as the acquisition strategy, business case, economic analysis, agreements between the partnering organizations, and interviewed cognizant officials, such as the Milestone Decision Authority, program manager, and Financial Management Service (FMS) and financial agent officials responsible for Navy Cash.

To address architectural alignment, we reviewed the program’s business enterprise architecture (BEA) compliance assessments and system architecture products as well as versions 4.0, 4.1, and 5.0 of the BEA and compared them to the BEA compliance requirements described in the Ronald W. Reagan National Defense Authorization Act for Fiscal Year 2005 and DOD’s BEA compliance guidance and evaluated the extent to which the compliance assessments addressed all relevant BEA products. We also reviewed DOD guidance for program architecture development, such as DOD’s Business Transformation Guidance, and compared Navy Cash’s program architecture development activities to this guidance. In addition, we interviewed Navy Cash and FMS officials, as well as Navy Cash’s Milestone Decision Authority, and requested related documentation on the potential duplication between Navy Cash and other DOD programs that involve the use of smart card functionality, such as the Air Force’s and Army’s Eagle Cash and EZpay programs.

To address the program’s economic justification, we reviewed the latest economic analysis to determine the basis for the cost and benefit estimates. This included evaluating the analysis against Office of Management and Budget guidance and GAO’s Cost Assessment Guide. In addition, we interviewed cognizant program officials, including the Navy Cash program manager and FMS, regarding their respective roles, responsibilities, and actual efforts in developing and/or reviewing the economic analysis and the extent to which measures and metrics showed that projected benefits in the economic analysis were actually being realized. We also interviewed cognizant officials such as the Milestone Decision Authority about the purpose and use of the program’s economic analysis for managing the investment in the Navy Cash program.

To address requirements development and management, we reviewed relevant program documentation, such as the concept of operations document, and interviewed relevant program officials and evaluated this information against relevant best practices. We also reviewed interface requirements documents, minutes of program management meetings, and traceability of security requirements. In addition, we interviewed program officials involved in the requirements management process to discuss the change control process they use and their roles and responsibilities for managing requirements.

To address risk management, we reviewed relevant risk management documentation, such as program management review meeting minutes and compared the program office’s activities with DOD’s risk management guidance and related best practices. We analyzed the effectiveness of the program’s management reviews in terms of managing risks. In doing so, we interviewed cognizant program officials responsible, such as the program manager, Milestone Decision Authority, and FMS officials to discuss their roles and responsibilities and obtain clarification on the program’s approach to managing risks associated with acquiring and implementing Navy Cash.

To address security management, we reviewed relevant security documentation, such as DOD and National Institute of Standards and Technology information security guidance, and the Navy Cash afloat and ashore system security authorization agreements. In addition, we observed the system in operation aboard the USS Theodore Roosevelt and discussed security issues with ship personnel, program office, FMS, and financial agent officials. We also reviewed USS Harry S. Truman contingency plan test results. Additionally, we reviewed a database used to maintain the inventory of Navy Cash hardware and software assets as a part of our analysis on the Navy Cash vulnerability management program. Furthermore, we interviewed cognizant DON, FMS, and financial agent officials to discuss their roles and responsibilities and obtain clarification on the program’s approach to protecting the confidentiality, integrity, and availability of Navy Cash systems and information.

To address system quality measurement, we reviewed program documentation, such as change request logs, and a plan of action and milestones for change requests. We also compared the program’s data collection and analysis practices relative to these areas to program guidance and best practices. We reviewed the plans for and results of surveys that were performed on user and shipboard merchant satisfaction with Navy Cash, and we interviewed program management and technical officials.

We conducted our work at DOD offices and program office and ship facilities in the Washington, D.C. metropolitan area, Norfolk, Virginia, and Mechanicsburg, Pennsylvania, between June 2007 and September 2008, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objective. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective.

Appendix II: Comments from the Department of Defense

Appendix III: Comments from the Department of the Treasury, Financial Management Service

Appendix IV: GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

In addition to the contact persons named above, key contributors to this report were Neelaxi Lakhmani (Assistant Director), Jenniffer Wilson (Assistant Director), Ed Glagola (Assistant Director), Monica Anatalio, Carolyn Boyce, Harold Brumm, West Coile, Neil Doherty, Cheryl Dottermusch, Joshua Hammerstein, Mustafa Hassan, Michael Holland, James Houtz, Ethan Iczkovitz, Rebecca LaPaze, Anh Le, Josh Leiling, Mary Marshall, Karen Richey, Melissa Schermerhorn, Karl Seifert, Jonathan Ticehurst, and Adam Vodraska.