Information Technology: Actions Needed to Fully Establish Program Management Capability for VA's Financial and Logistics Initiative
Released: 2009-12-01T00:00:00

Letter

Background

In recognition of their service to our country, the Department of Veterans Affairs (VA) provides medical care, benefits, social support, and lasting memorials to veterans and their families. It is the second-largest federal department with approximately 250,000 employees. In fiscal year 2008, VA reported incurring $97 billion in obligations for its overall operations.

VA provides services to veterans and their families primarily through its three line administrations: The Veterans Health Administration operates a nationwide network of 154 hospitals, 995 outpatient clinics, 135 community living centers, 49 residential rehabilitation treatment programs, and 232 community-based counseling centers.

The Veterans Benefits Administration provides assistance and benefits such as veterans’ compensation, survivors’ benefits, and employment assistance through 57 veterans’ benefits regional offices.

The National Cemetery Administration manages 130 national cemeteries.

To support its services to veterans and their families, VA relies on an assortment of business systems, including 13 different systems that currently support its asset and financial management. However, the department has long recognized that its business systems and processes are inefficient and do not effectively support the department’s mission. For example, according to the department, systems are not integrated, manual entry that involves labor-intensive accounting processes is business processes are not standardized, and processes and systems require multiple entry of business information and result in untimely financial reporting.

Since fiscal year 1991, the department has reported on the need for an integrated financial management system and has reported financial management system functionality as a material weakness. This weakness continues to exist because many of VA’s systems are outdated, leading to inefficiencies in the reliable, timely, and consistent preparation, processing, and analysis of financial information for the department’s consolidated financial statements. To address this weakness and to improve stewardship and accountability over its resources, VA has for over a decade been pursuing improvements in its business processes and replacement of its existing financial and asset management systems with an integrated financial management system.

Assessments of CoreFLS

The department’s first attempt to replace its financial and asset management systems, CoreFLS, began in 1998. The goal of this modernization effort was to develop a single system to integrate the many financial and asset management systems used across the department. VA had planned to complete CoreFLS in March 2006; however, it terminated development of the system in July 2004 after CoreFLS pilot tests determined it did not fully support the department’s operations and that the initiative suffered from significant project management weaknesses.

According to VA’s Office of Inspector General (OIG), the department had obligated about $249 million of the $472 million that had been budgeted for the initiative by the time of its termination.

Following the failed CoreFLS pilot tests, VA hired Carnegie Mellon University’s Software Engineering Institute (SEI) to perform an independent assessment of the project. In June 2004, SEI identified a number of management and technical deficiencies that had undermined the success of the initiative. SEI identified multiple findings related to problematic technical and functional execution, as well as poor management execution. Technical and functional problems included CoreFLS’s inability to perform essential financial management functions, security weaknesses, and usability. Management problems were identified in the areas of acquisition and program management, business process re- engineering, and transition planning.

In addition, in August 2004, VA’s OIG reported multiple findings related to CoreFLS deployment, such as inadequate training, inability to monitor fiscal and acquisition operations, inaccurate data, and project management and security weaknesses.

Further, in August 2007, VA’s Management Quality Assurance Service (MQAS) summarized findings from four CoreFLS reviews completed between August 2005 and August 2006. Among the findings, MQAS identified numerous fiscal and contract administration issues resulting from poor administrative internal controls such as improper reimbursements of task orders and travel expenses.

Collectively, VA identified 141 findings related to problems with the CoreFLS initiative, which the department categorized into functional areas of responsibility such as acquisition management, organizational change management, program management, and systems engineering. In a subsequent effort to capture lessons learned and ensure that mistakes from CoreFLS would not be repeated in later initiatives, VA developed a repository, in which it aggregated the findings from the three independent reviews of the initiative.

Establishment and Management of the FLITE Program

In September 2005, in a subsequent effort to replace its financial and asset management systems, VA began work on FLITE. In this regard, the department undertook activities related to planning and requirements development. For example, the department documented business requirements and business processes, initiated coordination for reporting and financial data warehouse development, conducted a market analysis of providers with the software and hosting capability to support VA’s existing financial management system, established key personnel requirements to provide program support and awarded a program support contract, and started developing numerous planning documents (e.g., program management plan, acquisition plan, and concept of operations).

According to VA’s planning documents, FLITE is a multiyear development effort that is projected to deliver a fully operational system by 2014 at a total estimated cost of $608.7 million. The overall objectives of the FLITE program are to implement accessible and enterprise-level standardized business processes that result in increased efficiencies and enhanced internal controls; provide VA executives and managers with timely, transparent financial and asset management information to make and implement effective policy, management, stewardship, and program decisions; and provide business data and information in a secure, shareable, open, and efficient manner to facilitate a service-oriented atmosphere.

The FLITE program includes two main projects to acquire the integrated asset and financial management system: an asset management component, referred to as the Strategic Asset Management (SAM) initiative, and the financial management component, referred to as the Integrated Financial Accounting System (IFAS). The program also includes a third project, to acquire a data warehouse that is intended to provide financial and logistics data reporting and analysis.

SAM is intended to consolidate the asset and inventory management functions and the associated work management processes currently performed by multiple legacy applications into an advanced integrated system. It is to be the system of record for VA’s physical assets and perform asset and inventory management, real property management, information technology (IT) asset management, and work order and project management functions currently performed by multiple legacy applications. VA has chosen IBM’s Maximo Enterprise Asset Management software suite to implement these capabilities.

IFAS is to be the financial, procurement, and accounting management component, and, together with SAM, is intended to replace VA’s legacy Financial Management System (FMS) and the Integrated Funds Distribution, Control Point Activity, Accounting, and Procurement (IFCAP) system.

The data warehouse is projected to consolidate data from multiple transactional systems, primarily SAM and IFAS, for improved reporting, querying, and analysis capability. It is also intended to allow users to run larger and more complex queries and reports faster, without affecting the performance of the source systems. ce systems.

Figure 1 shows a simplified view of the program’s components. Figure 1 shows a simplified view of the program’s components.

The program is a collaborative effort between the Assistant Secretary for Information and Technology, who serves as VA’s Chief Information Officer, and the Assistant Secretary for Management, who serves as VA’s Chief Financial Officer. Various groups within VA have different roles and responsibilities for overseeing and managing programs. Figure 2 depicts the relationships between these oversight groups and the FLITE program.

The roles and responsibilities of each oversight group are as follows: The VA Executive Board provides the Secretary of Veterans Affairs with a forum for discussing programs with senior leadership before decisions are made.

The Strategic Management Council makes recommendations about programs to the VA Executive Board.

The Programming and Long Term Issues Board focuses on long term multiyear program planning.

The Budgeting and Near Term Issues Board is responsible for overseeing budget formulation and execution activities.

The IT Leadership Board is responsible for adjudicating inter- and intraboard issues about programs that cannot be resolved between the Programming and Long Term Issues and Budgeting and Near Term Issues Boards.

The FLITE Oversight Board is responsible for making decisions regarding FLITE business requirements, policies, and standards.

The FLITE Program Office is responsible for overseeing and coordinating all aspects of the program. The office is responsible for performing these functions through the Program Director’s Office (PDO), which is responsible for business requirements and processes, and the IT Program Management Office (PMO), which is responsible for technical solutions. Project teams are responsible for managing SAM, IFAS, and the data warehouse. In addition, other VA organizations provide the office with quality assurance, acquisition, and technology support. These program- specific and VA supporting organizations are depicted in figure 3.

Table 1 describes the components that comprise the program office and supporting VA organizations.

Planned Implementation of FLITE

VA is employing a multiphase approach for both the SAM and IFAS projects, which are to be implemented by contractors using commercial off-the-shelf systems. Specifically, these components are to be implemented through sequenced acquisitions and phased deployment and integration. The systems are planned to be implemented initially at pilot sites and subsequently refined and validated at beta sites before national deployment. The purpose of the pilot phase is to perform a final validation of the selected commercial off-the-shelf system and associated business processes in a production environment, gain experience in deploying the system, and obtain acceptance from the user community. The beta phase is to further hone the rollout capabilities by deploying the system to a limited number of sites that span the range of VA’s organizational environments. Following the beta phase, the department plans to incorporate lessons learned from both phases and produce a set of repeatable processes that can be employed during national deployment of the system.

For SAM, the department’s plans include implementation at one pilot site and 15 beta sites. The SAM pilot contractor is to evaluate and analyze VA’s business processes and requirements for a fit with the Maximo software’s capabilities and produce updated business process documents based on the department’s needs. Also, the contractor is to train the users at the pilot site, as well as provide operations and maintenance and help desk services. The pilot phase is expected to last for an estimated 12 months. Subsequent to the pilot, the department plans to deploy SAM at 15 VA beta sites over a period of approximately 12 months. The component is expected to be deployed nationwide over 21 months, with its completion expected by May 2013.

Plans for IFAS include implementing the FMS replacement at five pilot/beta sites and implementing the IFCAP replacement at two pilot/beta sites. The IFAS pilot phase is currently scheduled to begin in the first quarter of fiscal year 2010. The department plans to deploy this component in two separate subphases over approximately 4 years. The first subphase, which will replace FMS with a commercial off-the-shelf financial management system, is expected to take about 2 years to complete. The second subphase, which is planned to be done concurrently with the first phase, will replace IFCAP with the IFAS commercial off-the-shelf financial management system and is expected to take just over 4 years to complete.

VA’s approach to implementing the data warehouse calls for developing the warehouse after the underlying data structures of SAM and IFAS are defined and stabilized. The department expects to complete the data warehouse in the first quarter of fiscal year 2014.

Figure 4 depicts the program timeline, from program proposal through deployment of the SAM, IFAS, and data warehouse components.

In 2009, the program office undertook various activities, including issuing the IFAS request for proposals (February), awarding a program management support contract (March), awarding the SAM pilot project contract and beginning work (April), issuing a request for proposals for independent verification and validation support (July), and initiating planning for the data warehouse (September).

According to program officials, as of September 2, 2009, the department had spent approximately $90.8 million on FLITE. This amount included $73.0 million for about 40 contract actions on behalf of the program office: $28.5 million for program management and technical support, $27.8 million for software licenses, $10.9 million for the SAM project, $5.5 million for analyses (e.g., requirements analyses), and $0.3 million for other program activities (e.g., training).

Prior GAO and OIG Reviews of FLITE

Both we and VA’s OIG have previously reported on the FLITE initiative. In a September 2008 report, we noted that key planning documents related to the initiative lacked specificity and detail, and that VA had not addressed all the findings in the CoreFLS findings repository. We recommended that VA add more specificity and details to key planning documents, suchas the concept of operations and work breakdown structure, and address all findings in the CoreFLS findings repository to minimize risk to th successful implementation of FLITE. In response to our report, as of September 2009, VA had updated key planning documents and reported that it had taken actions that addressed all of the findings identified in the repository.

In September 2009, VA’s OIG reported on VA’s effectiveness in managing the FLITE program. The office noted, among other things, that although program managers had taken steps toward addressing the CoreFLS findings, deficiencies similar to those found in CoreFLS were also evident in FLITE. For example, OIG reported that FLITE program functions were not fully staffed.

Pilot Project Is Behind Schedule

VA and its contractor have begun one of the two planned pilot systems— the SAM component. Specifically, in April 2009, the department contracted with General Dynamics Information Technology Inc. to implement Maximo at the VA Medical Center in Milwaukee, Wisconsin. Among the activities the contractor is expected to perform are analyzing business processes, documenting requirements, configuring Maximo, and performing system tests.

As of mid-September 2009, VA reported that, with the contractor only 5 months into the 1-year time period planned to complete the pilot, the project had fallen 2 months behind schedule. This 2-month schedule slip was a consequence of the contractor falling behind in its efforts to perform tasks and deliver products that are necessary to implement the pilot system. Specifically, of the 34 tasks planned to be undertaken by mid- September, the contractor reported that 11 had not yet been started— including conducting a security assessment and predeployment testing— and that of 23 tasks that had been initiated, 16 were behind schedule. For example, among the tasks that the contractor noted as behind schedule were analysis of security requirements, business process analysis, and system configuration. Regarding the seven remaining tasks, two had reportedly been completed and five were identified as being on schedule. The contractor reported that it had completed a requirements traceability matrix and was on schedule with respect to starting up a project management office, performing organizational change management activities, and developing quality assurance and control programs.

Further, with respect to the delivery of products, the contractor reported that it had delivered only 7 of 37 products due by mid-September. The SAM project management plan and the requirements management plan were among the products that were delivered. Products that had not yet been delivered included the Maximo system configuration document, intended to provide detailed instructions to enable a trained Maximo administrator to incorporate all VA configuration requirements, and the SAM system security plan.

VA attributed the project being 2 months behind schedule to a shortage of FLITE program office human capital resources and poor project management by the contractor. Specifically, according to the program director, the program did not have the personnel it needed during the initial months of the SAM pilot project to provide the contractor with the information it needed to make planned progress. Regarding the contractor’s project management, VA stated that the contractor provided a project manager who did not possess the skills necessary to deliver quality and timely products, delayed hiring a project scheduler and used an initial project scheduling approach that was incorrect, used an ineffective and inefficient approach to analyzing VA’s business processes and underestimated the time needed to obtain a thorough understanding of the processes, and underestimated the effort necessary to configure a database server used in the pilot’s development environment.

In mid-September, the FLITE program director stated that the department had filled almost all of the program office vacancies and that the contractor had begun to improve its project management weaknesses. Nevertheless, according to the program director, while the department does not expect any further delays in completing the SAM pilot, it does not expect to recover the 2-month schedule slippage that has already occurred. As a result, the department projected completion of the pilot in 14 months, instead of 12 months as originally planned.

Additionally, activities are under way to initiate the IFAS pilot. Specifically, the department issued a request for proposals for a pilot contractor in February 2009. A contract for the IFAS pilot is planned for award in late October 2009.

VA Has Recently Filled FLITE Program Staff Positions but Has Not Yet Fully Established Other Program Management Capabilities

VA has taken steps to institute effective management of the FLITE program; however, the department has not yet fully established key capabilities needed to ensure that system components will be implemented as planned. The department recently made progress toward filling program office staff vacancies. Nonetheless, more work is needed to fully establish program management capabilities in areas that are important to the development of its integrated financial and logistics system. Until VA completes efforts to develop and reconcile its cost estimate; comply with EVM system standards; implement performance measures for its schedule; include all relevant federal and system requirements; and perform effective, independent verification and validation, it will have increased risk that FLITE will experience cost overruns and schedule delays and will not provide the capabilities that users need.

VA Recently Filled Vacant Program Office Positions

Our past work has found that the success of federal programs depends on having effective strategic human capital management and, in particular, having the right number of people with the right mix of knowledge and skills. VA has recently taken steps to fill long-standing vacancies in the FLITE program that have adversely impacted the program’s ability to maintain schedules. Specifically, in mid-September, the program acquired 36 staff, filling 111 of 112 required positions. According to the Acting Assistant Secretary for the Office of Management, vacant FLITE program positions were filled by individuals who were reassigned, detailed, or newly hired when the VA Deputy Secretary became aware of the program’s need for staff resources. As a result of the department’s recent actions to fill vacant positions, the office should be better positioned to effectively manage the program.

VA’s FLITE Cost Estimate Is Incomplete

Federal guidelines recommend that operations and maintenance costs over the entire estimated life cycle of an investment be included in a cost estimate. Inclusion of these costs over the time period corresponding to the life of the investment is encouraged by the federal government’s guidance for managing capital assets because such costs are a key element for establishing the total cost of ownership. Further, our Cost Estimating and Assessment Guide describes effective cost-estimating practices, including performance of a risk and uncertainty analysis and development of an independent cost estimate that provides an unbiased test of whether the program’s estimate is reasonable. Typically, the two estimates are reconciled.

In August 2008, the FLITE program office developed a program cost estimate of $608.7 million for fiscal years 2007 through 2014—when FLITE systems are expected to achieve full operational capability. However, the office did not project operations and maintenance costs over the entire estimated life of the FLITE investment, and it did not perform a risk and uncertainty analysis as encouraged by best practices. Program officials stated that they did not consider life-cycle operations and maintenance costs in their estimate because they wanted to capture only the cost for developing the FLITE system up to its full operational capability. Also, rather than perform a risk and uncertainty analysis of their own, the program office planned to rely on risk analyses by an outside entity, the Department of the Navy Space and Naval Warfare Systems Command (SPAWAR), that the department engaged to generate a risk adjusted independent cost estimate.

Completed in April 2009, the SPAWAR estimate identified costs totaling $1.899 billion for the life of the program and included $1.061 billion of estimated operations and maintenance costs for fiscal year 2015 through fiscal year 2024, which represented the entire estimated life of the initiative. According to SPAWAR officials, they used our Cost Estimating and Assessment Guide as the method for developing the independent estimate. Also, to align with VA’s estimate, SPAWAR used standardized cost elements and definitions to develop a probability-based estimate of $837.8 million for fiscal years 2007 through 2014. This estimate was $229.1 million higher than the department’s estimate for this period. The department’s estimate was not derived based on standardized cost elements and probability-driven risk and uncertainty costs assessments.

VA has not yet reconciled its cost estimate with SPAWAR’s estimate. According to department officials, a significant number of end-of-fiscal- year procurement requests and the department’s prioritization of IT acquisitions had affected the timing of plans to reconcile the estimates. Program officials stated that they intend to incorporate federal polices and requirements, as well as address funding, budgetary, or contractual issues necessitated by the reconciliation. According to the officials, the department plans to initiate this work in December 2009 and to complete it in March 2010. Until the reconciliation is completed, effective administration of FLITE program planning, budgeting, acquisition, and performance management activities could be jeopardized if accurate cost data are not available to guide the execution of these functions. Completion of the reconciliation, which should include estimated operations and maintenance costs for the life of the program, is essential to increase the reliability of the FLITE cost estimate and reduce the risk that acquisition plans, budgets, and performance management activities will be unsuccessful or inefficient.

VA Has Not Performed Key Actions Necessary for Reliable EVM for FLITE

OMB and department policies require major programs to use EVM to measure and report program progress. EVM is a tool for measuring program progress by comparing the value of work accomplished with the amount of work expected to be accomplished. Such a comparison permits actual performance to be evaluated, based on variances from the cost and schedule baselines—collectively referred to as a performance measurement baseline. Identification of significant variances and analysis of their causes helps program managers determine the need for corrective actions. Before EVM analysis can be reliably performed, developing a credible cost estimate is necessary to provide program managers with a clear definition of the cost, schedule, and risks associated with the scope of work planned. These inputs are then used to create a performance measurement baseline for EVM analysis. In addition, federal policy requires that systems used to collect and process EVM data be compliant with the industry standard developed by the American National Standards Institute (ANSI) and Electronic Industries Alliance (EIA), ANSI/EIA Standard 748.

Program officials have recognized the importance of reliable EVM and finalized the FLITE Program Measurement Earned Value Management Plan in August 2009. The plan identified roles and responsibilities, applicable policy and guidance, and the program’s EVM implementation approach. According to program officials, programwide earned value reporting that will include government, program management support, and SAM project work activities is expected to begin in October 2009.

However, while VA plans to begin reporting earned value performance in October 2009, a reliable cost estimate, which is necessary for EVM reporting, is not expected to be completed by that time. Specifically, as noted earlier, the department has not reconciled its cost estimate for the program with SPAWAR’s independent cost estimate. Program officials do not expect reconciliation of the cost estimate to begin until 2 months after earned value reporting is scheduled to begin.

Additionally, VA officials have not yet ensured that all EVM systems for FLITE are certified for compliance with ANSI/EIA Standard 748. These compliance assessments are necessary to demonstrate the capability of providing reliable cost and schedule information for earned value reporting. Specifically, the compliance assessment for the SAM pilot contractor’s system has not yet been completed. While program officials did not provide information that explained why a compliance assessment of the contractor’s EVM system had not yet been completed, they stated that the contractor has a plan to obtain system certification. This activity is not expected to be complete until January 2010, 3 months after earned value reporting for the program is scheduled to begin.

Until the agency has completed reconciling its cost estimate and ensured that contractors comply with EVM system industry standards, VA will have an increased risk of reporting and managing the program based on unreliable performance data.

VA Has Not Established a Reliable Program Schedule for FLITE

GAO’s Cost Estimating and Assessment Guide states that the success of a program depends in part on having a reliable schedule that realistically depicts the program’s work activities to a specific degree of detail, reasonably indicates when those work activities will occur, estimates how long they will take to complete, and shows how the work activities are related to each other. For example, a reliable schedule would indicate when one work activity depends upon the completion of another before it can start and that required resources (e.g., labor and materials) are assigned to all activities. Overall, the schedule provides the road map for the orderly execution of a program, helps identify and address potential problems, provides a baseline to gauge progress, and promotes accountability.

VA has not yet established a schedule for the program that is reliable. Program officials stated that they baselined (i.e., formally established) an integrated master schedule in January 2009. However, in the program’s August and September 2009 Risk & Issues reports, program officials noted that the integrated master schedule was not complete and did not represent all program requirements. The reports also identified that the SAM pilot schedule (a key component of the overall program schedule) did not include sufficient detail to trace project tasks to contract requirements.

Our analysis also concluded that the schedule was unreliable and noted that, in addition to issues VA identified with the program schedule, the integrated master schedule did not include key program management activities for reconciling the program cost estimate and implementing EVM, nor did it identify resources assigned to activities already under way or expected to start in the near future. Further, the schedule did not identify all dependencies and activities and did not break down all dependencies and activities to a sufficient level of detail to measure performance. Program officials acknowledged these deficiencies and stated that program management staffing shortages and delays in receiving a reliable project schedule from the SAM contractor have affected their ability to produce a reliable schedule for the program. They stated that in July 2009, they began working with stakeholders to address schedule issues and plan to improve the reliability of their schedule by finalizing a revised integrated master schedule by October 2009.

Until VA completes a revised integrated master schedule that includes all key program activities broken down to a sufficient level of detail and identifies all resources and dependencies, the program’s efforts to measure progress and identify potential problems will be impaired, and the program will have increased risk of missing critical milestones for system delivery.

VA Has Not Identified and Effectively Managed All FLITE Requirements

According to SEI guidance, the requirements for a system should describe the functionality needed to meet user needs and perform as intended in the operational environment. Federal agencies also must ensure that their financial management systems comply with federal standards mandated by the Federal Financial Management Improvement Act of 1996. Also according to SEI guidance, an organization can ensure syste requirements are based on business requirements by tracking the requirements from inception of the project and agreement on a spe of business requirements to development of the system requirements, detailed design, code implementation, and test cases necessary for validating the requirements. Requirements must be traceable forwar backward (i.e., bidirectional traceability) through the development life cycle. Traceability helps reduce the risks of fielding a system that does n meet the needs of its users, incurring schedule delays, and increasing costs.

VA has developed an initial set of 1,700 requirements that need to be addressed in the development of the SAM and IFAS components. FLIT requirements consist of core financial and procurement requirements related to the IFAS project, as well as inventory, supply, and real prope requirements related to the SAM project. To develop the initial set of requirements for FLITE, program officials stated that they analyzed VA ’s current and planned financial and asset management business processes and researched the Financial Systems Integration Office’s (FSIO) Core Financial System Requirements and Inventory, Supplies, and Materials System Requirements publications. The initial set of requirements wasfurther defined and refined by obtaining input from consultants and VA financial and asset management experts. The department included all mandatory core financial system requirements in its IFAS requirements but did not include all mandatory inventory, supplies, and materials requirements in its SAM requirements. For example, our analysis sho that VA did not include requirements for recording whether goods and services are accepted or rejected and for performing a systematic review wed and follow-up of overdue in-transit items. Program officials explained that they did not include these requirements because they had not determined whether the requirements were applicable to the SAM project. The officials agreed to incorporate the missing requirements. VA is also in the process of finalizing its real property requirements for the SAM beta phase and still plans to develop additional requirements related to procurement for IFAS. Further, the department is identifying data analysis and reporting requirements for the data warehouse.

Regarding requirements traceability, SAM project officials acknowledge that mapping system requirements to the related business requirements is fundamental to effective requirements management. However, according to FLITE officials, they made a business decision not to establish bidirectional traceability between the business and system requirements included in the SAM pilot request for proposals. Instead, they decided to require the pilot contractor to establish traceability between the business and system requirements after the contractor analyzes and refines the requirements. According to the officials, the contractor plans to complete these tasks by December 2009. In addition, program officials stated that they plan to establish bidirectional traceability between IFAS business and system requirements under the IFAS implementation contract scheduled to be awarded in October 2009. In this regard, the IFAS request for proposals states that the implementation contractor will be required to finalize IFAS requirements, as well as maintain and document the traceability of all requirements to design, develop, integrate, and test specifications.

As the department develops its requirements, it is important that all relevant and applicable federal financial management system requirements be identified and incorporated into the program’s requirements to ensure its planned financial management systems meet users’ needs and comply with applicable federal laws. Further, until they have established traceability between the business and system requirements, VA will not be positioned to know whether the system requirements are complete and effectively address each business requirement.

VA Has Begun Independent Verification and Validation of the FLITE Program, but All Findings Have Not Been Addressed in a Timely Manner

According to recognized industry standards and our prior reports, the purpose of independent verification and validation is to provide an independent review of system processes and products to ensure that quality standards are being met. As we have previously noted, the use of independent verification and validation is a recognized best practice for large and complex system development and acquisition programs such as FLITE and involves an independent organization conducting unbiased reviews of processes, products, and results to verify and validate that they meet stated requirements and standards. VA policy recognizes the importance of addressing independent verification and validation results in a timely manner.

Recognizing the importance of independent verification and validation, the department’s Systems Quality Assurance Service was tasked with performing independent verification and validation activities for the FLITE program. In April 2009, this organization developed a Software Quality Assurance Plan to guide independent verification and validation activities for the program. The plan was developed consistent with industry standards and generally contained the required elements. The plan also outlined reviews that would be performed by the Systems Quality Assurance Service, including product (e.g., program and project deliverables), process, internal controls, test readiness, and production readiness reviews. In addition, the Systems Quality Assurance Service is responsible for advising and assisting with the program’s implementation of a suite of tools to support requirements management, change management, risk, and test management.

Independent verification and validation of the FLITE program has been focused primarily on the review of program and project deliverables. According to program officials, as of September 2009, the Systems Quality Assurance Service had reviewed 30 FLITE work products and provided findings and recommendations to document owners. Out of 1,064 total findings, 947 (approximately 89 percent) had been fully addressed by the program or had been identified as obsolete by the Systems Quality Assurance Service. Of the 117 remaining findings, 59 had been addressed but had not yet been reflected in revised documents, and 58 required additional attention. Of the 58 findings and recommendations that remained open, the SAM pilot site readiness plan accounted for 18 that were identified in December 2008. According to the Systems Quality Assurance Service, these findings focused on the need for consistency with other project documentation, clarity in the timing of site activities, and incorporation of planned site-level activities into the program work breakdown structure. In addition, according to department officials, the FLITE acquisition strategy has two findings and recommendations that were identified in December 2008 and that remain to be addressed. These findings are related to VA’s approach for acquiring SAM and IFAS integration support and the program’s focus on front-end acquisition activities, rather than full life cycle acquisition processes. Unknown or incomplete system integration requirements may result in significant rework and adversely impact the program’s cost, schedule, and quality.

According to FLITE program officials, they have not had the human capital resources they need to address all the independent verification and validation findings and recommendations in a timely manner. As a result, independent verification and validation findings that highlight important program issues (e.g., determining an approach for integrating SAM and IFAS) have not received the attention that they need. As discussed earlier, the staff resources recently added could help address the program’s inability to focus sufficient attention on resolving findings from initial independent verification and validation activities. It remains unclear whether the program office will be positioned to efficiently resolve findings raised when the scope of independent verification and validation activities expands to include system testing and production readiness reviews, which affect the extent to which FLITE components will meet stated requirements and quality standards.

Conclusions

The pilot for VA’s new asset management system has experienced a 2- month schedule delay just 5 months after award of the contract. While VA has recently taken steps to address the staffing shortages that have substantially contributed to this delay, it has not yet fully established the management capability necessary for FLITE to be successful. For example, the department’s program cost estimate did not represent total program costs, nor has the estimate been reconciled with an independent estimate—a process that could increase its reliability. Further, it has not conducted EVM that is needed to ensure the reliability of the department’s programwide reporting on the initiative. Also, VA has not yet made revisions that are needed to increase the reliability of the program’s integrated master schedule. In addition, the requirements for the two major program systems, SAM and IFAS, do not yet address all the functions expected of federal asset management and financial management systems. Finally, key findings from independent reviews of the program have not been fully addressed on a timely basis.

As a consequence, the department is faced with significant challenges in implementing FLITE’s pilot systems as planned, while simultaneously working to fully establish program management capabilities. Program officials recognize the importance of reconciling their cost estimate, ensuring compliance with EVM system standards, establishing a reliable schedule, ensuring all relevant federal and system requirements are identified and traceable, and addressing all independent verification and validation findings. Further, they have stated that they plan to take such actions. However, just as program officials needed the department’s support in filling long-standing program office vacancies, the full support of the department’s top management is critical to ensuring that planned actions are executed. If the program is not effective in addressing its management weaknesses, the department increases the risk of repeating its unsuccessful earlier attempt to modernize the department’s financial and logistics systems.

Recommendations for Executive Action

To help guide and ensure successful completion of FLITE, the Secretary of Veterans Affairs should direct and ensure that the Assistant Secretary for Management and the Assistant Secretary for Information and Technology take the following five actions: Improve the reliability of the program cost estimate by ensuring that the estimate includes system operations and maintenance costs and that the estimate is reconciled with the independent cost estimate.

Improve the reliability of program earned value management reporting by ensuring that contractor earned value management systems comply with industry standards.

Complete a revised integrated master schedule that includes all key program activities, including reconciliation of the program cost estimate and implementation of earned value management, and identifies all resources and dependencies.

Ensure that all relevant and applicable federal financial management system requirements are included in FLITE’s requirements and establish and maintain requirements traceability.

Ensure that all comments from independent verification and validation reviews are addressed.

Agency Comments and Our Evaluation

The VA Chief of Staff provided written comments on a draft of this report. In its comments, the department concurred with our recommendations and described actions to address them. For example, the department stated that it plans to reconcile the FLITE program cost estimate with the independent cost estimate by the second quarter of fiscal year 2010; ensure that future contractors’ EVM systems comply with industry standards and begin an independent review of the program’s EVM compliance by the first quarter of 2010; and include the reconciled program cost estimate in the integrated master schedule by the third quarter of fiscal year 2010. Further, the department stated that it plans to validate the completeness of FLITE requirements by mid-November 2009 and ensure that outstanding comments from independent verification and validation reviews are addressed by mid-December 2009. If the recommendations are properly implemented, they should better position VA to effectively manage the FLITE program.

The department also provided a technical comment, which we have addressed in the report as appropriate. The department’s written comments are reproduced in appendix II.

As agreed with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from the date of this letter. At that time, we will send copies of the report to interested congressional committees, the Secretary of Veterans Affairs, and other interested parties. In addition, the report will be available at no charge on our Web site at http://www.gao.gov.

If you or your staffs have questions about this report, please contact Valerie C. Melvin at (202) 512-6304 or melvinv@gao.gov, or Kay L. Daly at (202) 512-9095 or dalykl@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

As requested, the objectives of our study were to (1) determine the status of the Financial and Logistics Integrated Technology Enterprise’s (FLITE) pilot system development and (2) evaluate key program management processes, including the Department of Veterans Affairs’ (VA) efforts to institute effective human capital management, develop a reliable program cost estimate, use earned value management, establish a realistic program schedule, employ effective requirements development and management, and perform independent verification and validation.

To determine the status of the pilot system development, we obtained and analyzed program documentation, including program management plans, contracts, schedules, briefing slides, meeting minutes, and project status reports to identify from these reports the planned FLITE pilot activities and deliverables and determined to what extent these tasks had been completed; and supplemented department program documentation and our analyses by interviewing department and contractor officials, such as the program director, and observing project status meetings.

We also evaluated VA’s progress toward implementing our prior recommendations related to adding specificity and details to key planning documents by comparing updated documents, including the Program Management Plan and Strategic Asset Management (SAM) Concept of Operations to prior versions.

To evaluate key program management processes, we compared program staffing plans with the program’s staffing resource reports to determine the extent to which program human capital needs have been met; compared the program cost estimate and estimating activities to Office of Management and Budget guidance and GAO’s Cost Estimating and Assessment Guide to determine the estimate’s completeness and the effectiveness of the estimating activities; reviewed department documentation, such as the program’s plan for earned value management implementation, and compared them to federal policy and GAO’s Cost Estimating and Assessment Guide to determine the department’s preparedness for conducting reliable earned value management; reviewed the program schedule and compared it to planned activities, deliverables, and practices described in GAO’s Cost Estimating and Assessment Guide to assess the schedule’s reliability; analyzed program documentation, including the department’s business requirements, concept of operations for FLITE, traceability matrix, and requirements management plan, to determine the extent to which they reflect practices such as those recognized by SEI and include federal financial management system requirements; and reviewed program documentation, such as the software quality assurance plan, quality management plan, and technical review reports, to determine the extent to which the program has addressed independent verification and validation findings.

We conducted this performance audit at VA headquarters in Washington, D.C., from November 2008 through October 2009 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Veterans Affairs

Appendix III: GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

In addition to the contacts named above, key contributions to this report were made by Mark T. Bird, Assistant Director; Michael S. LaForge, Assistant Director; Heather A. Collins; Neil J. Doherty; Rebecca Eyler; David A. Hong; Jacqueline K. Mai; Yvonne D. Moss; Robert L. Williams, Jr.; and Leonard E. Zapata.