Information Technology: Actions Needed to Fully Establish Program Management Capability for VA's Financial and Logistics Initiative
Published: 2009-10-26T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Since 2005, the Department of Veterans Affairs (VA) has been undertaking an initiative to develop an integrated financial and asset management system known as the Financial and Logistics Integrated Technology Enterprise (FLITE). FLITE is the successor to an earlier initiative known as the Core Financial and Logistics System (CoreFLS) that the department undertook in 1998 and discontinued in 2004 because it failed to support VA's operations. In light of the past performance of CoreFLS and the Office of Management and Budget's designation of FLITE as high risk, GAO was asked to (1) determine the status of pilot system development and (2) evaluate key program management processes, including VA's efforts to institute effective human capital management, develop a reliable program cost estimate, use earned value management (a recognized means for measuring program progress), establish a realistic program schedule, employ effective requirements development and management, and perform independent verification and validation. To do so, GAO reviewed program documentation and interviewed relevant officials.
section_title: What GAO Found
paragraphs: Contract award and performance of work tasks had been started for one of two planned pilot systems--the Strategic Asset Management system. However, as of mid-September, the project had fallen behind (by 2 months) and the contractor had missed the deadline for initiating and completing planned tasks and delivering work products such as a system security plan. In particular, the contractor had not started 11 of 34 tasks, including conducting a security assessment, and was behind schedule on 16 of the remaining 23 tasks, including analyzing business processes. Program officials generally attributed the delays to VA having insufficient program and acquisition staff to perform necessary activities associated with awarding and executing the pilot contract and to poor project management by the pilot system contractor. A second project--for the Integrated Financial Accounting System pilot--is expected to start in October 2009. VA has taken steps to institute effective management of FLITE; however, the department has not yet fully established capabilities needed to ensure that the program will be successfully implemented. Specifically, VA has (1) recently filled long-standing staff vacancies, and only one program office staff opening remains; (2) not developed a cost estimate that includes total program costs or reconciled its estimate with an independent estimate; (3) not performed key actions necessary for reliable earned value management; (4) not yet established a schedule that is reliable; (5) not identified all mandatory federal financial management system requirements and ensured that system requirements are based on business requirements; and (6) not addressed all of the findings of its independent verification and validation organization in a timely manner. Until VA reconciles its cost estimate, ensures compliance with earned value management system standards, establishes a reliable schedule, ensures all relevant federal and system requirements are identified and traceable, and addresses all independent verification and validation findings, it could continue to experience schedule delays and further increase its risk of not providing the financial and asset management capabilities that users need.

section_title: Letter
paragraphs: Since 2005, the Department of Veterans Affairs (VA) has been undertaking  an initiative to develop an integrated financial management and  information system known as the Financial and Logistics Integrated  Technology Enterprise (FLITE). FLITE is the successor to an earlier  initiative that the department undertook known as the Core Financial and  Logistics System (CoreFLS). After having reportedly spent more than $249  million on its development, the department discontinued CoreFLS because  the pilot system failed to support VA’s operations.
According to the department, FLITE is intended to fulfill the critical need  for a modernized and integrated financial and asset management  capability. Such a capability would support VA’s strategic goal to deliver  world-class service to veterans and their families through effective  communication and management of people, technology, business  processes, and financial resources. In early 2007, the Office of  Management and Budget (OMB) designated FLITE as a high-risk  information technology investment.
In light of VA’s past performance with CoreFLS and the designation of  FLITE as high risk, you requested that we (1) determine the status of pilot  system development and (2) evaluate key program management processes  for the initiative, including VA’s efforts to institute effective human capital  management, develop a reliable program cost estimate, use earned value  management (EVM), establish a realistic program schedule, employ  effective requirements development and management, and perform  independent verification and validation.
To accomplish our objectives, we reviewed relevant program  documentation and interviewed appropriate VA and contractor officials.  Specifically, to determine the status of FLITE pilot system development,  we reviewed documentation such as program management plans and  project status reports. To evaluate key FLITE program management  processes, we compared VA’s activities to plans and best practices.
We performed our work at the Department of Veterans Affairs  headquarters in Washington, D.C., from November 2008 to October 2009 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives. See appendix I for a more complete  description of our objectives, scope, and methodology.
subsections: section_title: Background
paragraphs: In recognition of their service to our country, the Department of Veterans  Affairs (VA) provides medical care, benefits, social support, and lasting  memorials to veterans and their families. It is the second-largest federal  department with approximately 250,000 employees. In fiscal year 2008, VA  reported incurring $97 billion in obligations for its overall operations.
VA provides services to veterans and their families primarily through its  three line administrations:    The Veterans Health Administration operates a nationwide network of 154  hospitals, 995 outpatient clinics, 135 community living centers, 49  residential rehabilitation treatment programs, and 232 community-based  counseling centers.
The Veterans Benefits Administration provides assistance and benefits  such as veterans’ compensation, survivors’ benefits, and employment  assistance through 57 veterans’ benefits regional offices.
The National Cemetery Administration manages 130 national cemeteries.
To support its services to veterans and their families, VA relies on an  assortment of business systems, including 13 different systems that  currently support its asset and financial management. However, the  department has long recognized that its business systems and processes  are inefficient and do not effectively support the department’s mission.  For example, according to the department,    systems are not integrated,    manual entry that involves labor-intensive accounting processes is    business processes are not standardized, and    processes and systems require multiple entry of business information and  result in untimely financial reporting.
Since fiscal year 1991, the department has reported on the need for an  integrated financial management system and has reported financial  management system functionality as a material weakness. This weakness  continues to exist because many of VA’s systems are outdated, leading to  inefficiencies in the reliable, timely, and consistent preparation,  processing, and analysis of financial information for the department’s  consolidated financial statements. To address this weakness and to  improve stewardship and accountability over its resources, VA has for  over a decade been pursuing improvements in its business processes and  replacement of its existing financial and asset management systems with  an integrated financial management system.
subsections: section_title: Assessments of CoreFLS
paragraphs: The department’s first attempt to replace its financial and asset  management systems, CoreFLS, began in 1998. The goal of this  modernization effort was to develop a single system to integrate the many  financial and asset management systems used across the department. VA  had planned to complete CoreFLS in March 2006; however, it terminated  development of the system in July 2004 after CoreFLS pilot tests  determined it did not fully support the department’s operations and that  the initiative suffered from significant project management weaknesses.
According to VA’s Office of Inspector General (OIG), the department had  obligated about $249 million of the $472 million that had been budgeted  for the initiative by the time of its termination.
Following the failed CoreFLS pilot tests, VA hired Carnegie Mellon  University’s Software Engineering Institute (SEI) to perform an  independent assessment of the project. In June 2004, SEI identified a  number of management and technical deficiencies that had undermined  the success of the initiative. SEI identified multiple findings related to  problematic technical and functional execution, as well as poor  management execution. Technical and functional problems included  CoreFLS’s inability to perform essential financial management functions,  security weaknesses, and usability. Management problems were identified  in the areas of acquisition and program management, business process re- engineering, and transition planning.
In addition, in August 2004, VA’s OIG reported multiple findings related to  CoreFLS deployment, such as inadequate training, inability to monitor  fiscal and acquisition operations, inaccurate data, and project  management and security weaknesses.
Further, in August 2007, VA’s Management Quality Assurance Service  (MQAS) summarized findings from four CoreFLS reviews completed  between August 2005 and August 2006. Among the findings, MQAS  identified numerous fiscal and contract administration issues resulting  from poor administrative internal controls such as improper  reimbursements of task orders and travel expenses.
Collectively, VA identified 141 findings related to problems with the  CoreFLS initiative, which the department categorized into functional areas  of responsibility such as acquisition management, organizational change  management, program management, and systems engineering. In a  subsequent effort to capture lessons learned and ensure that mistakes  from CoreFLS would not be repeated in later initiatives, VA developed a  repository, in which it aggregated the findings from the three independent  reviews of the initiative.
section_title: Establishment and Management of the FLITE Program
paragraphs: In September 2005, in a subsequent effort to replace its financial and asset  management systems, VA began work on FLITE. In this regard, the  department undertook activities related to planning and requirements  development. For example, the department    documented business requirements and business processes,  initiated coordination for reporting and financial data warehouse  development,   conducted a market analysis of providers with the software and hosting  capability to support VA’s existing financial management system,   established key personnel requirements to provide program support and  awarded a program support contract, and   started developing numerous planning documents (e.g., program  management plan, acquisition plan, and concept of operations).
According to VA’s planning documents, FLITE is a multiyear development  effort that is projected to deliver a fully operational system by 2014 at a  total estimated cost of $608.7 million. The overall objectives of the FLITE  program are to  implement accessible and enterprise-level standardized business  processes that result in increased efficiencies and enhanced internal  controls;   provide VA executives and managers with timely, transparent financial  and asset management information to make and implement effective  policy, management, stewardship, and program decisions; and   provide business data and information in a secure, shareable, open, and  efficient manner to facilitate a service-oriented atmosphere.
The FLITE program includes two main projects to acquire the integrated  asset and financial management system: an asset management component,  referred to as the Strategic Asset Management (SAM) initiative, and the  financial management component, referred to as the Integrated Financial  Accounting System (IFAS). The program also includes a third project, to  acquire a data warehouse that is intended to provide financial and logistics  data reporting and analysis.
SAM is intended to consolidate the asset and inventory management  functions and the associated work management processes currently  performed by multiple legacy applications into an advanced integrated  system. It is to be the system of record for VA’s physical assets and  perform asset and inventory management, real property management,  information technology (IT) asset management, and work order and  project management functions currently performed by multiple legacy  applications. VA has chosen IBM’s Maximo Enterprise Asset Management  software suite to implement these capabilities.
IFAS is to be the financial, procurement, and accounting management  component, and, together with SAM, is intended to replace VA’s legacy  Financial Management System (FMS) and the Integrated Funds  Distribution, Control Point Activity, Accounting, and Procurement  (IFCAP) system.
The data warehouse is projected to consolidate data from multiple  transactional systems, primarily SAM and IFAS, for improved reporting,  querying, and analysis capability. It is also intended to allow users to run  larger and more complex queries and reports faster, without affecting the  performance of the source systems.  ce systems.
Figure 1 shows a simplified view of the program’s components.  Figure 1 shows a simplified view of the program’s components.
The program is a collaborative effort between the Assistant Secretary for  Information and Technology, who serves as VA’s Chief Information  Officer, and the Assistant Secretary for Management, who serves as VA’s  Chief Financial Officer. Various groups within VA have different roles and  responsibilities for overseeing and managing programs. Figure 2 depicts  the relationships between these oversight groups and the FLITE program.
The roles and responsibilities of each oversight group are as follows:    The VA Executive Board provides the Secretary of Veterans Affairs with a  forum for discussing programs with senior leadership before decisions are  made.
The Strategic Management Council makes recommendations about  programs to the VA Executive Board.
The Programming and Long Term Issues Board focuses on long term  multiyear program planning.
The Budgeting and Near Term Issues Board is responsible for overseeing  budget formulation and execution activities.
The IT Leadership Board is responsible for adjudicating inter- and  intraboard issues about programs that cannot be resolved between the  Programming and Long Term Issues and Budgeting and Near Term Issues  Boards.
The FLITE Oversight Board is responsible for making decisions regarding  FLITE business requirements, policies, and standards.
The FLITE Program Office is responsible for overseeing and coordinating  all aspects of the program. The office is responsible for performing these  functions through the Program Director’s Office (PDO), which is  responsible for business requirements and processes, and the IT Program  Management Office (PMO), which is responsible for technical solutions.  Project teams are responsible for managing SAM, IFAS, and the data  warehouse. In addition, other VA organizations provide the office with  quality assurance, acquisition, and technology support. These program- specific and VA supporting organizations are depicted in figure 3.
Table 1 describes the components that comprise the program office and  supporting VA organizations.
section_title: Planned Implementation of FLITE
paragraphs: VA is employing a multiphase approach for both the SAM and IFAS  projects, which are to be implemented by contractors using commercial  off-the-shelf systems. Specifically, these components are to be  implemented through sequenced acquisitions and phased deployment and  integration. The systems are planned to be implemented initially at pilot  sites and subsequently refined and validated at beta sites before national  deployment. The purpose of the pilot phase is to perform a final validation  of the selected commercial off-the-shelf system and associated business  processes in a production environment, gain experience in deploying the  system, and obtain acceptance from the user community. The beta phase  is to further hone the rollout capabilities by deploying the system to a  limited number of sites that span the range of VA’s organizational  environments. Following the beta phase, the department plans to  incorporate lessons learned from both phases and produce a set of  repeatable processes that can be employed during national deployment of  the system.
For SAM, the department’s plans include implementation at one pilot site  and 15 beta sites. The SAM pilot contractor is to evaluate and analyze VA’s  business processes and requirements for a fit with the Maximo software’s  capabilities and produce updated business process documents based on  the department’s needs. Also, the contractor is to train the users at the  pilot site, as well as provide operations and maintenance and help desk  services. The pilot phase is expected to last for an estimated 12 months.  Subsequent to the pilot, the department plans to deploy SAM at 15 VA beta  sites over a period of approximately 12 months. The component is  expected to be deployed nationwide over 21 months, with its completion  expected by May 2013.
Plans for IFAS include implementing the FMS replacement at five  pilot/beta sites and implementing the IFCAP replacement at two pilot/beta  sites. The IFAS pilot phase is currently scheduled to begin in the first  quarter of fiscal year 2010. The department plans to deploy this component  in two separate subphases over approximately 4 years. The first subphase,  which will replace FMS with a commercial off-the-shelf financial  management system, is expected to take about 2 years to complete. The  second subphase, which is planned to be done concurrently with the first  phase, will replace IFCAP with the IFAS commercial off-the-shelf financial  management system and is expected to take just over 4 years to complete.
VA’s approach to implementing the data warehouse calls for developing  the warehouse after the underlying data structures of SAM and IFAS are  defined and stabilized. The department expects to complete the data  warehouse in the first quarter of fiscal year 2014.
Figure 4 depicts the program timeline, from program proposal through  deployment of the SAM, IFAS, and data warehouse components.
In 2009, the program office undertook various activities, including issuing  the IFAS request for proposals (February), awarding a program  management support contract (March), awarding the SAM pilot project  contract and beginning work (April), issuing a request for proposals for  independent verification and validation support (July), and initiating  planning for the data warehouse (September).
According to program officials, as of September 2, 2009, the department  had spent approximately $90.8 million on FLITE. This amount included  $73.0 million for about 40 contract actions on behalf of the program office:  $28.5 million for program management and technical support, $27.8  million for software licenses, $10.9 million for the SAM project, $5.5  million for analyses (e.g., requirements analyses), and $0.3 million for  other program activities (e.g., training).
section_title: Prior GAO and OIG Reviews of FLITE
paragraphs: Both we and VA’s OIG have previously reported on the FLITE initiative. In  a September 2008 report, we noted that key planning documents related to  the initiative lacked specificity and detail, and that VA had not addressed  all the findings in the CoreFLS findings repository. We recommended  that VA add more specificity and details to key planning documents, suchas the concept of operations and work breakdown structure, and address  all findings in the CoreFLS findings repository to minimize risk to th successful implementation of FLITE. In response to our report, as of  September 2009, VA had updated key planning documents and reported  that it had taken actions that addressed all of the findings identified in the  repository.
In September 2009, VA’s OIG reported on VA’s effectiveness in managing  the FLITE program. The office noted, among other things, that although  program managers had taken steps toward addressing the CoreFLS  findings, deficiencies similar to those found in CoreFLS were also evident  in FLITE. For example, OIG reported that FLITE program functions were  not fully staffed.
section_title: Pilot Project Is Behind Schedule
paragraphs: VA and its contractor have begun one of the two planned pilot systems— the SAM component. Specifically, in April 2009, the department contracted  with General Dynamics Information Technology Inc. to implement  Maximo at the VA Medical Center in Milwaukee, Wisconsin. Among the  activities the contractor is expected to perform are analyzing business  processes, documenting requirements, configuring Maximo, and  performing system tests.
As of mid-September 2009, VA reported that, with the contractor only 5  months into the 1-year time period planned to complete the pilot, the  project had fallen 2 months behind schedule. This 2-month schedule slip  was a consequence of the contractor falling behind in its efforts to perform  tasks and deliver products that are necessary to implement the pilot  system. Specifically, of the 34 tasks planned to be undertaken by mid- September, the contractor reported that 11 had not yet been started— including conducting a security assessment and predeployment testing— and that of 23 tasks that had been initiated, 16 were behind schedule. For  example, among the tasks that the contractor noted as behind schedule  were analysis of security requirements, business process analysis, and  system configuration. Regarding the seven remaining tasks, two had  reportedly been completed and five were identified as being on schedule.  The contractor reported that it had completed a requirements traceability  matrix and was on schedule with respect to starting up a project  management office, performing organizational change management  activities, and developing quality assurance and control programs.
Further, with respect to the delivery of products, the contractor reported  that it had delivered only 7 of 37 products due by mid-September. The  SAM project management plan and the requirements management plan  were among the products that were delivered. Products that had not yet  been delivered included the Maximo system configuration document,  intended to provide detailed instructions to enable a trained Maximo  administrator to incorporate all VA configuration requirements, and the  SAM system security plan.
VA attributed the project being 2 months behind schedule to a shortage of  FLITE program office human capital resources and poor project  management by the contractor. Specifically, according to the program  director, the program did not have the personnel it needed during the  initial months of the SAM pilot project to provide the contractor with the  information it needed to make planned progress. Regarding the  contractor’s project management, VA stated that the contractor    provided a project manager who did not possess the skills necessary to  deliver quality and timely products,   delayed hiring a project scheduler and used an initial project scheduling  approach that was incorrect,    used an ineffective and inefficient approach to analyzing VA’s business  processes and underestimated the time needed to obtain a thorough  understanding of the processes, and   underestimated the effort necessary to configure a database server used in  the pilot’s development environment.
In mid-September, the FLITE program director stated that the department  had filled almost all of the program office vacancies and that the  contractor had begun to improve its project management weaknesses.  Nevertheless, according to the program director, while the department  does not expect any further delays in completing the SAM pilot, it does not  expect to recover the 2-month schedule slippage that has already  occurred. As a result, the department projected completion of the pilot in  14 months, instead of 12 months as originally planned.
Additionally, activities are under way to initiate the IFAS pilot.  Specifically, the department issued a request for proposals for a pilot  contractor in February 2009. A contract for the IFAS pilot is planned for  award in late October 2009.
section_title: VA Has Recently Filled FLITE Program Staff Positions but Has Not Yet Fully Established Other Program Management Capabilities
paragraphs: VA has taken steps to institute effective management of the FLITE  program; however, the department has not yet fully established key  capabilities needed to ensure that system components will be  implemented as planned. The department recently made progress toward  filling program office staff vacancies. Nonetheless, more work is needed to  fully establish program management capabilities in areas that are  important to the development of its integrated financial and logistics  system. Until VA completes efforts to develop and reconcile its cost  estimate; comply with EVM system standards; implement performance  measures for its schedule; include all relevant federal and system  requirements; and perform effective, independent verification and  validation, it will have increased risk that FLITE will experience cost  overruns and schedule delays and will not provide the capabilities that  users need.
subsections: section_title: VA Recently Filled Vacant Program Office Positions
paragraphs: Our past work has found that the success of federal programs depends on  having effective strategic human capital management and, in particular,  having the right number of people with the right mix of knowledge and  skills. VA has recently taken steps to fill long-standing vacancies in the  FLITE program that have adversely impacted the program’s ability to  maintain schedules. Specifically, in mid-September, the program acquired  36 staff, filling 111 of 112 required positions. According to the Acting  Assistant Secretary for the Office of Management, vacant FLITE program  positions were filled by individuals who were reassigned, detailed, or  newly hired when the VA Deputy Secretary became aware of the  program’s need for staff resources. As a result of the department’s recent  actions to fill vacant positions, the office should be better positioned to  effectively manage the program.
section_title: VA’s FLITE Cost Estimate Is Incomplete
paragraphs: Federal guidelines recommend that operations and maintenance costs  over the entire estimated life cycle of an investment be included in a cost  estimate. Inclusion of these costs over the time period corresponding to  the life of the investment is encouraged by the federal government’s  guidance for managing capital assets because such costs are a key element  for establishing the total cost of ownership. Further, our Cost Estimating  and Assessment Guide describes effective cost-estimating practices,  including performance of a risk and uncertainty analysis and development  of an independent cost estimate that provides an unbiased test of whether  the program’s estimate is reasonable. Typically, the two estimates are  reconciled.
In August 2008, the FLITE program office developed a program cost  estimate of $608.7 million for fiscal years 2007 through 2014—when FLITE  systems are expected to achieve full operational capability. However, the  office did not project operations and maintenance costs over the entire  estimated life of the FLITE investment, and it did not perform a risk and  uncertainty analysis as encouraged by best practices. Program officials  stated that they did not consider life-cycle operations and maintenance  costs in their estimate because they wanted to capture only the cost for  developing the FLITE system up to its full operational capability. Also,  rather than perform a risk and uncertainty analysis of their own, the  program office planned to rely on risk analyses by an outside entity, the  Department of the Navy Space and Naval Warfare Systems Command  (SPAWAR), that the department engaged to generate a risk adjusted  independent cost estimate.
Completed in April 2009, the SPAWAR estimate identified costs totaling  $1.899 billion for the life of the program and included $1.061 billion of  estimated operations and maintenance costs for fiscal year 2015 through  fiscal year 2024, which represented the entire estimated life of the  initiative. According to SPAWAR officials, they used our Cost Estimating  and Assessment Guide as the method for developing the independent  estimate. Also, to align with VA’s estimate, SPAWAR used standardized  cost elements and definitions to develop a probability-based estimate of  $837.8 million for fiscal years 2007 through 2014. This estimate was $229.1  million higher than the department’s estimate for this period. The  department’s estimate was not derived based on standardized cost  elements and probability-driven risk and uncertainty costs assessments.
VA has not yet reconciled its cost estimate with SPAWAR’s estimate.  According to department officials, a significant number of end-of-fiscal- year procurement requests and the department’s prioritization of IT  acquisitions had affected the timing of plans to reconcile the estimates.  Program officials stated that they intend to incorporate federal polices and  requirements, as well as address funding, budgetary, or contractual issues  necessitated by the reconciliation. According to the officials, the  department plans to initiate this work in December 2009 and to complete it  in March 2010. Until the reconciliation is completed, effective  administration of FLITE program planning, budgeting, acquisition, and  performance management activities could be jeopardized if accurate cost  data are not available to guide the execution of these functions.  Completion of the reconciliation, which should include estimated  operations and maintenance costs for the life of the program, is essential  to increase the reliability of the FLITE cost estimate and reduce the risk  that acquisition plans, budgets, and performance management activities  will be unsuccessful or inefficient.
section_title: VA Has Not Performed Key Actions Necessary for Reliable EVM for FLITE
paragraphs: OMB and department policies require major programs to use EVM to  measure and report program progress. EVM is a tool for measuring  program progress by comparing the value of work accomplished with the  amount of work expected to be accomplished. Such a comparison permits  actual performance to be evaluated, based on variances from the cost and  schedule baselines—collectively referred to as a performance  measurement baseline. Identification of significant variances and analysis  of their causes helps program managers determine the need for corrective  actions. Before EVM analysis can be reliably performed, developing a  credible cost estimate is necessary to provide program managers with a  clear definition of the cost, schedule, and risks associated with the scope  of work planned. These inputs are then used to create a performance  measurement baseline for EVM analysis. In addition, federal policy  requires that systems used to collect and process EVM data be compliant  with the industry standard developed by the American National Standards  Institute (ANSI) and Electronic Industries Alliance (EIA), ANSI/EIA  Standard 748.
Program officials have recognized the importance of reliable EVM and  finalized the FLITE Program Measurement Earned Value Management  Plan in August 2009. The plan identified roles and responsibilities,  applicable policy and guidance, and the program’s EVM implementation  approach. According to program officials, programwide earned value  reporting that will include government, program management support, and  SAM project work activities is expected to begin in October 2009.
However, while VA plans to begin reporting earned value performance in  October 2009, a reliable cost estimate, which is necessary for EVM  reporting, is not expected to be completed by that time. Specifically, as  noted earlier, the department has not reconciled its cost estimate for the  program with SPAWAR’s independent cost estimate. Program officials do  not expect reconciliation of the cost estimate to begin until 2 months after  earned value reporting is scheduled to begin.
Additionally, VA officials have not yet ensured that all EVM systems for  FLITE are certified for compliance with ANSI/EIA Standard 748. These  compliance assessments are necessary to demonstrate the capability of  providing reliable cost and schedule information for earned value  reporting. Specifically, the compliance assessment for the SAM pilot  contractor’s system has not yet been completed. While program officials  did not provide information that explained why a compliance assessment  of the contractor’s EVM system had not yet been completed, they stated  that the contractor has a plan to obtain system certification. This activity  is not expected to be complete until January 2010, 3 months after earned  value reporting for the program is scheduled to begin.
Until the agency has completed reconciling its cost estimate and ensured  that contractors comply with EVM system industry standards, VA will have  an increased risk of reporting and managing the program based on  unreliable performance data.
section_title: VA Has Not Established a Reliable Program Schedule for FLITE
paragraphs: GAO’s Cost Estimating and Assessment Guide states that the success of  a program depends in part on having a reliable schedule that realistically  depicts the program’s work activities to a specific degree of detail,  reasonably indicates when those work activities will occur, estimates how  long they will take to complete, and shows how the work activities are  related to each other. For example, a reliable schedule would indicate  when one work activity depends upon the completion of another before it  can start and that required resources (e.g., labor and materials) are  assigned to all activities. Overall, the schedule provides the road map for  the orderly execution of a program, helps identify and address potential  problems, provides a baseline to gauge progress, and promotes  accountability.
VA has not yet established a schedule for the program that is reliable.  Program officials stated that they baselined (i.e., formally established) an  integrated master schedule in January 2009. However, in the program’s  August and September 2009 Risk & Issues reports, program officials noted  that the integrated master schedule was not complete and did not  represent all program requirements. The reports also identified that the  SAM pilot schedule (a key component of the overall program schedule)  did not include sufficient detail to trace project tasks to contract  requirements.
Our analysis also concluded that the schedule was unreliable and noted  that, in addition to issues VA identified with the program schedule, the  integrated master schedule did not include key program management  activities for reconciling the program cost estimate and implementing  EVM, nor did it identify resources assigned to activities already under way  or expected to start in the near future. Further, the schedule did not  identify all dependencies and activities and did not break down all  dependencies and activities to a sufficient level of detail to measure  performance. Program officials acknowledged these deficiencies and  stated that program management staffing shortages and delays in receiving  a reliable project schedule from the SAM contractor have affected their  ability to produce a reliable schedule for the program. They stated that in  July 2009, they began working with stakeholders to address schedule  issues and plan to improve the reliability of their schedule by finalizing a  revised integrated master schedule by October 2009.
Until VA completes a revised integrated master schedule that includes all  key program activities broken down to a sufficient level of detail and  identifies all resources and dependencies, the program’s efforts to  measure progress and identify potential problems will be impaired, and  the program will have increased risk of missing critical milestones for  system delivery.
section_title: VA Has Not Identified and Effectively Managed All FLITE Requirements
paragraphs: According to SEI guidance, the requirements for a system should describe  the functionality needed to meet user needs and perform as intended in  the operational environment. Federal agencies also must ensure that  their financial management systems comply with federal standards  mandated by the Federal Financial Management Improvement Act of  1996. Also according to SEI guidance, an organization can ensure syste requirements are based on business requirements by tracking the  requirements from inception of the project and agreement on a spe of business requirements to development of the system requirements,  detailed design, code implementation, and test cases necessary for  validating the requirements. Requirements must be traceable forwar backward (i.e., bidirectional traceability) through the development life  cycle. Traceability helps reduce the risks of fielding a system that does n meet the needs of its users, incurring schedule delays, and increasing  costs.
VA has developed an initial set of 1,700 requirements that need to be  addressed in the development of the SAM and IFAS components. FLIT requirements consist of core financial and procurement requirements  related to the IFAS project, as well as inventory, supply, and real prope requirements related to the SAM project. To develop the initial set of  requirements for FLITE, program officials stated that they analyzed VA ’s  current and planned financial and asset management business processes  and researched the Financial Systems Integration Office’s (FSIO) Core  Financial System Requirements and Inventory, Supplies, and Materials  System Requirements publications. The initial set of requirements wasfurther defined and refined by obtaining input from consultants and VA  financial and asset management experts. The department included all  mandatory core financial system requirements in its IFAS requirements but did not include all mandatory inventory, supplies, and materials  requirements in its SAM requirements. For example, our analysis sho that VA did not include requirements for recording whether goods and  services are accepted or rejected and for performing a systematic review wed  and follow-up of overdue in-transit items. Program officials explained  that they did not include these requirements because they had not  determined whether the requirements were applicable to the SAM project.  The officials agreed to incorporate the missing requirements. VA is also in  the process of finalizing its real property requirements for the SAM beta  phase and still plans to develop additional requirements related to  procurement for IFAS. Further, the department is identifying data analysis  and reporting requirements for the data warehouse.
Regarding requirements traceability, SAM project officials acknowledge  that mapping system requirements to the related business requirements is  fundamental to effective requirements management. However, according  to FLITE officials, they made a business decision not to establish  bidirectional traceability between the business and system requirements  included in the SAM pilot request for proposals. Instead, they decided to  require the pilot contractor to establish traceability between the business  and system requirements after the contractor analyzes and refines the  requirements. According to the officials, the contractor plans to complete  these tasks by December 2009. In addition, program officials stated that  they plan to establish bidirectional traceability between IFAS business and  system requirements under the IFAS implementation contract scheduled  to be awarded in October 2009. In this regard, the IFAS request for  proposals states that the implementation contractor will be required to  finalize IFAS requirements, as well as maintain and document the  traceability of all requirements to design, develop, integrate, and test  specifications.
As the department develops its requirements, it is important that all  relevant and applicable federal financial management system requirements  be identified and incorporated into the program’s requirements to ensure  its planned financial management systems meet users’ needs and comply  with applicable federal laws. Further, until they have established  traceability between the business and system requirements, VA will not be  positioned to know whether the system requirements are complete and  effectively address each business requirement.
section_title: VA Has Begun Independent Verification and Validation of the FLITE Program, but All Findings Have Not Been Addressed in a Timely Manner
paragraphs: According to recognized industry standards and our prior reports, the  purpose of independent verification and validation is to provide an  independent review of system processes and products to ensure that  quality standards are being met. As we have previously noted, the use of  independent verification and validation is a recognized best practice for  large and complex system development and acquisition programs such as  FLITE and involves an independent organization conducting unbiased  reviews of processes, products, and results to verify and validate that they  meet stated requirements and standards. VA policy recognizes the  importance of addressing independent verification and validation results  in a timely manner.
Recognizing the importance of independent verification and validation, the  department’s Systems Quality Assurance Service was tasked with  performing independent verification and validation activities for the FLITE  program. In April 2009, this organization developed a Software Quality  Assurance Plan to guide independent verification and validation activities  for the program. The plan was developed consistent with industry  standards and generally contained the required elements. The plan also  outlined reviews that would be performed by the Systems Quality  Assurance Service, including product (e.g., program and project  deliverables), process, internal controls, test readiness, and production  readiness reviews. In addition, the Systems Quality Assurance Service is  responsible for advising and assisting with the program’s implementation  of a suite of tools to support requirements management, change  management, risk, and test management.
Independent verification and validation of the FLITE program has been  focused primarily on the review of program and project deliverables.  According to program officials, as of September 2009, the Systems Quality  Assurance Service had reviewed 30 FLITE work products and provided  findings and recommendations to document owners. Out of 1,064 total  findings, 947 (approximately 89 percent) had been fully addressed by the  program or had been identified as obsolete by the Systems Quality  Assurance Service. Of the 117 remaining findings, 59 had been addressed  but had not yet been reflected in revised documents, and 58 required  additional attention. Of the 58 findings and recommendations that  remained open, the SAM pilot site readiness plan accounted for 18 that  were identified in December 2008. According to the Systems Quality  Assurance Service, these findings focused on the need for consistency  with other project documentation, clarity in the timing of site activities,  and incorporation of planned site-level activities into the program work  breakdown structure. In addition, according to department officials, the  FLITE acquisition strategy has two findings and recommendations that  were identified in December 2008 and that remain to be addressed. These  findings are related to VA’s approach for acquiring SAM and IFAS  integration support and the program’s focus on front-end acquisition  activities, rather than full life cycle acquisition processes. Unknown or  incomplete system integration requirements may result in significant  rework and adversely impact the program’s cost, schedule, and quality.
According to FLITE program officials, they have not had the human  capital resources they need to address all the independent verification and  validation findings and recommendations in a timely manner. As a result,  independent verification and validation findings that highlight important  program issues (e.g., determining an approach for integrating SAM and  IFAS) have not received the attention that they need. As discussed earlier,  the staff resources recently added could help address the program’s  inability to focus sufficient attention on resolving findings from initial  independent verification and validation activities. It remains unclear  whether the program office will be positioned to efficiently resolve  findings raised when the scope of independent verification and validation  activities expands to include system testing and production readiness  reviews, which affect the extent to which FLITE components will meet  stated requirements and quality standards.
section_title: Conclusions
paragraphs: The pilot for VA’s new asset management system has experienced a 2- month schedule delay just 5 months after award of the contract. While VA  has recently taken steps to address the staffing shortages that have  substantially contributed to this delay, it has not yet fully established the  management capability necessary for FLITE to be successful. For  example, the department’s program cost estimate did not represent total  program costs, nor has the estimate been reconciled with an independent  estimate—a process that could increase its reliability. Further, it has not  conducted EVM that is needed to ensure the reliability of the department’s  programwide reporting on the initiative. Also, VA has not yet made  revisions that are needed to increase the reliability of the program’s  integrated master schedule. In addition, the requirements for the two  major program systems, SAM and IFAS, do not yet address all the  functions expected of federal asset management and financial  management systems. Finally, key findings from independent reviews of  the program have not been fully addressed on a timely basis.
As a consequence, the department is faced with significant challenges in  implementing FLITE’s pilot systems as planned, while simultaneously  working to fully establish program management capabilities. Program  officials recognize the importance of reconciling their cost estimate,  ensuring compliance with EVM system standards, establishing a reliable  schedule, ensuring all relevant federal and system requirements are  identified and traceable, and addressing all independent verification and  validation findings. Further, they have stated that they plan to take such  actions. However, just as program officials needed the department’s  support in filling long-standing program office vacancies, the full support  of the department’s top management is critical to ensuring that planned  actions are executed. If the program is not effective in addressing its  management weaknesses, the department increases the risk of repeating  its unsuccessful earlier attempt to modernize the department’s financial  and logistics systems.
section_title: Recommendations for Executive Action
paragraphs: To help guide and ensure successful completion of FLITE, the Secretary of  Veterans Affairs should direct and ensure that the Assistant Secretary for  Management and the Assistant Secretary for Information and Technology  take the following five actions:  Improve the reliability of the program cost estimate by ensuring that the  estimate includes system operations and maintenance costs and that the  estimate is reconciled with the independent cost estimate.
Improve the reliability of program earned value management reporting by  ensuring that contractor earned value management systems comply with  industry standards.
Complete a revised integrated master schedule that includes all key  program activities, including reconciliation of the program cost estimate  and implementation of earned value management, and identifies all  resources and dependencies.
Ensure that all relevant and applicable federal financial management  system requirements are included in FLITE’s requirements and establish  and maintain requirements traceability.
Ensure that all comments from independent verification and validation  reviews are addressed.
section_title: Agency Comments and Our Evaluation
paragraphs: The VA Chief of Staff provided written comments on a draft of this report.  In its comments, the department concurred with our recommendations  and described actions to address them. For example, the department  stated that it plans to reconcile the FLITE program cost estimate with the  independent cost estimate by the second quarter of fiscal year 2010;  ensure that future contractors’ EVM systems comply with industry  standards and begin an independent review of the program’s EVM  compliance by the first quarter of 2010; and include the reconciled  program cost estimate in the integrated master schedule by the third  quarter of fiscal year 2010. Further, the department stated that it plans to  validate the completeness of FLITE requirements by mid-November 2009  and ensure that outstanding comments from independent verification and  validation reviews are addressed by mid-December 2009. If the  recommendations are properly implemented, they should better position  VA to effectively manage the FLITE program.
The department also provided a technical comment, which we have  addressed in the report as appropriate. The department’s written  comments are reproduced in appendix II.
As agreed with your offices, unless you publicly announce its contents  earlier, we plan no further distribution of this report until 30 days from the  date of this letter. At that time, we will send copies of the report to  interested congressional committees, the Secretary of Veterans Affairs,  and other interested parties. In addition, the report will be available at no  charge on our Web site at http://www.gao.gov.
If you or your staffs have questions about this report, please contact  Valerie C. Melvin at (202) 512-6304 or melvinv@gao.gov, or Kay L. Daly at  (202) 512-9095 or dalykl@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. Key contributors to this report are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: As requested, the objectives of our study were to (1) determine the status  of the Financial and Logistics Integrated Technology Enterprise’s (FLITE)  pilot system development and (2) evaluate key program management  processes, including the Department of Veterans Affairs’ (VA) efforts to  institute effective human capital management, develop a reliable program  cost estimate, use earned value management, establish a realistic program  schedule, employ effective requirements development and management,  and perform independent verification and validation.
To determine the status of the pilot system development, we    obtained and analyzed program documentation, including program  management plans, contracts, schedules, briefing slides, meeting minutes,  and project status reports to identify from these reports the planned  FLITE pilot activities and deliverables and determined to what extent  these tasks had been completed; and   supplemented department program documentation and our analyses by  interviewing department and contractor officials, such as the program  director, and observing project status meetings.
We also evaluated VA’s progress toward implementing our prior  recommendations related to adding specificity and details to key planning  documents by comparing updated documents, including the Program  Management Plan and Strategic Asset Management (SAM) Concept of  Operations to prior versions.
To evaluate key program management processes, we    compared program staffing plans with the program’s staffing resource  reports to determine the extent to which program human capital needs  have been met;   compared the program cost estimate and estimating activities to Office of  Management and Budget guidance and GAO’s Cost Estimating and  Assessment Guide to determine the estimate’s completeness and the  effectiveness of the estimating activities;  reviewed department documentation, such as the program’s plan for  earned value management implementation, and compared them to federal  policy and GAO’s Cost Estimating and Assessment Guide to determine  the department’s preparedness for conducting reliable earned value  management; reviewed the program schedule and compared it to planned activities,  deliverables, and practices described in GAO’s Cost Estimating and  Assessment Guide to assess the schedule’s reliability;    analyzed program documentation, including the department’s business  requirements, concept of operations for FLITE, traceability matrix, and  requirements management plan, to determine the extent to which they  reflect practices such as those recognized by SEI and include federal  financial management system requirements; and reviewed program documentation, such as the software quality assurance  plan, quality management plan, and technical review reports, to determine  the extent to which the program has addressed independent verification  and validation findings.
We conducted this performance audit at VA headquarters in Washington,  D.C., from November 2008 through October 2009 in accordance with  generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on our  audit objectives.
section_title: Appendix II: Comments from the Department of Veterans Affairs
section_title: Appendix III: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: In addition to the contacts named above, key contributions to this report  were made by Mark T. Bird, Assistant Director; Michael S. LaForge,  Assistant Director; Heather A. Collins; Neil J. Doherty; Rebecca Eyler;  David A. Hong; Jacqueline K. Mai; Yvonne D. Moss; Robert L. Williams, Jr.;  and Leonard E. Zapata.