DHS Financial Management: Better Use of Best Practices Could Help Manage System Modernization Project Risks
Released: 2017-09-26T00:00:00

Letter

Background

Since DHS’s creation in 2003, significant internal control and financial management system deficiencies have hampered its ability to reasonably assure effective financial management and to manage operations. These deficiencies contributed to our decision to designate DHS’s management functions, including financial management, as high risk. To help address these deficiencies, DHS initiated a decentralized approach to upgrade or replace legacy financial management systems and has been evaluating various options for modernizing them, including the use of SSPs. DHS initiated three projects for modernizing the systems of selected DHS components, including its TRIO modernization project. The TRIO project has focused on migrating the financial management systems of Coast Guard, DNDO, and TSA to a modernized solution provided by IBC. DHS’s efforts to effectively assess and manage risks associated with this project are essential to DHS’s realizing its modernization goals.

In 2013, OMB issued a memorandum directing agencies to consider federal SSPs as part of their AAs. Also, in May 2014, Treasury and OMB designated IBC as one of four federal SSPs for financial management to provide core accounting and other services to federal agencies. This designation was based on Treasury and OMB’s evaluation of the four service providers’ ability to assist federal agencies in meeting their accounting and financial management needs, including experience with implementing financial management systems and providing other financial management services to customers, cost of services provided, compliance with financial management and internal control requirements, commitment to shared services, capacity, and long-term growth strategy. FIT’s responsibilities related to the governance and oversight of federal SSPs were subsequently transferred to USSM after USSM was established in October 2015.

TRIO Modernization Project

Because of concerns that its Core Accounting System (CAS) Suite was outdated, inefficient, and did not reliably meet requirements, Coast Guard completed an AA in January 2012 to assist in developing a path forward for modernizing its financial management system. In August 2012, Coast Guard established its CAS Replacement project team to further evaluate two of the alternatives considered in its AA and develop a recommended course of action. In addition, Coast Guard determined that hosting, owning, operating, and managing a financial management system were not among its core competencies. Because TSA and DNDO also relied on CAS as their primary accounting system, they also conducted AAs to identify the best alternative for transitioning to a modernized financial management system solution.

The AAs conducted by the TRIO components during 2012 and 2013 considered the use of federal and commercial SSPs and other options. In addition, Coast Guard completed additional market research including further analysis of commercial SSPs in June 2013. In July 2013, the TRIO components determined that migrating to a federal SSP was the best course of action and subsequently conducted discovery phase efforts with IBC from November 2013 through May 2014 to further explore the functional requirements for procurement, asset, and financial management services. Based on these efforts, in July 2014, the TRIO components recommended that they proceed with implementation of the IBC shared services solution. In August 2014, FIT and OMB concurred with this recommendation, and DHS entered into an interagency agreement (IAA) with IBC for implementation. Figure 1 shows a timeline of these key events.

The IAA for implementation and related performance work statement included a description of the services that IBC is to provide and the roles and responsibilities of DHS, the TRIO components, and IBC. The IAA also required IBC to prepare a detailed project management plan describing how the requirements would be managed and updated and an integrated master schedule (IMS) for identifying tasks to be completed, duration, percentage completed, dependencies, critical path, and milestones.

According to the February 2015 project management plan, DNDO, TSA, and Coast Guard were expected to go-live on the IBC solution in the first quarter of fiscal years 2016, 2017, and 2018, respectively. However, in May 2016, DHS and IBC determined that TSA’s and Coast Guard’s planned implementation dates were not viable because of various challenges impacting the TRIO project and recommended a 1-year delay for their respective implementation dates. Figure 2 summarizes planned and completed key implementation events for the TRIO project as of May 2016.

Best Practices for Conducting Analysis of Alternatives and Managing Risks

GAO, SEI, and other entities have developed and identified best practices to help guide organizations in effectively planning and managing various activities, including acquisitions of major information technology systems. These include GAO’s identified best practices for the AOA process and best practices identified by SEI for risk management.

GAO-identified best practices for AOA process. GAO identified 22 best practices for a reliable, high-quality AOA process that can be applied to a wide range of activities in which an alternative must be selected from a set of possible options, as well as to a broad range of capability areas, projects, and programs. These practices can provide a framework to help ensure that entities consistently and reliably select the project alternative that best meets mission needs. Not conforming to these best practices may lead to an unreliable process, and the entity will lack assurance that the preferred alternative best meets the mission needs. Appendix II provides additional details on GAO’s identified AOA process best practices and how they can be applied to a wide range of activities in which an alternative must be selected from a set of possible options, as well as to a broad range of capability areas, projects, and programs.

SEI’s risk management practices. SEI’s practices for the risk management process area call for the identification of potential problems before they occur so that risk-handling activities can be planned throughout the life of a project to mitigate adverse impacts on achieving objectives. These practices are determining risk sources and categories, defining parameters used to analyze and categorize risks and to control the risk management effort, establishing and maintaining the strategy to be used for risk identifying and documenting risks, evaluating and categorizing each identified risk using defined risk categories and parameters and determining its relative priority, developing a risk mitigation plan in accordance with the risk monitoring the status of each risk periodically and implementing the risk mitigation plan as appropriate.

DHS Did Not Always Follow Best Practices for Analyzing Alternatives for TRIO Components’ Choice of Modernized Financial Management System

Although the TRIO components conducted AAs to identify the preferred alternative for modernizing their financial management systems, their efforts did not always follow best practices. For example, Coast Guard’s and TSA’s AAs supporting their selection of migrating to a federal SSP for modernizing their financial management systems did not fully or substantially meet all four characteristics of a reliable, high-quality AOA process. In addition, we found that DHS guidance did not fully or substantially incorporate five of GAO’s identified best practices for conducting an AOA process. The TRIO components’ AAs included descriptions of the key factors, such as scores for each alternative against the selection criteria used to assess it. Based on these AAs, DHS and the TRIO components selected the federal SSP alternative as their preferred choice and subsequently selected IBC as their federal SSP. However, because Coast Guard’s and TSA’s AAs did not fully or substantially meet all four characteristics of a reliable, high-quality AOA process, they are at increased risk regarding their decision on the solution that represents the best alternative for meeting their mission needs.

DNDO Substantially, and Coast Guard and TSA Partially, Met Best Practices for Conducting AOAs

Based on the extent to which the DHS TRIO components followed the GAO-identified 22 best practices for conducting an AOA process, we found that DNDO’s AA process substantially met the four characteristics of a reliable, high-quality AOA process while the Coast Guard and TSA AA processes both substantially met one and partially met three of these four characteristics. For example, we found that TSA’s AA partially met the “well-documented” characteristic, in part, because risk mitigation strategies, assumptions, and constraints associated with each alternative were not discussed in its AA. In addition, we found that Coast Guard’s AA partially met the “credible” characteristic, in part, because there was no indication that it contained sensitivity analyses, an evaluation of the impact of changing assumptions on its overall costs or benefits analyses. Our overall assessment is summarized in table 1. Appendix III provides additional details on our assessment of the TRIO components’ AAs for each of the GAO-identified 22 AOA best practices.

Further, in comparing DHS AOA and AA guidance to the GAO-identified 22 AOA process best practices, we found that although DHS’s guidance for conducting both AOAs and AAs fully or substantially incorporated 17 of the identified best practices, the guidance did not fully or substantially incorporate 5 of these practices. For example, although the guidance addressed risk management in general terms, it did not detail the need to document risk mitigation strategies for each alternative. Not documenting the risks and related mitigation strategies for each alternative prevents decision makers from performing a meaningful trade-off analysis necessary to choose a recommended alternative. In addition, while DHS guidance describes the need for an AA or AOA review, it describes reviews conducted within the organizational chain of command and does not address the need for an independent review—one of the most reliable means to validate an AOA process. Further, although the guidance noted that weights for selection criteria may become more subjective when they cannot be derived analytically, additional guidance on weighting selection criteria was limited. Our overall assessment is summarized in table 2.

Because of these limitations in guidance, and because Coast Guard and TSA did not fully adhere to the GAO-identified best practices, Coast Guard’s and TSA’s AAs did not fully or substantially reflect all four characteristics of a reliable, high-quality AOA process. As a result, Coast Guard and TSA increased their risk of selecting a solution that may not represent the best alternative for meeting their mission needs.

TRIO Components Used Key Factors, Metrics, and Processes to Analyze Alternatives and Related Results

Documentation supporting TRIO components’ AA efforts included descriptions of the key factors, metrics, and processes involved in conducting their analyses, including the (1) alternatives considered, (2) market research conducted, (3) three alternatives evaluated, (4) selection criteria used by each and how the criteria were weighted, (5) scores for each alternative against the selection criteria, and (6) alternatives that scored the best under the AOA evaluation.

The TRIO components conducted market research to develop reasonable alternative solutions for consideration. For example, through its market research, TSA identified OMB-designated federal SSPs and commercial entities as potential alternatives for hosting and implementing a modernized and integrated financial management system. According to its AA, TSA was able to gain an understanding of the offerings, capabilities, and related costs associated with these alternatives through reviews of documentation and interviews.

After developing a diverse range of financial system modernization alternatives for consideration, each of the TRIO components assessed them for viability using various factors—such as measures of effectiveness, cost, risk, and value—and identified the three top-rated alternatives for further evaluation. For example, Coast Guard identified nine alternatives for consideration and analyzed, scored, and ranked them to determine its top three alternatives for further analysis: incrementally improve the current CAS Suite and remove certain outdated components, host the financial management system internally using software and tools already owned, and use an SSP to host the financial management system.

Each component identified its three alternatives for further evaluation and used defined selection criteria to rate them. For example, DNDO’s selection criteria included four categories of operational effectiveness that were weighted according to their level of importance. Based on their evaluations, each component identified the best alternative for its respective financial management system needs.

According to Coast Guard’s November 2012 decision memorandum, Coast Guard further narrowed the alternatives it focused on to (1) using an SSP to host its financial management system and (2) hosting the system internally using already-owned software and tools, and it also gathered rough order of magnitude cost estimates for both alternatives. Based on its evaluation, Coast Guard determined that the two alternatives were comparable. According to this memorandum, Coast Guard further determined that owning, hosting, operating, and managing a financial management system were not among its core competencies. Based on this determination, OMB direction to agencies to use (with limited exceptions) shared services, and other factors, Coast Guard decided that migrating to an SSP was the best alternative. TSA found in its February 2013 analysis that the differences between federal and commercial SSP alternatives were not significant and, as a result, recommended that a competitive procurement be conducted to better evaluate each alternative. However, DHS officials told us that TSA subsequently determined that a competitive procurement was not warranted and chose to migrate to a federal SSP. This determination was based on additional OMB guidance issued in March 2013 directing agencies to consider federal SSPs as part of their AAs and stating that commercial SSPs are an appropriate solution and would be funded by OMB only in instances in which the agency’s business case demonstrates that a commercial SSP can provide a better value for the federal government. In addition, DNDO determined that migrating to a federal SSP was its best alternative in May 2013.

Because its preliminary research focused primarily on the federal SSP marketplace, Coast Guard conducted additional market research to include a more robust analysis of commercial SSPs. Coast Guard’s June 2013 market research report described the results of this effort, including its evaluation of responses from 11 commercial SSPs. Coast Guard reported that none of the commercial SSPs that responded could meet all 44 specific financial management system requirements and the extent to which they could meet them varied significantly. Based on these results, Coast Guard determined that there was a lack of maturity in the commercial SSP market for federal financial management. According to the report, this overall assessment was based on various considerations of information provided by commercial SSP respondents, including the wide variety of proposed configurations, solutions, prices, and implementation schedules, the lack of federal experience and service for agency-wide capabilities, and insufficient length of service to establish positive trends in audit performance; the lack of similar offerings that implied a lack of strong competition between comparable products that would exert downward pressure on cost; and the lack of like product offerings, which increases the likelihood of higher switching costs in the case of poor performance because of increased difficulty in moving from one “turnkey” service to another.

In July 2013, the TRIO components and DHS selected the federal SSP alternative as their preferred choice and subsequently selected IBC as their federal SSP. DHS officials told us that IBC was selected based on (1) DHS’s reliance on OMB and Treasury’s designation of IBC as a federal SSP, (2) OMB guidance to consider the use of federal SSPs, and (3) a review of the availability of the four federal SSPs indicating that IBC was the only one available to meet the requirements and implementation schedule at that time. In August 2013, DHS notified OMB that the TRIO components had performed extensive market research and finalized their respective AAs and independently concluded that migrating to a federal SSP was in the best interests of the government. Also, in August 2013, FIT notified OMB regarding the TRIO components’ AA efforts and that the TRIO components would proceed to the discovery phase with IBC. According to FIT’s notification memorandum to OMB, the TRIO components’ AAs demonstrated that migrating to a federal SSP was the best value to the federal government and that the components identified IBC as a suitable partner based on the results of their market research into federal SSPs.

DHS Met Three and Partially Met Four Best Practices for Managing the Risks of Using IBC for the TRIO Project

Risk management best practices call for the identification of potential problems before they occur so that risk-handling activities can be planned throughout the life of the project to mitigate adverse impacts on achieving objectives. These best practices involve (1) preparing for risk management, (2) identifying and analyzing risks, and (3) mitigating identified risks. Preparing for risk management involves determining risk sources and categories and developing risk mitigation techniques. Identifying and analyzing risks includes determining those that are associated with cost, schedule, and performance and evaluating identified risks using defined risk parameters. Mitigating risks includes determining the levels and thresholds at which a risk becomes unacceptable and triggers the execution of a risk mitigation plan or contingency plan; determining the costs and benefits of implementing the risk mitigation plan for each risk; monitoring risk status; and providing a method for tracking open risk-handling action items to closure.

Based on our evaluation, we found that DHS processes generally reflected three of seven specific risk management best practices and partially reflected the remaining four practices. Table 3 summarizes the extent to which DHS followed these seven best practices for managing TRIO project risks. Additional details on DHS and TRIO component efforts to address these practices are summarized following this table.

Prepare for risk management. Key aspects of processes established by DHS and TRIO components related to the three best practices associated with preparing for risk management:

Determine risk sources and categories. This practice calls for a basis for systematically examining circumstances that affect the ability of the project to meet its objective and a mechanism for collecting and organizing risks. DHS and the TRIO components established processes that met this best practice. For example, DHS reviewed the integrated master schedule that IBC prepared to identify sources of risk and defined risk categories in TRIO project policies.

Define risk parameters. Risk parameters are used to provide common and consistent criteria for comparing risks to be managed. The best practice includes defining criteria for evaluating and quantifying risk likelihood and severity levels and defining thresholds for each risk category to determine whether risk is acceptable or unacceptable and to trigger management action. DHS partially met this best practice. DHS’s risk management program defined rating scales to provide consistent criteria for evaluating and quantifying risk likelihood and severity levels. However, DHS’s Risk Management Planning Handbook and related template for developing risk management plans for projects did not address the need for thresholds relevant to each category of risk to facilitate review of performance metrics in order to determine when risks become unacceptable or to invoke selected risk-handling options when monitored risks exceed defined thresholds.

Establish a risk management strategy. A risk management strategy addresses specific actions and the management approach used to apply and control the risk management program, including identifying sources of risk, the scheme used to categorize risks, and parameters used to evaluate and control risks for effective handling. DHS met this best practice. DHS and IBC established risk management policies and plans for the TRIO project based on DHS acquisition guidance, which provided a framework for a risk management program. Collectively, these policies and plans constitute a risk management strategy. DHS and IBC have periodically updated these documents to maintain the scope of the risk management effort; the methods and tools to be used for risk identification, risk analysis, risk mitigation, risk monitoring, and communication; the prioritization of risks; and the allocation of resources for risk mitigation.

Identify and analyze risks. Key aspects of processes established by DHS and the TRIO components related to the two best practices associated with identifying and analyzing risks: Identify risks. Risk identification should be an organized, thorough process to seek out probable or realistic risks to achieving objectives. This practice recognizes that risks should be identified and described understandably before they can be analyzed and managed properly. Using categories and parameters developed in the risk management strategy and identified sources of risk guides the identification of risks associated with cost, schedule, and performance. To identify risks, best practice elements include reviewing the work breakdown structure (WBS) and project plan to help ensure that all aspects of the work have been considered. Best practices for documenting risks include documenting the context, conditions, and potential consequences of each risk and identifying the relevant stakeholders associated with each risk.

DHS partially met this best practice. DHS’s July 2016 risk register contained a wide range of risks associated with defined risk categories. It also reflected DHS’s review of the TRIO project’s integrated master schedule that IBC prepared based on the WBS and work plans that IBC also developed. The risk register documented the context, conditions, potential consequences, and relevant stakeholders associated with each risk. However, DHS’s documented risk management processes did not identify all significant risks or reflect its efforts to revisit risks that had previously been closed. For example, DHS officials told us that IBC was unable to provide sufficient, reliable cost and schedule information for project monitoring; however, a risk reflecting these concerns was not included on its July 2016 risk register. Further, the risk register included certain closed risks related to the need for a governance structure and strategy for ensuring that IBC met performance, cost, and schedule objectives. Although DHS had ongoing concerns about its ability to ensure that IBC met these objectives, the risk register did not reflect efforts to revisit these risks to determine whether their status needed revision or if other risks should be included on the risk register to address its accountability concerns. In addition, DHS did not always take timely action to document its consideration of risks identified by its independent verification and validation (IV&V) contractor for potential inclusion on its risk register. For example, the IV&V contractor identified a risk related to inefficiencies in DHS’s document review process in June 2015 that was not included on DHS’s risk register until February 2016. DHS officials indicated that a crosswalk between the DHS risk register and IV&V contractor risk management observations was performed weekly; however, results of these weekly reviews were not documented.

Evaluate, categorize, and prioritize risks. Risk assessment uses defined categories and parameters to determine the priority of each risk to assist in determining when appropriate management attention is required. Best practices for analyzing risks include categorizing risks according to defined risk categories, evaluating identified risks using defined risk parameters, and prioritizing risks for mitigation. DHS’s processes met this practice. For example, the documented risk management program included application of defined risk categories and parameters for all identified risks, providing a means for reviewing risks and determining the likelihood and severity of risks being realized. The TRIO project’s Joint Risk Management Integrated Project Team provided consistency to the application of parameters by reviewing risk assessments when risks were first identified. By determining exposure ratings for each identified risk, DHS prioritized risks for monitoring and allocation of resources for risk mitigation.

Mitigate risks. Key aspects of processes established by DHS and the TRIO components related to the two best practices associated with mitigating risks:

Develop risk mitigation plans. Risk mitigation plans are developed in accordance with the risk management strategy and include a recommended course of action for each critical risk. The risk mitigation plan for a given risk includes techniques and methods used to avoid, reduce, and control the probability of risk occurrence; the extent of damage incurred should the risk occur; or both. Elements of this practice include determining the levels and thresholds that define when a risk becomes unacceptable and triggers the execution of a risk mitigation plan or contingency plan, identifying the person or group responsible for addressing each risk, determining the costs and benefits of implementing the risk mitigation plan for each risk, developing an overall risk mitigation plan for the work to orchestrate the implementation of individual risk mitigation plans, and developing contingency plans for selected critical risks in the event impacts associated with the risks are realized.

DHS partially met this best practice. DHS’s risk management program documentation reflected the development of risk response plans for most risks, including all those determined to be of medium and high exposure level. DHS identified those responsible for addressing each risk. However, DHS and IBC did not always develop sufficiently detailed risk mitigation plans including specific risk-handling action items, determination of the costs and benefits of implementing the risk mitigation plan for each risk, and developing contingency plans for selected critical risks in the event that their impacts are realized. For example, a risk associated with IBC’s capacity and experience for migrating large agencies the size of Coast Guard and TSA was identified in July 2014. Although DHS developed plans to help mitigate this risk, a contingency plan was not developed prior to realizing the adverse impact of not implementing Coast Guard and TSA on IBC’s modernized solution. Rather, a contingency plan working group (CPWG) to address this and other concerns was established in January 2017, over 2 years after the risk was initially identified. Further, thresholds were not used within the risk management program to define when a risk becomes unacceptable, triggering the execution of a risk mitigation plan or contingency plan.

Implement risk mitigation plans. Risk mitigation plans are implemented to facilitate a proactive program to regularly monitor risks and the status and results of risk-handling actions to effectively control and manage risks during the work effort. Best practice elements include revisiting and reevaluating risk status at regular intervals to support the discovery of new risks or new risk-handling options that can require reassessment of risks and re-planning of risk mitigation efforts. Elements also include providing a method for tracking open risk-handling action items to closure, establishing a schedule or period of performance for each risk-handling activity, invoking selected risk-handling options when monitored risks exceed defined thresholds, and providing a continued commitment of resources for each risk mitigation plan.

DHS partially met this best practice. Risk monitoring of the TRIO project consisted of reviews performed by DHS and TRIO component officials responsible for risk management and oversight functions. These reviews considered significant risks, risks approaching realization events, and the effect of management intervention on the resolution of risks. These reviews also relied, in part, on data contained in DHS’s risk register, which represents the official repository of TRIO project risks and information on the status of risks and related risk mitigation efforts.

However, other aspects of DHS’s efforts to implement risk mitigation plans did not fully adhere to certain elements associated with this best practice. For example, we identified certain issues that raised questions concerning the accuracy of data contained in the risk register, such as (1) the lack of clear markings indicating when the accuracy of data on each risk was last confirmed, including risk records that had not been modified in the previous 3 months, and (2) certain risks for which the estimated risk impact date had already occurred but its status risk according to DHS’s risk register did not reflect that it had been realized and become an issue. In addition, DHS officials stated that IBC did not provide sufficiently detailed, reliable cost and schedule information that could have been used to monitor TRIO project risks more effectively. DHS’s ability to monitor cost, schedule, and other performance metrics was also limited because of the lack of thresholds for management involvement, as noted above. DHS’s implementation of risk monitoring plans was further limited by other issues, including (1) a period of performance for each risk-handling activity, which includes a start date and anticipated completion date to control and monitor risk mitigation efforts, was not always established and (2) an inability to fully track open risk-handling action items to closure existed because of the lack of sufficient detail on specific risk-handling activities in the DHS risk register.

According to DHS officials, DHS relied heavily on IBC to manage risks associated with the TRIO project and, in particular, those for which IBC was assigned as the risk owner. They also acknowledged DHS’s responsibility for overseeing IBC’s TRIO project risk management efforts and described various actions taken to address growing concerns regarding IBC’s efforts. For example, DHS created the Joint Risk Management Integrated Project Team, in part, to provide a forum in which IBC could obtain assistance in developing risk responses and discuss DHS’s risk mitigation concerns. Further, to help reduce exposure of underlying risks, DHS offered assistance to IBC’s project management functions, such as developing the integrated master schedule and performing quality control checks on project deliverables. Despite these efforts, DHS officials stated that challenges associated with the IAA structure and terms of the performance work statement with IBC on the TRIO project limited DHS’s visibility into IBC’s overall cost, schedule, and performance controls and ability to oversee IBC’s risk management efforts. For example, they stated that the performance work statement did not specify the level of reporting to be provided by IBC on cost, schedule, and performance in sufficient detail to effectively monitor progress on achieving key project objectives.

Further, the limitations to managing risks related to the best practices we assessed as partially met were largely attributable to limitations in DHS and TRIO project guidance and policies. For example, DHS’s Risk Management Planning Handbook and related template for developing risk management plans for projects does not address the need to define thresholds to facilitate review of performance metrics to determine when risks become unacceptable. Also, TRIO project policies did not address the need to periodically revisit consideration of risk sources other than IMS-related milestones, specify periods of performance for specific risk- handling activities, or define an interval for updating and certifying risk statuses. In addition, DHS guidance and TRIO project policies did not describe the need to consider and document risks specifically related to the lack of sufficient, reliable cost and schedule information to properly manage and oversee the project or for timely disposition of risks that its IV&V contractor identified. Further, TRIO project risk management policies and management tools used to implement them address best practice elements such as determination of the costs and benefits of implementing risk mitigation plans, developing contingency plans, and developing specific risk-handling action items. However, these policies do not require, and the risk register was not designed to specifically capture, these elements in documented risk mitigation plans. By not adopting important elements of risk management best practices into project guidance, DHS and the TRIO components increase the risk that potential problems would not be identified before they occur and that activities to mitigate adverse impacts would not be effectively planned and initiated.

Key Factors and Challenges Impacting the TRIO Project and DHS’s Path Forward

Although DHS has taken various actions to manage the risks of using IBC for the TRIO project, including some that were consistent with best practices, the TRIO project has experienced challenges raising concerns regarding the extent to which its objectives will be achieved. In connection with these challenges, the TRIO components notified DHS during April 2016 through January 2017 that certain baseline cost and schedule objectives had not been, or were projected to not be, achieved as planned. According to these notifications and DHS officials we interviewed, several key factors and challenges significantly impacted DHS’s and IBC’s ability to achieve TRIO project objectives as intended. In addition, IBC, FIT, and USSM officials identified similar issues impacting the TRIO project. In connection with these challenges, DHS and IBC began contingency planning efforts in January 2017 to identify and assess viable options for improving program performance and addressing key TRIO project priorities. Plans for DHS’s path forward on the TRIO project, as of May 2017, involve significant changes, such as transitioning away from using IBC and a 2-year delay in completing Coast Guard and TSA’s migration to a modernized solution.

Key Factors and Challenges Impacting the TRIO Project

We grouped the key factors and challenges impacting the TRIO project that DHS, IBC, FIT, and USSM officials and OMB staff identified into five broad categories: (1) project resources, (2) project schedule, (3) complex requirements, (4) project costs, and (5) project management and communications. The key factors and challenges related to each category are summarized below.

Project resources: Concerns about IBC’s experience and its capacity to handle a modernization project involving agencies the size of Coast Guard and TSA were identified as significant risks in July 2014, resulting from discovery phase efforts completed prior to DHS and IBC’s entering the implementation phase in August 2014. According to DHS officials, status reports, and other documentation, key TRIO project challenges related to resources included concerns that (1) IBC encountered federal employee hiring challenges and was unable to ramp up and deploy the resources necessary to meet required deliverables, and (2) IBC experienced significant turnover of key stakeholders which adversely impacted its ability to achieve TRIO project objectives.

In connection with DHS’s decision to use IBC for the TRIO project, DHS officials told us that DHS relied heavily on OMB and Treasury’s designation of IBC as a federal SSP and their related assessment of IBC’s capacity and experience. DHS officials also told us that DHS relied on FIT’s federal agency migration evaluation model during discovery phase efforts that focused on assessing the functionality of the software rather than assessing IBC’s (1) capacity, experience, and capability; (2) ability to address more complex software configurations and interfaces associated with large agencies; and (3) cost, schedule, and performance metrics. DHS officials stated that issues related to IBC’s capacity and experience represented the most significant challenge impacting the TRIO project.

IBC officials acknowledged that IBC was unable to ramp up its resources until after the project had begun and that the IBC project team experienced significant turnover in key leadership and TRIO project positions over the course of the project. IBC officials also acknowledged that during its early efforts on the TRIO project, assigned IBC staff lacked the experience and expertise necessary for managing large-scale projects and, as a result, many of the risks initially identified were not effectively addressed. FIT and USSM officials and OMB staff also acknowledged that resource challenges significantly impacted the TRIO project. A FIT official acknowledged that assessing software functionality, rather than implementation, was emphasized during the discovery process. Although DHS relied on OMB and Treasury’s designation of IBC as a federal SSP, this FIT official also told us that because agencies’ specific needs can vary significantly, agencies are responsible for conducting sufficient due diligence to assess a federal SSP’s ability to meet their requirements.

Project schedule: DHS, IBC, FIT, and USSM officials acknowledged that migrating the TRIO components to IBC within original time frames was a significant challenge given the overall magnitude and complexity of the TRIO project. According to DHS officials and TRIO project documentation, DHS identified delays in completing various tasks and milestones including providing design phase technical documentation and design processing proposed change requests; meeting proposed baseline schedules for implementing Coast Guard and TSA on the modernized IBC solution; and achieving initial operating capability requirements and stabilizing the production environment after DNDO’s migration to IBC because of various issues related to reporting, invoice payment processing, contract management processes, and resolving help desk tickets in a timely manner.

DHS officials also stated that IBC did not consistently update the IMS to ensure that it accurately reflected all required tasks, the completion status, and the resources required to complete them. Concerns related to meeting milestones and updating the IMS were discussed during periodic status update meetings that included DHS, IBC, OMB, FIT, and USSM officials. IBC and DHS officials acknowledged that processes for communicating and resolving issues were not always efficient and contributed to schedule delays. In addition, in November 2016, USSM noted several concerns based on its review of a draft IMS supporting TSA’s re-planning efforts to go-live in October 2017. USSM’s concerns included an incomplete project scope and schedule and need for additional discovery to determine cost and level of effort, an extremely aggressive schedule with very limited contingencies for the lack of interim checkpoints or oversight on tasks exceeding 30 days, the need for a resource-loaded IMS that incorporates an appropriate level of detail, and the need for an expedited program governance strategy and escalation path that DHS and IBC leadership could use to make program decisions within the time allotted on the schedule.

Complex requirements: DHS, IBC, FIT, and USSM officials acknowledged the overall complexity of the TRIO project and that the lack of a detailed understanding of the components’ requirements earlier in the project impacted IBC’s and DHS’s ability to satisfy the requirements as planned. For example, USSM and FIT officials told us that under the shared services model, the approach for onboarding new customers usually involves migrating to a proven configuration of a solution that is already being used by the provider’s existing customers. However, rather than taking this approach, DHS and IBC agreed to implement a more recent version of Oracle Federal Financial software (version 12.2) with integrated contract life cycle and project modules. Under this approach, IBC’s plans included migrating other existing customers to this upgraded environment. USSM officials told us that migrating TRIO components to a new solution that required configuring new software and related applications and developing related interfaces introduced additional complexities that contributed to issues on the TRIO project. According to a FIT official, the functionality of this more recent version of software is very different than that of the version IBC’s existing customers used. This official stated that IBC did not have the needed government personnel with knowledge and experience associated with this new software, a condition that likely contributed to the challenges experienced on the TRIO project. IBC officials acknowledged that IBC’s lack of familiarity with Oracle 12.2 increased the complexity of the TRIO project.

In addition, DHS and IBC perspectives on the need for changes differed because of the lack of clarity regarding TRIO project requirements. DHS officials told us that many change requests on the TRIO project reflected the need for required functionality based on previously stated requirements. They also told us that they did not consider DNDO-related requirements to be overly complex when compared to those associated with IBC’s similarly sized customers. However, DHS officials stated that as of June 2017, IBC has not yet met DNDO’s needs to deliver a functioning travel system interface and other requirements. According to IBC officials, TRIO project change requests to address components’ requirements were extensive and included significant customizations to meet unique requirements that were not aligned with the federal shared service model. IBC officials noted additional challenges in addressing TRIO project requirements related to DHS’s efforts to address certain organizational change management and business process reengineering responsibilities. According to IBC officials, in some instances, the TRIO components provided conflicting requirements related to the same process that would have been more consistent had DHS completed more of its business process reengineering efforts prior to providing them to IBC.

Project costs: According to the July 2014 discovery report, proposed implementation costs for the TRIO project totaled $89.9 million. However, according to DHS officials and TRIO project documentation, estimated costs significantly increased because of schedule delays, unanticipated complexities, and other challenges. In January 2017, DHS prepared a summary of estimated TRIO project implementation costs associated with its IAA with IBC. According to this summary, estimated IBC-related TRIO project implementation costs through fiscal year 2017 increased by approximately $42.8 million (54 percent) from the $79.2 million provided in the original August 2014 IAA with IBC as a result of modifications required, in part, to address challenges impacting the project. DHS officials also expressed concerns regarding increases in estimated operations and maintenance costs for the IBC solution. For example, according to a December 2016 memorandum to DHS on action items associated with failing to meet the baseline schedule date for initial operational capability, DNDO stated that IBC’s updated projected costs of operations and maintenance of its system were unaffordable. In connection with these costs, DHS officials also stated that IBC determined that separate, rather than shared, help desk resources were required to support the TRIO project because it was significantly different from the solution that IBC’s existing customers used. As a result, the officials indicated that these costs were more than originally expected. However, IBC officials told us that a portion of the increase in help desk- related costs was also due to DNDO employees not using the system properly because they were not sufficiently trained on it before it was implemented. In addition, challenges impacting the TRIO project have contributed to significant changes in the path forward on the project; as a result, the extent to which overall TRIO project modernization costs will be impacted going forward has not yet been determined.

Project management and communication: According to DHS officials, various program management-related challenges impacted the TRIO project. For example, they expressed concerns regarding the effectiveness of IBC’s project management efforts including cost, schedule, and change management as well as IBC’s allocation of resources and slow decision-making process. They also stated that DHS provided significant time and resources to make up for fundamental project management activities that were under IBC’s control and not performed. In addition, DHS officials identified limitations associated with (1) poorly defined service level agreements and program performance metrics, (2) poor quality control plan, and (3) the lack of mechanisms for measuring delivery and addressing concerns regarding IBC’s performance. DHS officials told us that although various mechanisms can be used to hold commercial vendors accountable—such as cure notices, quality assurance surveillance plans, and incentives or disincentives to monitor performance—few mechanisms are available to hold federal agency service providers accountable for performance concerns.

DHS officials also acknowledged challenges in their project management and communication efforts and identified lessons learned to help improve future efforts, including the need to establish a performance-based contract to determine objective and enforceable activity level metrics; be more prepared for organizational changes; improve vendor, project, and schedule management efforts; better understand SSP resource plans and monitor SSP efforts to help ensure that sufficient resources are secured timely; and centralize program management for financial system modernization functions, rather than continuing with the structure used on the TRIO project—for example, the TRIO project’s program management structure consisted of program management offices at the component level performing cost, schedule, and technical monitoring activities with DHS headquarters’ involvement focused on governance and oversight, resulting in duplicate efforts across components.

IBC officials acknowledged challenges concerning IBC’s lack of sufficient resources and turnover, as described above. However, they told us that DHS’s approach to project management often resulted in duplicative meetings and a lengthy decision-making process involving several officials and multiple review and approval processes. According to USSM officials, the TRIO project team focused an unbalanced portion of its efforts on the delivery of technology at the expense of organizational change management, communication management, and other project management areas. For example, the failure to incorporate lessons learned from DNDO’s deployment adversely affected subsequent TRIO project implementation efforts, as change management activities did not address previously encountered risks. An OMB staff member concurred with the lessons learned that DHS identified, including those indicating the need for stronger project management. While the project is ongoing, the OMB staff member noted the importance of DHS having well-defined requirements for the project and better coordination to achieve the desired outcomes.

Significant TRIO Project Changes Resulting from Challenges and Steps Implemented for the Path Forward

In connection with TRIO project challenges, DHS officials told us that IBC notified DHS in April 2016 that it would not be able to meet the planned October 2016 implementation date for TSA. In response, DHS and IBC established the TSA Replan Tiger Team to perform a detailed assessment of potential courses of action. According to DHS officials, DHS and IBC subsequently took various actions to help address these and other challenges impacting the TRIO project, as summarized below.

May 2016: IBC requested additional funding for fiscal year 2016 for 14 additional IBC and contractor personnel to strengthen program coordination and management support. According to DHS officials, DHS provided this requested funding along with additional funding to establish a business integration office to help strengthen cross organizational communication. DHS determined that plans for migrating TSA and Coast Guard to IBC during the first quarter of fiscal years 2017 and 2018, respectively, were not viable. As a result, their planned migrations were each extended an additional year.

June 2016: DHS and IBC developed a comprehensive remediation plan to track progress on efforts to resolve numerous issues associated with DNDO’s production environment that continued to hamper its stability since going live in November 2015. According to DHS officials, these issues related to invoice payment and interest accruals, contract life cycle management, reporting, and other activities and have required numerous work-arounds to execute business processes.

August to October 2016: DHS, Coast Guard, and IBC determined that a similar replanning effort was needed for Coast Guard’s successful migration to IBC. According to DHS officials, IBC indicated that it was unable to simultaneously provide DNDO production and TSA implementation support while also addressing the complexities related to Coast Guard. DHS officials told us that another Tiger Team established to address Coast Guard issues failed to complete the scope of its charter, and as a result, Coast Guard was forced to assume a minimum of a 2- year delay (rather than the 1-year delay previously determined in May 2016) and that this significantly increased program costs. They further stated that some of the team’s deliverables have not been initiated or remain outstanding as of June 2017.

December 2016: IBC communicated to DHS that it cannot support the discovery phase with DHS’s CUBE modernization project. In addition, DHS approved the establishment of a Joint Program Management Office to serve as the overarching program management for DHS financial systems modernization projects. According to DHS officials, using a department-wide approach will enable DHS to more effectively leverage the resources and expertise across all modernization projects.

January 2017: IBC communicated to DHS that it cannot support Coast Guard implementation in October 2018, and DHS and IBC established a joint CPWG to assess viable options for improving program performance and addressing stakeholder concerns and key TRIO project priorities.

February 2017: DHS and IBC issued a joint memorandum to provide an update on contingency planning discussions. DHS and IBC shared commitments and determinations included (1) stabilizing the DNDO production environment and executing TSA implementation activities, (2) delivering the best value for the government and ensuring mutual success to the greatest extent possible, (3) preserving and protecting the current investment, and (4) making TSA implementation the first priority. In addition, DHS and IBC presented two options as representing the best opportunities for success in improving program performance and addressing stakeholder concerns: (1) continue with the status quo plan for Coast Guard implementation in October 2019, with significant improvements to program management and overall support capability and capacity, or (2) platform replacement. Platform replacement was presented as the preferred path toward meeting the needs of both DHS and IBC. Under this option, DHS and IBC would proceed with TSA implementation and work toward an orderly transition of TRIO components to an alternate service provider, hosting location, or both.

March 2017: According to DHS officials, DHS, IBC, and USSM officials met to review certain critical success criteria for TSA’s implementation. Based on these discussions, it was determined that TSA would not go live with IBC in fiscal year 2018 given the high-risk schedule and critical criteria involved and the Coast Guard implementation would also be delayed accordingly. Further, TSA release 3.0 would be delivered in October 2017 or as soon as possible thereafter. In addition, the CPWG would continue working to identify an alternative path forward, and DHS and IBC would identify and evaluate critical transition activities and timelines.

April 2017: The CPWG recommended moving away from IBC to a commercial service provider leveraging the cloud as the best course of action to complete TRIO project implementation and as the most fiscally responsible approach from a long-term sustainment and cost perspective. The CPWG’s recommendation was based on its analysis of six options and proposed a transition timeline, including key activities, as shown in figure 3.

May 2017: During its May 3, 2017 briefing of the Financial Systems Modernization Executive Steering Committee, DHS indicated that two of the options that the CPWG considered were no longer viable, including the CPWG’s recommendation to transition to a commercial cloud service provider because the software was not yet cloud-ready. DHS ranked the remaining four options using 13 OMB risk factors as selection criteria and determined that migrating the solution to a DHS data center represented the best option going forward. In addition, DHS decided to move forward with discovery efforts related to this option. According to its briefing presentation and DHS officials, the notional timeline of planned key events for the TRIO project included various items, as shown in figure 4.

DHS officials indicated that DHS expects to present the findings and recommendations resulting from discovery efforts associated with this new path forward to USSM and OMB for concurrence. As of August 2017, results of this effort were under review by DHS leadership.

Conclusions

The TRIO project represents a key element of DHS’s efforts to address long-standing deficiencies in its financial management systems and further improve financial management. Following best practices to manage risks effectively can help provide increased assurance that large, complex projects—such as the TRIO project—will achieve planned objectives. DNDO’s AA process substantially met the four characteristics of a reliable, high-quality AOA process. However, Coast Guard’s and TSA’s AAs substantially met one and partially met three of these four characteristics. Further, DHS did not always follow best practices for managing the risks of using IBC for the TRIO project. As a result, TRIO components faced an increased risk that the solution they chose would not represent the best alternative for meeting their mission needs and that the risks impacting the TRIO project would not be effectively managed to mitigate adverse impacts. In addition, significant challenges have impacted the TRIO project, raising concerns about the extent to which objectives will be achieved as planned. Plans for DHS’s path forward on the TRIO project, as of May 2017, involve significant changes, such as transitioning away from IBC and a 2-year delay in completing Coast Guard’s and TSA’s migration to a modernized solution. Without greater adherence to best practices for analyzing alternatives and managing project risks, DHS continues to face increased risk that its financial management system modernization project will not provide reasonable assurance of achieving its mission objectives.

Recommendations for Executive Action

We are making the following two recommendations to DHS: The DHS Under Secretary for Management should develop and implement effective processes and improve guidance to reasonably assure that future AAs fully follow AOA process best practices and reflect the four characteristics of a reliable, high-quality AOA process. (Recommendation 1)

The DHS Under Secretary for Management should improve the Risk Management Planning Handbook and other relevant guidance for managing risks associated with financial management system modernization projects to fully incorporate risk management best practices, including defining thresholds to facilitate review of performance metrics to determine when risks become unacceptable; identifying and analyzing risks to include periodically reconsidering risk sources, documenting risks specifically related to the lack of sufficient, reliable cost and schedule information needed to help properly manage and oversee the project, and timely disposition of IV&V contractor-identified risks; developing risk mitigation plans with specific risk-handling activities, the costs and benefits of implementing them, and contingency plans for selected critical risks; and implementing risk mitigation plans to include establishing periods of performance for risk-handling activities and defining time intervals for updating and certifying the accuracy and completeness of information on risks in DHS’s risk register. (Recommendation 2)

Agency Comments and Our Evaluation

We provided a draft of this product to DHS and the Department of the Interior for comment. In its comments, reprinted in appendix IV, DHS concurred with our recommendations and provided details on its implementation of the recommendations as discussed below. In addition, DHS provided technical comments, which we incorporated as appropriate. The Department of the Interior only provided technical comments, which we incorporated as appropriate.

DHS stated that it remains committed to its financial system modernization program. Specifically, regarding our first recommendation to develop and implement effective processes and improve guidance to reasonably assure that future AAs fully follow AOA process best practices and reflect the four characteristics of a reliable, high-quality AOA process, DHS stated that it agrees that effective processes and guidance are necessary to assure best practices. DHS also stated that it is important to note that the GAO-identified best practices were published more than 2 years after the TRIO components’ AAs were completed. While this is the case, as discussed in our report, these best practices are based on long- standing, fundamental tenets of sound decision making and economic analysis and were identified by compiling and reviewing commonly mentioned AOA policies and guidance that are known to and have been used by government and private sector entities.

DHS also stated that it has already implemented this recommendation through its issuance of guidance and instructions in 2016 and that a copy of this additional guidance and instructions was provided to GAO. However, the documentation provided by DHS does not fully address our recommendation. As part of our recommendation follow-up process, we will coordinate with DHS to obtain additional information on its efforts to address our recommendation.

With regard to our second recommendation to improve the Risk Management Planning Handbook and other relevant guidance, DHS stated that it concurred and agreed that the Risk Management Planning Handbook required updating to fully incorporate risk management best practices. In addition, DHS described actions it will take, and has taken, to revise and publish an updated handbook.

We are sending copies of this report to the appropriate congressional committees, the Acting Secretary of Homeland Security, the DHS Under Secretary for Management, the Acting DHS Chief Financial Officer, the Secretary of the Interior, and the Director of the Interior Business Center. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact me at (202) 512-9869 or khana@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix V.

Appendix I: Scope and Methodology

To determine the extent to which the Department of Homeland Security (DHS) followed best practices in analyzing the alternatives used in choosing the preferred alternative for modernizing TRIO components’ financial management systems, we reviewed information that the TRIO components provided as part of their alternatives analysis (AA) process, referred to as the AA body of work, which includes the AA and other supporting documentation that is not specifically included in the AA. In addition, we discussed the DHS AA process with the TRIO components and DHS officials. We evaluated each TRIO component’s AA body of work and assessed this information against the GAO-identified 22 analysis of alternatives (AOA) process best practices. We then scored each AA against those best practices. In appendix II, these GAO- identified best practices are described in detail.

Our evaluation comprised the following steps: (1) two GAO analysts separately examined the AA information received for each component, providing a score for each of 18 best practices; (2) a third GAO analyst resolved any differences between the two analysts’ initial scoring; and (3) a GAO specialist on AOA best practices, independent of the audit team, reviewed the team’s AA documentation, scores, and analyses for consistency. The GAO specialist also assessed the four best practices related to cost estimating.

We used the average scores for each best practice to determine an overall score for four summary characteristics—well-documented, comprehensive, unbiased, and credible—of a reliable, high-quality AOA process at each TRIO component. Next, we shared our preliminary analysis with the TRIO components and DHS, and requested their technical comments and any additional information for our further consideration. For those characteristics of the AA process that received a score of partially met or below, we met with TRIO component and DHS officials to discuss potential reasons that an AA did not always conform to best practices. Finally, using the same methodology and scoring process explained above, we performed a final assessment based on our preliminary analysis and the comments and additional information received. The best practices were not used to determine whether DHS made the correct decision in selecting Department of the Interior’s Interior Business Center (IBC) to implement the financial management systems modernization solution or whether the TRIO project would have arrived at a different conclusion had it more fully conformed to these best practices.

We also reviewed DHS guidance for conducting AOAs and AAs against the GAO-identified 22 AOA process best practices using the same methodology described above for reviewing the TRIO components’ AAs. In the course of applying these best practices to a TRIO component’s AA and to DHS guidance for the AA process, we assessed the reasonableness of the information we collected. We determined that the information from the DHS AA process was sufficiently reliable to use in assessing the TRIO components’ AAs and DHS guidance against these 22 best practices.

To determine the key factors, metrics, and processes used by the TRIO components in developing and evaluating DHS’s alternative solutions and final choice for financial system modernization, we reviewed each component’s AA, including a description of (1) the alternatives considered, (2) the market research conducted, (3) the three alternatives evaluated, (4) the selection criteria used and how the criteria were weighted, (5) how each alternative scored against the selection criteria, and (6) the alternative that scored the best according to the component’s evaluation.

To determine the extent to which DHS managed the risks of using IBC consistent with risk management best practices, we reviewed DHS’s and TRIO components’ risk management guidance and other documentation supporting their risk management efforts, including risk registers, mitigation plans, status reports, and risk management meeting minutes. We also met with officials to gain an understanding of the key processes and documents used for managing and reporting on TRIO project risks. We assessed the processes against best practices that the Software Engineering Institute (SEI) identified. The practices we selected are fundamental to effective risk management activities. These practices are identified in SEI’s Capability Maturity Model® Integration (CMMI®) for Acquisition, Version 1.3.

In particular, the key best practices for preparing for risk management are determine risk sources and categories, define risk parameters, and establish a risk management strategy.

The key best practices for identifying and analyzing risks are evaluate, categorize, and prioritize risks.

The key best practices for mitigating identified risks are develop risk mitigation plans and implement risk mitigation plans.

We applied the criteria from the CMMI risk management process area to determine the extent to which the expected practices were implemented, or future activities were planned for, by the program office. The rating system we used is as follows: (1) meets, or generally satisfies all elements of the specific practice; (2) partially meets, or generally satisfies a portion of specific practice elements; and (3) does not meet, or does not satisfy specific practice elements.

In the context of the best practices methodology, we assessed the reliability of TRIO project risk data contained in DHS’s risk register. We interviewed officials on how the risk register was developed and maintained, including key control activities used to provide reasonable assurance of the accuracy of the information reported in the register. We reviewed DHS’s July 2016 risk register and minutes from risk management committee meetings (one meeting per quarter, randomly selected). Of 120 TRIO project risks on the July 2016 risk register, we found 13 risks with missing data. Of 47 active risks identified, 28 risk records had not been modified in the previous 3 months and the register did not indicate when their accuracy was last confirmed and 35 risks were beyond their indicated impact dates but had not been marked as issues. We concluded that the pervasiveness of these data reliability problems decreased the usefulness of the risk register in connection with managing TRIO project risks.

To determine the key factors or challenges that have impacted the TRIO project and DHS’s plans for completing remaining key priorities, we met with DHS, IBC, Office of Financial Innovation and Transformation, and Unified Shared Services Management office officials and Office of Management and Budget staff to obtain their perspectives. In addition, we reviewed documentation provided by these officials, including TRIO project status reports and memorandums, leadership briefings, and other presentations.

We conducted this performance audit from March 2016 to September 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Best Practices for the Analysis of Alternatives Process

Many guides describe an approach to an analysis of alternatives (AOA); however, there is no single set of practices for the AOA process that has been broadly recognized by both the government and private sector entities. GAO has previously identified 22 best practices for an AOA process by (1) compiling and reviewing commonly mentioned AOA policies and guidance used by different government and private sector entities and (2) incorporating experts’ comments on a draft set of practices to develop a final set of practices. These practices are based on longstanding, fundamental tenets of sound decision making and economic analysis.

In addition, these practices can be applied to a wide range of activities in which an alternative must be selected from a set of possible options, as well as to a broad range of capability areas, projects, and programs. These practices can provide a framework to help ensure that entities consistently and reliably select the project alternative that best meets mission needs. The guidance below is an overview of the key principles that lead to a successful AOA process and not as a “how to” guide with detailed instructions for each best practice identified.

The 22 best practices that GAO identified are grouped into the following five phases: 1. Initialize the AOA process: Includes best practices that are applied before starting the process of identifying, analyzing, and selecting alternatives. This includes determining the mission need and functional requirements, developing the study time frame, creating a study plan, and determining who conducts the analysis. 2. Identify alternatives: Includes best practices that help ensure that the alternatives to be analyzed are sufficient, diverse, and viable. 3. Analyze alternatives: Includes best practices that compare the alternatives to be analyzed. The best practices in this category help ensure that the team conducting the analysis uses a standard, quantitative process to assess the alternatives. 4. Document and review the AOA process: Includes best practices that would be applied throughout the AOA process, such as documenting all steps taken to initialize, identify, and analyze alternatives and to select a preferred alternative in a single document. 5. Select a preferred alternative: Includes a best practice that is applied by the decision maker to compare alternatives and to select a preferred alternative.

The five phases address different themes of analysis necessary to complete the AOA process, and comprise the beginning of the AOA process (defining the mission needs and functional requirements) through the final step of the AOA process (selecting a preferred alternative).

We also identified four characteristics that relate to a reliable, high-quality AOA process—that the AOA process is well-documented, comprehensive, unbiased, and credible. Table 4 shows the four characteristics and their relevant AOA best practices.

Conforming to the 22 best practices helps ensure that the preferred alternative selected is the one that best meets the agency’s mission needs. Not conforming to the best practices may lead to an unreliable AOA process, and the agency will not have assurance that the preferred alternative best meets mission needs.

Appendix III: GAO Assessment of TRIO Components’ Alternatives Analyses

The Department of Homeland Security’s TRIO components—the U.S. Coast Guard (Coast Guard), Transportation Security Administration (TSA), and Domestic Nuclear Detection Office (DNDO)—conducted alternatives analyses (AA) during 2012 and 2013 to determine the best alternative for transitioning to a modernized financial management system solution. We evaluated the TRIO components’ AA processes against analysis of alternatives (AOA) best practices GAO identified as necessary characteristics of a reliable, high-quality AOA process (described in app. II).

GAO’s assessment of the extent to which Coast Guard’s, TSA’s, and DNDO’s AAs met each of the 22 best practices is detailed in tables 5, 6, and 7.

Appendix IV: Comments from the Department of Homeland Security

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, James Kernen (Assistant Director), William Brown, Courtney Cox, Eric Essig, Valerie Freeman, Matthew Gardner, Jason Lee, Jennifer Leotta, and Madhav Panwar made key contributions to this report.