prompt_front: You are given two reports by the government agency.



Question: Write a one-page summary of the report 1.

Report 1: VA’s mission is to promote the health, welfare, and dignity of all veterans in recognition of their service to the nation by ensuring that they receive medical care, benefits, social support, and lasting memorials  In carrying out this mission, the department operates one of the largest health care delivery systems in America, providing health care to millions of veterans and their families at more than 1,500 facilities  The department’s three major components—the Veterans Health Administration (VHA), the Veterans Benefits Administration (VBA), and the National Cemetery Administration (NCA)—are primarily responsible for carrying out its mission  More specifically, VHA provides health care services, including primary care and specialized care, and it performs research and development to improve veterans’ needs  VBA provides a variety of benefits to veterans and their families, including disability compensation, educational opportunities, assistance with home ownership, and life insurance  Further, NCA provides burial and memorial benefits to veterans and their families  Collectively, the three components rely on approximately 340,000 employees to provide services and benefits  These employees work in VA’s Washington, D C  headquarters, as well as 170 medical centers, approximately 750 community-based outpatient clinics, 300 veterans centers, 56 regional offices, and more than 130 cemeteries situated throughout the nation  The use of IT is critically important to VA’s efforts to provide benefits and services to veterans  As such, the department operates and maintains an IT infrastructure that is intended to provide the backbone necessary to meet the day-to-day operational needs of its medical centers, veteran- facing systems, benefits delivery systems, memorial services, and all other systems supporting the department’s mission  The infrastructure is to provide for data storage, transmission, and communications requirements necessary to ensure the delivery of reliable, available, and responsive support to all VA staff offices and administration customers, as well as veterans  According to department data as of October 2016, there were 576 active or in-development systems in VA’s inventory of IT systems  These systems are intended to be used for the determination of benefits, benefits claims processing, and access to health records, among other services  VHA is the parent organization for 319 of these systems  Of the 319 systems, 244 were considered mission-related and provide capabilities related to veterans’ health care delivery  For example, VHA’s systems provide capabilities to establish and maintain electronic health records that health care providers and other clinical staff use to view patient information in inpatient, outpatient, and long-term care settings  VistA serves an essential role in helping the department to fulfill its health care delivery mission  Specifically, VistA is an integrated medical information system for all veterans’ health information  It was developed in-house by the department’s clinicians and IT personnel and has been in operation since the early 1980s  As such, the system has long been vital to helping ensure the quality of health care received by the nation’s veterans and their dependents  VistA is comprised of more than 200 applications that assist in the delivery of health care and perform other important functions within the department, including financial management, enrollment, and registration  Some of these applications have been in operation for over 30 years and, according to VA, have become increasingly difficult and costly to maintain  As such, the department has expended extensive resources to modernize the system and increase its ability to allow for the viewing or exchange of patient information with the Department of Defense (DOD) and private sector health providers  In addition, as we recently reported, VHA has unaddressed needs that indicate its current health IT systems, including VistA, do not fully support the organization’s business functions  Specifically, about 39 percent of all requests related to health IT needs have remained unaddressed after more than 5 years  Electronic health records are particularly crucial for optimizing the health care provided to veterans, many of whom may have health records residing at multiple medical facilities within and outside the United States  Taking steps toward interoperability—that is, collecting, storing, retrieving, and transferring veterans’ health records electronically—is significant to improving the quality and efficiency of care  One of the goals of interoperability is to ensure that patients’ electronic health information is available from provider to provider, regardless of where it originated or resides  Since 2007, VA has been operating a centralized organization, the Office of Information and Technology (OI&T), in which most key functions intended for effective management of IT are performed  This office is led by the Assistant Secretary for Information and Technology—VA’s Chief Information Officer (CIO)  The office is responsible for providing strategy and technical direction, guidance, and policy related to how IT resources are to be acquired and managed for the department, and for working closely with its business partners—such as VHA—to identify and prioritize business needs and requirements for IT systems  Among other things, OI&T has responsibility for managing the majority of VA’s IT-related functions, including the maintenance and modernization of VistA  As of 2016, OI&T was comprised of more than 15,000 staff, with more than half of these positions filled by contractors  For fiscal year 2018, the department’s budget request included nearly $4 1 billion for IT  The department requested approximately $359 million for new systems development or modernization efforts, approximately $2 5 billion for maintaining existing systems, and approximately $1 2 billion for payroll and administration  For example, in its fiscal year 2018 budget submission, the department requested appropriations to support five IT portfolios, including the development and operations and maintenance for programs and projects related to the: Medical portfolio, which provides technology solutions to deliver modern, high-quality medical care capabilities to veterans ($944 2 million); Benefit portfolio, which addresses the technology needs managed by the Veterans Benefit Administration ($296 9 million); Memorial Affairs portfolio, which provides support for the modernization of applications and services for National Cemeteries at 133 locations nationwide ($24 5 million); Corporate portfolio, which consists of back office operations supporting the major business lines and department management ($270 6 million); and Enterprise IT, which provides the underlying infrastructure to enable the other portfolios to operate and includes such things as cybersecurity, data centers, cloud services, telephony, enterprise software, and data connectivity ($1 289 billion)  In 2015, we designated VA Health Care as a high-risk area for the federal government and, currently, we continue to be concerned about the department’s ability to ensure that its resources are being used cost- effectively and efficiently to improve veterans’ timely access to health care  In part, we identified limitations in the capacity of VA’s existing systems, including the outdated, inefficient nature of certain systems and a lack of system interoperability—that is, the ability to exchange and use electronic health information—as contributors to the department’s IT challenges related to health care  These challenges present risks to the timeliness, quality, and safety of the health care  While we recently reported that the department has begun to demonstrate leadership commitment to addressing IT challenges, more work remains  Also, in February 2015, we added Improving the Management of IT Acquisitions and Operations to our list of high-risk areas  Specifically, federal IT investments too frequently fail or incur cost overruns and schedule slippages while contributing little to mission-related outcomes  We have previously testified that the federal government has spent billions of dollars on failed IT investments, including, for example, VA’s Scheduling Replacement Project, which was terminated in September 2009 after spending an estimated $127 million over 9 years; and its Financial and Logistics Integrated Technology Enterprise program, which was intended to be delivered by 2014 at a total estimated cost of $609 million, but was terminated in October 2011 due to challenges in managing the program  This high-risk area highlighted several critical IT initiatives in need of additional congressional oversight, including (1) reviews of troubled projects; (2) efforts to increase the use of incremental development; (3) efforts to provide transparency relative to the cost, schedule, and risk levels for major IT investments; (4) reviews of agencies’ operational investments; (5) data center consolidation; and (6) efforts to streamline agencies’ portfolios of investments  We noted that agencies’ implementation of these initiatives was inconsistent and that more work remained to demonstrate progress in achieving IT acquisition and operation outcomes  We also recently issued an update to our high-risk report and noted that, while progress has been made in addressing the high-risk area of IT acquisitions and operations, significant work remains to be completed  For example, we noted, among other things, that additional work was needed to establish action plans for federal agencies to modernize or replace obsolete systems  Specifically, we pointed out that many federal systems use outdated software languages and hardware, which has increased spending on operations and maintenance of technology investments  VA was among a handful of departments with one or more archaic legacy systems  As discussed in our recent report on legacy systems used by federal agencies, we identified 2 of the department’s systems as being over 50 years old, and among the 10 oldest investments and/or systems that were reported by 12 selected agencies  Personnel and Accounting Integrated Data (PAID)—This 53-year old system automates time and attendance for employees, timekeepers, payroll, and supervisors  It is written in Common Business Oriented Language (COBOL), a programming language developed in the late 1950s and early 1960s, and runs on IBM mainframes  Benefits Delivery Network (BDN)—This 51-year old system tracks claims filed by veterans for benefits, eligibility, and dates of death  It is a suite of COBOL mainframe applications  Ongoing uses of antiquated systems, such as PAID and BDN, contribute to agencies spending a large, and increasing, proportion of their IT budgets on operations and maintenance of systems that have outlived their effectiveness and are consuming resources that outweigh their benefits  Accordingly, we have recommended that VA identify and plan to modernize or replace its legacy systems  The department concurred with our recommendation and stated that it plans to retire and replace PAID with the Human Resources Information System Shared Service Center in 2017  The department also stated that it has general plans to roll the capabilities of BDN into another system and to retire BDN in 2018  Congress enacted federal IT acquisition reform legislation (commonly referred to as the Federal Information Technology Acquisition Reform Act, or FITARA) in December 2014  This legislation was intended to improve agencies’ acquisitions of IT and enable Congress to monitor agencies’ progress and hold them accountable for reducing duplication and achieving cost savings  The law applies to VA and other covered agencies  It includes specific requirements related to seven areas, including data center consolidation and optimization, agency CIO authority, and government-wide software purchasing  Federal data center consolidation initiative (FDCCI)  Agencies are required to provide the Office of Management and Budget (OMB) with a data center inventory, a strategy for consolidating and optimizing their data centers (to include planned cost savings), and quarterly updates on progress made  The law also requires OMB to develop a goal for how much is to be saved through this initiative, and provide annual reports on cost savings achieved  Agency CIO authority enhancements  CIOs at covered agencies are required to (1) approve the IT budget requests of their respective agencies, (2) certify that IT investments are adequately implementing incremental development, as defined in capital planning guidance issued by OMB, (3) review and approve contracts for IT, and (4) approve the appointment of other agency employees with the title of CIO  Government-wide software purchasing program  The General Services Administration is to develop a strategic sourcing initiative to enhance government-wide acquisition and management of software  In doing so, the law requires that, to the maximum extent practicable, the General Services Administration should allow for the purchase of a software license agreement that is available for use by all executive branch agencies as a single user  Expanding upon FITARA, the Making Electronic Government Accountable by Yielding Tangible Efficiencies Act of 2016, or the “MEGABYTE Act,” further enhanced CIOs’ management of software licenses by requiring agency CIOs to establish an agency software licensing policy and a comprehensive software license inventory to track and maintain licenses, among other requirements  In June 2015, OMB released guidance describing how agencies are to implement FITARA  This guidance is intended to, among other things: assist agencies in aligning their IT resources with statutory establish government-wide IT management controls that will meet the law’s requirements, while providing agencies with flexibility to adapt to unique agency processes and requirements; clarify the CIO’s role and strengthen the relationship between agency CIOs and bureau CIOs; and strengthen CIO accountability for IT costs, schedules, performance, and security  In our draft report that is currently with VA for comments, we discuss the history of VA’s efforts to modernize its health information system, VistA  These four efforts—HealtheVet, the integrated Electronic Health Record (iEHR), VistA Evolution, and the Electronic Health Record Modernization (EHRM)—reflect varying approaches that the department has considered to achieve a modernized health care system over the course of nearly two decades  The modernization efforts are described as follows  In 2001, VA undertook its first VistA modernization project, the HealtheVet initiative, with the goals of standardizing the department’s health care system and eliminating the approximately 130 different systems used by its field locations at that time  HealtheVet was scheduled to be fully implemented by 2018 at a total estimated development and deployment cost of about $11 billion  As part of the effort, the department had planned to develop or enhance specific areas of system functionality through six projects, which were to be completed between 2006 and 2012  Specifically, these projects were to provide capabilities to support VA’s Health Data Repository and Patient Financial Services System, as well as the Laboratory, Pharmacy, Imaging, and Scheduling functions  In June 2008, we reported that the department had made progress on the HealtheVet initiative, but noted issues with project planning and governance  In June 2009, the Secretary of Veterans Affairs announced that VA would stop financing failed projects and improve the management of its IT development projects  Subsequently, in August 2010, the department reported that it had terminated the HealtheVet initiative  In February 2011, VA began its second modernization initiative, the iEHR program, in conjunction with DOD  The program was intended to replace the two separate electronic health record systems used by the two departments with a single, shared system  Moreover, because both departments would be using the same system, this approach was expected to largely sidestep the challenges that had been encountered in trying to achieve interoperability between their two separate systems  Initial plans called for the development of a single, joint system consisting of 54 clinical capabilities to be delivered in six increments between 2014 and 2017  Among the agreed-upon capabilities to be delivered were those supporting laboratory, anatomic pathology, pharmacy, and immunizations  According to VA and DOD, the single iEHR system had an estimated life cycle cost of $29 billion through the end of fiscal year 2029  However, in February 2013, the Secretaries of VA and DOD announced that they would not continue with their joint development of a single electronic health record system  This decision resulted from an assessment of the iEHR program that the secretaries had requested in December 2012 because of their concerns about the program facing challenges in meeting deadlines, costing too much, and taking too long to deliver capabilities  In 2013, the departments abandoned their plan to develop the integrated system and stated that they would again pursue separate modernization efforts  In December 2013, VA initiated its VistA Evolution program as a joint effort of VHA and OI&T that was to be completed by the end of fiscal year 2018  The program was to be comprised of a collection of projects and efforts focused on improving the efficiency and quality of veterans’ health care by modernizing the department’s health information systems, increasing the department’s data exchange and interoperability with DOD and private sector health care partners, and reducing the time it takes to deploy new health information management capabilities  Further, the program was intended to result in lower costs for system upgrades, maintenance, and sustainment  According to the department’s March 2017 cost estimate, VistA Evolution was to have a life cycle cost of about $4 billion through fiscal year 2028  Since initiating VistA Evolution in December 2013, VA has completed a number of key activities that were called for in its plans  For example, the department delivered capabilities, such as the ability for health providers to have an integrated, real-time view of electronic health record data through the Joint Legacy Viewer, as well as the ability for health care providers to view sensitive DOD notes and highlight abnormal test results for patients  VA also initiated work to standardize VistA across the 130 VA facilities and released enhancements to its legacy scheduling, pharmacy, and immunization systems  In addition, the department released the enterprise Health Management Platform, which is a web- based user interface that assembles patient clinical data from all VistA instances and DOD  Although VistA Evolution is ongoing, VA is currently in the process of revising its plan for the program as a result of the department recently announcing its pursuit of a fourth VistA modernization program (discussed below)  For example, the department determined that it would no longer pursue additional development or deployment of the enterprise Health Management Platform—a major VistA Evolution component— because the new modernization program is envisioned to provide similar capabilities  In June 2017, the VA Secretary announced a significant shift in the department’s approach to modernizing VistA  Specifically, rather than continue to use VistA, the Secretary stated that the department plans to acquire the same electronic health record system that DOD is implementing  In this regard, DOD has contracted with the Cerner Corporation to provide a new integrated electronic health record system  According to the Secretary, VA has chosen to acquire this same product because it would allow all of VA’s and DOD’s patient data to reside in one system, thus enabling seamless care between the department and DOD without the manual and electronic exchange and reconciliation of data between two separate systems  The VA Secretary added that this fourth modernization initiative is intended to minimize customization and system differences that currently exist within the department’s medical facilities, and ensure the consistency of processes and practices within VA and DOD  When fully operational, the system is intended to be the single source for patients to access their medical history and for clinicians to use that history in real time at any VA or DOD medical facility, which may result in improved health care outcomes  According to VA’s Chief Technology Officer, Cerner is expected to provide integration, configuration, testing, deployment, hosting, organizational change management, training, sustainment, and licenses necessary to deploy the system in a manner that meets the department’s needs  To expedite the acquisition, in June 2017, the Secretary signed a “Determination and Findings,” which noted a public interest exception to the requirement for full and open competition, and authorized VA to issue a solicitation directly to the Cerner Corporation  According to the Secretary, VA expects to award a contract to Cerner in December 2017, and deployment of the new system is anticipated to begin 18 months after the contract has been signed  VA’s Executive Director for the Electronic Health Records Modernization System stated that the department intends to incrementally deploy the new system to its medical facilities  Each facility is expected to continue using VistA until the new system has been deployed at that location  All VA medical facilities are anticipated to have the new system implemented within 7 to 8 years after the first deployment  Figure 1 shows a timeline of the four efforts that VA has pursued to modernize VistA since 2001  For iEHR and VistA Evolution, the two modernization initiatives for which VA could provide contract data, the department obligated approximately $1 1 billion for contracts with 138 different contractors during fiscal years 2011 through 2016  Specifically, the department obligated approximately $224 million and $880 million, respectively, for contracts associated with these efforts  Of the 138 contractors, 34 of them performed work supporting both iEHR and VistA Evolution  The remaining 104 contractors worked exclusively on either iEHR or VistA Evolution  Funding for the 34 contractors that worked on both iEHR and VistA Evolution totaled about $793 million of the $1 1 billion obligated for contracts on the two initiatives  Obligations for contracts awarded to the top 15 of these 34 contractors (which we designated as key contractors) accounted for about $741 million (about 67 percent) of the total obligated for contracts on the two initiatives  The remaining 123 contractors were obligated about $364 million for their contracts  The 15 key contractors were obligated about $564 million and $177 million for VistA Evolution and iEHR contracts, respectively  Table 1 identifies the key contractors and their obligated dollar totals for the two efforts  Additionally, we determined that, of the $741 million obligated to the key contractors, $411 million (about 55 percent) was obligated for contracts supporting the development of new system capabilities, $256 million (about 35 percent) was obligated for contracts supporting project management activities, and $74 million (about 10 percent) was obligated for contracts supporting operations and maintenance for iEHR and VistA Evolution  VA obligated funds to all 15 of the key contractors for system development, 13 of the key contractors for project management, and 12 of the key contractors for operations and maintenance  Figure 2 shows the amounts obligated for each of these areas  Further, based on the key contractors’ documentation, for the iEHR program, VA obligated $102 million for development, $65 million for project management, and $10 million for operations and maintenance  For the VistA Evolution Program, VA obligated $309 million for development, $191 million for project management, and $64 million for operations and maintenance  Figure 3 shows the amounts obligated for contracts on the VistA Evolution and iEHR programs for development, project management, and operations and maintenance  In addition, table 2 shows the amounts that each of the 15 key contractors were obligated for the three types of contract activities performed on iEHR and VistA Evolution  Industry best practices and IT project management principles stress the importance of sound planning for system modernization projects  These plans should identify key aspects of a project, such as the scope, responsible organizations, costs, schedules, and risks  Additionally, planning should begin early in the project’s lifecycle and be updated as the project progresses  Since the VA Secretary announced that the department would acquire the same electronic health record system as DOD, VA has begun planning for the transition from VistA Evolution to EHRM  However, the department is still early in its efforts, pending the contract award  In this regard, the department has begun developing plans that are intended to guide the new EHRM program  For example, the department has developed a preliminary description of the organizations that are to be responsible for governing the EHRM program  Further, the VA Secretary announced in congressional testimony in November 2017, a key reporting responsibility for the program—stating that the Executive Director for the Electronic Health Records Modernization System will report directly to the department’s Deputy Secretary  In addition, the department has developed a preliminary timeline for deploying its new electronic health record system to VA’s medical facilities, and a 90-day schedule that depicts key program activities  The department also has begun documenting the EHRM program risks  Beyond the aforementioned planning activities undertaken thus far, the Executive Director stated that the department intends to complete a full suite of planning and acquisition management documents to guide the program, including a life cycle cost estimate and an integrated master schedule to establish key milestones over the life of the project  To this end, the Executive Director told us that VA has awarded two program management contracts to support the development of these plans to MITRE Corporation and Booz Allen Hamilton  According to the Executive Director, VA also has begun reviewing the VistA Evolution Roadmap, which is the key plan that the department has used to guide VistA Evolution since 2014  This review is expected to result in an updated plan that is to prioritize any remaining VistA enhancements needed to support the transition from VistA Evolution to the new system  According to the Executive Director, the department intends to complete the development of its plans for EHRM within 90 days after award of the Cerner contract, which is anticipated to occur in December 2017  Further, beyond the development of plans, VA has begun to staff an organizational structure for the modernization initiative, with the Under Secretary of Health and the Assistant Secretary for Information and Technology (VA’s Chief Information Officer) designated as executive sponsors  It has also appointed a Chief Technology Officer from OI&T, and a Chief Medical Officer from VHA, both of whom are to report to the Executive Director  VA’s efforts to develop plans for EHRM and to staff an organization to manage the program encompass key aspects of project planning that are important to ensuring effective management of the department’s latest modernization initiative  However, the department remains early in its modernization planning efforts, many of which are dependent on the system acquisition contract award, which has not yet occurred  The department’s continued dedication to completing and effectively executing the planning activities that it has identified will be essential to helping minimize program risks and guide this latest electronic health record modernization initiative to a successful outcome—one which VA, for almost two decades, has yet to achieve  Beyond managing its system modernization efforts, such as VistA, VA has to ensure the effective implementation of the IT acquisition requirements called for in FITARA  Pursuant to FITARA, in August 2016, the Federal CIO issued a memorandum that announced the Data Center Optimization Initiative (DCOI)  According to OMB, this new initiative supersedes and builds on the results of FDCCI, and is also intended to improve the performance of federal data centers in areas such as facility utilization and power usage  Among other things, DCOI requires 24 federal departments and agencies, including VA, to develop plans and report on strategies (referred to as DCOI strategic plans) to consolidate inefficient infrastructure, optimize existing facilities, improve security posture, and achieve costs savings  Further, the memorandum establishes a set of five data center optimization metrics and performance targets intended to measure agency’s progress in the areas of (1) server utilization and automated monitoring, (2) energy metering, (3) power usage effectiveness, (4) facility utilization, and (5) virtualization  The guidance also indicates that OMB is to maintain a public dashboard that will display consolidation-related costs savings and optimization performance information for the agencies  However, in a series of reports that we issued from July 2011 through August 2017, we noted that, while data center consolidation could potentially save the federal government billions of dollars, weaknesses existed in several areas, including agencies’ data center consolidation plans, data center optimization, and OMB’s tracking and reporting on related cost savings  Further, we previously reported that VA’s progress toward closing data centers, and realizing the associated cost savings, lagged behind that of other covered agencies  More recently, VA reported a total inventory of 415 data centers, of which 39 had been closed as of August 2017  While the department anticipates another 10 data centers will be closed by the end of fiscal year 2018, these closures fall short of the targets set by OMB  Specifically, even if VA meets all of its planned targets for closure, it will only close about 9 percent of its tiered data centers and about 18 7 percent of its non-tiered data centers by the end of fiscal year 2018, which is short of the respective 25 and 60 percent targets set by OMB  Further, while VA has reported $23 61 million in data center-related cost savings and avoidances for 2012 through August 2017, the department does not expect to realize further savings from the additional 10 data center closures in the next year  In addition, in August 2017 we reported that agencies needed to address challenges in optimizing their data centers in order to achieve cost savings  Specifically, we noted that, according to the 24 agencies’ data center consolidation initiative strategic plans as of April 2017, most agencies were not planning to meet OMB’s optimization targets by the end of fiscal year 2018  As of February 2017, VA reported meeting one of the five data center optimization metrics related to power usage effectiveness  Also, the department’s data center optimization strategic plan indicates that the department plans to meet three of the five metrics by the end of fiscal year 2018  Further, while OMB directed agencies to replace manual collection and reporting of metrics with automated tools no later than fiscal year 2018, VA had only implemented automated tools at 6 percent of its data centers  OMB has emphasized the need to deliver investments in smaller parts, or increments, in order to reduce risk, deliver capabilities more quickly, and facilitate the adoption of emerging technologies  In 2010, it called for agencies’ major investments to deliver functionality every 12 months and, since 2012, every 6 months  Subsequently, FITARA codified a requirement that agency CIOs certify that IT investments are adequately implementing incremental development, as defined in the capital planning guidance issued by OMB  Later OMB guidance on the law’s implementation—issued in June 2015—directed agency CIOs to define processes and policies for their agencies which ensure that they certify that IT resources are adequately implementing incremental development  Between May 2014 and November 2017, we reported on agencies’ efforts to utilize incremental development practices for selected major investments  In November 2017, we noted that agencies reported that 62 percent of major IT software development investments were certified by the agency CIO as using adequate incremental development in fiscal year 2017, as required by FITARA  VA’s CIO certified the use of adequate incremental development for all 10 of its major IT investments  However, VA had not yet updated the department’s policy and process for the CIO’s certification of major IT investments’ adequate use of incremental development, in accordance with OMB’s guidance on the implementation of FITARA as we recommended  The department stated that it plans to address our recommendation to establish a policy and that the policy is targeted for completion in 2017  Federal agencies engage in thousands of licensing agreements annually  Effective management of software licenses can help organizations avoid purchasing too many licenses that result in unused software  In addition, effective management can help avoid purchasing too few licenses, which results in noncompliance with license terms and causes the imposition of additional fees  Federal agencies are responsible for managing their IT investment portfolios, including the risks from their major information system initiatives, in order to maximize the value of these investments to the agency  OMB developed a policy that requires agencies to conduct an annual, agency-wide IT portfolio review to, among other things, reduce commodity IT spending  Such areas of spending could include software licenses  We previously identified seven elements that a comprehensive software licensing policy should address: identify clear roles, responsibilities, and central oversight authority within the department for managing enterprise software license agreements and commercial software licenses; establish a comprehensive inventory (at least 80 percent of software license spending and/or enterprise licenses in the department) by identifying and collecting information about software license agreements using automated discovery and inventory tools; regularly track and maintain software licenses to assist the agency in implementing decisions throughout the software license management life cycle; analyze software usage and other data to make cost-effective provide training relevant to software license management; establish goals and objectives of the software license management consider the software license management life-cycle phases (i e , requisition, reception, deployment and maintenance, retirement, and disposal phases) to implement effective decision making and incorporate existing standards, processes, and metrics  We previously made recommendations to VA to (1) develop an agency- wide comprehensive policy for the management of software licenses that includes guidance for using analysis to better inform investment decision making, (2) employ a centralized software license management approach that is coordinated and integrated with key personnel, (3) establish a comprehensive inventory of software licenses using automated tools, (4) track and maintain a comprehensive inventory of software licenses using automated tools and metrics, (5) analyze agency-wide software license data to identify opportunities to reduce costs and better inform investment decision making, and (6) provide software license management training to appropriate personnel  Consistent with our recommendation, in July 2015, VA issued a comprehensive software licensing policy that addressed weaknesses we previously identified  The department also issued a directive that documents VA’s software license management policy and responsibilities for central management of agency-wide software licenses, consistent with our recommendations  By implementing our recommendations, VA should be better positioned to consistently and cost-effectively manage software throughout the agency  In August 2017, the department also provided documentation showing that it had generated a comprehensive inventory of software licenses using automated tools for the majority of agency software license spending or enterprise-wide licenses  This inventory can serve to reduce redundant applications and help identify other cost saving opportunities  Further, the department implemented a solution to analyze agency-wide software license data, including usage and costs  This solution should allow VA to identify cost saving opportunities and inform future investment decisions  In addition, the department has provided information indicating that appropriate personnel receive software license management training  In conclusion, VA has made extensive use of numerous contractors and has obligated more than $1 billion for contracts that supported two of four VistA modernization programs that the department has initiated  VA has recently begun the fourth modernization program in which it plans to replace VistA with the same commercially available electronic health record system that is used by DOD  However, the department’s latest modernization effort is in the early stages of planning and is dependent on the system acquisition contract award in December 2017  VA’s completion and effective execution of plans will be essential to guiding this latest electronic health record modernization initiative to a successful outcome  Beyond VistA, the department continues to make progress on key FITARA-related initiatives  Although the department has made progress in the area of software licensing, additional actions in the areas of data center consolidation and optimization, as well as incremental system development can better position VA to effectively manage its IT  We plan to continue to monitor the department’s progress on these important activities  Chairman Hurd, Ranking Member Kelly, and Members of the Subcommittee, this completes my prepared statement  I would be pleased to respond to any questions that you may have  If you or your staffs have any questions about this testimony, please contact David A  Powner at (202) 512-9286 or pownerd@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this testimony statement  GAO staff who made key contributions to this statement are Mark Bird (Assistant Director), Jacqueline Mai (Analyst in Charge), Justin Booth, Chris Businsky, Rebecca Eyler, Paris Hawkins, Valerie Hopkins, Brandon S  Pettis, Jennifer Stavros-Turner, Eric Trout, Christy Tyson, Eric Winter, and Charles Youman  This is a work of the U S  government and is not subject to copyright protection in the United States  The published product may be reproduced and distributed in its entirety without further permission from GAO  However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately 


Report 2: FEMA’s mission is to help people before, during, and after disasters  It provides assistance to those affected by emergencies and disasters by supplying immediate needs (e g , ice, water, food, and temporary housing) and providing financial assistance grants for damage to personal or public property  FEMA also provides non-disaster assistance grants to improve the nation’s preparedness, readiness, and resilience to all hazards  FEMA accomplishes a large part of its mission through awarding grants to state, local, and tribal governments and nongovernmental entities to help communities prevent, prepare for, protect against, mitigate the effects of, respond to, and recover from disasters and terrorist attacks  As previously mentioned, for fiscal years 2005 through 2014, the agency obligated about $104 5 billion in disaster relief grants  In addition, as of April 2018, the four major disasters in 2017—hurricanes Harvey, Irma, and Maria; and the California wildfires—had resulted in over $22 billion in FEMA grants  The current FEMA grants management environment is highly complex with many stakeholders, IT systems, and users  Specifically, this environment is comprised of 45 active disaster and non-disaster grant programs, which are grouped into 12 distinct grant categories  For example, one program in the Preparedness: Fire category is the Assistance to Firefighters Grants (AFG) program, which provides grants to fire departments, nonaffiliated emergency medical service organizations, and state fire training academies to support firefighting and emergency response needs  As another example, the Housing Assistance grant program is in the Recovery Assistance for Individuals category and provides financial assistance to individuals and households in geographical areas that have been declared an emergency or major disaster by the President  Table 1 lists FEMA’s non-disaster and disaster-based grant categories  According to FEMA, the processes for managing these different types of grants vary because the grant programs were developed independently by at least 18 separate authorizing laws that were enacted over a 62-year period (from 1947 through 2009)  The various laws call for different administrative and reporting requirements  For example, the Robert T  Stafford Disaster Relief and Emergency Assistance Act, as amended, established the statutory authority for 11 of the grant programs, such as the administration of Public Assistance and Individual Assistance grant programs after a presidentially declared disaster  The act also requires the FEMA Administrator to submit an annual report to the President and Congress covering FEMA’s expenditures, contributions, work, and accomplishments, pursuant to the act  As another example, the National Dam Safety Program Act established one of the grant programs aimed at providing financial assistance to improve dam safety  Key stakeholders in modernizing the IT grants management environment include the internal FEMA officials that review, approve, and monitor the grants awarded, such as grant specialists, program analysts, and supervisors  FEMA has estimated that it will need to support about 5,000 simultaneous internal users of its grants management systems  Other users include the grant recipients that apply for, receive, and submit reports on their grant awards; these are considered the external system users  These grant recipients can include individuals, states, local governments, Indian tribes, institutions of higher education, and nonprofit organizations  FEMA has estimated that there are hundreds of thousands of external users of its grants systems  The administration of the many different grant programs is distributed across four divisions within FEMA’s organizational structure  Figure 1 provides an overview of FEMA’s organizational structure and the divisions that are responsible for administering grants  Within three of the four divisions—Resilience, United States Fire Administration, and Office of Response and Recovery—16 different grant program offices are collectively responsible for administering the 45 grant programs  The fourth division consists of 10 regional offices that help administer grants within their designated geographical regions  For example, the Office of Response and Recovery division oversees three different offices that administer 13 grant programs that are largely related to providing assistance in response to presidentially declared disasters  Figure 2 shows the number of grant programs administered by each of the four divisions’ grant program and regional offices  In addition, appendix II lists the names of the 45 grant programs  FEMA’s OCIO is responsible for developing, enhancing, and maintaining the agency’s IT systems, and for increasing efficiencies and cooperation across the entire organization  However, we and the DHS Office of Inspector General (OIG) have previously reported that the grant programs and regional offices develop information systems independent of the OCIO and that this has contributed to the agency’s disparate IT environment  We and the DHS OIG have reported that this disparate IT environment was due, in part, to FEMA’s decentralized IT budget and acquisition practices  For example, from fiscal years 2010 through 2015, the OCIO’s budget represented about one-third of the agency’s IT budget, with the grant program offices accounting for the remaining two-thirds of that budget  In February 2018, the OIG found that FEMA had shown limited progress in improving its IT management and that many of the issues reported in prior audits remained unchanged  As such, the OIG initiated a more comprehensive audit of the agency’s IT management that is ongoing  FEMA has identified 10 primary legacy IT systems that support its grants management activities  According to the agency, most of these systems were developed to support specific grant programs or grant categories  Table 2 summarizes the 10 primary legacy systems  According to FEMA officials, the 10 primary grant systems are all in operation (several have been for decades) and are not interoperable  As a result, individual grant programs and regional offices have independently developed work arounds intended to address existing capability gaps with the primary systems  FEMA officials stated that while these work arounds have helped the agency partially address capability gaps with its primary systems, they are often nonstandardized processes, and introduce the potential for information security risks and errors  This environment has contributed to labor-intensive manual processes and an increased burden for grant recipients  The disparate systems have also led to poor information sharing and reporting capabilities, as well as difficulty reconciling financial data  The DHS OIG and we have previously highlighted challenges with FEMA’s past attempts to modernize its grant management systems  For example, In December 2006, the DHS OIG reported that EMMIE, an effort to modernize its grants management systems and provide a single grants processing solution, was being developed without a clear understanding and definition of the future solution  The report also identified the need to ensure crosscutting participation from headquarters, regions, and states in developing and maintaining a complete, documented set of FEMA business and system requirements  In April 2016, we found weaknesses in FEMA’s development of the EMMIE system  For example, we noted that the system was implemented without sufficient documentation of system requirements, an acquisition strategy, up-to-date cost estimate and schedule, total amount spent to develop the system, or a systems integration plan  In response to our findings and related recommendations, FEMA took action to address these issues  For example, the agency implemented a requirements management process that, among other things, provided guidance to programs on analyzing requirements to ensure that they are complete and verifiable  We reported in November 2017 that EMMIE lacked the ability to collect information on all pre-award activities and, as a result, agency officials said that they and applicants used ad hoc reports and personal tracking documents to manage and monitor the progress of grant applications  FEMA officials added that applicants often struggled to access the system and that the system was not user friendly  Due to EMMIE’s shortfalls, the agency had to develop another system in 2017 to supplement EMMIE with additional grant tracking and case management capabilities  FEMA initiated GMM in 2015, in part, due to EMMIE’s failed attempt to modernize the agency’s grants management environment  The program is intended to modernize and streamline the agency’s grants management environment  To help streamline the agency’s grants management processes, the program established a standard framework intended to represent a common grants management lifecycle  The framework consists of five sequential phases—pre-award, award, post-award, closeout, and post- closeout—along with a sixth phase dedicated to continuous grant program management activities, such as analyzing data and producing reports on grant awards and managing IT systems  FEMA also established 43 distinct business functions associated with these six lifecycle phases  Figure 3 provides the general activities that may occur in each of the grant lifecycle phases, but specific activities would depend on the type of grant being administered (i e , disaster versus non-disaster)  GMM is expected to be implemented within the complex IT environment that currently exists at FEMA  For example, the program is intended to replace the 10 legacy grants management systems, and potentially many additional subsystems, with a single IT system  Each of the 10 legacy systems was developed with its own database(s) and with no standardization of the grants management data and, according to FEMA officials, this legacy data has grown significantly over time  Accordingly, FEMA will need to migrate, analyze, and standardize the grants management data before transitioning it to GMM  The agency awarded a contract in June 2016 to support the data migration efforts for GMM  The agency also implemented a data staging environment in October 2017 to migrate the legacy data and identify opportunities to improve the quality of the data  Further, the GMM system is expected to interface with a total of 38 other systems  These include 19 systems external to DHS (e g , those provided by commercial entities or other federal government agencies) and 19 systems internal to DHS or FEMA  Some of the internal FEMA systems are undergoing their own modernization efforts and will need to be coordinated with GMM, such as the agency’s financial management systems, national flood insurance systems, and enterprise data warehouses  For example, FEMA’s Financial Systems Modernization Program was originally expected to deliver a new financial system in time to interface with GMM  However, the financial modernization has been delayed until after GMM is to be fully implemented; thus, GMM will instead need to interface with the legacy financial system  As a result, GMM is in the process of removing one of its key performance parameters in the acquisition program baseline related to financial systems interoperability and timeliness of data exchanged  In May 2017, DHS approved the acquisition program baseline for GMM  The baseline estimated the total lifecycle costs to be about $251 million, initial operational capability to be achieved by September 2019, and full operational capability to be achieved by September 2020  FEMA intends to develop and deploy its own software applications for GMM using a combination of commercial-off-the-shelf software, open source software, and custom developed code  The agency plans to rely on an Agile software development approach  According to FEMA planning documentation, the agency plans to fully deliver GMM by September 2020 over eight Agile development increments  Agile development is a type of incremental development, which calls for the rapid delivery of software in small, short increments  Many organizations, especially in the federal government, are accustomed to using a waterfall software development model  This type of model typically consists of long, sequential phases, and differs significantly from the Agile development approach  We have previously reported that DHS has sought to establish Agile software development as the preferred method for acquiring and delivering IT capabilities  However, the department has not yet completed critical actions necessary to update its guidance, policies, and practices for Agile programs, in areas such as, developing lifecycle cost estimates, managing IT requirements, testing and evaluation, oversight at key decision points, and ensuring cybersecurity  (See appendix III for more details on the Agile software development approach ) FEMA’s acquisition approach includes using contract support to assist with the development and deployment efforts  The agency selected a public cloud environment to host the computing infrastructure  In addition, from March through July 2017, the agency used a short-term contract aimed at developing prototypes of GMM functionality for grant tracking and monitoring, case management of disaster survivors, grant reporting, and grant closeout  The agency planned to award a second development contract by December 2017 to complete the GMM system (beyond the prototypes) and to begin this work in September 2018  However, due to delays in awarding the second contract to develop the complete GMM system, in January 2018, the program extended the scope and time frames of the initial short-term prototype contract for an additional year to develop the first increment of the GMM system— referred to as the AFG pilot  On August 31, 2018, FEMA awarded the second development contract, which is intended to deliver the remaining functionality beyond the AFG pilot (i e , increments 2 through 8)  FEMA officials subsequently issued a 90-day planning task order for the Agile development contractor to define the work that needs to be done to deliver GMM and the level of effort needed to accomplish that work  However, the planning task order was paused after a bid protest was filed with GAO in September 2018  According to FEMA officials, they resumed work on the planning task order after the bid protest was withdrawn by the protester on November 20, 2018, and then the work was paused again during the partial government shutdown from December 22, 2018, through January 25, 2019  FEMA began working on the AFG pilot—GMM’s first increment—in January 2018  This increment was intended to pilot GMM’s use of Agile development methods to replace core functionality for the AFG system (i e , one of the 10 legacy systems) This system supports three preparedness/fire-related grant programs—Assistance to Firefighters Grants Program, Fire Prevention and Safety Grant Program, and Staffing for Adequate Fire and Emergency Response Grant Program  According to FEMA officials, the AFG system was selected as the first system to be replaced because it is costly to maintain and the DHS OIG had identified cybersecurity concerns with the system  Among the 43 GMM business functions discussed earlier in this report, FEMA officials specified 19 functions to be delivered in the AFG pilot  Figure 4 shows the planned time frames for delivering the AFG pilot in increment 1 (which consisted of four 3-month Agile development sub- increments), as of August 2018  As of August 2018, the program was working on sub-increment 1C of the pilot  In September 2018, GMM deployed its first set of functionality to a total of 19 AFG users—which included seven of 169 total internal AFG users, and 12 of more than 153,000 external AFG users  The functionality supported four of the 19 business functions that are related to the closeout of grants (i e , the process by which all applicable administrative actions and all required work to award a grant have been completed)  This functionality included tasks such as evaluation of final financial reports submitted by grant recipients and final reconciliation of finances (e g , final disbursement to recipients and return of unobligated federal funds)  According to FEMA officials, closeout functionality was selected first for deployment because it was the most costly component of the legacy AFG system to maintain, as it is an entirely manual and labor-intensive process  The remaining AFG functionality and remaining AFG users are to be deployed by the end of the AFG pilot  The GMM program is executed by a program management office, which is overseen by a program manager and program executive  This office is responsible for directing the day-to-day operations and ensuring completion of GMM program goals and objectives  The program office resides within the Office of Response and Recovery, which is headed by an Associate Administrator who reports to the FEMA Administrator  In addition, the GMM program executive (who is also the Regional Administrator for FEMA Region IX) reports directly to the FEMA Administrator  GMM is designated as a level 2 major acquisition, which means that it is subject to oversight by the DHS acquisition review board  The board is chaired by the DHS Undersecretary for Management and is made up of executive-level members, such as the DHS Chief Information Officer  The acquisition review board serves as the departmental executive board that decides whether to approve GMM through key acquisition milestones and reviews the program’s progress and its compliance with approved documentation every 6 months  The board approved the acquisition program baseline for GMM in May 2017 (i e , estimated costs to be about $251 million and full operational capability to be achieved by September 2020)  In addition, the program is reviewed on a monthly basis by FEMA’s Grants Management Executive Steering Group  This group is chaired by the Deputy Administrator of FEMA  Further, DHS’s Financial Systems Modernization Executive Steering Committee, chaired by the DHS Chief Financial Officer, meets monthly and is to provide guidance, oversight, and support to GMM  For government organizations, including FEMA, cybersecurity is a key element in maintaining the public trust  Inadequately protected systems may be vulnerable to insider threats  Such systems are also vulnerable to the risk of intrusion by individuals or groups with malicious intent who could unlawfully access the systems to obtain sensitive information, disrupt operations, or launch attacks against other computer systems and networks  Moreover, cyber-based threats to federal information systems are evolving and growing  Accordingly, we designated cybersecurity as a government-wide high risk area 22 years ago, in 1997, and it has since remained on our high-risk list  Federal law and guidance specify requirements for protecting federal information and information systems  The Federal Information Security Modernization Act (FISMA) of 2014 requires executive branch agencies to develop, document, and implement an agency-wide cybersecurity program to provide security for the information and information systems that support operations and assets of the agency  The act also tasks NIST with developing, for systems other than those for national security, standards and guidelines to be used by all agencies to establish minimum cybersecurity requirements for information and information systems based on their level of cybersecurity risk  Accordingly, NIST developed a risk management framework of standards and guidelines for agencies to follow in developing cybersecurity programs  The framework addresses broad cybersecurity and risk management activities, including categorizing the system’s impact level; selecting, implementing, and assessing security controls; authorizing the system to operate (based on progress in remediating control weaknesses and an assessment of residual risk); and monitoring the efficacy of controls on an ongoing basis  Figure 5 provides an overview of this framework  Prior DHS OIG assessments, such as the annual evaluation of DHS’s cybersecurity program, have identified issues with FEMA’s cybersecurity practices  For example, in 2016, the OIG reported that FEMA was operating 111 systems without an authorization to operate  In addition, the agency had not created any corrective action plans for 11 of the systems that were classified as “Secret” or “Top Secret,” thus limiting its ability to ensure that all identified cybersecurity weaknesses were mitigated in a timely manner  The OIG further reported that, for several years, FEMA was consistently below DHS’s 90 percent target for remediating corrective action plans, with scores ranging from 73 to 84 percent  Further, the OIG reported that FEMA had a significant number of open corrective action plans (18,654) and that most of these plans did not contain sufficient information to address identified weaknesses  In 2017, the OIG reported that FEMA had made progress in addressing security weaknesses  For example, it reported that the agency had reduced the number of systems it was operating without an authorization to operate from 111 to 15 systems  According to GAO’s Business Process Reengineering Assessment Guide and the Software Engineering Institute’s Capability Maturity Model Integration® for Development, successful business process reengineering can enable agencies to replace their inefficient and outmoded processes with streamlined processes that can more effectively serve the needs of the public and significantly reduce costs and improve performance  Many times, new IT systems are implemented to support these improved business processes  Thus, effective management of IT requirements is critical for ensuring the successful design, development, and delivery of such new systems  These leading practices state that effective business process reengineering and IT requirements management involve, among other things, (1) ensuring strong executive leadership support for process reengineering; (2) assessing the current and target business environment and business performance goals; (3) establishing plans for implementing new business processes; (4) establishing clear, prioritized, and traceable IT requirements; (5) tracking progress in delivering IT requirements; and (6) incorporating input from end user stakeholders  Among these six selected leading practices for reengineering business processes and managing IT requirements, FEMA fully implemented four and partially implemented two of them for its GMM program  For example, the agency ensured strong senior leadership commitment to changing the way it manages its grants, took steps to assess and document its business environment and performance goals, defined initial IT requirements for GMM, took recent actions to better track progress in delivering planned IT requirements, and incorporated input from end user stakeholders  In addition, FEMA had begun planning for business process reengineering; however, it had not finalized plans for transitioning users to the new business processes  Further, while GMM took steps to establish clearly defined and prioritized IT requirements, key requirements were not always traceable  Table 3 summarizes the extent to which FEMA implemented the selected leading practices  According to GAO’s Business Process Reengineering Assessment Guide, the most critical factor for engaging in a reengineering effort is having strong executive leadership support to establish credibility regarding the seriousness of the effort and to maintain the momentum as the agency faces potentially extensive changes to its organizational structure and values  Without such leadership, even the best process design may fail to be accepted and implemented  Agencies should also ensure that there is ongoing executive support (e g , executive steering committee meetings headed by the agency leader) to oversee the reengineering effort from start to finish  FEMA senior leadership consistently demonstrated its commitment and support for streamlining the agency’s grants management business processes and provided ongoing executive support  For example, one of the Administrator’s top priorities highlighted in FEMA’s 2014 through 2022 strategic plans was to strengthen grants management through innovative systems and business processes to rapidly and effectively deliver the agency’s mission  In accordance with this strategic priority, FEMA initiated GMM with the intent to streamline and modernize grants management across the agency  In addition, FEMA established the Grants Management Executive Steering Group in September 2015  This group is responsible for transforming the agency’s grants management capabilities through its evaluation, prioritization, and oversight of grants management modernization programs, such as GMM  The group’s membership consists of FEMA senior leaders from across the agency’s program and business support areas, such as FEMA regions, Individual Assistance, Public Assistance, Preparedness, Office of the Chief Financial Officer, Office of Chief Counsel, OCIO, and the Office of Policy and Program Analysis  In this group’s ongoing commitment to reengineering grants management processes, it meets monthly to review GMM’s updates, risks, and action items, as well as the program’s budget, schedule, and acquisition activities  For example, the group reviewed the status of key acquisition activities and program milestones, such as the follow-on award for the pilot contractor and the program’s initial operational capability date  The group also reviewed GMM’s program risks, such as data migration challenges (discussed later in this report) and delays in the Agile development contract award  With this continuous executive involvement, FEMA is better positioned to maintain momentum for reengineering the new grants management business processes that the GMM system is intended to support  GAO’s Business Process Reengineering Assessment Guide states that agencies undergoing business process reengineering should develop a common understanding of the current environment by documenting existing core business processes to show how the processes work and how they are interconnected  The agencies should then develop a deeper understanding of the target environment by modeling the workflow of each target business process in enough detail to provide a common understanding of exactly what will be changed and who will be affected by a future solution  Agencies should also assess the performance of their current major business processes to identify problem areas that need to be changed or eliminated and to set realistically achievable, customer- oriented, and measurable business performance improvement goals  FEMA has taken steps to document the current and target grants management business processes  Specifically, The agency took steps to develop a common understanding of its grants management processes by documenting each of the 12 grant categories  For example, in 2016 and 2017, the agency conducted several nationwide user outreach sessions with representatives from FEMA headquarters, the 10 regional offices, and state and local grant recipients to discuss the grant categories and the current grants management business environment  In addition, FEMA’s Office of Chief Counsel developed a Grants Management Manual in January 2018 that outlined the authorizing laws, regulations, and agency policies for all of its grant programs  According to the Grants Management Executive Steering Group, the manual is intended to promote standardized grants management procedures across the agency  Additionally, the group expects grant program and regional offices to assess the manual against their own practices, make updates as needed, and ensure that their staff are properly informed and trained  FEMA also documented target grants management business process workflows for 18 of the 19 business functions that were notionally planned to be developed and deployed in the AFG pilot by December 2018  However, the program experienced delays in developing the AFG pilot (discussed later in this report) and, thus, deferred defining the remaining business function until the program gets closer to developing that function, which is now planned for August 2019  In addition, FEMA established measurable business performance goals for GMM that are aimed at addressing problem areas and improving grants management processes  Specifically, the agency established 14 business performance goals and associated thresholds in an October 2017 acquisition program baseline addendum, as well as 126 performance metrics for all 43 of the target grants management business functions in its March 2017 test and evaluation master plan  According to FEMA, the 14 business performance goals are intended to represent essential outcomes that will indicate whether GMM has successfully met critical, business-focused mission needs  GMM performance goals include areas such as improvements in the satisfaction level of users with GMM compared to the legacy systems and improvements in the timeliness of grant award processing  For example, one of GMM’s goals is to get at least 40 percent of users surveyed to agree or strongly agree that their grants management business processes are easier to accomplish with GMM, compared to the legacy systems  Program officials stated that they plan to work with the Agile development contractor to refine their performance goals and target thresholds, develop a plan for collecting the data and calculating the metrics, and establish a performance baseline with the legacy systems  Program officials also stated that they plan to complete these steps by September 2019—GMM’s initial operational capability date—which is when they are required to begin reporting these metrics to the DHS acquisition review board  According to GAO’s Business Process Reengineering Assessment Guide, agencies undergoing business process reengineering should (1) establish an overall plan to guide the effort (commonly referred to as an organizational change management plan) and (2) provide a common understanding for stakeholders of what to expect and how to plan for process changes  Agencies should develop the plan at the beginning of the reengineering effort and provide specific details on upcoming process changes, such as critical milestones and deliverables for an orderly transition, roles and responsibilities for change management activities, reengineering goals, skills and resource needs, key barriers to change, communication expectations, training, and any staff redeployments or reductions-in-force  The agency should develop and begin implementing its change management plan ahead of introducing new processes to ensure sufficient support among stakeholders for the reengineered processes  While FEMA has begun planning its business process reengineering activities, it has not finalized its plans or established time frames for their completion  Specifically, as of September 2018, program officials were in the process of drafting an organizational change management plan that is intended to establish an approach for preparing grants management stakeholders for upcoming changes  According to FEMA, this document is intended to help avoid uncertainty and confusion among stakeholders as changes are made to the agency’s grant programs, and ensure successful adoption of new business processes, strategies, and technologies  As discussed previously in this report, the transition to GMM will involve changes to FEMA’s disparate grants management processes that are managed by many different stakeholders across the agency  Program officials acknowledged that change management is the biggest challenge they face in implementing GMM and said they had begun taking several actions intended to support the agency’s change management activities  For example, program officials reported in October 2018 that they had recently created an executive-level working group intended to address FEMA’s policy challenges related to the standardization of grants management processes  Additionally, program officials reported that they planned to: (1) hire additional support staff focused on coordinating grants change management activities; and (2) pursue regional office outreach to encourage broad support among GMM’s decentralized stakeholders, such as state, local, and tribal territories  However, despite these actions, the officials were unable to provide time frames for completing the organizational change management plan or the additional actions  Until the plan and actions are complete, the program lacks assurance that it will have sufficient support among stakeholders for the reengineered processes  In addition, GMM did not establish plans and time frames for the activities that needed to take place prior to, during, and after the transition from the legacy AFG to GMM  Instead, program officials stated that they had worked collaboratively with the legacy AFG program and planned these details informally by discussing them in various communications, such as emails and meetings  However, this informal planning approach is not a repeatable process, which is essential to this program as FEMA plans to transition many sets of functionality to many different users during the lifecycle of this program  Program officials acknowledged that for future transitions they will need more repeatable transition planning and stated that they intend to establish such plans, but did not provide a time frame for when such changes would be made  Until FEMA develops a repeatable process, with established time frames for communicating the transition details to its customers prior to each transition, the agency risks that the transition from the legacy systems to GMM will not occur as intended  It also increases its risk that stakeholders will not support the implementation of reengineered grants management processes  Leading practices for software development efforts state that IT requirements are to be clearly defined and prioritized  This includes, among other things, maintaining bidirectional traceability as the requirements evolve, to ensure there are no inconsistencies among program plans and requirements  In addition, programs using Agile software development are to maintain a product vision, or roadmap, to guide the planning of major program milestones and provide a high-level view of planned requirements  Programs should also maintain a prioritized list (referred to as a backlog) of narrowly defined requirements (referred to as lower-level requirements) that are to be delivered  Programs should maintain this backlog with the product owner to ensure the program is always working on the highest priority requirements that will deliver the most value to the users  The GMM program established clearly defined and prioritized requirements and maintained bidirectional traceability among the various levels of requirements: Grant lifecycle phases: In its Concept of Operations document, the program established six grants management lifecycle phases that represent the highest level of GMM’s requirements, through which it derives lower-level requirements  Business functions: The Concept of Operations document also identifies the next level of GMM requirements—the 43 business functions that describe how FEMA officials, grant recipients, and other stakeholders are to manage grants  According to program officials, the 43 business functions are to be refined, prioritized, and delivered to GMM customers iteratively  Further, for the AFG pilot, the GMM program office prioritized 19 business functions with the product owner and planned the development of these functions in a roadmap  Epics: GMM’s business functions are decomposed into epics, which represent smaller portions of functionality that can be developed over multiple increments  According to program officials, GMM intends to develop, refine, and prioritize the epics iteratively  As of August 2018, the program had developed 67 epics in the program backlog  An example of one of the epics for the AFG pilot is to prepare and submit grant closeout materials  User stories: The epics are decomposed into user stories, which convey the customers’ requirements at the smallest and most discrete unit of work that must be done within a single sprint to create working software  GMM develops, refines, and prioritizes the user stories iteratively  As of August 2018, the program had developed 1,118 user stories in the backlog  An example of a user story is “As an external user, I can log in with a username and password ” Figure 6 provides an example of how GMM’s different levels of requirements are decomposed  Nevertheless, while we found requirements to be traceable at the sprint- level (i e , epics and user stories), traceability of requirements at the increment-level (i e , business functions) were inconsistent among different requirements planning documents  Specifically, the capabilities and constraints document shows that five business functions are planned to be developed within sub-increment 1A, whereas the other key planning document—the roadmap for the AFG pilot—showed one of those five functions as being planned for the sub-increment 1B  In addition, the capabilities and constraints document shows that nine business functions are planned to be developed within sub-increment 1B, but the roadmap showed one of those nine functions as being planned for the sub- increment 1C  Program officials stated that they decided to defer these functions to later sub-increments due to unexpected technical difficulties encountered when developing functionality and reprioritizing functions with the product owners  While the officials updated the roadmap to reflect the deferred functionality, they did not update the capabilities and constraints document to maintain traceability between these two important requirements planning documents  Program officials stated that they learned during the AFG pilot that the use of a capabilities and constraints document for increment-level scope planning was not ideal and that they intended to change the process for how they documented planned requirements for future increments  However, program officials did not provide a time frame for when this change would be made  Until the program makes this change and then ensures it maintains traceability of increment-level requirements between requirements planning documents, it will continue to risk confusion among stakeholders about what is to be delivered  In addition, until recently, GMM’s planning documents were missing up- to-date information regarding when most of the legacy systems will be transitioned to GMM  Specifically, while the program’s planning documents (including the GMM roadmap) provided key milestones for the entire lifecycle of the program and high-level capabilities to be delivered in the AFG pilot, these documents lacked up-to-date time frames for when FEMA planned to transition the nine remaining legacy systems  For example, in May 2017, GMM drafted notional time frames for transitioning the legacy systems, including plans for AFG to be the seventh system replaced by GMM  However, in December 2017, the program decided to reprioritize the legacy systems so that AFG would be replaced first—yet this major change was not reflected in the program’s roadmap  Moreover, while AFG program officials were informed of the decision to transition the AFG program first, in June 2018 officials from other grant programs told us that they had not been informed on when their systems were to be replaced  As a result, these programs were uncertain about when they should start planning for their respective transitions  In August 2018, GMM program officials acknowledged that they were delayed in deciding the sequencing order for the legacy system transitions  Program officials stated that the delay was due to their need to factor the Agile development contractor’s perspective into these decisions; yet, at that time, the contract award had been delayed by approximately 8 months  Subsequently, in October 2018, program officials identified tentative time frames for transitioning the remaining legacy systems  Program officials stated that they determined the tentative time frames for transitioning the legacy systems based on key factors, such as mission need, cost, security vulnerabilities, and technical obsolescence, and that they had shared these new time frames with grant program officials  The officials also stated that, once the Agile contractor begins contract performance, they expect to be able to validate the contractor’s capacity and finalize these time frames by obtaining approval from the Grants Management Executive Steering Group  By taking steps to update and communicate these important time frames, FEMA should be better positioned to ensure that each of the grant programs are prepared for transitioning to GMM  According to leading practices, Agile programs should track their progress in delivering planned IT requirements within a sprint (i e , short iterations that produce working software)  Given that sprints are very short cycles of development (e g , 2 weeks), the efficiency of completing planned work within a sprint relies on a disciplined approach that includes using a fixed pace, referred to as the sprint cadence, that provides a consistent and predictable development routine  A disciplined approach also includes identifying by the start of a sprint which user stories will be developed, developing those stories to completion (e g , fully tested and demonstrated to, and accepted by, the product owner), and tracking completion progress of those stories  Progress should be communicated to relevant stakeholders and used by the development teams to better understand their capacity to develop stories, continuously improve on their processes, and forecast how long it will take to deliver all remaining capabilities  The GMM program did not effectively track progress in delivering IT requirements during the first nine sprints, which occurred from January to June 2018  These gaps in tracking the progress of requirements, in part, had an impact on the program’s progress in delivering the 19 AFG business functions that were originally planned by December 2018 and are now deferred to August 2019  However, beginning in July 2018, in response to our ongoing review, the program took steps to improve in these areas  Specifically, GMM did not communicate the status of its Agile development progress to program stakeholders, such as the grant programs, the regional offices, and the development teams, during most of the first nine sprints  Program officials acknowledged that they should use metrics to track development progress and, in July 2018, they began reporting metrics to program stakeholders  For example, they began collecting and providing data on the number of stories planned and delivered, estimated capacity for development teams, and the number of days spent working on the sprint, as part of the program’s weekly status reports to program stakeholders, such as product owners  Rather than using a fixed, predictable sprint cadence, GMM allowed a variable development cadence, meaning that sprint durations varied from 1 to 4 weeks throughout the first nine sprints  Program officials noted that they had experimented with the use of a variable cadence to allow more time to complete complex technical work  Program officials stated that they realized that varying the sprints was not effective and, in July 2018 for sprint 10, they reverted back to a fixed, 2 week cadence  GMM added a significant amount of scope during its first nine sprints, after the development work had already begun  For example, the program committed to 28 user stories at the beginning of sprint eight, and then nearly doubled the work by adding 25 additional stories in the middle of the sprint  Program officials cited multiple reasons for adding more stories, including that an insufficient number of stories had been defined in the backlog when the sprint began, the realization that planned stories were too large and needed to be decomposed into smaller stories, and the realization that other work would be needed in addition to what was originally planned  Program officials recognized that, by the start of a sprint, the requirements should be sufficiently defined, such that they are ready for development without requiring major changes during the sprint  The program made recent improvements in sprints 11 and 12, which had only five stories added after the start of a sprint  By taking these steps to establish consistency among sprints, the program has better positioned itself to more effectively monitor and manage the remaining IT development work  In addition, this improvement in consistency should help the program avoid future deferments of functionality  Leading practices state that programs should regularly collaborate with, and collect input from, relevant stakeholders; monitor the status of stakeholder involvement; incorporate stakeholder input; and measure how well stakeholders’ needs are being met  For Agile programs, it is especially important to track user satisfaction to determine how well the program has met stakeholders’ needs  Consistent stakeholder participation ensures that the program meets its stakeholders’ needs  FEMA implemented its responsibilities in this area through several means, such as stakeholder outreach activities; development of a strategic communications plan; and continuous monitoring, solicitation, and recording of stakeholder involvement and feedback  For example, the agency conducted nationwide outreach sessions from January 2016 through August 2017 and began conducting additional outreach sessions in April 2018  These outreach sessions involved hundreds of representatives from FEMA headquarters, the 10 regional offices, and state and local grant recipients to collect information on the current grants management environment and opportunities for streamlining grants management processes  FEMA also held oversight and stakeholder outreach activities and actively solicited and recorded feedback from its stakeholders on a regular basis  For example, GMM regularly verified with users that the new functionality met their IT requirements, as part of the Agile development cycle  Additionally, we observed several GMM biweekly requirements validation sessions where the program’s stakeholders were involved and provided feedback as part of the requirements development and refinement process  In addition, FEMA identified GMM stakeholders and tracked its engagement with these stakeholders using a stakeholder register  The agency also defined processes for how the GMM program is to collaborate with its stakeholders in a stakeholder communication plan and Agile development team agreement  Also, while several officials from the selected grant program and regional offices that we interviewed indicated that the program could improve in communicating its plans for GMM and incorporating stakeholder input, most of the representatives from these offices stated that GMM is doing well at interacting with its stakeholders  Finally, in October 2018, program officials reported that they had recently begun measuring user satisfaction by conducting surveys and interviews with users that have utilized the new functionality within GMM  The program’s outreach activities, collection of stakeholder input, and measurement of user satisfaction demonstrate that the program is taking the appropriate steps to incorporate stakeholder input  Reliable cost estimates are critical for successfully delivering IT programs  Such estimates provide the basis for informed decision making, realistic budget formulation, meaningful progress measurement, and accountability for results  GAO’s Cost Estimating and Assessment Guide defines leading practices related to the following four characteristics of a high-quality, reliable estimate  Comprehensive

ground_truth: The use of IT is crucial to helping VA effectively serve the nation's veterans and, each year, the department spends billions of dollars on its information systems and assets. However, VA has faced challenges spanning a number of critical initiatives related to modernizing its major systems. To improve all major federal agencies' acquisitions and hold them accountable for reducing duplication and achieving cost savings, in December 2014 Congress enacted federal IT acquisition reform legislation (commonly referred to as the Federal Information Technology Acquisition Reform Act , or FITARA). GAO was asked to summarize its previous and ongoing work regarding VA's history of efforts to modernize VistA, including past use of contractors, and the department's recent effort to acquire a commercial electronic health record system to replace VistA. GAO was also asked to provide an update on VA's progress in key FITARA-related areas, including (1) data center consolidation and optimization, (2) incremental system development practices, and (3) software license management. VA generally agreed with the information upon which this statement is based. For nearly two decades, the Department of Veterans Affairs (VA) has undertaken multiple efforts to modernize its health information system—the Veterans Health Information Systems and Technology Architecture (known as VistA). Two of VA's most recent efforts included the Integrated Electronic Health Record (iEHR) program, a joint program with the Department of Defense (DOD) intended to replace separate systems used by VA and DOD with a single system; and the VistA Evolution program, which was to modernize VistA with additional capabilities and a better interface for all users. VA has relied extensively on assistance from contractors for these efforts. VA obligated over $1.1 billion for contracts with 138 contractors during fiscal years 2011 through 2016 for iEHR and VistA Evolution. Contract data showed that the 15 key contractors that worked on both programs accounted for $741 million of the funding obligated for system development, project management, and operations and maintenance to support the two programs (see figure). VA recently announced that it intends to change its VistA modernization approach and acquire the same electronic health record system that DOD is implementing. With respect to key FITARA-related areas, the department has reported progress on consolidating and optimizing its data centers, although this progress has fallen short of targets set by the Office of Management and Budget. VA has also reported $23.61 million in data center-related cost savings, yet does not expect to realize further savings from additional closures. In addition, VA's Chief Information Officer (CIO) certified the use of adequate incremental development for 10 of the department's major IT investments; however, VA has not yet updated its policy and process for CIO certification as GAO recommended. Finally, VA has issued a software licensing policy and has generated an inventory of its software licenses to inform future investment decisions. GAO has made multiple recommendations to VA aimed at improving the department's IT management. VA has generally agreed with the recommendations and begun taking responsive actions.

prompt_front: You are given two reports by the government agency.



Question: Write a one-page summary of the report 2.



Report 2:  The estimate accounts for all possible costs associated with a program, is structured in sufficient detail to ensure that costs are neither omitted nor double counted, and documents all cost-influencing assumptions  Well-documented  Supporting documentation explains the process, sources, and methods used to create the estimate; contains the underlying data used to develop the estimate; and is adequately reviewed and approved by management  Accurate  The estimate is not overly conservative or optimistic, is based on an assessment of the costs most likely to be incurred, and is regularly updated so that it always reflects the program’s current status  Credible  Discusses any limitations of the analysis because of uncertainty or sensitivity surrounding data or assumptions, the estimate’s results are cross-checked, and an independent cost estimate is conducted by a group outside the acquiring organization to determine whether other estimating methods produce similar results  In May 2017, DHS approved GMM’s lifecycle cost estimate of about $251 million for fiscal years 2015 through 2030  We found this initial estimate to be reliable because it fully or substantially addressed all the characteristics associated with a reliable cost estimate  For example, the estimate comprehensively included government and contractor costs, all elements of the program’s work breakdown structure, and all phases of the system lifecycle; and was aligned with the program’s technical documentation at the time the estimate was developed  GMM also fully documented the key assumptions, data sources, estimating methodology, and calculations for the estimate  Further, the program conducted a risk assessment and sensitivity analysis, and DHS conducted an independent assessment of the cost estimate to validate the accuracy and credibility of the cost estimate  However, key assumptions that FEMA made about the program changed soon after DHS approved the cost estimate in May 2017  Thus, the initial cost estimate no longer reflects the current approach for the program  For example, key assumptions about the program that changed include: Change in the technical approach: The initial cost estimate assumed that GMM would implement a software-as-a-service model, meaning that FEMA would rely on a service provider to deliver software applications and the underlying infrastructure to run them  However, in December 2017, the program instead decided to implement an infrastructure-as-a-service model, meaning that FEMA would develop and deploy its own software application and rely on a service provider to deliver and manage the computing infrastructure (e g , servers, software, storage, and network equipment)  According to program officials, this decision was made after learning from the Agile prototypes that the infrastructure-as-a-service model would allow GMM to develop the system in a more flexible environment  Increase in the number of system development personnel: A key factor with Agile development is the number of development teams (each consisting of experts in software development, testing, and cybersecurity) that are operating concurrently and producing separate portions of software functionality  Program officials initially assumed that they would need three to four concurrent Agile development teams, but subsequently realized that they would instead need to expend more resources to achieve GMM’s original completion date  Specifically, program officials now expect they will need to at least double, and potentially triple, the number of concurrent development teams to meet GMM’s original target dates  Significant delays and complexities with data migration: In 2016 and 2017, GMM experienced various technical challenges in its effort to transfer legacy system data to a data staging platform  This data transfer effort needed to be done to standardize the data before eventually migrating the data to GMM  These challenges resulted in significant delays and cost increases  Program officials reported that, by February 2018—at least 9 months later than planned—all legacy data had been transferred to a data staging platform so that FEMA officials could begin analyzing and standardizing the data prior to migrating it into GMM  FEMA officials reported that they anticipated the cost estimate to increase, and for this increase to be high enough to breach the $251 million threshold set in GMM’s May 2017 acquisition program baseline  Thus, consistent with DHS’s acquisition guidance, the program informed the DHS acquisition review board of this anticipated breach  The board declared that the program was in a cost breach status, as of September 12, 2018  As of October 2018, program officials stated that they were in the process of revising the cost estimate to reflect the changes in the program and to incorporate actual costs  In addition, the officials stated that the program was applying a new cost estimating methodology tailored for Agile programs that DHS’s Cost Analysis Division had been developing  In December 2018, program officials stated that they had completed the revised cost estimate but it was still undergoing departmental approval  Establishing an updated cost estimate should help FEMA better understand the expected costs to deliver GMM under the program’s current approach and time frames  The success of an IT program depends, in part, on having an integrated and reliable master schedule that defines when the program’s set of work activities and milestone events are to occur, how long they will take, and how they are related to one another  Among other things, a reliable schedule provides a roadmap for systematic execution of an IT program and the means by which to gauge progress, identify and address potential problems, and promote accountability  GAO’s Schedule Assessment Guide defines leading practices related to the following four characteristics that are vital to having a reliable integrated master schedule  Comprehensive  A comprehensive schedule reflects all activities for both the government and its contractors that are necessary to accomplish a program’s objectives, as defined in the program’s work breakdown structure  The schedule also includes the labor, materials, and overhead needed to do the work and depicts when those resources are needed and when they will be available  It realistically reflects how long each activity will take and allows for discrete progress measurement  Well-constructed  A schedule is well-constructed if all of its activities are logically sequenced with the most straightforward logic possible  Unusual or complicated logic techniques are used judiciously and justified in the schedule documentation  The schedule’s critical path represents a true model of the activities that drive the program’s earliest completion date and total float accurately depicts schedule flexibility  Credible  A schedule that is credible is horizontally traceable—that is, it reflects the order of events necessary to achieve aggregated products or outcomes  It is also vertically traceable—that is, activities in varying levels of the schedule map to one another and key dates presented to management in periodic briefings are consistent with the schedule  Data about risks are used to predict a level of confidence in meeting the program’s completion date  The level of necessary schedule contingency and high-priority risks are identified by conducting a robust schedule risk analysis  Controlled  A schedule is controlled if it is updated regularly by trained schedulers using actual progress and logic to realistically forecast dates for program activities  It is compared to a designated baseline schedule to measure, monitor, and report the program’s progress  The baseline schedule is accompanied by a baseline document that explains the overall approach to the program, defines ground rules and assumptions, and describes the unique features of the schedule  The baseline schedule and current schedule are subject to a configuration management control process  GMM’s schedule was unreliable because it minimally addressed three characteristics—comprehensive, credible, and controlled—and did not address the fourth characteristic of a reliable estimate—well-constructed  One of the most significant issues was that the program’s fast approaching, final delivery date of September 2020 was not informed by a realistic assessment of GMM development activities, and rather was determined by imposing an unsubstantiated delivery date  Table 4 summarizes our assessment of GMM’s schedule  In discussing the reasons for the shortfalls in these practices, program officials stated that they had been uncertain about the level of rigor that should be applied to the GMM schedule, given their use of Agile development  However, leading practices state that program schedules should meet all the scheduling practices, regardless of whether a program is using Agile development  As discussed earlier in this report, GMM has already experienced significant schedule delays  For example, the legacy data migration effort, the AFG pilot, and the Agile development contract have been delayed  Program officials also stated that the delay in awarding and starting the Agile contract has delayed other important activities, such as establishing time frames for transitioning legacy systems  A more robust schedule could have helped FEMA predict the impact of delays on remaining activities and identify which activities appeared most critical so that the program could ensure that any risks in delaying those activities were properly mitigated  In response to our review and findings, program officials recognized the need to continually enhance their schedule practices to improve the management and communication of program activities  As a result, in August 2018, the officials stated that they planned to add a master scheduler to the team to improve the program’s schedule practices and ensure that all of the areas of concern we identified are adequately addressed  In October 2018, the officials reported that they had recently added two master schedulers to GMM  According to the statement of objectives, the Agile contractor is expected to develop an integrated master schedule soon after it begins performance  However, program officials stated that GMM is schedule-driven—due to the Executive Steering Group’s expectation that the solution will be delivered by September 2020  The officials added that, if GMM encounters challenges in meeting this time frame, the program plans to seek additional resources to allow it to meet the 2020 target  GMM’s schedule-driven approach has already led to an increase in estimated costs and resources  For example, as previously mentioned, the program has determined that, to meet its original target dates, GMM needs to at least double, and possibly triple, the number of concurrent Agile development teams  In addition, we have previously reported that schedule pressure on federal IT programs can lead to omissions and skipping of key activities, especially system testing  In August 2018, program officials acknowledged that September 2020 may not be feasible and that the overall completion time frames established in the acquisition program baseline may eventually need to be rebaselined  Without a robust schedule to forecast whether FEMA’s aggressive delivery goal for GMM is realistic to achieve, leadership will be limited in its ability to make informed decisions on what additional increases in cost or reductions in scope might be needed to fully deliver the system  NIST’s risk management framework establishes standards and guidelines for agencies to follow in developing cybersecurity programs  Agencies are expected to use this framework to achieve more secure information and information systems through the implementation of appropriate risk mitigation strategies and by performing activities that ensure that necessary security controls are integrated into agencies’ processes  The framework addresses broad cybersecurity and risk management activities, which include the following: Categorize the system: Programs are to categorize systems by identifying the types of information used, selecting a potential impact level (e g , low, moderate, or high), and assigning a category based on the highest level of impact to the system’s confidentiality, integrity, and availability, if the system was compromised  Programs are also to document a description of the information system and its boundaries and should register the system with appropriate program management offices  System categorization is documented in a system security plan  Select and implement security controls: Programs are to determine protective measures, or security controls, to be implemented based on the system categorization results  These security controls are documented in a system security plan  For example, control areas include access controls, incident response, security assessment and authorization, identification and authentication, and configuration management  Once controls are identified, programs are to determine planned implementation actions for each of the designated controls  These implementation actions are also specified in the system security plan  Assess security controls: Programs are to develop, review, and approve a security assessment plan  The purpose of the security assessment plan approval is to establish the appropriate expectations for the security control assessment  Programs are to also perform a security control assessment by evaluating the security controls in accordance with the procedures defined in the security assessment plan, in order to determine the extent to which the controls were implemented correctly  The output of this process is intended to produce a security assessment report to document the issues, findings, and recommendations  Programs are to conduct initial remediation actions on security controls and reassess those security controls, as appropriate  Obtain an authorization to operate the system: Programs are to obtain security authorization approval in order to operate a system  Resolving weaknesses and vulnerabilities identified during testing is an important step leading up to achieving an authorization to operate  Programs are to establish corrective action plans to address any deficiencies in cybersecurity policies, procedures, and practices  DHS guidance also states that corrective action plans must be developed for every weakness identified during a security control assessment and within a security assessment report  Monitor security controls on an ongoing basis: Programs are to monitor their security controls on an ongoing basis after deployment, including determining the security impact of proposed or actual changes to the information system and assessing the security controls in accordance with a monitoring strategy that determines the frequency of monitoring the controls  For the GMM program’s engineering and test environment, which went live in February 2018, FEMA fully addressed three of the five key cybersecurity practices in NIST’s risk management framework and partially addressed two of the practices  Specifically, FEMA categorized GMM’s environment based on security risk, implemented select security controls, and monitored security controls on an ongoing basis  However, the agency partially addressed the areas of assessing security controls and obtaining an authorization to operate the system  Table 5 provides a summary of the extent to which FEMA addressed NIST’s key cybersecurity practices for GMM’s engineering and test environment  Consistent with NIST’s framework, GMM categorized the security risk of its engineering and test environment and identified it as a moderate- impact environment  A moderate-impact environment is one where the loss of confidentiality, integrity, or availability could be expected to have a serious or adverse effect on organizational operations, organizational assets, or individuals  GMM completed the following steps leading to this categorization: The program documented in its System Security Plan the various types of data and information that the environment will collect, process, and store, such as conducting technology research, building or enhancing technology, and maintaining IT networks  The program established three information types and assigned security levels of low, moderate, or high impact in the areas of confidentiality, availability, and integrity  A low-impact security level was assigned to two information types: (1) conducting technology research and (2) building or enhancing technology; and a moderate- impact security level was assigned to the third information type: maintaining IT networks  The engineering and test environment was categorized as an overall moderate-impact system, based on the highest security impact level assignment  GMM documented a description of the environment, including a diagram depicting the system’s boundaries, which illustrates, among other things, databases and firewalls  GMM properly registered its engineering and test environment with FEMA’s Chief Information Officer, Chief Financial Officer, and acting Chief Information Security Officer  By conducting the security categorization process, GMM has taken steps that should ensure that the appropriate security controls are selected for the program’s engineering and test environment  Consistent with NIST’s framework and the system categorization results, GMM appropriately determined which security controls to implement and planned actions for implementing those controls in its System Security Plan for the engineering and test environment  For example, the program utilized NIST guidance to select standard controls for a system categorized with a moderate-impact security level  These control areas include, for example, access controls, risk assessment, incident response, identification and authentication, and configuration management  Further, the program documented its planned actions to implement each control in its System Security Plan  For example, GMM documented that the program plans to implement its Incident Response Testing control by participating in an agency-wide exercise and unannounced vulnerability scans  As another example, GMM documented that the program plans to implement its Contingency Plan Testing control by testing the contingency plan annually, reviewing the test results, and preparing after action reports  By selecting and planning for the implementation of security controls, GMM has taken steps to mitigate its security risks and protect the confidentiality, integrity, and availability of the information system  Consistent with NIST’s framework, in January 2018, GMM program officials developed a security assessment plan for the engineering and test environment  According to GMM program officials, this plan was reviewed by the security assessment team  However, the security assessment plan lacked essential details  Specifically, while the plan included the general process for evaluating the environment’s security controls, the planned assessment procedures for all 964 security controls were not sufficiently defined  Specifically, GMM program officials copied example assessment procedures from NIST guidance and inserted them into its security assessment documentation for all of its 964 controls, without making further adjustments to explain the steps that should be taken specific to GMM  Table 6 shows an example of a security assessment procedure copied from the NIST guidance that should have been further adjusted for GMM  In addition, the actual assessment procedures that the GMM assessors used to evaluate the security controls were not documented  Instead, the program only documented whether each control passed or failed each test  GMM program officials stated that the planned assessment procedures are based on an agency template that was exported from a DHS compliance tool, and that FEMA security officials have been instructed by the DHS OCIO not to tailor or make any adjustments to the template language  However, the assessment procedures outlined in NIST’s guidance are to serve as a starting point for organizations preparing their program specific assessments  According to NIST, organizations are expected to select and tailor their assessment procedures for each security control from NIST’s list of suggested assessment options (e g , review, analyze, or inspect policies, procedures, and related documentation options)  DHS OCIO officials stated that, consistent with NIST’s guidance, they expect that components will ensure they are in compliance with the minimum standards and will also add details and additional rigor, as appropriate, to tailor the planned security assessment procedures to fit their unique missions or needs  In November 2018, in response to our audit, DHS OCIO officials stated that they were meeting with FEMA OCIO officials to understand why they did not document the planned and actual assessment procedures performed by the assessors for GMM  Until FEMA ensures that detailed planned evaluation methods and actual evaluation procedures specific to GMM are defined, the program risks assessing security controls incorrectly, having controls that do not work as intended, and producing undesirable outcomes with respect to meeting the security requirements  In addition, the security assessment plan was not approved by FEMA’s OCIO before proceeding with the security assessment  Program officials stated that approval was not required for the security assessment plan prior to the development of the security assessment report  However, NIST guidance states that the purpose of the security assessment plan approval is to establish the appropriate expectations for the security control assessment  By not getting the security assessment plan approved by FEMA’s OCIO before security assessment reviews were conducted, GMM risks inconsistencies with the plan and security objectives of the organization  Finally, consistent with NIST guidance, GMM performed a security assessment in December 2017 of the engineering and test environment’s controls, which identified 36 vulnerabilities (23 critical- and high-impact vulnerabilities and 13 medium- and low-impact vulnerabilities)  The program also documented these vulnerabilities and associated findings and recommendations in a security assessment report  GMM conducted initial remediation actions (i e , remediation of vulnerabilities that should be corrected immediately) for 12 of the critical- and high-impact vulnerabilities and a reassessment of those security controls confirmed that they were resolved by January 2018  Remediation of the remaining 11 critical- and high-impact vulnerabilities and 13 medium- and low- impact vulnerabilities were to be addressed by corrective action plans as part of the authorization to operate process, which is discussed in the next section  The authorization to operate GMM’s engineering and test environment was granted on February 5, 2018  Among other things, this decision was based on the important stipulation that the remaining 11 critical- and high- impact vulnerabilities associated with multifactor authentication would be addressed within 45 days, or by March 22, 2018  However, the program did not meet this deadline and, instead, approximately 2 months after this deadline passed, obtained a waiver to remediate these vulnerabilities by May 9, 2019  These vulnerabilities are related to a multifactor authentication capability  Program officials stated that they worked with FEMA OCIO officials to attempt to address these vulnerabilities by the initial deadline, but they were unsuccessful in finding a viable solution  Therefore, GMM program officials developed a waiver at the recommendation of the OCIO to provide additional time to develop a viable solution  However, a multifactor authentication capability is essential to ensuring that users are who they say they are, prior to granting users access to the GMM engineering and test environment, in order to reduce the risk of harmful actors accessing the system  In addition, as of September 2018, the program had not established corrective action plans for the 13 medium- and low-impact vulnerabilities  Program officials stated that they do not typically address low-impact vulnerabilities; however, this is in conflict with DHS guidance that specifies that corrective action plans must be developed for every weakness identified during a security control assessment and within a security assessment report  In response to our audit, in October 2018, GMM program officials developed these remaining corrective action plans  The plans indicated that these vulnerabilities were to be fully addressed by January 2019 and April 2019  While the program eventually took corrective actions in response to our audit by developing the missing plans, the GMM program initially failed to follow DHS’s guidance on preparing corrective actions plans for all security vulnerabilities  Until GMM consistently follows DHS’s guidance, it will be difficult for FEMA to determine the extent to which GMM’s security weaknesses identified during its security control assessments are remediated  Additionally, as we have reported at other agencies, vulnerabilities can be indicators of more significant underlying issues and, thus, without appropriate management attention or prompt remediation, GMM is at risk of unnecessarily exposing the program to potential exploits  Moreover, GMM was required to assess all untested controls by March 7, 2018, or no later than 30 days after the approval of the authorization to operate; however, it did not meet this deadline  Specifically, we found that, by October 2018, FEMA had not fully tested 190 security controls in the GMM engineering and test environment  These controls were related to areas such as security incident handling and allocation of resources required to protect an information system  In response to our findings, in October 2018, GMM program officials reported that they had since fully tested 27 controls and partially tested the remaining 163 controls  Program officials stated that testing of the 163 controls is a shared responsibility between GMM and other parties (e g , the cloud service provider)  They added that GMM had completed its portion of the testing but was in the process of verifying the completion of testing by other parties  Program officials stated that the untested controls were not addressed sooner, in part, because of errors resulting from configuration changes in the program’s compliance tool during a system upgrade, which have now been resolved  Until GMM ensures that all security controls have been tested, it remains at an increased risk of exposing programs to potential exploits  Consistent with the NIST framework, GMM established methods for assessing and monitoring security controls to be conducted after an authorization to operate has been approved  GMM has tailored its cybersecurity policies and practices for monitoring its controls to take into account the frequent and iterative pace with which system functionality is continuously being introduced into the GMM environment  Specifically, the GMM program established a process for assessing security impact changes to the system and conducting reauthorizations to operate within the rapid Agile delivery environment  As part of this process, GMM embedded cybersecurity experts on each Agile development team so that they are involved early and can impact security considerations from the beginning of requirements development through testing and deployment of system functionality  In addition, the process involves important steps for ensuring that the system moves from development to completion, while producing a secure and reliable system  For example, it includes procedures for creating, reviewing, and testing new system functionality  As the new system functionality is integrated with existing system functionality, it is to undergo automated testing and security scans in order to ensure that the integrity of the security of the system has not been compromised  Further, an automated process is to deploy the code if it passes all security scans, code tests, and code quality checks  GMM’s process for conducting a reauthorization to operate within the rapid delivery Agile development environment is to follow FEMA guidance that states that all high-level changes made to a FEMA IT system must receive approval from both a change advisory board and the FEMA Chief Information Officer  The board and FEMA Chief Information Officer are to focus their review and approval on scheduled releases and epics (i e , collections of user stories)  Additionally, the Information System Security Officer is to review each planned user story and, if it is determined that the proposed changes may impact the integrity of the authorization, the Information System Security Officer is to work with the development team to begin the process of updating the system authorization  Finally, GMM uses automated tools to track the frequency in which security controls are assessed and to ensure that required scanning data are received by FEMA for reporting purposes  Program officials stated that, in the absence of department-level and agency-level guidance, they have coordinated with DHS and FEMA OCIO officials to ensure that these officials are in agreement with GMM’s approach to continuous monitoring  By having monitoring control policies and procedures in place, FEMA management is positioned to more effectively prioritize and plan its risk response to current threats and vulnerabilities for the GMM program  Given FEMA’s highly complex grants management environment, with its many stakeholders, IT systems, and internal and external users, implementing leading practices for business process reengineering and IT requirements management is critical for success  FEMA has taken many positive steps, including ensuring executive leadership support for business process reengineering, documenting the agency’s grants management processes and performance improvement goals, defining initial IT requirements for the program, incorporating input from end user stakeholders into the development and implementation process, and taking recent actions to improve its delivery of planned IT requirements  Nevertheless, until the GMM program finalizes plans and time frames for implementing its organizational change management actions, plans and communicates system transition activities, and maintains clear traceability of IT requirements, FEMA will be limited in its ability to provide streamlined grants management processes and effectively deliver a modernized IT system to meet the needs of its large range of users  While GMM’s initial cost estimate was reliable, key assumptions about the program since the initial estimate had changed and, therefore, it no longer reflected the current approach for the program  The forthcoming updated cost schedule is expected to better reflect the current approach  However, the program’s unreliable schedule to fully deliver GMM by September 2020 is aggressive and unrealistic  The delays the program has experienced to date further compound GMM’s schedule issues  Without a robust schedule that has been informed by a realistic assessment of GMM’s development activities, leadership will be limited in its ability to make informed decisions on what additional increases in cost or reductions in scope might be needed to achieve their goals  Further, FEMA’s implementation of cybersecurity practices for GMM in the areas of system categorization, selection and implementation, and monitoring will help the program  However, GMM lacked essential details for evaluating security controls, did not approve the security assessment plan before proceeding with the security assessment, did not follow DHS’s guidance to develop corrective action plans for all security vulnerabilities, and did not fully test all security controls  As a result, the GMM engineering and test environment remains at an increased risk of exploitations  We are making eight recommendations to FEMA: The FEMA Administrator should ensure that the GMM program management office finalizes the organizational change management plan and time frames for implementing change management actions  (Recommendation 1) The FEMA Administrator should ensure that the GMM program management office plans and communicates its detailed transition activities to its affected customers before they transition to GMM and undergo significant changes to their processes  (Recommendation 2) The FEMA Administrator should ensure that the GMM program management office implements its planned changes to its processes for documenting requirements for future increments and ensures it maintains traceability among key IT requirements documents  (Recommendation 3) The FEMA Administrator should ensure that the GMM program management office updates the program schedule to address the leading practices for a reliable schedule identified in this report  (Recommendation 4) The FEMA Administrator should ensure that the FEMA OCIO defines sufficiently detailed planned evaluation methods and actual evaluation methods for assessing security controls  (Recommendation 5) The FEMA Administrator should ensure that the FEMA OCIO approves a security assessment plan before security assessment reviews are conducted  (Recommendation 6) The FEMA Administrator should ensure that the GMM program management office follows DHS guidance on preparing corrective action plans for all security vulnerabilities  (Recommendation 7) The FEMA Administrator should ensure that the GMM program management office fully tests all of its security controls for the system  (Recommendation 8) DHS provided written comments on a draft of this report, which are reprinted in appendix IV  In its comments, the department concurred with all eight of our recommendations and provided estimated completion dates for implementing each of them  For example, with regard to recommendation 4, the department stated that FEMA plans to update the GMM program schedule to address the leading practices for a reliable schedule by April 30, 2019  In addition, for recommendation 7, the department stated that FEMA plans to ensure that corrective action plans are prepared by July 31, 2019, to address all identified security vulnerabilities for GMM  If implemented effectively, the actions that FEMA plans to take in response to the recommendations should address the weaknesses we identified  We also received technical comments from DHS and FEMA officials, which we incorporated, as appropriate  We are sending copies of this report to the Secretary of Homeland Security and interested congressional committees  In addition, the report will be available at no charge on the GAO website at http://www gao gov  If you or your staff have any questions about this report, please contact me at (202) 512-4456 or harriscc@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report  GAO staff who made key contributions to this report are listed in appendix V  Our objectives were to (1) determine the extent to which the Federal Emergency Management Agency (FEMA) is implementing leading practices for reengineering its grants management business processes and incorporating business needs into Grants Management Modernization (GMM) information technology (IT) requirements; (2) assess the reliability of the program’s estimated costs and schedule; and (3) determine the extent to which FEMA is addressing key cybersecurity practices for GMM  To address the first objective, we reviewed GAO’s Business Process Reengineering Assessment Guide and Software Engineering Institute’s Capability Maturity Model for Integration for Development to identify practices associated with business process reengineering and IT requirements management  We then selected six areas that, in our professional judgment, represented foundational practices that were of particular importance to the successful implementation of an IT modernization effort that is using Agile development processes  We also selected the practices that were most relevant based on where GMM was in the system development lifecycle and we discussed the practice areas with FEMA officials  The practices are: Ensuring executive leadership support for process reengineering Assessing the current and target business environment and business Establishing plans for implementing new business processes Establishing clear, prioritized, and traceable IT requirements Tracking progress in delivering IT requirements Incorporating input from end user stakeholders We also reviewed selected chapters of GAO’s draft Agile Assessment Guide (Version 6A), which is intended to establish a consistent framework based on best practices that can be used across the federal government for developing, implementing, managing, and evaluating agencies’ IT investments that rely on Agile methods  To develop this guide, GAO worked closely with Agile experts in the public and private sector; some chapters of the guide are considered more mature because they have been reviewed by the expert panel  We reviewed these chapters to ensure that our expectations for how FEMA should apply the six practices for business process reengineering and IT requirements management are appropriate for an Agile program and are consistent with the draft guidance that is under development  Additionally, since Agile development programs may use different terminology to describe their software development processes, the Agile terms used in this report (e g , increment, sprint, epic, etc ) are specific to the GMM program  We obtained and analyzed FEMA grants management modernization documentation, such as current and target grants management business processes, acquisition program baseline, operational requirements document, concept of operations, requirements analyses workbooks, Grants Management Executive Steering Group artifacts, stakeholder outreach artifacts, Agile increment- and sprint-level planning and development artifacts, and the requirements backlog  We assessed the program documentation against the selected practices to determine the extent to which the agency had implemented them  We then assessed each practice area as: fully implemented—FEMA provided complete evidence that showed it fully implemented the practice area; partially implemented—FEMA provided evidence that showed it partially implemented the practice area; not implemented—FEMA did not provide evidence that showed it implemented any of the practice area  Additionally, we observed Agile increment and sprint development activities at GMM facilities in Washington, D C  We also observed a demonstration of how the program manages its lower level requirements (i e , user stories and epics) and maintains traceability of the requirements using an automated tool at GMM facilities in Washington, D C  We also interviewed FEMA officials, including the GMM Program Executive, GMM Program Manager, GMM Business Transformation Team Lead, and Product Owner regarding their efforts to streamline grants management business processes, collect and incorporate stakeholder input, and manage GMM’s requirements  In addition, we interviewed FEMA officials from four out of 16 grant program offices and two out of 10 regional offices to obtain contextual information and illustrative examples of FEMA’s efforts to reengineer grants management business processes and collect business requirements for GMM  Specifically, We selected the four grant program offices based on a range of grant programs managed, legacy systems used, and the amount of grant funding awarded  We also sought to select a cross section of different characteristics, such as selecting larger grant program offices, as well as smaller offices  In addition, we ensured that our selection included the Assistance to Firefighters Grants (AFG) program office because officials in this office represent the first GMM users and, therefore, are more actively involved with the program’s Agile development practices  Based on these factors, we selected: Public Assistance Division, Individual Assistance Division, AFG, and National Fire Academy  Additionally, the four selected grant program offices are responsible for 16 of the total 45 grant programs and are users of five of the nine primary legacy IT systems  The four selected grant program offices also represent about 68 percent of the total grant funding awarded by FEMA from fiscal years 2005 through 2016  We selected two regional offices based on (1) the largest amount of total FEMA grant funding for fiscal years 2005 through 2016—Region 6 located in Denton, Texas; and (2) the highest percentage of AFG funding compared to the office’s total grant funding awarded from fiscal years 2005 through 2016—Region 5 located in Chicago, Illinois  To assess the reliability of data from the program’s automated IT requirements management tool, we interviewed knowledgeable officials about the quality control procedures used by the program to assure accuracy and completeness of the data  We also compared the data to other relevant program documentation on GMM requirements  We determined that the data used were sufficiently reliable for the purpose of evaluating GMM’s practices for managing IT requirements  For our second objective, to assess the reliability of GMM’s estimated costs and schedule, we reviewed documentation on GMM’s May 2017 lifecycle cost estimate and on the program’s schedule, dated May 2018  To assess the reliability of the May 2017 lifecycle cost estimate, we evaluated documentation supporting the estimate, such as the cost estimating model, the report on GMM’s Cost Estimating Baseline Document and Life Cycle Cost Estimate, and briefings provided to the Department of Homeland Security (DHS) and FEMA management regarding the cost estimate  We assessed the cost estimating methodologies, assumptions, and results against leading practices for developing a comprehensive, accurate, well-documented, and credible cost estimate, identified in GAO’s Cost Estimating and Assessment Guide  We also interviewed program officials responsible for developing and reviewing the cost estimate to understand their methodology, data, and approach for developing the estimate  We found that the cost data were sufficiently reliable  To assess the reliability of the May 2018 GMM program schedule, we evaluated documentation supporting the schedule, such as the integrated master schedule, acquisition program baseline, and Agile artifacts  We assessed the schedule documentation against leading practices for developing a comprehensive, well-constructed, credible, and controlled schedule, identified in GAO’s Schedule Assessment Guide  We also interviewed GMM program officials responsible for developing and managing the program schedule to understand their practices for creating and maintaining the schedule  We noted in our report the instances where the quality of the schedule data impacted the reliability of the program’s schedule  For both the cost estimate and program schedule, we assessed each leading practice as: fully addressed—FEMA provided complete evidence that showed it implemented the entire practice area; substantially addressed—FEMA provided evidence that showed it implemented more than half of the practice area; partially addressed—FEMA provided evidence that showed it implemented about half of the practice area; minimally addressed—FEMA provided evidence that showed it implemented less than half of the practice area; not addressed—FEMA did not provide evidence that showed it implemented any of the practice area  Finally, we provided FEMA with draft versions of our detailed analyses of the GMM cost estimate and schedule  This was done to verify that the information on which we based our findings was complete, accurate, and up-to-date  Regarding our third objective, to determine the extent to which FEMA is addressing key cybersecurity practices for GMM, we reviewed documentation regarding DHS and FEMA cybersecurity policies and guidance, and FEMA’s authorization to operate for the program’s engineering and test environment  We evaluated the documentation against all six cybersecurity practices identified in the National Institute of Standards and Technology’s (NIST) Risk Management Framework  While NIST’s Risk Management Framework identifies six total practices, for reporting purposes, we combined two interrelated practices—selection of security controls and implementation of security controls—into a single practice  The resulting five practices were: categorizing the system based on security risk, selecting and implementing security controls, assessing security controls, obtaining an authorization to operate the system, and monitoring security controls on an ongoing basis  We obtained and analyzed key artifacts supporting the program’s efforts to address these risk management practices, including the program’s System Security Plan, the Security Assessment Plan and Report, Authorization to Operate documentation, and the program’s continuous monitoring documentation  We also interviewed officials from the GMM program office and FEMA’s Office of the Chief Information Officer, such as the GMM Security Engineering Lead, GMM Information System Security Officer, and FEMA’s Acting Chief Information Security Officer, regarding their efforts to assess, document, and review security controls for GMM  We assessed the evidence against the five practices to determine the extent to which the agency had addressed them  We then assessed each practice area as: fully addressed—FEMA provided complete evidence that showed it fully implemented the practice area; partially addressed—FEMA provided evidence that showed it partially implemented the practice area; not addressed—FEMA did not provide evidence that showed it implemented any of the practice area  To assess the reliability of data from the program’s automated security controls management tool, we interviewed knowledgeable officials about the quality control procedures used by the program to assure accuracy and completeness of the data  We also compared the data to other relevant program documentation on GMM security controls for the engineering and test environment  We found that some of the security controls data we examined were sufficiently reliable for the purpose of evaluating FEMA’s cybersecurity practices for GMM, and we noted in our report the instances where the accuracy of the data impacted the program’s ability to address key cybersecurity practices  We conducted this performance audit from December 2017 to April 2019 in accordance with generally accepted government auditing standards  Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives  We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives  The Federal Emergency Management Agency (FEMA) awards many different types of grants to state, local, and tribal governments and nongovernmental entities  These grants are to help communities prevent, prepare for, protect against, mitigate the effects of, respond to, and recover from disasters and terrorist attacks  Agile software development is a type of incremental development that calls for the rapid delivery of software in small, short increments  The use of an incremental approach is consistent with the Office of Management and Budget’s guidance as specified in its information technology (IT) Reform Plan, as well as the legislation commonly referred to as the Federal Information Technology Acquisition Reform Act  Many organizations, especially in the federal government, are accustomed to using a waterfall software development model, which typically consists of long, sequential phases, and differs significantly from the Agile development approach  Agile practices integrate planning, design, development, and testing into an iterative lifecycle to deliver software early and often  Figure 7 provides a depiction of software development using the Agile approach, as compared to a waterfall approach  The frequent iterations of Agile development are intended to effectively measure progress, reduce technical and programmatic risk, and respond to feedback from stakeholders in changes to IT requirements more quickly than traditional methods  Despite these intended benefits, organizations adopting Agile must overcome challenges in making significant changes to how they are accustomed to developing software  The significant differences between Agile and waterfall development impact how IT programs are planned, implemented, and monitored in terms of cost, schedule, and scope  For example, in waterfall development, significant effort is devoted upfront to document detailed plans and all IT requirements for the entire scope of work at the beginning of the program, and cost and schedule can be varied to complete that work  However, for Agile programs the precise details are unknown upfront, so initial planning of cost, scope, and timing would be conducted at a high level, and then supplemented with more specific plans for each iteration  While cost and schedule are set for each iteration, requirements for each iteration (or increment) can be variable as they are learned over time and revised to reflect experiences from completed iterations and to accommodate changing priorities of the end users  The differences in these two software development approaches are shown in figure 8  Looking at figure 8, the benefit provided from using traditional program management practices such as establishing a cost estimate or a robust schedule, is not obvious  However, unlike a theoretical environment, many government programs may not have the autonomy to manage completely flexible scope, as they must deliver certain minimal specifications with the cost and schedule provided  In those cases, it is vital for the team to understand and differentiate the IT requirements that are “must haves” from the “nice to haves” early in the planning effort  This would help facilitate delivery of the “must-haves” requirements first, thereby providing users with the greatest benefits as soon as possible  In addition to the contact named above, the following staff made key contributions to this report: Shannin G  O’Neill (Assistant Director), Jeanne Sung (Analyst in Charge), Andrew Beggs, Rebecca Eyler, Kendrick Johnson, Thomas J  Johnson, Jason Lee, Jennifer Leotta, and Melissa Melvin 
Report 2:  We previously found that results-oriented organizations set performance goals to clearly define desired program outcomes and develop performance measures that are clearly linked to the performance goals  Performance measures should focus on whether a program has achieved measurable standards toward achieving program goals, and allow agencies to monitor and report program accomplishments on an ongoing basis  Our previous work on performance metrics identified 10 attributes of effective performance  Table 7 identifies each key attribute of effective performance measures along with its definition  We evaluated the current performance measures included in the 2018 NSTS, the DHS fiscal year 2019 congressional budget justification, the CSR goals and priorities database, and the CFSR status database related to TSA’s Pipeline Security Branch  We primarily focused on key attributes which could be applied to individual measures  These include clarity, linkage, measurable targets, objectivity, reliability, and baseline and trend data  Our prior work on performance measurement found that all performance measure attributes are not equal and failure to have a particular attribute does not necessarily indicate that there is a weakness in that area or that the measure is not useful; rather, it may indicate an opportunity for further refinement  Based on our evaluation, the TSA-identified measures do not possess attributes that we have identified as being key to successful performance measures  As a result, TSA cannot fully determine the extent to which the Pipeline Security Branch has achieved desired outcomes, including the effectiveness of its efforts to reduce risks to pipelines  Specifically, many of TSA’s measures cover agency goals and mission, but they generally lack clarity and measurable targets, provide significantly overlapping information, and do not include baseline and trend data  Clarity  The pipeline-related measures in the 2018 NSTS are not clear because they do not describe the methodology used to calculate them, and the names and definitions are not clearly described  For example, NSTS goal 1 includes an objective to conduct training of employees responding to terrorist attacks  The desired outcome is to improve the capability of industry employees to respond and recover from terrorist attacks  However, the performance measure is the percentage of critical pipeline systems implementing the TSA Pipeline Security Guidelines  It is not clear if this measure is specific to the sections of the guidelines related to employee training or overall implementation of the guidelines  The CFSR status database measures include the percentage of recommendations implemented by topic, such as “Site Specific Security Measures,” “Signage,” or “Miscellaneous ” However, the database does not specifically define these topics or explain the methodology for calculating the measures  Unclear measures could be confusing and misleading to users  Core program activities  The pipeline-related measures in the 2018 NSTS cover some of the agency’s core program activities, such as conducting security exercises with the pipeline industry and providing intelligence and information products to the industry  However, the NSTS Pipeline Security Plan measures do not specifically include some core program activities, such as updating the TSA Pipeline Security Guidelines or the results of conducting CSRs and CFSRs in order to collect the information necessary for the existing performance measures  The CSR goals and priorities database and the CFSR status database include measures intended to track some of the results of pipeline security reviews, such as the average percentage compliance with the guidelines by fiscal year and the percentage of CFSR recommendations implemented  If core program activities are not covered, there may not be enough information available in those areas to managers and stakeholders  Limited overlap  The pipeline-related measures in the 2018 NSTS do not have limited overlap  As discussed previously, four of the five NSTS measures are based on the percentage of critical pipeline systems implementing TSA’s Pipeline Security Guidelines  The management measure is based on the number of complete pipeline security reviews  The CFSR status database measures are based on the percentage of recommendations implemented overall and by groups  Finally, the CSR goals and priorities database measures are based on the average compliance percentage of companies that had CSRs conducted in fiscal years 2016 and 2017  This is similar to four of the five NSTS measures  Significantly overlapping measures may lead to redundant, costly information that does not add value for TSA management  Linkage  The pipeline-related measures in the 2018 NSTS generally exhibited this key attribute  For example, all of the NSTS measures were arranged by agency strategic goals and risk-based priorities  However, the management measure in DHS’s fiscal year 2019 congressional budget justification and the CFSR status database measures did not specify the TSA goals and priorities to which they were aligned  If measures are not aligned with division and agency- wide goals and mission, the behaviors and incentives created by these measures do not support achieving those goals or mission  Measurable target  TSA’s measures generally did not include measurable targets in the form of a numerical goal and none of the NSTS measures had measurable targets  For example, the NSTS measure under the Security Planning priority, which tracks the percentage of critical pipeline systems implementing TSA’s Pipeline Security Guidelines, does not state what specific percentages would be considered an improvement in industry security plans  However, the management measure did include target numbers of pipeline security reviews by fiscal year  Both the CFSR status database measures and CSR goals and priorities database measures did not include measurable targets  Without measurable targets, TSA cannot tell if performance is meeting expectations  Objectivity  Because the pipeline-related measures in the 2018 NSTS, the CFSR status database, and the CSR goals and priorities database generally lack clarity and measurable targets, TSA cannot ensure its measures are free from bias or manipulation, and therefore, are not objective  If measures are not objective, the results of performance assessments may be systematically overstated or understated  Reliability  Because the pipeline-related measures in the 2018 NSTS, the CFSR status database, and the CSR goals and priorities database generally lack clarity, measurable targets, and baseline and trend data, it is not clear if TSA’s measures produce the same result under similar conditions; therefore, the pipeline-related measures are unreliable  If measures are not reliable, reported performance data may be inconsistent and add uncertainty  Baseline and trend data  TSA’s measures generally did not include baseline and trend data  For example, none of the NSTS measures included past results and compared them to measurable targets  TSA officials were unable to identify measures or goals to assess the extent to which pipeline operators have fully implemented the guidelines or increased pipeline security, but did say developing a feedback mechanism to measure progress in closing vulnerability gaps was important  However, the management measure did include the number of completed pipeline security reviews for each fiscal year from 2014 through 2017, as well as numerical goals  The CFSR status database includes information on CFSRs conducted from May 22, 2012, through June 29, 2017, but the measures are calculated for the entire time period rather than year-by-year  The CSR goals and priorities database measures include percentage compliance with the guidelines for CSRs conducted in fiscal years 2016 and 2017, as well as a combined measure  However, baseline and trend data are not tracked or reported in either database  Collecting, tracking, developing, and reporting baseline and trend data allows agencies to better evaluate progress being made and whether or not goals are being achieved  Pipeline Security Branch officials explained that in addition to the measures reported in the 2018 NSTS Pipeline Security Plan, they primarily rely on measures assessing CSR and CFSR implementation for assessing the value of its pipeline security program  TSA officials reported that they collect and analyze data and information collected from CSRs and CFSRs to, among other things, determine strengths and weaknesses at critical pipeline facilities, areas to target for risk reduction strategies, and pipeline industry implementation of the voluntary Pipeline Security Guidelines  For example, TSA officials reported that they analyzed information from approximately 734 CFSR recommendations that were made during fiscal years 2012 through 2016  They found that pipeline operators had made the strongest improvements in security training, public awareness outreach and law enforcement coordination, and site specific security measures  The most common areas in need of improvement were 24x7 monitoring, frequency of security vulnerability assessments, and proper signage  However, as described above, we found those measures also did not comport with key attributes for successful measures and we report below on reliability concerns for underlying data supporting those measures  In addition, while the Pipeline Security Branch may not rely on the measures included in the 2018 NSTS Pipeline Security Plan and the fiscal year 2019 congressional budget justification, they are important for reporting the status of pipeline security efforts to TSA as a whole and to external stakeholders such as Congress  Taking steps to ensure that the pipeline security program performance measures exhibit key attributes of successful performance measures could allow TSA to better assess the program’s effectiveness at reducing pipeline physical and cybersecurity risks  This could include steps such as modifying its suite of measures so they are clear, have measurable targets, and add baseline and trend data  Further examples include the following: Adding measurable targets consisting of numerical goals could allow TSA to better determine if the pipeline security program is meeting expectations  For example, measurable targets could be added to TSA’s existing measures by developing annual goals for the percentage of recommendations implemented to the CFSR status database and then reporting annual results  To make measures clearer, TSA could verify that each measure has a clearly stated name, definition, and methodology for how the measure is calculated  For example, the NSTS objective for security training mentioned above could have more specific language explaining how the measure is calculated and whether it applies to pipeline operators’ implementation of the training-related portions of the TSA Pipeline Security Guidelines or overall implementation  Finally, adding baseline and trend data could allow TSA to identify, monitor, and report changes in performance and help ensure that performance is viewed in context  For example, the NSTS measures, CFSR status database measures, and CSR goals and priorities database measures could have annual results from prior years  This could help TSA and external stakeholders evaluate the effectiveness of the pipeline security program and whether it is making progress toward its goals  According to TSA officials, the primary means for assessing the effectiveness of the agency’s efforts to reduce pipeline security risks is through conducting pipeline security reviews— Corporate Security Reviews (CSRs) and Critical Facility Security Reviews (CFSRs)  However, TSA has not tracked the status of CSR recommendations for over 5 years and related security review data are not sufficiently reliable  When conducting CSRs and CFSRs, TSA staff makes recommendations to operators, if appropriate  For example, a CSR recommendation might include a suggestion to conduct annual security-related drills and exercises, and a CFSR recommendation might include a suggestion to install barbed wire on the main gate of a pipeline facility  In response to recommendations that we made in our 2010 report, TSA developed three databases to track CSR and CFSR recommendations and their implementation status by pipeline facility, system, operator, and product type  In addition, the agency recently developed a fourth database to collect and analyze information gathered from pipeline operators’ responses to CSR questions  TSA officials reported that they use this database to assess the extent that TSA’s pipeline security program has met NSTS goals and Pipeline Security Branch priorities  TSA officials stated that they use the CSR goals and priorities database for follow-up on recommendations, indications of improvement in pipeline security, and as an input into TSA performance goals and measures, including the performance measures for the 2018 NSTS Pipeline Security Plan  We found several problems with the databases that indicate that the pipeline security program data are not sufficiently reliable and do not provide quality information that is current, complete, and accurate  First, the CSR recommendations database only included information for reviews conducted from November 2010 through February 2013  TSA officials stated that the agency stopped capturing CSR recommendations and status information in 2014  A TSA official stated that one factor was that the pipeline staffing level was one FTE in fiscal year 2014  However, the Pipeline Security Branch did not resume entering CSR recommendation-related information when staffing levels rose to 6 FTEs in the following year and beyond  As a result, TSA is missing over 5 years of data for the recommendations it made to pipeline operators when conducting CSRs  The agency collected some information from CSRs conducted in fiscal years 2016 and 2017 in the separate CSR goals and priorities database  However, this database does not include all of the information that TSA collects when conducting CSRs  Specifically, the CSR goals and priorities database does not state which companies were reviewed, what specific recommendations were made, or the current status of those recommendations, and only records operators’ responses to 79 of the 222 CSR questions  Second, our review identified instances of missing data, inconsistent data entry formats, and data entry errors in the four databases  For example: The CSR recommendations database had missing data in all 13 of the included fields and a data entry error shifted 50 observations into the wrong fields, impacting both the Status Date and Completion Code fields  The CSR goals and priorities database had seven entries with inconsistent data formatting and five of those entries were not taken into account when calculating summary figures  The CFSR recommendations database had missing data in 3 of 9 fields  There was also inconsistent data entry formats in 3 fields  The CFSR status database had missing data in 7 of 29 fields and inconsistent data entry formats in 4 fields  Finally, TSA has not documented its data entry and verification procedures, such as in a data dictionary or user manual, and does not have electronic safeguards for out-of-range or inconsistent entries for any of the databases it uses to track the status of CSR or CFSR recommendations and analyze operator responses to the CSR  TSA Pipeline Security Branch officials told us that they had not documented data entry and verification procedures and did not have electronic safeguards  This was for two reasons  First, the officials stated that the databases are small and maintained in a commercial spreadsheet program that does not allow for electronic safeguards  However, based on our review of the databases, the spreadsheet program does allow for a variety of electronic safeguards  For example, entries can be restricted to only allow selections from a drop-down list or only allow dates to be entered  Second, only a small number of TSA employees enter information into these databases  TSA officials explained that typically one TSA employee is responsible for entering information from pipeline security reviews, and another individual, usually whoever conducted the review, is tasked to verify the accuracy of the data entered  As a result, according to the officials, any errors would be self-evident and caught during these TSA employees’ reviews  Our work has emphasized the importance of quality information for management to make informed decisions and evaluate agencies’ performance in achieving key objectives and addressing risks  The Standards for Internal Control in the Federal Government states that management should use quality information to achieve agency objectives, where “quality” means, among other characteristics, current, complete, and accurate  In addition, DHS’s Information Quality Guidelines state that all DHS component agencies should treat information quality as integral to every step of the development of information, including creation, collection, maintenance, and dissemination  The guidelines also state that agencies should substantiate the quality of the information disseminated through documentation or other appropriate means  Without current, complete, and accurate information, it is difficult for TSA to evaluate the performance of the pipeline security program  Until TSA monitors and records the status of these reviews’ recommendations, it will be hindered in its efforts to determine whether its recommendations are leading to significant reduction in risk  By entering information on CSR recommendations and monitoring and recording their status, developing written documentation of its data entry and verification procedures and electronic safeguards, and improving the quality of its pipeline security program data, TSA could better ensure it has the information necessary to effectively monitor pipeline operators’ progress in improving their security posture, and evaluate its pipeline security program’s effectiveness in reducing security risks to pipelines  A successful pipeline attack could have dire consequences on public health and safety, as well as the U S  economy  Recent coordinated campaigns by environmental activists to disrupt pipeline operations, and the successful attempts by nation-state actors to infiltrate and obtain sensitive information from pipeline operators’ business and operating systems, demonstrate the dynamic and continuous threat to the security of our nation’s pipeline network  To help ensure the safety of our pipelines throughout the nation, it is important for TSA to address weaknesses in the management of its pipeline security program  TSA’s Pipeline Security Branch revised its security guidelines in March 2018 to, among other things, reflect the dynamic threat environment and incorporate NIST’s Cybersecurity Framework cybersecurity principles and practices  However, without a documented process defining how frequently TSA is to review and, if deemed necessary, revise its guidelines, TSA cannot ensure that its guidelines reflect the latest known standards and best practices for physical and cybersecurity, or address the persistent and dynamic security threat environment currently facing the nation’s pipeline system  Further, without clearly defined criteria for determining pipeline facilities’ criticality, TSA cannot ensure that pipeline operators are applying guidance uniformly and that all of the critical facilities across the pipeline sector have been identified; or that their vulnerabilities have been identified and addressed  TSA could improve its ability to conduct pipeline security reviews and the means that it uses to prioritize which pipeline systems to review based on their relative risk ranking  Establishing a strategic workforce plan could help TSA ensure that it has identified the necessary skills, competencies, and staffing allocations that the Pipeline Security Branch needs to carry out its responsibilities, including conducting security reviews of critical pipeline companies and facilities, as well as their cybersecurity posture  Better considering threat, vulnerability, and consequence elements in its risk assessment and incorporating an independent, external peer review in its process would provide more assurance that the Pipeline Security Branch ranks relative risk among pipeline systems using comprehensive and accurate data and methods  We are making 10 recommendations to TSA: The TSA Administrator should direct the Security Policy and Industry Engagement’s Surface Division to implement a documented process for reviewing, and if deemed necessary, for revising TSA’s Pipeline Security Guidelines at regular defined intervals  (Recommendation 1) The TSA Administrator should direct the Security Policy and Industry Engagement’s Surface Division to clarify TSA’s Pipeline Security Guidelines by defining key terms within its criteria for determining critical facilities  (Recommendation 2) The TSA Administrator should develop a strategic workforce plan for its Security Policy and Industry Engagement’s Surface Division, which could include determining the number of personnel necessary to meet the goals set for its Pipeline Security Branch, as well as the knowledge, skills, and abilities, including cybersecurity, that are needed to effectively conduct CSRs and CFSRs  (Recommendation 3) The TSA Administrator should direct the Security Policy and Industry Engagement’s Surface Division to update the Pipeline Relative Risk Ranking Tool to include up-to-date data to ensure it reflects industry conditions, including throughput and threat data  (Recommendation 4) The TSA Administrator should direct the Security Policy and Industry Engagement’s Surface Division to fully document the data sources, underlying assumptions and judgments that form the basis of the Pipeline Relative Risk Ranking Tool, including sources of uncertainty and any implications for interpreting the results from the assessment  (Recommendation 5) The TSA Administrator should direct the Security Policy and Industry Engagement’s Surface Division to identify or develop other data sources relevant to threat, vulnerability, and consequence consistent with the NIPP and DHS critical infrastructure risk mitigation priorities and incorporate that data into the Pipeline Relative Risk Ranking Tool to assess relative risk of critical pipeline systems, which could include data on prior attacks, natural hazards, feedback data on pipeline system performance, physical pipeline condition, and cross-sector interdependencies  (Recommendation 6) The TSA Administrator should direct the Security Policy and Industry Engagement’s Surface Division to take steps to coordinate an independent, external peer review of its Pipeline Relative Risk Ranking Tool, after the Pipeline Security Branch completes enhancements to its risk assessment approach  (Recommendation 7) The TSA Administrator should direct the Security Policy and Industry Engagement’s Surface Division to ensure that it has a suite of performance measures which exhibit key attributes of successful performance measures, including measurable targets, clarity, and baseline and trend data  (Recommendation 8) The TSA Administrator should direct the Security Policy and Industry Engagement’s Surface Division to take steps to enter information on CSR recommendations and monitor and record their status  (Recommendation 9) The TSA Administrator should direct the Security Policy and Industry Engagement’s Surface Division to improve the quality of its pipeline security program data by developing written documentation of its data entry and verification procedures, implementing standardized data entry formats, and correcting existing data entry errors  (Recommendation 10) We provided a draft of this report to DHS, DOE, DOT, and FERC  DHS provided written comments which are reproduced in appendix III  In its comments, DHS concurred with our recommendations and described actions planned to address them  DHS, DOE, DOT, FERC, also provided technical comments, which we incorporated as appropriate  We also provided draft excerpts of this product to the American Petroleum Institute (API), the Association of Oil Pipe Lines, the American Gas Association (AGA), the Interstate Natural Gas Association of America (INGAA), the American Public Gas Association, and the selected pipeline operators that we interviewed  For those who provided technical comments, we incorporated them as appropriate  With regard to our first recommendation, that TSA implement a documented process for reviewing, and if deemed necessary, for revising its Pipeline Security Guidelines at regular defined intervals, DHS stated that TSA will implement a documented process for reviewing and revising its Pipeline Security Guidelines at regular defined intervals, as appropriate  DHS estimated that this effort would be completed by March 31, 2019  This action, if fully implemented, should address the intent of the recommendation  With regard to our second recommendation, that TSA clarify its Pipeline Security Guidelines by defining key terms within its criteria for determining critical facilities, DHS stated that TSA will clarify its Pipeline Security Guidelines by defining key terms within its criteria for determining critical facilities  DHS estimated that this effort would be completed by May 31, 2019  This action, if fully implemented, should address the intent of the recommendation  With regard to our third recommendation, that TSA develop a strategic workforce plan for its Security Policy and Industry Engagement's Surface Division, DHS stated that TSA will develop a strategic workforce plan for the division, which includes determining the number of personnel necessary to meet the goals set for the Pipeline Security Branch, as well as the knowledge, skills, and abilities, including cybersecurity, that are needed to effectively conduct CSRs and CFSRs  DHS estimated that this effort would be completed by June 30, 2019  This action, if fully implemented, should address the intent of the recommendation  With regard to our fourth recommendation, that TSA update the Pipeline Relative Risk Ranking Tool to include up-to-date data in order to ensure it reflects industry conditions, including throughput and threat data, DHS stated that TSA will update the Pipeline Relative Risk Ranking Tool to include up-to-date data in order to ensure it reflects industry conditions, including throughput and threat data  DHS estimated that this effort would be completed by February 28, 2019  This action, if fully implemented, should address the intent of the recommendation  With regard to our fifth recommendation, that TSA fully document the data sources, underlying assumptions, and judgements that form the basis of the Pipeline Relative Risk Ranking Tool, including sources of uncertainty and any implications for interpreting the results from the assessment, DHS stated that TSA will fully document the data sources, underlying assumptions, and judgements that form the basis of the Pipeline Relative Risk Ranking Tool  According to DHS, this will include sources of uncertainty and any implications for interpreting the results from the assessment  DHS estimated that this effort would be completed by February 28, 2019  This action, if fully implemented, should address the intent of the recommendation  With regard to our sixth recommendation, that TSA identify or develop other data sources relevant to threat, vulnerability, and consequence consistent with the NIPP and DHS critical infrastructure risk mitigation priorities and incorporate that data into the Pipeline Relative Risk Ranking Tool to assess relative risk of critical pipeline systems, DHS stated that TSA will identify and/or develop other sources relevant to threat, vulnerability, and consequence consistent with the NIPP and DHS critical infrastructure risk mitigation priorities  DHS also stated that TSA will incorporate that data into the Pipeline Risk Ranking Tool to assess relative risk of critical pipeline systems, which could include data on prior attacks, natural hazards, feedback data on pipeline system performance, physical pipeline condition, and cross-sector interdependencies  DHS estimated that this effort would be completed by June 30, 2019  This action, if fully implemented, should address the intent of the recommendation  With regard to our seventh recommendation, that TSA take steps to coordinate an independent, external peer review of its Pipeline Relative Risk Ranking Tool, after the Pipeline Security Branch completes enhancements to its risk assessment approach, DHS stated that, after completing enhancements to its risk assessment approach, TSA will take steps to coordinate an independent, external peer review of its Pipeline Relative Risk Ranking Tool  DHS estimated that this effort would be completed by November 30, 2019  This action, if fully implemented, should address the intent of the recommendation  With regard to our eighth recommendation, that TSA ensure that the Security Policy and Industry Engagement's Surface Division has a suite of performance measures which exhibit key attributes of successful performance measures, including measurable targets, clarity, baseline, and trend data, DHS stated that TSA’s Surface Division’s Pipeline Section will develop both physical and cyber security performance measures, in consultation with pipeline stakeholders, to ensure that it has a suite of performance measures which exhibit key attributes of successful performance measures, including measurable targets, clarity, baseline, and trend data  DHS estimated that this effort would be completed by November 30, 2019  This action, if fully implemented, should address the intent of the recommendation  With regard to our ninth recommendation, that TSA take steps to enter information on CSR recommendations and monitor and record their status, DHS stated that TSA will enter information on CSR recommendations and monitor and record their status  DHS estimated that this effort would be completed by October 31, 2019  This action, if fully implemented, should address the intent of the recommendation  With regard to our tenth recommendation, that TSA take steps to improve the quality of its pipeline security program data by developing written documentation of its data entry and verification procedures, implementing standardized data entry formats, and correcting existing data entry errors, DHS stated that TSA will develop written documentation of its data entry and verification procedures, implementing standardized data entry formats, and correcting existing data entry errors  DHS estimated that this effort would be completed by July 31, 2019  This action, if fully implemented, should address the intent of the recommendation  As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until one day from the report date  At that time, we will send copies to the appropriate congressional committees; the Secretaries of Energy, Homeland Security, and Transportation; the Executive Director of the Federal Energy Regulatory Committee; and other interested parties  In addition, the report is available at no charge on the GAO website at http://www gao gov  If you or your staff have any questions about this report, please contact Chris Currie at (404) 679-1875 or curriec@gao gov, and Nick Marinos at (202) 512-9342 or marinosn@gao gov  Key contributors to this report are listed in appendix IV  This appendix lists security guidance and guidance-related tools that the pipeline operators and industry association officials we interviewed identified as adopted or available in order to secure their physical and cyber operations  This list should not be considered to include all physical and cybersecurity guidance that may be available or used by all pipeline operators nor do all operators use all guidance listed  The Transportation Security Administration’s (TSA) Pipeline Security Branch developed the Pipeline Relative Risk Ranking Tool (risk assessment) in 2007  The risk assessment calculates threat, vulnerability, and consequence on variables such as the amount of throughput in the pipeline system (consequence input)  Pipeline Security Branch officials told us that they use the pipeline risk assessment to rank relative risk of the top 100 critical pipeline systems, and the standard operating procedures for conducting Corporate Security Reviews (CSR) state the results of the risk ranking are the primary factor considered when prioritizing CSRs of pipeline companies  However, we identified several factors that likely limit the usefulness of the current assessment in calculating threat, vulnerability, and consequence to allow the Pipeline Security Branch to effectively prioritize reviews of pipeline systems  For example, because the risk assessment has not changed since 2014, information on threat may be outdated  Additionally, sources of data and underlying assumption and judgments regarding certain threat and vulnerability inputs to the assessment are not fully documented  For example, threats to cybersecurity are not specifically accounted for in the description of the risk assessment methodology, making it unclear if cybersecurity is part of the assessment’s threat factor  Further, the risk assessment does not include information that is consistent with the National Infrastructure Protection Plan (NIPP) and other Department of Homeland Security (DHS) priorities for critical infrastructure risk mitigation, such as information on natural hazards and the ability to measure risk reduction (feedback data)  According to Pipeline Security Branch officials, the risk ranking assessment is not intended to be a fully developed risk model detailing all pipeline factors influencing risk  Rather, officials said they are primarily interested in assessing risk data that impacts security  However, because TSA’s Pipeline Security Program is designed to enhance the security preparedness of the pipeline systems, incorporating additional factors that enhance security into their risk calculation of the most critical pipeline systems would better align their efforts with Presidential Policy Directive 21 (PPD-21)  For example, PPD-21 calls for agencies to integrate and analyze information to prioritize assets and manage risks to critical infrastructure, as well as anticipate interdependencies and cascading impacts  Below we present the various shortfalls in the risk assessment—outdated data, limited description of sources and methodology, and opportunities to better align with the NIPP and other DHS priorities for critical infrastructure risk mitigation—in the context of the components that comprise a risk assessment: threat, vulnerability, and consequence  Whereas in 2010 we made recommendations to improve the consequence component in the pipeline relative risk ranking tool, we have currently identified shortfalls that cut across all risk components: threat, vulnerability, and consequence  We identified several shortfalls in the pipeline risk assessment’s calculation of threat  First, while the risk assessment assesses consequence and vulnerability by pipeline system through use of multiple variables, it currently ranks threat for pipeline systems equally  Second, the evolving nature of threats to pipelines may not be reflected, since the risk assessment was last updated in 2014  Third, the threat calculation does not take into account natural hazards  Pipeline Security Branch officials said they currently rank threat equally across pipeline systems because they do not have granular enough threat information to distinguish threat by pipeline  However, ranking threat equally effectively has no effect on the risk calculation for pipeline systems  Further, this judgment is not documented in the risk assessment’s methodology  According to the NIPP, a risk assessment’s methodology must clearly document what information is used and how it is synthesized to generate a risk estimate, including any assumptions and judgments  Additionally, our analysis of the pipeline risk assessment found that it includes at least one field that TSA could use to differentiate threat by pipeline  Specifically, the risk assessment includes a field that accounts for whether a pipeline experienced a previous security threat (including failed attacks), and information provided by Pipeline Security Branch suggests some pipeline systems have experienced such threats  However, the Pipeline Security Branch did not capture these events in the risk assessment’s calculation, which Pipeline Security Branch officials said should be part of the threat calculation, but could not account for why they were not calculated for the systems in the risk assessment  These officials also clarified that incidents such as suspicious photography or vandalism do not constitute an attack to be accounted for in the threat calculation  Documenting such assumptions, judgments, or decisions to exclude information could provide increased transparency to those expected to interpret or use the results  Pipeline Security Branch officials also said that they ranked threat equally because TSA Intelligence and Analysis data show that threats to the oil and natural gas sector have been historically low, and Intelligence and Analysis does not conduct specific threat analysis against individual pipeline systems  However, the Pipeline Security Branch has not updated the risk assessment since June 2014; therefore, the threat information it used to determine threat calculations—and decide to rank threat equally—may be outdated and not reflect the threats to the industry that have emerged in recent years  In fact, pipeline operators we interviewed indicated that the types of threats that concern pipeline operators have evolved  For example, 5 of the 10 operators we interviewed indicated that environmental activists were an increased threat to the pipeline industry because they use sabotage techniques, such as valve turning and cutting in service pipelines with blow torches, against pipelines  Additionally, 6 of 10 pipeline operators we interviewed said cyber attacks from nation-state actors were a primary threat to their industry  Further, when TSA issued its revised Pipeline Security Guidelines in March 2018, it stated that its revisions to the guidelines were made to reflect the ever-changing threat environment in both the physical and cybersecurity realms  However, threats to cybersecurity are not specifically accounted for in the description of the risk assessment methodology  Recent Pipeline Modal and Cyber Modal Threat Assessments include cyber threats to the pipeline industry, but the description of the pipeline risk assessment’s methodology does not specify what types of threat assessments (sources) are used to calculate its threat score  To better align with the guidance in the NIPP for documenting sources of information when conducting risk assessments, the Pipeline Security Branch should document the information used  Keeping the risk assessment updated with current information, as well as documenting those data sources, could help the Pipeline Security Branch ensure it is using its limited resources to review the pipeline systems with greater risk  Natural Hazard Threats to Pipelines The Transportation Systems Sector, of which pipelines are a part, is critical to the Pacific Northwest, but also at risk from natural hazards, like earthquakes  For example, according to the Department of Homeland Security, an earthquake in the Puget Sound region—which relies on the transportation of crude oil from Alaska—could cripple the ports of Seattle and Tacoma, as well as the Olympic and Williams Pipelines greatly impacting the Pacific Northwest Economic Region  Hurricanes are the most frequent disruptive natural hazard for the oil and natural gas subsector and can cause the shutdown of facilities in an area, even when the facilities themselves are not directly affected by the storms  For example, according to the U S  Energy Information Administration, the flow of petroleum into the New York area via pipeline from the Gulf Coast relies on the ability to move it through major terminals  In August 2017, Hurricane Harvey caused major disruptions to crude oil and petroleum product supply chains, including those to New York Harbor from Houston, Texas via the Colonial Pipeline  Due to the hurricane, decreased supplies of petroleum products available for the pipeline in Houston forced Colonial Pipeline to limit operations temporarily  Finally, another shortfall in the current pipeline risk assessment methodology is that it does not account for natural hazards in its threat calculation, even though DHS’s definition of threat includes natural hazards, and security and resilience of critical infrastructure are often presented in the context of natural hazards  According to the NIPP, threat is a natural or manmade occurrence, individual, entity, or action that has or indicates the potential to harm life, information, operations, the environment, and/or property  As such, along with terrorism, criminal activity and cybersecurity, natural disasters are a key element of DHS’s critical infrastructure security and resilience mission  According to Pipeline Security Branch officials, there is not sufficient historical data available that would indicate a significant impact from natural disasters on specific pipeline systems  However, we identified possible sources of data for the Pipeline Security Branch to consider  For example, a 2016 RAND Corporation study examined national infrastructure systems’ exposure to natural hazards, including pipelines  Additionally, the Federal Emergency Management Agency (FEMA) has collaborated with stakeholders to develop the National Risk Index to, among other things, establish a baseline of natural hazards risk for the United States While there may not be historical data of natural hazard impact for every pipeline system, consulting other sources or experts could provide regional data or analysis to build a more comprehensive threat picture to help distinguish threats by pipeline system  According to the NIPP, hazard assessments should rely not only on historical information, but also future predictions about natural hazards to assess the likelihood or frequency of various hazards  We also identified multiple shortfalls in the vulnerability factors used in the risk assessment methodology, such as the potential uncertainty of the number of critical facilities and incorporating a feedback mechanism to calculate overall risk reduction  Other considerations for vulnerability calculations include physical condition of the pipeline system, cybersecurity activities, and interdependencies among sectors  The number of critical facilities a pipeline system has identified is used as an input for its vulnerability calculation in the Pipeline Security Branch’s risk assessment methodology  As discussed earlier, we identified deficiencies in TSA’s criteria for identifying critical facilities, and found that well-defined criteria and consistent application of the criteria for identifying critical facilities could improve the results of the Pipeline Security Branch’s risk assessment  Nevertheless, communicating in the risk assessment the uncertainty that may be inherent in this self-reported information would better align the risk assessment with the NIPP  Measuring Effectiveness in a Voluntary Environment According to the National Infrastructure Protection Plan, the use of performance metrics is an important step in the critical infrastructure risk management process to enable assessment of improvements in critical infrastructure security and resilience  The metrics provide a basis for the critical infrastructure community to establish accountability, document actual performance, promote effective management, and provide a feedback mechanism to inform decision making  By using metrics to evaluate the effectiveness of voluntary partnership efforts to achieve national and sector priorities, critical infrastructure partners can adjust and adapt their security and resilience approaches to account for progress achieved, as well as changes in the threat and other relevant environments  Metrics are used to focus attention on areas of security and resilience that warrant additional resources or other changes through an analysis of challenges and priorities at the national, sector, and owner/operator levels  Metrics also serve as a feedback mechanism for other aspects of the critical infrastructure risk management approach  Another shortfall in the risk assessment is its inability to reliably measure the progress a pipeline system made in addressing vulnerability gaps between security reviews  The current risk assessment includes a CSR score as part of its vulnerability calculation, which was developed in part in response to our 2010 recommendation to use more reliable data to measure a pipeline system’s vulnerability gap  However, during our review, Pipeline Security Branch officials said they plan to remove pipeline companies’ CSR scores from the risk assessment calculations, because they and industry partners do not have confidence that the score appropriately measures a pipeline system’s vulnerability  For example, Pipeline Security Branch officials explained that pipeline companies consider security factors differently, which can lead to variation in implementing risk reduction activities and by extension lead to different CSR scores  However, removing the CSR score eliminates the only feedback mechanism in the risk assessment from a pipeline company’s actual security review conducted by the Pipeline Security Branch  The NIPP and DHS’s Risk Management fundamentals emphasize the important role that such feedback mechanisms play in risk management  Officials from the Pipeline Security Branch agree on the importance of a feedback mechanism tying results of reviews to a revised vulnerability metric, but said they need a better measure than the current CSR score which is unreliable for comparative and analytic purposes  Developing a feedback mechanism based on implementation of TSA’s Pipeline Security Guidelines could be an important input to the risk assessment’s vulnerability calculation  This information would also inform the amount of risk pipeline companies are reducing by implementing the guidelines and could be used to inform overall risk reduction  The physical and cyber environments in which the pipeline sector operates also present vulnerabilities not accounted for in the pipeline risk assessment  In recent years, DHS has listed the potential for catastrophic losses to dramatically increase the overall risk associated with failing infrastructure and highlighted risks due to climate change and natural hazards to pipelines  For example, DHS reported extreme temperatures—such as higher and lower temperatures over prolonged periods of time—increase vulnerability to the critical infrastructure by causing elements to break and cease to function  Pipelines that freeze and then rupture can affect the energy and transportation systems sectors  As noted above, according to the NIPP, a natural or man-made occurrence or action with the potential to harm life is considered a threat, whereas vulnerability is defined as a physical feature or operational attribute that renders an entity open to exploitation or susceptible to a given threat or hazard  While pipeline physical condition is typically thought of in context of safety, pipeline condition or location (such as above or below ground) could touch upon pipeline security as it relates to system vulnerability  For example, a pipeline system or segment of a system with a compromised physical condition due to corrosion or age could affect the system’s vulnerability to threats and affect its ability to recover from such threats by potentially increasing the time a system is offline  According to the Transportation Systems Sector-Specific Plan, vulnerabilities to damage in aging transportation infrastructure—of which pipelines are a part—are projected to increase with the continued effects of climate change  Further, according to TSA’s Pipeline Security and Incident Recovery Protocol Plan, pipeline integrity efforts—including the design, construction, operation, and maintenance of pipelines—are important to pipeline security because well-maintained, safe pipelines are more likely to tolerate a physical attack  The Pipeline Security Branch already collects information from the Pipeline and Hazardous Materials Safety Administration (PHMSA) for its risk assessment, specifically information on High Consequence Area and High Threat Urban Area mileage  By considering additional information PHMSA collects on pipeline integrity, the Pipeline Security Branch could also use the information to help pipeline operators identify security measures to help reduce the consequences related to the comparatively higher vulnerability of an aging or compromised system  This would align with the Pipeline Security Branch’s efforts to improve security preparedness of pipeline systems and could better inform its vulnerability calculations for relative risk ranking of pipeline systems  Capturing cybersecurity in the risk assessment is also an area for improvement  Pipeline Security Branch officials told us they consulted with the National Cybersecurity and Communications Integration Center to revise TSA’s Pipeline Security Guidelines to align with the National Institute of Standards and Technology (NIST) Cybersecurity Framework and that absent data specific to pipelines on their cybersecurity vulnerabilities, they are unable to include a pipelines’ vulnerability to cyber attack in the risk assessment  However, the Pipeline Security Branch recently updated the security review questions asked of pipeline operators during corporate and critical facility reviews based on the recently updated Pipeline Security Guidelines  Using these updated questions related to companies’ cybersecurity posture, the Pipeline Security Branch could collect additional information on cybersecurity vulnerabilities which could inform the risk assessment  This could be an element of the feedback mechanism described above and emphasized in the NIPP  Additionally, NIST identified several supply chain vulnerabilities associated with cybersecurity, which are not currently accounted for in TSA’s Pipeline Security Guidelines  As pipeline operators implement increasing levels of network technologies to control their systems, the Pipeline Security Branch may not be fully accounting for pipeline systems’ cybersecurity posture by not including the cybersecurity-related vulnerabilities in its risk assessment inputs  Finally, we identified shortfalls in cross-sector interdependencies, which could affect vulnerability calculations  According to the NIPP, understanding and addressing risks from cross-sector dependencies and interdependencies is essential to enhancing critical infrastructure security and resilience  The Pipeline Security Branch’s pipeline risk assessment currently considers the effects of a pipeline system’s ability to service assets such as major airports, the electric grid, and military bases  However, consequence is calculated on the loss or disruption of the pipeline system to these other assets and does not capture the dependency of the pipeline system on other energy sources, such as electricity  Weather events such as Gulf of Mexico hurricanes and Superstorm Sandy highlighted the interdependencies between the pipeline and electrical sectors  Specifically, according to a 2015 DHS annual report on critical infrastructure, power failures during Superstorm Sandy in 2012 closed major pipelines for 4 days, reducing regional oil supplies by 35 to 40 percent  The report goes on to say that the interconnected nature of infrastructure systems can lead to cascading impacts and are increasing in frequency  Pipeline Security Branch officials are considering cross-sector interdependencies and said they discuss these factors with operators as they relate to system resiliency  Considering interdependencies of sectors in both directions—such as calculating the likelihood that an input like electricity could fail and cause disruptions to critical pipelines—could improve the calculations in the pipeline risk assessment  As previously discussed, the Pipeline Security Branch last calculated relative risk among the top 100 pipeline systems in 2014  When doing so, it used pipeline systems’ throughput data from 2010 to assess relative risk  According to Pipeline Security Branch officials, the amount of throughput in pipeline systems does not change substantially year to year  However, Standards for Internal Control in the Federal Government calls for management to use quality information to achieve the entity’s objectives, including using relevant data from reliable sources obtained in a timely manner  The Pipeline Security Branch uses throughput data as a consequence factor in the risk assessment to determine a pipeline system’s relative risk score  Throughput changes could affect relative risk ranking and the Pipeline Security Branch’s ability to accurately prioritize reviews based on relative risk  Chris P  Currie at (404) 679-1875 or curriec@gao gov Nick Marinos at (202) 512-9342 or marinosn@gao gov  In addition to the contacts named above, Ben Atwater, Assistant Director; Michael W  Gilmore, Assistant Director; and Michael C  Lenington, Analyst-in-Charge, managed this assignment  Chuck Bausell, David Blanding, Dominick Dale, Eric Hauswirth, Kenneth A  Johnson, Steve Komadina, Susanna Kuebler, Thomas Lombardi, David Plocher, and Janay Sam made significant contributions to this report 

ground_truth: FEMA, a component of DHS, annually awards billions of dollars in grants to help communities prepare for, mitigate the effects of, and recover from major disasters. However, FEMA's complex IT environment supporting grants management consists of many disparate systems. In 2008, the agency attempted to modernize these systems but experienced significant challenges. In 2015, FEMA initiated a new endeavor (the GMM program) aimed at streamlining and modernizing the grants management IT environment. GAO was asked to review the GMM program. GAO's objectives were to (1) determine the extent to which FEMA is implementing leading practices for reengineering its grants management processes and incorporating needs into IT requirements; (2) assess the reliability of the program's estimated costs and schedule; and (3) determine the extent to which FEMA is addressing key cybersecurity practices. GAO compared program documentation to leading practices for process reengineering and requirements management, cost and schedule estimation, and cybersecurity risk management, as established by the Software Engineering Institute, National Institute of Standards and Technology, and GAO. Of six important leading practices for effective business process reengineering and information technology (IT) requirements management, the Federal Emergency Management Agency (FEMA) fully implemented four and partially implemented two for the Grants Management Modernization (GMM) program (see table). Specifically, FEMA ensured senior leadership commitment, took steps to assess its business environment and performance goals, took recent actions to track progress in delivering IT requirements, and incorporated input from end user stakeholders. However, FEMA has not yet fully established plans for implementing new business processes or established complete traceability of IT requirements. Until FEMA fully implements the remaining two practices, it risks delivering an IT solution that does not fully modernize FEMA's grants management systems. While GMM's initial May 2017 cost estimate of about $251 million was generally consistent with leading practices for a reliable, high-quality estimate, it no longer reflects current assumptions about the program. FEMA officials stated in December 2018 that they had completed a revised cost estimate, but it was undergoing departmental approval. GMM's program schedule was inconsistent with leading practices; of particular concern was that the program's final delivery date of September 2020 was not informed by a realistic assessment of GMM development activities, and rather was determined by imposing an unsubstantiated delivery date. Developing sound cost and schedule estimates is necessary to ensure that FEMA has a clear understanding of program risks. Of five key cybersecurity practices, FEMA fully addressed three and partially addressed two for GMM. Specifically, it categorized GMM's system based on security risk, selected and implemented security controls, and monitored security controls on an ongoing basis. However, the program had not initially established corrective action plans for 13 medium- and low-risk vulnerabilities. This conflicts with the Department of Homeland Security's (DHS) guidance that specifies that corrective action plans must be developed for every weakness identified. Until FEMA, among other things, ensures that the program consistently follows the department's guidance on preparing corrective action plans for all security vulnerabilities, GMM's system will remain at increased risk of exploits. GAO is making eight recommendations to FEMA to implement leading practices related to reengineering processes, managing requirements, scheduling, and implementing cybersecurity. DHS concurred with all recommendations and provided estimated dates for implementing each of them.