prompt_front: You are given two reports by the government agency.



Question: Write a one-page summary of the report 1.

Report 1: Since 2001, the United States has made a commitment to building Afghanistan’s security and governance in order to prevent the country from once again becoming a sanctuary for terrorists  To achieve its security objectives, the United States currently has two missions in Afghanistan: a counterterrorism mission that it leads and the NATO-led Resolute Support train, advise, and assist mission, which it participates in with other coalition nations  The objective of Resolute Support, according to DOD reporting, is to establish self-sustaining Afghan security ministries and forces that work together to maintain security in Afghanistan  The United States is conducting these missions within a challenging security environment that has deteriorated since the January 2015 transition to Afghan-led security  The United Nations reported nearly 24,000 security incidents in Afghanistan in 2017—the most ever recorded—and, despite a slight decrease in the overall number of security incidents in early 2018, the United Nations noted significant security challenges, including a spike in high-casualty attacks in urban areas and coordinated attacks by the insurgency on ANDSF checkpoints  DOD provides both personnel and funding to support its efforts in Afghanistan  DOD documents indicate that the United States contributes more troops to Resolute Support than any other coalition nation  As of May 2018, the United States was contributing 54 percent of Resolute Support military personnel, according to DOD reporting  Of the approximately 14,000 U S  military personnel in Afghanistan as of June 2018, about 8,500 were assigned to Resolute Support to train, advise, and assist the ANDSF, according to DOD reporting  For fiscal year 2018, Congress appropriated about $4 67 billion for the Afghanistan Security Forces Fund—the primary mechanism of U S  financial support for manning, training, and equipping the ANDSF  Other international donors provided about $800 million, and the Afghan government committed to providing about $500 million, according to DOD reporting  Under Resolute Support and the International Security Assistance Force mission that preceded it, CSTC-A is the DOD organization responsible for (1) overseeing efforts to equip and train the ANA and ANP; (2) validating requirements, including equipment requirements; (3) validating existing supply levels; (4) submitting requests to DOD components to contract for procurement of materiel for the ANDSF; and (5) ensuring that the Afghan government appropriately uses and accounts for U S  funds provided as direct contributions from the Afghanistan Security Forces Fund  OSD-P is responsible for developing policy on and conducting oversight of the bilateral security relationship with Afghanistan focused on efforts to develop the Afghan security ministries and their forces  In August 2017, we reported that the United States had spent almost $18 billion on equipment and transportation for the ANDSF from fiscal years 2005 through April 2017, representing the second-largest expenditure category from the Afghanistan Security Forces Fund  In that report, we identified six types of key equipment the United States funded for the ANDSF in fiscal years 2003 through 2016, including approximately: 600,000 weapons, such as rifles, machine guns, grenade launchers, shotguns, and pistols; 163,000 tactical and nontactical radios, such as handheld radios and 76,000 vehicles, such as Humvees, trucks, recovery vehicles, and mine resistant ambush protected vehicles; 30,000 equipment items for detecting and disposing of explosives, such as bomb disposal robots and mine detectors; 16,000 equipment items for intelligence, surveillance, and reconnaissance, such as unmanned surveillance drones and night vision devices; and 208 aircraft, such as helicopters, light attack aircraft, and cargo airplanes  The Ministry of Defense oversees the ANA, and the Ministry of the Interior oversees the ANP  According to DOD reporting, the authorized force level for the ANDSF, excluding civilians, as of June 2018 was 352,000: 227,374 for the Ministry of Defense and 124,626 for the Ministry of Interior  The ANA includes the ANA corps, Afghan Air Force, Special Mission Wing, ANA Special Operations Command, and Ktah Khas (counterterrorism forces)  The ANP includes the Afghan Uniformed Police, Afghan Anti-Crime Police, Afghan Border Police, Public Security Police, Counter Narcotics Police of Afghanistan, and General Command of Police Special Units  The ANA Special Mission Wing, Ktah Khas, ANA Special Operations Command, and ANP General Command of Police Special Units are collectively referred to as the Afghan Special Security Forces  In this report, we refer to the Afghan Air Force and the Afghan Special Security Forces as specialized forces, and the other components of the ANDSF as conventional forces  According to DOD reporting, the combined authorized force level for the specialized forces as of June 2018 was approximately 34,500, or about 10 percent of the ANDSF’s total authorized force level of 352,000, compared with the conventional forces, which make up about 74 percent of the total authorized force level for the ANDSF  Figure 1 shows the ANDSF’s organization  U S  and coalition advisors from Resolute Support focus on capacity building at the Ministry of Defense, Ministry of Interior, and ANDSF regional headquarters, according to DOD reporting  Ministerial advisors are located at Resolute Support headquarters in Kabul  At the ministerial level, advisors provide assistance to improve institutional capabilities, focusing on several functional areas  Table 1 summarizes the indicators of effectiveness that ministerial advisors are to use to measure ministerial progress in developing functioning systems that can effectively execute each of the functional areas  Regional Resolute Support advisors from seven advising centers located throughout Afghanistan provide support to nearby ANA corps and ANP zone headquarters personnel, according to DOD reporting  Some advisors are embedded with their ANDSF counterparts, providing a continuous coalition presence, while others provide less frequent support, based on proximity to and capability of their ANDSF counterparts  Regional advisors are to track ANDSF capability development by assessing the progress of the ANA corps and ANP zone headquarters based on five capability pillars (see table 2)  DOD and other Resolute Support advisors are to document the results of these assessments each quarter in an ANDSF Assessment Report  According to DOD reporting, in addition to ministerial and regional advising, two tactical-level advisory commands provide continuous support for the ANDSF’s specialized forces: Train, Advise, and Assist Command–Air (TAAC-Air) advises the Afghan Air Force down to the unit level, and NATO Special Operations Component Command–Afghanistan (NSOCC-A) primarily provides tactical-level special operations advising for the Afghan Special Security Forces  TAAC-Air and NSOCC-A assess capabilities at the headquarters level based on the five capability pillars described above in table 2, and these assessments are included in the quarterly ANDSF Assessment Report  Figure 2 shows the levels of advising each Resolute Support advisory command type provides for the ANDSF conventional forces and specialized forces  Since Resolute Support began, the ANDSF have improved some capabilities related to the functional areas and capability pillars described above, but face several capability gaps that leave them reliant on coalition assistance, according to publicly available DOD reporting  DOD defines capability as the ability to execute a given task  A capability gap is the inability to execute a specified course of action, such as an ANDSF functional area or a capability pillar (see tables 1 and 2 above)  According to DOD guidance, a gap may occur because forces lack a materiel or non-materiel capability, lack proficiency or sufficiency in a capability, or need to replace an existing capability solution to prevent a future gap from occurring  According to DOD reporting on the Afghan security ministries, ANA corps, and ANP zones, the ANDSF generally have improved in some capability areas since Resolute Support began, with some components performing better than others  For example, DOD has reported that the Afghan ministries have improved in operational planning, strategic communications, and coordination between the Ministry of Interior and Ministry of Defense at the national level  In general, the ANA is more capable than the ANP, according to DOD reporting  According to DOD officials and SIGAR reporting, this is due, in part, to the ANA having more coalition advisors and monitoring than the ANP  DOD officials also noted that the Ministry of Interior, which oversees the ANP, and Afghanistan’s justice system are both underdeveloped, hindering the effectiveness of the ANP  Corruption, understaffing, and training shortfalls have also contributed to the ANP’s underdevelopment, according to DOD and SIGAR reporting  The Afghan Special Security Forces are the most capable within the ANDSF and can conduct the majority of their operations independently without coalition enablers, according to DOD reporting  DOD and SIGAR reports have attributed the Afghan Special Security Forces’ relative proficiency to factors such as low attrition rates, longer training, and close partnership with coalition forces  The Afghan Air Force is becoming increasingly capable, and can independently plan for and perform some operational tasks, such as armed overwatch and aerial escort missions, according to DOD reporting  However, DOD has reported that the ANDSF generally continue to need support in several key areas  For example, as of December 2017, DOD reported several ministerial capability gaps, including force management; logistics; and analyzing and integrating intelligence, surveillance, and reconnaissance information  DOD also reported that, as of December 2017, the ANA and ANP continued to have capability gaps in several key areas, such as weapons and equipment sustainment and integrating fire from aerial and ground forces  The ANDSF rely on support from contractors and coalition forces to mitigate capability gaps in these key areas  For some capability areas, such as aircraft and vehicle maintenance and logistics, the ANDSF is not expected to be self- sufficient until at least 2023, according to DOD reporting  According to DOD officials and SIGAR reporting, coalition and contractor support helps mitigate ANDSF capability gaps in the immediate term but may make it challenging to assess the ANDSF’s capabilities and gaps independent of such support  For example, vehicle and aircraft maintenance contractors are responsible for sustaining specific operational readiness rates for the equipment they service  While this helps ensure that ANDSF personnel have working equipment to accomplish their mission, thereby closing an immediate capability gap, it may mask the ANDSF’s underlying capabilities and potentially prolong reliance on such support, according to DOD officials and SIGAR reporting  DOD and the ANDSF have begun implementing plans and initiatives that aim to strengthen ANDSF capabilities  These include the following, among others: ANDSF Roadmap  In 2017, the Afghan government began implementing the ANDSF Roadmap—a series of developmental initiatives that seek to strengthen the ANDSF and increase security and governance in Afghanistan, according to DOD reporting  The Roadmap is structured to span 4 years, but DOD has reported that its full implementation will likely take longer than that  According to DOD reporting, the Roadmap aims to improve four key elements: (1) fighting capabilities; (2) leadership development; (3) unity of command and effort; and (4) counter-corruption efforts  Under the Roadmap’s initiative to increase the ANDSF’s fighting capabilities, DOD and the ANDSF have begun implementing plans to increase the size of the specialized forces  Specifically, DOD reports that the ANDSF plans to nearly double the size of the Afghan Special Security Forces by 2020 as an effort to bolster the ANDSF’s offensive reach and effectiveness  The Afghan Special Security Forces are to become the ANDSF’s primary offensive force, the conventional ANA forces are to focus on consolidating gains and holding key terrain and infrastructure, and the conventional ANP forces are to focus on community policing efforts  In addition, to provide additional aerial fire and airlift capabilities, the ANDSF began implementing an aviation modernization plan in 2017  The aim is to increase personnel strength and the size of the Afghan Air Force and Special Mission Wing fleets by 2023  Enhanced vehicle maintenance efforts  To help improve the ANDSF’s vehicle maintenance abilities, DOD awarded a National Maintenance Strategy Ground Vehicle Support contract, which, according to DOD officials, became fully operational in December 2017  The National Maintenance Strategy Ground Vehicle Support contract consolidated five separate vehicle maintenance and training contracts into a single contract and contains provisions for building the capacity of ANDSF and Afghan contractors to incrementally take control of vehicle maintenance over a 5-year period  Additional U S  military personnel  As part of the South Asia strategy, the United States committed 3,500 additional military personnel to increase support to its missions in Afghanistan  According to DOD reporting, most of the additional personnel will support the Resolute Support mission, providing more advising and combat enabler support to the ANDSF  Additionally, in March 2018, the United States began deploying a Security Force Assistance Brigade—a new type of unit made up of U S  Army personnel with expertise in training foreign militaries—to Afghanistan  The Security Force Assistance Brigade will advise conventional and specialized forces at and below the corps and zone levels and will accompany and support ANA conventional forces at the battalion level in ground operations as needed, according to DOD and SIGAR reporting  DOD collects some reliable information about the operation and maintenance abilities of ANDSF specialized forces, in part because advisors are embedded at the tactical level with the specialized forces, according to DOD officials  Specifically, U S  and coalition forces advise specialized forces at the tactical level under Resolute Support because building ANDSF aviation and special operations abilities are considered particularly important, according to DOD reporting  DOD officials told us that since U S  and coalition forces are embedded at the tactical level for specialized forces, they can monitor, assess, and report on tactical abilities, including the ability to operate and maintain equipment  Our analysis of information provided by DOD about the Afghan Air Force’s ability to operate and maintain MD-530 helicopters illustrates that DOD has some detailed information about specialized forces  TAAC-Air advisors help train Afghan pilots and maintainers and collect information on their tactical abilities  For example, TAAC-Air advisors track the percentage of maintenance performed by Afghan Air Force maintainers and aircraft operational readiness rates, according to DOD officials  According to DOD reporting and officials, as of December 2017, the Afghan Air Force could independently conduct MD-530 helicopter operations for short intervals without contractor support but relied on contractors to perform the majority of maintenance and sustainment activities  See appendix II for more information on the Afghan Air Force’s ability to operate and maintain MD-530 helicopters  U S  and coalition forces perform high-level assessments of the ANDSF conventional forces’ capabilities at the corps and zone levels but do not assess their tactical abilities, such as the ability to operate and maintain equipment, according to DOD officials  For example, U S  and coalition forces assess the ANA and ANP conventional forces in quarterly ANDSF Assessment Reports, but these reports are at the corps and zone headquarters levels, and are not meant to provide an evaluation of the entire ANDSF, according to DOD reporting  DOD officials stated that other U S - and coalition-produced reports and assessments, such as DOD’s semiannual Section 1225 reports to Congress, semiannual periodic mission reviews, and annual Afghanistan Plans of Record, provide some information on the ANDSF’s high-level capabilities  However, according to DOD officials, these reports do not routinely assess the conventional forces’ ability to operate and maintain equipment  According to DOD officials, DOD does not assess conventional forces’ tactical abilities because advisors have had little or no direct contact with conventional units below the corps and zone levels, and thus do not collect such information on conventional forces  Specifically, under Resolute Support, U S  and coalition forces have not embedded with the conventional forces below the corps and zone levels except in limited circumstances  Since U S  and coalition forces do not collect firsthand information on the conventional units’ tactical abilities, they rely on those units’ self-reporting for information on ANDSF abilities below the corps and zone levels, which, according to DOD officials, may be unreliable  ANDSF reporting is not verified by U S  officials and can be unreliable in its consistency, comprehensiveness, and credibility, according to DOD officials and SIGAR  For example, the ANDSF produce a monthly tracker on vehicle availability, maintenance backlog, repair times, and personnel productivity, but DOD officials told us that the trackers are of questionable accuracy  Our analysis of information provided by DOD about the ANDSF’s ability to operate and maintain tactical and nontactical radios illustrates the limited amount of information DOD has on ANDSF conventional forces’ tactical abilities  Specifically, DOD officials could not say how well ANDSF personnel on the front lines operate radios in the field and had only limited information on the ANDSF’s ability to maintain radios  For example, the officials noted that the ANA conventional forces can perform some unit-level radio repairs but that complex ANA radio maintenance and all ANP radio maintenance is conducted by contractors  DOD officials at Resolute Support headquarters told us that they provide ministerial- level advising on how to manage ANDSF radio systems and do not provide tactical advising or inventory control for radios  See appendix III for more information on the ANDSF’s ability to operate and maintain radios  Our analysis of information provided by DOD about the ANDSF’s ability to operate and maintain Mobile Strike Force Vehicles (MSFV) highlights the limited amount of information DOD has on ANDSF conventional forces’ tactical abilities compared with specialized forces  DOD officials were able to provide operation and maintenance information for MSFVs that had transferred to the specialized forces as of January 2018 but were unable to provide operation and maintenance information for any other MSFVs  The ANDSF began transferring one of the ANDSF’s two MSFV brigades from the conventional to specialized forces in August 2017, according to DOD officials  As part of this transfer, NSOCC-A advisors—who provide tactical-level advising for the Afghan Special Security Forces—assumed oversight for the first brigade from Resolute Support headquarters advisors  DOD officials stated that the ANDSF’s ability to operate and maintain MSFVs in this brigade prior to the transfer was unknown, as neither Resolute Support headquarters nor the ANA had assessed this  The operation and maintenance abilities of the second brigade, which is still in the conventional forces, remains unknown  DOD officials at NSOCC-A were able to provide information such as inventory and mission capability rates for the MSFVs that had transferred, but only for the short period of time the vehicles had been under the control of the specialized forces  DOD officials told us that NSOCC-A plans to collect more information on the specialized forces’ ability to operate and maintain MSFVs as they are transferred  See appendix IV for more information on the ANDSF’s ability to operate and maintain MSFVs  In the absence of embedded advisors at the tactical level, DOD has not implemented alternative approaches to collect reliable information about the conventional forces’ ability to operate and maintain equipment  Federal internal control standards state that U S  agencies should obtain and process reliable information to evaluate performance in achieving key objectives and assessing risks  DOD officials acknowledged that some of the plans described above that DOD and the ANDSF have begun implementing to address capability gaps may provide opportunities for DOD to collect more reliable information on the conventional forces’ ability to operate and maintain U S -purchased equipment  For example, the National Maintenance Strategy Ground Vehicle Support contract requires that contractors regularly report the total work orders received, work in progress, and completed maintenance work performed by ANDSF personnel as well as vehicle availability rates, which may be more reliable than the ANDSF’s monthly report on vehicle availability  In addition, the Security Force Assistance Brigade may be able to collect and report on the tactical abilities of units they advise and accompany on missions since they are being deployed at or below the corps and zone levels  However, as of June 2018, DOD officials had not decided which, if any, of these options to pursue  Without reliable information on the equipment operation and maintenance abilities of ANDSF conventional forces, which represent nearly 75 percent of the ANDSF, DOD may be unable to fully evaluate the success of its train, advise, assist, and equip efforts in Afghanistan  The United States invested nearly $84 billion in Afghan security in the 17- year period spanning fiscal years 2002 through 2018, but DOD continues to face challenges to developing a self-sustaining ANDSF  While DOD has reported the ANDSF have improved in several capability areas, they continue to face critical capability gaps, impeding their ability to maintain security and stability in Afghanistan independent of U S  and coalition forces  Moreover, DOD lacks reliable information about the degree to which conventional forces—which make up about three-quarters of the ANDSF—are able to operate and maintain U S -purchased equipment  This limits DOD’s ability to fully evaluate the success of its train, advise, assist, and equip efforts in Afghanistan  The Secretary of Defense should develop and, as appropriate, implement options for collecting reliable information on the ANDSF conventional forces’ ability to operate and maintain U S -purchased equipment  (Recommendation 1) We provided a draft of this report to DOD and State for comment  DOD declined to provide written comments specifically on this public version of the report, but DOD’s comments on the sensitive version of this report are reprinted in appendix V  The sensitive version of this report included two recommendations, which DOD cited in its comments on the draft of the sensitive report  One of those recommendations related to information that DOD deemed to be sensitive and that must be protected from public disclosure  Therefore, we have omitted that recommendation from DOD’s comment letter in appendix V  This omission did not have a material effect on the substance of DOD’s comments  In its comments, DOD concurred with the recommendation we made in this version of the report and stated it will take steps to implement it  DOD also provided technical comments, which we incorporated as appropriate  The Department of State had no comments  We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, and the Secretary of State  In addition, the report is available at no charge on the GAO website at http://www gao gov  If you or your staff has any questions about this report please contact me at (202) 512-7114 or farbj@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report  GAO staff who made key contributions to this report are listed in appendix VI  House Report 114-537 associated with the National Defense Authorization Act for Fiscal Year 2017 included a provision for us to review the Afghan National Defense and Security Forces’ (ANDSF) capability and capacity to operate and sustain U S -purchased weapon systems and equipment  This report is a public version of a sensitive report that we issued on September 20, 2018  Our September report included three objectives, including one on the extent to which DOD considers ANDSF input and meets their needs when identifying equipment requirements  DOD deemed the information related to that objective to be sensitive, which must be protected from public disclosure  Consequently, we removed that objective and a related recommendation from this public report  This version includes information on the other two objectives: (1) what has been reported about ANDSF capabilities and capability gaps and (2) the extent to which DOD has information about the ANDSF’s ability to operate and maintain U S -purchased equipment  Although the information provided in this report is more limited, the report uses the same methodology for the two objectives as the sensitive report  To identify what has been reported about ANDSF capabilities and capability gaps, we reviewed North Atlantic Treaty Organization (NATO) and DOD documents and reports, such as DOD’s semiannual Section 1225 reports to Congress, produced after the start of the NATO-led Resolute Support mission on January 1, 2015  To determine what steps DOD and NATO have taken to try to address gaps, we reviewed reports the Center for Naval Analyses produced for DOD, as well as DOD and NATO documents and reports produced after January 1, 2015, and reports from GAO, the Special Inspector General for Afghanistan Reconstruction (SIGAR), and the DOD Inspector General  We also interviewed Center for Naval Analyses representatives and DOD officials in the United States and Afghanistan, including DOD officials at the Combined Security Transition Command–Afghanistan (CSTC-A) and in the Office of the Undersecretary of Defense for Policy (OSD-P) who helped create the DOD reporting we reviewed  To determine the extent to which DOD has information about the ANDSF’s ability to operate and maintain U S -purchased equipment, we reviewed DOD documents and reports and interviewed DOD officials in the United States and Afghanistan, including DOD officials who advise the ANDSF  We also reviewed federal internal control standards to determine what responsibilities agencies have specifically related to information collection  To provide illustrative examples of information DOD has about the ANDSF’s ability to operate and maintain U S - purchased equipment and what that information indicates about the ANDSF’s abilities and challenges, we interviewed and analyzed written responses from DOD officials, including DOD officials who provide procurement and lifecycle management for some ANDSF aircraft and vehicles, about three equipment types—MD-530 helicopters, Mobile Strike Force Vehicles (MSFV), and radios  We selected these three equipment types from a list that we developed, for an August 2017 report, of key ANDSF equipment the United States purchased from fiscal years 2003 through 2016  We made our selections after reviewing DOD documentation and interviewing DOD officials regarding a number of considerations, such as (1) how critical the equipment is to the ANDSF’s ability to achieve its mission; (2) which ANDSF component uses the equipment (i e , Afghan National Police, Afghan National Army, or both); (3) whether DOD intends to continue procuring the equipment for the ANDSF; and (4) whether the equipment had been in use at least 5 years  We collected detailed information about the ANDSF’s ability to operate and maintain MD-530 helicopters, MSFVs, and radios, as well as other key statistics DOD provided about the equipment, such as inventory, average lifespan, average cost, role, and training  This information was based mainly on DOD responses collected from January 2018 to February 2018 as well as DOD documents and reports produced after January 1, 2015  The total amount of MD-530s and radios authorized for procurement was based on DOD data that we collected for our August 2017 report on key ANDSF equipment the United States purchased in fiscal years 2003 through 2016, which we supplemented with additional data DOD provided on U S -purchased equipment from October 1, 2016, through December, 31, 2017  The performance audit upon which this report is based was conducted from August 2016 to September 2018 in accordance with generally accepted government auditing standards  Those standards require that we plan and perform the audit to obtain sufficient, appropriate, evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives  We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives  We subsequently worked with DOD from September 2018 to October 2018 to prepare this public version of the original sensitive report for public release  This public version was also prepared in accordance with those standards  Manufacturer: MD Helicopters, Inc  U S  Program Management Office: U S  Army, Non-Standard Rotary Wing Aircraft Project Management Office Program Advising: Train Advise Assist Command–Air (TAAC-Air) The United States originally procured 6 unarmed MD-530s for the AAF for rotary wing training in 2011  In 2014, the United States purchased 12 armed MD-530s and began retrofitting the 5 remaining trainer helicopters with armament for operational missions to address a close air attack gap  MD- 530s were chosen to fill the gap over other aircraft, in part because they could be delivered relatively quickly as the AAF awaited A-29 light attack aircraft that were experiencing procurement delays, according to Department of Defense (DOD) officials  The United States procured additional MD-530s in 2015, 2016, and 2017 because of the aircraft’s positive impact on the battlefield, according to DOD officials (see fig  3)  Key Statistics Variants: All can be armed with  50-cal machine gun pods and/or 2 75 inch rocket pods  Total Authorized for Procurement: 60 as of December 31, 2017 Inventory: 25 as of January 2018 (30 are scheduled for delivery; attrition of 5 due to crashes and enemy fire) Average Lifespan: Absent mishaps, and with good maintenance, there is no defined lifespan limit for MD-530s, according to DOD officials  National Army and Afghan National Police, depending on the mission, in all but one region of Afghanistan, which is supported by other aircraft  MD-530s are typically tasked two at a time for missions, according to DOD officials  An MD-530 crew consists of a pilot and co-pilot, according to DOD  Average cost: $6 3 million per aircraft, including all electronic devices, weapons management systems, and weapons (excluding ordnance), according to DOD officials  o Division of labor is based on the individual crew members' capabilities, with one pilot handling navigation and communication while the other identifies targets and operates the weapon systems  Army pilot advisors at Kandahar Air Field, according to DOD officials  o MD-530 pilot training takes about 3 years (see fig  4)  1  The GAO report number cited in DOD’s letter refers to a draft of the sensitive version of this report, which we issued on September 20, 2018  Prior to issuing that version, we changed its report number to GAO-18- 662SU to reflect its sensitive nature  That version of this report included two recommendations  The second recommendation has been omitted from DOD’s letter in this public version because it was related to information that DOD deemed to be sensitive  In addition to the contact named above, Joyee Dasgupta (Assistant Director), Kara Marshall, Katherine Forsyth, and Bridgette Savino made key contributions to this report  The team also benefitted from the expert advice and assistance of David Dayton, Neil Doherty, Justin Fisher, Ashley Alley, Cary Russell, Marie Mak, James Reynolds, Sally Williamson, Ji Byun, and J  Kristopher Keener 


Report 2: Federal and state Medicaid spending on long-term care continues to increase; for example it increased from $146 billion in 2013 to $158 billion in 2015  Individuals seeking long-term care generally need care that is, by definition, longer term in nature and more costly than other types of care  Spending on long-term care services provided in home and community settings, including assisted living facilities, exceeds the amount spent on institutional settings such as nursing homes  State Medicaid programs may cover certain medical and non-medical services that assisted living facilities provide; however, the Medicaid statute does not provide for coverage of room and board charges of an assisted living facility  In their federal-state partnership, both CMS and states play important roles in the oversight of Medicaid  CMS is responsible for oversight of state Medicaid programs  To conduct this oversight, CMS issues program requirements in the form of regulations and guidance, approves changes states make to their programs, provides technical assistance to states, collects and reviews required information and data from states and, in some cases, reviews individual state programs  States are responsible for the day-to-day administration of their Medicaid programs, including monitoring and oversight of the different HCBS programs through which they cover assisted living services, within broad federal rules and requirements  Each state is required to identify and designate a single state agency to administer or supervise the administration of its Medicaid program  The state Medicaid agency may partially or fully delegate the administration and oversight of the state’s HCBS programs to another state agency or other entity, such as a state unit on aging, a mental health department, or other state departments or agencies with jurisdiction over a specific population or service  However, the state Medicaid agency is ultimately accountable to the federal government for compliance with the HCBS requirements  Under different authorizing provisions of federal law, states have considerable flexibility to establish multiple HCBS programs including those covering assisted living services  A state Medicaid program can have multiple HCBS programs operating under different federal authorities  CMS is responsible for ensuring that states meet the requirements associated with their HCBS programs under these different authorities  Key to states’ monitoring of the health and welfare of Medicaid beneficiaries is their tracking of, and response to, incidents that may cause harm to a beneficiary’s health or welfare, such as abuse, neglect, or exploitation—commonly referred to as critical incidents  Such monitoring is required for most HCBS programs; however, we previously found that requirements for states related to oversight of the health and welfare of beneficiaries in different types of HCBS programs varied, and recommended that CMS take steps to harmonize those requirements across programs  The most common HCBS programs with the most stringent federal requirements are HCBS waiver programs  These programs serve beneficiaries who are eligible for an institutional level of care; that is, beneficiaries must have needs that rise to the level of care usually provided in a nursing facility, hospital, or other institution  CMS oversees states’ HCBS waiver programs specifically by reviewing and approving applications and reviewing HCBS program reports that states submit  HCBS waiver program applications include specific requirements implementing various statutory and regulatory provisions  (See text box below ) One requirement is that states have the necessary safeguards in place to protect the health and welfare of beneficiaries receiving services covered by HCBS waiver programs  For each of their HCBS waiver programs, states must demonstrate to CMS that they are meeting various requirements CMS has established regarding beneficiary health and welfare  The Six Requirements States Must Demonstrate for Home- and Community-Based Services Waiver Programs 1  Administrative authority: The Medicaid agency retains ultimate administrative authority and responsibility for the operation of the waiver program by exercising oversight of the performance of waiver functions by other state and local/regional non-state agencies (if appropriate) and contracted entities  2  Level of care: The state demonstrates that it implements the processes and instrument(s) specified in its approved waiver for evaluating/re-evaluating an applicant’s/waiver participant’s level of care consistent with care provided in a hospital, nursing facility, or intermediate care facility  3  Qualified providers: The state demonstrates that it has designed and implemented an adequate system for assuring that all waiver services are provided by qualified providers  4  Service plan: The state demonstrates it has designed and implemented an effective system for reviewing the adequacy of service plans for the waiver participants  5  Health and welfare: The state demonstrates it has designed and implemented an effective system for assuring waiver participant health and welfare  6  Financial accountability: The state must demonstrate that it has designed and implemented an adequate system for insuring financial accountability of the waiver program  CMS also provides ongoing oversight of state HCBS programs through annual reports that states must submit for each of their HCBS waiver programs as well as renewal reports submitted about two years before an HCBS waiver is scheduled to end  The state reports are intended to provide CMS with information on the operation of state HCBS waiver programs  In contrast to long-term care services provided in nursing facilities, less is known at the federal level about the oversight and quality of care in assisted living facilities  Generally, states establish their own licensing and oversight requirements for assisted living facilities  As a result, the requirements for assisted living facilities and the type and frequency of oversight can vary across states  In contrast, nursing homes must meet a comprehensive set of federal requirements in order to receive payment for long-term care services for Medicaid and Medicare beneficiaries in addition to state requirements  CMS contracts with state entities to regularly inspect nursing facilities and investigate complaints to assess whether nursing homes meet these federal quality requirements  Annually CMS publishes a comprehensive report on nursing homes that serve Medicaid and Medicare beneficiaries, including the extent that beneficiaries are at risk for harm, based on these investigations and inspections  In addition, CMS publicly reports a summary of each nursing home’s quality data using a five-star quality rating based on health inspection results, staffing data, and quality measure data  The goal of this rating system is to help consumers make meaningful distinctions among high- and low-performing nursing homes  This type of standardized framework for oversight, investigation and inspections, and reporting on quality of care concerns does not exist for assisted living facilities and other types of HCBS providers  Forty-eight state Medicaid agencies reported collectively spending about $10 billion in state and federal Medicaid funds for assisted living services in 2014, according to our survey  The other 3 states reported that they did not pay for assisted living services  We estimate that this spending for services provided by assisted living facilities represents 12 4 percent of the $80 6 billion Medicaid spent on HCBS in all settings that year  More than 330,000 Medicaid beneficiaries received assisted living services, based on data reported to us by the 48 states  Nationally, the average spending per beneficiary on assisted living services in the 48 states in 2014 was about $30,000; states provided these HCBS services through fee-for-service and managed care delivery models  Fee-for-service spending comprised 81 percent of total spending on assisted living services and managed care spending was about 19 percent of the total  The cost per beneficiary reported by surveyed states also varied based on payment type; average per beneficiary cost was $31,000 for fee-for-service and $27,000 for managed care  About 21 percent of Medicaid assisted living enrollment was for beneficiaries receiving these services under a managed care delivery model  (See table 1 ) Average per-beneficiary spending varied significantly across the states  For example, for the nine states with the lowest spending per beneficiary, average Medicaid spending ranged from about $1,700 to about $9,500 per beneficiary  In contrast, in the nine states with the highest per- beneficiary spending, the average spending ranged from about $43,000 to $108,000 per beneficiary  (See Figure 1 ) For more information on each state’s enrollment, total spending, and average per beneficiary spending on assisted living services, see appendix I  The 48 states that reported covering assisted living services in 2014 said they did so through 132 different programs  The majority of the states, 31 of the 48, reported administering more than one program that covered assisted living services  As illustrated in table 2 below, of the different types of HCBS programs under which states can provide coverage for assisted living services, HCBS waivers were the most common type of program they used  Specifically, 39 states and 69 percent of the programs that provided assisted living services, were operated under the HCBS waiver program  (See appendix II for additional details on each state’s number of programs by program type and total number of HCBS programs that covered assisted living facility services in 2014 ) Almost all of the 48 states that covered assisted living services did so for two groups of Medicaid beneficiaries eligible through their programs  In 45 of 48 states, aged beneficiaries received services provided by assisted living facilities  Similarly, in 43 of 48 states, physically disabled beneficiaries received services  (See Figure 2 ) In 38 or more of the 48 states that covered assisted living services, six types of services were provided  For example, 45 states covered assistance with activities of daily living, such as bathing and dressing; 44 states covered medication administration; and 41 states covered coordination of meals  (See Figure 3 ) State Medicaid agency approaches for oversight of assisted living services varied widely in terms of who provided the oversight for their largest programs, according to their responses to our survey  Thirteen of the 48 state Medicaid agencies reported delegating administrative responsibilities, including oversight of beneficiary health and welfare, to other state or local agencies  State Medicaid agencies may delegate the administration of programs to government or other agencies through a written agreement; however, state Medicaid agencies retain the ultimate oversight responsibility for those delegated functions  For example, among the 13 states that delegated HCBS program administration, the administering agencies were those that provided services to the aged, disabled, or both of these populations, such as the states’ Departments of Aging  (See text box, below, for examples of states’ delegation ) Examples of State Medicaid Agencies’ Delegation of Authority for Administration of Home- and Community-based Services’ Programs Covering Assisted Living Services Georgia’s Elderly & Disabled Waiver Program was operated in 2014 by the Georgia Department of Human Services Division of Aging Services, a separate agency of the state that was not a division/unit of the Medicaid agency  The Georgia Medicaid Agency maintained a formal interagency agreement with the Division of Aging Services which describes by function the required deliverables to support compliance and a schedule for delivery of reports  Nebraska’s Waiver for Aged and Adults and Children with Disabilities is operated by the state Medicaid agency Division of Medicaid and Long Term Care  The majority of services are provided by independent contractors in order to allow service delivery in the rural and frontier areas of the state  The state Medicaid agency contracts with the Area Agencies on Aging, Independent Living Centers, and Early Development Network agencies to perform a variety of operational and administrative functions including authorizing services and monitoring the delivery of services  States also varied in the types of information they reported reviewing as part of the oversight of assisted living services, and the extent to which state Medicaid agencies review the information when another agency is responsible for administration  For example, other entities outside the state Medicaid agency—such as the agency delegated to administer an HCBS program, or a contractor that manages provider enrollment—may check to ensure a provider is allowed to deliver services to Medicaid beneficiaries; in such cases, however, the state Medicaid agency might not be aware of the results of such checks  As illustrated in table 3, in all 48 states the types of information generally reviewed by either the state Medicaid agency, the agency delegated administrative responsibilities, or other agencies were: critical incident reports, the HHS Office of Inspector General’s list of excluded providers, patient service plans, and information on concerns about care received directly from patients, relatives, caregivers or the assisted living facility itself  In many cases, the state Medicaid agency did not review all information sources reviewed by other agencies  For example, although all critical incident reports were reviewed in the 48 states by either the state Medicaid agency, the agency delegated administrative responsibilities, or another agency; in 16 of those states, the state Medicaid agency was not involved in those reviews, according to responses to our survey  Instead, the critical incident reports were reviewed by another entity designated responsible for the HCBS program in the state or another state entity with regulatory responsibility over the assisted living facility  Such reviews, including any critical incidents found, may not have been communicated back to the state Medicaid agency, according to responses to our survey  State Medicaid agencies also varied in reporting the extent to which they were made aware or notified when enforcement actions were taken as a result of concerns with beneficiary care identified by other entities  Various oversight actions may be taken by the state Medicaid agency, the agency delegated to administer an HCBS program, or a state regulatory agency, such as a state agency responsible for licensing and inspecting various types of HCBS providers  When delegated agencies or other licensing agencies take corrective action, the state Medicaid agency may not be aware unless notified by the agencies taking that action  For example, in 23 states, the investigation of potential incidents related to beneficiary health and welfare was delegated to another agency but in only 6 of these states was the state Medicaid agency always notified of such an investigation based on our survey  (See table 4 and text box below ) Example of a Collaborative Approach to Monitoring and Ensuring Quality Care Specifically for Assisted Living Facilities In 2009, the Wisconsin Coalition for Collaborative Excellence in Assisted Living was formed to redesign the way quality is ensured and improved for individuals residing in assisted living communities  This public/private coalition utilizes a collective impact model approach that brings together the state, the industry, the consumer, and academia to identify and implement agreed upon approaches designed to improve the outcomes of individuals living in Wisconsin assisted living communities  The core of the coalition is the implementation of an association developed, department approved, comprehensive quality assurance, quality improvement program  For their largest HCBS programs that covered assisted living services, the 48 states varied in how they monitored “critical incidents” that caused actual or potential harm to Medicaid beneficiaries in assisted living facilities  Specifically, the 48 states varied in their ability to report the number of critical incidents; how they defined incidents, and the extent to which they made information on such incidents readily available to the public  These states varied in whether they could provide us the number of critical incidents involving beneficiaries for their largest programs covering assisted living services, and for those that could report, the number of incidents they reported varied widely  In 26 of the 48 states the Medicaid agencies were unable to report, for their largest program covering assisted living services, the number of critical incidents that had occurred in assisted living facilities in 2014  The remaining 22 states reported a total of 22,921 critical incidents involving Medicaid beneficiaries in their largest programs covering assisted living services  The number of critical incidents reported in these states ranged from 1 to 8,900  For six of these states the number of critical incidents reported was more than 1,000, (See text box, below, for examples of selected state processes managing critical incidents ) Selected States’ Processes for Managing Beneficiary Harm or Potential Harm in Assisted Living Facilities Georgia: According to state officials in 2014 there was no centralized or comprehensive system for capturing and tracking the data on actual and potential violations  State officials acknowledged the lack of a centralized system prevents the Division of Community Health from tracking the status of each problem  Nebraska: According to state officials, Nebraska’s Adult Protective Services operates an electronic system that coordinates across state social service programs  When Adult Protective Services initiates an investigation of reported harm to an assisted living resident, the state Medicaid agency is automatically notified  Reasons state Medicaid agencies reported for being unable to provide us with the number of critical incidents included limitations in the data or data systems for tracking them  Nine states reported an inability to track incidents by provider type, and thus distinguish critical incidents in assisted living facilities from other providers of home and community based services  States also cited lacking a system to collect critical incidents (9 states), and that the system for reporting could not identify whether a resident was a Medicaid beneficiary (5 states)  Even in the 32 states where the state Medicaid agencies reported reviewing information about critical incidents, 20 states were unable to provide the actual number of critical incidents that occurred in assisted living facilities  State Medicaid agencies’ definitions of critical incidents also varied  As illustrated in Figure 4, all 48 states cited physical assault, emotional abuse, and sexual assault or abuse as a critical incident in their largest programs providing assisted living services in 2014  However, for other types of incidents, several states did not identify the incident as critical, including discharge and eviction from the facility (not a critical incident in 24 states), medication errors (not a critical incident in 7 states), and unauthorized use of seclusion, (not a critical incident in 6 states)  For other serious incidents, a relatively small number of states did not identify the incident as critical, such as unexplained death (not a critical incident in 3 states) and missing beneficiaries (not a critical incident in 2 states)  See appendix IV for a full list of the beneficiary-related incidents and the number of states that identify each as critical  Although half of the 48 states that cover assisted living services did not consider discharges or evictions to be critical incidents, according to state responses to our survey, 42 states offered certain protections related to involuntary discharge of Medicaid residents who live in assisted living facilities  The majority of protections consisted of a lease agreement requirement that applied to other housing contracts in the state, such as providing residents with eviction notices  Other protections included an appeals process (10 states) and a requirement for the facility to find an alternative location for the resident (10 states)  State Medicaid agencies also varied in whether they made information on critical incidents and other key information readily available to the public  (See table 5 ) Beneficiaries seeking care in an assisted living facility may want to know the number of critical incidents related to a particular facility  Through our survey we found that states differed in the availability of information related to health and welfare that was available to the public  For example, 34 of the 48 states reported that they made critical incident information available to the public by phone, website, or in person, and the remaining 14 states did not have such information available at all  Although all 48 states had information in some form on which assisted facilities accepted Medicaid beneficiaries, 8 states could not provide this information by phone and 22 states could not provide the information in person  In recent years, CMS has taken steps to improve oversight of beneficiary health and welfare in HCBS programs by adding new HCBS waiver application requirements for state monitoring of beneficiary health and welfare  CMS requires state waiver applications to include specific requirements that implement various statutory and regulatory provisions, including a provision that states assure that they will safeguard the health and welfare of Medicaid beneficiaries  In March 2014, CMS added unexplained death to the events that states must be able to identify and address on an ongoing basis, as part of their efforts to prevent instances of abuse, neglect, and exploitation, and added four new requirements for states to protect beneficiary health and welfare  (See table 6 ) In its guidance implementing the 2014 requirements, CMS noted that state associations and state representatives’ work groups had agreed that “health and welfare is one of the most important assurances to track, and requires more extensive tracking to benefit the individuals receiving services, for instance by using data to prevent future incidents ” As a condition for approval of their HCBS waiver applications for each of the requirements, states must identify and agree with CMS on the type of information they will collect to provide as evidence that they will meet the requirements  However, according to CMS officials, each state Medicaid agency has wide discretion over the information it will collect and report to demonstrate that it is meeting the health and welfare requirements and protecting beneficiaries  Although CMS added the additional requirements in 2014 for safeguarding beneficiary health and welfare, the agency generally did not change requirements for how it oversees state monitoring efforts once HCBS waivers are approved  We found a number of limitations in CMS’s oversight of approved HCBS waivers that undermine the agency’s ability to effectively monitor state oversight of HCBS waivers  These limitations include: unclear guidance on what states should identify and report annually related to any identified program deficiencies; lack of requirements on states to regularly provide CMS information on critical incidents; and CMS’s inconsistent enforcement of the requirement that states submit annual reports  Unclear guidance on what states should identify and report annually related to any identified program deficiencies  Federal law requires states to provide CMS with information annually on an HCBS waiver’s impact on (1) the type and amount, and cost of services provided and (2) the health and welfare of Medicaid beneficiaries receiving waiver services  CMS reporting requirements give states latitude to determine what to report as health and welfare deficiencies found through state monitoring of their HCBS programs  With respect to health and welfare, CMS’s State Medicaid Manual directs states when preparing their annual reports to “check the appropriate boxes regarding the impact of the waiver on the health and welfare” of beneficiaries and to describe relevant information  States are required to provide a brief description of the state process for monitoring beneficiary safeguards, use check boxes to indicate that beneficiary health and welfare safeguards have been met, and identify whether deficiencies were detected during the monitoring process  If states determine that deficiencies were identified through monitoring, states are required to “provide a summary of the significant areas where deficiencies were detected” and an explanation of the actions taken to address deficiencies and ensure the deficiencies do not recur  CMS’s written instructions for completing the HCBS annual report do not provide further guidance regarding reporting of deficiencies  For example, the reporting instructions do not describe or identify 1) what states are supposed to report as deficiencies, 2) how they are to identify which deficiencies are most significant, and 3) the extent to which states need to explain the steps taken to ensure that deficiencies do not recur  The lack of clarity is inconsistent with federal internal control standards, in particular, the need for federal agencies to have processes that identify information needed to achieve objectives and address risk  Without clear instructions as to what states must report, states’ annual reports may not identify deficiencies with states’ HCBS waiver programs that may affect the health and welfare of beneficiaries  States may determine that issues or problems they identified through monitoring do not represent reportable deficiencies and therefore may not report those deficiencies to CMS, increasing the risk that problems are not elevated to CMS’s attention  In the case of one of the selected states we reviewed, no problems were included on the annual reports submitted to CMS between 2011 and 2015  However, when CMS completed its review in the fourth year of the state’s waiver— for purpose of renewing the waiver—it determined the state was not assuring beneficiary health and welfare  CMS found that the information the state submitted for purpose of renewal suggested a “pervasive failure” by the state to assure the health and welfare of beneficiaries receiving services, including assisted living services  In particular, CMS noted the state provided insufficient information regarding the number of unexpected or suspicious beneficiary deaths  CMS concluded that the state failed to demonstrate that it has effective systems and processes for ensuring the health and welfare of beneficiaries  Lack of requirements on states to annually provide CMS information on critical incidents  Despite the importance of state critical incident management and reporting systems to protecting the health and welfare of beneficiaries, CMS lacks written requirements that states provide information needed for the agency oversight of state monitoring of critical incidents  According to CMS, a critical element of effective state oversight is the operation of data systems that support the identification of trends and patterns in the occurrence of critical incidents to identify needed improvements  Such a system is also consistent with federal internal controls standards which specify, in particular, the need for federal agencies to have processes that identify information needed to achieve objectives and address risk  CMS requires states to operate a critical incident reporting system  On their waiver applications states must check a box indicating they operate a system and also describe their system—including who must report and when, and what must be reported  Despite this requirement for states to have critical incident reporting systems, CMS does not require states to report to CMS any data from these systems on critical incidents as part of their required annual reports  Specifically, states are not required to include, in their annual reports, the number of critical incidents reported or substantiated that involve Medicaid beneficiaries  As a result, CMS does not have a method to confirm what states describe about critical incident management systems, which is a required component of states’ waiver applications or to assess the capabilities of states’ systems  For example, CMS cannot confirm whether the state systems can report incidents by location or type of residential provider, such as assisted living facilities; the type and severity of critical incidents that occurred; and the number of incidents that involved Medicaid beneficiaries  Without annual critical incident reporting, CMS may be at risk of (1) not having adequate evidence that states are meeting CMS requirements to have an effective critical incident management and reporting system and of (2) being unaware of problems with states’ abilities to identify, track, and address critical incidents involving Medicaid beneficiaries  Our prior work has shown that the lack of explicit reporting requirements on critical incidents not only impacts HCBS waiver programs but also impacts other types of Medicaid long-term services programs as well  Specifically, In a November 2016 report, we found that CMS requirements for states to report on their critical incident monitoring systems for the HCBS waiver program were more stringent than those for other types of HCBS programs, potentially leaving those other programs at even greater risk  We recommended that CMS take steps to harmonize requirements across different types of HCBS programs  HHS concurred with the recommendation stating it would seek input from states, stakeholders, and the public regarding harmonizing requirements across programs  In an August 2017 report we found similar issues in critical incident reporting requirements for other types of long term services programs, particularly those used to provide HCBS and other long term services under managed care  We found that CMS was not always requiring states that contracted with managed care organizations to provide long term services and supports to report to CMS sufficient information on critical incidents and other key areas needed to monitor beneficiary access and quality  We recommended that CMS take steps to identify and obtain key information needed to better oversee states’ efforts to monitor beneficiary access to quality services in their managed long-term services and supports programs  HHS concurred with this recommendation and stated that the agency would take this recommendation into account as part of an ongoing review of its 2016 Medicaid managed care rule  We continue to believe that the implementation of our prior recommendations is needed to help improve CMS oversight of states monitoring of beneficiary safety  CMS’s inconsistent enforcement of the requirement that states submit annual reports  States must prepare and submit an annual report for each HCBS waiver as a condition of waiver approval  According to CMS guidance, the agency’s review of the annual report is part of the ongoing oversight of HCBS waiver programs and not submitting an annual report jeopardizes the states renewal of HCBS waiver programs  However, some states have not been timely in submitting the required annual reports for their HCBS waivers  A review of 2013 HCBS annual reports by a CMS contractor, published in 2016, found that annual reports were missing for 29 HCBS waivers and multiple years’ of annual reports were missing for 8 waivers  In 2014, CMS adopted new strategies to ensure compliance with HCBS waiver requirements, including the requirement that states submit annual reports on a timely basis  These strategies include withholding federal funding, placing a moratorium on enrollment in the waiver, or other actions the agency determines necessary  CMS officials reported that the agency had not used these new strategies with states that were delinquent in submitting their annual reports  Officials said they were in the process of reviewing how to implement these new strategies in the case of one state; however, as of August 2017 officials had not finalized a decision  CMS’s ability to provide effective oversight of state programs and protect beneficiary health and welfare is undermined by the lack of enforcement and receipt of required annual waiver reports  Effective state and federal oversight is necessary to ensure that the health and welfare of Medicaid beneficiaries receiving assisted living services are protected, especially given the particular vulnerability of many of these beneficiaries to abuse, neglect, or exploitation  CMS has taken steps to strengthen beneficiary health and welfare protections in states’ HCBS waiver programs, the most common type of program that covers assisted living services and one that serves the most vulnerable beneficiaries  In particular, CMS now has multiple requirements for states to safeguard beneficiaries’ health and welfare, including requirements to operate an effective critical incident management and reporting system to identify, investigate, and address incidents of beneficiary abuse, neglect, exploitation, and unexplained death  However, CMS’s ability to effectively monitor how well states are assuring beneficiary health and welfare is limited by gaps in state reporting to CMS  CMS has not provided clear guidance to states on what information to include in annual reports on deficiencies they identify  As a result, CMS lacks assurance that it is receiving consistent, complete, and relevant information on deficiencies that is needed to oversee beneficiary health and welfare  Lacking clear guidance on the reporting of deficiencies may result in a delayed recognition of problems that may affect beneficiary health and welfare  Further, for years, states have been required to check a box attesting that they operate a critical incident management system, but have not always been required to report information on incidents of potential or actual harm to beneficiaries  Given the increasing prevalence of assisted living facilities as a provider of services to Medicaid beneficiaries, it is unclear why more than half of states responding to our survey could not provide us information on the number of critical incidents that occurred in these facilities in their states  Reporting data from their critical incident systems, such as the number of incidents, the type and severity of the incidents, or the location or type of facility in which the incident occurred would provide evidence that an effective system is in place, provide information on the extent beneficiaries are subject to actual or potential harm, and allow for tracking trends over time  Finally, CMS has not ensured that all states submit annual reports on their HCBS waiver programs as required  Without improvements to state reporting, CMS cannot ensure states are meeting their commitments to protect the health and welfare of Medicaid beneficiaries receiving assisted living services, potentially jeopardizing their care  We are making the following three recommendations to CMS: The Administrator of CMS should provide guidance and clarify requirements regarding the monitoring and reporting of deficiencies that states using HCBS waivers are required to report on their annual reports  (Recommendation 1) The Administrator of CMS should establish standard Medicaid reporting requirements for all states to annually report key information on critical incidents, considering, at a minimum, the type of critical incidents involving Medicaid beneficiaries, and the type of residential facilities, including assisted living facilities, where critical incidents occurred  (Recommendation 2) The Administrator of CMS should ensure that all states submit annual reports for HCBS waivers on time as required  (Recommendation 3) We provided a draft of this report to HHS for review and comment  HHS provided written comments, which are reproduced in Appendix V  The department also provided technical comments, which we incorporated as appropriate  In its written comments, the department concurred with two of our three recommendations, specifically, that CMS will clarify requirements for state reporting of program deficiencies and ensure that all states submit required annual reports on time  HHS did not explicitly agree or disagree with our third recommendation to require all states to report information on critical incidents to CMS annually  The department noted it has established a workgroup to learn more about states’ health and welfare systems and that it will use the results of this workgroup to determine which additional reporting requirements would be beneficial  The workgroup’s review will continue through calendar year 2018  In technical comments, HHS indicated that after the workgroup’s review is complete it will consider annual reporting of critical incidents  We believe establishing the workgroup is a positive first step towards improving oversight and state reporting and encourage HHS to require annual reporting on critical incidents when developing additional reporting requirements  As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date  At that time, we will send copies to the Secretary of Health and Human Services, the Administrator of CMS, the Administrator of the Administration for Community Living, appropriate congressional committees, and other interested parties  In addition, the report will be available at no charge on the GAO website at http://www gao gov  If you or your staff members have any questions about this report, please contact me at (202) 512-7114 or at iritanik@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report  GAO staff that made key contributions to this report are listed in appendix VI  Our survey of state Medicaid agencies regarding coverage, spending, enrollment, and oversight of assisted living services in 2014, obtained information on challenges for Medicaid beneficiaries to access assisted living services in their states  States provided information related to factors that create challenges for Medicaid beneficiaries’ ability to access and receive assisted living services and the extent states had policies to help beneficiaries with the cost of room and board  A number of states in our survey cited common factors as creating the greatest challenges to a beneficiary’s ability to access assisted living services, including the number of assisted living facilities willing to accept Medicaid beneficiaries (13 states or 27 percent of the 48 states) program enrollment caps (9 states or 19 percent of the 48 states) beneficiaries’ inability to pay for assisted living facility room and board (9 states or 19 percent of the 48 states), which Medicaid typically does not cover low rates the state Medicaid program paid assisted living facilities (8 states or 17 percent of the 48 states)  A number of states reported that they had policies to assist Medicaid beneficiaries with the costs of room and board charged by assisted living facilities, which Medicaid does not typically cover  Two common policies, cited by at least half of the states, were aimed at limiting how much assisted living facilities could charge Medicaid beneficiaries for room and board  For example, 30 of 48 states, limited the amount facilities could charge for room and board to the amount of income certain beneficiaries receive as Supplemental Security Income  The other commonly cited policies focused on providing financial assistance to the beneficiaries to defray the room and board costs  (See table 9 ) In addition to the contact named above, Tim Bushfield and Christine Brudevold (Assistant Directors), Jennie Apter, Shirin Hormozi, Anne Hopewell, Kelsey Kreider, Perry Parsons, Vikki Porter, and Jennifer Whitworth made key contributions to this report 

ground_truth: Developing independently capable ANDSF is a key component of U.S. and coalition efforts to create sustainable security and stability in Afghanistan under the North Atlantic Treaty Organization (NATO)-led Resolute Support mission. The United States is the largest contributor of funding and personnel to Resolute Support, providing and maintaining ANDSF equipment, along with training, advising, and assistance to help the ANDSF effectively use and sustain the equipment in the future. House Report 114-537 included a provision for GAO to review the ANDSF's capability and capacity to operate and sustain U.S.-purchased weapon systems and equipment. This report addresses (1) what has been reported about ANDSF capabilities and capability gaps and (2) the extent to which DOD has information about the ANDSF's ability to operate and maintain U.S.-purchased equipment. To conduct this work, GAO analyzed DOD and NATO reports and documents, examined three critical equipment types, and interviewed DOD officials in the United States and Afghanistan. This is a public version of a sensitive report issued in September 2018. Information that DOD deemed sensitive has been omitted. Since the Resolute Support mission began in 2015, the Afghan National Defense and Security Forces (ANDSF) have improved some fundamental capabilities, such as high-level operational planning, but continue to rely on U.S. and coalition support to fill several key capability gaps, according to Department of Defense (DOD) reporting. DOD has initiatives to address some ANDSF capability gaps, such as a country-wide vehicle maintenance and training effort, but DOD reports it does not expect the ANDSF to develop and sustain independent capabilities in some areas, such as logistics, for several years. While DOD has firsthand information on the abilities of the Afghan Air Force and Special Security Forces to operate and maintain U.S.-purchased equipment, it has little reliable information on the equipment proficiency of conventional ANDSF units. U.S. and coalition advisors are embedded at the tactical level for the Air Force and Special Security Forces, enabling DOD to directly assess those forces' abilities. However, the advisors have little direct contact with conventional ANDSF units on the front lines. As a result, DOD relies on those units' self-assessments of tactical abilities, which, according to DOD officials, can be unreliable. GAO's analysis of three critical equipment types illustrated the varying degrees of DOD's information (see figure above). For example, DOD provided detailed information about the Air Force's ability to operate and maintain MD-530 helicopters and the Special Security Forces' ability to operate and maintain Mobile Strike Force Vehicles; however, DOD had limited information about how conventional forces operate and maintain radios and Mobile Strike Force Vehicles. DOD's lack of reliable information on conventional forces' equipment operations and maintenance abilities adds to the uncertainty and risk in assessing the progress of DOD efforts in Afghanistan. GAO recommends that DOD develop options for collecting reliable information on conventional ANDSF units' ability to operate and maintain U.S.-purchased equipment. DOD concurred with this recommendation.

prompt_front: You are given two reports by the government agency.



Question: Write a one-page summary of the report 2.

Report 1: FEMA’s mission is to help people before, during, and after disasters  It provides assistance to those affected by emergencies and disasters by supplying immediate needs (e g , ice, water, food, and temporary housing) and providing financial assistance grants for damage to personal or public property  FEMA also provides non-disaster assistance grants to improve the nation’s preparedness, readiness, and resilience to all hazards  FEMA accomplishes a large part of its mission through awarding grants to state, local, and tribal governments and nongovernmental entities to help communities prevent, prepare for, protect against, mitigate the effects of, respond to, and recover from disasters and terrorist attacks  As previously mentioned, for fiscal years 2005 through 2014, the agency obligated about $104 5 billion in disaster relief grants  In addition, as of April 2018, the four major disasters in 2017—hurricanes Harvey, Irma, and Maria; and the California wildfires—had resulted in over $22 billion in FEMA grants  The current FEMA grants management environment is highly complex with many stakeholders, IT systems, and users  Specifically, this environment is comprised of 45 active disaster and non-disaster grant programs, which are grouped into 12 distinct grant categories  For example, one program in the Preparedness: Fire category is the Assistance to Firefighters Grants (AFG) program, which provides grants to fire departments, nonaffiliated emergency medical service organizations, and state fire training academies to support firefighting and emergency response needs  As another example, the Housing Assistance grant program is in the Recovery Assistance for Individuals category and provides financial assistance to individuals and households in geographical areas that have been declared an emergency or major disaster by the President  Table 1 lists FEMA’s non-disaster and disaster-based grant categories  According to FEMA, the processes for managing these different types of grants vary because the grant programs were developed independently by at least 18 separate authorizing laws that were enacted over a 62-year period (from 1947 through 2009)  The various laws call for different administrative and reporting requirements  For example, the Robert T  Stafford Disaster Relief and Emergency Assistance Act, as amended, established the statutory authority for 11 of the grant programs, such as the administration of Public Assistance and Individual Assistance grant programs after a presidentially declared disaster  The act also requires the FEMA Administrator to submit an annual report to the President and Congress covering FEMA’s expenditures, contributions, work, and accomplishments, pursuant to the act  As another example, the National Dam Safety Program Act established one of the grant programs aimed at providing financial assistance to improve dam safety  Key stakeholders in modernizing the IT grants management environment include the internal FEMA officials that review, approve, and monitor the grants awarded, such as grant specialists, program analysts, and supervisors  FEMA has estimated that it will need to support about 5,000 simultaneous internal users of its grants management systems  Other users include the grant recipients that apply for, receive, and submit reports on their grant awards; these are considered the external system users  These grant recipients can include individuals, states, local governments, Indian tribes, institutions of higher education, and nonprofit organizations  FEMA has estimated that there are hundreds of thousands of external users of its grants systems  The administration of the many different grant programs is distributed across four divisions within FEMA’s organizational structure  Figure 1 provides an overview of FEMA’s organizational structure and the divisions that are responsible for administering grants  Within three of the four divisions—Resilience, United States Fire Administration, and Office of Response and Recovery—16 different grant program offices are collectively responsible for administering the 45 grant programs  The fourth division consists of 10 regional offices that help administer grants within their designated geographical regions  For example, the Office of Response and Recovery division oversees three different offices that administer 13 grant programs that are largely related to providing assistance in response to presidentially declared disasters  Figure 2 shows the number of grant programs administered by each of the four divisions’ grant program and regional offices  In addition, appendix II lists the names of the 45 grant programs  FEMA’s OCIO is responsible for developing, enhancing, and maintaining the agency’s IT systems, and for increasing efficiencies and cooperation across the entire organization  However, we and the DHS Office of Inspector General (OIG) have previously reported that the grant programs and regional offices develop information systems independent of the OCIO and that this has contributed to the agency’s disparate IT environment  We and the DHS OIG have reported that this disparate IT environment was due, in part, to FEMA’s decentralized IT budget and acquisition practices  For example, from fiscal years 2010 through 2015, the OCIO’s budget represented about one-third of the agency’s IT budget, with the grant program offices accounting for the remaining two-thirds of that budget  In February 2018, the OIG found that FEMA had shown limited progress in improving its IT management and that many of the issues reported in prior audits remained unchanged  As such, the OIG initiated a more comprehensive audit of the agency’s IT management that is ongoing  FEMA has identified 10 primary legacy IT systems that support its grants management activities  According to the agency, most of these systems were developed to support specific grant programs or grant categories  Table 2 summarizes the 10 primary legacy systems  According to FEMA officials, the 10 primary grant systems are all in operation (several have been for decades) and are not interoperable  As a result, individual grant programs and regional offices have independently developed work arounds intended to address existing capability gaps with the primary systems  FEMA officials stated that while these work arounds have helped the agency partially address capability gaps with its primary systems, they are often nonstandardized processes, and introduce the potential for information security risks and errors  This environment has contributed to labor-intensive manual processes and an increased burden for grant recipients  The disparate systems have also led to poor information sharing and reporting capabilities, as well as difficulty reconciling financial data  The DHS OIG and we have previously highlighted challenges with FEMA’s past attempts to modernize its grant management systems  For example, In December 2006, the DHS OIG reported that EMMIE, an effort to modernize its grants management systems and provide a single grants processing solution, was being developed without a clear understanding and definition of the future solution  The report also identified the need to ensure crosscutting participation from headquarters, regions, and states in developing and maintaining a complete, documented set of FEMA business and system requirements  In April 2016, we found weaknesses in FEMA’s development of the EMMIE system  For example, we noted that the system was implemented without sufficient documentation of system requirements, an acquisition strategy, up-to-date cost estimate and schedule, total amount spent to develop the system, or a systems integration plan  In response to our findings and related recommendations, FEMA took action to address these issues  For example, the agency implemented a requirements management process that, among other things, provided guidance to programs on analyzing requirements to ensure that they are complete and verifiable  We reported in November 2017 that EMMIE lacked the ability to collect information on all pre-award activities and, as a result, agency officials said that they and applicants used ad hoc reports and personal tracking documents to manage and monitor the progress of grant applications  FEMA officials added that applicants often struggled to access the system and that the system was not user friendly  Due to EMMIE’s shortfalls, the agency had to develop another system in 2017 to supplement EMMIE with additional grant tracking and case management capabilities  FEMA initiated GMM in 2015, in part, due to EMMIE’s failed attempt to modernize the agency’s grants management environment  The program is intended to modernize and streamline the agency’s grants management environment  To help streamline the agency’s grants management processes, the program established a standard framework intended to represent a common grants management lifecycle  The framework consists of five sequential phases—pre-award, award, post-award, closeout, and post- closeout—along with a sixth phase dedicated to continuous grant program management activities, such as analyzing data and producing reports on grant awards and managing IT systems  FEMA also established 43 distinct business functions associated with these six lifecycle phases  Figure 3 provides the general activities that may occur in each of the grant lifecycle phases, but specific activities would depend on the type of grant being administered (i e , disaster versus non-disaster)  GMM is expected to be implemented within the complex IT environment that currently exists at FEMA  For example, the program is intended to replace the 10 legacy grants management systems, and potentially many additional subsystems, with a single IT system  Each of the 10 legacy systems was developed with its own database(s) and with no standardization of the grants management data and, according to FEMA officials, this legacy data has grown significantly over time  Accordingly, FEMA will need to migrate, analyze, and standardize the grants management data before transitioning it to GMM  The agency awarded a contract in June 2016 to support the data migration efforts for GMM  The agency also implemented a data staging environment in October 2017 to migrate the legacy data and identify opportunities to improve the quality of the data  Further, the GMM system is expected to interface with a total of 38 other systems  These include 19 systems external to DHS (e g , those provided by commercial entities or other federal government agencies) and 19 systems internal to DHS or FEMA  Some of the internal FEMA systems are undergoing their own modernization efforts and will need to be coordinated with GMM, such as the agency’s financial management systems, national flood insurance systems, and enterprise data warehouses  For example, FEMA’s Financial Systems Modernization Program was originally expected to deliver a new financial system in time to interface with GMM  However, the financial modernization has been delayed until after GMM is to be fully implemented; thus, GMM will instead need to interface with the legacy financial system  As a result, GMM is in the process of removing one of its key performance parameters in the acquisition program baseline related to financial systems interoperability and timeliness of data exchanged  In May 2017, DHS approved the acquisition program baseline for GMM  The baseline estimated the total lifecycle costs to be about $251 million, initial operational capability to be achieved by September 2019, and full operational capability to be achieved by September 2020  FEMA intends to develop and deploy its own software applications for GMM using a combination of commercial-off-the-shelf software, open source software, and custom developed code  The agency plans to rely on an Agile software development approach  According to FEMA planning documentation, the agency plans to fully deliver GMM by September 2020 over eight Agile development increments  Agile development is a type of incremental development, which calls for the rapid delivery of software in small, short increments  Many organizations, especially in the federal government, are accustomed to using a waterfall software development model  This type of model typically consists of long, sequential phases, and differs significantly from the Agile development approach  We have previously reported that DHS has sought to establish Agile software development as the preferred method for acquiring and delivering IT capabilities  However, the department has not yet completed critical actions necessary to update its guidance, policies, and practices for Agile programs, in areas such as, developing lifecycle cost estimates, managing IT requirements, testing and evaluation, oversight at key decision points, and ensuring cybersecurity  (See appendix III for more details on the Agile software development approach ) FEMA’s acquisition approach includes using contract support to assist with the development and deployment efforts  The agency selected a public cloud environment to host the computing infrastructure  In addition, from March through July 2017, the agency used a short-term contract aimed at developing prototypes of GMM functionality for grant tracking and monitoring, case management of disaster survivors, grant reporting, and grant closeout  The agency planned to award a second development contract by December 2017 to complete the GMM system (beyond the prototypes) and to begin this work in September 2018  However, due to delays in awarding the second contract to develop the complete GMM system, in January 2018, the program extended the scope and time frames of the initial short-term prototype contract for an additional year to develop the first increment of the GMM system— referred to as the AFG pilot  On August 31, 2018, FEMA awarded the second development contract, which is intended to deliver the remaining functionality beyond the AFG pilot (i e , increments 2 through 8)  FEMA officials subsequently issued a 90-day planning task order for the Agile development contractor to define the work that needs to be done to deliver GMM and the level of effort needed to accomplish that work  However, the planning task order was paused after a bid protest was filed with GAO in September 2018  According to FEMA officials, they resumed work on the planning task order after the bid protest was withdrawn by the protester on November 20, 2018, and then the work was paused again during the partial government shutdown from December 22, 2018, through January 25, 2019  FEMA began working on the AFG pilot—GMM’s first increment—in January 2018  This increment was intended to pilot GMM’s use of Agile development methods to replace core functionality for the AFG system (i e , one of the 10 legacy systems) This system supports three preparedness/fire-related grant programs—Assistance to Firefighters Grants Program, Fire Prevention and Safety Grant Program, and Staffing for Adequate Fire and Emergency Response Grant Program  According to FEMA officials, the AFG system was selected as the first system to be replaced because it is costly to maintain and the DHS OIG had identified cybersecurity concerns with the system  Among the 43 GMM business functions discussed earlier in this report, FEMA officials specified 19 functions to be delivered in the AFG pilot  Figure 4 shows the planned time frames for delivering the AFG pilot in increment 1 (which consisted of four 3-month Agile development sub- increments), as of August 2018  As of August 2018, the program was working on sub-increment 1C of the pilot  In September 2018, GMM deployed its first set of functionality to a total of 19 AFG users—which included seven of 169 total internal AFG users, and 12 of more than 153,000 external AFG users  The functionality supported four of the 19 business functions that are related to the closeout of grants (i e , the process by which all applicable administrative actions and all required work to award a grant have been completed)  This functionality included tasks such as evaluation of final financial reports submitted by grant recipients and final reconciliation of finances (e g , final disbursement to recipients and return of unobligated federal funds)  According to FEMA officials, closeout functionality was selected first for deployment because it was the most costly component of the legacy AFG system to maintain, as it is an entirely manual and labor-intensive process  The remaining AFG functionality and remaining AFG users are to be deployed by the end of the AFG pilot  The GMM program is executed by a program management office, which is overseen by a program manager and program executive  This office is responsible for directing the day-to-day operations and ensuring completion of GMM program goals and objectives  The program office resides within the Office of Response and Recovery, which is headed by an Associate Administrator who reports to the FEMA Administrator  In addition, the GMM program executive (who is also the Regional Administrator for FEMA Region IX) reports directly to the FEMA Administrator  GMM is designated as a level 2 major acquisition, which means that it is subject to oversight by the DHS acquisition review board  The board is chaired by the DHS Undersecretary for Management and is made up of executive-level members, such as the DHS Chief Information Officer  The acquisition review board serves as the departmental executive board that decides whether to approve GMM through key acquisition milestones and reviews the program’s progress and its compliance with approved documentation every 6 months  The board approved the acquisition program baseline for GMM in May 2017 (i e , estimated costs to be about $251 million and full operational capability to be achieved by September 2020)  In addition, the program is reviewed on a monthly basis by FEMA’s Grants Management Executive Steering Group  This group is chaired by the Deputy Administrator of FEMA  Further, DHS’s Financial Systems Modernization Executive Steering Committee, chaired by the DHS Chief Financial Officer, meets monthly and is to provide guidance, oversight, and support to GMM  For government organizations, including FEMA, cybersecurity is a key element in maintaining the public trust  Inadequately protected systems may be vulnerable to insider threats  Such systems are also vulnerable to the risk of intrusion by individuals or groups with malicious intent who could unlawfully access the systems to obtain sensitive information, disrupt operations, or launch attacks against other computer systems and networks  Moreover, cyber-based threats to federal information systems are evolving and growing  Accordingly, we designated cybersecurity as a government-wide high risk area 22 years ago, in 1997, and it has since remained on our high-risk list  Federal law and guidance specify requirements for protecting federal information and information systems  The Federal Information Security Modernization Act (FISMA) of 2014 requires executive branch agencies to develop, document, and implement an agency-wide cybersecurity program to provide security for the information and information systems that support operations and assets of the agency  The act also tasks NIST with developing, for systems other than those for national security, standards and guidelines to be used by all agencies to establish minimum cybersecurity requirements for information and information systems based on their level of cybersecurity risk  Accordingly, NIST developed a risk management framework of standards and guidelines for agencies to follow in developing cybersecurity programs  The framework addresses broad cybersecurity and risk management activities, including categorizing the system’s impact level; selecting, implementing, and assessing security controls; authorizing the system to operate (based on progress in remediating control weaknesses and an assessment of residual risk); and monitoring the efficacy of controls on an ongoing basis  Figure 5 provides an overview of this framework  Prior DHS OIG assessments, such as the annual evaluation of DHS’s cybersecurity program, have identified issues with FEMA’s cybersecurity practices  For example, in 2016, the OIG reported that FEMA was operating 111 systems without an authorization to operate  In addition, the agency had not created any corrective action plans for 11 of the systems that were classified as “Secret” or “Top Secret,” thus limiting its ability to ensure that all identified cybersecurity weaknesses were mitigated in a timely manner  The OIG further reported that, for several years, FEMA was consistently below DHS’s 90 percent target for remediating corrective action plans, with scores ranging from 73 to 84 percent  Further, the OIG reported that FEMA had a significant number of open corrective action plans (18,654) and that most of these plans did not contain sufficient information to address identified weaknesses  In 2017, the OIG reported that FEMA had made progress in addressing security weaknesses  For example, it reported that the agency had reduced the number of systems it was operating without an authorization to operate from 111 to 15 systems  According to GAO’s Business Process Reengineering Assessment Guide and the Software Engineering Institute’s Capability Maturity Model Integration® for Development, successful business process reengineering can enable agencies to replace their inefficient and outmoded processes with streamlined processes that can more effectively serve the needs of the public and significantly reduce costs and improve performance  Many times, new IT systems are implemented to support these improved business processes  Thus, effective management of IT requirements is critical for ensuring the successful design, development, and delivery of such new systems  These leading practices state that effective business process reengineering and IT requirements management involve, among other things, (1) ensuring strong executive leadership support for process reengineering; (2) assessing the current and target business environment and business performance goals; (3) establishing plans for implementing new business processes; (4) establishing clear, prioritized, and traceable IT requirements; (5) tracking progress in delivering IT requirements; and (6) incorporating input from end user stakeholders  Among these six selected leading practices for reengineering business processes and managing IT requirements, FEMA fully implemented four and partially implemented two of them for its GMM program  For example, the agency ensured strong senior leadership commitment to changing the way it manages its grants, took steps to assess and document its business environment and performance goals, defined initial IT requirements for GMM, took recent actions to better track progress in delivering planned IT requirements, and incorporated input from end user stakeholders  In addition, FEMA had begun planning for business process reengineering; however, it had not finalized plans for transitioning users to the new business processes  Further, while GMM took steps to establish clearly defined and prioritized IT requirements, key requirements were not always traceable  Table 3 summarizes the extent to which FEMA implemented the selected leading practices  According to GAO’s Business Process Reengineering Assessment Guide, the most critical factor for engaging in a reengineering effort is having strong executive leadership support to establish credibility regarding the seriousness of the effort and to maintain the momentum as the agency faces potentially extensive changes to its organizational structure and values  Without such leadership, even the best process design may fail to be accepted and implemented  Agencies should also ensure that there is ongoing executive support (e g , executive steering committee meetings headed by the agency leader) to oversee the reengineering effort from start to finish  FEMA senior leadership consistently demonstrated its commitment and support for streamlining the agency’s grants management business processes and provided ongoing executive support  For example, one of the Administrator’s top priorities highlighted in FEMA’s 2014 through 2022 strategic plans was to strengthen grants management through innovative systems and business processes to rapidly and effectively deliver the agency’s mission  In accordance with this strategic priority, FEMA initiated GMM with the intent to streamline and modernize grants management across the agency  In addition, FEMA established the Grants Management Executive Steering Group in September 2015  This group is responsible for transforming the agency’s grants management capabilities through its evaluation, prioritization, and oversight of grants management modernization programs, such as GMM  The group’s membership consists of FEMA senior leaders from across the agency’s program and business support areas, such as FEMA regions, Individual Assistance, Public Assistance, Preparedness, Office of the Chief Financial Officer, Office of Chief Counsel, OCIO, and the Office of Policy and Program Analysis  In this group’s ongoing commitment to reengineering grants management processes, it meets monthly to review GMM’s updates, risks, and action items, as well as the program’s budget, schedule, and acquisition activities  For example, the group reviewed the status of key acquisition activities and program milestones, such as the follow-on award for the pilot contractor and the program’s initial operational capability date  The group also reviewed GMM’s program risks, such as data migration challenges (discussed later in this report) and delays in the Agile development contract award  With this continuous executive involvement, FEMA is better positioned to maintain momentum for reengineering the new grants management business processes that the GMM system is intended to support  GAO’s Business Process Reengineering Assessment Guide states that agencies undergoing business process reengineering should develop a common understanding of the current environment by documenting existing core business processes to show how the processes work and how they are interconnected  The agencies should then develop a deeper understanding of the target environment by modeling the workflow of each target business process in enough detail to provide a common understanding of exactly what will be changed and who will be affected by a future solution  Agencies should also assess the performance of their current major business processes to identify problem areas that need to be changed or eliminated and to set realistically achievable, customer- oriented, and measurable business performance improvement goals  FEMA has taken steps to document the current and target grants management business processes  Specifically, The agency took steps to develop a common understanding of its grants management processes by documenting each of the 12 grant categories  For example, in 2016 and 2017, the agency conducted several nationwide user outreach sessions with representatives from FEMA headquarters, the 10 regional offices, and state and local grant recipients to discuss the grant categories and the current grants management business environment  In addition, FEMA’s Office of Chief Counsel developed a Grants Management Manual in January 2018 that outlined the authorizing laws, regulations, and agency policies for all of its grant programs  According to the Grants Management Executive Steering Group, the manual is intended to promote standardized grants management procedures across the agency  Additionally, the group expects grant program and regional offices to assess the manual against their own practices, make updates as needed, and ensure that their staff are properly informed and trained  FEMA also documented target grants management business process workflows for 18 of the 19 business functions that were notionally planned to be developed and deployed in the AFG pilot by December 2018  However, the program experienced delays in developing the AFG pilot (discussed later in this report) and, thus, deferred defining the remaining business function until the program gets closer to developing that function, which is now planned for August 2019  In addition, FEMA established measurable business performance goals for GMM that are aimed at addressing problem areas and improving grants management processes  Specifically, the agency established 14 business performance goals and associated thresholds in an October 2017 acquisition program baseline addendum, as well as 126 performance metrics for all 43 of the target grants management business functions in its March 2017 test and evaluation master plan  According to FEMA, the 14 business performance goals are intended to represent essential outcomes that will indicate whether GMM has successfully met critical, business-focused mission needs  GMM performance goals include areas such as improvements in the satisfaction level of users with GMM compared to the legacy systems and improvements in the timeliness of grant award processing  For example, one of GMM’s goals is to get at least 40 percent of users surveyed to agree or strongly agree that their grants management business processes are easier to accomplish with GMM, compared to the legacy systems  Program officials stated that they plan to work with the Agile development contractor to refine their performance goals and target thresholds, develop a plan for collecting the data and calculating the metrics, and establish a performance baseline with the legacy systems  Program officials also stated that they plan to complete these steps by September 2019—GMM’s initial operational capability date—which is when they are required to begin reporting these metrics to the DHS acquisition review board  According to GAO’s Business Process Reengineering Assessment Guide, agencies undergoing business process reengineering should (1) establish an overall plan to guide the effort (commonly referred to as an organizational change management plan) and (2) provide a common understanding for stakeholders of what to expect and how to plan for process changes  Agencies should develop the plan at the beginning of the reengineering effort and provide specific details on upcoming process changes, such as critical milestones and deliverables for an orderly transition, roles and responsibilities for change management activities, reengineering goals, skills and resource needs, key barriers to change, communication expectations, training, and any staff redeployments or reductions-in-force  The agency should develop and begin implementing its change management plan ahead of introducing new processes to ensure sufficient support among stakeholders for the reengineered processes  While FEMA has begun planning its business process reengineering activities, it has not finalized its plans or established time frames for their completion  Specifically, as of September 2018, program officials were in the process of drafting an organizational change management plan that is intended to establish an approach for preparing grants management stakeholders for upcoming changes  According to FEMA, this document is intended to help avoid uncertainty and confusion among stakeholders as changes are made to the agency’s grant programs, and ensure successful adoption of new business processes, strategies, and technologies  As discussed previously in this report, the transition to GMM will involve changes to FEMA’s disparate grants management processes that are managed by many different stakeholders across the agency  Program officials acknowledged that change management is the biggest challenge they face in implementing GMM and said they had begun taking several actions intended to support the agency’s change management activities  For example, program officials reported in October 2018 that they had recently created an executive-level working group intended to address FEMA’s policy challenges related to the standardization of grants management processes  Additionally, program officials reported that they planned to: (1) hire additional support staff focused on coordinating grants change management activities; and (2) pursue regional office outreach to encourage broad support among GMM’s decentralized stakeholders, such as state, local, and tribal territories  However, despite these actions, the officials were unable to provide time frames for completing the organizational change management plan or the additional actions  Until the plan and actions are complete, the program lacks assurance that it will have sufficient support among stakeholders for the reengineered processes  In addition, GMM did not establish plans and time frames for the activities that needed to take place prior to, during, and after the transition from the legacy AFG to GMM  Instead, program officials stated that they had worked collaboratively with the legacy AFG program and planned these details informally by discussing them in various communications, such as emails and meetings  However, this informal planning approach is not a repeatable process, which is essential to this program as FEMA plans to transition many sets of functionality to many different users during the lifecycle of this program  Program officials acknowledged that for future transitions they will need more repeatable transition planning and stated that they intend to establish such plans, but did not provide a time frame for when such changes would be made  Until FEMA develops a repeatable process, with established time frames for communicating the transition details to its customers prior to each transition, the agency risks that the transition from the legacy systems to GMM will not occur as intended  It also increases its risk that stakeholders will not support the implementation of reengineered grants management processes  Leading practices for software development efforts state that IT requirements are to be clearly defined and prioritized  This includes, among other things, maintaining bidirectional traceability as the requirements evolve, to ensure there are no inconsistencies among program plans and requirements  In addition, programs using Agile software development are to maintain a product vision, or roadmap, to guide the planning of major program milestones and provide a high-level view of planned requirements  Programs should also maintain a prioritized list (referred to as a backlog) of narrowly defined requirements (referred to as lower-level requirements) that are to be delivered  Programs should maintain this backlog with the product owner to ensure the program is always working on the highest priority requirements that will deliver the most value to the users  The GMM program established clearly defined and prioritized requirements and maintained bidirectional traceability among the various levels of requirements: Grant lifecycle phases: In its Concept of Operations document, the program established six grants management lifecycle phases that represent the highest level of GMM’s requirements, through which it derives lower-level requirements  Business functions: The Concept of Operations document also identifies the next level of GMM requirements—the 43 business functions that describe how FEMA officials, grant recipients, and other stakeholders are to manage grants  According to program officials, the 43 business functions are to be refined, prioritized, and delivered to GMM customers iteratively  Further, for the AFG pilot, the GMM program office prioritized 19 business functions with the product owner and planned the development of these functions in a roadmap  Epics: GMM’s business functions are decomposed into epics, which represent smaller portions of functionality that can be developed over multiple increments  According to program officials, GMM intends to develop, refine, and prioritize the epics iteratively  As of August 2018, the program had developed 67 epics in the program backlog  An example of one of the epics for the AFG pilot is to prepare and submit grant closeout materials  User stories: The epics are decomposed into user stories, which convey the customers’ requirements at the smallest and most discrete unit of work that must be done within a single sprint to create working software  GMM develops, refines, and prioritizes the user stories iteratively  As of August 2018, the program had developed 1,118 user stories in the backlog  An example of a user story is “As an external user, I can log in with a username and password ” Figure 6 provides an example of how GMM’s different levels of requirements are decomposed  Nevertheless, while we found requirements to be traceable at the sprint- level (i e , epics and user stories), traceability of requirements at the increment-level (i e , business functions) were inconsistent among different requirements planning documents  Specifically, the capabilities and constraints document shows that five business functions are planned to be developed within sub-increment 1A, whereas the other key planning document—the roadmap for the AFG pilot—showed one of those five functions as being planned for the sub-increment 1B  In addition, the capabilities and constraints document shows that nine business functions are planned to be developed within sub-increment 1B, but the roadmap showed one of those nine functions as being planned for the sub- increment 1C  Program officials stated that they decided to defer these functions to later sub-increments due to unexpected technical difficulties encountered when developing functionality and reprioritizing functions with the product owners  While the officials updated the roadmap to reflect the deferred functionality, they did not update the capabilities and constraints document to maintain traceability between these two important requirements planning documents  Program officials stated that they learned during the AFG pilot that the use of a capabilities and constraints document for increment-level scope planning was not ideal and that they intended to change the process for how they documented planned requirements for future increments  However, program officials did not provide a time frame for when this change would be made  Until the program makes this change and then ensures it maintains traceability of increment-level requirements between requirements planning documents, it will continue to risk confusion among stakeholders about what is to be delivered  In addition, until recently, GMM’s planning documents were missing up- to-date information regarding when most of the legacy systems will be transitioned to GMM  Specifically, while the program’s planning documents (including the GMM roadmap) provided key milestones for the entire lifecycle of the program and high-level capabilities to be delivered in the AFG pilot, these documents lacked up-to-date time frames for when FEMA planned to transition the nine remaining legacy systems  For example, in May 2017, GMM drafted notional time frames for transitioning the legacy systems, including plans for AFG to be the seventh system replaced by GMM  However, in December 2017, the program decided to reprioritize the legacy systems so that AFG would be replaced first—yet this major change was not reflected in the program’s roadmap  Moreover, while AFG program officials were informed of the decision to transition the AFG program first, in June 2018 officials from other grant programs told us that they had not been informed on when their systems were to be replaced  As a result, these programs were uncertain about when they should start planning for their respective transitions  In August 2018, GMM program officials acknowledged that they were delayed in deciding the sequencing order for the legacy system transitions  Program officials stated that the delay was due to their need to factor the Agile development contractor’s perspective into these decisions; yet, at that time, the contract award had been delayed by approximately 8 months  Subsequently, in October 2018, program officials identified tentative time frames for transitioning the remaining legacy systems  Program officials stated that they determined the tentative time frames for transitioning the legacy systems based on key factors, such as mission need, cost, security vulnerabilities, and technical obsolescence, and that they had shared these new time frames with grant program officials  The officials also stated that, once the Agile contractor begins contract performance, they expect to be able to validate the contractor’s capacity and finalize these time frames by obtaining approval from the Grants Management Executive Steering Group  By taking steps to update and communicate these important time frames, FEMA should be better positioned to ensure that each of the grant programs are prepared for transitioning to GMM  According to leading practices, Agile programs should track their progress in delivering planned IT requirements within a sprint (i e , short iterations that produce working software)  Given that sprints are very short cycles of development (e g , 2 weeks), the efficiency of completing planned work within a sprint relies on a disciplined approach that includes using a fixed pace, referred to as the sprint cadence, that provides a consistent and predictable development routine  A disciplined approach also includes identifying by the start of a sprint which user stories will be developed, developing those stories to completion (e g , fully tested and demonstrated to, and accepted by, the product owner), and tracking completion progress of those stories  Progress should be communicated to relevant stakeholders and used by the development teams to better understand their capacity to develop stories, continuously improve on their processes, and forecast how long it will take to deliver all remaining capabilities  The GMM program did not effectively track progress in delivering IT requirements during the first nine sprints, which occurred from January to June 2018  These gaps in tracking the progress of requirements, in part, had an impact on the program’s progress in delivering the 19 AFG business functions that were originally planned by December 2018 and are now deferred to August 2019  However, beginning in July 2018, in response to our ongoing review, the program took steps to improve in these areas  Specifically, GMM did not communicate the status of its Agile development progress to program stakeholders, such as the grant programs, the regional offices, and the development teams, during most of the first nine sprints  Program officials acknowledged that they should use metrics to track development progress and, in July 2018, they began reporting metrics to program stakeholders  For example, they began collecting and providing data on the number of stories planned and delivered, estimated capacity for development teams, and the number of days spent working on the sprint, as part of the program’s weekly status reports to program stakeholders, such as product owners  Rather than using a fixed, predictable sprint cadence, GMM allowed a variable development cadence, meaning that sprint durations varied from 1 to 4 weeks throughout the first nine sprints  Program officials noted that they had experimented with the use of a variable cadence to allow more time to complete complex technical work  Program officials stated that they realized that varying the sprints was not effective and, in July 2018 for sprint 10, they reverted back to a fixed, 2 week cadence  GMM added a significant amount of scope during its first nine sprints, after the development work had already begun  For example, the program committed to 28 user stories at the beginning of sprint eight, and then nearly doubled the work by adding 25 additional stories in the middle of the sprint  Program officials cited multiple reasons for adding more stories, including that an insufficient number of stories had been defined in the backlog when the sprint began, the realization that planned stories were too large and needed to be decomposed into smaller stories, and the realization that other work would be needed in addition to what was originally planned  Program officials recognized that, by the start of a sprint, the requirements should be sufficiently defined, such that they are ready for development without requiring major changes during the sprint  The program made recent improvements in sprints 11 and 12, which had only five stories added after the start of a sprint  By taking these steps to establish consistency among sprints, the program has better positioned itself to more effectively monitor and manage the remaining IT development work  In addition, this improvement in consistency should help the program avoid future deferments of functionality  Leading practices state that programs should regularly collaborate with, and collect input from, relevant stakeholders; monitor the status of stakeholder involvement; incorporate stakeholder input; and measure how well stakeholders’ needs are being met  For Agile programs, it is especially important to track user satisfaction to determine how well the program has met stakeholders’ needs  Consistent stakeholder participation ensures that the program meets its stakeholders’ needs  FEMA implemented its responsibilities in this area through several means, such as stakeholder outreach activities; development of a strategic communications plan; and continuous monitoring, solicitation, and recording of stakeholder involvement and feedback  For example, the agency conducted nationwide outreach sessions from January 2016 through August 2017 and began conducting additional outreach sessions in April 2018  These outreach sessions involved hundreds of representatives from FEMA headquarters, the 10 regional offices, and state and local grant recipients to collect information on the current grants management environment and opportunities for streamlining grants management processes  FEMA also held oversight and stakeholder outreach activities and actively solicited and recorded feedback from its stakeholders on a regular basis  For example, GMM regularly verified with users that the new functionality met their IT requirements, as part of the Agile development cycle  Additionally, we observed several GMM biweekly requirements validation sessions where the program’s stakeholders were involved and provided feedback as part of the requirements development and refinement process  In addition, FEMA identified GMM stakeholders and tracked its engagement with these stakeholders using a stakeholder register  The agency also defined processes for how the GMM program is to collaborate with its stakeholders in a stakeholder communication plan and Agile development team agreement  Also, while several officials from the selected grant program and regional offices that we interviewed indicated that the program could improve in communicating its plans for GMM and incorporating stakeholder input, most of the representatives from these offices stated that GMM is doing well at interacting with its stakeholders  Finally, in October 2018, program officials reported that they had recently begun measuring user satisfaction by conducting surveys and interviews with users that have utilized the new functionality within GMM  The program’s outreach activities, collection of stakeholder input, and measurement of user satisfaction demonstrate that the program is taking the appropriate steps to incorporate stakeholder input  Reliable cost estimates are critical for successfully delivering IT programs  Such estimates provide the basis for informed decision making, realistic budget formulation, meaningful progress measurement, and accountability for results  GAO’s Cost Estimating and Assessment Guide defines leading practices related to the following four characteristics of a high-quality, reliable estimate  Comprehensive


Report 2:  The estimate accounts for all possible costs associated with a program, is structured in sufficient detail to ensure that costs are neither omitted nor double counted, and documents all cost-influencing assumptions  Well-documented  Supporting documentation explains the process, sources, and methods used to create the estimate; contains the underlying data used to develop the estimate; and is adequately reviewed and approved by management  Accurate  The estimate is not overly conservative or optimistic, is based on an assessment of the costs most likely to be incurred, and is regularly updated so that it always reflects the program’s current status  Credible  Discusses any limitations of the analysis because of uncertainty or sensitivity surrounding data or assumptions, the estimate’s results are cross-checked, and an independent cost estimate is conducted by a group outside the acquiring organization to determine whether other estimating methods produce similar results  In May 2017, DHS approved GMM’s lifecycle cost estimate of about $251 million for fiscal years 2015 through 2030  We found this initial estimate to be reliable because it fully or substantially addressed all the characteristics associated with a reliable cost estimate  For example, the estimate comprehensively included government and contractor costs, all elements of the program’s work breakdown structure, and all phases of the system lifecycle; and was aligned with the program’s technical documentation at the time the estimate was developed  GMM also fully documented the key assumptions, data sources, estimating methodology, and calculations for the estimate  Further, the program conducted a risk assessment and sensitivity analysis, and DHS conducted an independent assessment of the cost estimate to validate the accuracy and credibility of the cost estimate  However, key assumptions that FEMA made about the program changed soon after DHS approved the cost estimate in May 2017  Thus, the initial cost estimate no longer reflects the current approach for the program  For example, key assumptions about the program that changed include: Change in the technical approach: The initial cost estimate assumed that GMM would implement a software-as-a-service model, meaning that FEMA would rely on a service provider to deliver software applications and the underlying infrastructure to run them  However, in December 2017, the program instead decided to implement an infrastructure-as-a-service model, meaning that FEMA would develop and deploy its own software application and rely on a service provider to deliver and manage the computing infrastructure (e g , servers, software, storage, and network equipment)  According to program officials, this decision was made after learning from the Agile prototypes that the infrastructure-as-a-service model would allow GMM to develop the system in a more flexible environment  Increase in the number of system development personnel: A key factor with Agile development is the number of development teams (each consisting of experts in software development, testing, and cybersecurity) that are operating concurrently and producing separate portions of software functionality  Program officials initially assumed that they would need three to four concurrent Agile development teams, but subsequently realized that they would instead need to expend more resources to achieve GMM’s original completion date  Specifically, program officials now expect they will need to at least double, and potentially triple, the number of concurrent development teams to meet GMM’s original target dates  Significant delays and complexities with data migration: In 2016 and 2017, GMM experienced various technical challenges in its effort to transfer legacy system data to a data staging platform  This data transfer effort needed to be done to standardize the data before eventually migrating the data to GMM  These challenges resulted in significant delays and cost increases  Program officials reported that, by February 2018—at least 9 months later than planned—all legacy data had been transferred to a data staging platform so that FEMA officials could begin analyzing and standardizing the data prior to migrating it into GMM  FEMA officials reported that they anticipated the cost estimate to increase, and for this increase to be high enough to breach the $251 million threshold set in GMM’s May 2017 acquisition program baseline  Thus, consistent with DHS’s acquisition guidance, the program informed the DHS acquisition review board of this anticipated breach  The board declared that the program was in a cost breach status, as of September 12, 2018  As of October 2018, program officials stated that they were in the process of revising the cost estimate to reflect the changes in the program and to incorporate actual costs  In addition, the officials stated that the program was applying a new cost estimating methodology tailored for Agile programs that DHS’s Cost Analysis Division had been developing  In December 2018, program officials stated that they had completed the revised cost estimate but it was still undergoing departmental approval  Establishing an updated cost estimate should help FEMA better understand the expected costs to deliver GMM under the program’s current approach and time frames  The success of an IT program depends, in part, on having an integrated and reliable master schedule that defines when the program’s set of work activities and milestone events are to occur, how long they will take, and how they are related to one another  Among other things, a reliable schedule provides a roadmap for systematic execution of an IT program and the means by which to gauge progress, identify and address potential problems, and promote accountability  GAO’s Schedule Assessment Guide defines leading practices related to the following four characteristics that are vital to having a reliable integrated master schedule  Comprehensive  A comprehensive schedule reflects all activities for both the government and its contractors that are necessary to accomplish a program’s objectives, as defined in the program’s work breakdown structure  The schedule also includes the labor, materials, and overhead needed to do the work and depicts when those resources are needed and when they will be available  It realistically reflects how long each activity will take and allows for discrete progress measurement  Well-constructed  A schedule is well-constructed if all of its activities are logically sequenced with the most straightforward logic possible  Unusual or complicated logic techniques are used judiciously and justified in the schedule documentation  The schedule’s critical path represents a true model of the activities that drive the program’s earliest completion date and total float accurately depicts schedule flexibility  Credible  A schedule that is credible is horizontally traceable—that is, it reflects the order of events necessary to achieve aggregated products or outcomes  It is also vertically traceable—that is, activities in varying levels of the schedule map to one another and key dates presented to management in periodic briefings are consistent with the schedule  Data about risks are used to predict a level of confidence in meeting the program’s completion date  The level of necessary schedule contingency and high-priority risks are identified by conducting a robust schedule risk analysis  Controlled  A schedule is controlled if it is updated regularly by trained schedulers using actual progress and logic to realistically forecast dates for program activities  It is compared to a designated baseline schedule to measure, monitor, and report the program’s progress  The baseline schedule is accompanied by a baseline document that explains the overall approach to the program, defines ground rules and assumptions, and describes the unique features of the schedule  The baseline schedule and current schedule are subject to a configuration management control process  GMM’s schedule was unreliable because it minimally addressed three characteristics—comprehensive, credible, and controlled—and did not address the fourth characteristic of a reliable estimate—well-constructed  One of the most significant issues was that the program’s fast approaching, final delivery date of September 2020 was not informed by a realistic assessment of GMM development activities, and rather was determined by imposing an unsubstantiated delivery date  Table 4 summarizes our assessment of GMM’s schedule  In discussing the reasons for the shortfalls in these practices, program officials stated that they had been uncertain about the level of rigor that should be applied to the GMM schedule, given their use of Agile development  However, leading practices state that program schedules should meet all the scheduling practices, regardless of whether a program is using Agile development  As discussed earlier in this report, GMM has already experienced significant schedule delays  For example, the legacy data migration effort, the AFG pilot, and the Agile development contract have been delayed  Program officials also stated that the delay in awarding and starting the Agile contract has delayed other important activities, such as establishing time frames for transitioning legacy systems  A more robust schedule could have helped FEMA predict the impact of delays on remaining activities and identify which activities appeared most critical so that the program could ensure that any risks in delaying those activities were properly mitigated  In response to our review and findings, program officials recognized the need to continually enhance their schedule practices to improve the management and communication of program activities  As a result, in August 2018, the officials stated that they planned to add a master scheduler to the team to improve the program’s schedule practices and ensure that all of the areas of concern we identified are adequately addressed  In October 2018, the officials reported that they had recently added two master schedulers to GMM  According to the statement of objectives, the Agile contractor is expected to develop an integrated master schedule soon after it begins performance  However, program officials stated that GMM is schedule-driven—due to the Executive Steering Group’s expectation that the solution will be delivered by September 2020  The officials added that, if GMM encounters challenges in meeting this time frame, the program plans to seek additional resources to allow it to meet the 2020 target  GMM’s schedule-driven approach has already led to an increase in estimated costs and resources  For example, as previously mentioned, the program has determined that, to meet its original target dates, GMM needs to at least double, and possibly triple, the number of concurrent Agile development teams  In addition, we have previously reported that schedule pressure on federal IT programs can lead to omissions and skipping of key activities, especially system testing  In August 2018, program officials acknowledged that September 2020 may not be feasible and that the overall completion time frames established in the acquisition program baseline may eventually need to be rebaselined  Without a robust schedule to forecast whether FEMA’s aggressive delivery goal for GMM is realistic to achieve, leadership will be limited in its ability to make informed decisions on what additional increases in cost or reductions in scope might be needed to fully deliver the system  NIST’s risk management framework establishes standards and guidelines for agencies to follow in developing cybersecurity programs  Agencies are expected to use this framework to achieve more secure information and information systems through the implementation of appropriate risk mitigation strategies and by performing activities that ensure that necessary security controls are integrated into agencies’ processes  The framework addresses broad cybersecurity and risk management activities, which include the following: Categorize the system: Programs are to categorize systems by identifying the types of information used, selecting a potential impact level (e g , low, moderate, or high), and assigning a category based on the highest level of impact to the system’s confidentiality, integrity, and availability, if the system was compromised  Programs are also to document a description of the information system and its boundaries and should register the system with appropriate program management offices  System categorization is documented in a system security plan  Select and implement security controls: Programs are to determine protective measures, or security controls, to be implemented based on the system categorization results  These security controls are documented in a system security plan  For example, control areas include access controls, incident response, security assessment and authorization, identification and authentication, and configuration management  Once controls are identified, programs are to determine planned implementation actions for each of the designated controls  These implementation actions are also specified in the system security plan  Assess security controls: Programs are to develop, review, and approve a security assessment plan  The purpose of the security assessment plan approval is to establish the appropriate expectations for the security control assessment  Programs are to also perform a security control assessment by evaluating the security controls in accordance with the procedures defined in the security assessment plan, in order to determine the extent to which the controls were implemented correctly  The output of this process is intended to produce a security assessment report to document the issues, findings, and recommendations  Programs are to conduct initial remediation actions on security controls and reassess those security controls, as appropriate  Obtain an authorization to operate the system: Programs are to obtain security authorization approval in order to operate a system  Resolving weaknesses and vulnerabilities identified during testing is an important step leading up to achieving an authorization to operate  Programs are to establish corrective action plans to address any deficiencies in cybersecurity policies, procedures, and practices  DHS guidance also states that corrective action plans must be developed for every weakness identified during a security control assessment and within a security assessment report  Monitor security controls on an ongoing basis: Programs are to monitor their security controls on an ongoing basis after deployment, including determining the security impact of proposed or actual changes to the information system and assessing the security controls in accordance with a monitoring strategy that determines the frequency of monitoring the controls  For the GMM program’s engineering and test environment, which went live in February 2018, FEMA fully addressed three of the five key cybersecurity practices in NIST’s risk management framework and partially addressed two of the practices  Specifically, FEMA categorized GMM’s environment based on security risk, implemented select security controls, and monitored security controls on an ongoing basis  However, the agency partially addressed the areas of assessing security controls and obtaining an authorization to operate the system  Table 5 provides a summary of the extent to which FEMA addressed NIST’s key cybersecurity practices for GMM’s engineering and test environment  Consistent with NIST’s framework, GMM categorized the security risk of its engineering and test environment and identified it as a moderate- impact environment  A moderate-impact environment is one where the loss of confidentiality, integrity, or availability could be expected to have a serious or adverse effect on organizational operations, organizational assets, or individuals  GMM completed the following steps leading to this categorization: The program documented in its System Security Plan the various types of data and information that the environment will collect, process, and store, such as conducting technology research, building or enhancing technology, and maintaining IT networks  The program established three information types and assigned security levels of low, moderate, or high impact in the areas of confidentiality, availability, and integrity  A low-impact security level was assigned to two information types: (1) conducting technology research and (2) building or enhancing technology; and a moderate- impact security level was assigned to the third information type: maintaining IT networks  The engineering and test environment was categorized as an overall moderate-impact system, based on the highest security impact level assignment  GMM documented a description of the environment, including a diagram depicting the system’s boundaries, which illustrates, among other things, databases and firewalls  GMM properly registered its engineering and test environment with FEMA’s Chief Information Officer, Chief Financial Officer, and acting Chief Information Security Officer  By conducting the security categorization process, GMM has taken steps that should ensure that the appropriate security controls are selected for the program’s engineering and test environment  Consistent with NIST’s framework and the system categorization results, GMM appropriately determined which security controls to implement and planned actions for implementing those controls in its System Security Plan for the engineering and test environment  For example, the program utilized NIST guidance to select standard controls for a system categorized with a moderate-impact security level  These control areas include, for example, access controls, risk assessment, incident response, identification and authentication, and configuration management  Further, the program documented its planned actions to implement each control in its System Security Plan  For example, GMM documented that the program plans to implement its Incident Response Testing control by participating in an agency-wide exercise and unannounced vulnerability scans  As another example, GMM documented that the program plans to implement its Contingency Plan Testing control by testing the contingency plan annually, reviewing the test results, and preparing after action reports  By selecting and planning for the implementation of security controls, GMM has taken steps to mitigate its security risks and protect the confidentiality, integrity, and availability of the information system  Consistent with NIST’s framework, in January 2018, GMM program officials developed a security assessment plan for the engineering and test environment  According to GMM program officials, this plan was reviewed by the security assessment team  However, the security assessment plan lacked essential details  Specifically, while the plan included the general process for evaluating the environment’s security controls, the planned assessment procedures for all 964 security controls were not sufficiently defined  Specifically, GMM program officials copied example assessment procedures from NIST guidance and inserted them into its security assessment documentation for all of its 964 controls, without making further adjustments to explain the steps that should be taken specific to GMM  Table 6 shows an example of a security assessment procedure copied from the NIST guidance that should have been further adjusted for GMM  In addition, the actual assessment procedures that the GMM assessors used to evaluate the security controls were not documented  Instead, the program only documented whether each control passed or failed each test  GMM program officials stated that the planned assessment procedures are based on an agency template that was exported from a DHS compliance tool, and that FEMA security officials have been instructed by the DHS OCIO not to tailor or make any adjustments to the template language  However, the assessment procedures outlined in NIST’s guidance are to serve as a starting point for organizations preparing their program specific assessments  According to NIST, organizations are expected to select and tailor their assessment procedures for each security control from NIST’s list of suggested assessment options (e g , review, analyze, or inspect policies, procedures, and related documentation options)  DHS OCIO officials stated that, consistent with NIST’s guidance, they expect that components will ensure they are in compliance with the minimum standards and will also add details and additional rigor, as appropriate, to tailor the planned security assessment procedures to fit their unique missions or needs  In November 2018, in response to our audit, DHS OCIO officials stated that they were meeting with FEMA OCIO officials to understand why they did not document the planned and actual assessment procedures performed by the assessors for GMM  Until FEMA ensures that detailed planned evaluation methods and actual evaluation procedures specific to GMM are defined, the program risks assessing security controls incorrectly, having controls that do not work as intended, and producing undesirable outcomes with respect to meeting the security requirements  In addition, the security assessment plan was not approved by FEMA’s OCIO before proceeding with the security assessment  Program officials stated that approval was not required for the security assessment plan prior to the development of the security assessment report  However, NIST guidance states that the purpose of the security assessment plan approval is to establish the appropriate expectations for the security control assessment  By not getting the security assessment plan approved by FEMA’s OCIO before security assessment reviews were conducted, GMM risks inconsistencies with the plan and security objectives of the organization  Finally, consistent with NIST guidance, GMM performed a security assessment in December 2017 of the engineering and test environment’s controls, which identified 36 vulnerabilities (23 critical- and high-impact vulnerabilities and 13 medium- and low-impact vulnerabilities)  The program also documented these vulnerabilities and associated findings and recommendations in a security assessment report  GMM conducted initial remediation actions (i e , remediation of vulnerabilities that should be corrected immediately) for 12 of the critical- and high-impact vulnerabilities and a reassessment of those security controls confirmed that they were resolved by January 2018  Remediation of the remaining 11 critical- and high-impact vulnerabilities and 13 medium- and low- impact vulnerabilities were to be addressed by corrective action plans as part of the authorization to operate process, which is discussed in the next section  The authorization to operate GMM’s engineering and test environment was granted on February 5, 2018  Among other things, this decision was based on the important stipulation that the remaining 11 critical- and high- impact vulnerabilities associated with multifactor authentication would be addressed within 45 days, or by March 22, 2018  However, the program did not meet this deadline and, instead, approximately 2 months after this deadline passed, obtained a waiver to remediate these vulnerabilities by May 9, 2019  These vulnerabilities are related to a multifactor authentication capability  Program officials stated that they worked with FEMA OCIO officials to attempt to address these vulnerabilities by the initial deadline, but they were unsuccessful in finding a viable solution  Therefore, GMM program officials developed a waiver at the recommendation of the OCIO to provide additional time to develop a viable solution  However, a multifactor authentication capability is essential to ensuring that users are who they say they are, prior to granting users access to the GMM engineering and test environment, in order to reduce the risk of harmful actors accessing the system  In addition, as of September 2018, the program had not established corrective action plans for the 13 medium- and low-impact vulnerabilities  Program officials stated that they do not typically address low-impact vulnerabilities; however, this is in conflict with DHS guidance that specifies that corrective action plans must be developed for every weakness identified during a security control assessment and within a security assessment report  In response to our audit, in October 2018, GMM program officials developed these remaining corrective action plans  The plans indicated that these vulnerabilities were to be fully addressed by January 2019 and April 2019  While the program eventually took corrective actions in response to our audit by developing the missing plans, the GMM program initially failed to follow DHS’s guidance on preparing corrective actions plans for all security vulnerabilities  Until GMM consistently follows DHS’s guidance, it will be difficult for FEMA to determine the extent to which GMM’s security weaknesses identified during its security control assessments are remediated  Additionally, as we have reported at other agencies, vulnerabilities can be indicators of more significant underlying issues and, thus, without appropriate management attention or prompt remediation, GMM is at risk of unnecessarily exposing the program to potential exploits  Moreover, GMM was required to assess all untested controls by March 7, 2018, or no later than 30 days after the approval of the authorization to operate; however, it did not meet this deadline  Specifically, we found that, by October 2018, FEMA had not fully tested 190 security controls in the GMM engineering and test environment  These controls were related to areas such as security incident handling and allocation of resources required to protect an information system  In response to our findings, in October 2018, GMM program officials reported that they had since fully tested 27 controls and partially tested the remaining 163 controls  Program officials stated that testing of the 163 controls is a shared responsibility between GMM and other parties (e g , the cloud service provider)  They added that GMM had completed its portion of the testing but was in the process of verifying the completion of testing by other parties  Program officials stated that the untested controls were not addressed sooner, in part, because of errors resulting from configuration changes in the program’s compliance tool during a system upgrade, which have now been resolved  Until GMM ensures that all security controls have been tested, it remains at an increased risk of exposing programs to potential exploits  Consistent with the NIST framework, GMM established methods for assessing and monitoring security controls to be conducted after an authorization to operate has been approved  GMM has tailored its cybersecurity policies and practices for monitoring its controls to take into account the frequent and iterative pace with which system functionality is continuously being introduced into the GMM environment  Specifically, the GMM program established a process for assessing security impact changes to the system and conducting reauthorizations to operate within the rapid Agile delivery environment  As part of this process, GMM embedded cybersecurity experts on each Agile development team so that they are involved early and can impact security considerations from the beginning of requirements development through testing and deployment of system functionality  In addition, the process involves important steps for ensuring that the system moves from development to completion, while producing a secure and reliable system  For example, it includes procedures for creating, reviewing, and testing new system functionality  As the new system functionality is integrated with existing system functionality, it is to undergo automated testing and security scans in order to ensure that the integrity of the security of the system has not been compromised  Further, an automated process is to deploy the code if it passes all security scans, code tests, and code quality checks  GMM’s process for conducting a reauthorization to operate within the rapid delivery Agile development environment is to follow FEMA guidance that states that all high-level changes made to a FEMA IT system must receive approval from both a change advisory board and the FEMA Chief Information Officer  The board and FEMA Chief Information Officer are to focus their review and approval on scheduled releases and epics (i e , collections of user stories)  Additionally, the Information System Security Officer is to review each planned user story and, if it is determined that the proposed changes may impact the integrity of the authorization, the Information System Security Officer is to work with the development team to begin the process of updating the system authorization  Finally, GMM uses automated tools to track the frequency in which security controls are assessed and to ensure that required scanning data are received by FEMA for reporting purposes  Program officials stated that, in the absence of department-level and agency-level guidance, they have coordinated with DHS and FEMA OCIO officials to ensure that these officials are in agreement with GMM’s approach to continuous monitoring  By having monitoring control policies and procedures in place, FEMA management is positioned to more effectively prioritize and plan its risk response to current threats and vulnerabilities for the GMM program  Given FEMA’s highly complex grants management environment, with its many stakeholders, IT systems, and internal and external users, implementing leading practices for business process reengineering and IT requirements management is critical for success  FEMA has taken many positive steps, including ensuring executive leadership support for business process reengineering, documenting the agency’s grants management processes and performance improvement goals, defining initial IT requirements for the program, incorporating input from end user stakeholders into the development and implementation process, and taking recent actions to improve its delivery of planned IT requirements  Nevertheless, until the GMM program finalizes plans and time frames for implementing its organizational change management actions, plans and communicates system transition activities, and maintains clear traceability of IT requirements, FEMA will be limited in its ability to provide streamlined grants management processes and effectively deliver a modernized IT system to meet the needs of its large range of users  While GMM’s initial cost estimate was reliable, key assumptions about the program since the initial estimate had changed and, therefore, it no longer reflected the current approach for the program  The forthcoming updated cost schedule is expected to better reflect the current approach  However, the program’s unreliable schedule to fully deliver GMM by September 2020 is aggressive and unrealistic  The delays the program has experienced to date further compound GMM’s schedule issues  Without a robust schedule that has been informed by a realistic assessment of GMM’s development activities, leadership will be limited in its ability to make informed decisions on what additional increases in cost or reductions in scope might be needed to achieve their goals  Further, FEMA’s implementation of cybersecurity practices for GMM in the areas of system categorization, selection and implementation, and monitoring will help the program  However, GMM lacked essential details for evaluating security controls, did not approve the security assessment plan before proceeding with the security assessment, did not follow DHS’s guidance to develop corrective action plans for all security vulnerabilities, and did not fully test all security controls  As a result, the GMM engineering and test environment remains at an increased risk of exploitations  We are making eight recommendations to FEMA: The FEMA Administrator should ensure that the GMM program management office finalizes the organizational change management plan and time frames for implementing change management actions  (Recommendation 1) The FEMA Administrator should ensure that the GMM program management office plans and communicates its detailed transition activities to its affected customers before they transition to GMM and undergo significant changes to their processes  (Recommendation 2) The FEMA Administrator should ensure that the GMM program management office implements its planned changes to its processes for documenting requirements for future increments and ensures it maintains traceability among key IT requirements documents  (Recommendation 3) The FEMA Administrator should ensure that the GMM program management office updates the program schedule to address the leading practices for a reliable schedule identified in this report  (Recommendation 4) The FEMA Administrator should ensure that the FEMA OCIO defines sufficiently detailed planned evaluation methods and actual evaluation methods for assessing security controls  (Recommendation 5) The FEMA Administrator should ensure that the FEMA OCIO approves a security assessment plan before security assessment reviews are conducted  (Recommendation 6) The FEMA Administrator should ensure that the GMM program management office follows DHS guidance on preparing corrective action plans for all security vulnerabilities  (Recommendation 7) The FEMA Administrator should ensure that the GMM program management office fully tests all of its security controls for the system  (Recommendation 8) DHS provided written comments on a draft of this report, which are reprinted in appendix IV  In its comments, the department concurred with all eight of our recommendations and provided estimated completion dates for implementing each of them  For example, with regard to recommendation 4, the department stated that FEMA plans to update the GMM program schedule to address the leading practices for a reliable schedule by April 30, 2019  In addition, for recommendation 7, the department stated that FEMA plans to ensure that corrective action plans are prepared by July 31, 2019, to address all identified security vulnerabilities for GMM  If implemented effectively, the actions that FEMA plans to take in response to the recommendations should address the weaknesses we identified  We also received technical comments from DHS and FEMA officials, which we incorporated, as appropriate  We are sending copies of this report to the Secretary of Homeland Security and interested congressional committees  In addition, the report will be available at no charge on the GAO website at http://www gao gov  If you or your staff have any questions about this report, please contact me at (202) 512-4456 or harriscc@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report  GAO staff who made key contributions to this report are listed in appendix V  Our objectives were to (1) determine the extent to which the Federal Emergency Management Agency (FEMA) is implementing leading practices for reengineering its grants management business processes and incorporating business needs into Grants Management Modernization (GMM) information technology (IT) requirements; (2) assess the reliability of the program’s estimated costs and schedule; and (3) determine the extent to which FEMA is addressing key cybersecurity practices for GMM  To address the first objective, we reviewed GAO’s Business Process Reengineering Assessment Guide and Software Engineering Institute’s Capability Maturity Model for Integration for Development to identify practices associated with business process reengineering and IT requirements management  We then selected six areas that, in our professional judgment, represented foundational practices that were of particular importance to the successful implementation of an IT modernization effort that is using Agile development processes  We also selected the practices that were most relevant based on where GMM was in the system development lifecycle and we discussed the practice areas with FEMA officials  The practices are: Ensuring executive leadership support for process reengineering Assessing the current and target business environment and business Establishing plans for implementing new business processes Establishing clear, prioritized, and traceable IT requirements Tracking progress in delivering IT requirements Incorporating input from end user stakeholders We also reviewed selected chapters of GAO’s draft Agile Assessment Guide (Version 6A), which is intended to establish a consistent framework based on best practices that can be used across the federal government for developing, implementing, managing, and evaluating agencies’ IT investments that rely on Agile methods  To develop this guide, GAO worked closely with Agile experts in the public and private sector; some chapters of the guide are considered more mature because they have been reviewed by the expert panel  We reviewed these chapters to ensure that our expectations for how FEMA should apply the six practices for business process reengineering and IT requirements management are appropriate for an Agile program and are consistent with the draft guidance that is under development  Additionally, since Agile development programs may use different terminology to describe their software development processes, the Agile terms used in this report (e g , increment, sprint, epic, etc ) are specific to the GMM program  We obtained and analyzed FEMA grants management modernization documentation, such as current and target grants management business processes, acquisition program baseline, operational requirements document, concept of operations, requirements analyses workbooks, Grants Management Executive Steering Group artifacts, stakeholder outreach artifacts, Agile increment- and sprint-level planning and development artifacts, and the requirements backlog  We assessed the program documentation against the selected practices to determine the extent to which the agency had implemented them  We then assessed each practice area as: fully implemented—FEMA provided complete evidence that showed it fully implemented the practice area; partially implemented—FEMA provided evidence that showed it partially implemented the practice area; not implemented—FEMA did not provide evidence that showed it implemented any of the practice area  Additionally, we observed Agile increment and sprint development activities at GMM facilities in Washington, D C  We also observed a demonstration of how the program manages its lower level requirements (i e , user stories and epics) and maintains traceability of the requirements using an automated tool at GMM facilities in Washington, D C  We also interviewed FEMA officials, including the GMM Program Executive, GMM Program Manager, GMM Business Transformation Team Lead, and Product Owner regarding their efforts to streamline grants management business processes, collect and incorporate stakeholder input, and manage GMM’s requirements  In addition, we interviewed FEMA officials from four out of 16 grant program offices and two out of 10 regional offices to obtain contextual information and illustrative examples of FEMA’s efforts to reengineer grants management business processes and collect business requirements for GMM  Specifically, We selected the four grant program offices based on a range of grant programs managed, legacy systems used, and the amount of grant funding awarded  We also sought to select a cross section of different characteristics, such as selecting larger grant program offices, as well as smaller offices  In addition, we ensured that our selection included the Assistance to Firefighters Grants (AFG) program office because officials in this office represent the first GMM users and, therefore, are more actively involved with the program’s Agile development practices  Based on these factors, we selected: Public Assistance Division, Individual Assistance Division, AFG, and National Fire Academy  Additionally, the four selected grant program offices are responsible for 16 of the total 45 grant programs and are users of five of the nine primary legacy IT systems  The four selected grant program offices also represent about 68 percent of the total grant funding awarded by FEMA from fiscal years 2005 through 2016  We selected two regional offices based on (1) the largest amount of total FEMA grant funding for fiscal years 2005 through 2016—Region 6 located in Denton, Texas; and (2) the highest percentage of AFG funding compared to the office’s total grant funding awarded from fiscal years 2005 through 2016—Region 5 located in Chicago, Illinois  To assess the reliability of data from the program’s automated IT requirements management tool, we interviewed knowledgeable officials about the quality control procedures used by the program to assure accuracy and completeness of the data  We also compared the data to other relevant program documentation on GMM requirements  We determined that the data used were sufficiently reliable for the purpose of evaluating GMM’s practices for managing IT requirements  For our second objective, to assess the reliability of GMM’s estimated costs and schedule, we reviewed documentation on GMM’s May 2017 lifecycle cost estimate and on the program’s schedule, dated May 2018  To assess the reliability of the May 2017 lifecycle cost estimate, we evaluated documentation supporting the estimate, such as the cost estimating model, the report on GMM’s Cost Estimating Baseline Document and Life Cycle Cost Estimate, and briefings provided to the Department of Homeland Security (DHS) and FEMA management regarding the cost estimate  We assessed the cost estimating methodologies, assumptions, and results against leading practices for developing a comprehensive, accurate, well-documented, and credible cost estimate, identified in GAO’s Cost Estimating and Assessment Guide  We also interviewed program officials responsible for developing and reviewing the cost estimate to understand their methodology, data, and approach for developing the estimate  We found that the cost data were sufficiently reliable  To assess the reliability of the May 2018 GMM program schedule, we evaluated documentation supporting the schedule, such as the integrated master schedule, acquisition program baseline, and Agile artifacts  We assessed the schedule documentation against leading practices for developing a comprehensive, well-constructed, credible, and controlled schedule, identified in GAO’s Schedule Assessment Guide  We also interviewed GMM program officials responsible for developing and managing the program schedule to understand their practices for creating and maintaining the schedule  We noted in our report the instances where the quality of the schedule data impacted the reliability of the program’s schedule  For both the cost estimate and program schedule, we assessed each leading practice as: fully addressed—FEMA provided complete evidence that showed it implemented the entire practice area; substantially addressed—FEMA provided evidence that showed it implemented more than half of the practice area; partially addressed—FEMA provided evidence that showed it implemented about half of the practice area; minimally addressed—FEMA provided evidence that showed it implemented less than half of the practice area; not addressed—FEMA did not provide evidence that showed it implemented any of the practice area  Finally, we provided FEMA with draft versions of our detailed analyses of the GMM cost estimate and schedule  This was done to verify that the information on which we based our findings was complete, accurate, and up-to-date  Regarding our third objective, to determine the extent to which FEMA is addressing key cybersecurity practices for GMM, we reviewed documentation regarding DHS and FEMA cybersecurity policies and guidance, and FEMA’s authorization to operate for the program’s engineering and test environment  We evaluated the documentation against all six cybersecurity practices identified in the National Institute of Standards and Technology’s (NIST) Risk Management Framework  While NIST’s Risk Management Framework identifies six total practices, for reporting purposes, we combined two interrelated practices—selection of security controls and implementation of security controls—into a single practice  The resulting five practices were: categorizing the system based on security risk, selecting and implementing security controls, assessing security controls, obtaining an authorization to operate the system, and monitoring security controls on an ongoing basis  We obtained and analyzed key artifacts supporting the program’s efforts to address these risk management practices, including the program’s System Security Plan, the Security Assessment Plan and Report, Authorization to Operate documentation, and the program’s continuous monitoring documentation  We also interviewed officials from the GMM program office and FEMA’s Office of the Chief Information Officer, such as the GMM Security Engineering Lead, GMM Information System Security Officer, and FEMA’s Acting Chief Information Security Officer, regarding their efforts to assess, document, and review security controls for GMM  We assessed the evidence against the five practices to determine the extent to which the agency had addressed them  We then assessed each practice area as: fully addressed—FEMA provided complete evidence that showed it fully implemented the practice area; partially addressed—FEMA provided evidence that showed it partially implemented the practice area; not addressed—FEMA did not provide evidence that showed it implemented any of the practice area  To assess the reliability of data from the program’s automated security controls management tool, we interviewed knowledgeable officials about the quality control procedures used by the program to assure accuracy and completeness of the data  We also compared the data to other relevant program documentation on GMM security controls for the engineering and test environment  We found that some of the security controls data we examined were sufficiently reliable for the purpose of evaluating FEMA’s cybersecurity practices for GMM, and we noted in our report the instances where the accuracy of the data impacted the program’s ability to address key cybersecurity practices  We conducted this performance audit from December 2017 to April 2019 in accordance with generally accepted government auditing standards  Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives  We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives  The Federal Emergency Management Agency (FEMA) awards many different types of grants to state, local, and tribal governments and nongovernmental entities  These grants are to help communities prevent, prepare for, protect against, mitigate the effects of, respond to, and recover from disasters and terrorist attacks  Agile software development is a type of incremental development that calls for the rapid delivery of software in small, short increments  The use of an incremental approach is consistent with the Office of Management and Budget’s guidance as specified in its information technology (IT) Reform Plan, as well as the legislation commonly referred to as the Federal Information Technology Acquisition Reform Act  Many organizations, especially in the federal government, are accustomed to using a waterfall software development model, which typically consists of long, sequential phases, and differs significantly from the Agile development approach  Agile practices integrate planning, design, development, and testing into an iterative lifecycle to deliver software early and often  Figure 7 provides a depiction of software development using the Agile approach, as compared to a waterfall approach  The frequent iterations of Agile development are intended to effectively measure progress, reduce technical and programmatic risk, and respond to feedback from stakeholders in changes to IT requirements more quickly than traditional methods  Despite these intended benefits, organizations adopting Agile must overcome challenges in making significant changes to how they are accustomed to developing software  The significant differences between Agile and waterfall development impact how IT programs are planned, implemented, and monitored in terms of cost, schedule, and scope  For example, in waterfall development, significant effort is devoted upfront to document detailed plans and all IT requirements for the entire scope of work at the beginning of the program, and cost and schedule can be varied to complete that work  However, for Agile programs the precise details are unknown upfront, so initial planning of cost, scope, and timing would be conducted at a high level, and then supplemented with more specific plans for each iteration  While cost and schedule are set for each iteration, requirements for each iteration (or increment) can be variable as they are learned over time and revised to reflect experiences from completed iterations and to accommodate changing priorities of the end users  The differences in these two software development approaches are shown in figure 8  Looking at figure 8, the benefit provided from using traditional program management practices such as establishing a cost estimate or a robust schedule, is not obvious  However, unlike a theoretical environment, many government programs may not have the autonomy to manage completely flexible scope, as they must deliver certain minimal specifications with the cost and schedule provided  In those cases, it is vital for the team to understand and differentiate the IT requirements that are “must haves” from the “nice to haves” early in the planning effort  This would help facilitate delivery of the “must-haves” requirements first, thereby providing users with the greatest benefits as soon as possible  In addition to the contact named above, the following staff made key contributions to this report: Shannin G  O’Neill (Assistant Director), Jeanne Sung (Analyst in Charge), Andrew Beggs, Rebecca Eyler, Kendrick Johnson, Thomas J  Johnson, Jason Lee, Jennifer Leotta, and Melissa Melvin 

ground_truth: The number of individuals receiving long term care services from Medicaid in community residential settings is expected to grow. These settings, which include assisted living facilities, provide a range of services that allow aged and disabled beneficiaries, who might otherwise require nursing home care, to remain in the community. State Medicaid programs and CMS, the federal agency responsible for overseeing the state programs, share responsibility for ensuring that beneficiaries' health and welfare is protected. GAO was asked to examine state and federal oversight of assisted living services in Medicaid. This report (1) describes state spending on and coverage of these services, (2) describes how state Medicaid agencies oversee the health and welfare of beneficiaries in these settings, and (3) examines the extent that CMS oversees state Medicaid agency monitoring of assisted living services. GAO surveyed all state Medicaid agencies and interviewed officials in a nongeneralizeable sample of three states with varied oversight processes for their assisted living programs. GAO reviewed regulations and guidance, and interviewed CMS officials. State Medicaid agencies in 48 states that covered assisted living services reported spending more than $10 billion (federal and state) on assisted living services in 2014. These 48 states reported covering these services for more than 330,000 beneficiaries through more than 130 different programs. Most programs were operated under Medicaid waivers that allow states to target certain populations, limit enrollment, or restrict services to certain geographic areas. With respect to oversight of their largest assisted living programs, state Medicaid agencies reported varied approaches to overseeing beneficiary health and welfare, particularly in how they monitored critical incidents involving beneficiaries receiving assisted living services. State Medicaid agencies are required to protect beneficiary health and welfare and operate systems to monitor for critical incidents—cases of potential or actual harm to beneficiaries such as abuse, neglect, or exploitation. Twenty-six state Medicaid agencies could not report to GAO the number of critical incidents that occurred in assisted living facilities, citing reasons including the inability to track incidents by provider type (9 states), lack of a system to collect critical incidents (9 states), and lack of a system that could identify Medicaid beneficiaries (5 states). State Medicaid agencies varied in what types of critical incidents they monitored. All states identified physical, emotional, or sexual abuse as a critical incident. A number of states did not identify other incidents that may indicate potential harm or neglect such as medication errors (7 states) and unexplained death (3 states). State Medicaid agencies varied in whether they made information on critical incidents and other key information available to the public. Thirty-four states made critical incident information available to the public by phone, website, or in person, while another 14 states did not have such information available at all. Oversight of state monitoring of assisted living services by the Centers for Medicare & Medicaid Services (CMS), an agency within the Department of Health and Human Services (HHS), is limited by gaps in state reporting. States are required to annually report to CMS information on deficiencies affecting beneficiary health and welfare for the most common program used to provide assisted living services. However, states have latitude in what they consider a deficiency. States also must describe their systems for monitoring critical incidents, but CMS does not require states to annually report data from their systems. Under federal internal control standards, agencies should have processes to identify information needed to achieve objectives and address risk. Without clear guidance on reportable deficiencies and no requirement to report critical incidents, CMS may be unaware of problems. For example, CMS found, after an in-depth review in one selected state seeking to renew its program, that the state lacked an effective system for assuring beneficiary health and welfare, including reporting insufficient information on the number of unexpected or suspicious beneficiary deaths. The state had not reported any deficiencies in annual reports submitted to CMS in 5 prior years. GAO recommendations to CMS include clarifying state requirements for reporting program deficiencies and requiring annual reporting of critical incidents. HHS concurred with GAO's recommendations to clarify deficiency reporting and stated that it would consider annual reporting requirements for critical incidents after completing an ongoing review.