prompt_front: You are given two reports by the government agency.



Question: Write a one-page summary of the report 1.



Report 2: Major disaster declarations can trigger a variety of federal response and recovery programs for government and nongovernmental entities, households, and individuals  FEMA’s Office of Response and Recovery manages the PA grant program, providing funds to states, territorial governments, local government agencies, Indian tribes, authorized tribal organizations, and certain private nonprofit organizations in response to presidentially declared disaster declarations to repair damaged public infrastructure such as roads, schools, and bridges  Figure 1 shows the total amount of PA funds obligated by county from January 2009 through February 2017 for federal disaster declarations  To implement the PA program, FEMA’s staff includes a mix of temporary, reservist, and permanent employees under two authorities, the Stafford Act and Title 5  Reservists make up the largest share of the PA workforce, which consisted of 1,852 employees––1,041 reservists, 634 full-time equivalents, and 177 temporary Cadre of On-Call Response/Recovery Employees––as of June 2017, according to PA officials  Figure 2 summarizes the key characteristics for each type of employee  After a disaster, FEMA sends PA program staff to the affected area to work with state and local officials to assess the damage prior to a disaster declaration  FEMA officials establish a temporary Joint Field Office (JFO) to house staff who will manage response and recovery functions after a declared disaster (including operations, emergency response and support teams, planning, administration, finance, and logistics)  Once the President has declared a disaster, PA staff work with grant applicants to help them document damages, identify eligible costs and work, and prepare requests for PA grant funds by developing project proposals  These proposals may include proposals for hazard mitigation if the hazard mitigation work is related to the repair of damaged facilities, referred to as permanent work projects  Immediate emergency measures, such as debris removal, are not eligible for hazard mitigation  Officials then review and obtain approval of the projects prior to FEMA obligating funds to state grantees  Figure 3 describes the process used to develop, review, and obligate PA projects  In addition to rebuilding and restoring infrastructure to its predisaster state, the PA program can be used to fund hazard mitigation measures that will reduce future risk to the infrastructure in conjunction with the repair of disaster-damaged facilities  There is no preset limit to the amount of PA funds a community may receive; however, PA hazard mitigation measures must be determined to be cost effective  Some examples of hazard mitigation measures that FEMA has predetermined to be cost effective, if they meet certain requirements, include installing shut-off valves on underground pipelines so that damaged sections can be isolated during or following a disaster; securing a roof using straps, clips, or other anchoring systems in locations subject to high winds; and installing shutters on windows or replacing glass with impact-resistant material  Applicants can also propose mitigation measures that are separate from the damaged portions of a facility, such as constructing floodwalls around damaged facilities to avoid future flooding  FEMA evaluates these proposals, considering how the proposed measure protects damaged portions of a facility and whether the measure is reasonable based on the extent of the damage, and determines eligibility on a case-by-case basis  FEMA’s Federal Insurance and Mitigation Administration (FIMA) deploys a cadre of mitigation staff to help coordinate and implement hazard mitigation activities during disaster recovery, including PA hazard mitigation  A primary task of these staff is to identify and assess opportunities to incorporate hazard mitigation into PA projects  Generally, if an applicant seeks to incorporate hazard mitigation measures into a PA project, FIMA’s hazard mitigation staff develop a hazard mitigation proposal  We, the DHS OIG, and others have reported past challenges with FEMA’s management of the PA program related to workforce management, information sharing, and hazard mitigation  For example, we reported in 2008 that the PA program had a shortage of experienced and knowledgeable staff, relied on temporary rotating staff, and provided limited training to their workforce, which impaired PA program delivery and delayed recovery efforts after Hurricanes Katrina and Rita  We found that staff turnover, coupled with information sharing challenges, delayed projects when applicants had to provide the same information each time FEMA assigned new staff and that poorly trained staff provided incomplete and inaccurate information during their initial meetings with applicants or made inaccurate eligibility determinations, which also caused processing delays  We recommended that FEMA strengthen continuity among staff involved in administering the PA program by developing protocols to improve information and document sharing among FEMA staff  In response, in 2013 FEMA instituted a PA Consistency Initiative, which included hiring new managers for FEMA regional offices, stakeholder training on PA program administration, and using a newly developed internal website to allow staff to post and share information to address continuity and knowledge sharing concerns during disaster operations  FEMA also developed the Public Assistance Program Delivery Transition Standard Operating Procedure to facilitate the transfer of responsibility for PA program activities during cases of staff turnover during recovery operations  Despite FEMA’s efforts to implement our recommendations, the DHS-OIG, in 2016, found continuing challenges after Hurricane Sandy with workforce levels, skills, and performance of reservists, who make up the majority of the PA workforce  Regarding information sharing, in 2008, we also identified difficulties sharing documents among federal, state, and local participants in the PA process and difficulties tracking the status of projects  We recommended that FEMA improve information sharing within the PA process by identifying and disseminating practices that facilitate more effective communication among federal, state, and local entities  In response, FEMA proceeded with the implementation of a grant tracking and management system, called EMMIE, which was used previously in 2007  However, in subsequent years we found weaknesses in how FEMA developed the system and the DHS-OIG found that information sharing problems similar to the ones identified in our 2008 report persisted  Regarding hazard mitigation, we reported in 2015 that state and local officials experienced challenges in using PA hazard mitigation during the Hurricane Sandy recovery efforts because PA officials did not consistently prioritize hazard mitigation, and in some cases discouraged mitigation projects during the PA grant application process, among other challenges  We recommended that FEMA assess the challenges state and local officials reported, including the extent to which they can be addressed, and implement corrective actions, as needed  In response to our recommendation, FEMA developed a corrective action plan that included actions and milestones for reviewing, updating, and implementing PA hazard mitigation policy  FEMA also identified the PA new delivery model as a solution for some of the challenges state and local officials reported  Previously, the OIG also reported that PA program officials did not consistently identify eligible PA hazard mitigation projects, and that PA officials did not prioritize the identification of PA hazard mitigation opportunities at the onset of recovery efforts after the 2005 Gulf Coast hurricanes  See appendix I for a summary of findings and the status of our past recommendations on challenges with workforce management, information sharing, and hazard mitigation related to the PA program since our last review in December 2008  FEMA’s own internal reviews and outreach efforts have also identified similar challenges  For example, at FEMA’s request the Homeland Security Studies and Analysis Institute assessed the effectiveness and efficiency of the PA program in 2011  The institute’s report outlined 3 key findings and 23 recommendations relating to the PA preaward process  For example, the report found that FEMA could enhance training programs to develop a skilled and experienced workforce; utilize technology and employ web-based tools to support centralized processing, transparency, and efficient decision making; and identify and address potential special considerations, such as hazard mitigation proposals, as early as possible in the preaward process to improve consistency  In 2014, PA program officials analyzed the PA grant process and used input from agency staff and officials involved in various aspects of the program to identify potential improvements  The resulting Public Assistance Program Realignment report found that challenges in workforce management, information sharing, and hazard mitigation continued, and included recommendations for improvement  For example, the report concluded that a shortage of qualified staff, high turnover, unclear organizational responsibilities, and inconsistent training were long-standing and continuing challenges that impaired the PA pre-award process  In addition, from January 2015 to April 2015, FEMA conducted extensive outreach with more than 260 stakeholders across FEMA headquarters, all 10 regions, 43 states, and 4 tribal nations to discuss challenges in the PA program and opportunities for improvement  For example, stakeholders identified challenges with ineffective information collection during the preaward process and suggested identifying special considerations, such as hazard mitigation, earlier in the PA process as an idea for improvement  In response, FEMA began redesigning the PA preaward process to operationalize the results of its 2014 report and address areas for improvement identified through its outreach efforts  FEMA awarded a contract for program support to help PA officials implement a redesigned PA program in 2015  This included a new process to develop and review grant applications, and obligate PA funds to states affected by disasters; new positions, such as a new program delivery manager who is the single point of contact throughout the grant application process; a new Consolidated Resource Center (CRC) to support field operations by supplementing project development, validation, and review of proposed PA project applications; and a new information system to maintain and share PA grant application documents  As part of the new process, PA program officials identified the need to ensure that staff emphasize special considerations, such as hazard mitigation, earlier in the process  Taken together, these efforts represent FEMA’s “new delivery model” for awarding PA program grants  Enhancements in the PA program under the new delivery model are presented in figure 4  Regarding the new delivery model process, FEMA introduced several changes to enhance outreach to applicants during the “exploratory call”— the first contact between FEMA and local officials—and during the first in- person meeting, called the “recovery scoping meeting ” FEMA also revised decision points during the process, when program officials can request more information from applicants, and applicants can review and approve the completion of project development steps  FEMA also incorporated special considerations, such as hazard mitigation, earlier in the new process during the exploratory calls and recovery scoping meetings  The changes and enhancements to the PA grant award process in the new delivery model are presented in figure 5  The new process divides proposed PA projects based on complexity and type of work into three categories—100 percent completed, standard, and specialized—that PA staff manage to expedite review or assign skilled staff to technical projects as needed  If the applicant has already completed work following a disaster, such as debris removal, it is considered “100 percent completed” and JFO staff collect the necessary documents and provide the information to the CRC staff who complete the development of project applications, validate the information, and complete all necessary reviews  Projects that require repairs and further assistance from PA program staff at the JFO include “standard” and “specialized” projects, which include a site inspection to document damages, before the JFO staff provide the information to the CRC  Further, PA program officials assign PA staff based on their skills and experience to standard projects, which are less technically complex to develop, and specialized projects, which are more technically complex and costly  We discuss the new workforce positions FEMA developed for JFOs and CRCs, the new information system FEMA developed to maintain and share PA grant documents with applicants, and FEMA’s efforts to incorporate hazard mitigation into PA projects later in this report  Since 2015, FEMA has invested almost $9 million to redesign the PA program through the reengineering and implementation of the new delivery model, including about $4 7 million for contract support for implementation, and $4 million for acquisition of the new information system  FEMA tested the new delivery model in a series of selected disasters, using a continuous process improvement approach to assess and improve the process, workforce changes, and information system requirements, prior to implementing the new model for all future disasters  For example, FEMA first tested the new process in Iowa in July 2015 and, in February 2016, PA program officials expanded their test to include all of the new staff positions  In October 2016, PA program officials added the new information system to achieve a comprehensive implementation of all of the elements of the new delivery model for the agency’s response to Hurricane Matthew in Georgia, two additional disasters in Georgia in January 2017, and in Missouri, North Dakota, Wyoming, Vermont, and two disasters in New Hampshire from June through August 2017  The timeline for PA’s implementation of the new delivery model is shown in figure 6  According to program officials, FEMA planned to implement the new model for all future disasters beginning in January 2018  However, historic disaster activity during the 2017 hurricane season accelerated full implementation  As a result, on September 12, 2017, FEMA officials announced that, unless officials determined it would be infeasible in an individual disaster, the program would use the new delivery model in all future disasters  According to FEMA’s 2014 PA Program Realignment report and other program documents, PA officials designed the new delivery model to respond to persistent workforce management challenges related to identifying the required number of staff and needed skills and training, among other things, to improve the efficiency and effectiveness of the PA preaward process  To address these challenges, PA program officials centralized much of the responsibility for processing PA projects in the CRCs, created additional new positions with specialized roles and responsibilities in JFOs, and established training and mentoring programs to help build the new staffs’ skills  In 2016, PA program officials centralized some of the project activities that otherwise were being carried out at individual JFOs at FEMA’s first new CRC in Denton, Texas  Officials did so by establishing 18 new positions, many of which directly correlated with positions that FEMA deployed to individual JFOs in the legacy PA delivery model  According to PA officials, centralizing positions will improve standardization in project processing, and result in a higher quality work product  As part of the new delivery model, PA program officials created a new hazard mitigation liaison position for PA program staff at the CRC that did not previously exist at individual JFOs  The other new positions that PA program officials either created or centralized at the CRC included two specialized positions responsible for costing and validating PA projects  Previously, the PA project specialist deployed to the JFO would complete these tasks and others; however, the consistency of project development varied across the regions and disasters  In contrast, CRC staff are full-time employees who receive training to specialize in completing standardized project development steps for PA projects from multiple disasters on an ongoing basis  Program officials anticipate that centralizing new specialized staff at the CRCs will also reduce PA administrative costs and staffing levels at the JFOs  For example, staff at the CRCs, such as the new hazard mitigation liaisons and insurance and costing specialists, could support project development for multiple disasters and regions simultaneously, whereas PA previously needed to deploy staff to each JFO to fulfill these roles  In addition, once JFOs operating under the new model send projects to the CRCs for processing and review, FEMA can more rapidly close its JFOs, reducing associated administrative costs  For example, following Hurricane Matthew, FEMA credited the new delivery model, in part, with its ability to close the JFO in Georgia sooner than several other JFOs in neighboring states not involved in the implementation of the new delivery model  PA program officials created new positions with more specialized roles and responsibilities to help PA staff at JFOs provide more consistency in the project development process and guidance to applicants  Program officials split the broad responsibilities previously managed at the JFOs by PA crew leaders and project specialists, into two new, specialized positions—the program delivery manager and site inspector  The program delivery manager serves as the applicant’s single point-of-contact throughout the preaward process, manages communication with the applicant, and oversees document collection  All three PA grant applicants we spoke to following Hurricane Matthew in Georgia greatly appreciated the knowledge and assistance provided by their program delivery managers  Site inspectors are responsible for conducting the site inspection to document all disaster-related damages; determining the applicant’s plans for recovery, coordinating with other specialists, and verifying the information collected with the applicant  Officials expect deployed staff at JFOs can complete the fieldwork faster and provide greater continuity of service to applicants  Further, program officials believe that specializing roles will enable them to provide more targeted training, and improve employee satisfaction  Site inspection, hazard mitigation, and environmental and historic preservation specialists, along with a new Public Assistance program mentor, conduct a site inspection with the applicant to document damages to a historic cemetery in Savannah, Georgia, following Hurricane Matthew in 2016  PA program officials designed new training and mentoring programs for the new positions at the CRCs and JFOs and used a continuous feedback process to update and improve the training, position guides, and task books throughout the implementation of the new delivery model, according to PA officials  According to a June 2017 update of the PA Cadre Training Plan, training for the new model has five major focuses: required training and skills for position qualification; on-site refresher training; mentor training; regional-based state, local, tribal, and territorial training; and training on the new information system  Specifically, officials developed six new training courses, and identified which are required for each position under the new delivery model  For example, a program delivery manager at the JFO is required to complete both the program delivery manager and site inspector specialist courses  As of June 2017, PA program officials had provided at least one new model training course to 93 percent of their cadre (including program delivery manager training to 366 individuals and site inspector training to 1,172 individuals) and planned to provide 28 additional courses through September 2017 to the PA cadre  According to regional and CRC officials, the training courses and mentoring from experienced staff helped maximize new staff’s capabilities in the new process  Throughout the third implementation of the new delivery model, JFO and CRC staff, as well as regional PA staff, stakeholders, and applicants, identified staff skills and training as a key area that needed more attention for full implementation of the new delivery model  Our work and FEMA’s after-action reports from the third test in Georgia identified problems with site inspector skills, which affected the timeliness and accuracy of projects  Specialists and managers at the CRC noted that poorly trained site inspectors did not consistently provide the necessary information from the field, which resulted in delays for the CRC staff to process projects, and after-action reports also identified challenges with site inspector skills  According to a PA applicant in Georgia, the inconsistency of skills and experience of their site inspector resulted in the need to conduct a “do-over” site inspection on one of the applicant’s projects, causing delays  PA staff and state officials attribute much of the site inspectors’ skill gaps to their lack of training and experience in the program  According to PA Region officials, providing timely training will be a resource-intensive challenge for implementing the new delivery model for all future disasters  For example, it can be difficult to train reservists before FEMA deploys them to disasters, and many of the program’s experienced reservists have retired or resigned, resulting in few mentors for the program and a high need to provide training to inexperienced and newly hired staff  PA officials and stakeholders also emphasized the need for FEMA to provide additional training for state and local officials to build capacity and support the goals of the new delivery model  For example, according to JFO officials at the third implementation, the new delivery model increases responsibilities for applicants, who will require more applicant training than FEMA currently provides  According to state officials, applicant capabilities vary, and FEMA should provide training to state and local officials on the new delivery model and the information system before a disaster  Skill gaps among applicants could result in inconsistent implementation of the new process, according to PA staff and stakeholders, and PA staff said that training was important to prevent applicants from reverting back to the legacy PA grant application process  To support full implementation of the new delivery model for all disasters, PA program officials have updated training courses for PA staff and applicants, and planned additional training to address these challenges and other lessons learned through the test implementation  For example, PA officials told us they updated the site inspector training program in May 2017 and scheduled a new site inspector training session in August 2017 to include more hands-on training to help address the skill gaps identified for site inspectors  PA officials created a new training course for FEMA’s regional offices, in part to enable regional PA staff to provide new delivery model training to state and local officials  PA officials also planned to develop a self-paced, online course for state and local officials by the end of 2017  PA officials have not fully assessed the workforce needed for JFO field operations, CRC staff, or FIMA’s hazard mitigation staff to support implementation of the new delivery model for all future disasters  PA program officials developed an initial assessment of the total number of staff needed in the field and the CRCs in 2016 to estimate cost savings associated with consolidating and specializing positions at the CRCs and deploying fewer staff to JFOs  However, the assessment did not identify the number of staff required to fill specific positions, including program delivery managers and hazard mitigation specialists, needed to support the new delivery model for full implementation  In reviewing the test implementations of the new delivery model, we found that inadequate staffing levels at the JFOs and CRCs, and with FIMA’s hazard mitigation staff, affected staffs’ ability to achieve the goals of the new delivery model  Staff levels at the JFO  We identified challenges with having the right number of program delivery managers and site inspection specialists to achieve program goals for customer satisfaction, efficiency, and quality in test implementations of the new delivery model  For example, in the second test implementation of the new delivery model in Oregon in 2016, PA did not deploy enough program delivery managers to the disaster, which resulted in unmanageable caseloads for program delivery managers, according to state and PA officials  PA program officials assigned program delivery managers an average caseload of 12 PA applicants, which was more than they could effectively manage, according to PA staff, and program officials aim for a caseload of 8 to 10 applicants  According to state officials, local officials reported they did not always receive the support they needed from program delivery managers who managed caseloads consisting of dozens of projects at multiple sites for each applicant during the Oregon implementation  As a result of overwhelmed program delivery managers, local officials faced challenges understanding their responsibilities, such as recognizing when they needed to provide information for the project development to proceed, according to state officials  PA staff involved with the third test implementation in Georgia in 2016 and 2017 said there were not enough site inspectors or program delivery managers to fully manage the workload at the JFO  Because of the specialization of roles, projects could not move forward when there were not enough staff to execute the next step in the process  For example, PA staff at the JFO said program delivery managers completed recovery scoping meetings rapidly, but faced a bottleneck in scheduling site inspections because there were more applicants awaiting site inspections than could be fulfilled by the number of site inspection specialists available  Staff levels at the CRC  Staff at the CRC reported challenges with staffing levels during the Oregon and Georgia test implementations, and expressed concerns about when PA officials will staff the CRCs to support full implementation of the new model for all disasters  During the Oregon test implementation, a CRC specialist said there were not enough technical specialists to manage the workload and, as a result, PA program officials had to redeploy site inspectors from their JFO field operations to the CRC to complete costing estimates  During the third test in Georgia, quality assurance specialists said that their workload resulted in added stress trying to complete the work in time while adhering to quality standards  According to CRC specialists in Denton, Texas, PA officials had not determined required staff levels for full implementation, but agreed that workload was too high and program officials needed to determine the appropriate staff levels for each CRC to support full implementation  PA officials were still evaluating CRC processing times and workload management from the Oregon and Georgia test implementations to determine staffing needs, according to PA officials  Further, PA program officials plan to establish a second CRC in Winchester, Virginia, before the end of 2017, but have not determined the number of additional permanent full-time staff needed to support the CRCs for full implementation of the new delivery model  Staff levels for the hazard mitigation specialists  PA officials have not identified the number of hazard mitigation specialists in FIMA’s hazard mitigation cadre needed for full implementation of the new delivery model  According to JFO staff, current hazard mitigation staff levels are insufficient to provide the desired in-person participation of hazard mitigation staff on all recovery scoping meetings to share information on hazard mitigation with applicants and help them identify potential mitigation opportunities  A PA program official said officials missed opportunities to pursue hazard mitigation during the test implementation after Hurricane Matthew in Georgia due to lack of hazard mitigation specialists  In addition, for the test implementation in Oregon, there were not enough hazard mitigation specialists to cover all site inspections and implement their new delivery model responsibilities, according to FEMA’s after-action reports  The absence of hazard mitigation specialists in the early stages of PA project development may cause delays in officials’ identifying hazard mitigation opportunities, according to a FIMA official  PA program officials said they did not work with FIMA to determine the appropriate levels of hazard mitigation staff under the new delivery model because they were refining the new process, but as of June 2017 were working with FIMA to do so  One of the key implementation activities in our Business Process Reengineering Assessment Guide includes addressing workforce management issues  Specifically, this includes identifying how many and which employees will be affected by the position changes and retraining  Further, our prior work has found that high-performing organizations identify their current and future workforce needs—including the appropriate number and deployment of staff across the organization— and address workforce gaps, to improve the contribution of critical skills and competencies needed for mission success  According to a PA program official, their initial workforce assessment was not comprehensive because they were still collecting data required to make informed decisions  PA officials agreed that updating their workforce assessments prior to full implementation could be helpful, and acknowledged that program officials needed to be more proactive applying the lessons learned as they pivot from testing to full implementation of the new delivery model in 2018  FEMA also conducts a standard agency wide workforce structure review every 2 to 3 years, which helps officials determine the appropriate disaster workforce levels  As of June 2017, PA officials were working with other offices within FEMA to expedite the agency-wide assessment of the PA and FIMA hazard mitigation cadres, but did not know when they would complete the assessment  PA officials also acknowledged that they faced an aggressive schedule to complete various planned activities for workforce management, training, and other efforts, in support of full implementation, and that they may not be able to complete all efforts as thoroughly as they would like in order to expedite the transition of the PA program to the new delivery model  The gaps in PA workforce assessment in the JFOs, CRCs, and for FIMA’s hazard mitigation cadre present a risk that PA program managers will not have a sufficient workforce to support the goals of the new delivery model  In addition, the timing and implementation of the hiring and training activities for new PA program staff could take multiple months, and program officials will need to know what staff levels are necessary for full implementation of the new delivery model to inform resource decisions for the program in coordination with other agency offices  According to PA program officials, workforce assessment efforts have been delayed as a result of disaster response and recovery efforts related to Hurricanes Harvey, Irma, and Maria  Completing a workforce assessment will help program officials identify gaps in their workforce and skills, which could help PA program officials minimize the effects of long- standing workforce staffing and training challenges on the PA program delivery and inform full implementation for all disasters  costs  For example, EMMIE does not collect information on all of the preaward activities that are part of the PA grant application process  As a result, PA program officials said they, and applicants, must use ad hoc reports and personal tracking documents to manage and monitor the progress of grant applications  PA officials added that EMMIE is not user- friendly and applicants often struggle to access the system  In response to these ongoing challenges, PA program officials developed FAC-Trax— a separate information system from EMMIE—with new capabilities designed to improve transparency, efficiency, and management of the PA program  Specifically, FAC-Trax allows FEMA staff (PA Grants Manager) and applicants (PA Grants Portal), to review, manage, and track current PA project status and documentation  For example, applicants can use FAC- Trax to submit requests for public assistance, upload required project documentation, approve grant application items, and send and receive notifications on grant progress and activities  In addition, the FAC-Trax system includes standardized forms, as well as required fields and tasks that PA program staff and applicants must complete before moving on to the next steps in the PA preaward process  According to PA officials, these capabilities increase transparency, encourage greater applicant involvement, and enhance collaboration and communication between FEMA and grant applicants, to improve efficiency in processing and awarding grant applications and enhance the quality of project development  Further, PA officials said that FAC-Trax could reduce challenges associated with staff turnover during the project development process because the system stores and maintains applicant information and project documentation, making it easier for transitioning staff to assist an applicant  They also said they use FAC-Trax to gather and analyze data that supports management of the PA process, including measuring the timeliness of the grant application process  For example, during the test implementation of the new delivery model in Georgia following Hurricane Matthew, officials were able to document that, on average, program delivery managers took 5 days to conduct the exploratory call and 14 days to hold the recovery scoping meeting with applicants, and CRC officials took 33 days to develop and review grant proposals  Managers use this data to assess staffing needs and identify bottlenecks in the PA process, according to PA officials  FAC-Trax is critical to the new PA delivery model and will be a primary means of sharing grant application documents, tracking ongoing PA projects, and ensuring that FEMA staff and applicants follow PA grant policies and procedures  Given the importance of developing and testing this new information sharing system, we evaluated its development against four key IT management controls—(1) project planning; (2) risk management; (3) requirements development; and (4) systems testing and integration  When implemented effectively, these controls provide assurance that IT systems will be delivered within cost and schedule and meet the capabilities needed by its users  We found that FEMA’s development of FAC-Trax fully satisfied best practices for project planning and risk management, but additional steps are needed to fully satisfy the areas of requirements development and systems testing and integration, as discussed below  See appendix II for the full assessment of each IT management control  PA program officials fully satisfied all five practices in the project planning control area, according to our assessment  Key project planning practices are (1) establishing and maintaining the program’s acquisition strategy, (2) developing and maintaining the overall project plan and obtaining commitment from relevant stakeholders, (3) developing and maintaining the program’s cost estimate, (4) establishing and maintaining the program’s schedule estimate, and (5) identifying the necessary knowledge and skills needed to carry out the program  To address the first and second practices, program officials established detailed plans that describe the acquisition strategy and objectives, the program’s scope, and its framework for using an Agile software development approach, among other key actions  Agile is a method of software development that utilizes an iterative process and constantly improves software based on user needs and feedback  Program officials also developed a plan detailing the program’s approach to deploy and maintain FAC-Trax and established stakeholder groups and an integrated product team to support and oversee the development of FAC-Trax  To address the third and fourth practices, they developed and maintained a master schedule of all implementation tasks and milestones through project completion, and developed a life-cycle cost estimate of over $19 million  Additionally, FAC-Trax’s acquisition performance baseline describes the system’s minimum acceptable and desired baselines for performance, schedule, and cost  Lastly, in regards to the fifth practice, program officials identified the knowledge and skills needed to carry out the program in the FAC-Trax Request for Proposal and FAC-Trax Capability Development Plan  PA program officials fully satisfied all four practices in the risk management control area, according to our assessment  Key risk management practices are (1) identifying risks, threats, and vulnerabilities that could negatively affect work efforts, (2) evaluating and categorizing each identified risk using defined risk categories and parameters, (3) developing risk mitigation plans for selected risks, and (4) monitoring the status of each risk periodically and implementing the risk mitigation plan as appropriate  To address the first and second practices, program officials identified key risks that could negatively affect FAC-Trax in a “risk register”—an online site used to track risks, issues, and mitigating actions  As of May 2017, officials had identified 13 risks in the risk register—four open and nine closed—and evaluated and categorized the identified risks based on the probability of occurrence and scope, schedule, and cost impacts  For example, program officials reported that two of its open risks have a “medium” risk rating—meaning the risk has the potential to slightly affect project cost, schedule, or performance  To address the third and fourth practices, program officials developed and documented risk mitigation plans for all identified risks  For example, program officials planned to mitigate the risk of limited engagement of subject matter experts by identifying and engaging with appropriate experts through workshops, and monitoring the capability development process to identify any issues that may cause project delays  In addition, PA program officials documented the responsible officials, reevaluation date, and risk status, among other things, for each risk in the register, and reviewed and updated risks during weekly and monthly program reviews with stakeholders throughout FEMA  PA program officials fully satisfied four out of five practices in the requirements development control area, according to our assessment  Key requirements development practices are (1) eliciting stakeholder needs, expectations, and constraints, and transforming them into prioritized customer requirements; (2) developing and reviewing operational concepts and scenarios to refine and discover requirements; (3) analyzing requirements to ensure that they are complete, feasible, and verifiable; (4) analyzing requirements to balance stakeholder needs and constraints; and (5) testing and validating the system as it is being developed  To address the first and second practices, program officials developed a requirements management plan outlining how officials capture, assess, and plan for FAC-Trax enhancements, and established a change control process to review, prioritize, and verify user requests for changes to the system and feedback  As of May 2017, the PA program office received 734 change requests related to FAC-Trax, of which program officials completed 420 changes and planned to address an additional 277 entries  Program officials also developed a functional requirements document outlining the high-level requirements for FAC- Trax and detailed operational concepts and scenarios for each phase of the preaward process in the system’s concept of operations  To address the fourth practice, program officials created a standard template to analyze and document the user needs and acceptance criteria for planned system capabilities in March 2017  In addition, PA program officials identified risks and dependencies for recommended changes to FAC-Trax, and balanced the cost and priority of system enhancements as part of the change control process  Lastly, regarding the fifth practice, program officials tested and evaluated FAC-Trax during development, which included validating system enhancements through user acceptance testing  However, program officials did not fully address the third practice— analyzing requirements to ensure they are complete, feasible, and verifiable—because they did not ensure detailed user requirements were necessary and sufficient by tracking them back to higher-level requirements  For example, although program officials reviewed change requests for completeness and followed up with users to verify requirements, officials did not track system enhancements, made in response to detailed user requirements (e g , allowing users to search PA projects by project number), back to the high-level requirements (e g , storing data and information provided by the applicant) identified in the FAC-Trax functional requirements document and performance work statement  Officials did not track system enhancements back to high-level requirements because they did not have a complete understanding of basic user needs and system requirements at the beginning of the FAC- Trax effort, according to the PA program manager  A PA official also said the change control process was a way to identify the basic capabilities FAC-Trax needed to have and that tracking enhancements back to high- level requirements could have made the change control process more difficult to manage, and reduced user participation if, for example, users needed to understand how their change requests related to high-level requirements  However, program officials could have tracked enhancements back to high-level requirements themselves using the change control process without putting any additional burden on users  Despite not having a complete understanding of user needs and system requirements at the beginning of the FAC-Trax effort, analyzing whether users’ change requests satisfy higher-level requirements identified in key design and planning documents would have provided officials with a basis for more detailed and precise requirements throughout project development and helped them better manage the project, according to IT management controls  Further, according to the PMBOK® Guide, tracking or measuring system capabilities against approved requirements is a key process for managing a project’s scope, measuring project completion, and ensuring the project meets user needs and expectations  Program officials acknowledged the importance of tracking system enhancements back to documented system requirements  Ensuring that FAC-Trax meets user needs and expectations is especially important because the information system is key to the success of the new delivery model, according to PA officials  By analyzing progress made on documented, high-level requirements, a step that reflects a key IT management control for requirements development, the PA program will have greater assurance that FAC-Trax will provide functionality that meets user needs and expectations  PA program officials did not fully satisfy either of the two practices in the systems testing and integration control area, according to our assessment  Key systems testing and integration practices are (1) developing test plans and test cases, which include a description of the overall approach for system testing, the set of tasks necessary to prepare for and perform testing, the roles and responsibilities for individuals or groups responsible for testing, and criteria to determine whether the system has passed or failed testing; and (2) developing a systems integration plan to identify all systems to be integrated, describe how integration problems are to be documented and resolved, define roles and responsibilities of all relevant participants, and establish a sequence and schedule for every integration step  In regards to the first practice, PA program officials and the FAC-Trax contractor established a test plan that identifies the method and strategy to perform testing, including the necessary tasks, such as responding to user feedback and testing errors, and incorporating necessary resolutions into future work, testing parameters, and the roles and responsibilities of the individuals responsible for testing  However, program officials have not developed system testing criteria to evaluate FAC-Trax, which would align with the practice described above of using criteria to determine whether the system has passed or failed testing  A key feature of Agile software development is the “definition of done”—a set of clear, comprehensive, and objective criteria, that the government should use to evaluate software after each iteration of development  PA program officials said they did not establish a definition of done because officials initially managing the FAC-Trax effort lacked familiarity with system development in the Agile environment  Officials acknowledged the importance of establishing a definition of done and said they are planning to develop one, but have not identified how or when to incorporate it into the development process  According to the TechFAR—the government’s handbook for procuring digital services using Agile processes—the government and vendor should establish this definition after contract award at the beginning of each cycle of software development  By establishing criteria, such as a definition of done, to evaluate the system—a step that reflects a key IT management control for system testing and is an effective practice for applying Agile to software development—the PA program will have greater assurance that FAC- Trax is usable and responsive to specified requirements  In regards to the second practice, PA program officials developed a systems integration plan in June 2017 that identified the potential for integration of FAC-Trax with four FEMA systems, including EMMIE  In addition, program officials included a description of how staff should document integration problems and the resolution of problems in FAC- Trax development and test plans  However, the systems integration plan does not define roles and responsibilities of all participants for system integration activities or establish a sequence and schedule for every integration step for the four FEMA systems  PA officials said that system integration planning for FAC-Trax is in the early stages, but acknowledged the importance of these elements of system integration planning  Officials plan to define roles and responsibilities of all participants for system integration activities and develop the sequence and schedule for every integration step as they add new systems to the FAC-Trax development plan and obtain funding needed for their integration  Nonetheless, FEMA has used FAC-Trax for selected PA disasters since October 2016 and plans to use FAC-Trax for all future disasters  According to IT management controls, agencies should establish the systems integration plan early in the project and revise it to reflect evolving and emerging user needs  By ensuring that the FAC- Trax systems integration plan defines the roles and responsibilities of relevant participants for all integration relationships and establishes a sequence and schedule for every integration step, the PA program will have greater assurance that FAC-Trax functions properly with other systems and meets user needs  FEMA’s new delivery model enhances participation of hazard mitigation staff with the goal of identifying opportunities for mitigation earlier in the PA preaward process, according to PA officials  Two key changes related to hazard mitigation under the new model include (1) an emphasis on engaging with hazard mitigation specialists at the JFO earlier in the PA process and involving them in specific PA preaward activities and (2) the establishment of the PA program’s hazard mitigation liaison at the CRC  For example, position guides direct program delivery managers to coordinate with FIMA’s hazard mitigation specialists prior to recovery scoping meetings, and site inspectors to coordinate with hazard mitigation specialists prior to site inspections to discuss a PA grant applicant’s damages and any potential mitigation opportunities  PA program officials also developed guidance for conducting the exploratory call and the recovery scoping meeting with applicants, which include questions for PA staff to ask on the applicant’s interest in or plans for incorporating hazard mitigation into potential projects  In addition, a new hazard mitigation liaison at the CRC is responsible for reviewing PA projects for hazard mitigation opportunities and serving as a mitigation subject matter expert for the PA program  According to data provided by FEMA, PA grant applicants incorporated hazard mitigation into approximately 18 percent of permanent work projects for all disasters nationwide from 2012 to 2015  During test implementation of the new delivery model, state, PA, and FIMA officials all reported an increase in the number of hazard mitigation activities on PA permanent work projects  For example, state officials who participated in the second new model test in Oregon said that effective communication and coordination between PA and hazard mitigation staff resulted in applicants incorporating hazard mitigation into over 60 percent of permanent work projects  Furthermore, PA officials reported an increase in hazard mitigation during the third test implementation of the new model in Georgia following Hurricane Matthew, where approximately 16 percent of permanent work projects included mitigation, as of June 2017  This represents an increase compared to the PA program’s estimate for the proportion of projects that incorporate hazard mitigation among previous PA hurricane disasters in Georgia, which was about 3 percent, according to PA officials  While PA officials are trying to increase hazard mitigation through the new delivery model, not all disasters present the same number of opportunities to incorporate hazard mitigation  First, the PA program only incorporates hazard mitigation measures for permanent work projects, such as repairs to roads, bridges, and buildings  For example, as of June 2017, approximately 60 percent of the projects FEMA funded in Georgia for the third test implementation after Hurricane Matthew were for emergency work, which is not eligible for hazard mitigation measures  Second, the PA program only funds mitigation measures that officials determine to be cost-effective  In addition, we have previously reported on other factors that affect whether applicants incorporate hazard mitigation into PA projects, such as their capacity to manage and ability to fund hazard mitigation projects  National Planning for Hazard Mitigation In our 2015 report on disaster resilience following Hurricane Sandy, we noted that disaster affected areas have different threats and vulnerabilities, and local stakeholders make the ultimate determination whether or not to incorporate hazard mitigation into a project  Further, without a strategic approach to making disaster resilience investments, the federal government and its nonfederal partners may be unable to fully capitalize on opportunities for mitigation on the greatest known threats and hazards  We recommended that the Mitigation Framework Leadership Group develop an investment strategy to help ensure that federal funds expended to enhance disaster resilience achieve the goal of reducing the nation’s fiscal exposure because of climate change and the rise in the number of federal major disaster declarations as effectively and efficiently as possible  In response, the Federal Emergency Management Agency (FEMA) plans to issue a final National Mitigation Investment Strategy in 2018  The goals of this strategy include increasing the effectiveness of investments in reducing disaster losses and increasing resilience, and improving coordination of disaster risk management among federal, state, local, tribal, territorial, and private entities  Although the new model establishes hazard mitigation activities for PA and FIMA staff in the preaward process, it does not standardize and prioritize hazard mitigation planning at JFOs in the way FEMA has done with prior PA program policy  Specifically, FEMA’s 2007 PA program policy standardized planning for hazard mitigation across PA recovery efforts by stating that agency and state officials should issue a memorandum of understanding (MOU) early in the disaster, outlining how PA hazard mitigation will be addressed for the disaster, including what mitigation measures will be emphasized, and identifying applicable codes and standards, and any potential integration with other mitigation grant programs  However, PA program officials did not include guidance that standardizes planning for hazard mitigation, such as encouraging the use of an MOU, in FEMA’s 2010 PA program policy, the most recent update to the Public Assistance Program and Policy Guide in April 2017, or the New Delivery Model Operations Manual  As a result, FIMA officials said FEMA and state officials do not consistently issue MOUs that outline how FEMA and the state plan to promote PA hazard mitigation during the recovery effort, explaining that the use of the MOU is based on the preferences and priorities of the FEMA officials involved  When not issuing an MOU, FIMA hazard mitigation staff and PA officials at the JFO meet to determine the extent which hazard mitigation staff interact directly with applicants regarding PA hazard mitigation during the recovery process, according to a FIMA official  Having a consistent approach to planning for and prioritizing hazard mitigation across all disasters is important for FEMA, given that FEMA experienced challenges consistently prioritizing and integrating hazard mitigation across PA recovery efforts, according to GAO and others  For example, in our 2015 report on resilience in the Hurricane Sandy recovery, we found that state and local officials experienced challenges maximizing disaster resilience in the recovery effort because PA officials did not consistently prioritize hazard mitigation during project development  According to FEMA’s National Mitigation Framework, planning is vital for mitigation efforts during disaster recovery, and federal, state, and local officials should establish procedures that emphasize a coordinated delivery of mitigation activities and capitalize on opportunities to reduce future disaster losses  Similarly, the Recovery Federal Interagency Operational Plan, which supports FEMA’s National Disaster Recovery Framework, identifies planning as a key task for identifying mitigation opportunities and integrating risk reduction considerations into decisions and investments during the recovery process  FIMA officials agreed that including the development of a formal plan, such as the historical 2007 PA program policy regarding the use of MOUs, for PA hazard mitigation in operations guidance would help program officials plan for and prioritize hazard mitigation  They noted that FIMA’s hazard mitigation field operations guide includes procedures for implementing proposed MOUs to achieve mitigation goals  PA program officials said that, in light of changes to the PA process under the new model and subsequent updates to program policies, the MOU policy from the 2007 PA program policy was outdated  But officials agreed that planning for and prioritizing hazard mitigation at the operational level is important and said they were examining additional ways to incorporate these activities early in the PA process  As FEMA continues to implement the new model, establishing procedures to standardize hazard mitigation planning for each disaster, as it did through prior policy, could improve the prioritization of hazard mitigation in PA recovery efforts and increase the effectiveness of mitigation for reducing disaster losses and increasing resilience  PA program officials developed performance objectives and measures for hazard mitigation in the new delivery model, but could add measures to better align performance assessment for the PA program with FEMA’s broader strategic goals for hazard mitigation  In its strategic plan for 2014–2018, FEMA established an agency-wide goal to increase the percentage of FEMA-funded disaster projects, such as those under the PA program, that provide mitigation above local, state, and federal building code requirements by 5 percentage points by the end of fiscal year 2018  For example, local building codes may require measures for new construction to mitigate against future damage  To align with FEMA’s strategic goal, PA officials would also need to measure the number of PA projects that included mitigation measures that bring any repaired infrastructure to a level above applicable building codes  However, under the new model, FEMA officials developed performance objectives (and associated measures) to increase the number of projects that include hazard mitigation by 5 percent, and increase the total dollars spent on hazard mitigation by 2 percent  While these measures could help to incentivize mitigation, they are not tied to building codes and do not include specific information that FEMA could use to continually assess the PA program’s contributions to meeting the agency’s strategic goal  According to Standards for Internal Control in the Federal Government, agency management should design control activities, such as establishing and reviewing performance measures, to achieve the agency’s objectives  In addition, our work on leading public sector organizations has found that such organizations assess the extent to which their programs and activities contribute to meeting their mission and desired outcomes, and strive to establish clear hierarchies of performance goals and measures  A clear connection between performance measures and program offices helps to both reinforce accountability and ensure that, in their day-to-day activities, managers keep in mind the outcomes their organization is striving to achieve  FEMA’s ability to evaluate and report on PA hazard mitigation data is constrained, but officials are addressing this challenge through the development of data reporting and analytics capabilities for the PA program’s new information system, according to PA officials  PA program officials developed measures they could use to evaluate the new model during test implementation and compare new model performance to the legacy PA process, and agreed that aligning PA program hazard mitigation goals with FEMA’s agency-wide strategic goals would be helpful  As FEMA continues to develop and implement the new model, developing performance measures and objectives to better inform and support the agency’s broader strategic goals could help to ensure that FEMA capitalizes on hazard mitigation opportunities in PA recovery efforts  FEMA’s Public Assistance grant program is a complicated, multi-billion dollar program that is critical to helping state and local communities rebuild and recover after a major disaster  In recent years, FEMA has undertaken a major reengineering effort to make the PA preaward process simpler and more efficient for applicants and to address challenges encountered during recovery from past disasters  FEMA’s new delivery model represents a significant opportunity to strengthen the PA program and address these past challenges, and growing pains are to be expected when implementing any large reengineering effort  Further, FEMA officials work to implement these changes while supporting response and recovery following disasters, including the catastrophic flooding from Hurricane Harvey in August 2017 and widespread damages from Hurricanes Irma and Maria in September 2017  As such, it is critical that feedback obtained and lessons learned while testing the new model inform decisions and actions as FEMA proceeds with full implementation for all disasters, including the complex recovery efforts in the states and territories affected by Hurricanes Harvey, Irma, and Maria  FEMA has redesigned the PA delivery model to address various challenges related to workforce management, information sharing with state and local grantees, and incorporating hazard mitigation into PA projects  FEMA has developed new workforce processes, training, and positions to address past challenges, but completing a workforce assessment that identifies the number of staff needed will inform workforce management and resource allocation decisions to help FEMA ensure a more successful implementation  This is particularly important as FEMA is using the new model for the long-term recovery from the 2017 hurricanes, and FEMA faces capacity challenges as its workforce is stretched thin  Further, FEMA’s new FAC-Trax information sharing system provides FEMA and state and local applicants and grantees with better capabilities to address past challenges in managing and tracking PA projects  In developing FAC-Trax, FEMA implemented many of the key IT management controls that can help ensure that new IT systems are implemented effectively  However, additional steps are needed to fully satisfy the areas of requirements development and systems testing and integration  Finally, FEMA has taken some actions to better promote hazard mitigation as part of its new PA model  However, more consistent planning for hazard mitigation following a PA disaster and developing specific performance measures and objectives that better align with and support the agency’s broader strategic goals related to hazard mitigation could help to ensure that mitigation is incorporated into recovery efforts, which presents an opportunity to encourage disaster resilience and reduce federal fiscal exposure from recurring catastrophic natural disasters  We are making the following five recommendations to FEMA’s Assistant Administrator for Recovery: The FEMA Assistant Administrator for Recovery should complete a workforce staffing assessment that identifies the appropriate number of staff needed at joint field offices, Consolidated Resource Centers, and in FIMA’s hazard mitigation cadre to implement the new delivery model nationwide  (Recommendation 1) The FEMA Assistant Administrator for Recovery should establish controls for tracking FAC-Trax capabilities to the system’s functional and operational requirements to more fully satisfy requirements development controls and ensure that the new information system provides capabilities that meets users’ needs and expectations  (Recommendation 2) The FEMA Assistant Administrator for Recovery should establish system testing criteria, such as a “definition of done,” to assess FAC- Trax as it is developed; define the roles and responsibilities of all participants; and develop the sequence and schedule for integration of other systems with FAC-Trax to more fully satisfy systems testing and integration controls  (Recommendation 3) The FEMA Assistant Administrator for Recovery, in coordination with the Associate Administrator of the Federal Insurance and Mitigation Administration, should implement procedures to standardize planning for addressing PA hazard mitigation at the joint field offices, for example, by requiring FEMA and state officials to develop a memorandum of understanding outlining how they will prioritize and address hazard mitigation following a disaster as it did through prior policy  (Recommendation 4) The FEMA Assistant Administrator for Recovery, in coordination with the Associate Administrator of the Federal Insurance and Mitigation Administration, should develop performance measures and associated objectives for the new delivery model to better align with FEMA’s strategic goal for hazard mitigation in the recovery process  (Recommendation 5) We provided a draft of this report to DHS and FEMA for review and comment  DHS provided written comments, which are reproduced in appendix III  In its comments, DHS concurred with our recommendations and described actions planned to address them  FEMA also provided technical comments, which we incorporated as appropriate  With regard to our first recommendation, that FEMA complete a workforce staffing assessment that identifies the number of staff needed at joint field offices, Consolidated Resource Centers, and FIMA’s hazard mitigation cadre, DHS stated that PA, in coordination with the Field Operations Directorate and FIMA, will continue to refine and evaluate staffing needs and update the cadre force structures under the new delivery model  DHS estimated that this effort would be completed by June 28, 2019  This action, if fully implemented, should address the intent of the recommendation  With regard to our second recommendation, that FEMA establish controls for tracking FAC-Trax capabilities to ensure the new information system meets users’ needs, DHS stated that Recovery program managers will update the FAC-Trax Requirements Management Plan and the FAC-Trax Release Plan to ensure the tracking and traceability of FAC-Trax functional and operational requirements  DHS estimated that this effort would be completed by January 31, 2018  This action, if fully implemented, should address the intent of the recommendation  With regard to our third recommendation, that FEMA establish systems testing criteria to assess the development of FAC-Trax; and define the roles and responsibilities, and sequence and schedule for system integration, DHS stated that Recovery program managers will update the FAC-Trax System Integration Plan to include integration with the Deployment Tracking System, Enterprise Data Warehouse, Preliminary Damage Assessment interface, and State Grants Management system interface  DHS estimated that this effort would be completed by June 29, 2018  This action, if fully implemented, should address the intent of the recommendation  With regard to our fourth recommendation, that FEMA implement procedures to standardize planning for addressing PA hazard mitigation at the JFO, DHS stated that PA will update current process documents or develop new documents to better incorporate mitigation into the operational planning phase of the new delivery model  DHS estimated that this effort would be completed by July 31, 2018  This action, if fully implemented, should address the intent of the recommendation  With regard to our fifth recommendation, that PA coordinate with FIMA to develop performance measures and associated objectives for the new delivery model that better align with FEMA’s strategic goals for hazard mitigation in the recovery process, DHS stated that PA will reconvene the PA-Mitigation working group to develop and refine PA related hazard mitigation performance measures  DHS estimated that this effort would be completed by June 29, 2018  This action, if fully implemented, should address the intent of the recommendation  We are sending copies of this report to the Secretary of Homeland Security and interested congressional committees  If you or your staff have any questions about this report, please contact me at (404) 679-1875 or CurrieC@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report  GAO staff who made key contributions to this report are listed in appendix IV  Appendix II: Assessment of Information Technology Management Controls for the FEMA Applicant Case Tracker (FAC-Trax) Table 2 shows details on the Federal Emergency Management Agency (FEMA) Public Assistance (PA) program office’s implementation of key practices across four information technology (IT) management control areas for its new information system, the FEMA Applicant Case Tracker (FAC-Trax)  PA developed FAC-Trax as a web-based project tracking and case management system to supplement the Emergency Management Mission Integrated Environment (EMMIE) and help resolve long-standing information sharing challenges  To determine the extent to which the FAC-Trax program office implemented IT management controls, we reviewed documentation from the FAC-Trax program and compared it to key management best practices, including the Software Engineering Institute’s Capability Maturity Model® Integration for Acquisition and Development, the Project Management Institute’s Guide to the Project Management Body of Knowledge (PMBOK® Guide), and the Institute of Electrical and Electronics Engineers’ Standard for Software and System Test Documentation  We assessed the program as having fully implemented a practice if the agency provided evidence that it fully addressed the practice; partially implemented if the agency provided evidence that it addressed some, but not all, portions of the practice; and not implemented if the agency did not provide any evidence that it addressed the practice  Table 2  Public Assistance (PA) Program Office’s Implementation of Key Information Technology Management Controls for FAC-Trax PA program officials developed an acquisition plan for FAC-Trax identifying the capabilities the system is intended to deliver, the acquisition approach, and acquisition objectives  Additionally, program officials developed a capability development plan outlining a strategy for the program to obtain approval to acquire FAC-Trax  Lastly, program officials developed a systems engineering plan describing the program’s scope and its framework for using an Agile development approach, as well as a deployment, support, and maintenance plan for FAC-Trax  PA program officials developed an acquisition program baseline detailing FAC-Trax’s cost parameters and a life-cycle cost estimate for the system  As of May 2017, the life- cycle cost estimate for FAC-Trax through fiscal year (FY) 2023 is approximately $19 3 million  PA program officials updated the life-cycle cost estimate for FYs 2016 and 2017 after price negotiations with the FAC-Trax contractor, and will continue to update the estimate as annual budgets are approved, according to the Integrated Logistic Support Plan  The contracting officer’s representative for FAC-Trax performs a cost review at the end of each month, according to program officials  Furthermore, the contractor’s weekly status report includes information on the number of hours worked and the percent of contract value spent  Program officials also review program costs with Office of Response and Recovery, PA, Office of the Chief Information Officer (OCIO), and other program office stakeholders during a weekly program review  PA program officials developed an acquisition program baseline detailing FAC-Trax’s schedule parameters, as well as an integrated master schedule for the system  The integrated master schedule identifies tasks, major milestones, and task dependencies  The PA program manager reviews and updates the integrated master schedule on a weekly basis  Program officials also review FAC-Trax’s schedule with Office of Response and Recovery, PA, OCIO, and other program office stakeholders during a weekly program review  PA program officials identified the knowledge and skills needed to carry out the program in FAC-Trax contract documentation and the capability development plan  Specifically, program officials included an attachment to the FAC-Trax contract listing the required labor categories and corresponding functional position descriptions  Program officials also described the role, position type, minimum grade, and minimum certification for required personnel resources for the acquisition, development, and implementation of FAC-Trax  PA program officials developed, reviewed, and maintained project planning documents and obtained commitment from relevant stakeholders  For example, program officials reviewed and updated the integrated master schedule and costs on a weekly and monthly basis, respectively  Further, program officials reviewed the status of project elements, such as the schedule, quality and technical issues, stakeholders, staffing, cost, and risks, with Office of Response and Recovery, PA, OCIO, and other program office stakeholders during a weekly program review  PA program officials also established tactical, functional, and stakeholder groups, as well as an Integrated Product Team to support and oversee the development of FAC-Trax  FEMA’s Recovery Technology Programs Division (RTPD) has a division-level risk management plan that serves as guidance for all Recovery systems, including FAC- Trax  Program officials identified key risks that could negatively affect FAC-Trax work efforts in RTPD’s “risk register”—an online site used to track risks, issues, and mitigating actions for the division and each program office  Program officials also identified five technical, cost, and schedule risks in the FAC-Trax acquisition plan  Program officials included one of these risks in the risk register, while the remaining four were managed outside of the register  As of May 2017, program officials had identified 13 risks in its risk register—four open and nine closed  The four open risks were (1) limited subject matter expert engagement during requirements development, (2) vacancies in program management office support positions, (3) unresolved service level agreement support and funding issues, and (4) the loss of the authority to operate due to a Trusted Internet Connection that is not compliant with Department of Homeland Security security policy  Program officials evaluated and categorized the identified risks based on the probability of occurrence and scope, schedule, and cost impacts  These four points of measurement are used to calculate an overall risk score  The risk score helps program officials determine a risk’s risk rating—low, medium, or high  For example, program officials reported that two of its open risks have a “medium” risk rating—meaning the risk has the potential to slightly impact project cost, schedule, or performance  In addition, program officials detailed the risk category, probability, and impact for the five risks identified in the FAC-Trax acquisition plan  Program officials developed risk mitigation and contingency plans for each risk in the risk register  For example, program officials planned to mitigate the open risk concerning subject matter expert engagement, by identifying and engaging with appropriate subject matter experts through requirements development workshops scheduled in advance of the sprint they are to support, and monitoring the development of user stories to identify any issues that may cause delays  In addition, program officials described the risk management plan and responsible officials for the five risks identified in the FAC-Trax acquisition plan  PA program officials review and update program risks during a monthly program meeting  Program officials also review program risks with Office of Response and Recovery, PA, OCIO, and other program office stakeholders during a weekly program review  Furthermore, the FAC-Trax contractor provides a weekly status update which includes a section on identified risks  Program officials established re-evaluation dates and recorded updates, including any actions taken, for each risk in the risk register  In addition, program officials were able to provide updates on the four risks identified in the FAC-Trax acquisition plan and managed outside of the register  According to PA officials, these risks were addressed and closed by the approval of program planning documents, such as the mission needs statement, concept of operations, and operational requirements document, following the solutions engineering review, which demonstrates the readiness of the program to proceed with the procurement, in September 2016  Program officials established a requirements management plan outlining how it captures, assesses, and plans for FAC-Trax enhancements, and established a change control process to review, prioritize, and verify user requests for changes to the system and feedback  As of May 2017, the PA program office received 734 change requests related to FAC-Trax, of which program officials completed 420 changes and planned to address an additional 277 entries  PA program officials also facilitated workshops to gather requirements for specific user groups and obtained additional requirements for FAC-Trax through customer feedback on a temporary technology tool— an Access database referred to as the Public Assistance Recovery Information System—used to support an early stage of the new model implementation  Further, program officials developed a functional requirements document outlining the high-level functional and operational requirements for FAC-Trax  PA program officials developed a concept of operations for FAC-Trax detailing operating concepts and scenarios for each phase of the PA preaward process  Program officials also detailed the workflow, phases, business functions, and data inputs and outputs for the re-engineered PA process in FAC-Trax’s functional requirements document  In March 2017, program officials developed a standard template to describe the process, tasks, and data inputs and outputs for specific system capabilities  As part of the change control process, PA program officials meet three times a week to discuss and prioritize change requests  Specifically, program officials review submissions to the change control form to ensure completeness, validate impacts and root cause, and research details for incoming requests  PA program officials also follow up with users to understand and verify requirements  In March 2017, program officials developed a standard template to capture acceptance criteria for specific requirements  However, PA program officials do not track system enhancements back to the high-level requirements identified in FAC-Trax’s operational and functional requirements documentation and performance work statement  PA program officials identified system requirements and constraints in the FAC-Trax concept of operations and functional and operational requirements documents  Further, through its change control process, program officials collect suggestions, issues, and feedback on FAC-Trax and system enhancements from stakeholders, identify risks for change requests, and balance prioritized requirements and estimated level of efforts with projected costs prior to each sprint  In March 2017, program officials developed a standard template to analyze and document the urgency and need for specific requirements  PA program officials and the FAC-Trax contractor established a testing and evaluation plan for the system, developed acceptance criteria for user stories, and obtained feedback from users during and after testing  The testing process concludes with user acceptance testing (UAT)  If a change request fails during UAT or a new requirement is discovered during development, the PA program will capture the failed request or new requirement in the product backlog for implementation in a future product release  Key practices Systems testing and integration Developing test plans and test cases PA program officials and the FAC-Trax contractor tested and evaluated the system during development  The FAC-Trax test plan identifies the method and strategy to perform the testing, including the necessary tasks, testing parameters, and the roles and responsibilities of the individuals responsible for testing  However, program officials did not develop system testing criteria to evaluate FAC-Trax  A key feature of Agile software development is the “definition of done”—a set of clear, comprehensive, and objective criteria, that the government should use to evaluate software after each iteration of development  PA program officials developed a systems integration plan in June 2017 that identifies potential integration of FAC-Trax and four FEMA systems, including the Emergency Management Mission Integrated Environment  Specifically, the plan includes data requirements and standards; descriptions of the four systems FEMA plans to integrate with FAC-Trax and the proposed relationship for each connection; and security and access management requirements  In addition, program officials included a description of how integration problems are to be documented and resolved in FAC-Trax development and test plans  However, the systems integration plan does not define roles and responsibilities of all participants for system integration activities or establish a sequence and schedule for every integration step for the four FEMA systems  ● Fully implemented: The agency provided evidence that it fully addressed this practice  ◐ Partially implemented: The agency provided evidence that it addressed some, but not all, portions of this practice  ◌ Not implemented: The agency did not provide any evidence that it addressed this practice  In addition to the contact named above, Chris Keisling (Assistant Director), Amanda R  Parker (Analyst-in-Charge), Mathew Bader, Allison Bawden, Anthony Bova, Eric Hauswirth, Susan Hsu, Rianna Jansen, Justin Jaynes, Tracey King, Matthew T  Lowney, Heidi Nielson, Claire Peachey, Brenda Rabinowitz, Ryan Siegel, Martin Skorczynski, Niti Tandon, Walter K  Vance, James T  Williams, and Eric Winter made key contributions to this report 
Report 2:  The processes for developing and updating estimates are designed to inform management about program progress and the use of program resources, supporting cost-driven planning efforts and well-informed decision making  Our work has identified a number of best practices for use in developing guidance related to cost estimation and analysis that are the basis of effective program cost estimating and should result in reliable and valid cost estimates that management can use for making informed decisions  In 2012 we reported that the Bureau had not yet established guidance for developing cost estimates  We recommended that the Bureau establish guidance, policies, and procedures for developing cost estimates that would meet best practice criteria  The Bureau agreed with the theme of the report but did not specifically agree with the recommendation  Moreover, in June 2016, we also reported that the cost estimation team did not record how and why it changed assumptions that were provided to it and did not document the sources of all data it used  The documentation of these changes to assumptions did not happen because the Bureau lacked written guidance and procedures for the cost estimation team to follow  During this review we found the Bureau has since established reliable guidance, processes, and policies for developing cost estimates and managing the cost estimation process  The following documents, shown in table 2, establish roles and responsibilities for oversight and approval of cost estimation processes, provide a detailed description of the steps taken to produce a high-quality cost estimate, and clarify the process for updating the cost estimate and associated documents over the life of a project  The Decennial Census Program’s Cost Estimate and Analysis Process, which provides a detailed description of the steps taken to produce a high-quality estimate, is reliable as it met the criteria for 8 steps and substantially met the criteria for 4 steps of the 12 best steps outlined in our Cost Estimating and Assessment Guide, as shown below in figure 8  To avoid cost overruns and to support high performance, it will be important for the Bureau to abide by their newly developed policies and guidance and continue to use the life-cycle cost estimate as a management tool  The 2017 life-cycle cost estimate includes significantly higher costs than those included in the 2015 estimate  In 2015, the Bureau estimated that they could conduct the operation at a cost of $12 3 billion in constant 2020 dollars  The Bureau’s latest cost estimate, announced in October 2017, reflects the same design, but at an expected cost of $15 6 billion  Figure 9 below shows the change in cost by WBS category for 2015 and 2017  The largest increases occurred in the Response, Managerial Contingency, and Census/Survey Engineering categories  Increased costs of $1 3 billion in the response category (costs related to collecting, maintaining, and processing survey response data) were in part due to reduced assumptions for self-response rates, leading to increases in the amount of data collected in the field, which is more costly to the Bureau  Contingency allocations increased overall from $1 35 billion in 2015 to $2 6 billion in 2017, as the Bureau gained a greater understanding of risks facing the 2020 Census  Increases of $838 million in the Census/Survey Engineering category were due mainly to the cost of an IT contract for integrating decennial survey systems that was not included in the 2015 cost estimate  Bureau officials attribute a decrease of $551 million in estimated costs for Program Management to changes in the categorization of costs associated with risks: In the 2017 version of the estimate, estimated costs related to program risks were allocated to their corresponding WBS element  More generally, factors that contributed to cost fluctuations between the 2015 and 2017 cost estimates include: changes in assumptions for census operations, improved ability to anticipate and quantify risk, an overall increase in IT costs, and more defined contract requirements  Several assumptions for the implementation of the 2020 Census have changed since the 2015 cost estimate  Some assumptions contributing to cost changes, mainly in the Response (related to collecting and processing response data) and Frame (the mapping and collecting addresses to frame enumeration activities) categories, include the following: Self-response rates  Changes in assumptions for expected self- response rates contributed to increases in the response category, as the assumed rate decreased from 63 5 percent in 2015 to 60 5 percent in 2017, thereby increasing the anticipated percentage and associated cost of nonresponse follow-up  When the Bureau does not receive responses by mail, phone, or Internet, census enumerators visit each nonresponding household to obtain data  Thus, reduced self-response rates lead to increases in the amount of data collected in the field, which is more costly to the Bureau  Bureau officials attributed this decrease to a forecasted reduction in Internet response due to added authentication steps at log in and the elimination of the function allowing users to save their responses and return later to complete the survey  Productivity rates  The productivity of enumerators collecting data for NRFU is another variable in the cost estimate that was updated, contributing to cost increases in the response category  Expected productivity rates for NRFU decreased from the 2015 estimate of 4 attempts per hour to 2 9  According to Bureau documentation, this more conservative estimate is based on historical data, rather than research and test data  In-office address canvassing rates  The Bureau will not go door-to- door to conduct in-field address canvassing across the country to update address and map information for every housing unit, as it has in prior decennial censuses  Rather, some areas would only need a review of their address and map information using computer imagery and third-party data sources—what the Bureau calls “in-office” address canvassing procedures  However, in March 2017, citing budget uncertainty the Bureau decided to discontinue one of the phases of in-office review address canvassing for the 2020 Census  The cancellation of that phase of in-office review is expected to increase the number of housing units canvassed in-field by 5 percent (from 25 to 30 percent of all canvassed housing units)  In-field canvassing is more labor intensive compared to in office procedures  The 2017 version of the cost estimate reflects this increase in workload for in-field address canvassing, though overall changes in estimated costs for the Frame category, of which Address Canvassing is a part, were minimal  Staffing  Updated analysis resulted in changes to several staffing assumptions, which resulted in decreases across WBS categories  Changes included reduced pay rates for field data collection staff based on current labor market conditions and reductions in the length of staff engagement  In general, contingency allocations increased overall from $1 35 billion in 2015 to $2 6 billion in 2017  This increase in contingency can be attributed, in part, to the Bureau gaining a clearer understanding of risk and uncertainty in the 2020 Census as it approaches  The Bureau developed some of its contingency based on proven risk management techniques, including Monte Carlo analysis and allocated funding for known risk scenarios  The 2017 estimate includes close to $1 4 billion in estimated costs for these risks, almost three times the amount included in the 2015 estimate  The basis of estimate contains detail on the various risks and the process for calculating the associated contingency  The 2017 version also includes a contingency amount of $1 2 billion for general risks, or unknown-unknowns, such as natural disasters and cyber-attacks  Contingency amounts were reallocated within the WBS to more closely reflect the nature of the risk: Bureau officials attribute a decrease from the 2015 estimate of $551 million in estimated costs for program management to changes in the categorization of costs associated with risks  Officials stated that, in 2015, discrete program risks were previously consolidated as program management costs  In 2017, these discrete costs were reallocated to associate risks with the appropriate WBS element  For example, contingency amounts related to the likelihood of achieving a certain response rate previously included in the program management work breakdown category are now a part of the “response” work breakdown category  Increases in IT costs, totaling $1 59 billion, represented almost 50 percent of the total cost increase from 2015 to 2017  The total share of IT costs as a percentage of total census costs increased from 28 percent in 2015 to 32 percent in 2017, or from $3 41 billion to approximately $5 billion  Increases in IT costs are spread across seven cost categories  Figure 10 shows the IT and non-IT cost by WBS for the 2017 cost estimate  IT costs in infrastructure, response data, and census/survey WBSs account for the majority of the approximately $5 billion  The Bureau’s October 2015 cost estimate included IT costs for, among other things, system engineering, test and evaluation, and infrastructure, as well as for a portion of the Census Enterprise Data Collection and Processing (CEDCaP) program  The 2017 estimated IT cost increases were due, in large part, to the Bureau (1) updating the cost estimate for CEDCaP; (2) including an estimate for technical integration services that contributed to increases in the Census and Survey Engineering category; and (3) updating costs related to other major contracts (such as mobile device as a service, field IT services, and payroll systems)  Bureau documents described an overall improvement in the Bureau’s ability to define and specify contract requirements  This resulted in updated estimates for several contracts, including for the Census Questionnaire Assistance (CQA) contract  Assumptions regarding call volume to the CQA were increased by 5 percent to account for expected response by phone after the elimination of the option to save Internet responses and return to complete the form later  The Bureau also cited updated cost data and the results of reconciliation with independent cost estimates as factors contributing to the increased costs of other major contracts, including for the procurement of data collection devices  The Secretary of Commerce provided comments on a draft of this report on August 2, 2018  The comments are reprinted in appendix II  The Department of Commerce generally agreed with our findings regarding the improvements the Census Bureau has made in its cost estimates  However, Commerce did not agree with our assessment that the Bureau’s 2017 lifecycle cost estimate is “not reliable ” Commerce noted that it had conducted two independent cost analyses and was satisfied that the cost estimate was reliable  The Bureau also provided technical comments that we incorporated, as appropriate  We maintain that, to be considered reliable, a cost estimate must meet or substantially meet the criteria for each of the four characteristics outlined in our Cost Estimating and Assessment Guide  These characteristics are derived from measures consistently applied by cost estimating organizations throughout the federal government and industry and are considered best practices for the development of reliable cost estimates  Without a reliable cost estimate, the Bureau is limited in its ability to make informed decisions about program resources and to effectively measure progress against operational objectives  Thus, while the Bureau has made considerable progress in all four of the characteristics, it has only partially met the criteria for the characteristic of being well-documented  Until the Bureau meets or substantially meets the criteria for this characteristic, the cost estimate cannot be considered reliable  As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date  At that time, we will send copies of the report to the appropriate congressional committees, the Secretary of Commerce, the Under Secretary of Economic Affairs, the Acting Director of the U S  Census Bureau, and other interested parties  In addition, this report is available at no charge on the GAO website at http://gao gov  If you or your staff have any questions about this report, please contact me at (202) 512-2757 or goldenkoffr@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report  GAO staff who made key contributions to this report are listed in appendix III  The purpose of our review was to evaluate the reliability of the Census Bureau’s (Bureau) life-cycle cost estimate using our Cost Estimating and Assessment Guide  We reviewed (1) the extent to which the Bureau’s life-cycle cost estimate and associated guidance met our best practices for cost estimation using documentation and information obtained in discussions with the Bureau related to the 2020 life-cycle cost estimate and (2) compared the 2015 and 2017 life-cycle cost estimates to describe key drivers of cost growth  For both objectives we reviewed documentation from the Bureau on the 2020 life-cycle cost estimate and interviewed Bureau and Department of Commerce officials  For the first objective, we relied on our Cost Estimating and Assessment Guide as criteria  Our cost specialists assessed measures consistently applied by cost-estimating organizations throughout the federal government and industry and considered best-practices for developing reliable cost estimates  We analyzed the cost estimating practices used by the Bureau against these best practices and evaluated them in four categories: comprehensive, well-documented, accurate, and credible  Comprehensive  The cost estimate should include both government and contractor costs of the program over its full life-cycle, from inception of the program through design, development, deployment, and operation and maintenance to retirement of the program  It should also completely define the program, reflect the current schedule, and be technically reasonable  Comprehensive cost estimates should be structured in sufficient detail to ensure that cost elements are neither omitted nor double counted  Specifically, the cost estimate should be based on a product-oriented work breakdown structure (WBS) that allows a program to track cost and schedule by defined deliverables, such as hardware or software components  Finally, where information is limited and judgments are made, the cost estimate should document all cost-influencing assumptions  Well-documented  A good cost estimate—while taking the form of a single number—is supported by detailed documentation that describes how it was derived and how the expected funding will be spent in order to achieve a given objective  Therefore, the documentation should capture in writing such things as the source data used, the calculations performed and their results, and the estimating methodology used to derive each WBS element’s cost  Moreover, this information should be captured in such a way that the data used to derive the estimate can be traced back to, and verified against, their sources so that the estimate can be easily replicated and updated  The documentation should also discuss the technical baseline description and how the data were normalized  Finally, the documentation should include evidence that the cost estimate was reviewed and accepted by management  Accurate  The cost estimate should provide for results that are unbiased, and it should not be overly conservative or optimistic  An estimate is accurate when it is based on an assessment of most likely costs; adjusted properly for inflation; and contains few, if any, minor mistakes  In addition, a cost estimate should be updated regularly to reflect significant changes in the program—such as when schedules or other assumptions change—and actual costs, so that it is always reflecting current status  During the update process, variances between planned and actual costs should be documented, explained, and reviewed  Among other things, the estimate should be grounded in a historical record of cost estimating and actual experiences on other comparable programs  Credible  The cost estimate should discuss any limitations of the analysis because of uncertainty or biases surrounding data or assumptions  Major assumptions should be varied, and other outcomes recomputed to determine how sensitive they are to changes in the assumptions  Risk and uncertainty analysis should be performed to determine the level of risk associated with the estimate  Further, the estimate’s cost drivers should be cross-checked, and an independent cost estimate conducted by a group outside the acquiring organization should be developed to determine whether other estimating methods produce similar results  If any of the characteristics are not met, minimally met, or partially met, then the cost estimate does not fully reflect the characteristics of a high- quality estimate and cannot be considered reliable  We also analyzed the Bureau’s cost estimation and analysis guidance and evaluated them against a 12-step process outlined in our Cost Estimation and Assessment Guide  A high-quality cost estimating process integrates the following: 1  Define estimate’s purpose  2  Develop estimating plan  3  Define program characteristics  4  Determine estimating structure  5  Identify ground rules and assumptions  6  Obtain data  7  Develop point estimate and compare it to an independent cost estimate  8  Conduct sensitivity analysis  9  Conduct risk and uncertainty analysis  10  Document the estimate  11  Present estimate to management for approval  12  Update the estimate to reflect actual costs and changes  These 12 steps, when followed correctly, should result in reliable and valid cost estimates that management can use for making informed decisions  If any of the steps in the Bureau’s process do not meet, minimally meet, or partially meet the 12 steps, then the cost estimate guidance does not fully reflect best practices for developing a high-quality estimate and cannot be considered reliable  Lastly, to describe key drivers of cost growth, we compared cost information included in the 2015 and 2017 cost estimates  We analyzed both summary and detailed cost information to assess key changes in totals overall, by WBS category, and by information technology (IT) vs  Non-IT costs  We used this analysis in conjunction with information received from the Bureau during interviews and through document transfers to describe overall changes in the cost estimate from 2015 to 2017  We conducted this performance audit from December 2017 to August 2018 in accordance with generally accepted government auditing standards  Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives  We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives  In addition to the contact named above, Lisa Pearson (Assistant Director), Karen Cassidy (Analyst in Charge), Brian Bothwell, Jackie Chapin, Ann Czapiewski, Jason Lee, Ty Mitchell, Kayla Robinson, and Tim Wexler made significant contributions to this report 

ground_truth: The SFSP, a federal nutrition assistance program, is intended to provide food to children in low-income areas during periods when area schools are closed for vacation. In the last decade, federal expenditures for SFSP have increased as the program has expanded, according to USDA data. GAO was asked to review the SFSP. This report examines (1) what is known about SFSP participation, (2) other programs that help feed low-income children over the summer, and (3) challenges, if any, in providing summer meals to children and the extent to which USDA provides assistance to address these challenges. GAO reviewed relevant federal laws, regulations, and guidance; analyzed USDA's SFSP data for fiscal years 2007 through 2016; surveyed state agencies responsible for administering the SFSP in 50 states and the District of Columbia; visited a nongeneralizable group of 3 states and 30 meal sites, selected based on Census data on child poverty rates and urban and rural locations; analyzed meal site data from the 3 states; and interviewed USDA, state and national organization officials, and SFSP providers, including sponsors and site operators. Nationwide, the total number of meals served to children in low-income areas through the Summer Food Service Program (SFSP) increased from 113 to 149 million (about 32 percent) from fiscal year 2007 through 2016. The U.S. Department of Agriculture (USDA) directs states to use the number of meals served, along with other data, to estimate the number of children participating in the SFSP. However, participation estimates have been calculated inconsistently from state to state and year to year. In 2017, USDA took steps to improve the consistency of participation estimates, noting they are critical for informing program implementation and strategic planning. However, GAO determined that the method USDA directs states to use will continue to provide unreliable estimates of participation, hindering USDA's ability to use them for these purposes. Other federal and nonfederal programs help feed low-income children over the summer to some extent, according to states GAO surveyed and SFSP providers and others GAO interviewed. For example, in July 2016, USDA data indicate about 26 million meals were served through a separate federal program that allows school meal providers to serve summer meals. Some children also received summer meals through nonfederal programs operated by faith-based organizations and foodbanks, though GAO's state survey and interviews with providers and national organizations indicate the reach of such efforts is limited. States and SFSP providers reported challenges with meal sites, participation, and program administration; USDA has taken steps to address these areas. Specifically, in GAO's survey, a majority of states reported challenges with availability and awareness of meal sites, as well as limited program participation and administrative capacity. National, state, and local officials have taken steps to address these issues, such as increasing outreach and offering activities to attract participation. In addition, 17 states in GAO's survey and providers in the states GAO visited reported a challenge with ensuring meal sites are in safe locations. To address this safety issue, USDA has granted some states and sponsors flexibility from the requirement that children consume meals on-site. However, USDA has not broadly communicated the circumstances it considers when granting this flexibility. Further, some states and sponsors that have requested this flexibility reported difficulty obtaining data to show these circumstances exist, hampering their ability to ensure safe meal delivery. GAO is making four recommendations, including that USDA improve estimates of children's participation in SFSP and communicate the circumstances it considers when granting flexibilities to ensure safe meal delivery. USDA generally agreed with GAO's recommendations.

prompt_front: You are given two reports by the government agency.



Question: Write a one-page summary of the report 2.

Report 1: Created in 1968, the SFSP is authorized under the Richard B  Russell National School Lunch Act and generally provides free meals to children age 18 and under in low-income areas during certain periods when school is not in session  Specifically, the SFSP operates during school summer vacation periods between May through September, vacation periods in any month for programs operating on a continuous school calendar, and certain other times for areas affected by an unanticipated school closure, such as for a natural disaster  However, the majority of SFSP meals are served to children during the summer months  In fiscal years 2007 through 2016, federal expenditures on SFSP increased, according to FNS data, though there was a slight decrease between fiscal years 2015 and 2016 (see fig  1)  The SFSP is administered at the federal level by FNS through its national and regional offices  FNS is responsible for issuing regulations, instructions, and guidance; reviewing states’ program management and administration plans; overseeing program administration; and reimbursing states for meals served that meet program requirements  At the state level, the program is administered by state agencies and locally operated by state-approved sponsors, such as school districts, local government entities, or private nonprofit organizations  State agencies are responsible for approving, providing training to, and inspecting and monitoring sponsors and meal sites  Sponsors, in turn, are responsible for monitoring their SFSP meal sites, managing the meal service, and providing training to administrative staff and site operators  A sponsor may operate one site or multiple sites  Sites are physical locations in the community where children receive and consume meals in a supervised setting  According to FNS guidance, sites may be located in a variety of settings, including schools, parks, community centers, health clinics, hospitals, apartment complexes, churches, and migrant camps  States may approve different types of SFSP meal sites, including open sites, closed enrolled sites, and camps  Open sites operate in an area where at least half of the children are eligible for free or reduced-price school meals (referred to as “area eligible”), according to data from entities such as schools or the U S  Census Bureau  Children are generally eligible for free or reduced-price school meals if their households have incomes at or below 185 percent of federal poverty guidelines  At open sites, meals are made available to all children in the area, and all meals served that meet program requirements are reimbursable  Closed enrolled sites, on the other hand, are open only to enrolled children, as opposed to the community at large  At closed enrolled sites, meals served to all children in attendance are reimbursable as long as at least half of the enrolled children are eligible for free or reduced-price school lunch  Unlike other types of sites, camps are reimbursed only for meals served to children who have been individually determined to be eligible for free or reduced-price school meals  SFSP meals must meet certain requirements in order to be eligible for federal reimbursement; for example, the meals must be served and consumed on-site at an approved site  Federal reimbursements for summer meals are provided for each breakfast, lunch, supper, or snack served to an eligible child at an eligible site that also meets federal requirements for menu components, scheduled meal times, and nutrition  For example, to meet nutritional requirements, a lunch or a supper must, at a minimum, include four components: 2 ounces of meat or a comparable serving of a meat alternate, 3/4 cup of fruits and/or vegetables (at least two kinds), a slice of bread or a comparable serving of another grain, and a cup of milk  In 2017, the federal reimbursement rate was $3 83 or $3 77 for each eligible SFSP lunch or supper served, depending on the type of meal site  Each site may serve up to two meals or one meal and one snack per day  Some flexibilities are available to FNS in implementing the SFSP program, under its waiver and demonstration authorities  Specifically, the National School Lunch Act authorizes the Secretary of Agriculture to waive, upon request of a state or eligible service provider, certain program requirements established under the National School Lunch Act or the Child Nutrition Act of 1966, as amended, including some for the SFSP  In order to grant a waiver request, the Secretary must determine that the waiver would facilitate the state or service provider’s ability to carry out the purpose of the program, and that the waiver will not increase the overall cost of the program to the federal government, among other things  In the event a waiver request is submitted, the Secretary is required to act promptly and state in writing whether the waiver request is granted or denied, and why  The Secretary is also required to periodically review the performance of waiver recipients, and submit an annual report to Congress summarizing the use of waivers and their effectiveness, among other details  In addition to this waiver authority, the Secretary is also authorized to carry out demonstration projects to develop and test methods of providing access to summer meals for low-income children in urban and rural areas, to reduce or eliminate the food insecurity and hunger of low-income children and improve their nutritional status  The Secretary is required to provide for an independent evaluation of the demonstration projects carried out under this authority, and submit an annual report to Congress on the status of each project and the results of the evaluations  The total number of SFSP meals served nationwide during the summer— one indicator of program participation—increased from 113 million meals in fiscal year 2007 to 149 million meals in fiscal year 2016, or by 32 percent, according to our analysis of FNS data  The number of SFSP meals served has generally increased from year to year over this 10-year period  Most recently, meals decreased by 6 percent from 156 million meals in summer 2015 to 149 million meals in summer 2016, according to our analysis of FNS data (see fig  2)  Factors that may have affected year-to-year fluctuations include changes in funding for summer programs, sponsor participation, weather, and the number of weekdays available for sites to serve meals within a given summer, according to FNS and state agency officials we interviewed  For example, state agency officials in one of the three selected states we visited said they believe that reductions in state and local funding for summer programs that also provide meals, and turnover of sponsors, including losing one of the state’s largest sponsors in a recent summer, affected the total number of SFSP meals served in their state in 2016  According to our analysis of FNS data, SFSP lunches served in the summer months increased by over 17 million from fiscal year 2007 through fiscal year 2016, accounting for almost half of the total increase in the number of SFSP meals served in that period  However, when comparing across each of the meal types, supper and breakfast had the largest percentage increases over the 10-year period, 50 and 48 percent, respectively (see table 1)  In comparison, the number of SFSP lunches served increased by 26 percent from fiscal years 2007 through fiscal year 2016  From fiscal year 2007 through fiscal year 2016, there were increases in the numbers of meals served in both SFSP and NSLP, the largest child nutrition assistance program  Specifically, SFSP lunches served in July increased from 32 million to 40 million, or 24 percent, from fiscal year 2007 to 2016, and NSLP lunches served in March increased from 328 million to 376 million meals, or 15 percent, according to our analysis of FNS data  Although the programs generally serve similar populations, different factors likely affected the number of meals served by each program, in part because NSLP serves children in schools during the school year and SFSP serves children in a variety of settings during the summer months  Although states report the actual number of SFSP meals served to FNS, they estimate the number of children participating in SFSP, and information obtained from our state survey and FNS indicate that these participation estimates have been calculated inconsistently  FNS instructs state agencies on how to calculate a statewide estimate of children’s participation in the SFSP, referred to as average daily attendance (ADA), using sponsor-reported information on the number of meals served and days of operation in July of each year  However, states’ methods for calculating ADA have differed from state to state and from year to year, according to our review of states’ survey responses and FNS documents  For example, although FNS directed states to include the number of meals served in each site’s primary meal service— which may or may not be lunch—some states, according to our survey and FNS data, were calculating ADA using only meals served at lunch  FNS officials told us that these states were therefore not following the agency’s instructions  Further, some states have changed their methods for calculating ADA over time—five states reported in our survey that the method they used to calculate ADA in fiscal year 2016 differed from the one they used previously  While FNS clarified its instructions in May 2017 to help improve the consistency of states’ ADA calculations moving forward, ADA remains an unreliable estimate of children’s daily participation in SFSP for at least two reasons, according to our analysis  (See sidebar for the revised ADA calculation instructions ) First, ADA is based on summary data that does not account for existing variation in site days of operation, and second, it is based on July data, which does not reflect the month with the greatest number of meals served in every state  According to our analysis, ADA is an unreliable estimate of children’s participation in SFSP because it currently does not account for existing variation in the number of days that each site serves meals to children  Specifically, because FNS’s instructions indicate that sites’ ADAs are to be combined to provide a statewide ADA estimate, differences in the number of days of meal service are disregarded  As a result, ADA does not reflect the average number of children served SFSP meals daily throughout the month  Our analysis of site-level data from one of the selected states illustrates this limitation  In this state, multiple sites reported an ADA of 60 for July, yet two of those sites served meals to children on only 1 day of the month and another site served meals to children on 20 days  Although 120 children were served SFSP meals only 1 day in July across two of these sites, the combined ADA across all three sites, which we calculated following FNS’s instructions, inaccurately suggests an average of 180 children were participating in SFSP at these sites on a daily basis in July  According to our analysis, ADA is also an unreliable estimate of children’s participation in SFSP because it currently does not account for state variation in the month with the greatest number of SFSP meals served, potentially leading to an underestimate  According to FNS officials, the agency instructs states to calculate ADA for July because officials identified this as the month with the largest number of meals served nationwide  However, because of reasons such as state variations in school calendars, July is not the month with the largest number of meals served in every state  In one of the selected states, Arizona, using July to calculate ADA cuts the estimate almost in half  Specifically, we followed FNS’s instructions and calculated that Arizona’s ADA was 14,987 in July 2016 compared to 26,772 in June 2016  Nationwide, in summer 2016, 26 states served more SFSP meals in June or August than in July, according to our analysis of FNS data  However, without site level data on meals served and operating days, the extent to which these states had higher ADAs in June or August as compared to July is unknown  In its May 2017 memo to states revising the ADA calculation instructions, FNS said that it is critical that the agency’s means of estimating children’s participation in the SFSP is as accurate as possible because it helps inform program implementation at the national level and facilitates strategic planning and outreach to areas with low participation  In addition, Standards for Internal Control in the Federal Government state that agencies should maintain quality data and process it into quality information that is shared with stakeholders to help achieve agency goals  Although FNS has also collected information on other data that states collect on the SFSP, the agency has not yet used this information to help improve its estimate of children’s participation in the program  In 2015, FNS published a Request for Information, asking whether states or sponsors collect any SFSP data that are not reported to FNS  While FNS received responses from only 15 states, these responses suggest that some states collect additional data, such as site-level data that may allow for an improved estimate of children’s SFSP participation, potentially addressing the issues we found in our analysis  In response to the information FNS received, they followed up with up to 9 of the 15 states in 2016 and 2017 to explore the feasibility of collecting additional data and improving estimates of children’s participation  Although they took these steps, FNS officials told us they are cognizant of the burden on states and site operators that would be associated with additional reporting requirements  At this time, the agency has not taken further action to improve the estimate, such as addressing the reliability issues caused by variation in the number of operating days of meal sites and in the months with the greatest number of meals served by state  As a result, FNS’s understanding of children’s participation in the SFSP remains limited, which impairs its ability to both inform program implementation and facilitate strategic planning and outreach to areas with low participation  Other federal programs that operate solely in the summer, as well as those operating year-round, help feed low-income children in the summer months  These programs include the NSLP Seamless Summer Option, which provides nutrition assistance benefits solely in the summer, and several federal programs that operate year-round  In July 2016, in addition to the 70 million meals provided through the SFSP, 26 million meals were provided to low-income children through school food authorities participating in the NSLP’s Seamless Summer Option, according to FNS data  The Seamless Summer Option was established in 2004, and according to FNS, streamlines administrative requirements to encourage school food authorities providing free or reduced-price meals during the school year under the NSLP and SBP to continue providing meals to low-income children when school is not in session  For example, officials from a national organization involved in summer meals told us the Seamless Summer Option makes it easier for school food authorities to provide summer meals because they continue working with the same state agency, reporting the same information to the state, and operating without having to transition to a separate program  Nonetheless, school food authorities can choose to provide free summer meals to children through either the SFSP or Seamless Summer Option, and the majority of states (34) reported in our survey that a greater proportion of school food authorities participated in the SFSP than the Seamless Summer Option in summer 2016  According to FNS and selected state officials, this may be related to the generally lower meal reimbursement rates school food authorities participating in the Seamless Summer Option receive compared to the rates received by those participating in the SFSP  In summer 2016, the Seamless Summer Option added to the geographic availability of summer meal sites in two of the three states we visited as part of our review  School food authorities provided summer meals through the Seamless Summer Option in Arizona and Illinois, but not in Massachusetts, based on our analysis of data provided by these states  In Arizona and Illinois, school food authorities participating in the Seamless Summer Option added 643 and 298 summer meal sites, respectively, in the month with the largest number of SFSP meals served in each state (see fig  3)  In addition, some of the Seamless Summer Option sites in these two states provided meals to children in areas where there were no SFSP sites  For example, Seamless Summer Option sites provided meals in areas near the northeastern and southwestern corners of Arizona that lacked nearby SFSP sites  In addition to the SFSP and the Seamless Summer Option, the Summer Electronic Benefit Transfer for Children (Summer EBT) demonstration provided nutrition assistance benefits to 209,000 low-income children in summer 2016 in select areas across 6 states and 2 Indian Tribal Organizations, according to FNS officials  Since the summer of 2011, Summer EBT benefits have been provided to eligible households on an electronic benefits transfer card, which households use to purchase eligible foods at authorized retailers  Specifically, the demonstration has provided monthly benefits of $30 or $60 per eligible child to households with children in areas with a perceived high level of need, based on the demonstration grantees’ assessments of the percentage of children eligible for free or reduced-price school meals and the availability of the SFSP  Consistent with this, three of the states that participated in Summer EBT in 2016 reported through our survey that these benefits helped children who were unable to access summer meals through the SFSP or the Seamless Summer Option  Further, according to an FNS- funded evaluation, Summer EBT improved food security among low- income children who participated in the demonstration  Specifically, the evaluation found the receipt of these benefits reduced the number of children in the demonstration experiencing very low food security between 2011 and 2013 by one-third  Some low-income children also receive nutrition assistance in the summer through federal programs that operate year-round  According to FNS data, in June 2016, 5 8 million infants and children participated in the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC) and 3 million children participated in the Child and Adult Care Food Program (CACFP)  In addition, an average of 19 2 million children participated each month in the Supplemental Nutrition Assistance Program (SNAP) in fiscal year 2016, according to FNS data  These benefits are provided year-round, including when school is in session and children may also be eligible to receive school meals  In our previous work on federal domestic food assistance programs, we reported that no one program alone is intended to meet a household’s full nutritional needs  At that time, several officials and providers told us that the variety of food assistance programs offers eligible individuals and households different types of assistance and can help households fill the gaps and address the specific needs of individual members  For example, a mother with two children may rely on SNAP for her household’s basic groceries, the NSLP to feed a school-age child during the school year, and WIC to obtain supplemental foods for herself and an infant  Some low-income children also receive summer meals through nonfederal programs, according to our state survey and interviews with organizations involved in summer meals  Twenty-seven states reported in our survey that they were aware of other state- or non-state-funded programs that provided children of low-income households with meals in their states during the summer months  According to our analysis of state survey responses, local faith-based organizations and foodbanks were the most common types of entities operating these types of programs  Similarly, officials from FNS and two regional organizations we interviewed said they were aware of children receiving summer meals through nonfederal programs operated by faith- based and other community organizations  In addition, SFSP site operators at 6 of the 30 meal sites we visited in the selected states told us nearby foodbanks and faith-based organizations may also be providing children with free meals to some extent  For example, one of the meal sites we visited was operated by a foodbank that, in addition to the SFSP, provided food boxes to those in need and distributed food to other local community organizations to provide to persons in need of immediate assistance, including families with children  Although FNS and the majority of states do not collect data on nonfederal programs, results from our state survey and interviews with SFSP providers and organizations involved in summer meals indicate the reach of nonfederal programs is limited  In our survey, states reported that the geographic coverage of these nonfederal programs varied by state, with 11 states indicating that they operated in some portions of the state—the most common state response  In addition, 16 states reported that they were not aware of any nonfederal programs providing summer meals to children in their state (see fig  4)  Similarly, SFSP site operators at 24 of the 30 meal sites we visited were unaware of nonfederal programs providing meals to children in the areas in which they operated  In addition, officials from several national organizations involved in summer meals told us children have very few options for receiving summer meals beyond the federal summer meals programs  Specifically, officials from one national organization explained that food is often a significant part of the cost of a summer activity program for children and suggested that is one reason why organizations choose to participate in the SFSP  Although the SFSP provides for federal reimbursement of eligible meals and certain administrative and operating costs, nonfederal programs that provide children with summer meals may choose not to participate in the SFSP for several reasons, according to officials we interviewed from several organizations involved in summer meals  For example, some nonfederal program providers may not participate in the SFSP because they are unaware the program exists  Additionally, some nonfederal program providers may be aware of the SFSP, but choose not to participate because they do not want to follow certain program requirements, such as the nutrition or meal pattern standards  In addition, some providers may not participate in the program because they do not think they can handle certain aspects of the administrative workload associated with the SFSP  For example, a state official we interviewed told us the administrative workload associated with the SFSP can be particularly challenging, especially for smaller sponsors  Similarly, officials from a regional organization involved in summer meals told us one of the providers they work with who operated 10 meal sites chose to leave the SFSP because the paperwork required to operate the sites was too administratively burdensome for their volunteer site operators  States and SFSP providers reported challenges with meal sites, participation, and administration, though federal, state, and local entities have taken steps to improve these areas  Half or more of states reported in our survey that SFSP issues related to meal site availability, such as in rural areas, increasing children’s participation, and program administration were moderately to extremely challenging (see fig  5)  Overall, 41 states reported facing at least one challenge with the SFSP, while 9 reported facing none  Availability of transportation, low population density, and limited meal sites pose challenges for SFSP in rural areas, according to states we surveyed, selected national organizations, and state and local officials in the three selected states we visited  More than two-thirds of states in our survey reported they faced a moderate to extreme challenge with limited options in rural areas to transport children to summer meal sites (37), as well as with the distance to summer meal sites in rural areas resulting in low child turnout that affects the financial viability of site sponsorship (36)  As officials from one national organization explained, it may not be cost- effective for sponsors to operate in remote or rural areas if there are not enough meal sites or children participating in the program  Similarly, a sponsor in one of the selected states indicated that there are large parts of the state where the distances between meal sites are substantial, and travel between them takes several hours  An official from one of the selected states said transportation challenges can lead to underserved rural areas, including Indian reservations  Of the three states we reviewed, each had rural areas with few or no federally funded meal sites in summer 2016  However, a majority of the children in some of those areas were eligible for free or reduced price school meals, according to Census data provided by FNS, and would therefore be “area eligible” for the purposes of SFSP  For example, as shown in figure 6, “area eligible” locations in rural western parts of Arizona did not have any SFSP or Seamless Summer Option meals sites in June 2016, the month with the greatest number of summer meals served in that state  States and SFSP providers have responded to challenges with meal sites in rural areas by using other meal delivery approaches—efforts that FNS has supported through information sharing and grants  For example, according to one national organization involved in summer meals, some SFSP providers offer vans or buses to transport children to meal sites or partner with local bus authorities to give children free rides to meal sites  Instead of transporting children to sites, other sponsors transport meals to children through mobile meal delivery, an alternative summer meal model used in 48 states according to our survey  In this model, sponsors deliver meals by bus, using a route with state-approved stops in a community, and children consume the meal at the stop under a supervised setting  According to FNS officials and representatives from national organizations, this approach can be particularly helpful for providing summer meals to children in rural areas  State officials in two selected states told us they use mobile meal delivery to help fill gaps in meal service and help children overcome the lack of transportation or resources in their community  To serve children in very remote areas with limited resources, a sponsor in one of the selected states reported piloting a model involving delivering frozen meals every other week to such areas and supplying equipment, such as freezers and microwaves, to support meal service  To help sponsors address challenges related to meal sites in rural areas, FNS has shared information on alternative delivery models through its SFSP toolkit and webinars and has also provided related grant funding  For example, in summer 2011 and 2012, FNS funded the Meal Delivery demonstration project to provide meals to children in rural areas where low population density, long distances, and transportation issues made it difficult for children to get to SFSP sites, making site and sponsor operation financially unsustainable  The demonstration project funded meals to children in rural areas of Delaware, Massachusetts, and New York, providing food delivery to homes or drop-off sites near homes of eligible children  More than half the states (30) in our survey reported they faced a moderate to extreme challenge reaching low-income children in communities that are not area eligible  Areas in which fewer than 50 percent of children qualify for free or reduced-price meals during the school year are not eligible to have open summer meal sites at which all children who come to the site can receive a free meal  As a result, some children who are eligible for free and reduced-price meals during the school year do not have open summer meal sites located in close proximity to their residences, according to several national organization officials and SFSP providers  Eligible children in these areas may instead be limited to other types of SFSP sites, such as closed enrolled summer meal sites, or nonfederal programs providing meals, if available  For example, in one of the selected states, a sponsor of SFSP sites funded meals without federal support at one site that they operated as an open site in order to serve low-income children residing in low-income housing  These children did not otherwise have access to a federally funded summer meals site, according to these officials, because the broader area was part of a school district that had a greater than 50 percent proportion of children from higher-income families  Recognizing that some children may reside in an area that is not area eligible but is immediately adjacent to such an area, FNS has allowed additional flexibility in establishing area eligibility for open meal sites  Specifically, in 2014 and 2016 policy memos, FNS expanded the ways in which states and sponsors can use Census data to establish area eligibility  For example, FNS has allowed states and sponsors to average Census data across adjacent geographic areas to determine area eligibility  FNS noted that these additional flexibilities help ensure meal sites can be located in more areas in which poor economic conditions exist  Nearly all states (50) reported in our survey that the availability of meal sites throughout the summer months was a factor critical to the success of the SFSP, yet more than half the states (27) also reported they faced a moderate to extreme challenge with limited meal site days of operation  Nineteen of the 40 states that provided information about site days of operation reported 1 day as the shortest length of operation for SFSP sites in their state in fiscal year 2016  Limited meal site days of operation was a significant challenge in one of the three selected states we visited, as almost one-quarter of sites operated for only 1 to 2 weeks across a 2-month period in summer 2016, and an additional half of sites operated for 3 to 4 weeks across that same period, according to our analysis of state data  In contrast, in the other two selected states, the majority of sites (64 and 76 percent, respectively) operated for 5 or more weeks during a 2-month period  SFSP sites may have limited days of operation for various reasons, such as constraints with program administration and costs, according to interviews with a national organization official and a sponsor in one of the selected states  Some SFSP providers and national organizations involved in summer meals have responded to these challenges by working to extend the days of operation of meal sites—efforts that FNS has supported through related grant funding  Officials from one meal site located at a school in one of the selected states told us that 2017 was the first year the site stayed open an additional 4 weeks after summer school classes ended in an effort to expand participation, an extension made possible through support from an experienced sponsor  In addition, officials from a national organization involved in sponsoring summer meals told us they encourage their local sites to operate in August—a month where there are generally fewer summer meal service offerings—to meet children’s needs  At the federal level, under its demonstration authority, FNS funded the Extending Length of Operation Incentive project, a grant which provided an additional 50-cent reimbursement for all lunch meals served at sites in Arkansas in 2010 that offered meals for 40 or more days  Two-thirds of states (34) reported through our survey that they also faced a moderate to extreme challenge with a lack of awareness of summer meal sites among children and families, a challenge also mentioned by SFSP providers in the selected states  Meal site operators in one selected state noted that making families aware that all children may receive a meal for free at open sites can be a challenge  For example, one sponsor operating a meal site in a school said the perception among some is that the meal program is only for children attending summer school, and not for others in the community  Although that site had outside banners and advertising to help address that misperception, another SFSP provider explained that having sufficient funds to market the SFSP and increase awareness among families is also a challenge  To address these challenges, state agencies, some SFSP providers, and FNS have taken steps to help promote awareness of the SFSP  For example, nearly all states (47) reported in our survey that they have increased their outreach efforts for the SFSP in the last 5 years  More than half of states (36) also reported increases in overall SFSP participation during that time, which they believe were related to their outreach efforts  The majority of states in our survey reported conducting outreach on the SFSP to groups including children, parents and guardians, and schools, among others, using methods such as flyers, email, newspapers, and social media (see fig  7)  Further, state agency officials and sponsors in the selected states reported that they have developed partnerships with state and local advocacy groups and community leaders, among others, to promote the SFSP  For example, one state agency official said they partner with local advocacy organizations to field calls from parents seeking information about summer meal sites through their hunger hotline  FNS has promoted the use of such partnerships, as well as traditional and social media, to raise awareness of the SFSP  In addition, FNS developed the Summer Meals Site Finder, an online mapping tool that provides information on summer meal sites nationwide  Attracting children of all ages to SFSP meal sites can also be a challenge, according to states and SFSP providers  More than half of the states (31) reported in our survey that they faced a moderate to extreme challenge with limited youth and teen participation at summer meal sites, and an official from a national organization involved in the SFSP explained that it is difficult to attract children to a meal site when the site is focused solely on food  Similarly, 46 states in our survey reported that providing age- appropriate programming and enrichment activities for children at summer meal sites is a factor critical to the success of the SFSP  However, some meal sites may lack the resources to add activities, according to some SFSP providers in the selected states as well as FNS and national organization officials  Attracting teens can be particularly challenging, in part because of meal service time periods, a lack of age- appropriate activities, and stigma, according to national organizations and providers we interviewed  For example, early morning meal sites generally attract younger kids as teens may be apt to sleep later in the summer, and teens may also perceive a stigma in participating in a free meal program and may face peer pressure not to eat  In addition, meal offerings at SFSP sites may also present challenges to teen participation  Specifically, because FNS bases minimum portion size requirements for meals on the needs of younger children, meals are not always adequate to meet the nutritional needs of teens, according to one sponsor we interviewed  Across the 30 meal sites in the 3 states we visited in summer 2016, we observed variety in the meals served during different meal services  (see fig  8 ) States and SFSP providers have collaborated with others and sought specific types of sites to help provide enrichment activities and attract certain age groups—efforts that FNS has supported through information sharing and related grant funding  Sponsors in the selected states said they have focused on partnerships with groups such as those focused on youth development, churches, libraries, and police or fire departments, to offer age-appropriate activities for children (see fig  9)  For example, programs with local police departments, such as Cops N Kids in one selected state, or libraries in two selected states, provided meal services in combination with youth development or other enrichment activities  (See sidebar for highlights on the Cops N Kids program ) One national organization official said activities at SFSP sites can help take away the stigma around the program because children are not just there for the meal  Efforts to rebrand the SFSP as a community event where entire families can participate at the meal site also can have this effect, which is why some sponsors in the selected states said they partnered with foodbanks to donate meals for adults  In addition, a sponsor in one selected state told us they adjusted their meal offerings to match the needs of children of different age groups, for example, by serving meals to younger children earlier in the day and meals to teens later in the day  To support participation from children of all ages, FNS has shared information on age-appropriate activities through its SFSP toolkit and provided related grant funding  For example, in 2010, FNS funded the Activity Incentive demonstration project, in which sponsors in Mississippi were provided with mini-grants to increase enrichment and recreational activities, such as education, tutoring, sports and games, arts and other activities, to draw children to meal sites  More than half the states reported in our survey that they faced a moderate to extreme challenge with limited state agency staffing (27), a limited amount of federal funding for SFSP administration (27), as well as ensuring sponsor participation to meet needs (28)  In addition, 28 states reported in our survey that they faced a moderate to extreme challenge with sponsors not following program requirements  Limited staffing can affect a state agency’s ability to conduct efforts aimed at increasing participation, identifying potential sponsors, and reviewing and monitoring sponsors, according to national organization and state officials we interviewed  For example, increases in sponsors and sites requires additional staff and time to conduct pre-approval visits, sponsor and site reviews, vendor reviews, and technical assistance visits, which directly affects the amount of funding needed to support staff salaries and travel reimbursement, according to one state in our survey  However, because the SFSP administrative funds FNS provides to states are based on the number of meals served in the previous year, increasing the number of staff to help increase SFSP participation is difficult, according to a national organization official we interviewed  States reported a moderate to extreme challenge with the following issues related to ensuring sponsor participation: a lack of sponsors to meet summer meal needs, a lack of awareness of the summer meal program among potential sponsors or sites, completing federal requirements for monitoring of SFSP sponsors, and identifying potential sponsors  State agencies responsible for administering the SFSP reported relying on other resources and partners to help with program administration— strategies that FNS has supported through information sharing and its online tools  As discussed earlier, all three selected state agencies we interviewed told us they partner with advocacy groups to help expand and conduct outreach on the SFSP  Additionally, more than half the states in our survey reported several factors—which may ease the administrative burden on states—as critical to the success of the SFSP, including partnerships with SFSP sponsors (49) and retaining sponsors and sites over multiple summers (51)  To support states’ use of alternative funding sources to help administer the SFSP, FNS has shared information on federal, state, and private funding and grant opportunities  FNS also developed the online Capacity Builder tool, which 35 states reported in our survey was moderately to extremely useful in identifying or confirming meal site eligibility in fiscal year 2017  Seventeen states reported in our survey that ensuring summer meal sites are in safe locations was moderately to very challenging, a challenge that some states and SFSP providers have taken steps to help address  State officials and SFSP providers in the selected states reported that when crime has occurred near a site, there are concerns about ensuring children’s safety while they are consuming meals at the site, as well as the safety of site staff delivering meals  Some sponsors noted, in particular, parents’ concerns for the safety of their children at meal sites in light of criminal activities in the surrounding area  To ensure children continue to have access to meals, some sponsors noted that in the event of an immediate threat at an outdoor meal site, site staff are sometimes able to bring children to a nearby indoor space instead  States and SFSP provider officials in two selected states told us they have also used other strategies, including partnerships with local law enforcement agencies, to help address safety concerns during the meal service and ensure children have access to meals  For example, national organizations involved in summer meals and sponsor officials in the selected states said they encourage partnerships with local police departments to use police escorts at meal sites or to follow mobile meal routes in situations where safety at the meal site is a concern  When violence or crime has occurred near a site, some states and SFSP sponsors have also sought flexibility from FNS with respect to the federal requirement that children consume summer meals on site, according to state and local officials  FNS has used its available authorities to grant some states and sponsors flexibility with respect to the requirement that children consume summer meals on site, such as when safety at the site is a concern; however, FNS has not clearly communicated to all states and sponsors the circumstances it considers when deciding whether to grant this flexibility  According to our review of letters FNS sent to multiple states approving their requests for this type of flexibility, the agency identified a consistent set of circumstances that needed to be met for it to grant this flexibility  These circumstances were described in the letters the agency sent to states and generally included verification that violent crime activities occurred within both a 6-block radius of the meal site and 72 hours prior to the meal service  FNS’s letters to states indicate that when documentation was provided to the agency showing that these circumstances existed at a summer meals site on a particular day or days, meals consumed by children off site on those days were eligible for federal reimbursement  Although FNS has issued guidance on the general processes for requesting flexibility from program requirements under its waiver and demonstration authorities, these guidance documents do not detail the specific circumstances that the agency considers when deciding whether to grant flexibility from the on-site requirement due to safety concerns  FNS has communicated this information only in its responses to specific state and sponsor requests, and it has not communicated these circumstances more broadly to all states and sponsors  FNS officials explained that they review state and sponsor requests for flexibility due to safety concerns on a case-by-case basis  However, they also acknowledged that the set of circumstances used for approval of state and sponsor requests for flexibility, which we identified in their letters to states, has been used repeatedly  Further, states and sponsors reported challenges obtaining the specific data needed for approval of a site for this type of flexibility, hampering some providers’ efforts to ensure safe delivery of meals  For example, state agency and sponsor officials in one selected state said obtaining the crime data needed to qualify for the flexibility can be an administrative burden on sponsors, and these data are not consistently available in a timely manner  According to state agency and sponsor officials in one of the selected states, daily crime statistics are not available in all areas, and while a sponsor can sometimes access current data on crime in a city, the most recent available data on crime in suburban areas are sometimes one year old  FNS is aware of state and local challenges obtaining the necessary crime data, according to our discussions with FNS officials  FNS officials acknowledged that while they have granted some state and sponsor requests to allow children to consume meals off site in certain areas where violence or crime has occurred, some sponsors were unable to implement the flexibility because they could not obtain the necessary crime data  To help achieve agency objectives and address related risks, the Standards for Internal Control in the Federal Government state that agencies should communicate key information to their internal and external stakeholders  Although FNS officials told us they do not have one set of circumstances under which they approve these requests, our review found only one set of circumstances under which this type of flexibility has been approved  However, FNS has not broadly communicated the circumstances it considers in deciding whether to approve requests for flexibility with respect to the requirement that children consume summer meals on site in areas with violence or crime  Unless FNS shares this information with all states and sponsors, states and sponsors will likely continue to be challenged to use this flexibility, hindering its usefulness in ensuring safe summer meal delivery to children  In addition, FNS has issued reports to Congress evaluating some of its demonstration projects, as required under its statutory authorities, but the agency has not issued any such reports to Congress specifically on the use of flexibilities with respect to the on-site requirement in areas where safety is a concern  As previously discussed, the agency is required to annually submit certain reports to Congress regarding the use of waivers and evaluations of projects carried out under its demonstration authority  Furthermore, Standards for Internal Control in the Federal Government state that management should use quality information to make informed decisions and evaluate the entity’s performance in achieving key objectives and addressing risks  Yet, FNS has not evaluated nor reported on the use of waivers and demonstration projects in cases where safety was a concern  Although FNS requests reports from state agencies or sponsors that have received flexibility with summer meals delivery under FNS’s demonstration and waiver authorities, FNS officials told us they have not assessed whether their use of these flexibilities to address safety issues has been effective in ensuring safe meal delivery  FNS officials told us that they have not evaluated or reported on these flexibilities, in part, because they have limited information on their outcomes  Without understanding the impact of its use of these flexibilities, neither FNS nor Congress knows whether these flexibilities are helping provide meals to children  In addition to the challenges with safety at meal sites, sponsors also sometimes face administrative challenges when participating in multiple child nutrition programs that are operated by different state agencies or divisions within the same agency, according to officials from national and regional organizations and sponsors we interviewed  For example, officials from national organizations involved in summer meals told us the management of each child nutrition program and processes related to applications, funding, and oversight are fragmented in many states  For example, a sponsor in one of the selected states told us aspects of the SFSP and CACFP sponsor applications are highly duplicative and estimated it took 42 hours last year to complete duplicative paperwork  Another sponsor that provides school meals during the school year told us they had to fill out 60 additional pages of paperwork to provide summer meals, which coupled with having a state contact for the SFSP that was different from the one they worked with for the NSLP, was a significant burden for them  Officials from one national organization told us a lack of interoperability of some state agencies’ data systems has caused challenges and administrative burden for some sponsors  For example, in some states, different agencies oversee child nutrition programs, yet are unable to share data on sponsor approval, and therefore, sponsors are required to submit similar information to both, according to these officials  Duplicative paperwork can be particularly burdensome for some SFSP providers, as national organization officials and SFSP providers in the selected states said completing SFSP application paperwork can be especially challenging when a sponsor has staff shortages or no dedicated SFSP staff  Some selected states have worked with SFSP sponsors to help minimize the administrative burden  For example, state agency officials from one of the selected states said they have connected less-experienced sponsors to more-experienced sponsors in the community to help them with program administration  In one case, an experienced SFSP sponsor partnered with a small sponsor new to the program to help with SFSP administration, including helping them understand program rules and paperwork requirements  One SFSP sponsor also noted that their state agency took additional steps to ease administrative burden, such as making the forms for the CACFP more consistent with those for the SFSP and streamlining certain requirements for large and experienced sponsors, which the sponsor found helpful  At the federal level, FNS has established program and policy simplifications to help lessen the administrative burden on sponsors participating in multiple child nutrition programs, though the persistence of these challenges indicate that information about these simplifications has not reached all relevant state agencies  While FNS officials told us that some of the duplicative requirements may be a function of differences in statute, FNS provided guidance to states in 2011 and 2014 on simplified application procedures for institutions participating in CACFP that also wish to apply for SFSP  FNS noted in its guidance that in states where CACFP and SFSP are administered by different state agencies, state agencies are encouraged to work together to share information and streamline the application and agreement process as much as possible  FNS also addressed these simplifications in a state agency meeting in November 2017  Additionally, FNS provided guidance to states in 2012 on simplified application and review procedures for school food authorities participating in the NSLP that wish to also participate in the SFSP  Although FNS has shared this information with states in an attempt to make them aware of streamlining options, FNS officials noted that some states may choose not to implement them  Standards for Internal Control in the Federal Government state that management should externally communicate the necessary quality information to achieve the entity’s objectives, as well as periodically evaluate the methods of communication to ensure communication is effective and appropriate  FNS’s existing guidance addresses options for streamlining administrative requirements for sponsors participating in multiple child nutrition programs  However, information on program and policy simplifications available for sponsors participating in both NSLP and SFSP has not been shared with states recently, and challenges in this area persist, indicating this information has not reached all relevant state agencies  Without further efforts from FNS to disseminate information on current options for streamlining administrative requirements across child nutrition programs, overlapping and duplicative administrative requirements may limit children’s access to meals by discouraging sponsor participation in child nutrition programs  The purpose of the SFSP is to continue to provide children in low-income areas with nutritious meals over the summer when school is no longer in session, and to that end, the program provided 149 million SFSP meals to children in fiscal year 2016  Although meals served are one indicator of participation, FNS’s current estimates of children participating in SFSP are unreliable  Without additional understanding of children’s participation in the SFSP, FNS lacks information critical for informing program implementation, strategic planning, and outreach  The majority of states nationwide and SFSP providers in the three states we visited reported experiencing a number of challenges with the SFSP, and FNS has taken important steps to address these challenges  Two key challenges identified by officials in the selected states and national organizations we interviewed are ensuring summer meal sites are in safe locations, and meeting administrative requirements when participating in multiple child nutrition programs  FNS has taken steps to address these challenges by providing flexibilities in how meals are delivered to children and streamlining options for those providers participating in more than one child nutrition program  However, a lack of clarity concerning the circumstances under which FNS grants flexibilities in areas of violence and crime, and a lack of information on its use of these flexibilities and their impact on program administration, hinder efforts to ensure program goals are met  Furthermore, absent a reminder to states regarding existing options for streamlining administration across multiple nutrition programs, some providers may continue to be discouraged from participating in these programs due to duplicative and burdensome administrative requirements, which may ultimately limit the provision of nutritious meals to children  We are making the following four recommendations to FNS: The Administrator of FNS should improve its estimate of children’s participation in the SFSP by focusing on addressing, at a minimum, data reliability issues caused by variations in the number of operating days of meal sites and in the months in which states see the greatest number of meals served  (Recommendation 1) The Administrator of FNS should communicate to all SFSP stakeholders the circumstances it considers in approving requests for flexibility with respect to the requirement that children consume SFSP meals on-site in areas that have experienced crime and violence, taking into account the feasibility of accessing data needed for approval, to ensure safe delivery of meals to children  (Recommendation 2) The Administrator of FNS should evaluate and annually report to Congress, as required by statute, on its use of waivers and demonstration projects to grant states and sponsors flexibility with respect to the requirement that children consume SFSP meals on-site in areas experiencing crime or violence, to improve its understanding of the use and impact of granting these flexibilities on meeting program goals  (Recommendation 3) The Administrator of FNS should disseminate information about existing flexibilities available to state agencies to streamline administrative requirements for sponsors participating in the SFSP and other child nutrition programs to help lessen the administrative burden  For example, FNS could re-distribute existing guidance to state agencies that explains available flexibilities and encourage information sharing  (Recommendation 4) We provided a draft of this report to the Secretary of the USDA for review and comment  FNS officials provided technical comments, which we incorporated as appropriate  In addition, in oral comments, FNS officials, including the Deputy Administrator for Child Nutrition Programs, generally agreed with the recommendations in the report  As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date  At that time, we will send copies to the Secretary of the USDA and interested congressional committees  The report will also be available at no charge on the GAO website at www gao gov  If you or your staff have any questions about this report, please contact me at (202) 512-7215 or larink@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report  GAO staff who made key contributions to this report are listed in appendix IV  This appendix discusses in detail our methodology for addressing three research objectives: (1) What is known about participation in the Summer Food Service Program (SFSP) and how has it changed in the last 10 years? (2) What other programs help feed low-income children over the summer? and (3) What challenges exist, if any, in providing summer meals to children, and to what extent does the U S  Department of Agriculture’s (USDA) Food and Nutrition Service (FNS) provide assistance to states and sponsors to address these challenges? In addition to the methods we discuss below, to address all three research objectives, we reviewed relevant federal laws, regulations, and guidance; interviewed FNS officials in its headquarters and seven regional offices; and interviewed a broad range of regional and nationwide organizations involved in the SFSP  In addition, we coordinated with officials in USDA’s Office of Inspector General on their ongoing work in this area  To address our first objective about participation in the SFSP, we analyzed FNS data on meals served for fiscal years 2007 through 2016  Specifically, we analyzed the total number of meals served nationwide through the SFSP from fiscal year 2007 through fiscal year 2016  Each month, states report to FNS the number of meals served by meal type (breakfast, lunch, snack, and supper) and the number of meals served by meal and sponsor type (e g , government, nonprofit, etc ) using the FNS- 418 form  To add context on these trends, we also analyzed and compared the number of SFSP lunches served in July with the number of free and reduced-price lunches served to children in March through the National School Lunch Program (NSLP), the largest child nutrition assistance program, from fiscal year 2007 through fiscal year 2016  Each month, states report to FNS the number of meals served through the NSLP using the FNS-10 form  To assess the reliability of SFSP and NSLP data, we (1) performed electronic testing of relevant data elements, (2) reviewed existing information about the data and the system that produced them, and (3) interviewed agency officials knowledgeable about the data  Electronic testing included, but was not limited to, checks for missing data elements, duplicative records, and values outside a designated range or valid time period  We determined that these data were sufficiently reliable to identify the number of SFSP meals served and assess change over time  To further examine what is known about participation in the SFSP, we also reviewed FNS’s data on estimates of children’s participation in the program and determined that these estimates have been calculated inconsistently and are unreliable  To assess the reliability of these data, we reviewed documentation about the estimates, interviewed FNS officials, and asked states about the estimate calculation in our survey  As described in our findings, FNS does not collect data on the number of children participating in the SFSP  Instead, FNS relies on states’ estimates of children’s participation, which are based on other data reported by sponsors, such as the number of meals served and meal service days in July  To address our second objective about other programs that help feed children in the summer, we reviewed FNS’s estimate of the number of meals served through the NSLP’s Seamless Summer Option in fiscal year 2016  FNS does not collect data on the number of meals served through the Seamless Summer Option  Instead, FNS annually estimates the number of Seamless Summer Option meals served nationally by aggregating the number of free and reduced-price breakfasts, lunches, and snacks served through the School Breakfast Program (SBP) and NSLP in July  As previously noted, states report these data monthly to FNS  Although FNS does not know the actual number of meals served through the Seamless Summer Option, agency officials told us they believe the number of summer meals provided through the NSLP is small relative to the number of meals served through the Seamless Summer Option during the summer months  They noted that their use of July NSLP data to estimate the Seamless Summer Option meals likely overestimates the number of these meals for July and underestimates the number of these meals for the entire summer  To assess the reliability of the July NSLP data, we (1) performed electronic testing of relevant data elements, (2) reviewed existing information about the data and the system that produced them, and (3) interviewed agency officials knowledgeable about the data  Electronic testing included, but was not limited to, checking for missing data and data that fell outside of a reasonable range or date for the specific time period (July)  We determined that these data were sufficiently reliable to describe the number of meals served  In addition to the data FNS requires states to report, some states collect summer meals data at the meal site level and we used such data from the three selected states to address all three objectives  For objective one, to examine the number of meals served and days of operation at each summer meals site, we analyzed site-level data for 2 months from summer 2016, including the month with the largest number of SFSP meals served in each selected state: Arizona (June and July 2016), Illinois (July and August 2016), and Massachusetts (July and August 2016)  Each state also provided us with data on the number and types of meals served at each SFSP site, the site location, and the duration of time each site operated over the summer  Using the data provided by the states, we calculated the average daily attendance (ADA) for each meal site based on FNS’s instructions and examined the variation in ADA across sites and months  For our second objective on other programs, these selected states provided similar site level data for the state’s Seamless Summer Option sites, if applicable  We assessed the reliability of these data by (1) performing electronic testing of relevant data elements, (2) reviewing existing information about the data and the system that produced them, and (3) interviewing agency officials knowledgeable about the data  We determined that the data were sufficiently reliable for the purposes of this report  For both our second objective on other programs and third objective about challenges in providing summer meals to children, we also examined meal site availability in the three selected states by mapping the locations of meal sites  On the maps, we included fiscal year 2016 area eligibility data from FNS’s Capacity Builder mapping tool, as provided by FNS  The site area eligibility data from FNS’s Capacity Builder is based on the U S  Census Bureau’s 5-Year American Community Survey (ACS) estimates of children ages 0-12 and 0-18 eligible for free and reduced-price meals by Census block group and tract  According to FNS officials, FNS obtains 5-Year ACS estimates annually from the U S  Census Bureau and updates its site area eligibility in the Capacity Builder accordingly  For fiscal year 2016, FNS used 2009- 2013 ACS data to identify and include site area eligibility in its Capacity Builder  To help inform all of our research objectives, we conducted a survey of the state agencies that oversee the SFSP in the 50 states and the District of Columbia  We administered our web-based survey between August and October 2017 and received 100 percent response rate  The survey included questions about participation in the SFSP, factors critical to the overall success of the SFSP, outreach efforts, federal technical assistance, barriers and challenges in providing summer meals, alternative summer feeding models, the NSLP’s Seamless Summer Option and the federal Summer Electronic Benefit Transfer for Children demonstration, and nonfederal programs that provide children of low- income households with meals during the summer months  The survey also requested data on SFSP sites participating in the program in fiscal year 2016 and the method state agencies used to calculate ADA in SFSP on the FNS-418 form in fiscal year 2016  Because this was not a sample survey, there are no sampling errors  However, the practical difficulties of conducting any survey may introduce nonsampling errors, such as variations in how respondents interpret questions and their willingness to offer accurate responses  We took steps to minimize nonsampling errors, including pretesting draft instruments and using a web-based administration system  Specifically, during survey development, we pretested draft instruments with SFSP staff from four states (Michigan, New Mexico, North Carolina, and South Dakota) in May 2017  We selected the pretest states based on information provided by officials from FNS’s regional offices and national organizations involved in summer meals about state administration of summer meals programs, with the goal of selecting a group of states with varied experiences  In the pretests, we were generally interested in the clarity, precision, and objectivity of the questions, as well as the flow and layout of the survey  For example, we wanted to ensure definitions used in the surveys were clear and known to the respondents, categories provided in close-ended questions were complete and exclusive, and the ordering of survey sections and the questions within each section were appropriate  We revised the final survey based on pretest results  Another step we took to minimize nonsampling errors was using a web-based survey  Allowing respondents to enter their responses directly into an electronic instrument created a record for each respondent in a data file and eliminated the need for and the errors associated with a manual data entry process  We did not fully validate specific information that states reported through our survey  To help inform all of our objectives and gather information about the SFSP directly at the local-level, we conducted 30 site visits in three states: Arizona (12 sites), Illinois (8 sites), and Massachusetts (10 sites) between June and July 2017, and interviewed organizations involved with the SFSP in each site visit state  We used U S  Census Bureau data to select states and local areas within those states based on a high proportion of children in poverty, a mix of urban and rural locations, as well as a mix of sponsor and site type and diverse locations  We visited a wide variety of site locations including, but not limited to, schools, parks, community recreation areas, and libraries  At each SFSP site, we gathered information on local level factors related to SFSP participation and administration by interviewing the organization sponsoring the site, the site operators and staff, and those participating at the site using semi-structured questions  While interviewing SFSP sponsor organizations, we collected information on the sponsors’ roles in the SFSP, characteristics of the sites the organizations sponsored, outreach efforts, any challenges or barriers to SFSP administration and any efforts to address such challenges, relationships with the state agencies that administer the SFSP, relationships with FNS (national and regional offices), and the availability of nonfederally funded programs that provide meals to low-income children over the summer  During the interviews with site operators and staff, we collected information about site operation (e g , site operating days, meals offered, etc ), any challenges to providing SFSP meals to children and any efforts to address such challenges, outreach efforts, and the proximity of the next closest meal site  The information we collected from those participating at the sites included their perspectives on the SFSP food, site food consumption habits, ease of travel to the site, and access to other SFSP sites  At each site, we made observations as to how the food was provided to the children, food consumption and waste, the approximate age range of the children being served, and availability of programs or activities (e g , recreational sports)  Using semi-structured questions, we also interviewed the state agencies responsible for administering the SFSP in the site visit states to gather further information on how the SFSP is administered in each state, statewide participation in the program, related data collection activities, any challenges to administering the program and any efforts to address such challenges, related outreach efforts, alternative meal delivery models being employed by SFSP sponsors, FNS guidance or technical assistance, and the availability of nonfederally funded programs that provide meals to low-income children over the summer  For states that indicated there were other challenge(s), we provided an open-ended question that requested a description of the challenge(s) and 14 states provided descriptions of other challenges, not shown here  For states that indicated there were other challenge(s), we provided an open-ended question that requested a description of the challenge(s) and 8 states provided descriptions of other challenges, not shown here  In addition to the contact named above, Rachel Frisk (Assistant Director), Claudine Pauselli (Analyst-in-Charge), Melissa Jaynes, and Matthew Nattinger made key contributions to this report  Also contributing to this report were Susan Aschoff, Sarah Cornetto, Ying Long, Jean McSween, Mimi Nguyen, Almeta Spencer, and Ashanta Williams 


Report 2:  The processes for developing and updating estimates are designed to inform management about program progress and the use of program resources, supporting cost-driven planning efforts and well-informed decision making  Our work has identified a number of best practices for use in developing guidance related to cost estimation and analysis that are the basis of effective program cost estimating and should result in reliable and valid cost estimates that management can use for making informed decisions  In 2012 we reported that the Bureau had not yet established guidance for developing cost estimates  We recommended that the Bureau establish guidance, policies, and procedures for developing cost estimates that would meet best practice criteria  The Bureau agreed with the theme of the report but did not specifically agree with the recommendation  Moreover, in June 2016, we also reported that the cost estimation team did not record how and why it changed assumptions that were provided to it and did not document the sources of all data it used  The documentation of these changes to assumptions did not happen because the Bureau lacked written guidance and procedures for the cost estimation team to follow  During this review we found the Bureau has since established reliable guidance, processes, and policies for developing cost estimates and managing the cost estimation process  The following documents, shown in table 2, establish roles and responsibilities for oversight and approval of cost estimation processes, provide a detailed description of the steps taken to produce a high-quality cost estimate, and clarify the process for updating the cost estimate and associated documents over the life of a project  The Decennial Census Program’s Cost Estimate and Analysis Process, which provides a detailed description of the steps taken to produce a high-quality estimate, is reliable as it met the criteria for 8 steps and substantially met the criteria for 4 steps of the 12 best steps outlined in our Cost Estimating and Assessment Guide, as shown below in figure 8  To avoid cost overruns and to support high performance, it will be important for the Bureau to abide by their newly developed policies and guidance and continue to use the life-cycle cost estimate as a management tool  The 2017 life-cycle cost estimate includes significantly higher costs than those included in the 2015 estimate  In 2015, the Bureau estimated that they could conduct the operation at a cost of $12 3 billion in constant 2020 dollars  The Bureau’s latest cost estimate, announced in October 2017, reflects the same design, but at an expected cost of $15 6 billion  Figure 9 below shows the change in cost by WBS category for 2015 and 2017  The largest increases occurred in the Response, Managerial Contingency, and Census/Survey Engineering categories  Increased costs of $1 3 billion in the response category (costs related to collecting, maintaining, and processing survey response data) were in part due to reduced assumptions for self-response rates, leading to increases in the amount of data collected in the field, which is more costly to the Bureau  Contingency allocations increased overall from $1 35 billion in 2015 to $2 6 billion in 2017, as the Bureau gained a greater understanding of risks facing the 2020 Census  Increases of $838 million in the Census/Survey Engineering category were due mainly to the cost of an IT contract for integrating decennial survey systems that was not included in the 2015 cost estimate  Bureau officials attribute a decrease of $551 million in estimated costs for Program Management to changes in the categorization of costs associated with risks: In the 2017 version of the estimate, estimated costs related to program risks were allocated to their corresponding WBS element  More generally, factors that contributed to cost fluctuations between the 2015 and 2017 cost estimates include: changes in assumptions for census operations, improved ability to anticipate and quantify risk, an overall increase in IT costs, and more defined contract requirements  Several assumptions for the implementation of the 2020 Census have changed since the 2015 cost estimate  Some assumptions contributing to cost changes, mainly in the Response (related to collecting and processing response data) and Frame (the mapping and collecting addresses to frame enumeration activities) categories, include the following: Self-response rates  Changes in assumptions for expected self- response rates contributed to increases in the response category, as the assumed rate decreased from 63 5 percent in 2015 to 60 5 percent in 2017, thereby increasing the anticipated percentage and associated cost of nonresponse follow-up  When the Bureau does not receive responses by mail, phone, or Internet, census enumerators visit each nonresponding household to obtain data  Thus, reduced self-response rates lead to increases in the amount of data collected in the field, which is more costly to the Bureau  Bureau officials attributed this decrease to a forecasted reduction in Internet response due to added authentication steps at log in and the elimination of the function allowing users to save their responses and return later to complete the survey  Productivity rates  The productivity of enumerators collecting data for NRFU is another variable in the cost estimate that was updated, contributing to cost increases in the response category  Expected productivity rates for NRFU decreased from the 2015 estimate of 4 attempts per hour to 2 9  According to Bureau documentation, this more conservative estimate is based on historical data, rather than research and test data  In-office address canvassing rates  The Bureau will not go door-to- door to conduct in-field address canvassing across the country to update address and map information for every housing unit, as it has in prior decennial censuses  Rather, some areas would only need a review of their address and map information using computer imagery and third-party data sources—what the Bureau calls “in-office” address canvassing procedures  However, in March 2017, citing budget uncertainty the Bureau decided to discontinue one of the phases of in-office review address canvassing for the 2020 Census  The cancellation of that phase of in-office review is expected to increase the number of housing units canvassed in-field by 5 percent (from 25 to 30 percent of all canvassed housing units)  In-field canvassing is more labor intensive compared to in office procedures  The 2017 version of the cost estimate reflects this increase in workload for in-field address canvassing, though overall changes in estimated costs for the Frame category, of which Address Canvassing is a part, were minimal  Staffing  Updated analysis resulted in changes to several staffing assumptions, which resulted in decreases across WBS categories  Changes included reduced pay rates for field data collection staff based on current labor market conditions and reductions in the length of staff engagement  In general, contingency allocations increased overall from $1 35 billion in 2015 to $2 6 billion in 2017  This increase in contingency can be attributed, in part, to the Bureau gaining a clearer understanding of risk and uncertainty in the 2020 Census as it approaches  The Bureau developed some of its contingency based on proven risk management techniques, including Monte Carlo analysis and allocated funding for known risk scenarios  The 2017 estimate includes close to $1 4 billion in estimated costs for these risks, almost three times the amount included in the 2015 estimate  The basis of estimate contains detail on the various risks and the process for calculating the associated contingency  The 2017 version also includes a contingency amount of $1 2 billion for general risks, or unknown-unknowns, such as natural disasters and cyber-attacks  Contingency amounts were reallocated within the WBS to more closely reflect the nature of the risk: Bureau officials attribute a decrease from the 2015 estimate of $551 million in estimated costs for program management to changes in the categorization of costs associated with risks  Officials stated that, in 2015, discrete program risks were previously consolidated as program management costs  In 2017, these discrete costs were reallocated to associate risks with the appropriate WBS element  For example, contingency amounts related to the likelihood of achieving a certain response rate previously included in the program management work breakdown category are now a part of the “response” work breakdown category  Increases in IT costs, totaling $1 59 billion, represented almost 50 percent of the total cost increase from 2015 to 2017  The total share of IT costs as a percentage of total census costs increased from 28 percent in 2015 to 32 percent in 2017, or from $3 41 billion to approximately $5 billion  Increases in IT costs are spread across seven cost categories  Figure 10 shows the IT and non-IT cost by WBS for the 2017 cost estimate  IT costs in infrastructure, response data, and census/survey WBSs account for the majority of the approximately $5 billion  The Bureau’s October 2015 cost estimate included IT costs for, among other things, system engineering, test and evaluation, and infrastructure, as well as for a portion of the Census Enterprise Data Collection and Processing (CEDCaP) program  The 2017 estimated IT cost increases were due, in large part, to the Bureau (1) updating the cost estimate for CEDCaP; (2) including an estimate for technical integration services that contributed to increases in the Census and Survey Engineering category; and (3) updating costs related to other major contracts (such as mobile device as a service, field IT services, and payroll systems)  Bureau documents described an overall improvement in the Bureau’s ability to define and specify contract requirements  This resulted in updated estimates for several contracts, including for the Census Questionnaire Assistance (CQA) contract  Assumptions regarding call volume to the CQA were increased by 5 percent to account for expected response by phone after the elimination of the option to save Internet responses and return to complete the form later  The Bureau also cited updated cost data and the results of reconciliation with independent cost estimates as factors contributing to the increased costs of other major contracts, including for the procurement of data collection devices  The Secretary of Commerce provided comments on a draft of this report on August 2, 2018  The comments are reprinted in appendix II  The Department of Commerce generally agreed with our findings regarding the improvements the Census Bureau has made in its cost estimates  However, Commerce did not agree with our assessment that the Bureau’s 2017 lifecycle cost estimate is “not reliable ” Commerce noted that it had conducted two independent cost analyses and was satisfied that the cost estimate was reliable  The Bureau also provided technical comments that we incorporated, as appropriate  We maintain that, to be considered reliable, a cost estimate must meet or substantially meet the criteria for each of the four characteristics outlined in our Cost Estimating and Assessment Guide  These characteristics are derived from measures consistently applied by cost estimating organizations throughout the federal government and industry and are considered best practices for the development of reliable cost estimates  Without a reliable cost estimate, the Bureau is limited in its ability to make informed decisions about program resources and to effectively measure progress against operational objectives  Thus, while the Bureau has made considerable progress in all four of the characteristics, it has only partially met the criteria for the characteristic of being well-documented  Until the Bureau meets or substantially meets the criteria for this characteristic, the cost estimate cannot be considered reliable  As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date  At that time, we will send copies of the report to the appropriate congressional committees, the Secretary of Commerce, the Under Secretary of Economic Affairs, the Acting Director of the U S  Census Bureau, and other interested parties  In addition, this report is available at no charge on the GAO website at http://gao gov  If you or your staff have any questions about this report, please contact me at (202) 512-2757 or goldenkoffr@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report  GAO staff who made key contributions to this report are listed in appendix III  The purpose of our review was to evaluate the reliability of the Census Bureau’s (Bureau) life-cycle cost estimate using our Cost Estimating and Assessment Guide  We reviewed (1) the extent to which the Bureau’s life-cycle cost estimate and associated guidance met our best practices for cost estimation using documentation and information obtained in discussions with the Bureau related to the 2020 life-cycle cost estimate and (2) compared the 2015 and 2017 life-cycle cost estimates to describe key drivers of cost growth  For both objectives we reviewed documentation from the Bureau on the 2020 life-cycle cost estimate and interviewed Bureau and Department of Commerce officials  For the first objective, we relied on our Cost Estimating and Assessment Guide as criteria  Our cost specialists assessed measures consistently applied by cost-estimating organizations throughout the federal government and industry and considered best-practices for developing reliable cost estimates  We analyzed the cost estimating practices used by the Bureau against these best practices and evaluated them in four categories: comprehensive, well-documented, accurate, and credible  Comprehensive  The cost estimate should include both government and contractor costs of the program over its full life-cycle, from inception of the program through design, development, deployment, and operation and maintenance to retirement of the program  It should also completely define the program, reflect the current schedule, and be technically reasonable  Comprehensive cost estimates should be structured in sufficient detail to ensure that cost elements are neither omitted nor double counted  Specifically, the cost estimate should be based on a product-oriented work breakdown structure (WBS) that allows a program to track cost and schedule by defined deliverables, such as hardware or software components  Finally, where information is limited and judgments are made, the cost estimate should document all cost-influencing assumptions  Well-documented  A good cost estimate—while taking the form of a single number—is supported by detailed documentation that describes how it was derived and how the expected funding will be spent in order to achieve a given objective  Therefore, the documentation should capture in writing such things as the source data used, the calculations performed and their results, and the estimating methodology used to derive each WBS element’s cost  Moreover, this information should be captured in such a way that the data used to derive the estimate can be traced back to, and verified against, their sources so that the estimate can be easily replicated and updated  The documentation should also discuss the technical baseline description and how the data were normalized  Finally, the documentation should include evidence that the cost estimate was reviewed and accepted by management  Accurate  The cost estimate should provide for results that are unbiased, and it should not be overly conservative or optimistic  An estimate is accurate when it is based on an assessment of most likely costs; adjusted properly for inflation; and contains few, if any, minor mistakes  In addition, a cost estimate should be updated regularly to reflect significant changes in the program—such as when schedules or other assumptions change—and actual costs, so that it is always reflecting current status  During the update process, variances between planned and actual costs should be documented, explained, and reviewed  Among other things, the estimate should be grounded in a historical record of cost estimating and actual experiences on other comparable programs  Credible  The cost estimate should discuss any limitations of the analysis because of uncertainty or biases surrounding data or assumptions  Major assumptions should be varied, and other outcomes recomputed to determine how sensitive they are to changes in the assumptions  Risk and uncertainty analysis should be performed to determine the level of risk associated with the estimate  Further, the estimate’s cost drivers should be cross-checked, and an independent cost estimate conducted by a group outside the acquiring organization should be developed to determine whether other estimating methods produce similar results  If any of the characteristics are not met, minimally met, or partially met, then the cost estimate does not fully reflect the characteristics of a high- quality estimate and cannot be considered reliable  We also analyzed the Bureau’s cost estimation and analysis guidance and evaluated them against a 12-step process outlined in our Cost Estimation and Assessment Guide  A high-quality cost estimating process integrates the following: 1  Define estimate’s purpose  2  Develop estimating plan  3  Define program characteristics  4  Determine estimating structure  5  Identify ground rules and assumptions  6  Obtain data  7  Develop point estimate and compare it to an independent cost estimate  8  Conduct sensitivity analysis  9  Conduct risk and uncertainty analysis  10  Document the estimate  11  Present estimate to management for approval  12  Update the estimate to reflect actual costs and changes  These 12 steps, when followed correctly, should result in reliable and valid cost estimates that management can use for making informed decisions  If any of the steps in the Bureau’s process do not meet, minimally meet, or partially meet the 12 steps, then the cost estimate guidance does not fully reflect best practices for developing a high-quality estimate and cannot be considered reliable  Lastly, to describe key drivers of cost growth, we compared cost information included in the 2015 and 2017 cost estimates  We analyzed both summary and detailed cost information to assess key changes in totals overall, by WBS category, and by information technology (IT) vs  Non-IT costs  We used this analysis in conjunction with information received from the Bureau during interviews and through document transfers to describe overall changes in the cost estimate from 2015 to 2017  We conducted this performance audit from December 2017 to August 2018 in accordance with generally accepted government auditing standards  Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives  We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives  In addition to the contact named above, Lisa Pearson (Assistant Director), Karen Cassidy (Analyst in Charge), Brian Bothwell, Jackie Chapin, Ann Czapiewski, Jason Lee, Ty Mitchell, Kayla Robinson, and Tim Wexler made significant contributions to this report 

ground_truth: FEMA, an agency of the Department of Homeland Security (DHS), has obligated more than $36 billion in PA grants to state, local, and tribal governments to help communities recover and rebuild after major disasters since 2009. Further, costs are rising with disasters, such as Hurricanes Harvey, Irma, and Maria in 2017. FEMA recently redesigned how the PA program delivers assistance to state and local grantees to improve operations and address past challenges identified by GAO and others. FEMA tested the new delivery model in selected disasters and announced implementation in September 2017. GAO was asked to assess the redesigned PA program. This report examines, among other things, the extent to which FEMA's new delivery model addresses (1) past workforce management challenges and assesses future workforce needs; and (2) past information sharing challenges and key IT management controls. GAO reviewed FEMA policy, strategy, and implementation documents; interviewed FEMA and state officials, PA program applicants, and other stakeholders; and observed implementation of the new model at one test location following Hurricane Matthew in 2016. The Federal Emergency Management Agency (FEMA) redesigned the Public Assistance (PA) grant program delivery model to address past challenges in workforce management, but has not fully assessed future workforce staffing needs. GAO and others have previously identified challenges related to shortages in experienced and trained FEMA PA staff and high turnover among these staff. These challenges often led to applicants receiving inconsistent guidance and to PA project delays. As part of its new model, FEMA is creating consolidated resource centers to standardize and centralize PA staff responsible for managing grant applications, and new specialized positions, such as hazard mitigation liaisons, program delivery managers, and site inspectors, to ensure more consistent guidance to applicants. However, FEMA has not assessed the workforce needed to fully implement the new model, such as the number of staff needed to fill certain new positions, or to achieve staffing goals for supporting hazard mitigation on PA projects. Fully assessing workforce needs will help to ensure that FEMA has the people and the skills needed to fully implement the new PA model and help to avoid the long-standing workforce challenges the program encountered in the past. FEMA designed a new PA information and case management system—called the FEMA Applicant Case Tracker (FAC-Trax)—to address past information sharing challenges, such as difficulties in sharing grant documentation among FEMA, state, and local officials and tracking the status of PA projects, but additional actions could better ensure effective implementation. Both FEMA and state officials involved in testing of the new model stated that the new information system allows them to better manage and track PA applications and documentation, which could lead to greater transparency and efficiencies in the program. Further, GAO found that this new system fully addresses two of four key information technology (IT) management controls—project planning and risk management—that are necessary to ensure systems work effectively and meet user needs. However, GAO found that FEMA has not fully addressed the other two controls—requirements development and systems testing and integration. By better analyzing progress on high-level user requirements, for example, FEMA will have greater assurance that FAC-Trax will meet user needs and achieve the goals of the new delivery model. GAO is making five recommendations, including that FEMA assess the workforce needed for the new delivery model and improve key IT management controls for its new information sharing and case management system, FAC-Trax. DHS concurred with all recommendations.