prompt_front: You are given two reports by the government agency.



Question: Write a one-page summary of the report 1.

Report 1: NASA’s Commercial Crew Program is a multi-phased effort that began in 2010  Across the phases, NASA has engaged several companies, using both agreements and contract vehicles to develop and demonstrate crew transportation capabilities  As the program has passed through these phases, NASA has generally narrowed down the number of participants  The early phases of the program were under Space Act agreements, which is what NASA calls the agreements entered into pursuant to its other transaction authority  These types of agreements are generally not subject to the Federal Acquisition Regulation (FAR) and allow the government and its contractors greater flexibility in many areas  Under these Space Act agreements, NASA relied on the commercial companies to propose specifics related to their crew transportation systems, including their design, the capabilities they would provide, and the level of private investment  In these phases, NASA provided technical support and determined whether the contractors met certain technical milestones  In most cases, NASA also provided funding  For the final two phases of the program, NASA awarded FAR-based contracts  By using FAR-based contracts, NASA gained the ability to procure missions to the ISS, while continuing to provide technical expertise and funding to the contractors  NASA levied two sets of requirements on the contractors: the ISS program requirements, which must be met by all spacecraft visiting the ISS whether they carry cargo or crew; and the Commercial Crew Program requirements, which have a focus on system capabilities and safety rather than design  The program also established a verification closure notice process, in which the contractors submit data to NASA to verify they have met all the requirements to be certified  This certification must occur before contractors are allowed to fly initial crewed missions to the ISS  In September 2014, NASA awarded firm-fixed-price contracts to Boeing and SpaceX, valued at up to $4 2 billion and $2 6 billion, respectively, for the Commercial Crew Transportation Capability phase  Under a firm- fixed-price contract, the contractor must perform a specified amount of work for the price negotiated by the contractor and government  This is in contrast to a cost-reimbursement contract, in which the government generally agrees to pay the contractor’s allowable costs regardless of whether work is completed  During this phase, the contractors will complete development of crew transportation systems  Boeing’s spacecraft—CST-100 Starliner—is composed of a crew module and a service module  The crew module will carry the crew and cargo  It also includes communication systems, docking mechanisms, and return systems for Earth landing  The service module provides propulsion on-orbit and in abort scenarios as well as radiators for thermal control  SpaceX’s spacecraft—Dragon 2—is composed of a capsule, which we refer to as the crew module, and a trunk, which we refer to as the support module  The crew module is composed of a pressure section and a service section  This module will carry the crew and cargo  It also includes avionics, docking mechanisms, and return systems for a water landing  The support module includes solar arrays for on-orbit power and guidance fins for escape abort scenarios  Figure 1 shows the spacecraft and launch vehicles for Boeing and SpaceX’s crew transportation systems  The Commercial Crew Transportation Capability phase contracts include three types of services: Contract Line Item 001 encompasses the firm-fixed-price design, development, test, and evaluation work needed to support NASA’s certification of the contractor’s spacecraft, launch vehicle, and ground support systems  Contract Line Item 002 covers any service missions that NASA orders to transport astronauts to and from the ISS  Under this indefinite-delivery, indefinite-quantity line item, NASA has ordered six post-certification missions from each contractor  Each service mission is its own firm-fixed-price task order  NASA must certify the contractors’ systems before they can fly these missions  Contract Line Item 003 is an indefinite-delivery, indefinite-quantity line item for any special studies, tests, and analyses that NASA may request  These tasks do not include any work necessary to accomplish the requirements under contract line item 001 and 002  As of April 2018, NASA had funded studies worth approximately $30 million to Boeing, including approximately $27 million for additional testing of the parachute system  NASA had funded studies worth approximately $44 million to SpaceX, including approximately $34 million for additional testing of the parachute system  For each contractor, the maximum value of this contract line item is $150 million  NASA has made changes to the contracts that have increased their value  While the contracts are fixed-price, their values can increase if NASA adds work or otherwise changes requirements, among other means  As of April 2018, NASA requirement changes had increased the value of contract line item 001 for Boeing by approximately $191 million and for SpaceX by approximately $91 million  NASA divided the certification work under contract line item 001 into two acceptance events: the design certification review and the certification milestone  An acceptance event occurs when NASA approves a contractor’s designs and acknowledges that the contractor’s work is complete and meets the requirements of the contract  The first acceptance event—the design certification review—verifies the contractor’s crew transportation system’s capability to safely approach, dock, mate, and depart from the ISS, among other requirements  After the contractor has successfully completed all of its flight tests, as well as various other activities, the second acceptance event—the certification milestone—determines whether the crew transportation system meets the Commercial Crew Program’s requirements  Following this contract milestone is an agency certification review, which authorizes the use of a contractor’s system to transport NASA crew to and from the ISS  Figure 2 shows a notional path leading up to the agency certification review  The Commercial Crew Program’s certification plan outlines how the program will incrementally review required deliverables leading up to, and supporting, the agency certification review  For each review, the plan describes the information that the contractor and the program will present  At the agency certification review, which is chaired by the Associate Administrator of the Human Exploration and Operations Mission Directorate, the agency will review the program’s formal recommendation to certify the contractor’s crew transportation system  Program officials said that their goal is to develop and review certification evidence incrementally in order to reduce the risk that issues will be identified during the agency certification review  In our February 2017 report, we evaluated the progress made by the two contractors on the Commercial Crew Program and found the following: Both of the Commercial Crew Program’s contractors had made progress developing their crew transportation systems, but both also had aggressive development schedules that were increasingly under pressure  We reported that both Boeing and SpaceX had determined that they would not be able to meet their original 2017 certification dates, and both expected certification to be delayed until 2018  We found that the schedule pressures were amplified by NASA’s need to provide a viable crew transportation option because its contract with Russia’s space agency was to provide crew transportation to the ISS for six astronauts through 2018 with rescue and return through late spring 2019  Purchasing additional seats from Russia involves a contracting process that typically takes 3 years  Without a viable contingency option for ensuring uninterrupted access to the ISS in the event of further Commercial Crew delays, we concluded that NASA was at risk of not being able to maximize the return on its multibillion dollar investment in the space station  The Commercial Crew Program was using mechanisms laid out in its contracts to gain a high level of visibility into the contractors’ crew transportation systems, but maintaining that level of visibility through certification could add schedule pressures  We noted that, for example, due to NASA’s acquisition strategy for this program, its personnel were less involved in the testing, launching, and operation of the crew transportation system  While the program developed productive working relationships with both contractors, obtaining the level of visibility that the program required had also taken more time than the program or contractors had anticipated  Ultimately, we noted that the program had the responsibility for ensuring the safety of U S  astronauts, and its contracts gave it deference to determine the level of visibility required to do so  We concluded that the program office could face difficult choices moving forward about how to maintain the level of visibility it feels it needs without adding to the program’s schedule pressures  In order to ensure that the United States had continued access to the ISS if the Commercial Crew Program’s contractors experienced additional schedule delays, we recommended in our February 2017 report that the NASA Administrator develop a contingency plan for maintaining a presence on the ISS beyond 2018, including options to purchase additional Russian Soyuz seats, and report to Congress on the results  NASA concurred with this recommendation, and in February 2017, NASA executed a contract modification that purchased two seats and included an option to purchase three additional crewmember seats from Boeing on the Russian Soyuz vehicle  These seats represent a contingency plan for U S  access to the ISS through 2019  In April 2017, NASA informed the Congress of this action  Boeing and SpaceX continue to make progress developing their crew transportation systems, but both contractors have further delayed the certification milestone to early 2019  These changes have occurred as the contractors continue to work to aggressive schedules, and they have had to delay key events regularly  Further delays are likely as the Commercial Crew Program’s schedule risk analysis shows that the certification milestone is likely to further slip  In addition, as of mid-June 2018, NASA officials told us that these dates may change soon but that both contractors have not yet provided official updates to their schedules to NASA  NASA has not fully shared information with Congress regarding the risks of future schedule delays for the contractors and, as a result, Congress lacks insight into when the contractors will be certified  Also, there may be a gap in access to the ISS if the Commercial Crew Program experiences additional delays  While NASA has begun to discuss potential options, it currently does not have a contingency plan for how to ensure an uninterrupted presence on the ISS beyond 2019  Boeing and SpaceX have continued to make progress finalizing their designs and building hardware as they work toward their certification milestones  The contractors are manufacturing test articles to demonstrate system performance and flight spacecraft to support the uncrewed and crewed flight tests, which are expected to demonstrate the ability to meet contract requirements  As table 1 shows, these test articles and spacecraft vary in levels of completion  Some are built and undergoing testing while others are starting the manufacturing phase  Should any issues arise during integration and test or the flight tests, the contractors may have to complete rework on the spacecraft already under construction  While both contractors are making progress, the Commercial Crew Program is tracking risks that each contractor has to address through testing and other means as they work towards the certification milestone  As we have previously reported, these types of risks are inherent in NASA’s major acquisitions, which are highly complex, specialized, and often pushing the state of the art in space technology, but they could also delay the contractors’ progress if issues arise during testing  The Commercial Crew Program’s top programmatic risks identified for Boeing include challenges related to its abort system performance, parachutes, and launch vehicle  Abort System: Boeing is addressing a risk that its abort system, which it needs for human spaceflight certification, may not meet the program’s requirement to have sufficient control of the vehicle through an abort  In some abort scenarios, Boeing has found that the spacecraft may tumble, which could pose a threat to the crew’s safety  To validate the effectiveness of its abort system, Boeing has conducted extensive wind tunnel testing and plans to complete a pad abort test in July 2018  Parachute System: Boeing is also addressing a risk that during descent, a portion of the spacecraft’s forward heat shield may re- contact the spacecraft after it is jettisoned and damage the parachute system  Boeing’s analysis indicates the risk exists only if one of two parachutes that pull the forward heat shield away from the spacecraft does not deploy as expected, and that potential re-contact is non- detrimental  However, NASA’s independent analysis indicates that this may occur even if both parachutes deploy as expected  If the program determines this risk is unacceptable, Boeing would need to redesign the parachute system, which the program estimates could result in at least a 6-month delay  Launch Vehicle Data: One of the program’s top programmatic and safety concerns is that it may not have enough information from Boeing’s launch vehicle provider, United Launch Alliance, to assess whether the Atlas V launch vehicle prevents or controls cracking that could lead to catastrophic failures  NASA estimates that unfinished work in this area could take Boeing and the United Launch Alliance until the fourth quarter of 2018 to complete  Additionally, the first stage of the Atlas V is powered by the Russian built RD-180 engine, and, according to program and Boeing officials, access to its data is highly restricted by agreements between the U S  and Russian governments  Since our last report, the Commercial Crew Program has lowered the risk that certification of the launch vehicle might not occur by negotiating steps to access necessary data, but work is still ongoing  The Commercial Crew Program’s top programmatic risks identified for SpaceX are in part related to ongoing design and development efforts related to its launch vehicle design, the Falcon 9 Block 5  Composite Overwrap Pressure Vessel: This Block 5 design includes SpaceX’s redesign of the composite overwrap pressure vessel, which is intended to contain a gas under high pressure  SpaceX officials stated the newly designed vessel aims to eliminate risks identified in the older design, which was involved in an anomaly that caused a mishap in September 2016  SpaceX plans to qualify the updated design for flight prior to the uncrewed flight test design certification review  Engine Turbine Cracking: The Block 5 design also includes design changes to address cracks in the turbine of its engine identified during development testing  NASA program officials told us that they had informed SpaceX that the cracks were an unacceptable risk for human spaceflight  SpaceX officials told us that they have made design changes to this Block 5 upgrade that did not result in any cracking during initial testing  However, this risk will not be closed until SpaceX successfully completes qualification testing in accordance with NASA’s standards without any cracks  As of March 2018, SpaceX had not yet completed this testing  Propellant Loading Procedures: Both the program and a NASA advisory group have raised SpaceX’s plan to fuel the launch vehicle after the astronauts are on board the spacecraft to be a potential safety risk  In the May 2018 meeting minutes, however, the Aerospace Safety Advisory Panel stated that with appropriate controls in place, this approach could be a viable option for the program to consider  SpaceX’s perspective is that this operation may be a lower risk to the crew because it reduces the crew exposure time while the launch vehicle is being loaded with propellant  To better understand the propellant loading procedures, the program and SpaceX agreed to demonstrate the loading process five times from the launch site in the final crew configuration prior to the crewed flight test  The five events include the uncrewed flight test and the in-flight abort test  Therefore, delays to those events would lead to delays to the agreed upon demonstrations, which could in turn delay the crewed flight test and certification milestone  Both contractors have notified NASA that their certification milestones have slipped to January 2019 for Boeing and February 2019 for SpaceX, but the Commercial Crew Program’s schedule risk analysis indicates more delays are likely  This analysis identifies a range for each contractor, with an earliest and latest possible completion date, as well as an average  In April 2018, the program’s schedule risk analysis found there was zero percent chance that either contractor would achieve its current proposed certification milestone  The analysis’s average certification date was December 2019 for Boeing and January 2020 for SpaceX  Figure 3 shows the original Boeing and SpaceX contract schedules and the current proposed schedule for five key events in each contract, as well as NASA’s schedule risk analysis for the certification milestone  Each month, the program updates its schedule risk analysis based on the contractors’ internal schedules as well as program officials’ perspectives and insight into specific technical risks  The Commercial Crew Program manager told us that differences between the contractors’ proposed schedules and the program’s schedule risk analysis include: The contractors are aggressive and use their schedule dates to motivate their teams, while NASA adds additional schedule margin for testing  Both contractors assume an efficiency factor in getting to the crewed flight test that NASA does not factor into its analysis  The program manager also told us that the program meets with each contractor monthly to discuss schedules and everyone agrees to the relationships between events in the schedule even if they disagree on the length of time required to complete events  The program manager added, however, that she relies on her prior experience to estimate schedule time frames as opposed to relying on the contractors’ schedules, which are often optimistic  Our analysis also shows that the contractors often delay their schedules  Both contractors have repeatedly stated that their schedules are aggressive and have set ambitious—rather than realistic—dates, only to frequently delay them  Since the current contracts were awarded in 2014, the Commercial Crew Program has held 13 quarterly reviews for each contractor  For the five key events identified above, Boeing has reported a delay at 7 of those quarterly reviews and SpaceX has reported a delay at 9 of them  In mid-June 2018, NASA officials told us that the dates for these key events may change soon  The information presented in Figure 3 above is based on first quarter calendar year 2018 data  NASA officials stated both contractors have not yet officially communicated new schedule dates to NASA as of the second quarter calendar year 2018  We found that both contractors have updated schedules that indicate delays are forthcoming for at least one key event, but NASA officials told us they lack confidence in those dates until they are officially communicated to NASA by the contractors  As a result, NASA is managing a multibillion dollar program without confidence in its schedule information as it approaches several big events, including uncrewed and crewed flight tests  The risk of future delays in the contractors’ schedules is critical information that NASA has not fully shared with Congress  Moreover, NASA has not yet developed a contingency plan to address the potential gaps that these delays could have on U S  access to the ISS after 2019  Specifically, in the Explanatory Statement accompanying the fiscal year 2018 Consolidated and Further Continuing Appropriations Act, the House Appropriations Committee stated its expectation that NASA report quarterly to the Senate and House Committees on Appropriations on the status of the Commercial Crew Program contracts  Previously, members of Congress had asked for this information in order to ensure that Congress had adequate insight into this program  While NASA includes both contractors’ proposed schedules in its quarterly report to Congress, NASA does not include the results of its own schedule risk analysis  Given the frequency with which the contractors delay key events in their schedules, the program’s schedule risk assessment provides valuable insight into potential delays that NASA currently is not providing to Congress  In addition, as previously mentioned, NASA executed a contract modification that purchased two seats and included an option to purchase three additional crew member seats through Boeing for an undisclosed value and reported this action to Congress in April 2017  Ultimately, the option was exercised, and NASA purchased a total of five seats on four different Soyuz flights  Boeing obtained these seats through a separate settlement with the Russian firm RSC Energia, which manufactures the Soyuz  These seats were intended to serve as a contingency plan based on schedule information available at that time  However, subsequent delays, as well as the risk of future delays as discussed above, indicate that this contingency plan will likely no longer be sufficient  The earliest and latest possible completion dates for certification in NASA’s April 2018 schedule risk analysis indicate it is possible that neither contractor would be ready before August 2020, leaving a potential gap in access of at least 9 months  We calculated the potential gap based on the contractor certification milestone dates, but there could be some additional time required between that review and the first post-certification service mission to the ISS  As seen in figure 4, if the contractors can maintain their current proposed schedules for their respective certification milestones, a gap in access to the ISS is not expected  However, there would be a gap in access to the ISS if neither contractor has its certification milestone before November 2019, which is when NASA expects the final Russian Soyuz seat for a U S  astronaut to return  Senior NASA officials told us that sustaining a U S  presence on the ISS is essential to maintain and operate integral systems, without which the ISS cannot function  Given the importance of maintaining a U S  presence on the ISS, NASA officials have stated they are working on options to address the potential gap in access  However, officials told us that planning for contingencies is difficult given the extensive international negotiations required for some options  Obtaining additional Soyuz seats seems unlikely, as the process for manufacturing the spacecraft and contracting for those seats typically takes 3 years—meaning additional seats would not be available before 2021  As a result, according to NASA’s Associate Administrator for Human Exploration and Operations, the options NASA is considering include: Refine the remaining Soyuz launch schedule to allow for a return in January 2020, as opposed to November 2019  This would provide 2 additional months of access to the ISS before the commercial crew flights need to start  Use the crewed flight tests as operational flights to transport U S  astronauts to and from the ISS  In March 2018, NASA modified Boeing’s contract to allow NASA to add a third crew member and extend the length of the flight test, if NASA chooses to do so  This would have limited usefulness, however, in filling a potential gap in access to the ISS if the schedule for Boeing’s crewed flight test slips past the return date for the last Soyuz flight and SpaceX also continues to experience delays  NASA’s Associate Administrator for Human Exploration and Operations stated that he is “brainstorming” other options to ensure access to the ISS but does not have a formal plan  While options are not unlimited and decisions have to be made within the context of the current geopolitical environment, Congress stated in the NASA Authorization Act of 2005 that it is U S  policy to possess the capability for human access to space on a continuous basis  In 2010, Congress further stated that one of the key objectives of the United States’ human spaceflight policy is to sustain the capability for long-duration presence in low-Earth orbit through full utilization of the ISS  If NASA does not develop options for ensuring access to the ISS in the event of further Commercial Crew delays, it will not be able to ensure that the U S  policy goal and objective for the ISS will be met  The Commercial Crew Program relies on several contractual mechanisms to assess safety throughout the certification process, and those mechanisms are in varying stages of completion  The program itself, its contractors, and two of NASA’s independent review organizations have raised concerns about the program’s ability to assess and evaluate all of the deliverables in a timely manner  In addition, one of the key safety requirements levied by the program is loss of crew, which captures the probability of death or disability to a crew member  NASA does not have a consistent approach for how to incorporate key inputs to assess this metric, which means the agency as a whole may not clearly capture or document its risk tolerance with respect to loss of crew  Further, the program’s chief safety and mission assurance officer is dual hatted to serve simultaneously in a programmatic position as well as the program’s safety technical authority  This approach creates an environment of competing interests because it relies on the same individual to manage technical and safety aspects on behalf of the program while also serving as the independent oversight of those same areas  The contractors are required to provide several key deliverables to the Commercial Crew Program, which inform the agency certification review and help NASA determine the level of risk it is accepting with respect to safety of each spacecraft  As described below, these deliverables are in varying stages of completion and the program itself, its contractors, and two of NASA’s independent review organizations have raised concerns about the program’s ability to assess and evaluate all of the deliverables in a timely manner  Certification Data Package  Among other things, the certification data package includes a list of seven system safety assessments  For example, the certification data package includes a fault tolerance assessment, which describes the system’s ability to sustain a certain number of undesired events, such as software or operational anomalies  A human error analysis—one of the seven assessments in the data package—evaluates human errors to minimize their negative effects on the system  Boeing held its uncrewed flight test design certification review in December 2017 and submitted its certification data package for NASA approval  Boeing plans three more updates to this data package prior to the final certification milestone  SpaceX has begun to submit data and plans to submit its final certification data package as part of its crewed flight test design certification review, which is scheduled for September 2018  According to the program’s certification review plan, program officials will review and approve the contractors’ certification data packages, which will be used to inform the agency certification review  Phased Safety Review Process  A three-phased safety review process informs the program’s quality assurance activities, and it is intended to ensure that the contractors have identified all safety-critical hazards and implemented associated controls prior to the first crewed flight test  In phase one, the contractors identified risks in their designs and developed reports on potential hazards, the controls they put in place to mitigate them, and explanations for how the controls will mitigate the hazards  In phase two, which is nearing completion, the program reviews and approves the contractors’ hazard reports and develops strategies to verify and validate that the controls are effective  For example, if a control requires that an item be waterproofed, verification and validation strategies could include inspections and tests to confirm that the item is waterproof  As of April 2018, the program had yet to complete this phase, having approved 97 percent of Boeing’s phase two reports and 72 percent of SpaceX’s phase two reports  In phase three, the contractors will conduct the verification activities and submit the hazard reports to the program for approval  The program has begun phase three, including approving 19 percent of Boeing’s phase three reports  Program Requirements  While the program manager told us that all of the requirements contribute to the safety of the commercial systems, safety officials are required to approve a subset of these requirements  Examples of requirements approved by safety officials include the ability to leave the spacecraft in an emergency or to abort a launch  When a contractor is ready for NASA to verify that it has met a requirement, the contractor submits data for NASA to review through a verification closure notice  We define “safety-specific notices” as those requiring safety officials’ approval  As shown in table 2, as of March 2018, the program had approved 2 percent of Boeing’s safety-specific notices and 0 percent of SpaceX’s safety-specific notices  Testing  The program also requires testing to verify and validate the crew transportation system  Agency officials emphasized the importance of testing to safety, stating that testing reduces uncertainty about a system’s performance and can uncover unknown problems  As noted above, both contractors will be conducting an uncrewed and a crewed flight test prior to being certified  While a certain level of risk needs to be accepted to conduct human spaceflight, these flight tests help to mitigate this risk by validating the integrated performance of the hardware and software  Agency and program officials stated that the contractors’ flight tests are critical evidence to support certification of a safe and reliable system  As evidenced by the data above, the program still has a significant amount of work ahead with respect to approving certification packages and closing hazard reports and verification closure notices  We have previously found that the program’s workload was an emerging schedule risk, and the contractors have continued to express concern about program officials’ ability to process and approve certification paperwork in a timely manner  Workload has also been a concern for two of NASA’s independent review organizations  For example, the Aerospace Safety Advisory Panel noted in its January 2018 annual report that the sheer volume of work that remains for the program in terms of closing hazard reports and verification closure notices is significant  In addition, the program’s safety and mission assurance office identified the upcoming bow wave of work in a shrinking time period as a top risk to achieving certification  One mechanism the program put in place to assess the overall safety of each spacecraft—loss of crew—has been a focus of the Aerospace Safety Advisory Panel, Members of Congress, our prior work, and the program itself  Loss of crew captures the probability of death or permanent disability to one or more crew members  It has received a lot of attention, in part, because it has been a top risk for the program since 2015  Specifically, the program has been concerned that neither contractor would be able to meet the contract requirement of a 1 in 270 probability of incurring loss of crew  We identified two key concerns with how NASA is using the loss of crew metric: (1) inconsistent approaches to assess the loss of crew metric and (2) no identified plan to share lessons learned about using the loss of crew metric as a safety threshold  A loss of crew value is generated through a probabilistic safety analysis, which models scenarios that could result in the loss of crew using various inputs  According to the program’s analysis, the probability of on-orbit debris damaging the vehicle has the greatest effect on a loss of crew value  This probability is informed by an orbital debris (debris) model, which was updated in 2014, after the loss of crew requirement was established  The updated debris model makes it harder to meet a loss of crew value, in part, because the modeling environment where the contractors’ systems will operate has changed  For example, the updated model includes a larger span of orbit, greater range of debris sizes, and the addition of material density classifications, which were not included in the former model  Further, the probabilistic safety analysis may include operational mitigations, such as on-orbit inspections that would include using cameras on the ISS to visually survey the spacecraft for damage, which, according to officials, makes it easier to meet a loss of crew value  NASA describes the probabilistic safety analysis as a powerful tool that should be used as part of the overall risk management process to ensure the risk associated with development and operation of a system is understood, evaluated, managed, and mitigated  However, we found differences in the approaches that officials plan to use to assess loss of crew as well as in the loss of crew value being measured that could limit the usefulness of this tool  Agency Certification  The agency certification review for each contractor will include an assessment of whether its crew transportation system meets a loss of crew threshold of 1 in 150 for missions to the ISS, which is based on a May 2011 safety memo from the Office of Safety and Mission Assurance  A loss of crew value with a higher denominator, such as 1 in 270, is harder to meet than with a lower denominator, such as 1 in 150  According to the Chief of the Office of Safety and Mission Assurance, he will assess the 1 in 150 threshold using a probabilistic safety analysis that includes the updated debris model and operational mitigations, such as the on- orbit inspections cited above  Program Office  According to program officials, they will assess whether either contractor meets a 1 in 270 loss of crew value based on a probabilistic safety analysis using the former debris model (not the updated model) and not including operational mitigations  Contracting Officer  According to the contracting officer, each contractor’s loss of crew requirement is 1 in 270 without including operational mitigations  The contracting officer stated that SpaceX’s contract requirement uses the updated debris model in the probabilistic safety analysis, whereas Boeing’s contract requirement uses the former debris model in the probabilistic safety analysis  Program’s Chief Safety and Mission Assurance Officer  According to the program’s chief safety and mission assurance officer, he will conduct a probabilistic safety analysis using the updated debris model and will not include operational mitigations to assess whether each contractor meets a 1 in 200 loss of crew value  This loss of crew value stems from a program update that occurred after the initial contracts were signed  These different approaches are summarized in table 3 below  Agency policy requires human spaceflight programs to set a safety threshold, which NASA did for the Commercial Crew Program when it identified the 1 in 150 loss of crew threshold in the May 2011 safety memo  Subsequently, the program set more rigorous loss of crew values in contract and program documents  NASA also updated the debris model, which we previously noted makes it more difficult to meet a loss of crew value  As a result, NASA does not have a consistent approach for how to incorporate key inputs to the probabilistic safety analysis, including changes to the debris model  Instead, the risk tolerance that NASA is accepting with loss of crew varies based upon which entity is presenting the results of its probabilistic safety analysis  For example, it is possible that the program’s assessment will determine that neither contractor will meet the 1 in 270 contract requirement, but that the agency’s assessment will determine that the contractors meet the 1 in 150 agency certification value because that analysis will include operational mitigations  Or the program’s assessment could determine that Boeing meets the 1 in 270 contractual loss of crew requirement, but the agency’s assessment may determine that Boeing does not meet the 1 in 150 agency certification value because that analysis will use the updated debris model  Federal internal controls state that agency management should define objectives clearly to enable the identification of risks and define risk tolerances  Specifically, management should define risk tolerances in specific and measurable terms, so they are clearly stated and can be measured  In this case, because there will be multiple analyses conducted using different inputs, NASA risks not clearly capturing or documenting, in a coherent manner, its overall risk tolerance with respect to loss of crew before a final decision must be made on whether to certify either crew transportation system  Moreover, capturing the challenges and lessons learned from using the loss of crew metric is critical, particularly because agency officials told us that this is the first time this metric has been used as a safety threshold  Also, there are different viewpoints about the utility of the metric as a safety threshold across the agency  The program manager repeatedly told us that loss of crew is best used as a design tool  For example, program officials told us that both contractors incorporated additional orbital debris shielding into their designs to mitigate the orbital debris risk and improve their loss of crew values  In addition, the Aerospace Safety Advisory Panel reported in 2018 that loss of crew should not be viewed as an absolute measure of actual risk during operations  However, the May 2011 agency safety memo states that a breach of the loss of crew threshold would initiate a termination review of the Commercial Crew Program, which is a more strict application of the loss of crew metric  Both program and safety officials told us that, after the agency certification is complete and lessons learned are available to be compiled, sharing those lessons learned across NASA would be a good idea given the complexities associated with assessing the loss of crew metric  As of April 2018, however, agency officials said they did not have a plan for loss of crew knowledge-sharing  We have previously found that lessons learned provide a powerful method of sharing good ideas for improving work processes, facility or equipment design and operation, quality, safety, and cost-effectiveness  Further, according to NASA’s Knowledge Policy on Program and Projects, which is managed through the Office of the Chief Engineer, a principle of each center and mission directorate’s knowledge strategy is that knowledge is the cornerstone of NASA’s ability to achieve mission success  The policy acknowledges that NASA faces continuous challenges in using what it knows effectively  These challenges include, but are not limited to, enabling the identification and flow of knowledge across organizational boundaries; preserving knowledge at risk of being lost; and providing means for individuals, teams, and the organization to learn from experiences  If NASA does not capture lessons learned from the Commercial Crew Program on using the loss of crew requirement to set a program’s safety threshold and whether it met the agency’s intended goal, future programs will not be able to benefit from the knowledge gained from this multibillion dollar investment  NASA’s governance model prescribes a management structure that employs checks and balances among key organizations to ensure that decisions have the benefit of different points of view and are not made in isolation  As part of this structure, NASA established the technical authority process as a system of checks and balances to provide independent oversight of programs and projects in support of safety and mission success through the selection of specific individuals with delegated levels of authority  The technical authority process has been used in other parts of the government for acquisitions, including the Department of Defense and Department of Homeland Security  The Commercial Crew Program is organizationally connected to three technical authorities within NASA: the Office of the Chief Engineer technical authority, the Office of Chief Health and Medical technical authority, and the Office of Safety and Mission Assurance (safety) technical authority  The safety technical authority is responsible for ensuring from an independent standpoint that the program’s products and processes satisfy NASA’s safety, reliability, and mission assurance policies  The NASA safety technical authority has delegated authority through the Kennedy Space Center Director to the Chief Safety and Mission Assurance Officer for the Commercial Crew Program  We have previously reviewed how NASA has organized its technical authorities for its Exploration Systems Development organization—an organization that oversees the development of the Space Launch System, Orion crew capsule, and associated ground systems that have the goal of extending human presence beyond low-Earth orbit  In October 2017, we found that the Exploration Systems Development organization had established an organizational structure in which the technical authorities for engineering and safety and mission assurance were dual hatted simultaneously in programmatic positions  We found that having the same individual simultaneously fill both a technical authority role and a program role created an environment of competing interests, where the technical authority’s ability to impartially and objectively assess the programs while at the same time acting on behalf of the Exploration Systems Development organization in programmatic capacities may be subject to impairments  We found that this was in contrast to a recommendation from the Columbia Accident Investigation Board report—the result of an in-depth assessment of the technical and organizational causes of the 2003 Space Shuttle Columbia accident—for NASA to establish a technical authority to serve independently of the Space Shuttle program, so that employees would not feel hampered to bring forward safety concerns or disagreements with programmatic decisions  The board’s findings that led to this recommendation included a broken safety culture in which it was difficult for minority and dissenting opinions to percolate up through the hierarchy; dual center and programmatic roles vested in one person that had confused lines of authority, responsibility, and accountability and made the oversight process susceptible to conflicts of interest; and oversight personnel in positions within the program, increasing the risk that these staffs’ perspectives would be hindered by too much familiarity with the programs they were overseeing  In October 2017, we recommended that the division no longer dual hat two individuals who had both programmatic and technical authority responsibilities  As of April 2018, NASA had taken steps to separate the engineering technical authority position from the programmatic position, and NASA’s Chief of Safety and Mission Assurance said he planned to separate the safety position but had not yet completed that action  The Commercial Crew Program employs a similar structure to the Exploration Systems Development organization in that the safety technical authority is dual hatted simultaneously in a programmatic position as the Commercial Crew Program’s Safety and Mission Assurance Manager  According to the program’s safety technical authority, in his programmatic role for the program, he helps set priorities for safety issues, including how staff will be utilized to meet those priorities  In the technical authority role, he provides independent oversight in support of safety and mission success  In his dual-hatted role, this official will be responsible for endorsing the program’s certification recommendations in two different capacities: as the technical authority and as a program authority  As a result, this structure relies on the same individual to completely separate two roles—one to manage the Commercial Crew Program’s safety issues within programmatic cost and schedule constraints, and the other to assess the same issues in an independent oversight role  While the Commercial Crew Program may have an additional level of separation between the safety technical authority and the program’s involvement in the design of commercial systems due to its shared assurance model with the commercial providers, the Commercial Crew Program still maintains a structure where one individual simultaneously serves in both technical authority and programmatic roles  Figure 5 describes some of the conflicting roles and responsibilities of this official in his two different positions  During our review, officials cited several factors in support of a dual- hatted approach: The safety technical authority retains independence because his technical authority reporting path and performance reviews are not under the purview of the Commercial Crew Program chain of command  Due to the Commercial Crew Program’s shared assurance model with commercial providers, the program is operating in a quality assurance role that provides an additional level of separation between the safety technical authority and the program’s involvement in the design of commercial systems  For safety decisions involving cost and schedule where the individual who is dual hatted with both technical authority and programmatic responsibilities may feel conflicted, he stated that he would discuss these matters with his management to validate the logic behind his decision  There are cost and knowledge efficiencies gained from one individual serving in both programmatic and safety technical authority capacities  NASA’s Chief of Safety and Mission Assurance stated that he has great confidence in the individual currently serving in the dual hatted role for the Commercial Crew Program, but acknowledged there is inherent conflict even with the program’s shared assurance model  In December 2017, he stated that, based on our previous work and current discussions, he intends to decouple the programmatic and technical authority responsibilities for the Commercial Crew Program but had not done so as of April 2018  Federal internal control standards state that an agency should design control activities to achieve objectives and respond to risks, which includes segregation of key duties and responsibilities to reduce the risk of error, misuse, or fraud  By overlapping technical authority and programmatic responsibilities, NASA will continue to run the risk of creating an environment of competing interests for the Commercial Crew Program’s safety technical authority  NASA’s Commercial Crew Program is a multibillion dollar effort to facilitate the commercial development of a crew transportation system that can end the United States’ reliance on Russia to maintain an uninterrupted presence on the ISS  Boeing and SpaceX continue to make progress developing a capability to fly to the ISS, but both have continued to experience delays  Program analysis indicates risks of further delays in each contractor’s current schedule, but NASA has not provided that information to Congress in its routine briefings  Without this information, Congress does not know the full extent of potential delays to inform decision making  Additional delays could also disrupt U S  access to the ISS  While NASA is working on potential solutions, there is no contingency plan in place to address this potential gap  Without a viable contingency plan, NASA puts at risk achievement of the U S  goal and objective for the ISS  NASA must balance safety with acceptable risk for human spaceflight  As part of the certification process for each contractor’s spacecraft, NASA has developed one key safety metric, loss of crew  However, the complicated nature of this metric is further muddled by the inconsistent approaches being used across NASA about what inputs to be considered  As a result, there is no clear articulation of what level of risk NASA will accept with respect to this program  In addition, NASA does not have plans to capture lessons learned from how the Commercial Crew Program has used this metric to assess safety and is missing an opportunity to capture this knowledge for future human spaceflight programs  Finally, a space program’s management and oversight approach is an integral part of ensuring that human spaceflight is as safe and successful as possible  Independence of the program management and oversight functions is key to achieving the balance between safety and success  The Commercial Crew Program’s approach, however, burdens the safety technical authority with both programmatic and independent technical authority responsibilities  As a result, NASA has limited assurance that independence can be maintained as part of its institutional process to ensure safety and success  We are making the following five recommendations to NASA: The NASA Associate Administrator for Human Exploration and Operations should direct the Commercial Crew Program to include the results of its schedule risk analysis in its mandatory quarterly reports to Congress  (Recommendation 1) The NASA Administrator should develop and maintain a contingency plan for ensuring a presence on the ISS until a Commercial Crew Program contractor is certified  (Recommendation 2) The NASA Administrator should direct the Chief of Safety and Mission Assurance, the NASA Associate Administrator for Human Exploration and Operations, the Commercial Crew Program Manager, and the Commercial Crew Program Contracting Officer to collectively determine and document before the agency certification review how the agency will determine its risk tolerance level with respect to loss of crew  (Recommendation 3) After completing the agency certification review, NASA’s Chief Engineer and Chief of Safety and Mission Assurance, with support from the NASA Associate Administrator for Human Exploration and Operations and the Commercial Crew Program Manager, should document lessons learned related to loss of crew as a safety threshold for future crewed spaceflight missions, given the complexity of the metric  (Recommendation 4) The NASA Chief of Safety and Mission Assurance should restructure the technical authority within the Commercial Crew Program to ensure that the technical authority for the Office of Safety and Mission Assurance is no longer dual hatted with programmatic and independent technical authority responsibilities  (Recommendation 5) We provided a draft of this report to NASA for review and comment  NASA provided written comments that are reprinted in appendix II  In its response, NASA concurred with three of our recommendations, did not concur with one, and partially concurred with another  NASA concurred with our recommendation to develop and maintain a contingency plan to ensure a U S  presence on the ISS and expects to take action to close this recommendation by the end of December 2018  NASA concurred with our recommendation to document lessons learned related to the loss of crew requirement and expects to take action to close this recommendation by the end of May 2019  NASA concurred with our recommendation to restructure the safety technical authority so that it is no longer dual hatted with programmatic and independent technical authority responsibilities  NASA expects to take action to close this recommendation by the end of August 2018  NASA did not concur with our recommendation that the Commercial Crew Program should include the results of its schedule risk analysis in its quarterly reports to Congress  NASA stated that it uses the contractors’ schedules as a baseline to provide qualitative statements in the NASA summary that accompanies each contractor’s quarterly reports to Congress  NASA believes that this approach is appropriate and is in accordance with the explanatory statement accompanying the Consolidated and Further Continuing Appropriations Act, 2015  NASA also stated that it will be working to ensure that the contractors’ schedules and the program’s internal assessments sync up as the program gets closer to launch  As a result, NASA explained that there will not be a requirement for a detailed NASA assessment, because the contractors’ schedule will either match NASA’s analysis or NASA will discuss its position as it has done in previous reports to Congress  We continue to believe the recommendation is valid because the program’s schedule risk analysis would provide Congress with valuable insight into potential delays, which are likely  Both contractors have repeatedly stated that their schedules are aggressive and that the dates are ambitious  As a result, we found that the contractors frequently delay dates for key events  For example, Boeing has delayed its certification milestone by 17 months and SpaceX by 22 months since the original schedules were established  The program’s recent schedule risk analysis indicates that more delays to certification are likely, but that information is not presented to Congress in NASA’s quarterly reports  Without this information, Congress does not know the full extent of potential delays to inform decision making  NASA partially concurred with our recommendation that the Chief of Safety and Mission Assurance, the NASA Associate Administrator for Human Exploration and Operations, the Commercial Crew Program Manager, and the Commercial Crew Program Contracting Officer should collectively determine and document how the agency will determine its risk tolerance level with respect to loss of crew before the agency certification review  In its response, NASA stated that it documented the agency’s risk tolerance level with respect to loss of crew for the program in its May 2011 safety memo  Further, NASA stated that it documented the requirement to limit risks to the loss of crew in a certification requirements document  NASA stated that ultimately the Commercial Crew Program is accountable for ensuring that the contractors’ systems meet the loss of crew value in this certification requirements document, which is a loss of crew value of 1 in 270  If a contractor’s system cannot meet that loss of crew value, or any other requirement, the program will request a waiver as part of the human rating certification process to ensure transparency  NASA acknowledged in its response that the existence of multiple documents defining residual risk requirements and an agency threshold for loss of crew can be confusing  NASA’s response, however, does not address our finding that it does not have a consistent approach for how to incorporate key inputs, including which debris model should be used or whether to include operational mitigations  NASA stated that it had taken action to address this recommendation; however, NASA did not outline any steps it took to resolve the concern that the risk tolerance for the loss of crew requirement depends on which entity is presenting the results of its analysis  We continue to believe that, before the agency certification review, the key parties must collectively determine how the agency will determine its risk tolerance with respect to loss of crew  We believe this approach will reduce confusion and increase transparency  We are sending copies of this report to NASA Administrator and interested congressional committees  In addition, the report is available at no charge on the GAO website at http://www gao gov  If you or your staff have any questions about this report, please contact me at (202)512-4841 or chaplainc@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report  GAO staff who made major contributions to this report are listed in appendix III  The objectives of our review were to assess (1) the extent to which the contractors have made progress towards meeting the National Aeronautics and Space Administration’s (NASA) certification requirements and NASA’s plans to help ensure continued access to the International Space Station (ISS); and (2) how NASA’s certification process addresses safety of the contractors’ crew transportation systems  To assess the contractors’ progress towards certification, we obtained and reviewed program and contractor documents, including monthly and quarterly updates from April 2017 through May 2018  We interviewed program and contractor officials to discuss the contractors’ recent progress, including upcoming events and any expected delays, and to understand technical risks, potential consequences, and planned mitigation activities  To identify total delays to date, we compared original contract schedules to Boeing and SpaceX’s calendar year 2018 first quarter proposed schedules, which are the most recent  Based on our review of program and contractor documents, we defined the contractors’ key events as: the uncrewed and crewed flight tests, the design certification reviews for each of those flights, and the certification milestone  We selected the two flight tests for each contractor as key events because they are intended to test key system capabilities, including the ability to launch, dock with the ISS, and return safely to Earth  We selected the design certification reviews because they verify the contractors’ crew transportation systems’ capability to safely approach, dock, mate, and depart from the ISS, among other requirements  We selected the certification milestone because it determines whether the crew transportation system meets the Commercial Crew Program’s requirements  To determine the extent to which contractors have delayed these key events over time, we analyzed the contractors’ schedule data from the 13 quarterly progress reports to date, from first quarter 2015 through first quarter calendar year 2018  We also obtained the results of the program’s April 2018 schedule risk analysis  We presented the schedule analysis range from the end of the month of the earliest possible completion date to the end of the month of the latest possible completion date  We reviewed the program’s Congressional requirements to report on cost, schedule, and technical status  Finally, to assess the potential effects of any certification delays on NASA’s access to ISS, we reviewed NASA’s contracts with Boeing and the Russian Federal Space Agency for transportation on the Soyuz vehicle  We interviewed officials from the ISS program and NASA’s Human Exploration and Operations Mission Directorate to determine if the agency had developed contingency plans to mitigate the effects of any certification delays on its access to the ISS  To assess how NASA’s certification process addresses safety of the contractors’ crew transportation systems, we reviewed agency safety policies, program plans, and contract documents to identify what safety assessments were required of the program, which safety factors would be considered in certification reviews, and when certification approval would be granted  We also interviewed program officials and the contractors about their safety policies and procedures as well as about the certification process  We identified the loss of crew requirements for the program and the contractors, and interviewed program and agency officials to determine how loss of crew would be assessed and considered throughout the certification process  To gain a broader understanding of the relative importance of loss of crew, we reviewed NASA safety policies, prior GAO reports, and annual reports from the Aerospace Safety Advisory Panel  To determine how the Orbital Debris Engineering Model (ORDEM) was updated and to assess differences between the former and updated models, we reviewed NASA documentation about the ORDEM 2000 and ORDEM 3 0 models and obtained information about the models from NASA’s Orbital Debris Program Office  To assess the program’s progress in closing requirements, including those requirements specifically related to safety, we reviewed program data for each contractor on contract requirements and closure status  We limited this analysis to requirements included in the CCT-REQ-1130 ISS Crew Transportation and Services Requirements Document because these requirements are verified by the Commercial Crew Program, whereas requirements contained within SSP 50808 ISS to Commercial Orbital Transportation Services Interface Requirements Document are managed by the ISS Transportation and Integration Office  We classified requirements as “safety-specific” when the program’s Office of Safety and Mission Assurance was listed as a verification closure notice signatory (i e , approver)  We then analyzed the program’s data to determine how many verification closure notices had been approved for all requirements and for the subset of safety-specific requirements  We reviewed program and agency documentation, such as organizational charts, program plans, and safety policies as well as interviewed program and agency officials, to determine the role of the safety and mission assurance technical authority in the program  We also reviewed the 2003 Columbia Accident Investigation Board’s Report’s findings and recommendations related to culture and organizational management of human spaceflight programs  We reviewed annual briefings and reports and met with representatives from two organizations that provide NASA with independent assessments of the program, the program’s standing review board and the Aerospace Safety Advisory Panel, to gain their perspectives on the contractor’s progress and how NASA addresses safety in its certification process  We also met with representatives from the National Transportation Safety Board and three experts with background on safety in human spaceflight in order to increase our contextual understanding of the role of safety in human spaceflight missions  We conducted this performance audit from April 2017 to July 2018 in accordance with generally accepted government auditing standards  Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives  We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives  Cristina T  Chaplain, (202) 512-4841 or chaplainc@gao gov  In addition to the contact named above, Molly Traci, Assistant Director; Kazue Chinen; Lorraine Ettaro; Lisa Fisher; Laura Greifner; Kurt Gurka; Miranda Riemer; Juli Steinhouse; Roxanna T  Sun; Hai Tran; Kristin Van Wychen; and Alyssa Weir made significant contributions to this report 


Report 2: Risk management, as applied to security of federal facilities, entails a continuous process of applying a series of mitigating actions—assessing risk through the evaluation of threats, vulnerabilities, and consequences; responding to risks with appropriate countermeasures; and monitoring risks using quality information (see fig  1)  In 1995, Executive Order 12977 established the ISC after the bombing of the Oklahoma City Alfred P  Murrah Federal Building in April 1995  The ISC’s mandate is to enhance the quality and effectiveness of security in and protection of federal facilities in the United States occupied by federal employees for nonmilitary activities  The order directs the ISC to develop and evaluate security standards for federal facilities, develop a strategy to ensure executive agencies and departments comply with such standards, and oversee the implementation of appropriate security measures in federal facilities  The ISC has released a body of standards, including the ISC Standard, designed to apply to the physical security efforts of all federal, non-military agencies  The ISC Standard prescribes a process for agencies to follow in developing their risk assessment methodologies (see fig  2)  Most federal departments and agencies are generally responsible for protecting their own facilities and have physical security programs in place to do so  The ISC Standard requires executive departments and agencies to follow the risk-management process when conducting risk assessments for each of their facilities  That process begins with determining the facility security level, ranging from level I (lowest risk) for facilities generally having 100 or fewer employees to level V (highest risk) for the most critical facilities and generally having greater than 750 employees  The security level designation determines the facility’s baseline countermeasures  For each facility, departments and agencies are required to (a) consider all of the “undesirable events” that could pose a risk to their facilities— such as active shooters, vandalism, and explosions—and (b) assess three factors of risk (threats, vulnerabilities, and consequences) to specific undesirable events  Subsequently, agencies are to combine all three factors to yield a measurable level of risk for each undesirable event (see app  III)  Based on the results of these assessments, agencies should customize (either increase or decrease) the countermeasures to adequately reflect the assessed level of risk  In addition, as part of planning for physical security resources within an agency’s budget process, the ISC has identified the need to balance allocations for countermeasures with other operational needs and with competing priorities  The ISC Best Practices have some similarities with leading practices in capital decision-making  For example, both state that the allocation of resources should be integrated into the agency’s mission, objectives, goals, and budget process  However, beyond the ISC Best Practices, the Office of Management and Budget and we have developed more comprehensive leading practices in capital decision- making that provide agencies with guidance for prioritizing budget decisions such as for countermeasure projects  The Office of Management and Budget and our guidance also emphasize evaluating a full range of alternatives, informed by agency asset inventories that contain condition information, to bridge any identified performance gap  Furthermore, the guidance calls for a comprehensive decision-making framework to review, rank, and select from among competing project proposals  Such a framework should include the appropriate levels of management review, and selections should be based on the use of established criteria  The following describes the mission and physical security program characteristics for the agencies in our review: CBP, the nation’s largest law enforcement agency, has responsibility for securing the country’s borders  It also has responsibility for conducting security assessments at about 1,200 facilities, including approximately 215 federally owned and agency-controlled higher-level facilities (facility security levels III and IV)  These facilities include border patrol stations with holding cells for people detained at the border, office buildings, and canine-training centers  CBP conducts these assessments  FAA’s mission is to provide a safe and efficient aerospace system for the country  According to agency data, FAA has 55 federally owned and agency-controlled higher-level facilities—including critical air traffic control towers  According to FAA officials, FAA specialists conduct security assessments  ARS conducts research related to agriculture and disseminates information to ensure high-quality safe food and to sustain a competitive agricultural economy  According to agency data, ARS has security responsibility for four domestic federally owned and agency- controlled higher-level facilities—including laboratories for research to improve food and crop quality, office buildings, and warehouses  ARS security personnel have responsibility for conducting security assessments  The Forest Service sustains the health, diversity, and productivity of the nation’s forests and grasslands  According to agency officials, the Forest Service has one federally owned and agency-controlled higher- level facility—a regional headquarters office building  The Forest Service’s security officials have responsibility for conducting security assessments, but at the time of our review, USDA security officials conducted the assessment at Forest Service’s one higher-level facility  None of the four selected agencies’ security assessment methodologies fully aligned with the ISC Standard  The ISC gives agencies some flexibility to design their own security-assessment methodologies for identifying necessary countermeasures as long as the chosen methodology adheres to fundamental principles of a sound risk- management methodology  Specifically, methodologies must: consider all of the undesirable events identified in the ISC Standard as possible risks to federal facilities, and assess three factors of risk (threats, vulnerabilities, and consequences) for each of the events  Furthermore, the ISC Standard requires executive departments and agencies to document decisions that deviate from the ISC Standard  Agencies’ policies and methodologies reference the ISC Standard  However, none of the agencies’ methodologies considered all of the undesirable events during assessments although they used some type of risk assessment methodology  In addition, the agencies did not always adhere to these principles of risk management (see table 1)  At the time of our review, CBP’s methodology did not fully align with the ISC Standard because it did not consider all of the 33 undesirable events nor assess threat and consequence  CBP security specialists assessed vulnerabilities at building entrances and exits, in interior rooms, and around the perimeter using a yes/no checklist during the assessment process  However, assessment reports showed that specialists did not assess the threats and consequences of undesirable events at each facility  According to security officials, the gap occurred because they designed the checklist to meet requirements in the 2009 CBP Security Policy and Procedures Handbook, which predates the first edition of the ISC Standard issued in 2010  CBP officials told us that as of January 2017, they began using an improved methodology to assess the threats, vulnerabilities, and consequences for 30 of 33 undesirable events— omitting three now identified in the November 2016 revision to the ISC Standard  However, CBP has not yet updated its handbook to align with the ISC Standard, even though it started this effort over 3 years ago in December 2013  CBP officials did not provide a draft of its updated handbook, but they provided a plan with milestone dates for issuing the handbook by September 2018  CBP officials also told us that updates to the handbook may have to wait due to competing priorities, including efforts to address the backlog of assessments (which we discuss later in this report)  Delays in updating the handbook mean that CBP’s policy will continue to not align with the ISC Standard  Furthermore, although CBP security officials told us that all of the agency’s security specialists have been trained to use the improved assessment methodology, without documentation of the methodology in agency policy, there may be greater risk of its inconsistent application  Standards for Internal Control emphasize the importance of agencies developing and documenting policies to ensure agency-wide objectives are met  Documentation serves to retain institutional knowledge over time when questions about previous decisions arise  Without an updated policy handbook that requires a methodology that assesses all undesirable events consistent with the ISC Standard, CBP cannot reasonably ensure that its facilities will have levels of protection commensurate to their risk  FAA’s methodology does not fully align with the ISC Standard because it does not consider all of the 33 undesirable events nor does it assess all three factors of risk  FAA security specialists assess vulnerabilities to the site perimeter, entryways, and interior rooms using a yes/no checklist, but the checklist does not assess the consequences from each of the undesirable events at each facility  With respect to threat, FAA applies the ISC’s baseline threat—a general federal facilities threat level that relates directly to a set of baseline countermeasures—across all its higher-level facilities because FAA policy states that there is no agency-specific threat that exceeds the current baseline threat  According to FAA officials, the baseline threat standardizes the security needs across their facilities rather than addressing the security needs of individual facilities from specific threats  When necessary, FAA policy allows specialists to modify countermeasures based on an evaluation of conditions at the facility  FAA realized that this approach was no longer appropriate given the agency-wide goal to make risk-based decisions, a review of the assessment process after a 2014 Chicago fire incident that destroyed critical FAA equipment, and an awareness of ISC initiatives to assess compliance  To address the resulting methodological gaps, FAA hired a contractor to design, develop, test, and validate an improved risk- assessment methodology  Subsequently, FAA improved its methodology in January 2017 to assess the threats, vulnerabilities, and consequences for 30 of the 33 undesirable events identified in the November 2016 revision to the ISC Standard —and tested the methodology at lower- and higher-level facilities  This revised methodology addresses the need to assess individual facility needs rather than using a standardized baseline approach  In April 2017, FAA officials told us of their plan for implementing this methodology and provided tentative milestone dates to conduct further testing, training, and analysis before deciding to use the improved methodology, which they expect to complete by January 2018  However, their plan lacks the necessary information to ensure successful implementation, such as detail on how many facilities they will test and how they will use the results of testing, training, and analysis to implement the improved methodology within the identified 9-month time frame  Furthermore, the improved methodology does not address undesirable events for which ISC issued countermeasures in May 2017  Without a detailed implementation plan to assess the methodology’s impact on its security program, FAA cannot reasonably ensure that its facilities have the proper countermeasures  With ongoing changes to its security program, FAA has an opportunity to fully align its improved methodology with the ISC Standard by including all 33 undesirable events and to update its policy requiring the use of such a methodology  Unlike CBP and FAA—which developed their own methodologies separate from their parent departments (Department of Homeland Security (DHS) and Department of Transportation (DOT), respectively)— ARS and the Forest Service follow an assessment methodology developed by USDA  USDA’s methodology does not fully align with the ISC Standard because it does not consider all of the 33 undesirable events for which ISC issued countermeasures in May 2017  Security specialists from USDA headquarters typically assess ARS’s and the Forest Service’s higher-level facilities using a risk-based methodology that considers the 31 undesirable events listed in the previous version of the ISC Standard dated August 2013  However, until recently, USDA did not assign ratings to each of the three risk factors—threat, vulnerability, and consequence—and then combine these ratings to yield a measurable level of risk for each undesirable event  USDA security officials said that they have revised the assessment-reporting format to include this risk calculation and trained their specialists to measure risk in this way  USDA officials provided us with a new assessment template that addresses all 33 undesirable events and includes measuring risk  Additionally, USDA officials said that they are revising their outdated physical security manual and expect to complete it by April 2018  With a revised manual and application of the new assessment template, USDA should be better positioned to assess risk at its facilities  When agencies do not use methodologies that fully align with the ISC Standard, they could face deleterious effects, ranging from facilities having inappropriate levels of protection to agencies having an inability to make informed resource allocation decisions for their physical security needs  Specifically, the ISC Standard states that facilities may face the effect of either having (1) less protection than needed resulting in inadequate security or (2) more protection than needed resulting in an unnecessary use of resources  The ISC Standard also states that these effects can be negated by determining the proper protection according to a risk assessment  Identified excess resources in one risk area then can be reallocated to underserved areas, thus ensuring the most cost- effective security program is implemented  As an illustration of such potential effects, we found that two agencies assessing two higher-level facilities came to two different conclusions in terms of their need for X-ray machines to screen for guns, knives, and other prohibitive items in federal facilities  Specifically, one agency based its decision on a policy that does not deviate from the ISC’s baseline set of countermeasures, and the other agency based its decision on professional judgement that deviated from the ISC’s baseline set of countermeasures  Neither agency based its decision on a risk assessment nor documented its decision—both ISC requirements, specifically: Without conducting a risk assessment, FAA recently expanded a policy requirement calling for all higher-level facilities to have X-ray machines and magnetometers  This new requirement poses a potentially sizeable investment for the agency with an estimated cost of X-ray machines of about $24,000 and magnetometers of about $4,000 each  FAA may need such equipment at all its higher-level facilities  However, the ISC Standard requires that agencies conduct risk assessments first to justify their needs  Without conducting risk assessments, FAA managers could unnecessarily use resources by installing such equipment in all higher-level air traffic facilities when there may be higher priority needs A USDA security specialist decided, despite an ISC baseline requirement that higher–level facilities have X-ray machines, not to recommend an X-ray machine at a higher-level Forest Service facility  The specialist reasoned that unlike other federal buildings with numerous unknown visitors, this facility receives mostly known individuals and a limited number of visitors  The ISC Standard allows for professional judgement; however, the ISC requires that agencies document deviations from the baseline set of countermeasures  Reducing the facility’s level of protection without documenting an assessment of risk could result in no record of the basis of the decision for current and future facility managers and security officials to review or use as justification in the case of a question of compliance  In another case, we found that one higher-level facility did not have access control for employees or visitors nor did it have armed guard patrols  The facility manager told us that intelligence and a history without incidents gave leadership reason to believe that these measures were not needed and that therefore the agency did not require and would not fund such protective measures for this facility—in effect, accepting the risks to the facility  Security officials said they also had the same understanding and did not document the matter in the assessment report even though agency policy and the ISC Standard require written documentation when officials deviate from the baseline requirement  Without security assessments that fully align with the ISC Standard and provide measureable levels of risk, agencies do not have the information they need to determine priorities and make informed resource allocation decisions  For example, they may not be able to assess whether to acquire or forego costly physical-security countermeasures—such as, X- ray machines, access control systems, and closed-circuit television systems—for facilities  Additionally, after determining the need to acquire a countermeasure, agencies must fund the countermeasure  As previously discussed, leading practices in capital decision-making include a comprehensive framework to review, rank, and select from competing project proposals for funding  In conducting risk assessments that do not fully align with the ISC Standard (i e , not assessing threats, vulnerabilities, and consequences and measuring risks), agencies miss the opportunity for more informed funding decisions  Three of the four agencies (CBP, ARS, and the Forest Service) currently prioritize funding for operational needs over physical security needs (see table 2) when agencies’ priorities might be different if they based their decisions on an aligned risk assessment  Standards for Internal Control state that agencies should use quality information on an ongoing basis as a means to monitor program activities and take corrective action, as necessary  The ISC requires that agencies assess higher-level facilities at least once every 3 years—an interval requirement to identify and address evolving risks  We found that three of the four agencies (CBP, ARS, and the Forest Service) did not meet this requirement  Officials reported various challenges including (1) assessments competing with other security activities, (2) an insufficient number of qualified staff to conduct assessments when compared to the number of facilities, or (3) not knowing of the required assessment schedule  An “information system” is the people, processes, data, and technology that management organizes to obtain, communicate, or dispose of information  that had not been reassessed since 2010  CBP security officials attributed the backlog to (1) having too few security specialists assigned to assess about 1,200 facilities and (2) the specialists working on competing priorities, such as revising the security handbook, conducting technical inspections, and reviewing new construction designs and renovation projects  According to CBP security officials, they have developed a plan to eliminate the backlog by the end of fiscal year 2018 by prioritizing the completion of assessments  While we found the plan comprehensive, the schedule did not seem feasible  For example, the plan assumes that one specialist can complete six assessments in 3 consecutive days and that another specialist can complete three assessments in 1 day  In contrast, security officials told us specialists take about 20 work hours (or 2½ days) to conduct an on-site assessment of one facility  CBP officials said that they believe they can meet the time frames of the plan because they have set aside other priorities and have a thorough understanding of the scope of work involved at the facilities  They added that it will not be easy to meet the timeline, but they can accomplish it with a motivated and committed workforce, adequate financial resources, and absent activities that would otherwise require shifting of resources  We question the feasibility of setting aside important priorities, such as updating the policy manual and reviewing physical security elements in new construction designs, as well as the workload assumptions for completing the assessments  Further, these other priorities are also key to securing facilities  Without balancing assessments with competing priorities, CBP’s time frames for completing the assessments by the end of fiscal year 2018 may not be feasible and may also result in the agency’s not addressing other important physical security responsibilities  Since the ISC issued its standard in 2010, ARS and the Forest Service have assessed their higher-level facilities at least once  However, these agencies have not reassessed all of their higher-level facilities within the 3-year interval requirement  Specifically, security specialists have not conducted required reassessments of two ARS and one Forest Service higher-level facilities  The ARS headquarters official explained that the agency had not reassessed the two facilities due to competing priorities and insufficient internal resources  During the course of our review, ARS headquarters officials said they began assessing one of the two ARS facilities in May 2017 and will begin assessing the second facility in October 2017  The Forest Service official explained that the agency missed its security reassessment of the regional office because the facility staff had not requested one  During our visit, facility staff responsible for security told us that they were not aware of the ISC’s 3- year interval requirement  Facility staff requested a reassessment, and security officials told us that they expected to complete it by mid-June 2017  Completing this one-time assessment may address the facility’s security needs temporarily  However, ARS and the Forest Service have not implemented a long-term schedule with key milestones and lack a means to monitor completion of assessments of higher-level facilities at least once every 3 years  Consequently, these agencies cannot reasonably ensure that they have full knowledge of the risks to their facilities  FAA data from 2010 through 2016 show that FAA has assessed its 55 higher-level facilities at least once every 3 years  FAA policy requires that specialists schedule assessments of higher-level facilities every 12– 18 months depending on whether the facility has met FAA physical security standards  The ISC Standard states that to make appropriate resource decisions, agencies need information, such as what is being accomplished, what needs management attention, and what is performing at expected levels  We found that agencies’ methods of collecting and storing security information had limitations that affected agency and facility officials’ oversight of the physical security of their facilities (see table 3)  Without having long-term, agency-wide information to monitor whether assessments are conducted on schedule, ARS and the Forest Service may not meet the ISC Standard, resulting in not adequately protecting their facilities and employees  The ISC Standard also states that agencies should measure their security program’s capabilities and effectiveness to demonstrate the need to fund facility security and to make appropriate decisions for allocating resources  However, the agencies in our review were unable to demonstrate appropriate oversight of their physical security programs because: CBP’s handbook does not include requirements for data collection and analysis for monitoring physical-security program activities  Facility managers and security officials do not enter assessment results, such as the countermeasures recommended for facilities, in the real property database  Consequently, they do not have comprehensive data to manage their security program, assess overall performance, and take any necessary corrective actions  A CBP official told us that a comprehensive database would allow CBP to set priorities for addressing countermeasures  Without including data collection and analysis requirements in its updated handbook, CBP may be unable to monitor the performance of its physical security program  FAA’s policy does not require ongoing monitoring of physical security information, such as the status of recommended countermeasures or assessment schedules  As a result, FAA officials do not proactively use physical security information to assess the overall performance of its physical security program and take corrective actions before an incident occurs  Without a policy requiring ongoing monitoring of information—an internal control activity, FAA may be unable to assess the overall performance of its security program and take necessary corrective actions  USDA has a decentralized security program and places the responsibility on agencies to create their physical security programs  Security officials from ARS and the Forest Service told us that USDA does not have a policy for collecting and managing agency-wide information; however, they said that USDA is drafting a new departmental regulation and manual that will specify (1) the roles and responsibilities of agency and facility managers and (2) electronic- data-reporting requirements for monitoring the performance of the physical security program  USDA officials provided a draft of USDA’s regulation and manual for our review  The draft regulation did not mention data reporting and monitoring, while the draft manual only contained a table of contents that included a section entitled “Facility Tracking Database ” USDA officials expect to issue new policies sometime between October 2017 and April 2018  In the absence of new departmental regulation and manual, USDA and Forest Service officials told us that they have begun to develop a Forest Service system for storing electronic copies of agency-wide assessments and that they plan to expand the use of this system to track site specific assessment dates and status of recommended countermeasures  Forest Service officials provided milestone dates and described the capabilities for a future information system, which they expect to complete in September 2017  However, we could not determine whether the manual will have information system requirements to monitor agencies’ physical security program, an internal control activity  Without USDA’s including data collection and analysis requirements in its manual, its agencies may not be able to monitor the performance of their physical security programs  Without agencies having information to monitor security activities, they were unable to provide us information on the status of countermeasures across their entire portfolio  In order to better understand the status of countermeasures implemented and facilities’ experiences when implementing countermeasures, we determined the status of countermeasures at 13 facilities we visited  As previously noted, risk management, as it pertains to physical security, involves agency officials monitoring their physical security programs  During our visits to 13 selected facilities, we found the four agencies differed in the number of countermeasures that they had not implemented  Facility officials provided us with some information on why countermeasures had not been implemented, specifically: CBP had a significant number of recommended countermeasures from 2010 through 2016 that remained open at the eight selected CBP facilities  CBP facility officials gave reasons why recommended countermeasures had not been implemented  At one facility, officials did not know about the recommended countermeasures from its last 2010 assessment because the individuals previously knowledgeable about the assessments left the organization without communicating the results  By taking action to improve facility security, they implemented some needed countermeasures  However, at the time of our review, a large number of the recommendations remained open  At another facility, officials told us that they too had not known (for the same reason mentioned above) of their 2010 assessment, which contained recommended countermeasures  However, these officials told us that they submitted a funding request a few weeks before our visit to address all except one of the open countermeasures  In other cases, facilities have not implemented needed countermeasures due to resource constraints or physical site limitations  FAA had a large number of recommended countermeasures from 2010 through 2016 that remained open at the time of our review for the two FAA facilities visited  In this case, the most recent security assessment, completed in late 2016, resulted in one facility’s having little time to implement countermeasures by the time we conducted our analysis  While ARS had closed almost all recommended countermeasures at two facilities at the time of our review, one Forest Service facility had not yet implemented a recommendation (to secure its entrance doors) that was identified in a 2013 security assessment (see bottom center photo, fig  3)  This countermeasure remained open because facility officials said they continued to explore alternatives to address the recommendation  Figure 3 shows examples of countermeasures not fully implemented at selected facilities we visited  During our site visits and discussions with facility staff, we found that physical site limitations or other priorities can make it difficult for facility managers to implement countermeasures  For example, a countermeasure might involve correcting a clear zone violation—that is, moving an object (such as a brick wall) a certain distance away from the facility’s perimeter fence to prevent a potential intruder from using the object to climb over the fence  However, when the object near the fence is a building and the property outside of the fence is not federally owned (see bottom right photo, fig  3), it may not be cost effective to correct the clear zone violation  In this situation, the agency bears the responsibility for exploring ways to address the vulnerability  In following the ISC Standard, as previously noted, managers are required to justify and document why they could not implement recommended countermeasures—what the ISC calls risk acceptance  Selected agencies carry a great responsibility for protecting facilities that support border protection activities, provide safe and efficient air traffic around the country, and protect the quality of the nation’s food supply  With this responsibility comes the need to appropriately assess risk to ensure the security of these agencies’ facilities  However, 7 years after the ISC issued its initial risk-management process standard, each of four selected agencies continued to use assessment methodologies that did not fully align with this standard  During our review, agencies improved their methodologies to better align with the ISC Standard, but the agencies had not yet incorporated the methodologies into their policies and procedures  Without updated policies and procedures requiring a methodology that adheres to the ISC Standard (including all 33 undesirable events now identified in the November 2016 revision to the ISC Standard), agencies may not collect the information needed to assess risk and determine priorities for improved security  This situation could hamper the agencies’ ability to make informed resource allocation decisions or to recommend countermeasures commensurate to the needs at specific facilities  To address challenges in conducting timely assessments, agencies that had a backlog developed plans to address them, but the assumptions used in CBP’s plans and time frames did not appear to fully reflect the agency’s competing priorities and actual experience  Additionally, ARS and Forest Service have not implemented a long-term assessment schedule with key milestones to ensure that higher-level facilities are reassessed at least once every 3 years  Further, in cases where the agencies may have had risk assessment information, CBP, ARS, and the Forest Service lack the means to collect, store, and analyze this information in order to monitor the status of a facility’s security  Without these key aspects of a comprehensive security program—a methodology that meets the standard, policies, and procedures that incorporate that methodology; the ability to complete assessments on time; and information to perform monitoring—agencies remain vulnerable to substantial security risks  To improve agencies’ physical security programs’ alignment with the ISC Risk Management Process for Federal Facilities and Standards for Internal Control in the Federal Government for information and monitoring, we recommend that the Commissioner of U S  Customs and Border Protection take the following three actions: with regard to the updated Security Policy and Procedures Handbook, the ISC’s Risk Management Process for Federal Facilities requirement to assess all undesirable events, consider all three factors of risk, and document deviations from the standard, and data collection and analysis requirements for monitoring the performance of CBP’s physical security program  revise the assumptions used in the plan to address the backlog to balance assessments with competing priorities, such as updating the policy manual and reviewing new construction design, to develop a feasible time frame for completing the assessment backlog  Secretary of Transportation direct the FAA Administrator to take the following three actions: develop a plan that provides sufficient details on the activities needed and time frames within the date when FAA will implement an improved methodology; update FAA’s policy to require the use of a methodology that fully aligns with the ISC’s Risk Management Process for Federal Facilities for assessing all undesirable events, considering all three factors of risk, and documenting all deviations from the standard countermeasures; and update FAA’s policy to include ongoing monitoring of physical security information  Secretary of Agriculture take the following two actions: include data collection and analysis requirements for monitoring the performance of agencies’ physical security programs, in the department’s revised physical-security manual, and direct the Administrator of the Agricultural Research Service and the Chief of the Forest Service to implement and monitor a long-term assessment schedule with key milestones to ensure that higher-level facilities are reassessed at least once every 3 years  We provided a draft of this report to the Departments of Homeland Security, Transportation, and Agriculture for review and comment  All three departments agreed with the findings and recommendations for their respective agencies  DHS agreed with our recommendations and provided actions and timeframes for completion  With regard to our recommendation to update the Security Policy and Procedures Handbook, DHS stated that CBP is updating the handbook to include: (1) a discussion and diagram of the ISC risk management process and its application within CBP’s assessment processes; (2) specific guidance for conducting risk assessments in accordance with the ISC’s Risk Management Process for Federal Facilities; and (3) a requirement and guidance for data collection and analysis in support of a robust physical security program  With regard to our recommendation to revise the assumptions used in the plan to address the assessment backlog, DHS stated that CBP has reevaluated current priorities and believes the current plan to eliminate the risk assessment backlog by the end of fiscal year 2018 is achievable  DHS also provided technical comments, which we incorporated as appropriate  DHS’s official written response is reprinted in appendix IV  DOT also agreed with our recommendations and by e-mail requested that we publish the response to the sensitive version of this report  DOT stated that FAA continues to refine its policy and develop processes that address the ISC threats, vulnerabilities, and consequences  Further, DOT stated that FAA would either validate that current mitigation strategies address those risks or apply additional appropriate countermeasures  DOT stated that it will provide a detailed response to each recommendation within 60 days from the date of this report  DOT’s official written response is reprinted in appendix V  USDA agreed with our recommendations and provided the agency-wide actions for completion  USDA provided a plan to ensure compliance with the ISC’s Risk Management Process for Federal Facilities by development of a standard physical-security assessment process and by initiation of a compliance program to track assessments and monitor the installation of countermeasures  In an e-mail, USDA provided milestone dates and planned completion by January 2019  USDA’s official written response is reprinted in appendix VI  If you or your staff has any questions about this report, please contact me at (202) 512-2834 or RectanusL@gao gov  GAO staff who made key contributions to this report are listed in appendix VI  This report examines: (1) how selected agencies’ assessment methodologies align with the Interagency Security Committee’s (ISC) risk management standard for identifying necessary countermeasures and (2) what management challenges, if any, selected agencies reported facing in conducting physical security assessments and monitoring the results  To determine how selected agencies’ assessment methodologies align with ISC standards for identifying the necessary countermeasures, we identified federal executive branch departments and agencies reported by the Department of Homeland Security (DHS) to have received delegations of authority to protect their own buildings  We reviewed the Federal Real Property Council’s data on the Federal Real Property Profile to identify federally owned and agency-controlled buildings  We determined that these data were sufficiently reliable for the purpose of our reporting objectives based upon our recent report that reviewed these data fields  We selected four agencies based upon their large quantity of reported federally owned and agency-controlled buildings: DHS, U S  Customs and Border Protection (CBP); Department of Transportation (DOT), Federal Aviation Administration (FAA); United States Department of Agriculture (USDA), Agricultural Research Service (ARS) and USDA’s United States Forest Service (Forest Service)  This methodology purposely does not include federal buildings protected by FPS and under the control of the General Services Administration as well as other agencies that we reported on in our previous work  We obtained and reviewed one particular ISC standard, The Risk Management Process for Federal Facilities (the ISC Standard) and its related appendices for assessing physical security and providing recommended countermeasures at federal facilities  We obtained and analyzed the selected departments’ and agencies’ facility-security policies and procedures for a risk assessment methodology  According to the ISC Standard, agencies’ risk assessment methodologies must: consider all of the undesirable events identified in the ISC Standard as possible risks to federal facilities as listed in appendix III; assess the threat, consequences, and vulnerability to specific produce similar or identical results when applied by various security provide sufficient justification for deviations from the ISC-defined security baseline  We limited the scope of this review to the first two standards above because agencies’ adherence to these standards could be objectively verified by reviewing and analyzing agency documentation and interviewing agency officials, and their adherence to the two additional standards could not be verified in this manner  We did not conduct risk assessments with independent security professionals to evaluate: 1) the results from prior agency evaluations and 2) the sufficiency of justifications for deviations from the ISC-defined security baseline, as both evaluations were outside of the scope of the engagement  Therefore, for the purposes of this report, risk assessment policies, procedures and resulting methodology that align with ISC standards are those that consider all of the undesirable events and assess the threats, consequences, and vulnerabilities to specific undesirable events  We reviewed and analyzed information to answer the following five questions: 1  Do the policies and procedures mention the ISC standards? 2  Do the policies and procedures consider all of the undesirable events? 3  Do the policies and procedures assess the threat of specific undesirable events? 4  Do the policies and procedures assess the consequences of specific undesirable events? 5  Do the policies and procedures assess the vulnerability to specific undesirable events? We answered each of these questions as either a “Yes” or “No” for our selected agencies  The “No” answer to questions 3, 4, and 5 includes the following two possibilities: (a) the agency’s threat, consequence, or vulnerability ratings are not tied to specific undesirable events, or (b) the agency does not have a framework or formalized steps within which it collects and analyzes threat-, consequence-, or vulnerability-related information  If the answer to each of the five questions was “Yes,” then the agency’s overall risk assessment methodology aligns with ISC risk assessment standards for the purposes of this report  If the answer to one or more of the five questions was “No”, then the agency’s methodology does not to align with ISC standards for the purposes of this report  We interviewed security officials at ISC; three departments (DHS, DOT, and USDA); and four agencies (CBP, FAA, ARS, and the Forest Service)  We obtained and analyzed agency guidance on prioritizing physical security needs and interviewed agencies’ facility maintenance and budget officials  We reviewed the ISC’s best practices for planning for physical security resources within an agency budget process  Additionally, we reviewed the Office of Management and Budget’s and our leading practices in capital decision-making that provide agencies with guidance for prioritizing budget decisions such as “countermeasure projects ” We also reviewed Standards for Internal Control in the Federal Government because internal controls play a significant role in helping agencies achieve their mission-related responsibilities  Our findings from our review of the selected agencies are not generalizable to all ISC member agencies, but provide insight into and illustrative examples about selected agencies’ facility risk-assessment methodologies  To determine what management challenges selected agencies reported facing in conducting physical security assessments and monitoring results, we interviewed agencies’ security, maintenance, and budget officials  We requested agency security officials to provide portfolio- wide data on facility security assessments for our review in order to select sites to visit and analyze data for dates of assessments and the status of findings  We assessed the reliability of this data through interviews with knowledgeable agency staff and a review for completeness and any unexpected values  We compiled information from physical security assessments when no portfolio-wide agency data were available  We determined that these data were sufficient for the purpose of our reporting objectives and selected geographically dispersed sites with buildings with higher reported security levels per the ISC Standard, as these higher security levels have greater requirements and therefore the potential for greater resource needs  See appendix II for the 13 sites we selected  For these selected sites, we interviewed agency staff concerning the assessment process, site-specific findings, recommendations, justification for deviations from ISC’s baseline standards, and management challenges faced in addressing physical security needs  We observed and photographed the status of the findings from the site physical security assessments  We did not independently determine what constitutes a management challenge or a physical security finding  Rather, we relied on these stakeholders to determine these physical security concerns as defined in their own standards and guidance  The information from our selected sites is illustrative and cannot be generalized to sites agency- wide  The performance audit upon which this report is based was conducted from June 2016 to August 2017 in accordance with generally accepted government auditing standards  Those standards require that we plan and perform the audit to obtain sufficient, appropriate, evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives  We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives  We subsequently worked with DHS, DOT and USDA from August 2017 to October 2017 to prepare this version of the original report for public release  This public version was also prepared in accordance with these standards  Error! No text of specified style in document  Error! No text of specified style in document  Appendix VII: GAO Contact and Staff Acknowledgments Error! No text of specified style in document  In addition to the contact named above, Amelia Shachoy (Assistant Director), Steve Martinez (Analyst-in-Charge), Jennifer Clayborne, George Depaoli, Geoffrey Hamilton, Joshua Ormond, Alison Snyder, Amelia Michelle Weathers, and Elizabeth Wood made key contributions to this report 

ground_truth: In 2014, NASA awarded two firm-fixed-price contracts to Boeing and SpaceX, worth a combined total of up to $6.8 billion, to develop crew transportation systems and conduct initial missions to the ISS. In February 2017, GAO found that both contractors had made progress, but their schedules were under mounting pressure. The contractors were originally required to provide NASA all the evidence it needed to certify that their systems met its requirements by 2017. A House report accompanying H.R. 5393 included a provision for GAO to review the progress of NASA's human exploration programs. This report examines the Commercial Crew Program, including (1) the extent to which the contractors have made progress towards certification and (2) how NASA's certification process addresses safety of the contractors' crew transportation systems. GAO analyzed contracts, schedules, and other documentation and spoke with officials from NASA, the Commercial Crew Program, Boeing, SpaceX, and two of NASA's independent review bodies that provide oversight. Both of the Commercial Crew Program's contractors, Boeing and Space Exploration Technologies Corporation (SpaceX), are making progress finalizing designs and building hardware for their crew transportation systems, but both contractors continue to delay their certification milestone (see figure). Certification is the process that the National Aeronautics and Space Administration (NASA) will use to ensure that each contractor's system meets its requirements for human spaceflight for the Commercial Crew Program. Further delays are likely as the Commercial Crew Program's schedule risk analysis shows that the certification milestone is likely to slip. The analysis identifies a range for each contractor, with an earliest and latest possible completion date, as well as an average. The average certification date was December 2019 for Boeing and January 2020 for SpaceX, according to the program's April 2018 analysis. Since the Space Shuttle was retired in 2011, the United States has been relying on Russia to carry astronauts to and from the International Space Station (ISS). Additional delays could result in a gap in U.S. access to the space station as NASA has contracted for seats on the Russian Soyuz spacecraft only through November 2019. NASA is considering potential options, but it does not have a contingency plan for ensuring uninterrupted U.S. access. NASA's certification process addresses the safety of the contractors' crew transportation systems through several mechanisms, but there are factors that complicate the process. One of these factors is the loss of crew metric that was put in place to capture the probability of death or permanent disability to an astronaut. NASA has not identified a consistent approach for how to assess loss of crew. As a result, officials across NASA have multiple ways of assessing the metric that may yield different results. Consequently, the risk tolerance level that NASA is accepting with loss of crew varies based upon which entity is presenting the results of its assessment. Federal internal controls state that management should define risk tolerances so they are clear and measurable. Without a consistent approach for assessing the metric, the agency as a whole may not clearly capture or document its risk tolerance with respect to loss of crew. GAO is making five recommendations, including that NASA develop a contingency plan for ensuring a U.S. presence on the ISS and clarify how it will determine its risk tolerance for loss of crew. NASA concurred with three recommendations; partially concurred on the recommendation related to loss of crew; and non-concurred with a recommendation to report its schedule analysis to Congress. GAO believes these recommendations remain valid, as discussed in the report.

prompt_front: You are given two reports by the government agency.



Question: Write a one-page summary of the report 2.

Report 1: Major disaster declarations can trigger a variety of federal response and recovery programs for government and nongovernmental entities, households, and individuals  FEMA’s Office of Response and Recovery manages the PA grant program, providing funds to states, territorial governments, local government agencies, Indian tribes, authorized tribal organizations, and certain private nonprofit organizations in response to presidentially declared disaster declarations to repair damaged public infrastructure such as roads, schools, and bridges  Figure 1 shows the total amount of PA funds obligated by county from January 2009 through February 2017 for federal disaster declarations  To implement the PA program, FEMA’s staff includes a mix of temporary, reservist, and permanent employees under two authorities, the Stafford Act and Title 5  Reservists make up the largest share of the PA workforce, which consisted of 1,852 employees––1,041 reservists, 634 full-time equivalents, and 177 temporary Cadre of On-Call Response/Recovery Employees––as of June 2017, according to PA officials  Figure 2 summarizes the key characteristics for each type of employee  After a disaster, FEMA sends PA program staff to the affected area to work with state and local officials to assess the damage prior to a disaster declaration  FEMA officials establish a temporary Joint Field Office (JFO) to house staff who will manage response and recovery functions after a declared disaster (including operations, emergency response and support teams, planning, administration, finance, and logistics)  Once the President has declared a disaster, PA staff work with grant applicants to help them document damages, identify eligible costs and work, and prepare requests for PA grant funds by developing project proposals  These proposals may include proposals for hazard mitigation if the hazard mitigation work is related to the repair of damaged facilities, referred to as permanent work projects  Immediate emergency measures, such as debris removal, are not eligible for hazard mitigation  Officials then review and obtain approval of the projects prior to FEMA obligating funds to state grantees  Figure 3 describes the process used to develop, review, and obligate PA projects  In addition to rebuilding and restoring infrastructure to its predisaster state, the PA program can be used to fund hazard mitigation measures that will reduce future risk to the infrastructure in conjunction with the repair of disaster-damaged facilities  There is no preset limit to the amount of PA funds a community may receive; however, PA hazard mitigation measures must be determined to be cost effective  Some examples of hazard mitigation measures that FEMA has predetermined to be cost effective, if they meet certain requirements, include installing shut-off valves on underground pipelines so that damaged sections can be isolated during or following a disaster; securing a roof using straps, clips, or other anchoring systems in locations subject to high winds; and installing shutters on windows or replacing glass with impact-resistant material  Applicants can also propose mitigation measures that are separate from the damaged portions of a facility, such as constructing floodwalls around damaged facilities to avoid future flooding  FEMA evaluates these proposals, considering how the proposed measure protects damaged portions of a facility and whether the measure is reasonable based on the extent of the damage, and determines eligibility on a case-by-case basis  FEMA’s Federal Insurance and Mitigation Administration (FIMA) deploys a cadre of mitigation staff to help coordinate and implement hazard mitigation activities during disaster recovery, including PA hazard mitigation  A primary task of these staff is to identify and assess opportunities to incorporate hazard mitigation into PA projects  Generally, if an applicant seeks to incorporate hazard mitigation measures into a PA project, FIMA’s hazard mitigation staff develop a hazard mitigation proposal  We, the DHS OIG, and others have reported past challenges with FEMA’s management of the PA program related to workforce management, information sharing, and hazard mitigation  For example, we reported in 2008 that the PA program had a shortage of experienced and knowledgeable staff, relied on temporary rotating staff, and provided limited training to their workforce, which impaired PA program delivery and delayed recovery efforts after Hurricanes Katrina and Rita  We found that staff turnover, coupled with information sharing challenges, delayed projects when applicants had to provide the same information each time FEMA assigned new staff and that poorly trained staff provided incomplete and inaccurate information during their initial meetings with applicants or made inaccurate eligibility determinations, which also caused processing delays  We recommended that FEMA strengthen continuity among staff involved in administering the PA program by developing protocols to improve information and document sharing among FEMA staff  In response, in 2013 FEMA instituted a PA Consistency Initiative, which included hiring new managers for FEMA regional offices, stakeholder training on PA program administration, and using a newly developed internal website to allow staff to post and share information to address continuity and knowledge sharing concerns during disaster operations  FEMA also developed the Public Assistance Program Delivery Transition Standard Operating Procedure to facilitate the transfer of responsibility for PA program activities during cases of staff turnover during recovery operations  Despite FEMA’s efforts to implement our recommendations, the DHS-OIG, in 2016, found continuing challenges after Hurricane Sandy with workforce levels, skills, and performance of reservists, who make up the majority of the PA workforce  Regarding information sharing, in 2008, we also identified difficulties sharing documents among federal, state, and local participants in the PA process and difficulties tracking the status of projects  We recommended that FEMA improve information sharing within the PA process by identifying and disseminating practices that facilitate more effective communication among federal, state, and local entities  In response, FEMA proceeded with the implementation of a grant tracking and management system, called EMMIE, which was used previously in 2007  However, in subsequent years we found weaknesses in how FEMA developed the system and the DHS-OIG found that information sharing problems similar to the ones identified in our 2008 report persisted  Regarding hazard mitigation, we reported in 2015 that state and local officials experienced challenges in using PA hazard mitigation during the Hurricane Sandy recovery efforts because PA officials did not consistently prioritize hazard mitigation, and in some cases discouraged mitigation projects during the PA grant application process, among other challenges  We recommended that FEMA assess the challenges state and local officials reported, including the extent to which they can be addressed, and implement corrective actions, as needed  In response to our recommendation, FEMA developed a corrective action plan that included actions and milestones for reviewing, updating, and implementing PA hazard mitigation policy  FEMA also identified the PA new delivery model as a solution for some of the challenges state and local officials reported  Previously, the OIG also reported that PA program officials did not consistently identify eligible PA hazard mitigation projects, and that PA officials did not prioritize the identification of PA hazard mitigation opportunities at the onset of recovery efforts after the 2005 Gulf Coast hurricanes  See appendix I for a summary of findings and the status of our past recommendations on challenges with workforce management, information sharing, and hazard mitigation related to the PA program since our last review in December 2008  FEMA’s own internal reviews and outreach efforts have also identified similar challenges  For example, at FEMA’s request the Homeland Security Studies and Analysis Institute assessed the effectiveness and efficiency of the PA program in 2011  The institute’s report outlined 3 key findings and 23 recommendations relating to the PA preaward process  For example, the report found that FEMA could enhance training programs to develop a skilled and experienced workforce; utilize technology and employ web-based tools to support centralized processing, transparency, and efficient decision making; and identify and address potential special considerations, such as hazard mitigation proposals, as early as possible in the preaward process to improve consistency  In 2014, PA program officials analyzed the PA grant process and used input from agency staff and officials involved in various aspects of the program to identify potential improvements  The resulting Public Assistance Program Realignment report found that challenges in workforce management, information sharing, and hazard mitigation continued, and included recommendations for improvement  For example, the report concluded that a shortage of qualified staff, high turnover, unclear organizational responsibilities, and inconsistent training were long-standing and continuing challenges that impaired the PA pre-award process  In addition, from January 2015 to April 2015, FEMA conducted extensive outreach with more than 260 stakeholders across FEMA headquarters, all 10 regions, 43 states, and 4 tribal nations to discuss challenges in the PA program and opportunities for improvement  For example, stakeholders identified challenges with ineffective information collection during the preaward process and suggested identifying special considerations, such as hazard mitigation, earlier in the PA process as an idea for improvement  In response, FEMA began redesigning the PA preaward process to operationalize the results of its 2014 report and address areas for improvement identified through its outreach efforts  FEMA awarded a contract for program support to help PA officials implement a redesigned PA program in 2015  This included a new process to develop and review grant applications, and obligate PA funds to states affected by disasters; new positions, such as a new program delivery manager who is the single point of contact throughout the grant application process; a new Consolidated Resource Center (CRC) to support field operations by supplementing project development, validation, and review of proposed PA project applications; and a new information system to maintain and share PA grant application documents  As part of the new process, PA program officials identified the need to ensure that staff emphasize special considerations, such as hazard mitigation, earlier in the process  Taken together, these efforts represent FEMA’s “new delivery model” for awarding PA program grants  Enhancements in the PA program under the new delivery model are presented in figure 4  Regarding the new delivery model process, FEMA introduced several changes to enhance outreach to applicants during the “exploratory call”— the first contact between FEMA and local officials—and during the first in- person meeting, called the “recovery scoping meeting ” FEMA also revised decision points during the process, when program officials can request more information from applicants, and applicants can review and approve the completion of project development steps  FEMA also incorporated special considerations, such as hazard mitigation, earlier in the new process during the exploratory calls and recovery scoping meetings  The changes and enhancements to the PA grant award process in the new delivery model are presented in figure 5  The new process divides proposed PA projects based on complexity and type of work into three categories—100 percent completed, standard, and specialized—that PA staff manage to expedite review or assign skilled staff to technical projects as needed  If the applicant has already completed work following a disaster, such as debris removal, it is considered “100 percent completed” and JFO staff collect the necessary documents and provide the information to the CRC staff who complete the development of project applications, validate the information, and complete all necessary reviews  Projects that require repairs and further assistance from PA program staff at the JFO include “standard” and “specialized” projects, which include a site inspection to document damages, before the JFO staff provide the information to the CRC  Further, PA program officials assign PA staff based on their skills and experience to standard projects, which are less technically complex to develop, and specialized projects, which are more technically complex and costly  We discuss the new workforce positions FEMA developed for JFOs and CRCs, the new information system FEMA developed to maintain and share PA grant documents with applicants, and FEMA’s efforts to incorporate hazard mitigation into PA projects later in this report  Since 2015, FEMA has invested almost $9 million to redesign the PA program through the reengineering and implementation of the new delivery model, including about $4 7 million for contract support for implementation, and $4 million for acquisition of the new information system  FEMA tested the new delivery model in a series of selected disasters, using a continuous process improvement approach to assess and improve the process, workforce changes, and information system requirements, prior to implementing the new model for all future disasters  For example, FEMA first tested the new process in Iowa in July 2015 and, in February 2016, PA program officials expanded their test to include all of the new staff positions  In October 2016, PA program officials added the new information system to achieve a comprehensive implementation of all of the elements of the new delivery model for the agency’s response to Hurricane Matthew in Georgia, two additional disasters in Georgia in January 2017, and in Missouri, North Dakota, Wyoming, Vermont, and two disasters in New Hampshire from June through August 2017  The timeline for PA’s implementation of the new delivery model is shown in figure 6  According to program officials, FEMA planned to implement the new model for all future disasters beginning in January 2018  However, historic disaster activity during the 2017 hurricane season accelerated full implementation  As a result, on September 12, 2017, FEMA officials announced that, unless officials determined it would be infeasible in an individual disaster, the program would use the new delivery model in all future disasters  According to FEMA’s 2014 PA Program Realignment report and other program documents, PA officials designed the new delivery model to respond to persistent workforce management challenges related to identifying the required number of staff and needed skills and training, among other things, to improve the efficiency and effectiveness of the PA preaward process  To address these challenges, PA program officials centralized much of the responsibility for processing PA projects in the CRCs, created additional new positions with specialized roles and responsibilities in JFOs, and established training and mentoring programs to help build the new staffs’ skills  In 2016, PA program officials centralized some of the project activities that otherwise were being carried out at individual JFOs at FEMA’s first new CRC in Denton, Texas  Officials did so by establishing 18 new positions, many of which directly correlated with positions that FEMA deployed to individual JFOs in the legacy PA delivery model  According to PA officials, centralizing positions will improve standardization in project processing, and result in a higher quality work product  As part of the new delivery model, PA program officials created a new hazard mitigation liaison position for PA program staff at the CRC that did not previously exist at individual JFOs  The other new positions that PA program officials either created or centralized at the CRC included two specialized positions responsible for costing and validating PA projects  Previously, the PA project specialist deployed to the JFO would complete these tasks and others; however, the consistency of project development varied across the regions and disasters  In contrast, CRC staff are full-time employees who receive training to specialize in completing standardized project development steps for PA projects from multiple disasters on an ongoing basis  Program officials anticipate that centralizing new specialized staff at the CRCs will also reduce PA administrative costs and staffing levels at the JFOs  For example, staff at the CRCs, such as the new hazard mitigation liaisons and insurance and costing specialists, could support project development for multiple disasters and regions simultaneously, whereas PA previously needed to deploy staff to each JFO to fulfill these roles  In addition, once JFOs operating under the new model send projects to the CRCs for processing and review, FEMA can more rapidly close its JFOs, reducing associated administrative costs  For example, following Hurricane Matthew, FEMA credited the new delivery model, in part, with its ability to close the JFO in Georgia sooner than several other JFOs in neighboring states not involved in the implementation of the new delivery model  PA program officials created new positions with more specialized roles and responsibilities to help PA staff at JFOs provide more consistency in the project development process and guidance to applicants  Program officials split the broad responsibilities previously managed at the JFOs by PA crew leaders and project specialists, into two new, specialized positions—the program delivery manager and site inspector  The program delivery manager serves as the applicant’s single point-of-contact throughout the preaward process, manages communication with the applicant, and oversees document collection  All three PA grant applicants we spoke to following Hurricane Matthew in Georgia greatly appreciated the knowledge and assistance provided by their program delivery managers  Site inspectors are responsible for conducting the site inspection to document all disaster-related damages; determining the applicant’s plans for recovery, coordinating with other specialists, and verifying the information collected with the applicant  Officials expect deployed staff at JFOs can complete the fieldwork faster and provide greater continuity of service to applicants  Further, program officials believe that specializing roles will enable them to provide more targeted training, and improve employee satisfaction  Site inspection, hazard mitigation, and environmental and historic preservation specialists, along with a new Public Assistance program mentor, conduct a site inspection with the applicant to document damages to a historic cemetery in Savannah, Georgia, following Hurricane Matthew in 2016  PA program officials designed new training and mentoring programs for the new positions at the CRCs and JFOs and used a continuous feedback process to update and improve the training, position guides, and task books throughout the implementation of the new delivery model, according to PA officials  According to a June 2017 update of the PA Cadre Training Plan, training for the new model has five major focuses: required training and skills for position qualification; on-site refresher training; mentor training; regional-based state, local, tribal, and territorial training; and training on the new information system  Specifically, officials developed six new training courses, and identified which are required for each position under the new delivery model  For example, a program delivery manager at the JFO is required to complete both the program delivery manager and site inspector specialist courses  As of June 2017, PA program officials had provided at least one new model training course to 93 percent of their cadre (including program delivery manager training to 366 individuals and site inspector training to 1,172 individuals) and planned to provide 28 additional courses through September 2017 to the PA cadre  According to regional and CRC officials, the training courses and mentoring from experienced staff helped maximize new staff’s capabilities in the new process  Throughout the third implementation of the new delivery model, JFO and CRC staff, as well as regional PA staff, stakeholders, and applicants, identified staff skills and training as a key area that needed more attention for full implementation of the new delivery model  Our work and FEMA’s after-action reports from the third test in Georgia identified problems with site inspector skills, which affected the timeliness and accuracy of projects  Specialists and managers at the CRC noted that poorly trained site inspectors did not consistently provide the necessary information from the field, which resulted in delays for the CRC staff to process projects, and after-action reports also identified challenges with site inspector skills  According to a PA applicant in Georgia, the inconsistency of skills and experience of their site inspector resulted in the need to conduct a “do-over” site inspection on one of the applicant’s projects, causing delays  PA staff and state officials attribute much of the site inspectors’ skill gaps to their lack of training and experience in the program  According to PA Region officials, providing timely training will be a resource-intensive challenge for implementing the new delivery model for all future disasters  For example, it can be difficult to train reservists before FEMA deploys them to disasters, and many of the program’s experienced reservists have retired or resigned, resulting in few mentors for the program and a high need to provide training to inexperienced and newly hired staff  PA officials and stakeholders also emphasized the need for FEMA to provide additional training for state and local officials to build capacity and support the goals of the new delivery model  For example, according to JFO officials at the third implementation, the new delivery model increases responsibilities for applicants, who will require more applicant training than FEMA currently provides  According to state officials, applicant capabilities vary, and FEMA should provide training to state and local officials on the new delivery model and the information system before a disaster  Skill gaps among applicants could result in inconsistent implementation of the new process, according to PA staff and stakeholders, and PA staff said that training was important to prevent applicants from reverting back to the legacy PA grant application process  To support full implementation of the new delivery model for all disasters, PA program officials have updated training courses for PA staff and applicants, and planned additional training to address these challenges and other lessons learned through the test implementation  For example, PA officials told us they updated the site inspector training program in May 2017 and scheduled a new site inspector training session in August 2017 to include more hands-on training to help address the skill gaps identified for site inspectors  PA officials created a new training course for FEMA’s regional offices, in part to enable regional PA staff to provide new delivery model training to state and local officials  PA officials also planned to develop a self-paced, online course for state and local officials by the end of 2017  PA officials have not fully assessed the workforce needed for JFO field operations, CRC staff, or FIMA’s hazard mitigation staff to support implementation of the new delivery model for all future disasters  PA program officials developed an initial assessment of the total number of staff needed in the field and the CRCs in 2016 to estimate cost savings associated with consolidating and specializing positions at the CRCs and deploying fewer staff to JFOs  However, the assessment did not identify the number of staff required to fill specific positions, including program delivery managers and hazard mitigation specialists, needed to support the new delivery model for full implementation  In reviewing the test implementations of the new delivery model, we found that inadequate staffing levels at the JFOs and CRCs, and with FIMA’s hazard mitigation staff, affected staffs’ ability to achieve the goals of the new delivery model  Staff levels at the JFO  We identified challenges with having the right number of program delivery managers and site inspection specialists to achieve program goals for customer satisfaction, efficiency, and quality in test implementations of the new delivery model  For example, in the second test implementation of the new delivery model in Oregon in 2016, PA did not deploy enough program delivery managers to the disaster, which resulted in unmanageable caseloads for program delivery managers, according to state and PA officials  PA program officials assigned program delivery managers an average caseload of 12 PA applicants, which was more than they could effectively manage, according to PA staff, and program officials aim for a caseload of 8 to 10 applicants  According to state officials, local officials reported they did not always receive the support they needed from program delivery managers who managed caseloads consisting of dozens of projects at multiple sites for each applicant during the Oregon implementation  As a result of overwhelmed program delivery managers, local officials faced challenges understanding their responsibilities, such as recognizing when they needed to provide information for the project development to proceed, according to state officials  PA staff involved with the third test implementation in Georgia in 2016 and 2017 said there were not enough site inspectors or program delivery managers to fully manage the workload at the JFO  Because of the specialization of roles, projects could not move forward when there were not enough staff to execute the next step in the process  For example, PA staff at the JFO said program delivery managers completed recovery scoping meetings rapidly, but faced a bottleneck in scheduling site inspections because there were more applicants awaiting site inspections than could be fulfilled by the number of site inspection specialists available  Staff levels at the CRC  Staff at the CRC reported challenges with staffing levels during the Oregon and Georgia test implementations, and expressed concerns about when PA officials will staff the CRCs to support full implementation of the new model for all disasters  During the Oregon test implementation, a CRC specialist said there were not enough technical specialists to manage the workload and, as a result, PA program officials had to redeploy site inspectors from their JFO field operations to the CRC to complete costing estimates  During the third test in Georgia, quality assurance specialists said that their workload resulted in added stress trying to complete the work in time while adhering to quality standards  According to CRC specialists in Denton, Texas, PA officials had not determined required staff levels for full implementation, but agreed that workload was too high and program officials needed to determine the appropriate staff levels for each CRC to support full implementation  PA officials were still evaluating CRC processing times and workload management from the Oregon and Georgia test implementations to determine staffing needs, according to PA officials  Further, PA program officials plan to establish a second CRC in Winchester, Virginia, before the end of 2017, but have not determined the number of additional permanent full-time staff needed to support the CRCs for full implementation of the new delivery model  Staff levels for the hazard mitigation specialists  PA officials have not identified the number of hazard mitigation specialists in FIMA’s hazard mitigation cadre needed for full implementation of the new delivery model  According to JFO staff, current hazard mitigation staff levels are insufficient to provide the desired in-person participation of hazard mitigation staff on all recovery scoping meetings to share information on hazard mitigation with applicants and help them identify potential mitigation opportunities  A PA program official said officials missed opportunities to pursue hazard mitigation during the test implementation after Hurricane Matthew in Georgia due to lack of hazard mitigation specialists  In addition, for the test implementation in Oregon, there were not enough hazard mitigation specialists to cover all site inspections and implement their new delivery model responsibilities, according to FEMA’s after-action reports  The absence of hazard mitigation specialists in the early stages of PA project development may cause delays in officials’ identifying hazard mitigation opportunities, according to a FIMA official  PA program officials said they did not work with FIMA to determine the appropriate levels of hazard mitigation staff under the new delivery model because they were refining the new process, but as of June 2017 were working with FIMA to do so  One of the key implementation activities in our Business Process Reengineering Assessment Guide includes addressing workforce management issues  Specifically, this includes identifying how many and which employees will be affected by the position changes and retraining  Further, our prior work has found that high-performing organizations identify their current and future workforce needs—including the appropriate number and deployment of staff across the organization— and address workforce gaps, to improve the contribution of critical skills and competencies needed for mission success  According to a PA program official, their initial workforce assessment was not comprehensive because they were still collecting data required to make informed decisions  PA officials agreed that updating their workforce assessments prior to full implementation could be helpful, and acknowledged that program officials needed to be more proactive applying the lessons learned as they pivot from testing to full implementation of the new delivery model in 2018  FEMA also conducts a standard agency wide workforce structure review every 2 to 3 years, which helps officials determine the appropriate disaster workforce levels  As of June 2017, PA officials were working with other offices within FEMA to expedite the agency-wide assessment of the PA and FIMA hazard mitigation cadres, but did not know when they would complete the assessment  PA officials also acknowledged that they faced an aggressive schedule to complete various planned activities for workforce management, training, and other efforts, in support of full implementation, and that they may not be able to complete all efforts as thoroughly as they would like in order to expedite the transition of the PA program to the new delivery model  The gaps in PA workforce assessment in the JFOs, CRCs, and for FIMA’s hazard mitigation cadre present a risk that PA program managers will not have a sufficient workforce to support the goals of the new delivery model  In addition, the timing and implementation of the hiring and training activities for new PA program staff could take multiple months, and program officials will need to know what staff levels are necessary for full implementation of the new delivery model to inform resource decisions for the program in coordination with other agency offices  According to PA program officials, workforce assessment efforts have been delayed as a result of disaster response and recovery efforts related to Hurricanes Harvey, Irma, and Maria  Completing a workforce assessment will help program officials identify gaps in their workforce and skills, which could help PA program officials minimize the effects of long- standing workforce staffing and training challenges on the PA program delivery and inform full implementation for all disasters  costs  For example, EMMIE does not collect information on all of the preaward activities that are part of the PA grant application process  As a result, PA program officials said they, and applicants, must use ad hoc reports and personal tracking documents to manage and monitor the progress of grant applications  PA officials added that EMMIE is not user- friendly and applicants often struggle to access the system  In response to these ongoing challenges, PA program officials developed FAC-Trax— a separate information system from EMMIE—with new capabilities designed to improve transparency, efficiency, and management of the PA program  Specifically, FAC-Trax allows FEMA staff (PA Grants Manager) and applicants (PA Grants Portal), to review, manage, and track current PA project status and documentation  For example, applicants can use FAC- Trax to submit requests for public assistance, upload required project documentation, approve grant application items, and send and receive notifications on grant progress and activities  In addition, the FAC-Trax system includes standardized forms, as well as required fields and tasks that PA program staff and applicants must complete before moving on to the next steps in the PA preaward process  According to PA officials, these capabilities increase transparency, encourage greater applicant involvement, and enhance collaboration and communication between FEMA and grant applicants, to improve efficiency in processing and awarding grant applications and enhance the quality of project development  Further, PA officials said that FAC-Trax could reduce challenges associated with staff turnover during the project development process because the system stores and maintains applicant information and project documentation, making it easier for transitioning staff to assist an applicant  They also said they use FAC-Trax to gather and analyze data that supports management of the PA process, including measuring the timeliness of the grant application process  For example, during the test implementation of the new delivery model in Georgia following Hurricane Matthew, officials were able to document that, on average, program delivery managers took 5 days to conduct the exploratory call and 14 days to hold the recovery scoping meeting with applicants, and CRC officials took 33 days to develop and review grant proposals  Managers use this data to assess staffing needs and identify bottlenecks in the PA process, according to PA officials  FAC-Trax is critical to the new PA delivery model and will be a primary means of sharing grant application documents, tracking ongoing PA projects, and ensuring that FEMA staff and applicants follow PA grant policies and procedures  Given the importance of developing and testing this new information sharing system, we evaluated its development against four key IT management controls—(1) project planning; (2) risk management; (3) requirements development; and (4) systems testing and integration  When implemented effectively, these controls provide assurance that IT systems will be delivered within cost and schedule and meet the capabilities needed by its users  We found that FEMA’s development of FAC-Trax fully satisfied best practices for project planning and risk management, but additional steps are needed to fully satisfy the areas of requirements development and systems testing and integration, as discussed below  See appendix II for the full assessment of each IT management control  PA program officials fully satisfied all five practices in the project planning control area, according to our assessment  Key project planning practices are (1) establishing and maintaining the program’s acquisition strategy, (2) developing and maintaining the overall project plan and obtaining commitment from relevant stakeholders, (3) developing and maintaining the program’s cost estimate, (4) establishing and maintaining the program’s schedule estimate, and (5) identifying the necessary knowledge and skills needed to carry out the program  To address the first and second practices, program officials established detailed plans that describe the acquisition strategy and objectives, the program’s scope, and its framework for using an Agile software development approach, among other key actions  Agile is a method of software development that utilizes an iterative process and constantly improves software based on user needs and feedback  Program officials also developed a plan detailing the program’s approach to deploy and maintain FAC-Trax and established stakeholder groups and an integrated product team to support and oversee the development of FAC-Trax  To address the third and fourth practices, they developed and maintained a master schedule of all implementation tasks and milestones through project completion, and developed a life-cycle cost estimate of over $19 million  Additionally, FAC-Trax’s acquisition performance baseline describes the system’s minimum acceptable and desired baselines for performance, schedule, and cost  Lastly, in regards to the fifth practice, program officials identified the knowledge and skills needed to carry out the program in the FAC-Trax Request for Proposal and FAC-Trax Capability Development Plan  PA program officials fully satisfied all four practices in the risk management control area, according to our assessment  Key risk management practices are (1) identifying risks, threats, and vulnerabilities that could negatively affect work efforts, (2) evaluating and categorizing each identified risk using defined risk categories and parameters, (3) developing risk mitigation plans for selected risks, and (4) monitoring the status of each risk periodically and implementing the risk mitigation plan as appropriate  To address the first and second practices, program officials identified key risks that could negatively affect FAC-Trax in a “risk register”—an online site used to track risks, issues, and mitigating actions  As of May 2017, officials had identified 13 risks in the risk register—four open and nine closed—and evaluated and categorized the identified risks based on the probability of occurrence and scope, schedule, and cost impacts  For example, program officials reported that two of its open risks have a “medium” risk rating—meaning the risk has the potential to slightly affect project cost, schedule, or performance  To address the third and fourth practices, program officials developed and documented risk mitigation plans for all identified risks  For example, program officials planned to mitigate the risk of limited engagement of subject matter experts by identifying and engaging with appropriate experts through workshops, and monitoring the capability development process to identify any issues that may cause project delays  In addition, PA program officials documented the responsible officials, reevaluation date, and risk status, among other things, for each risk in the register, and reviewed and updated risks during weekly and monthly program reviews with stakeholders throughout FEMA  PA program officials fully satisfied four out of five practices in the requirements development control area, according to our assessment  Key requirements development practices are (1) eliciting stakeholder needs, expectations, and constraints, and transforming them into prioritized customer requirements; (2) developing and reviewing operational concepts and scenarios to refine and discover requirements; (3) analyzing requirements to ensure that they are complete, feasible, and verifiable; (4) analyzing requirements to balance stakeholder needs and constraints; and (5) testing and validating the system as it is being developed  To address the first and second practices, program officials developed a requirements management plan outlining how officials capture, assess, and plan for FAC-Trax enhancements, and established a change control process to review, prioritize, and verify user requests for changes to the system and feedback  As of May 2017, the PA program office received 734 change requests related to FAC-Trax, of which program officials completed 420 changes and planned to address an additional 277 entries  Program officials also developed a functional requirements document outlining the high-level requirements for FAC- Trax and detailed operational concepts and scenarios for each phase of the preaward process in the system’s concept of operations  To address the fourth practice, program officials created a standard template to analyze and document the user needs and acceptance criteria for planned system capabilities in March 2017  In addition, PA program officials identified risks and dependencies for recommended changes to FAC-Trax, and balanced the cost and priority of system enhancements as part of the change control process  Lastly, regarding the fifth practice, program officials tested and evaluated FAC-Trax during development, which included validating system enhancements through user acceptance testing  However, program officials did not fully address the third practice— analyzing requirements to ensure they are complete, feasible, and verifiable—because they did not ensure detailed user requirements were necessary and sufficient by tracking them back to higher-level requirements  For example, although program officials reviewed change requests for completeness and followed up with users to verify requirements, officials did not track system enhancements, made in response to detailed user requirements (e g


Report 2: , allowing users to search PA projects by project number), back to the high-level requirements (e g , storing data and information provided by the applicant) identified in the FAC-Trax functional requirements document and performance work statement  Officials did not track system enhancements back to high-level requirements because they did not have a complete understanding of basic user needs and system requirements at the beginning of the FAC- Trax effort, according to the PA program manager  A PA official also said the change control process was a way to identify the basic capabilities FAC-Trax needed to have and that tracking enhancements back to high- level requirements could have made the change control process more difficult to manage, and reduced user participation if, for example, users needed to understand how their change requests related to high-level requirements  However, program officials could have tracked enhancements back to high-level requirements themselves using the change control process without putting any additional burden on users  Despite not having a complete understanding of user needs and system requirements at the beginning of the FAC-Trax effort, analyzing whether users’ change requests satisfy higher-level requirements identified in key design and planning documents would have provided officials with a basis for more detailed and precise requirements throughout project development and helped them better manage the project, according to IT management controls  Further, according to the PMBOK® Guide, tracking or measuring system capabilities against approved requirements is a key process for managing a project’s scope, measuring project completion, and ensuring the project meets user needs and expectations  Program officials acknowledged the importance of tracking system enhancements back to documented system requirements  Ensuring that FAC-Trax meets user needs and expectations is especially important because the information system is key to the success of the new delivery model, according to PA officials  By analyzing progress made on documented, high-level requirements, a step that reflects a key IT management control for requirements development, the PA program will have greater assurance that FAC-Trax will provide functionality that meets user needs and expectations  PA program officials did not fully satisfy either of the two practices in the systems testing and integration control area, according to our assessment  Key systems testing and integration practices are (1) developing test plans and test cases, which include a description of the overall approach for system testing, the set of tasks necessary to prepare for and perform testing, the roles and responsibilities for individuals or groups responsible for testing, and criteria to determine whether the system has passed or failed testing; and (2) developing a systems integration plan to identify all systems to be integrated, describe how integration problems are to be documented and resolved, define roles and responsibilities of all relevant participants, and establish a sequence and schedule for every integration step  In regards to the first practice, PA program officials and the FAC-Trax contractor established a test plan that identifies the method and strategy to perform testing, including the necessary tasks, such as responding to user feedback and testing errors, and incorporating necessary resolutions into future work, testing parameters, and the roles and responsibilities of the individuals responsible for testing  However, program officials have not developed system testing criteria to evaluate FAC-Trax, which would align with the practice described above of using criteria to determine whether the system has passed or failed testing  A key feature of Agile software development is the “definition of done”—a set of clear, comprehensive, and objective criteria, that the government should use to evaluate software after each iteration of development  PA program officials said they did not establish a definition of done because officials initially managing the FAC-Trax effort lacked familiarity with system development in the Agile environment  Officials acknowledged the importance of establishing a definition of done and said they are planning to develop one, but have not identified how or when to incorporate it into the development process  According to the TechFAR—the government’s handbook for procuring digital services using Agile processes—the government and vendor should establish this definition after contract award at the beginning of each cycle of software development  By establishing criteria, such as a definition of done, to evaluate the system—a step that reflects a key IT management control for system testing and is an effective practice for applying Agile to software development—the PA program will have greater assurance that FAC- Trax is usable and responsive to specified requirements  In regards to the second practice, PA program officials developed a systems integration plan in June 2017 that identified the potential for integration of FAC-Trax with four FEMA systems, including EMMIE  In addition, program officials included a description of how staff should document integration problems and the resolution of problems in FAC- Trax development and test plans  However, the systems integration plan does not define roles and responsibilities of all participants for system integration activities or establish a sequence and schedule for every integration step for the four FEMA systems  PA officials said that system integration planning for FAC-Trax is in the early stages, but acknowledged the importance of these elements of system integration planning  Officials plan to define roles and responsibilities of all participants for system integration activities and develop the sequence and schedule for every integration step as they add new systems to the FAC-Trax development plan and obtain funding needed for their integration  Nonetheless, FEMA has used FAC-Trax for selected PA disasters since October 2016 and plans to use FAC-Trax for all future disasters  According to IT management controls, agencies should establish the systems integration plan early in the project and revise it to reflect evolving and emerging user needs  By ensuring that the FAC- Trax systems integration plan defines the roles and responsibilities of relevant participants for all integration relationships and establishes a sequence and schedule for every integration step, the PA program will have greater assurance that FAC-Trax functions properly with other systems and meets user needs  FEMA’s new delivery model enhances participation of hazard mitigation staff with the goal of identifying opportunities for mitigation earlier in the PA preaward process, according to PA officials  Two key changes related to hazard mitigation under the new model include (1) an emphasis on engaging with hazard mitigation specialists at the JFO earlier in the PA process and involving them in specific PA preaward activities and (2) the establishment of the PA program’s hazard mitigation liaison at the CRC  For example, position guides direct program delivery managers to coordinate with FIMA’s hazard mitigation specialists prior to recovery scoping meetings, and site inspectors to coordinate with hazard mitigation specialists prior to site inspections to discuss a PA grant applicant’s damages and any potential mitigation opportunities  PA program officials also developed guidance for conducting the exploratory call and the recovery scoping meeting with applicants, which include questions for PA staff to ask on the applicant’s interest in or plans for incorporating hazard mitigation into potential projects  In addition, a new hazard mitigation liaison at the CRC is responsible for reviewing PA projects for hazard mitigation opportunities and serving as a mitigation subject matter expert for the PA program  According to data provided by FEMA, PA grant applicants incorporated hazard mitigation into approximately 18 percent of permanent work projects for all disasters nationwide from 2012 to 2015  During test implementation of the new delivery model, state, PA, and FIMA officials all reported an increase in the number of hazard mitigation activities on PA permanent work projects  For example, state officials who participated in the second new model test in Oregon said that effective communication and coordination between PA and hazard mitigation staff resulted in applicants incorporating hazard mitigation into over 60 percent of permanent work projects  Furthermore, PA officials reported an increase in hazard mitigation during the third test implementation of the new model in Georgia following Hurricane Matthew, where approximately 16 percent of permanent work projects included mitigation, as of June 2017  This represents an increase compared to the PA program’s estimate for the proportion of projects that incorporate hazard mitigation among previous PA hurricane disasters in Georgia, which was about 3 percent, according to PA officials  While PA officials are trying to increase hazard mitigation through the new delivery model, not all disasters present the same number of opportunities to incorporate hazard mitigation  First, the PA program only incorporates hazard mitigation measures for permanent work projects, such as repairs to roads, bridges, and buildings  For example, as of June 2017, approximately 60 percent of the projects FEMA funded in Georgia for the third test implementation after Hurricane Matthew were for emergency work, which is not eligible for hazard mitigation measures  Second, the PA program only funds mitigation measures that officials determine to be cost-effective  In addition, we have previously reported on other factors that affect whether applicants incorporate hazard mitigation into PA projects, such as their capacity to manage and ability to fund hazard mitigation projects  National Planning for Hazard Mitigation In our 2015 report on disaster resilience following Hurricane Sandy, we noted that disaster affected areas have different threats and vulnerabilities, and local stakeholders make the ultimate determination whether or not to incorporate hazard mitigation into a project  Further, without a strategic approach to making disaster resilience investments, the federal government and its nonfederal partners may be unable to fully capitalize on opportunities for mitigation on the greatest known threats and hazards  We recommended that the Mitigation Framework Leadership Group develop an investment strategy to help ensure that federal funds expended to enhance disaster resilience achieve the goal of reducing the nation’s fiscal exposure because of climate change and the rise in the number of federal major disaster declarations as effectively and efficiently as possible  In response, the Federal Emergency Management Agency (FEMA) plans to issue a final National Mitigation Investment Strategy in 2018  The goals of this strategy include increasing the effectiveness of investments in reducing disaster losses and increasing resilience, and improving coordination of disaster risk management among federal, state, local, tribal, territorial, and private entities  Although the new model establishes hazard mitigation activities for PA and FIMA staff in the preaward process, it does not standardize and prioritize hazard mitigation planning at JFOs in the way FEMA has done with prior PA program policy  Specifically, FEMA’s 2007 PA program policy standardized planning for hazard mitigation across PA recovery efforts by stating that agency and state officials should issue a memorandum of understanding (MOU) early in the disaster, outlining how PA hazard mitigation will be addressed for the disaster, including what mitigation measures will be emphasized, and identifying applicable codes and standards, and any potential integration with other mitigation grant programs  However, PA program officials did not include guidance that standardizes planning for hazard mitigation, such as encouraging the use of an MOU, in FEMA’s 2010 PA program policy, the most recent update to the Public Assistance Program and Policy Guide in April 2017, or the New Delivery Model Operations Manual  As a result, FIMA officials said FEMA and state officials do not consistently issue MOUs that outline how FEMA and the state plan to promote PA hazard mitigation during the recovery effort, explaining that the use of the MOU is based on the preferences and priorities of the FEMA officials involved  When not issuing an MOU, FIMA hazard mitigation staff and PA officials at the JFO meet to determine the extent which hazard mitigation staff interact directly with applicants regarding PA hazard mitigation during the recovery process, according to a FIMA official  Having a consistent approach to planning for and prioritizing hazard mitigation across all disasters is important for FEMA, given that FEMA experienced challenges consistently prioritizing and integrating hazard mitigation across PA recovery efforts, according to GAO and others  For example, in our 2015 report on resilience in the Hurricane Sandy recovery, we found that state and local officials experienced challenges maximizing disaster resilience in the recovery effort because PA officials did not consistently prioritize hazard mitigation during project development  According to FEMA’s National Mitigation Framework, planning is vital for mitigation efforts during disaster recovery, and federal, state, and local officials should establish procedures that emphasize a coordinated delivery of mitigation activities and capitalize on opportunities to reduce future disaster losses  Similarly, the Recovery Federal Interagency Operational Plan, which supports FEMA’s National Disaster Recovery Framework, identifies planning as a key task for identifying mitigation opportunities and integrating risk reduction considerations into decisions and investments during the recovery process  FIMA officials agreed that including the development of a formal plan, such as the historical 2007 PA program policy regarding the use of MOUs, for PA hazard mitigation in operations guidance would help program officials plan for and prioritize hazard mitigation  They noted that FIMA’s hazard mitigation field operations guide includes procedures for implementing proposed MOUs to achieve mitigation goals  PA program officials said that, in light of changes to the PA process under the new model and subsequent updates to program policies, the MOU policy from the 2007 PA program policy was outdated  But officials agreed that planning for and prioritizing hazard mitigation at the operational level is important and said they were examining additional ways to incorporate these activities early in the PA process  As FEMA continues to implement the new model, establishing procedures to standardize hazard mitigation planning for each disaster, as it did through prior policy, could improve the prioritization of hazard mitigation in PA recovery efforts and increase the effectiveness of mitigation for reducing disaster losses and increasing resilience  PA program officials developed performance objectives and measures for hazard mitigation in the new delivery model, but could add measures to better align performance assessment for the PA program with FEMA’s broader strategic goals for hazard mitigation  In its strategic plan for 2014–2018, FEMA established an agency-wide goal to increase the percentage of FEMA-funded disaster projects, such as those under the PA program, that provide mitigation above local, state, and federal building code requirements by 5 percentage points by the end of fiscal year 2018  For example, local building codes may require measures for new construction to mitigate against future damage  To align with FEMA’s strategic goal, PA officials would also need to measure the number of PA projects that included mitigation measures that bring any repaired infrastructure to a level above applicable building codes  However, under the new model, FEMA officials developed performance objectives (and associated measures) to increase the number of projects that include hazard mitigation by 5 percent, and increase the total dollars spent on hazard mitigation by 2 percent  While these measures could help to incentivize mitigation, they are not tied to building codes and do not include specific information that FEMA could use to continually assess the PA program’s contributions to meeting the agency’s strategic goal  According to Standards for Internal Control in the Federal Government, agency management should design control activities, such as establishing and reviewing performance measures, to achieve the agency’s objectives  In addition, our work on leading public sector organizations has found that such organizations assess the extent to which their programs and activities contribute to meeting their mission and desired outcomes, and strive to establish clear hierarchies of performance goals and measures  A clear connection between performance measures and program offices helps to both reinforce accountability and ensure that, in their day-to-day activities, managers keep in mind the outcomes their organization is striving to achieve  FEMA’s ability to evaluate and report on PA hazard mitigation data is constrained, but officials are addressing this challenge through the development of data reporting and analytics capabilities for the PA program’s new information system, according to PA officials  PA program officials developed measures they could use to evaluate the new model during test implementation and compare new model performance to the legacy PA process, and agreed that aligning PA program hazard mitigation goals with FEMA’s agency-wide strategic goals would be helpful  As FEMA continues to develop and implement the new model, developing performance measures and objectives to better inform and support the agency’s broader strategic goals could help to ensure that FEMA capitalizes on hazard mitigation opportunities in PA recovery efforts  FEMA’s Public Assistance grant program is a complicated, multi-billion dollar program that is critical to helping state and local communities rebuild and recover after a major disaster  In recent years, FEMA has undertaken a major reengineering effort to make the PA preaward process simpler and more efficient for applicants and to address challenges encountered during recovery from past disasters  FEMA’s new delivery model represents a significant opportunity to strengthen the PA program and address these past challenges, and growing pains are to be expected when implementing any large reengineering effort  Further, FEMA officials work to implement these changes while supporting response and recovery following disasters, including the catastrophic flooding from Hurricane Harvey in August 2017 and widespread damages from Hurricanes Irma and Maria in September 2017  As such, it is critical that feedback obtained and lessons learned while testing the new model inform decisions and actions as FEMA proceeds with full implementation for all disasters, including the complex recovery efforts in the states and territories affected by Hurricanes Harvey, Irma, and Maria  FEMA has redesigned the PA delivery model to address various challenges related to workforce management, information sharing with state and local grantees, and incorporating hazard mitigation into PA projects  FEMA has developed new workforce processes, training, and positions to address past challenges, but completing a workforce assessment that identifies the number of staff needed will inform workforce management and resource allocation decisions to help FEMA ensure a more successful implementation  This is particularly important as FEMA is using the new model for the long-term recovery from the 2017 hurricanes, and FEMA faces capacity challenges as its workforce is stretched thin  Further, FEMA’s new FAC-Trax information sharing system provides FEMA and state and local applicants and grantees with better capabilities to address past challenges in managing and tracking PA projects  In developing FAC-Trax, FEMA implemented many of the key IT management controls that can help ensure that new IT systems are implemented effectively  However, additional steps are needed to fully satisfy the areas of requirements development and systems testing and integration  Finally, FEMA has taken some actions to better promote hazard mitigation as part of its new PA model  However, more consistent planning for hazard mitigation following a PA disaster and developing specific performance measures and objectives that better align with and support the agency’s broader strategic goals related to hazard mitigation could help to ensure that mitigation is incorporated into recovery efforts, which presents an opportunity to encourage disaster resilience and reduce federal fiscal exposure from recurring catastrophic natural disasters  We are making the following five recommendations to FEMA’s Assistant Administrator for Recovery: The FEMA Assistant Administrator for Recovery should complete a workforce staffing assessment that identifies the appropriate number of staff needed at joint field offices, Consolidated Resource Centers, and in FIMA’s hazard mitigation cadre to implement the new delivery model nationwide  (Recommendation 1) The FEMA Assistant Administrator for Recovery should establish controls for tracking FAC-Trax capabilities to the system’s functional and operational requirements to more fully satisfy requirements development controls and ensure that the new information system provides capabilities that meets users’ needs and expectations  (Recommendation 2) The FEMA Assistant Administrator for Recovery should establish system testing criteria, such as a “definition of done,” to assess FAC- Trax as it is developed; define the roles and responsibilities of all participants; and develop the sequence and schedule for integration of other systems with FAC-Trax to more fully satisfy systems testing and integration controls  (Recommendation 3) The FEMA Assistant Administrator for Recovery, in coordination with the Associate Administrator of the Federal Insurance and Mitigation Administration, should implement procedures to standardize planning for addressing PA hazard mitigation at the joint field offices, for example, by requiring FEMA and state officials to develop a memorandum of understanding outlining how they will prioritize and address hazard mitigation following a disaster as it did through prior policy  (Recommendation 4) The FEMA Assistant Administrator for Recovery, in coordination with the Associate Administrator of the Federal Insurance and Mitigation Administration, should develop performance measures and associated objectives for the new delivery model to better align with FEMA’s strategic goal for hazard mitigation in the recovery process  (Recommendation 5) We provided a draft of this report to DHS and FEMA for review and comment  DHS provided written comments, which are reproduced in appendix III  In its comments, DHS concurred with our recommendations and described actions planned to address them  FEMA also provided technical comments, which we incorporated as appropriate  With regard to our first recommendation, that FEMA complete a workforce staffing assessment that identifies the number of staff needed at joint field offices, Consolidated Resource Centers, and FIMA’s hazard mitigation cadre, DHS stated that PA, in coordination with the Field Operations Directorate and FIMA, will continue to refine and evaluate staffing needs and update the cadre force structures under the new delivery model  DHS estimated that this effort would be completed by June 28, 2019  This action, if fully implemented, should address the intent of the recommendation  With regard to our second recommendation, that FEMA establish controls for tracking FAC-Trax capabilities to ensure the new information system meets users’ needs, DHS stated that Recovery program managers will update the FAC-Trax Requirements Management Plan and the FAC-Trax Release Plan to ensure the tracking and traceability of FAC-Trax functional and operational requirements  DHS estimated that this effort would be completed by January 31, 2018  This action, if fully implemented, should address the intent of the recommendation  With regard to our third recommendation, that FEMA establish systems testing criteria to assess the development of FAC-Trax; and define the roles and responsibilities, and sequence and schedule for system integration, DHS stated that Recovery program managers will update the FAC-Trax System Integration Plan to include integration with the Deployment Tracking System, Enterprise Data Warehouse, Preliminary Damage Assessment interface, and State Grants Management system interface  DHS estimated that this effort would be completed by June 29, 2018  This action, if fully implemented, should address the intent of the recommendation  With regard to our fourth recommendation, that FEMA implement procedures to standardize planning for addressing PA hazard mitigation at the JFO, DHS stated that PA will update current process documents or develop new documents to better incorporate mitigation into the operational planning phase of the new delivery model  DHS estimated that this effort would be completed by July 31, 2018  This action, if fully implemented, should address the intent of the recommendation  With regard to our fifth recommendation, that PA coordinate with FIMA to develop performance measures and associated objectives for the new delivery model that better align with FEMA’s strategic goals for hazard mitigation in the recovery process, DHS stated that PA will reconvene the PA-Mitigation working group to develop and refine PA related hazard mitigation performance measures  DHS estimated that this effort would be completed by June 29, 2018  This action, if fully implemented, should address the intent of the recommendation  We are sending copies of this report to the Secretary of Homeland Security and interested congressional committees  If you or your staff have any questions about this report, please contact me at (404) 679-1875 or CurrieC@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report  GAO staff who made key contributions to this report are listed in appendix IV  Appendix II: Assessment of Information Technology Management Controls for the FEMA Applicant Case Tracker (FAC-Trax) Table 2 shows details on the Federal Emergency Management Agency (FEMA) Public Assistance (PA) program office’s implementation of key practices across four information technology (IT) management control areas for its new information system, the FEMA Applicant Case Tracker (FAC-Trax)  PA developed FAC-Trax as a web-based project tracking and case management system to supplement the Emergency Management Mission Integrated Environment (EMMIE) and help resolve long-standing information sharing challenges  To determine the extent to which the FAC-Trax program office implemented IT management controls, we reviewed documentation from the FAC-Trax program and compared it to key management best practices, including the Software Engineering Institute’s Capability Maturity Model® Integration for Acquisition and Development, the Project Management Institute’s Guide to the Project Management Body of Knowledge (PMBOK® Guide), and the Institute of Electrical and Electronics Engineers’ Standard for Software and System Test Documentation  We assessed the program as having fully implemented a practice if the agency provided evidence that it fully addressed the practice; partially implemented if the agency provided evidence that it addressed some, but not all, portions of the practice; and not implemented if the agency did not provide any evidence that it addressed the practice  Table 2  Public Assistance (PA) Program Office’s Implementation of Key Information Technology Management Controls for FAC-Trax PA program officials developed an acquisition plan for FAC-Trax identifying the capabilities the system is intended to deliver, the acquisition approach, and acquisition objectives  Additionally, program officials developed a capability development plan outlining a strategy for the program to obtain approval to acquire FAC-Trax  Lastly, program officials developed a systems engineering plan describing the program’s scope and its framework for using an Agile development approach, as well as a deployment, support, and maintenance plan for FAC-Trax  PA program officials developed an acquisition program baseline detailing FAC-Trax’s cost parameters and a life-cycle cost estimate for the system  As of May 2017, the life- cycle cost estimate for FAC-Trax through fiscal year (FY) 2023 is approximately $19 3 million  PA program officials updated the life-cycle cost estimate for FYs 2016 and 2017 after price negotiations with the FAC-Trax contractor, and will continue to update the estimate as annual budgets are approved, according to the Integrated Logistic Support Plan  The contracting officer’s representative for FAC-Trax performs a cost review at the end of each month, according to program officials  Furthermore, the contractor’s weekly status report includes information on the number of hours worked and the percent of contract value spent  Program officials also review program costs with Office of Response and Recovery, PA, Office of the Chief Information Officer (OCIO), and other program office stakeholders during a weekly program review  PA program officials developed an acquisition program baseline detailing FAC-Trax’s schedule parameters, as well as an integrated master schedule for the system  The integrated master schedule identifies tasks, major milestones, and task dependencies  The PA program manager reviews and updates the integrated master schedule on a weekly basis  Program officials also review FAC-Trax’s schedule with Office of Response and Recovery, PA, OCIO, and other program office stakeholders during a weekly program review  PA program officials identified the knowledge and skills needed to carry out the program in FAC-Trax contract documentation and the capability development plan  Specifically, program officials included an attachment to the FAC-Trax contract listing the required labor categories and corresponding functional position descriptions  Program officials also described the role, position type, minimum grade, and minimum certification for required personnel resources for the acquisition, development, and implementation of FAC-Trax  PA program officials developed, reviewed, and maintained project planning documents and obtained commitment from relevant stakeholders  For example, program officials reviewed and updated the integrated master schedule and costs on a weekly and monthly basis, respectively  Further, program officials reviewed the status of project elements, such as the schedule, quality and technical issues, stakeholders, staffing, cost, and risks, with Office of Response and Recovery, PA, OCIO, and other program office stakeholders during a weekly program review  PA program officials also established tactical, functional, and stakeholder groups, as well as an Integrated Product Team to support and oversee the development of FAC-Trax  FEMA’s Recovery Technology Programs Division (RTPD) has a division-level risk management plan that serves as guidance for all Recovery systems, including FAC- Trax  Program officials identified key risks that could negatively affect FAC-Trax work efforts in RTPD’s “risk register”—an online site used to track risks, issues, and mitigating actions for the division and each program office  Program officials also identified five technical, cost, and schedule risks in the FAC-Trax acquisition plan  Program officials included one of these risks in the risk register, while the remaining four were managed outside of the register  As of May 2017, program officials had identified 13 risks in its risk register—four open and nine closed  The four open risks were (1) limited subject matter expert engagement during requirements development, (2) vacancies in program management office support positions, (3) unresolved service level agreement support and funding issues, and (4) the loss of the authority to operate due to a Trusted Internet Connection that is not compliant with Department of Homeland Security security policy  Program officials evaluated and categorized the identified risks based on the probability of occurrence and scope, schedule, and cost impacts  These four points of measurement are used to calculate an overall risk score  The risk score helps program officials determine a risk’s risk rating—low, medium, or high  For example, program officials reported that two of its open risks have a “medium” risk rating—meaning the risk has the potential to slightly impact project cost, schedule, or performance  In addition, program officials detailed the risk category, probability, and impact for the five risks identified in the FAC-Trax acquisition plan  Program officials developed risk mitigation and contingency plans for each risk in the risk register  For example, program officials planned to mitigate the open risk concerning subject matter expert engagement, by identifying and engaging with appropriate subject matter experts through requirements development workshops scheduled in advance of the sprint they are to support, and monitoring the development of user stories to identify any issues that may cause delays  In addition, program officials described the risk management plan and responsible officials for the five risks identified in the FAC-Trax acquisition plan  PA program officials review and update program risks during a monthly program meeting  Program officials also review program risks with Office of Response and Recovery, PA, OCIO, and other program office stakeholders during a weekly program review  Furthermore, the FAC-Trax contractor provides a weekly status update which includes a section on identified risks  Program officials established re-evaluation dates and recorded updates, including any actions taken, for each risk in the risk register  In addition, program officials were able to provide updates on the four risks identified in the FAC-Trax acquisition plan and managed outside of the register  According to PA officials, these risks were addressed and closed by the approval of program planning documents, such as the mission needs statement, concept of operations, and operational requirements document, following the solutions engineering review, which demonstrates the readiness of the program to proceed with the procurement, in September 2016  Program officials established a requirements management plan outlining how it captures, assesses, and plans for FAC-Trax enhancements, and established a change control process to review, prioritize, and verify user requests for changes to the system and feedback  As of May 2017, the PA program office received 734 change requests related to FAC-Trax, of which program officials completed 420 changes and planned to address an additional 277 entries  PA program officials also facilitated workshops to gather requirements for specific user groups and obtained additional requirements for FAC-Trax through customer feedback on a temporary technology tool— an Access database referred to as the Public Assistance Recovery Information System—used to support an early stage of the new model implementation  Further, program officials developed a functional requirements document outlining the high-level functional and operational requirements for FAC-Trax  PA program officials developed a concept of operations for FAC-Trax detailing operating concepts and scenarios for each phase of the PA preaward process  Program officials also detailed the workflow, phases, business functions, and data inputs and outputs for the re-engineered PA process in FAC-Trax’s functional requirements document  In March 2017, program officials developed a standard template to describe the process, tasks, and data inputs and outputs for specific system capabilities  As part of the change control process, PA program officials meet three times a week to discuss and prioritize change requests  Specifically, program officials review submissions to the change control form to ensure completeness, validate impacts and root cause, and research details for incoming requests  PA program officials also follow up with users to understand and verify requirements  In March 2017, program officials developed a standard template to capture acceptance criteria for specific requirements  However, PA program officials do not track system enhancements back to the high-level requirements identified in FAC-Trax’s operational and functional requirements documentation and performance work statement  PA program officials identified system requirements and constraints in the FAC-Trax concept of operations and functional and operational requirements documents  Further, through its change control process, program officials collect suggestions, issues, and feedback on FAC-Trax and system enhancements from stakeholders, identify risks for change requests, and balance prioritized requirements and estimated level of efforts with projected costs prior to each sprint  In March 2017, program officials developed a standard template to analyze and document the urgency and need for specific requirements  PA program officials and the FAC-Trax contractor established a testing and evaluation plan for the system, developed acceptance criteria for user stories, and obtained feedback from users during and after testing  The testing process concludes with user acceptance testing (UAT)  If a change request fails during UAT or a new requirement is discovered during development, the PA program will capture the failed request or new requirement in the product backlog for implementation in a future product release  Key practices Systems testing and integration Developing test plans and test cases PA program officials and the FAC-Trax contractor tested and evaluated the system during development  The FAC-Trax test plan identifies the method and strategy to perform the testing, including the necessary tasks, testing parameters, and the roles and responsibilities of the individuals responsible for testing  However, program officials did not develop system testing criteria to evaluate FAC-Trax  A key feature of Agile software development is the “definition of done”—a set of clear, comprehensive, and objective criteria, that the government should use to evaluate software after each iteration of development  PA program officials developed a systems integration plan in June 2017 that identifies potential integration of FAC-Trax and four FEMA systems, including the Emergency Management Mission Integrated Environment  Specifically, the plan includes data requirements and standards; descriptions of the four systems FEMA plans to integrate with FAC-Trax and the proposed relationship for each connection; and security and access management requirements  In addition, program officials included a description of how integration problems are to be documented and resolved in FAC-Trax development and test plans  However, the systems integration plan does not define roles and responsibilities of all participants for system integration activities or establish a sequence and schedule for every integration step for the four FEMA systems  ● Fully implemented: The agency provided evidence that it fully addressed this practice  ◐ Partially implemented: The agency provided evidence that it addressed some, but not all, portions of this practice  ◌ Not implemented: The agency did not provide any evidence that it addressed this practice  In addition to the contact named above, Chris Keisling (Assistant Director), Amanda R  Parker (Analyst-in-Charge), Mathew Bader, Allison Bawden, Anthony Bova, Eric Hauswirth, Susan Hsu, Rianna Jansen, Justin Jaynes, Tracey King, Matthew T  Lowney, Heidi Nielson, Claire Peachey, Brenda Rabinowitz, Ryan Siegel, Martin Skorczynski, Niti Tandon, Walter K  Vance, James T  Williams, and Eric Winter made key contributions to this report 

ground_truth: Protecting federal employees and facilities from security threats is of critical importance. Most federal agencies are generally responsible for their facilities and have physical security programs to do so. GAO was asked to examine how federal agencies assess facilities' security risks. This report examines: (1) how selected agencies' assessment methodologies align with the ISC's risk management standard for identifying necessary countermeasures and (2) what management challenges, if any, selected agencies reported facing in conducting physical security assessments and monitoring the results. GAO selected four agencies—CBP, FAA, ARS, and the Forest Service—based on their large number of facilities and compared each agency's assessment methodology to the ISC Standard; analyzed facility assessment schedules and results from 2010 through 2016; and interviewed security officials. GAO also visited 13 facilities from these four agencies, selected based on geographical dispersion and their high risk level. None of the four agencies GAO reviewed—U.S. Customs and Border Protection (CBP), the Federal Aviation Administration (FAA), the Agricultural Research Service (ARS), and the Forest Service—used security assessment methodologies that fully aligned with the Interagency Security Committee's Risk Management Process for Federal Facilities standard (the ISC Standard). This standard requires that methodologies used to identify necessary facility countermeasures—such as fences and closed-circuit televisions—must: 1. Consider all of the undesirable events (i.e., arson and vandalism) identified by the ISC Standard as possible risks to facilities. 2. Assess three factors—threats, vulnerabilities, and consequences—for each of these events and use these three factors to measure risk. All four agencies used methodologies that included some ISC requirements when conducting assessments. CBP and FAA assessed vulnerabilities but not threats and consequences. ARS and the Forest Service assessed threats, vulnerabilities, and consequences, but did not use these factors to measure risk. In addition, the agencies considered many, but not all 33 undesirable events related to physical security as possible risks to their facilities. Agencies are taking steps to improve their methodologies. For example, ARS and the Forest Service now use a methodology that measures risk and plan to incorporate the methodology into policy. Although CBP and FAA have updated their methodologies, their policies do not require methodologies that fully align with the ISC standard. As a result, these agencies miss the opportunity for a more informed assessment of the risk to their facilities. All four agencies reported facing management challenges in conducting physical security assessments or monitoring assessment results. Specifically, CBP, ARS, and the Forest Service have not met the ISC's required time frame of every 3 years for conducting assessments. For example, security specialists have not conducted required reassessments of two ARS and one Forest Service higher-level facilities. While these three agencies have plans to address backlogs, CBP's plan does not balance conducting risk assessments with other competing security priorities, such as updating its policy manual, and ARS and the Forest Service lack a means to monitor completion of future assessments. Furthermore, CBP, ARS, and the Forest Service did not have the data or information systems to monitor assessment schedules or the status of countermeasures at facilities, and their policies did not specify such data requirements. For example, ARS and the Forest Service do not collect and analyze security-related data, such as countermeasures' implementation. FAA does not routinely monitor the performance of its physical security program. Without improved monitoring, agencies are not well equipped to prioritize their highest security needs, may leave facilities' vulnerabilities unaddressed, and may not take corrective actions to meet physical security program objectives. This is a public version of a sensitive report that GAO issued in August 2017. Information that the agencies under review deemed sensitive has been omitted. GAO recommends: (1) that CBP and FAA update policies to require the use of methodologies fully aligned with the ISC Standard; (2) that CBP revise its plan to eliminate the assessments backlog; and (3) that all four agencies improve monitoring of their physical security programs. All four agencies agreed with the respective recommendations.