prompt_front: You are given two reports by the government agency.



Question: Write a one-page summary of the report 1.



Report 2: NASA’s Commercial Crew Program is a multi-phased effort that began in 2010  Across the phases, NASA has engaged several companies, using both agreements and contract vehicles to develop and demonstrate crew transportation capabilities  As the program has passed through these phases, NASA has generally narrowed down the number of participants  The early phases of the program were under Space Act agreements, which is what NASA calls the agreements entered into pursuant to its other transaction authority  These types of agreements are generally not subject to the Federal Acquisition Regulation (FAR) and allow the government and its contractors greater flexibility in many areas  Under these Space Act agreements, NASA relied on the commercial companies to propose specifics related to their crew transportation systems, including their design, the capabilities they would provide, and the level of private investment  In these phases, NASA provided technical support and determined whether the contractors met certain technical milestones  In most cases, NASA also provided funding  For the final two phases of the program, NASA awarded FAR-based contracts  By using FAR-based contracts, NASA gained the ability to procure missions to the ISS, while continuing to provide technical expertise and funding to the contractors  NASA levied two sets of requirements on the contractors: the ISS program requirements, which must be met by all spacecraft visiting the ISS whether they carry cargo or crew; and the Commercial Crew Program requirements, which have a focus on system capabilities and safety rather than design  The program also established a verification closure notice process, in which the contractors submit data to NASA to verify they have met all the requirements to be certified  This certification must occur before contractors are allowed to fly initial crewed missions to the ISS  In September 2014, NASA awarded firm-fixed-price contracts to Boeing and SpaceX, valued at up to $4 2 billion and $2 6 billion, respectively, for the Commercial Crew Transportation Capability phase  Under a firm- fixed-price contract, the contractor must perform a specified amount of work for the price negotiated by the contractor and government  This is in contrast to a cost-reimbursement contract, in which the government generally agrees to pay the contractor’s allowable costs regardless of whether work is completed  During this phase, the contractors will complete development of crew transportation systems  Boeing’s spacecraft—CST-100 Starliner—is composed of a crew module and a service module  The crew module will carry the crew and cargo  It also includes communication systems, docking mechanisms, and return systems for Earth landing  The service module provides propulsion on-orbit and in abort scenarios as well as radiators for thermal control  SpaceX’s spacecraft—Dragon 2—is composed of a capsule, which we refer to as the crew module, and a trunk, which we refer to as the support module  The crew module is composed of a pressure section and a service section  This module will carry the crew and cargo  It also includes avionics, docking mechanisms, and return systems for a water landing  The support module includes solar arrays for on-orbit power and guidance fins for escape abort scenarios  Figure 1 shows the spacecraft and launch vehicles for Boeing and SpaceX’s crew transportation systems  The Commercial Crew Transportation Capability phase contracts include three types of services: Contract Line Item 001 encompasses the firm-fixed-price design, development, test, and evaluation work needed to support NASA’s certification of the contractor’s spacecraft, launch vehicle, and ground support systems  Contract Line Item 002 covers any service missions that NASA orders to transport astronauts to and from the ISS  Under this indefinite-delivery, indefinite-quantity line item, NASA has ordered six post-certification missions from each contractor  Each service mission is its own firm-fixed-price task order  NASA must certify the contractors’ systems before they can fly these missions  Contract Line Item 003 is an indefinite-delivery, indefinite-quantity line item for any special studies, tests, and analyses that NASA may request  These tasks do not include any work necessary to accomplish the requirements under contract line item 001 and 002  As of April 2018, NASA had funded studies worth approximately $30 million to Boeing, including approximately $27 million for additional testing of the parachute system  NASA had funded studies worth approximately $44 million to SpaceX, including approximately $34 million for additional testing of the parachute system  For each contractor, the maximum value of this contract line item is $150 million  NASA has made changes to the contracts that have increased their value  While the contracts are fixed-price, their values can increase if NASA adds work or otherwise changes requirements, among other means  As of April 2018, NASA requirement changes had increased the value of contract line item 001 for Boeing by approximately $191 million and for SpaceX by approximately $91 million  NASA divided the certification work under contract line item 001 into two acceptance events: the design certification review and the certification milestone  An acceptance event occurs when NASA approves a contractor’s designs and acknowledges that the contractor’s work is complete and meets the requirements of the contract  The first acceptance event—the design certification review—verifies the contractor’s crew transportation system’s capability to safely approach, dock, mate, and depart from the ISS, among other requirements  After the contractor has successfully completed all of its flight tests, as well as various other activities, the second acceptance event—the certification milestone—determines whether the crew transportation system meets the Commercial Crew Program’s requirements  Following this contract milestone is an agency certification review, which authorizes the use of a contractor’s system to transport NASA crew to and from the ISS  Figure 2 shows a notional path leading up to the agency certification review  The Commercial Crew Program’s certification plan outlines how the program will incrementally review required deliverables leading up to, and supporting, the agency certification review  For each review, the plan describes the information that the contractor and the program will present  At the agency certification review, which is chaired by the Associate Administrator of the Human Exploration and Operations Mission Directorate, the agency will review the program’s formal recommendation to certify the contractor’s crew transportation system  Program officials said that their goal is to develop and review certification evidence incrementally in order to reduce the risk that issues will be identified during the agency certification review  In our February 2017 report, we evaluated the progress made by the two contractors on the Commercial Crew Program and found the following: Both of the Commercial Crew Program’s contractors had made progress developing their crew transportation systems, but both also had aggressive development schedules that were increasingly under pressure  We reported that both Boeing and SpaceX had determined that they would not be able to meet their original 2017 certification dates, and both expected certification to be delayed until 2018  We found that the schedule pressures were amplified by NASA’s need to provide a viable crew transportation option because its contract with Russia’s space agency was to provide crew transportation to the ISS for six astronauts through 2018 with rescue and return through late spring 2019  Purchasing additional seats from Russia involves a contracting process that typically takes 3 years  Without a viable contingency option for ensuring uninterrupted access to the ISS in the event of further Commercial Crew delays, we concluded that NASA was at risk of not being able to maximize the return on its multibillion dollar investment in the space station  The Commercial Crew Program was using mechanisms laid out in its contracts to gain a high level of visibility into the contractors’ crew transportation systems, but maintaining that level of visibility through certification could add schedule pressures  We noted that, for example, due to NASA’s acquisition strategy for this program, its personnel were less involved in the testing, launching, and operation of the crew transportation system  While the program developed productive working relationships with both contractors, obtaining the level of visibility that the program required had also taken more time than the program or contractors had anticipated  Ultimately, we noted that the program had the responsibility for ensuring the safety of U S  astronauts, and its contracts gave it deference to determine the level of visibility required to do so  We concluded that the program office could face difficult choices moving forward about how to maintain the level of visibility it feels it needs without adding to the program’s schedule pressures  In order to ensure that the United States had continued access to the ISS if the Commercial Crew Program’s contractors experienced additional schedule delays, we recommended in our February 2017 report that the NASA Administrator develop a contingency plan for maintaining a presence on the ISS beyond 2018, including options to purchase additional Russian Soyuz seats, and report to Congress on the results  NASA concurred with this recommendation, and in February 2017, NASA executed a contract modification that purchased two seats and included an option to purchase three additional crewmember seats from Boeing on the Russian Soyuz vehicle  These seats represent a contingency plan for U S  access to the ISS through 2019  In April 2017, NASA informed the Congress of this action  Boeing and SpaceX continue to make progress developing their crew transportation systems, but both contractors have further delayed the certification milestone to early 2019  These changes have occurred as the contractors continue to work to aggressive schedules, and they have had to delay key events regularly  Further delays are likely as the Commercial Crew Program’s schedule risk analysis shows that the certification milestone is likely to further slip  In addition, as of mid-June 2018, NASA officials told us that these dates may change soon but that both contractors have not yet provided official updates to their schedules to NASA  NASA has not fully shared information with Congress regarding the risks of future schedule delays for the contractors and, as a result, Congress lacks insight into when the contractors will be certified  Also, there may be a gap in access to the ISS if the Commercial Crew Program experiences additional delays  While NASA has begun to discuss potential options, it currently does not have a contingency plan for how to ensure an uninterrupted presence on the ISS beyond 2019  Boeing and SpaceX have continued to make progress finalizing their designs and building hardware as they work toward their certification milestones  The contractors are manufacturing test articles to demonstrate system performance and flight spacecraft to support the uncrewed and crewed flight tests, which are expected to demonstrate the ability to meet contract requirements  As table 1 shows, these test articles and spacecraft vary in levels of completion  Some are built and undergoing testing while others are starting the manufacturing phase  Should any issues arise during integration and test or the flight tests, the contractors may have to complete rework on the spacecraft already under construction  While both contractors are making progress, the Commercial Crew Program is tracking risks that each contractor has to address through testing and other means as they work towards the certification milestone  As we have previously reported, these types of risks are inherent in NASA’s major acquisitions, which are highly complex, specialized, and often pushing the state of the art in space technology, but they could also delay the contractors’ progress if issues arise during testing  The Commercial Crew Program’s top programmatic risks identified for Boeing include challenges related to its abort system performance, parachutes, and launch vehicle  Abort System: Boeing is addressing a risk that its abort system, which it needs for human spaceflight certification, may not meet the program’s requirement to have sufficient control of the vehicle through an abort  In some abort scenarios, Boeing has found that the spacecraft may tumble, which could pose a threat to the crew’s safety  To validate the effectiveness of its abort system, Boeing has conducted extensive wind tunnel testing and plans to complete a pad abort test in July 2018  Parachute System: Boeing is also addressing a risk that during descent, a portion of the spacecraft’s forward heat shield may re- contact the spacecraft after it is jettisoned and damage the parachute system  Boeing’s analysis indicates the risk exists only if one of two parachutes that pull the forward heat shield away from the spacecraft does not deploy as expected, and that potential re-contact is non- detrimental  However, NASA’s independent analysis indicates that this may occur even if both parachutes deploy as expected  If the program determines this risk is unacceptable, Boeing would need to redesign the parachute system, which the program estimates could result in at least a 6-month delay  Launch Vehicle Data: One of the program’s top programmatic and safety concerns is that it may not have enough information from Boeing’s launch vehicle provider, United Launch Alliance, to assess whether the Atlas V launch vehicle prevents or controls cracking that could lead to catastrophic failures  NASA estimates that unfinished work in this area could take Boeing and the United Launch Alliance until the fourth quarter of 2018 to complete  Additionally, the first stage of the Atlas V is powered by the Russian built RD-180 engine, and, according to program and Boeing officials, access to its data is highly restricted by agreements between the U S  and Russian governments  Since our last report, the Commercial Crew Program has lowered the risk that certification of the launch vehicle might not occur by negotiating steps to access necessary data, but work is still ongoing  The Commercial Crew Program’s top programmatic risks identified for SpaceX are in part related to ongoing design and development efforts related to its launch vehicle design, the Falcon 9 Block 5  Composite Overwrap Pressure Vessel: This Block 5 design includes SpaceX’s redesign of the composite overwrap pressure vessel, which is intended to contain a gas under high pressure  SpaceX officials stated the newly designed vessel aims to eliminate risks identified in the older design, which was involved in an anomaly that caused a mishap in September 2016  SpaceX plans to qualify the updated design for flight prior to the uncrewed flight test design certification review  Engine Turbine Cracking: The Block 5 design also includes design changes to address cracks in the turbine of its engine identified during development testing  NASA program officials told us that they had informed SpaceX that the cracks were an unacceptable risk for human spaceflight  SpaceX officials told us that they have made design changes to this Block 5 upgrade that did not result in any cracking during initial testing  However, this risk will not be closed until SpaceX successfully completes qualification testing in accordance with NASA’s standards without any cracks  As of March 2018, SpaceX had not yet completed this testing  Propellant Loading Procedures: Both the program and a NASA advisory group have raised SpaceX’s plan to fuel the launch vehicle after the astronauts are on board the spacecraft to be a potential safety risk  In the May 2018 meeting minutes, however, the Aerospace Safety Advisory Panel stated that with appropriate controls in place, this approach could be a viable option for the program to consider  SpaceX’s perspective is that this operation may be a lower risk to the crew because it reduces the crew exposure time while the launch vehicle is being loaded with propellant  To better understand the propellant loading procedures, the program and SpaceX agreed to demonstrate the loading process five times from the launch site in the final crew configuration prior to the crewed flight test  The five events include the uncrewed flight test and the in-flight abort test  Therefore, delays to those events would lead to delays to the agreed upon demonstrations, which could in turn delay the crewed flight test and certification milestone  Both contractors have notified NASA that their certification milestones have slipped to January 2019 for Boeing and February 2019 for SpaceX, but the Commercial Crew Program’s schedule risk analysis indicates more delays are likely  This analysis identifies a range for each contractor, with an earliest and latest possible completion date, as well as an average  In April 2018, the program’s schedule risk analysis found there was zero percent chance that either contractor would achieve its current proposed certification milestone  The analysis’s average certification date was December 2019 for Boeing and January 2020 for SpaceX  Figure 3 shows the original Boeing and SpaceX contract schedules and the current proposed schedule for five key events in each contract, as well as NASA’s schedule risk analysis for the certification milestone  Each month, the program updates its schedule risk analysis based on the contractors’ internal schedules as well as program officials’ perspectives and insight into specific technical risks  The Commercial Crew Program manager told us that differences between the contractors’ proposed schedules and the program’s schedule risk analysis include: The contractors are aggressive and use their schedule dates to motivate their teams, while NASA adds additional schedule margin for testing  Both contractors assume an efficiency factor in getting to the crewed flight test that NASA does not factor into its analysis  The program manager also told us that the program meets with each contractor monthly to discuss schedules and everyone agrees to the relationships between events in the schedule even if they disagree on the length of time required to complete events  The program manager added, however, that she relies on her prior experience to estimate schedule time frames as opposed to relying on the contractors’ schedules, which are often optimistic  Our analysis also shows that the contractors often delay their schedules  Both contractors have repeatedly stated that their schedules are aggressive and have set ambitious—rather than realistic—dates, only to frequently delay them  Since the current contracts were awarded in 2014, the Commercial Crew Program has held 13 quarterly reviews for each contractor  For the five key events identified above, Boeing has reported a delay at 7 of those quarterly reviews and SpaceX has reported a delay at 9 of them  In mid-June 2018, NASA officials told us that the dates for these key events may change soon  The information presented in Figure 3 above is based on first quarter calendar year 2018 data  NASA officials stated both contractors have not yet officially communicated new schedule dates to NASA as of the second quarter calendar year 2018  We found that both contractors have updated schedules that indicate delays are forthcoming for at least one key event, but NASA officials told us they lack confidence in those dates until they are officially communicated to NASA by the contractors  As a result, NASA is managing a multibillion dollar program without confidence in its schedule information as it approaches several big events, including uncrewed and crewed flight tests  The risk of future delays in the contractors’ schedules is critical information that NASA has not fully shared with Congress  Moreover, NASA has not yet developed a contingency plan to address the potential gaps that these delays could have on U S  access to the ISS after 2019  Specifically, in the Explanatory Statement accompanying the fiscal year 2018 Consolidated and Further Continuing Appropriations Act, the House Appropriations Committee stated its expectation that NASA report quarterly to the Senate and House Committees on Appropriations on the status of the Commercial Crew Program contracts  Previously, members of Congress had asked for this information in order to ensure that Congress had adequate insight into this program  While NASA includes both contractors’ proposed schedules in its quarterly report to Congress, NASA does not include the results of its own schedule risk analysis  Given the frequency with which the contractors delay key events in their schedules, the program’s schedule risk assessment provides valuable insight into potential delays that NASA currently is not providing to Congress  In addition, as previously mentioned, NASA executed a contract modification that purchased two seats and included an option to purchase three additional crew member seats through Boeing for an undisclosed value and reported this action to Congress in April 2017  Ultimately, the option was exercised, and NASA purchased a total of five seats on four different Soyuz flights  Boeing obtained these seats through a separate settlement with the Russian firm RSC Energia, which manufactures the Soyuz  These seats were intended to serve as a contingency plan based on schedule information available at that time  However, subsequent delays, as well as the risk of future delays as discussed above, indicate that this contingency plan will likely no longer be sufficient  The earliest and latest possible completion dates for certification in NASA’s April 2018 schedule risk analysis indicate it is possible that neither contractor would be ready before August 2020, leaving a potential gap in access of at least 9 months  We calculated the potential gap based on the contractor certification milestone dates, but there could be some additional time required between that review and the first post-certification service mission to the ISS  As seen in figure 4, if the contractors can maintain their current proposed schedules for their respective certification milestones, a gap in access to the ISS is not expected  However, there would be a gap in access to the ISS if neither contractor has its certification milestone before November 2019, which is when NASA expects the final Russian Soyuz seat for a U S  astronaut to return  Senior NASA officials told us that sustaining a U S  presence on the ISS is essential to maintain and operate integral systems, without which the ISS cannot function  Given the importance of maintaining a U S  presence on the ISS, NASA officials have stated they are working on options to address the potential gap in access  However, officials told us that planning for contingencies is difficult given the extensive international negotiations required for some options  Obtaining additional Soyuz seats seems unlikely, as the process for manufacturing the spacecraft and contracting for those seats typically takes 3 years—meaning additional seats would not be available before 2021  As a result, according to NASA’s Associate Administrator for Human Exploration and Operations, the options NASA is considering include: Refine the remaining Soyuz launch schedule to allow for a return in January 2020, as opposed to November 2019  This would provide 2 additional months of access to the ISS before the commercial crew flights need to start  Use the crewed flight tests as operational flights to transport U S  astronauts to and from the ISS  In March 2018, NASA modified Boeing’s contract to allow NASA to add a third crew member and extend the length of the flight test, if NASA chooses to do so  This would have limited usefulness, however, in filling a potential gap in access to the ISS if the schedule for Boeing’s crewed flight test slips past the return date for the last Soyuz flight and SpaceX also continues to experience delays  NASA’s Associate Administrator for Human Exploration and Operations stated that he is “brainstorming” other options to ensure access to the ISS but does not have a formal plan  While options are not unlimited and decisions have to be made within the context of the current geopolitical environment, Congress stated in the NASA Authorization Act of 2005 that it is U S  policy to possess the capability for human access to space on a continuous basis  In 2010, Congress further stated that one of the key objectives of the United States’ human spaceflight policy is to sustain the capability for long-duration presence in low-Earth orbit through full utilization of the ISS  If NASA does not develop options for ensuring access to the ISS in the event of further Commercial Crew delays, it will not be able to ensure that the U S  policy goal and objective for the ISS will be met  The Commercial Crew Program relies on several contractual mechanisms to assess safety throughout the certification process, and those mechanisms are in varying stages of completion  The program itself, its contractors, and two of NASA’s independent review organizations have raised concerns about the program’s ability to assess and evaluate all of the deliverables in a timely manner  In addition, one of the key safety requirements levied by the program is loss of crew, which captures the probability of death or disability to a crew member  NASA does not have a consistent approach for how to incorporate key inputs to assess this metric, which means the agency as a whole may not clearly capture or document its risk tolerance with respect to loss of crew  Further, the program’s chief safety and mission assurance officer is dual hatted to serve simultaneously in a programmatic position as well as the program’s safety technical authority  This approach creates an environment of competing interests because it relies on the same individual to manage technical and safety aspects on behalf of the program while also serving as the independent oversight of those same areas  The contractors are required to provide several key deliverables to the Commercial Crew Program, which inform the agency certification review and help NASA determine the level of risk it is accepting with respect to safety of each spacecraft  As described below, these deliverables are in varying stages of completion and the program itself, its contractors, and two of NASA’s independent review organizations have raised concerns about the program’s ability to assess and evaluate all of the deliverables in a timely manner  Certification Data Package  Among other things, the certification data package includes a list of seven system safety assessments  For example, the certification data package includes a fault tolerance assessment, which describes the system’s ability to sustain a certain number of undesired events, such as software or operational anomalies  A human error analysis—one of the seven assessments in the data package—evaluates human errors to minimize their negative effects on the system  Boeing held its uncrewed flight test design certification review in December 2017 and submitted its certification data package for NASA approval  Boeing plans three more updates to this data package prior to the final certification milestone  SpaceX has begun to submit data and plans to submit its final certification data package as part of its crewed flight test design certification review, which is scheduled for September 2018  According to the program’s certification review plan, program officials will review and approve the contractors’ certification data packages, which will be used to inform the agency certification review  Phased Safety Review Process  A three-phased safety review process informs the program’s quality assurance activities, and it is intended to ensure that the contractors have identified all safety-critical hazards and implemented associated controls prior to the first crewed flight test  In phase one, the contractors identified risks in their designs and developed reports on potential hazards, the controls they put in place to mitigate them, and explanations for how the controls will mitigate the hazards  In phase two, which is nearing completion, the program reviews and approves the contractors’ hazard reports and develops strategies to verify and validate that the controls are effective  For example, if a control requires that an item be waterproofed, verification and validation strategies could include inspections and tests to confirm that the item is waterproof  As of April 2018, the program had yet to complete this phase, having approved 97 percent of Boeing’s phase two reports and 72 percent of SpaceX’s phase two reports  In phase three, the contractors will conduct the verification activities and submit the hazard reports to the program for approval
Report 2:  See appendix III for a sample RVAR template  We analyzed 81 RVARs developed by the 10 FEMA regions from July 2012 through December 2016 and found that regions did not consistently obtain and document information on all elements related to each of the six regulatory factors in their RVARs  As shown in table 6, all 81 RVARs had at least some elements documented but not all for each of the IA regulatory factors  For example, for the IA concentration of damages regulatory factor, the six elements to be addressed include the number of homes destroyed, damaged or affected, damage concentration, and damage to critical facilities  While 44 of the 81 RVARs documented all of the six elements, 37 documented some but not all of the elements  Similarly, for the trauma regulatory factor, the four elements to be addressed include injuries, death, power outages, and disruption of community functions  While 30 of the 81 RVARs documented all of the four elements, 51 documented some but not all of the elements  For the insurance coverage factor, while five RVARs documented all of the elements, 73 RVARs documented some but not all of the elements  Elements under this factor include home ownership, insurance, and flood insurance, when applicable  None of the six regulatory factors were fully documented across all RVARs  See appendix IV for detailed information on the extent to which all of the elements of the six regulatory factors were documented in the RVARs from July 2012 through December 2016  FEMA headquarters officials acknowledged that information related to all the elements for each of the IA regulatory factors were missing from the RVARs  They stated that they had not collected all information on all factors because one factor may have more weight than another based on the specific incident that has occurred  However, they also indicated that they do not fully know and have not evaluated all of the reasons why a region may have omitted information on an element of a factor  FEMA headquarters officials agreed that having complete information on all elements of the regulatory factors in the RVARs would assist in their recommendation process  Standards for Internal Control in the Federal Government suggest that agencies should establish and operate monitoring activities to ensure that internal controls—such as the documentation of all of the elements of the IA regulatory factors FEMA regions considered—are effective, and to take corrective actions as appropriate  Because it is unclear why regions are not completely documenting all elements related to the current six regulatory factors, such an evaluation could help FEMA identify whether any corrective steps are needed  Doing so could help FEMA ensure it is achieving its stated goals in providing consistency in the evaluation process and in the types of factors it considers  Officials we interviewed in 9 of the 10 FEMA regions and state emergency management offices in all 11 states reported the positive relationship they maintain with each other as a strength in the IA declaration process  For example, both FEMA regional officials and state emergency management officials stated that they have a good working relationship and are in regular communication via telephone or in-person meetings with each other  Also, state emergency management officials we spoke to stated that whenever they are in need of assistance, they know they can reach out to FEMA regional officials for assistance  However, FEMA regional and state emergency management officials we spoke to also reported various challenges with the process  These include the subjective nature of the IA regulatory factors given the lack of eligibility thresholds, the lack of transparency in the decision-making process, and difficulty gathering information on IA regulatory factors  Subjective nature of the IA factors and lack of eligibility thresholds  Officials from 9 of 10 FEMA regions stated the subjective nature of the IA program is a challenge; and officials in 6 of the 10 regions also said they found the lack of eligibility thresholds a challenge  An official in one region stated that unlike FEMA’s Public Assistance program, which has minimum thresholds for eligibility, it is unclear when states should apply for IA funds  Under the Public Assistance program, for example, for states or tribes to qualify for assistance, they must demonstrate that they have sustained a minimum of $1 million in damages and the impact of damages must amount to $1 00 per capita in the state  An official in another region explained that although the subjectivity of the IA factors provides flexibility in determining the type of IA program needed, having some quantifiable criteria could help officials explain to states why their requests were denied or approved  Similarly, officials we interviewed in 7 of the 11 states said they found the subjective nature of the factors with no threshold to be a challenge  A state emergency management official in one state said this subjectivity makes it difficult to determine whether or not the state should make an IA request  A state emergency management official in another state reported that the subjectivity can cause the IA declaration process to be inconsistent, and it is not always clear how or why certain declarations were approved and others were not  Further, a state emergency management official in an additional state also pointed to the subjective nature of the factors with no threshold as a reason for not being able to provide a more detailed rationale behind a declaration denial  To illustrate this, table 7 shows how four states requested IA declarations related to the same tornado in 2012 and varied in what they reported across the six IA factors, such as the levels of damages incurred, special populations among their residents, and insurance coverage  Two of these four states—Kentucky and Indiana—received IA declarations and the other two—Ohio and Illinois—were denied  Lack of transparency  Another challenge reported by FEMA regional and state emergency management officials was the lack of transparency in how FEMA evaluates and provides a recommendation to the President on whether a declaration is warranted  For example, officials we interviewed in 4 of 10 regions indicated the lack of transparency as a challenge  A FEMA official in one region stated that the region would like more transparency regarding what FEMA headquarters recommends to the President and whether the President’s decision aligns with FEMA’s recommendation  State emergency management officials we interviewed in 10 of 11 states also reported that lack of transparency with the IA process is a challenge  For example, an emergency management official in one state said it is not clear how or if FEMA considers all of the factors  Also, an emergency management official in another state reported that it was unclear to him why his state’s declaration request was denied while the requests of other states with similar incidents were declared  Difficulty gathering information on IA regulatory factors  Officials in 4 of 10 FEMA regions reported difficulty gathering information, such as income or insurance coverage, as a challenge  An official in one region stated that it is difficult to obtain information related to IA factors from states  For example, the official said that calculating the concentration of damages is difficult absent technical guidance from FEMA headquarters, as the current guidance only accounts for the number of structure damage but not the impact of damage  Further, officials in two FEMA regions stated that states lack a dedicated IA official, making it difficult for state officials, who play multiple roles, to provide the necessary information related to the IA factors in their IA declaration request  Additionally, a state emergency management official in one state also reported that lack of staff resources in her state makes it difficult to verify all the local damage assessments prior to making a declaration request  Pursuant to the Sandy Recovery Improvement Act of 2013, in November 2015, FEMA issued a Notice of Proposed Rulemaking to revise the six current IA regulatory factors to the following proposed factors: state fiscal capacity and resource availability; uninsured home and personal property losses; disaster-impacted population profile; impact to community infrastructure; casualties; and disaster-related unemployment  According to FEMA headquarters officials, the revisions aim to provide more objective criteria, clarify the threshold for eligibility, and speed the declaration  The officials said the proposed rule also seeks to provide additional clarity and guidance for all the established factors  Table 8 shows FEMA’s description of current and proposed IA factors  FEMA received public comments from 14 states in the Federal Register during the comment period for the proposed rule and proposed guidance  The 14 states expressed concern about the proposed factor for state fiscal capacity and resource availability, including the reliability and relevance of data sources such as total taxable resources  These states expressed concern that the data collection necessary to meet the new requirements would fall upon them, adding to the cost burden of completing an IA disaster declaration request  They also explained that the use of total taxable resources and other similar data is not an effective way to assess a state’s current ability to provide resources following a disaster  Also, these states indicated that the data points such as total taxable resources and per capita personal income that would be used to evaluate state fiscal capacity are outdated and inaccurate and would be an inefficient way to evaluate a state’s true fiscal capacity to respond to a disaster  Regarding the other five proposed factors, several states in their comments raised questions about ambiguities in interpreting the factors or the feasibility and cost of gathering related data  For example, in regards to the factor on disaster impacted population, five states expressed concern that the data required for the disaster-impacted population factor would be a cost burden to the state or that the data would be inappropriate for evaluation  Additionally, two states said unemployment related to a disaster incident for the disaster-related unemployment factor would be hard to quantify in the first 30 days following a disaster  They stated that this was especially an issue given that states work to submit an IA disaster declaration request as soon as possible following a disaster  According to the Office of Management and Budget’s Office of Information and Regulatory Affairs website, the projected date for finalization of the proposed rule is September 2018; however, as of April 2018, FEMA officials stated that they were not certain whether that timeframe would be met  Until the proposed rule is finalized, we will not know the extent to which the various challenges FEMA regions and state officials raised in our interviews and in comments on the proposed rule will be addressed  FEMA has obligated over $8 6 billion nationwide in IA from calendar years 2008 through 2016, highlighting the importance of FEMA’s evaluation of states’ and tribes’ IA declaration requests  FEMA’s regional offices evaluate the request and make a regional recommendation through the Regional Administrator’s Validation and Recommendation, which documents information on all relevant IA regulatory factors  FEMA has developed the Regional Administrator’s Validation and Recommendation to ensure regions consistently obtain and document the information needed by FEMA to make a disaster declaration recommendation to the President based on the IA regulatory factors  However, FEMA’s regional offices do not consistently obtain and document information on all elements of the current IA regulatory factors  Because it is unclear why regions are not always documenting all of the elements related to these factors, evaluating the reasons why could help FEMA identify if any corrective steps are needed  Doing so could also help FEMA ensure it is meeting its stated goals in providing consistency in the evaluation process and in the types of factors it considers  We recommend that the Administrator of FEMA evaluate why regions are not completing the Regional Administrator’s Validation and Recommendations for each element of the current IA regulatory factors and take corrective steps, if necessary  We provided a draft of this report to DHS for its review and comment  DHS provided written comments, which are summarized below and reproduced in full in appendix V  DHS concurred with the recommendation and described planned actions to address it  In addition, DHS provided written technical comments, which we incorporated into the report as appropriate  DHS concurred with our recommendation that FEMA evaluate why regions are not completing the Regional Administrator’s Validation and Recommendations for each element of the IA regulatory factors and take corrective steps, if necessary  DHS stated that a FEMA working group consisting of headquarters stakeholders will draft survey questions for FEMA region officials to identify the common reasons why an element of an IA regulatory factor may not be addressed within a RVAR  According to DHS, the working group will also analyze, assess, and present the findings of the survey responses to FEMA senior leadership, and if needed, FEMA will develop and send a memorandum to the regions with additional guidance regarding the appropriate preparation of RVARs  DHS stated that the estimated completion date is in the fall of 2018  These actions, if implemented effectively, should address the intent of our recommendation  We will send copies of this report to the Secretary of Homeland Security, the FEMA Administrator, and the appropriate congressional committees  If you or your staff have any questions about this report, please contact me at (404) 679-1875 or curriec@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report  Key contributors to this report are listed in appendix VI  Table 9 provides the total number of Individual Assistance declaration requests made, declared, and denied, by region, state, and tribe for disaster declarations requested from calendar years 2008 through 2016  Table 10 provides Federal Emergency Management Agency’s (FEMA) Individual Assistance (IA) actual obligations for declarations made from calendar years 2008 through 2016 by state and type of disaster  As part of the Federal Emergency Management Agency’s (FEMA) declaration process, FEMA’s regional offices are to evaluate states’ or tribes’ declaration requests, including the IA declaration request, and make a recommendation called the Regional Administrator’s Validation and Recommendation (RVAR) and submit the RVAR to FEMA headquarters  In June 2012, FEMA headquarters issued a template for FEMA regional offices to use in developing the RVAR as identified in figure 3  Tables 11 through 16 provide information on each element of the 6 Individual Assistance (IA) regulatory factors documented in the Regional Administrator’s Validation and Recommendation (RVAR) from July 2012 through December 2016 by the Federal Emergency Management Agency region  In addition to the contact named above, Aditi Archer (Assistant Director), Su Jin Yon (Analyst-In-Charge), Hiwotte Amare, Eric Hauswirth, Susan Hsu, Jun S  (Joyce) Kang, Christopher Keisling, Heidi Nielson, Hadley Nobles, Anne Rhodes-Kline, and Jerome (Jerry) Sandau made significant contributions to this report 

ground_truth: Congress created several authorities that provide DOD research labs with ways to increase efficiency and foster innovation. Senate report 114-255 contained a provision for GAO to study governance models used by federal labs. This report evaluates DOD labs' use of authorities to foster innovation and efficiency. GAO selected four authorities that recent work on best practices for science and technology management and expedited defense lab hiring have shown to be the most crucial for supporting innovation; administered a survey to 44 lab directors to gain insight into their use of the authorities; interviewed key lab officials and contractors; and reviewed relevant policies and guidance. Congress has provided the Department of Defense's (DOD) research labs with several authorities to enhance management and operations. Four authorities that GAO examined provide lab directors with greater ability to make their own decisions regarding the funding of projects, hiring, lab management, and purchasing of equipment or services. 1. Laboratory initiated research authority. This authority, as implemented, provides labs with a means to fund new science and technology projects that they consider a priority. Labs may use a percentage of all funds available to the lab and are permitted to charge customers of the lab a percentage fee of the costs for activities performed by the lab for the customer. 2. Direct hire authority. This authority enables labs to compete with private industry for high-quality talent. For example, it provides for streamlined hiring of applicants with relevant advanced degrees, or students enrolled in science, technology, engineering, and mathematics programs. 3. Laboratory enhancement pilot program authority. This authority generally allows lab directors to propose alternative methods that might lead to more effective lab management, and waive certain policies or procedures that might affect implementation of these methods. 4. Micro-purchase authority. This authority raises the threshold for small purchases for DOD research lab activities from $3,500 to $10,000 to facilitate acquisitions. While labs have used these authorities, their use has sometimes been limited, particularly with the laboratory initiated research authority. DOD lab directors at Air Force, Navy, and Army cited several obstacles that impede wider use of that authority, specifically: Air Force: Financial management officials at the Air Force stated that the service's accounting system does not currently have an automated capability to transfer the allowable percentage fee of costs to a central account at the Air Force Research Laboratory. This lack of capability, officials noted, creates a significant administrative burden related to charging these fees. Navy: In fiscal year 2017, Navy labs invested $7.3 million in lab infrastructure projects, compared to $32.9 million and $53.7 million at the Air Force and Army, respectively. Navy lab officials told us that they were restricted in their use of infrastructure funds available under the laboratory initiated research authority due to a lack of clear guidance as to whether and how to use this authority within the Capital Investment Program of the Navy Working Capital Fund. Army: The Army requires its labs to use a similar percentage of funds from two sources: (1) what it refers to as directly appropriated funds and (2) funds labs charge for customer activities. Some Army lab directors reported assessing a lower rate on customer funds than allowed so as not to drive customers away. The labs then generally charge a lower than desired rate on their directly appropriated funds, which further constrains the total funding available to them. GAO is making three recommendations to enhance DOD's use of laboratory initiated research authority, including that the Air Force assess potential accounting system improvements, the Navy clarify how labs can use the authority for infrastructure improvements, and the Army assess its policy to determine whether changes are needed to remove disincentives for labs to use the authority. DOD concurred with the recommendations.

prompt_front: You are given two reports by the government agency.



Question: Write a one-page summary of the report 2.

Report 1: The defense lab enterprise consists of 63 labs, warfare centers, and engineering centers across the Departments of the Army, Navy, and Air Force, as shown in Figure 1 below  About 50,000 federally employed scientists and engineers work at these defense labs to support warfighter needs and develop transformative capabilities  Defense labs are managed and operated within the military service chain of command  DOD budgets for technology and product development activities under its research, development, test, and evaluation budget, which DOD groups into seven budget activity categories for its annual budget estimates  Air Force and Army labs rely on appropriated funding provided from the service—often referred to as mission funding—or from customers (or some combination thereof)  Customers, such as program offices, provide funding to defense labs for technology development activities and related research  The Air Force and Army funding structure is in contrast to Navy research and development activities, which operate under the Navy Working Capital Fund—a revolving fund that finances Department of the Navy activities on a reimbursable basis  Under this funding model, the Navy employs a Capital Investment Program to obtain capital assets, including minor military construction projects for labs  The program provides the framework for planning, coordinating, and controlling Navy working capital funds and expenditures to obtain capital assets  Figure 2 illustrates the varying funding models used by the military service labs  In addition to its labs, DOD sponsors other entities to provide for its technology development needs  Specifically, these include: FFRDCs are operated by universities, other not-for-profit or nonprofit organizations, or private firms under long-term contracts and provide special research and development services that generally cannot be readily satisfied by government personnel or private contractors  For example, the Massachusetts Institute of Technology Lincoln Laboratory develops key radar and electronic warfare technologies for integrated air and missile defense systems  In addition, the Software Engineering Institute operated by Carnegie Mellon University provides cybersecurity solutions for defense entities  While DOD sponsors 10 FFRDCs in total, it designates 3 FFRDCs as research and development labs, which maintain long-term competencies in key technology areas  In addition to these, DOD sponsors 2 systems engineering and integration FFRDCs and 5 studies and analysis FFRDCs  UARCs provide specialized research and development services similar to FFRDCs and also operate under long-term contracts  However, unlike FFRDCs, DOD requires that UARCs be affiliated with a university  Generally, UARCs may not compete against industry in response to a competitive Request for Proposals for development or production that involves engineering expertise  DOD currently sponsors 13 UARCs  Key DOD offices provide oversight to the defense labs: The Under Secretary of Defense for Research and Engineering (USD(R&E))—the principal advisor to the Secretary of Defense for research, engineering, and technology development activities and programs—serves as DOD’s chief technology officer  The powers and duties of this office include establishing policies and providing oversight for DOD’s research, engineering, and technology development activities  The Defense Laboratories Office—within the Office of the USD(R&E)—supports DOD’s research and engineering mission by helping to ensure comprehensive department-level insight into the activities and capabilities of the defense labs  This office carries out a range of core functions related to the defense labs, including analysis of capabilities, alignment of activities, and advocacy  Congress has granted authorities that address hiring, infrastructure, and technology transition challenges to defense labs since 1995  These authorities provide defense lab directors with certain flexibilities within the established legal framework to manage their operations  While Congress has provided a number of authorities, in this report we focus on four authorities that our prior work on best practices in science and technology management and expedited lab hiring has shown are, or have the potential to be, the most crucial for supporting innovation within DOD labs  Laboratory Initiated Research Authority  This authority provides lab directors with the means to fund some of the research projects that the lab will pursue  The authority provided in Section 219 of the Duncan Hunter National Defense Authorization Act for Fiscal Year 2009, as implemented, provides lab directors with a means to fund projects they consider to be a priority in four allowable categories: (1) basic and applied research, (2) technology transition, (3) workforce development, and (4) revitalization, recapitalization, or repair or minor construction of lab infrastructure  These projects include those not specifically tied to defined requirements, outside of the normal 2-year budget planning process  The authority directs the Secretary of Defense to establish mechanisms under which lab directors may use an amount of funds equal to not less than 2 percent and not more than 4 percent of all funds available to the defense lab for projects under the four allowable categories  Further, lab directors are permitted to obtain additional funding by charging customers a fixed percentage fee that may not exceed 4 percent of costs  Direct Hire Authorities  These authorities provide lab directors with a streamlined and accelerated hiring process  Congress has enacted four types of direct hire authorities since 2008, which help labs compete with private industry and academia for high-quality scientific, engineering, and technical talent  Specific types of direct hire authorities include hiring: (1) candidates with advanced degrees; (2) candidates with bachelor’s degrees; (3) veterans; and (4) students currently enrolled in graduate or undergraduate science, technology, engineering, and mathematics (STEM) programs , Laboratory Enhancement Pilot Program  This authority provides methods for effective lab management operations  Section 233 of the National Defense Authorization Act for Fiscal Year 2017 established a pilot program for lab directors to propose alternative and innovative methods that might lead to more effectively managing labs, and authorized lab directors to waive any regulation, restriction, requirement, guidance, policy, procedure, or departmental instruction that would affect implementation of these methods, unless such implementation would be prohibited by a provision of an existing statute or common law  Micro-purchase Authority  This authority facilitates the purchasing process for labs  The FAR states a preference for government agencies to purchase and pay for micro-purchases of supplies or services using the government-wide commercial purchase card up to and at the micro-purchase threshold, but micro-purchases may be conducted using any of the simplified acquisition methods  This facilitates the ability of lab officials to quickly and easily acquire needed items for their activities and reduce the administrative costs associated with such small purchases  While the FAR micro-purchase was generally $3,500 during our review, Congress increased it to $10,000 for activities of the science and technology reinvention labs in Section 217 of the National Defense Authorization Act for Fiscal Year 2017  As we found in June 2018, the federal government spends approximately $137 billion annually government-wide on research and development (R&D) to help further agencies’ missions, including at federal labs  From fiscal years 2015 to 2017, DOD, Energy, and NASA represented three of the top four federal agencies with the highest annual federal R&D spending, accounting for about 66 percent of total federal R&D spending on average, as shown in Figure 3  While the labs primarily support the agencies that directly fund them, DOD, Energy, and NASA research entities also collaborate extensively to support activities of shared interest  For example, DOD and NASA research centers have collaborated to develop hypersonic vehicle capabilities  Further, Energy’s national labs help provide critical national security capabilities for DOD and support NASA’s deep space mission radioisotope requirements  In 2017, Energy reported performing about $2 6 billion of work per year from fiscal years 2011 through 2015 for other federal agencies and other customers, including DOD  Most defense labs have used the selected authorities since 2008, but their use has sometimes been limited for a variety of reasons  According to lab directors, this is because of DOD legal and policy restrictions and stakeholder concerns  For example: Use of the laboratory initiated research authority was limited by DOD’s military construction funding and financial management policies  Use of the direct hire authority was limited, in part, by personnel- related delays, security clearance challenges, and military hiring restrictions  Use of the laboratory enhancement pilot program was limited by stakeholder uncertainty about how to use this authority effectively  Use of the increased micro-purchase authority was limited by stakeholder concerns about the authority’s potential effect on small businesses  We found that most defense labs have used the laboratory initiated research authority  Twenty-three of 31 of respondents to our survey— about 74 percent—reported obligating funds under this authority  However, we found that most labs are not using the full 4 percent of all funds available to each lab, or charging customers the full fixed percentage fee of 4 percent of costs, as allowed by law  Specifically, we found that, as of September 2018: Navy labs reported charging customers a percentage fee of about 2 percent of costs as of fiscal year 2018  Prior to this, Navy labs only charged a 1 percent fixed fee on these costs  Because Navy labs are working capital funded organizations, they can use payments from customers for goods delivered or services performed  Army labs reported using between 2 and 3 percent of all funds available to the lab for projects under the four allowable categories and charging customers a fixed fee of between zero and 3 percent of costs to fund such activities  Only the Air Force Research Laboratory reported using the full 4 percent of all funds available to the lab  According to agency officials, the lab is using 3 percent of all funds available to the lab and is allowing individual technology directorates the option to use the additional 1 percent of funds available  In fiscal year 2018, three of the lab’s nine technology directorates chose to use this additional 1 percent  However, the lab has not charged customers a fixed percentage fee on their costs at all  As figure 4 shows, in fiscal year 2017, the aggregate fixed percentage fee charged by labs in each of the military departments totaled under the full 4 percent allowed by law for each funding source  Decisions to charge lower percentages are decisions to forego additional potential funding, although agencies have various reasons why this can happen, as we will discuss later  In total, DOD reported that this authority provided almost $300 million to labs in fiscal year 2017 and funded more than 1,750 projects across the four allowable categories, as Figure 5 illustrates  We previously found, in June 2017, that the laboratory initiated research authority provides defense lab directors with limited flexibility to initiate science and technology projects  These projects include those that are not road mapped or tied to defined requirements outside of the normal 2- year budget planning process, and are focused on both near- and long- term needs  For this review, defense lab officials we interviewed stated that the laboratory initiated research authority enables their scientists and researchers to pursue projects not necessarily tied to requirements and provides necessary funds for workforce development and lab infrastructure projects  Further, as shown in Figure 6, lab directors we surveyed generally view the authority as both fostering innovation and increasing efficiency across the four allowable categories on which funds can be used  In accordance with the one of the statutory purposes for the use of the funds, lab directors have developed new, innovative technologies using this authority  For example, DOD reported that: In fiscal year 2017, the Naval Surface Warfare Center, Crane Division, developed and fielded a solution to an urgent requirement for defeating small unmanned aerial vehicles that attack Navy assets or surveil naval activities  The center delivered this technology to the warfighter in May 2017 just 7 weeks after the Navy submitted the requirement  The Army Research Laboratory used the authority to fund a project that eventually developed a material that could increase the speed and lower the power needs of future generations of computer chips, thereby supporting Army networks  The Navy invested more than $700 thousand in laboratory initiated research authority funds to commission a Ballast Water Research Lab at Naval Surface Warfare Center, Carderock Division  Through the use of this new facility, engineers will be able to study ways to treat ballast water to prevent introduction of non- native aquatic species into a new environment that can be disastrous for the marine life that already inhabit that environment, and ensure that the Navy is able to meet various port regulations around the world for its ships  The Air Force Research Laboratory invested funds in fiscal year 2017 to renovate an existing facility to provide high performance computing capability to aid the rapid development of “game-changing” technologies and weapon systems  Officials at the Army’s Space and Missile Defense Command Technical Center noted they used the laboratory initiated research authority for the first time in fiscal year 2018 because the current executive director, who assumed the position in 2017, prioritized implementing this authority  Most of the Center’s planned investments are focused on workforce development and laboratory infrastructure projects; officials cited a high energy laser technology lab as one of the projects being supported by the revitalization, recapitalization, or minor military construction portion of this authority  Although the majority of defense labs reported using the laboratory initiated research authority, interviews we conducted throughout our review, along with other DOD reports, identified certain obstacles that have, at times, impeded wider usage  DOD-wide military construction funding restrictions  DOD restrictions limit the amount of laboratory initiated research authority funds that labs can spend on lab infrastructure  DOD’s limit is $6 million for the revitalization and recapitalization projects that can be funded under the laboratory initiated research authority  Lab officials stated that this amount is often insufficient to construct advanced lab facilities  Air Force Research Laboratory officials indicated that it is nearly impossible to construct lab facilities for less than $6 million  Officials at the Army’s Aviation and Missile Research, Development and Engineering Center echoed this sentiment and noted that they have primarily used funds to renovate existing buildings rather than fund new lab facility construction  In January 2017, the Defense Science Board identified lab infrastructure challenges, including that the average age of research and development facilities was nearly 50 years  Further, the Board reported that the labs are usually not successful in competing against broader service needs for military construction funds  Air Force does not charge customers a fixed percentage fee of costs  The Air Force Research Laboratory reported that it is not charging customers the allowable fixed percentage fee of costs to fund science and technology activities because it does not have a mechanism in place to do so  Air Force Research Laboratory officials estimated the lab would collect approximately $3 million a year if the lab charged customer activities the maximum allowable fee (4 percent)  Air Force financial management officials stated that the service’s accounting system does not currently have an automated capability to transfer the allowable percentage fee of costs to a central account at the Air Force Research Laboratory  This lack of capability, officials noted, creates a significant administrative burden for charging these fees  The officials stated that they have not yet estimated the cost to add an automated capability  Although it is possible for the Air Force Research Laboratory to charge customer work orders manually—outside of the Air Force’s accounting system—officials with the Office of the Assistant Secretary of the Air Force for Financial Management and Comptroller perceive that the resources (time and people) required to manage such a process would be cost prohibitive  However, according to these officials, the Air Force has not assessed the costs required to improve the accounting system to do so, nor has it identified the potential benefits any improvements would provide  Federal internal control standards state that changes in condition affecting an entity and its environment often require changes to the entity’s internal control system, as existing controls may not be effective for meeting objectives (or addressing risks) under changed conditions  Further, these standards state that any internal control deficiencies require further evaluation and remediation by management  By not assessing the potential costs and benefits related to the options for collecting these allowable fees, the Air Force could be missing out on a potential source of funding to support its needs  DOD lacks clear guidance on how the Navy should use the initiated research authority for some infrastructure investments within the Capital Investment Program  In our review of DOD documentation, we found that, among the military departments, Navy labs funded recapitalization and revitalization projects using the laboratory initiated research authority the least  As recently as early 2017, a DOD-commissioned study found that defense labs face substantial infrastructure deficiencies that it has not yet identified funding to address  In fiscal year 2017, Navy labs invested $7 3 million in lab recapitalization projects, compared to $32 9 million and $53 7 million at the Air Force and Army, respectively  Navy lab officials told us that their ability to fund lab recapitalization and revitalization projects using funds available under the laboratory initiated research authority is limited because they have not been provided with clear guidance as to whether and how to use the laboratory initiated research authority within the Capital Investment Program of the Navy Working Capital Fund  Some Navy lab officials stated that they have found ways to use the initiated research authority for certain infrastructure investments  These officials stated that they used authority outside of the Capital Investment Program of the Navy Working Capital Fund, for instance, for projects below applicable thresholds because using the authority within the Program creates a bureaucratic and financial burden for them  For example, officials at two separate warfare centers—Naval Surface Warfare Center, Crane Division, and the Naval Air Warfare Center, Aircraft Division, noted that they did not expend funds in either fiscal year 2016 or fiscal year 2017 for recapitalization and revitalization projects  Both cited the Capital Investment Program as a significant barrier to their desired use of the laboratory initiated research authority  Officials from the Office of Budget, within the Office of the Assistant Secretary of the Navy for Financial Management and Comptroller agreed that, to date, clarifying guidance on the use of the laboratory initiated research authority within the Capital Investment Program has not been issued, effectively limiting the extent to which the labs can use it for infrastructure needs  According to these officials, the Office of the Secretary of Defense (OSD) Comptroller—in coordination with the Office of Financial Policy and Systems within the Office of the Assistant Secretary of the Navy for Financial Management and Comptroller—is responsible for developing the clarifying guidance their office has sought  This persistent lack of guidance on whether or how Navy labs should use the laboratory initiated research authority within the context of the Capital Investment Program presents an opportunity cost  Namely, the Navy’s labs have missed out on, and continue to miss, opportunities to invest in needed improvements to its aging lab infrastructure  The Army requires its laboratories to apply similar percentages to what is refers to as “Army direct appropriations” and “customer funds ” The Army requires that the percentage fee applied to direct appropriations not vary from the percentage fee applied to customer funds by more than 1 percent  The Army implemented this policy to maximize the laboratory initiated research authority’s effect on its 17 laboratories  However, the Office of the USD(R&E) reported in March 2018 that the policy was having a significant limiting effect on the breadth and scope of activities executed under this authority  Similarly, we found that the policy may, in practice, create a disincentive for Army lab directors to use the authority  In their responses to our survey, Army lab directors, representing key capability areas, acknowledged their concern about the percentage fee they assessed on customer funds affecting their ability to increase or maintain their customer bases  Further, some Army lab directors reported assessing a lower percentage fee on customer funds than allowed, which could help retain customers that might otherwise be driven away with higher assessed fees to carry out activities  As a result, these labs generally are setting a lower percentage fee on their directly appropriated funds, thereby lowering the overall laboratory initiated research funding available to them  Nonetheless, the Army has not assessed its policy to determine whether changes are needed to eliminate these disincentives  Continuing to operate without such an assessment could result in Army labs using the laboratory initiated research authority to fund fewer self-initiated projects—with the downstream effect that fewer new technologies for warfighters are available  The Navy applies a consistent fixed percentage fee of costs across its labs  Within the Navy, senior leadership has set the fixed percentage fee of costs the labs charge on customer funds at 2 percent  A senior Navy science and technology official stated that Navy leadership set a uniform fixed percentage fee to charge to customer activities across the Navy lab enterprise, in part, to ensure the labs were not inadvertently competing against one another for customer funds  For example, without a uniform rate, a Navy warfare center could offer a lower fee to entice a customer to use it rather than another center  The use of a fixed percentage fee facilitates program offices selecting warfare centers on the basis of best available match in capabilities  On the other hand, the Navy’s fixed 2 percentage fee of costs does limit—by half, as compared to the maximum 4 percent allowable—the amount of fees that Navy labs can collect  Consequently, several Navy lab directors told us that they would like to have the ability to increase the fixed percentage fee of costs above the Navy’s 2 percent to provide their labs with additional resources they said they need for innovation-related investments  Among the lab directors that responded to our survey, 30 of 31 replied that their lab had used at least one of the four types of direct hire authorities previously discussed since fiscal year 2014  Officials view direct hire authority as allowing the labs to compete with private industry for qualified applicants  Lab directors reported they generally believe that each type of direct hire is extremely or very useful for fostering innovation and increasing efficiency, as shown in Figure 7  Selected Officials’ Testimony on the Value of Direct Hire Authority: The U S  Army Engineer Research and Development Center “was able to meet this important goal [of annually hiring more than 160 new researchers] in large part because of its direct hiring authorities, which save time, effort, and costs, and allow the organization to more effectively hire the best and brightest minds available ” – Dr  Jeffrey P  Holland, Past Director, U S  Army Engineer Research and Development Center, in testimony before the Senate Committee on Armed Services (Emerging Threats and Capabilities Subcommittee), May 3, 2017  “The Air Force’s ability to recruit, retain, and develop the STEM workforce is vital toward building the future Air Force; Congress has been greatly supportive of these efforts…the addition of direct hire for candidates has been extremely useful in hiring qualified scientists and engineers in less than half the time of traditional hiring methods ” – Jeffrey Stanley, Air Force Deputy Assistant Secretary— Science, Technology and Engineering in testimony before the House Committee on Armed Services (Emerging Threats and Capabilities Subcommittee), March 14, 2018  Although participation in the laboratory enhancement pilot program is open to the DOD labs—and 19 of the 31 lab directors, or 61 percent, that responded to our survey reported they were participating—to date, only the Navy has formally established a pilot program for its labs  The Army and Air Force have not yet used this relatively new authority  A senior Navy science and technology official told us the Navy took important steps to facilitate the implementation of that service’s pilot program  According to the Navy official: The Office of the Deputy Assistant Secretary of the Navy for Research, Development, Test and Evaluation led the effort across the Navy labs, compiling—from each lab’s submission—a single list of proposals to forward to Navy leadership that would apply to all participating Navy labs  The Navy pursued a three-phased approach with its pilot program, with Phase 1 primarily focused on contracting and acquisition policy- related matters  Senior Navy research and development officials perceived these matters as being the easiest from which to obtain buy-in from Navy policy officials and attorneys, as well as Navy leadership  Phase 2 will include proposals related to Information Technology systems for research and development networks, while Phase 3 will most likely address personnel issues  Navy research and development officials deferred proposals— including information technology network enhancements—that might require extensive discussions with policy officials and attorneys stakeholders across the Navy  These proposals were pushed back to allow time for those stakeholders to see how the pilot program was being implemented and executed by the labs  None of the Army and Air Force labs has yet established a laboratory enhancement pilot program  Consistent with Army policy, the Medical Research and Materiel Command and the Space and Missile Defense Command Technical Center submitted proposals; however, they have yet to establish a pilot program  The Army’s Research, Development and Engineering Command, with input from its subordinate labs and engineering centers, developed a list of lab enhancement proposals but, as of September 2018, had yet to formally submit these final proposals to Army leadership for approval  These include initiatives in business operations, contracting, finance, information technology, and personnel management  A senior Army science and technology acknowledged that organizations across the military department have concerns about providing the labs with too much autonomy to use this new authority  Air Force Research Laboratory officials said they previously submitted a list of approximately 30 proposals to the Defense Laboratories Office in September 2017, but ultimately pulled back those requests because of stakeholder concerns within the Air Force  Specifically, officials with the Office of the Deputy Assistant Secretary of the Air Force for Science, Technology, and Engineering stated that the Air Force Materiel Command, to which the lab is a subordinate organization, had not seen the proposals before they were submitted  In addition, these officials identified concerns about how various stakeholders throughout the Air Force—such as those from financial management and personnel—would react to these proposals  These proposals could potentially sidestep the stakeholders’ oversight function of related lab activities  A senior Air Force Research Laboratory official stated that the lab re-submitted its proposals to the Air Force Materiel Command and that Air Force leadership was still reviewing them at the time of this report  Twenty-six of 31 labs directors—84 percent—reported having used the $10,000 micro-purchase threshold authority granted by Congress in 2016  However, we found that contracting and small business management officials’ concerns with this authority have created implementation challenges at some defense labs  For instance, a senior Navy official indicated that multiple stakeholders from across the Navy—including its Office of Small Business Programs—raised concerns about the authority’s potential impact on small businesses as micro-purchasing allows defense labs to bypass small business set asides  Several labs reported similar stakeholder concerns that prevented implementation of the micro-purchase threshold increase  At the same time, however, lab officials we interviewed expressed the view that the increased threshold will be beneficial, consistent with their opinions about the laboratory enhancement pilot program  For example, officials at the Naval Research Laboratory stated that increasing the threshold to $10,000 allows their scientists and engineers to directly purchase necessary equipment and materials through simplified procedures  They identified examples of projects that had been delayed by as much as several months because scientists and engineers used other than simplified acquisition procedures to purchase a relatively inexpensive piece of equipment, such as a specialized microscope, because the cost was above the previous threshold of $3,500  Similarly, the Army’s Armament Research, Development and Engineering Center reported that the micro-purchase threshold increase enables the lab to use simplified acquisition procedures for more items  As a result, they noted that the new authority increases efficiency by reducing contracting time and cost for those additional items  The Navy’s Space and Naval Warfare Systems Center Atlantic similarly reported that requirements, which were previously procured using other than simplified acquisition procedures, took up to 60 to 90 days to procure, while it took as little as 3 to 4 days under this new authority, which enabled its scientists and engineers to purchase materials needed for critical, time sensitive projects  However, lab officials acknowledged that the $10,000 micro-purchase threshold authority—like the laboratory enhancement pilot program—is too new to fully understand how it will increase efficiency and foster innovation over the long term  DOD sponsors several research centers, which are governed through noncompetitive agreements, including contracts  These centers provide the department with access to scientific experts employed by universities and other non-profit organizations  Scientists employed by these external to DOD research centers—specifically, three lab FFRDCs and 13 UARCs—execute DOD-funded science and technology development projects in emerging technical areas  DOD staff oversee these centers using routine oversight of funded research tasks and comprehensive reviews, which help DOD determine whether the centers’ funding should continue  DOD and research center officials told us that their ability to authorize work at the FFRDCs that DOD sponsors is limited by legislative restrictions on the staffing levels at these centers, as well as by infrastructure modernization challenges they face  DOD sponsors three research and development FFRDC labs that were established under noncompetitive procedures  Two of the three lab FFRDCs are operated by universities and one is operated by a nonprofit company  DOD also has contracts with 13 UARCs that fulfill a similar scientific role as the lab FFRDCs, while also differing from them in other respects  These differences are described in more detail in table 1  DOD’s contractor-operated research centers received about $1 3 billion annually in DOD funding in fiscal year 2016 and fiscal year 2017, according to DOD data  The two largest research and development FFRDCs, the Lincoln Laboratory and the Software Engineering Institute, received about 67 percent of total research center funding from DOD in 2017  UARCs received an average of $27 million in DOD funding, which was a 15 percent decrease from 2016  Research centers may also receive work and funding from other federal departments and private companies after obtaining DOD sponsor approval  Appendix II provides an overview of DOD FFRDC and UARC funding in fiscal years 2016 and 2017  DOD Sponsorship and Contract Awards: We reported in 2014 that FFRDCs in the federal government are defined through the sponsoring agreement between the agency and the contractor retained to operate the FFRDC  A written agreement of sponsorship between the government and the FFRDC must be prepared when the FFRDC is established, which may be included in a contract between the government and the FFRDC, or in another legal instrument under which an FFRDC accomplishes effort, or it may be in a separate written agreement  Historically, DOD sponsors retain contractors for many years or decades as FFRDC operators  We found that research centers undertake DOD-sponsored projects and, in some limited instances, scientific projects initiated by centers that are overseen by DOD staff  Individual sponsors enter into noncompetitive contracts with FFRDCs and UARCs  DOD uses noncompetitive contracts to establish or maintain an essential engineering, research, or development capability to be provided by an educational or other nonprofit institution or a federally funded research and development center  Scientific Project Funding: We found that project sponsors provide funding to existing contracts  For example, the government issues orders for requirements under Lincoln Laboratory’s indefinite delivery indefinite quantity base contract as funding sponsors approve new projects  Individual project sponsors, along with the primary sponsor, oversee how project funds are spent by the centers  Project sponsors decide whether they will continue to work with these entities based on perceived performance success  This effectively provides an incentive for FFRDCs and UARCS to perform successfully  This work and review cycle is described in Figure 9 below  FFRDCs and UARCs also partner with DOD government-operated labs to plan and execute technology development projects  For example, according to Navy officials, Naval Surface Warfare Center, Carderock Division collaborated with Navy-sponsored UARCs, such as Penn State’s Applied Research Laboratory, to help develop Navy submarine propeller and propulsion designs  Self-initiated Projects: Research center officials said that DOD provides some research centers with limited funds to self-initiate innovative projects  This funding helps the centers ensure that development projects are not limited to just satisfying near-term DOD requirements  Instead, future generations of DOD technologies can be funded  For example, officials at Johns Hopkins University Applied Physics Laboratory proactively conducted work on advanced naval defense technologies in response to similar technology development in adversary countries  Although Navy sponsors did not fund this initial work, they subsequently provided funding in this area after Hopkins’ research identified a risk reduction strategy for the Navy, according to the Johns Hopkins officials  This allowed the UARC to move relatively quickly on a new science and technology project idea  DOD uses 13 UARCs and three lab FFRDCs to obtain direct access to scientific expertise in emerging technical areas, supplementing research conducted at DOD’s government-owned and operated labs  These research centers provide DOD with additional scientific capabilities and the ability to expand quickly into new technical fields  Hiring Scientific Personnel: Although FFRDCs are largely federally funded, they are generally operated, managed, and administered by either a university or consortium of universities, other not-for-profit or nonprofit organization, or an industrial firm, as an autonomous organization or as an identifiable separate operating unit of a parent organization  The contractor operating the FFRDC exercises primary control over its FFRDC’s business concerns, such as personnel policies and compensation  DOD-funded research centers have flexibility in hiring scientists that leverage a parent institution’s expertise in emerging scientific fields  For example, leadership officials at the Army Institute for Soldier Nanotechnologies UARC at MIT and the Software Engineering Institute FFRDC at Carnegie-Melon University noted that projects they have conducted for DOD have benefitted from university experts in fields such as dark matter physics and artificial intelligence  Personnel Compensation: Research center officials we spoke with noted that their workforce policies permit them to flexibly hire, fire, and compensate staff as needed  Although employee salaries are established separately from the government schedule, they are approved by the government  Further, officials noted that university centers typically offer salaries in line with the labor market, but do not attempt to compete on a salary basis with relatively high, unaffordable private sector company salaries  Instead, they compete on the basis of other factors, such as offering scientists the opportunity to work for a prestigious university conducting science and technology research  Research Center Infrastructure: As with personnel matters, research centers have discretion to manage infrastructure in accordance with the policies and procedures of their parent institutions  While one center, Lincoln Laboratory, is located on government property, others primarily reside on property owned or leased by their parent institutions  According to agency officials, DOD contributes funding for the use and repair of these facilities through their contracts with research centers  Officials noted that Lincoln Laboratory uses military construction funding to pay for new buildings as it is located on government property  Trusted Advisor Role: FFRDCs and UARCs function as trusted advisors for the government and operate in the public interest with objectivity and independence  FFRDCs are independent, private-sector, non-profit organization units required to be free from personal or organizational conflicts of interest, as the FFRDCs answer to the government customer  As a result, DOD’s lab FFRDCs perform tasks that are closely associated with the performance of inherently governmental functions and have access to sensitive and proprietary data  Research center officials noted challenges limiting their work providing scientific expertise to DOD  FFRDCs are also limited in executing infrastructure investments  Limitation on Available Work Hours: DOD FFRDCs are limited by an annual ceiling set by Congress on the amount of staff years of technical effort (STE) that may be funded for defense FFRDCs  We previously found in October 2008 these limits were imposed in response to concerns that DOD was inefficiently using its FFRDCs  We found that the STE workload limitation aimed to ensure that FFRDC work was appropriate and limited resources were being used for DOD’s highest priorities  As a result, Software Engineering Institute officials said they decline many DOD programs’ requests for assistance due to the annual work hour limitation  Further, officials at the Office of the Secretary of Defense’s Studies and FFRDC Management Office reported that this limit significantly constrains the use of DOD’s FFRDCs and that DOD customer demand for their services is significantly greater than the annual STE limit  OSD officials indicated that FFRDC related work must be deferred to later years when these limits are reached, since there are no other legally compliant alternatives capable of fulfilling these requirements  Infrastructure: FFRDC officials we interviewed identified infrastructure challenges—including aging facilities and equipment—as hindering their research and development efforts  For example, many buildings at the Massachusetts Institute of Technology (MIT) Lincoln Laboratory are over 60 years old; MIT considers over half of them to be in substandard condition  According to an MIT official, these facilities, located on government property, were not structurally designed for modern research and have relatively poor vibration isolation, resulting in inefficient workarounds or work that could not be performed  Officials from the Defense Laboratories Office noted that the MIT Lincoln Laboratory is unique among DOD’s FFRDCs in that it is operated on government- owned property  A 2013 study, conducted on behalf of the White House Office of Science and Technology Policy, found that lab infrastructure project funding proposals must compete with hospitals, barracks, runways, and roads and, therefore, tend to be lower on the priority list for military construction funding  A 2017 Defense Science Board report and DOD officials we spoke with indicated this continues to be true  While contract research centers have significant flexibility to execute infrastructure work, they are still affected by limited availability of military construction funding  Officials at another center noted that in some instances, DOD sponsors have been unable or slow to provide required secure facilities and equipment within needed time frames  Delays of this nature can affect the research centers’ ability to deliver the technologies or related services needed by DOD  The Department of Energy (Energy) primarily relies on contractor- operated FFRDCs to operate its labs, while the majority of NASA labs and centers are government-operated  Energy’s national labs form the core of the agency’s scientific work and mission  This is in contrast to DOD-funded labs, which constitute a relatively small aspect of DOD’s overall mission  We have previously found that Energy’s labs can use funding for minor infrastructure improvements  NASA centers can also approve and fund certain facility projects, in accordance with NASA policies, and they have encountered significant challenges with aging infrastructure  Also, in some cases, energy and space research centers have significant challenges with hiring replacement staff and competing with private sector employers for staff  Energy’s labs can hire scientific personnel with the flexibility of private companies, while NASA centers were previously provided hiring flexibilities by Congress in 2004 to facilitate staff hiring  While Energy and NASA’s research entities follow their specific governance models, there are broad characteristics common across these agencies as well as DOD  Table 2 illustrates that while research centers are largely government-owned, the government is not always the operator  As we have reported, Department of Energy national labs are primarily operated by for-profit, non-profit and university FFRDC contractors using management and operating contracts, which are competed on a limited basis  Energy’s funding sponsors and headquarters officials are required to reevaluate FFRDC performance in increments not to exceed 5 years by federal acquisition regulations, which inform future decisions to renew the agreement  In 1990, we designated Energy’s contract management—including both contract administration and project management—a high-risk area because of Energy’s inadequate management and oversight of contractors, leaving the department vulnerable to fraud, waste, abuse, and mismanagement  In 2009, we subsequently narrowed the focus of Energy’s high-risk designation to the National Nuclear Security Administration and Office of Environmental Management, which together oversee four national labs  Further, in our 2017 High Risk report, we found that these two agencies had made progress in addressing our contract management concerns, but we identified continued problems with the agencies having sufficient capacity to mitigate contract and project management risks  Also, we found that they had demonstrated little progress in addressing contract management challenges, particularly in the area of financial management  The Department of Energy uses performance-based management and operating contracts, which have been subject to limited competition, with universities, non-profit companies and for-profit companies to operate the national labs on government-owned property  These contractor-operated FFRDCs provide the vast majority of Energy’s science and technology capacity, rather than supplementing the work of government-operated labs like DOD’s FFRDCs  Energy has depended on the expertise of private organizations to execute its science and technology work since the Manhattan Project produced the first atomic bomb during World War II  The Spallation Neutron Source is an experimental research facility at Oak Ridge National Laboratory—a government-owned contractor-operated laboratory  The Spallation Neutron Source includes the world’s most powerful pulsed-neutron sources and provides information about the structure and properties of materials that cannot be obtained by other means  The Spallation Neutron Source is a user facility whereby researchers from universities, national laboratories, and industry submit proposals, which are peer- reviewed and must compete for time at the user facility  The primary focus of each lab varies based on its expertise and facilities  Energy largely oversees its lab contractors through its headquarters program offices, which include the National Nuclear Security Administration, Office of Science, the Office of Fossil Energy, as well as co-located government field offices  Office of Science-sponsored labs primarily support scientific research for energy and physical sciences, while the National Nuclear Security Administration-sponsored (NNSA) labs primarily focus on nuclear weapons and related science and technologies  Energy also oversees its lab contractors’ activities through on-site Energy oversight offices that work alongside lab management at each FFRDC  Some labs specialize in earlier-phase science, while other labs work on later-phase nuclear weapons technologies in addition to earlier-phase science  As Figure 10 shows, these labs are spread across the United States  Energy has only one government-operated and government-owned lab, the National Energy Technology Laboratory  Key differences between Energy’s contractor-operated and government-operated governance models are described in table 3  Energy’s FFRDCs use their own personnel systems, which Energy officials stated provide more flexibility for hiring and retaining qualified staff  Management within these FFRDCs can move staff in or out of scientific areas more quickly than government labs can, thereby providing greater agility to meet Energy’s needs in emerging science areas  For example, Energy’s lab oversight staff at Oak Ridge National Laboratory told us that use of lab contractors’ human resources management systems allows for workforce flexibilities to meet Energy’s needs  While these contractors have leeway in managing their human resources systems, Energy’s headquarters maintains oversight—through its contracting officers—over employee compensation  Energy’s FFRDC contractors manage and operate nearly all of the department’s government-owned national lab facilities—including day-to- day management of government-controlled facilities and real property  Lab operators used funding to complete minor construction projects, which cost $10 million or less  This funding comes from a percentage of science and technology projects’ funding, requires local Energy oversight office approval, and has streamlined project management requirements  In contrast, major infrastructure upgrades are funded through relatively long and complex line-item funding processes, and projects over $50 million are subject to more rigorous project management requirements  Energy’s labs use a small portion of their funding to initiate discretionary projects for science and technologies that will benefit sponsors in the long-term by maintaining the scientific and technical vitality of the laboratories  To maintain and enhance lab expertise, the National Defense Authorization Act for Fiscal Year 1991 authorized Energy’s contractor-operated labs receiving funding for national security programs to use a percentage of lab funds to perform lab-directed R&D of a creative and innovative nature  The actual percentages allowed to be used for lab-directed R&D are subject to Energy’s approval  Energy’s entities sponsor most national lab projects based on their needs and lab expertise  Typically, earlier foundational science projects are funded through a process whereby funding sponsors issue calls for proposals to Energy’s national labs  Interested scientific teams at labs provide proposals to conduct these projects for sponsor consideration  Sponsors then assess proposals for scientific merit and decide which teams receive funding to execute their projects  NNSA provides funding for later-phase nuclear technology development projects to its labs after agreement is made regarding objectives and deliverables for specific projects, according to Lawrence Livermore National Laboratory officials  Despite their flexibilities with regard to hiring and infrastructure decisions compared to government operated labs, Energy’s lab leadership and government oversight officials noted human resource and facilities related challenges, such as: Sufficiently compensating staff located in high-cost of living areas  For example, the labor market of the San Francisco area, where several Department of Energy national labs are located, is highly competitive for employers  Commercial firms offer salaries and compensation that typically exceed those of government-funded, contractor-operated labs, although Energy’s contractors have more pay flexibility than is allowed for Energy’s government employees  Obtaining government clearances in a timely manner  Energy’s NNSA oversight officials and lab management staff, in particular, cited this challenge, which they stated has led to a backlog of people needing clearances  Government hiring freeze constraining overall hiring  Officials at Energy’s government-operated National Energy Technology Laboratory reported that as a result of a government hiring freeze, the lab has increasingly hired private contractor staff to the point that more than half of the total lab staff is now comprised of contractor employees  Major infrastructure challenges at Energy labs  Energy reported in July 2018 that over half of all national lab buildings are in either substandard or inadequate condition  The Energy Inspector General also identified infrastructure modernization as one of Energy’s top management challenges  This finding followed a mandated commission’s report in 2015 that facilities and infrastructure across Energy’s national lab network were hampered by high levels of deferred maintenance and excess facilities  The majority of NASA’s science and technology facilities are operated within the governance framework of government-operated research centers, similarly to most DOD labs  While government-operated, they have been granted additional legislative flexibilities for hiring employees beyond those normally available to government entities  NASA locates its science and technology staff at four government- operated research centers, one contractor-operated FFRDC, and at five NASA centers assisting space and space flight development  These centers and the Jet Propulsion Laboratory—NASA’s sole sponsored FFRDC—execute NASA’s research missions including technology development in exploration and aeronautics  The differences between these two governance approaches are described in Table 4  NASA also works with Johns Hopkins University Applied Physics Lab, a UARC, to develop major space flight missions  The NASA Glenn Research Center—a government- operated laboratory—is currently developing solar electric propulsion technologies intended to allow manned and unmanned spacecraft to be propelled far beyond earth orbit using solar power  This project is developing large, flexible, radiation-resistant solar arrays that can be unfurled to capture solar energy powering fuel-efficient electrostatic thrusters  Scientists expect a system-level flight test within the next decade to demonstrate key technologies supporting NASA’s Lunar Orbital Platform-Gateway project, a platform to mature necessary short- and long-duration deep space exploration capabilities  Headquarters, including funding sponsors providing oversight for their individual projects  Title 51, Chapters 201 and 203 of U S  Code and technology efforts with DOD and other organizations, including use of NASA lab and test facilities  Operated by university contractor having sole source contract  Not permitted to compete against industry, except for operation of an FFRDC  property (originally part of DOD)  5-year contract renewable to 10 years total  Headquarters, including funding sponsors providing oversight for their individual projects  Title 51, Chapters 201 and 203 of U S  Code; 10 U S C  § 2304 (c) (3)(B) ; Federal Acquisition Regulation § 35 017 based NASA oversight staff  sponsors seeking FFRDC assistance with NASA approval  “Lab” as used in this context refers to science and technology organizations equivalent to NASA Research Centers, DOD UARCs and FFRDCs  Mission leadership officials at NASA Headquarters—including the Associate Administrators for Aeronautics Research, Human Exploration, Science and Space Technology—oversee NASA’s research centers as well as the Jet Propulsion Laboratory  These officials are responsible for technology programs providing funds to research centers and the Jet Propulsion Laboratory to support their specific mission areas  NASA’s science and technology project portfolios are based on the requirements and priorities established by NASA’s leaders in collaboration with key stakeholders in academia and industry among others  In planning their science and technology work, NASA’s Glenn Research Center officials noted that NASA research center directors consider the capabilities and resources—including staff and facilities—of other research centers to minimize redundant work  NASA depends on a highly skilled civil servant and contractor workforce to plan and execute its missions  Congress provided NASA with additional human resource authorities beyond those otherwise allowed for federal government personnel through the NASA Flexibility Act of 2004  We found in September 2008 that NASA sought this flexibility to ensure that it could hire and retain the workforce it desired  This law consisted of multiple provisions to address a range of human capital challenges and to strengthen all levels of the workforce  The provisions included incentives—including compensation—to allow NASA to compete successfully in the labor market with the private sector and reshape its workforce more effectively to support the Agency’s mission  NASA also employs a significant contractor workforce across its different centers  Glenn Research Center officials we interviewed stated their portfolio of science and technology projects—and funding—mostly aligns with NASA’s top requirements and priorities  They, along with NASA sponsors, create technology roadmaps and investment plans to determine their future projects  NASA policy requires that NASA’s scientific teams offer proposals for potential research and science and technology projects  This is similar in some ways to how many DOD and Energy centers must find sponsors willing to fund specific technology development projects, rather than receiving technology development funding for a given year  These proposals are reviewed by peer review teams, who identify for selecting officials those proposals they believe have the most scientific merit  Ames Research Center officials said they believe this process can foster innovation, encourage employees to keep skills honed, and mitigate complacency  Glenn Research Center officials said that while most of the work they conduct is for sponsored applied research or advanced technology development, about 2 percent of their science and technology budget is spent on early-stage scientific innovation  Recommended projects of this nature proposed by the research center are typically approved by headquarters officials, according to these Glenn officials  NASA provides technical grants for basic research and applied science to university scientists nationwide on a competitive basis, and also funds similar research done internally at research centers  As with DOD and Energy’s research centers, NASA officials have identified some key operating challenges, including: Aging infrastructure and facilities  The NASA Inspector General listed infrastructure area as one of the top five management and performance challenges facing NASA  Further, the Inspector General identified deficiencies with facilities planning and reported that about 80 percent of facilities at three of four NASA research centers are over 50 years old, while about half of the facilities at the Jet Propulsion Laboratory and the fourth research center are that old  Infrastructure projects and upgrades of $1 million or less are undertaken by research center management instead of at the NASA headquarters level  Construction above this threshold has significantly more requirements and is approved by NASA headquarters  Glenn Research Center officials indicated it is difficult to obtain funding for projects that exceed the minor infrastructure threshold, in part, due competition with major construction of facilities proposals from across the agency for limited funds  As a result, they put most of their efforts into sustaining existing infrastructure  Workforce shortages in key technical areas  As we found in May 2018, NASA has experienced workforce challenges on several major projects such as the Mars 2020 and Europa Clipper projects  Also, over 40 percent of NASA’s workforce is either eligible to retire now or will be eligible in the next 5 years  NASA headquarters officials noted that NASA’s workforce is aging because NASA has a low attrition rate—about 4 percent annually—and high numbers of staff stay several years beyond retirement  Further, in 2017, the NASA Inspector General found gaps in NASA’s workforce planning for specific capability areas and how workforce plans would meet future needs, and recommended that NASA establish standardized guidance defining the data and analyses for these planning efforts  NASA concurred with and identified its plan to implement this recommendation  However, NASA has not implemented this recommendation, according to the NASA Inspector General’s latest semiannual report to Congress  Congress provided DOD lab directors with key authorities to foster targeted, timely investments in the most pressing technology areas  Lab directors have used these authorities—such as laboratory initiated research and direct hire authorities—to varying degrees, but more needs to be done to facilitate innovation and efficiency  Specifically, service specific obstacles in the Air Force, Navy, and Army impede lab directors from capitalizing on laboratory initiated research authority to a greater extent  Service leadership can take actions to better understand and potentially remove barriers to more fully use laboratory initiated research tools  We are making the following three recommendations to DOD: The Secretary of the Air Force should assess the potential costs and benefits of implementing accounting system improvements that would allow the Air Force Research Laboratory to charge customers a fixed percentage fee on provided science and technology activities to the extent allowed under the laboratory initiated research authority  (Recommendation 1) The Secretary of the Navy should clarify whether and how to use the laboratory initiated research authority within the Capital Investment Program  (Recommendation 2) The Secretary of the Army should assess existing Army policy for laboratory initiated research authority and determine whether to implement changes to eliminate disincentives for lab usage of the authority  (Recommendation 3) We provided a draft of this report to DOD, Energy, and NASA for review and comment  Energy and NASA did not provide any comments on the draft report  In DOD’s written comments, reproduced in appendix III, DOD concurred with our three recommendations  Further, in its response to our third recommendation, DOD stated that the Army plans to initiate a study by January 2, 2019, regarding its use of the laboratory initiated research authority  According to DOD, the Army’s study will identify potential opportunities for policy improvements  We are sending copies of this report to the appropriate congressional committees and offices; the Secretary of Defense; the Secretaries of the Army, Navy, and Air Force; the Secretary of Energy; and the NASA Administrator  In addition, the report will be made available at no charge on the GAO website at http://www gao gov  If you or your staff have any questions concerning this report, please contact me at (202) 512-4841  Contact points for our offices of Congressional Relations and Public Affairs may be found on the last page of this report  Staff members making key contributions to the report are listed in appendix IV  This report examines (1) how the Department of Defense (DOD) labs have used selected legislative authorities to foster innovation and efficiency and identify what barriers impede their use; (2) identifies and describes governance models used by selected DOD-sponsored federally funded research centers and university affiliated research centers; and (3) identifies and describes governance models used non-defense labs, specifically at the Department of Energy (Energy) and National Aeronautics and Space Administration (NASA)  To address the first objective, we selected four specific authorities for our review based on previous work identifying science and technology best practices and expedited lab hiring: Laboratory Initiated Research Authority  The authority provided in Section 219 of the Duncan Hunter National Defense Authorization Act for Fiscal Year 2009, as implemented, provides lab directors with flexibility to fund projects in four allowable categories: basic and applied research; technology transition; workforce development; and revitalization, recapitalization, or repair or minor military construction of lab infrastructure  Laboratory Enhancement Pilot Program  Section 233 of the National Defense Authorization Act for Fiscal Year 2017 established a pilot program for lab directors to propose alternative and innovative methods that might lead to more effectively managing and operating labs and authorized lab directors to waive any regulation, restriction, requirement, guidance, policy, procedure, or departmental instruction that would affect implementation of these methods unless such implementation would be prohibited by a provision of an existing statute or common law  Direct Hire Authority  Four types of direct hire authorities authorized by Congress since 2008 are intended to provide a streamlined and accelerated hiring process to allow the labs to successfully compete with private industry and academia for high-quality scientific, engineering, and technician talent  Micro-purchase Authority  The Federal Acquisition Regulation states a preference for government agencies, to purchase and pay for micro-purchases of supplies or services using the government-wide commercial purchase card up to and at the micro-purchase threshold, but micro-purchases may be conducted using any of the simplified acquisition methods  While the FAR micro-purchase threshold was generally $3,500 at the time of our review, Congress increased this threshold to $10,000 for activities of DOD science and technology reinvention laboratories in Section 217 the National Defense Authorization Act for Fiscal Year 2017  Although Congress has provided additional legislative authorities to defense lab directors to address hiring, infrastructure, and technology transition challenges, the authorities that we covered in our review are the ones that our prior and current work have shown are currently, or have the potential to be, the most critical for supporting science and technology reinvention laboratories’ innovation mission within DOD labs  DOD lab leaders use these authorities to flexibly fund projects intended to facilitate research and development; propose alternative and innovative methods that might lead to more effective lab management; directly hire personnel at DOD labs including students currently enrolled in science, technology, engineering, and mathematics (STEM) programs; and expand critical science and technology purchases using simplified acquisition methods  To identify the extent to which DOD laboratories have used these authorities as well as to identify what potential barriers existed to using these authorities, we administered a survey to 44 STRL directors (or their equivalent) to collect information on the use of these specific authorities, their perceptions about the effectiveness of those authorities, and their perceptions about any barriers to using these authorities  The members of the population surveyed were the 44 defense laboratories defined as science and technology reinvention laboratories  For the purposes of our review, we defined laboratories as inclusive of Air Force technical directorates (10), Army warfare centers (17), and Navy warfare centers (17)  We emailed questionnaires to the laboratories beginning in late March 2018, and survey data collection ended in early May 2018, with 31 labs returning completed questionnaires, for an overall response rate of 71 percent at the laboratory level  We took steps to minimize the potential errors that the practical difficulties of conducting any survey may introduce  Nonresponse error can result when a survey fails to capture information from all population members selected into a survey sample  Of the 13 questionnaires not returned, 4 were Army warfare centers, and 9 were Air Force research directorates  Throughout the data collection period, we made multiple follow-up attempts by email and phone to those labs not yet responding  The Air Force Research Laboratory (AFRL) provided a single survey response for the entire laboratory enterprise  Not all returned questionnaires may have answers to every question applicable to a respondent  However, this question-level nonresponse did not exceed one for any of the questions applicable to all 31 labs  Because we selected the entire population of laboratories for our survey, our estimates are not subject to sampling error  We developed our list of the 44 labs in our population in consultation with DOD, and are confident that none were left out, so our or survey has no known sources of coverage error  We conducted pretests of the draft questionnaire with 3 laboratories in the population and made revisions to reduce the possibility of measurement error from differences in how questions were interpreted and the sources of information available to respondents  After reviewing the answers received, we also followed up as necessary with respondents to clarify apparent inconsistencies or other possible misreports, and made changes to responses where corrections were needed  A second, independent analyst checked the accuracy of all computer analyses to minimize the likelihood of errors in data processing  To obtain additional information on this objective, we reviewed relevant legislation which established or amended these authorities and reviewed applicable DOD and service policy documentation  Further, we collected military service related information on the usage of two authorities, such as: Spending data on the use of the laboratory initiated research authority  We gathered this information from DOD-mandated reports to Congress on the use of this authority and military service officials  We determined these data to be reliable based on reviews of agency documentation collected and interviews with agency officials  Data on the usage of direct hire authorities by the service laboratories  We collected direct hire data from each of the military services including the number of direct hire authority candidates hired as well as the number of direct hire positions the laboratories were authorized to hire  We determined these data to be reliable based on reviews of agency documentation collected and interviews with agency officials  We also used select findings from our May 2018 report where we evaluated DOD’s use of hiring authorities, including direct hire authority  More information about the scope and methodology of our prior work can be found in that report  In addition, we also collected information on military service proposals to utilize the laboratory enhancement pilot program authority  To obtain further information on department- and service-level involvement in and perspectives of defense laboratory authorities and challenges, we interviewed officials responsible for the management, execution, and oversight of DOD’s science and technology enterprise, including military service labs  At the Office of the Secretary of Defense and military department headquarters level, those responsible for the management and oversight of science and technology activities, we met with officials from the: Office of the Assistant Secretary of Defense for Research and DOD Defense Laboratories Office; Office of the Deputy Assistant Secretary of the Army for Research and Office of the Deputy Assistant Secretary of the Air Force for Science, Technology, and Engineering; Office of the Assistant Secretary of the Air Force for Financial Office of the Deputy Assistant Secretary of the Navy for Research, Development, Test, and Evaluation; and Office of the Budget, within the Office of the Assistant Secretary of the Navy for Financial Management and Comptroller We also met with military department lab officials responsible for the management and execution of science and technology activities from the: Army Research, Development and Engineering Command; Army Research Laboratory; Army Aviation and Missile Research, Development, and Engineering Air Force Research Laboratory; Naval Research Laboratory; Naval Surface Warfare Center, Headquarters; and Naval Surface Warfare Center, Carderock Division To identify and describe governance models used by selected DOD- sponsored federally funded research centers (FFRDCs) and university affiliated research centers (UARCs), we focused our review on the 3 FFRDCs designated as research and development labs as well as all 13 UARCS sponsored by DOD entities  We reviewed appropriate sections of the FAR language related to FFRDCs and UARCs, DOD guidance for working with FFRDCs and UARCs, relevant contracts, and performance assessments  Further, we met with officials from the office of the Deputy Director, OSD Studies and Federally Funded Research & Development Centers Management and Office to discuss overall FFRDC and UARC management, policies, and challenges facing FFRDCs and UARCs  We interviewed officials at selected research and development FFRDCs and UARCS to discuss their experience conducting DOD research and interactions with their customers, such as defense program executive offices  We met with officials at the two major research and development lab FFRDCs—The Lincoln Laboratory at the Massachusetts Institute of Technology (MIT) and the Software Engineering Institute at Carnegie Mellon University  We also selected a university affiliated research center sponsored by the Army and Navy: The Applied Physics Laboratory at Johns Hopkins University and the Institute for Soldier Nanotechnologies also at the MIT  To identify and describe governance models by non-defense labs, we selected Energy and NASA to focus our efforts  We identified 17 Energy national labs and 4 NASA research centers conducting basic and applied research similar to DOD labs  These agencies, along with DOD, represent 3 of the top 4 agencies in terms of average federal research and development spending from fiscal years 2015 to 2017  In our August 2016 GAO Technology Readiness Assessment Guide, we drew heavily from DOD, NASA, and Energy for best practices, terminology, and examples  This contributed to our decision to focus on Energy and NASA’s research entities in this laboratory governance review  We did not include the fourth agency—the National Institutes of Health—in our review because it is not as similar to DOD  We also reviewed relevant Energy and NASA guidance as well as relevant FAR sections  At Energy, we met with officials from the National Nuclear Security Administration which is semi-autonomous entity within Energy responsible for managing the nation’s nuclear weapons and nuclear security  We also met with Officials from the Office of Science, a program office responsible for supporting energy related fundamental science and research  To gain further insights on operating structures, funding arrangements, and their overall experience we met with lab leadership at selected Energy labs which were chosen based on initial discussions with agency officials and our review of past GAO work: Oak Ridge National Laboratory, Lawrence Berkeley National Laboratory, Lawrence Livermore National Laboratory, and National Energy Technology Laboratory (the sole Energy government owned and operated laboratory) We also met with leadership from Battelle Memorial Institute, which is the sole or joint contract manager for five Energy national labs including Oak Ridge National Laboratory  In addition, Battelle is an integrated subcontractor at Lawrence Livermore National Laboratory  At NASA, we met with officials with NASA’s Science Mission Directorate and Mission Support Directorate to discuss overall research center management and operations  We also leveraged ongoing and recently completed work at GAO to gain additional insight on NASA’s operations such as human capital management  Almost all of NASA’s research, space, and space flight centers conduct research and development activities  However, we focused our review on four research centers where NASA primarily conducts its aeronautics research, which has substantial overlap with DOD activities  To gain additional insight into the experience of lab leaders at NASA research centers, we met with officials at NASA’s Glenn Research Center and Ames Research Center  In addition, we also met with officials at the NASA Jet Propulsion Center, which is the only NASA-sponsored FFRDC  We conducted this performance audit from July 2017 to December 2018 in accordance with generally accepted government auditing standards  Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives  We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives  Office of the Secretary of Defense (OSD) In addition to the contact named above, Christopher R  Durbin (Assistant Director); Charlie Shivers, III (Analyst-in-Charge); Emily Bond; Lorraine Ettaro; Carl Ramirez; Sylvia Schatz; Sean Seales; Brian Smith; and Robin Wilson made significant contributions to this report 


Report 2:  The program has begun phase three, including approving 19 percent of Boeing’s phase three reports  Program Requirements  While the program manager told us that all of the requirements contribute to the safety of the commercial systems, safety officials are required to approve a subset of these requirements  Examples of requirements approved by safety officials include the ability to leave the spacecraft in an emergency or to abort a launch  When a contractor is ready for NASA to verify that it has met a requirement, the contractor submits data for NASA to review through a verification closure notice  We define “safety-specific notices” as those requiring safety officials’ approval  As shown in table 2, as of March 2018, the program had approved 2 percent of Boeing’s safety-specific notices and 0 percent of SpaceX’s safety-specific notices  Testing  The program also requires testing to verify and validate the crew transportation system  Agency officials emphasized the importance of testing to safety, stating that testing reduces uncertainty about a system’s performance and can uncover unknown problems  As noted above, both contractors will be conducting an uncrewed and a crewed flight test prior to being certified  While a certain level of risk needs to be accepted to conduct human spaceflight, these flight tests help to mitigate this risk by validating the integrated performance of the hardware and software  Agency and program officials stated that the contractors’ flight tests are critical evidence to support certification of a safe and reliable system  As evidenced by the data above, the program still has a significant amount of work ahead with respect to approving certification packages and closing hazard reports and verification closure notices  We have previously found that the program’s workload was an emerging schedule risk, and the contractors have continued to express concern about program officials’ ability to process and approve certification paperwork in a timely manner  Workload has also been a concern for two of NASA’s independent review organizations  For example, the Aerospace Safety Advisory Panel noted in its January 2018 annual report that the sheer volume of work that remains for the program in terms of closing hazard reports and verification closure notices is significant  In addition, the program’s safety and mission assurance office identified the upcoming bow wave of work in a shrinking time period as a top risk to achieving certification  One mechanism the program put in place to assess the overall safety of each spacecraft—loss of crew—has been a focus of the Aerospace Safety Advisory Panel, Members of Congress, our prior work, and the program itself  Loss of crew captures the probability of death or permanent disability to one or more crew members  It has received a lot of attention, in part, because it has been a top risk for the program since 2015  Specifically, the program has been concerned that neither contractor would be able to meet the contract requirement of a 1 in 270 probability of incurring loss of crew  We identified two key concerns with how NASA is using the loss of crew metric: (1) inconsistent approaches to assess the loss of crew metric and (2) no identified plan to share lessons learned about using the loss of crew metric as a safety threshold  A loss of crew value is generated through a probabilistic safety analysis, which models scenarios that could result in the loss of crew using various inputs  According to the program’s analysis, the probability of on-orbit debris damaging the vehicle has the greatest effect on a loss of crew value  This probability is informed by an orbital debris (debris) model, which was updated in 2014, after the loss of crew requirement was established  The updated debris model makes it harder to meet a loss of crew value, in part, because the modeling environment where the contractors’ systems will operate has changed  For example, the updated model includes a larger span of orbit, greater range of debris sizes, and the addition of material density classifications, which were not included in the former model  Further, the probabilistic safety analysis may include operational mitigations, such as on-orbit inspections that would include using cameras on the ISS to visually survey the spacecraft for damage, which, according to officials, makes it easier to meet a loss of crew value  NASA describes the probabilistic safety analysis as a powerful tool that should be used as part of the overall risk management process to ensure the risk associated with development and operation of a system is understood, evaluated, managed, and mitigated  However, we found differences in the approaches that officials plan to use to assess loss of crew as well as in the loss of crew value being measured that could limit the usefulness of this tool  Agency Certification  The agency certification review for each contractor will include an assessment of whether its crew transportation system meets a loss of crew threshold of 1 in 150 for missions to the ISS, which is based on a May 2011 safety memo from the Office of Safety and Mission Assurance  A loss of crew value with a higher denominator, such as 1 in 270, is harder to meet than with a lower denominator, such as 1 in 150  According to the Chief of the Office of Safety and Mission Assurance, he will assess the 1 in 150 threshold using a probabilistic safety analysis that includes the updated debris model and operational mitigations, such as the on- orbit inspections cited above  Program Office  According to program officials, they will assess whether either contractor meets a 1 in 270 loss of crew value based on a probabilistic safety analysis using the former debris model (not the updated model) and not including operational mitigations  Contracting Officer  According to the contracting officer, each contractor’s loss of crew requirement is 1 in 270 without including operational mitigations  The contracting officer stated that SpaceX’s contract requirement uses the updated debris model in the probabilistic safety analysis, whereas Boeing’s contract requirement uses the former debris model in the probabilistic safety analysis  Program’s Chief Safety and Mission Assurance Officer  According to the program’s chief safety and mission assurance officer, he will conduct a probabilistic safety analysis using the updated debris model and will not include operational mitigations to assess whether each contractor meets a 1 in 200 loss of crew value  This loss of crew value stems from a program update that occurred after the initial contracts were signed  These different approaches are summarized in table 3 below  Agency policy requires human spaceflight programs to set a safety threshold, which NASA did for the Commercial Crew Program when it identified the 1 in 150 loss of crew threshold in the May 2011 safety memo  Subsequently, the program set more rigorous loss of crew values in contract and program documents  NASA also updated the debris model, which we previously noted makes it more difficult to meet a loss of crew value  As a result, NASA does not have a consistent approach for how to incorporate key inputs to the probabilistic safety analysis, including changes to the debris model  Instead, the risk tolerance that NASA is accepting with loss of crew varies based upon which entity is presenting the results of its probabilistic safety analysis  For example, it is possible that the program’s assessment will determine that neither contractor will meet the 1 in 270 contract requirement, but that the agency’s assessment will determine that the contractors meet the 1 in 150 agency certification value because that analysis will include operational mitigations  Or the program’s assessment could determine that Boeing meets the 1 in 270 contractual loss of crew requirement, but the agency’s assessment may determine that Boeing does not meet the 1 in 150 agency certification value because that analysis will use the updated debris model  Federal internal controls state that agency management should define objectives clearly to enable the identification of risks and define risk tolerances  Specifically, management should define risk tolerances in specific and measurable terms, so they are clearly stated and can be measured  In this case, because there will be multiple analyses conducted using different inputs, NASA risks not clearly capturing or documenting, in a coherent manner, its overall risk tolerance with respect to loss of crew before a final decision must be made on whether to certify either crew transportation system  Moreover, capturing the challenges and lessons learned from using the loss of crew metric is critical, particularly because agency officials told us that this is the first time this metric has been used as a safety threshold  Also, there are different viewpoints about the utility of the metric as a safety threshold across the agency  The program manager repeatedly told us that loss of crew is best used as a design tool  For example, program officials told us that both contractors incorporated additional orbital debris shielding into their designs to mitigate the orbital debris risk and improve their loss of crew values  In addition, the Aerospace Safety Advisory Panel reported in 2018 that loss of crew should not be viewed as an absolute measure of actual risk during operations  However, the May 2011 agency safety memo states that a breach of the loss of crew threshold would initiate a termination review of the Commercial Crew Program, which is a more strict application of the loss of crew metric  Both program and safety officials told us that, after the agency certification is complete and lessons learned are available to be compiled, sharing those lessons learned across NASA would be a good idea given the complexities associated with assessing the loss of crew metric  As of April 2018, however, agency officials said they did not have a plan for loss of crew knowledge-sharing  We have previously found that lessons learned provide a powerful method of sharing good ideas for improving work processes, facility or equipment design and operation, quality, safety, and cost-effectiveness  Further, according to NASA’s Knowledge Policy on Program and Projects, which is managed through the Office of the Chief Engineer, a principle of each center and mission directorate’s knowledge strategy is that knowledge is the cornerstone of NASA’s ability to achieve mission success  The policy acknowledges that NASA faces continuous challenges in using what it knows effectively  These challenges include, but are not limited to, enabling the identification and flow of knowledge across organizational boundaries; preserving knowledge at risk of being lost; and providing means for individuals, teams, and the organization to learn from experiences  If NASA does not capture lessons learned from the Commercial Crew Program on using the loss of crew requirement to set a program’s safety threshold and whether it met the agency’s intended goal, future programs will not be able to benefit from the knowledge gained from this multibillion dollar investment  NASA’s governance model prescribes a management structure that employs checks and balances among key organizations to ensure that decisions have the benefit of different points of view and are not made in isolation  As part of this structure, NASA established the technical authority process as a system of checks and balances to provide independent oversight of programs and projects in support of safety and mission success through the selection of specific individuals with delegated levels of authority  The technical authority process has been used in other parts of the government for acquisitions, including the Department of Defense and Department of Homeland Security  The Commercial Crew Program is organizationally connected to three technical authorities within NASA: the Office of the Chief Engineer technical authority, the Office of Chief Health and Medical technical authority, and the Office of Safety and Mission Assurance (safety) technical authority  The safety technical authority is responsible for ensuring from an independent standpoint that the program’s products and processes satisfy NASA’s safety, reliability, and mission assurance policies  The NASA safety technical authority has delegated authority through the Kennedy Space Center Director to the Chief Safety and Mission Assurance Officer for the Commercial Crew Program  We have previously reviewed how NASA has organized its technical authorities for its Exploration Systems Development organization—an organization that oversees the development of the Space Launch System, Orion crew capsule, and associated ground systems that have the goal of extending human presence beyond low-Earth orbit  In October 2017, we found that the Exploration Systems Development organization had established an organizational structure in which the technical authorities for engineering and safety and mission assurance were dual hatted simultaneously in programmatic positions  We found that having the same individual simultaneously fill both a technical authority role and a program role created an environment of competing interests, where the technical authority’s ability to impartially and objectively assess the programs while at the same time acting on behalf of the Exploration Systems Development organization in programmatic capacities may be subject to impairments  We found that this was in contrast to a recommendation from the Columbia Accident Investigation Board report—the result of an in-depth assessment of the technical and organizational causes of the 2003 Space Shuttle Columbia accident—for NASA to establish a technical authority to serve independently of the Space Shuttle program, so that employees would not feel hampered to bring forward safety concerns or disagreements with programmatic decisions  The board’s findings that led to this recommendation included a broken safety culture in which it was difficult for minority and dissenting opinions to percolate up through the hierarchy; dual center and programmatic roles vested in one person that had confused lines of authority, responsibility, and accountability and made the oversight process susceptible to conflicts of interest; and oversight personnel in positions within the program, increasing the risk that these staffs’ perspectives would be hindered by too much familiarity with the programs they were overseeing  In October 2017, we recommended that the division no longer dual hat two individuals who had both programmatic and technical authority responsibilities  As of April 2018, NASA had taken steps to separate the engineering technical authority position from the programmatic position, and NASA’s Chief of Safety and Mission Assurance said he planned to separate the safety position but had not yet completed that action  The Commercial Crew Program employs a similar structure to the Exploration Systems Development organization in that the safety technical authority is dual hatted simultaneously in a programmatic position as the Commercial Crew Program’s Safety and Mission Assurance Manager  According to the program’s safety technical authority, in his programmatic role for the program, he helps set priorities for safety issues, including how staff will be utilized to meet those priorities  In the technical authority role, he provides independent oversight in support of safety and mission success  In his dual-hatted role, this official will be responsible for endorsing the program’s certification recommendations in two different capacities: as the technical authority and as a program authority  As a result, this structure relies on the same individual to completely separate two roles—one to manage the Commercial Crew Program’s safety issues within programmatic cost and schedule constraints, and the other to assess the same issues in an independent oversight role  While the Commercial Crew Program may have an additional level of separation between the safety technical authority and the program’s involvement in the design of commercial systems due to its shared assurance model with the commercial providers, the Commercial Crew Program still maintains a structure where one individual simultaneously serves in both technical authority and programmatic roles  Figure 5 describes some of the conflicting roles and responsibilities of this official in his two different positions  During our review, officials cited several factors in support of a dual- hatted approach: The safety technical authority retains independence because his technical authority reporting path and performance reviews are not under the purview of the Commercial Crew Program chain of command  Due to the Commercial Crew Program’s shared assurance model with commercial providers, the program is operating in a quality assurance role that provides an additional level of separation between the safety technical authority and the program’s involvement in the design of commercial systems  For safety decisions involving cost and schedule where the individual who is dual hatted with both technical authority and programmatic responsibilities may feel conflicted, he stated that he would discuss these matters with his management to validate the logic behind his decision  There are cost and knowledge efficiencies gained from one individual serving in both programmatic and safety technical authority capacities  NASA’s Chief of Safety and Mission Assurance stated that he has great confidence in the individual currently serving in the dual hatted role for the Commercial Crew Program, but acknowledged there is inherent conflict even with the program’s shared assurance model  In December 2017, he stated that, based on our previous work and current discussions, he intends to decouple the programmatic and technical authority responsibilities for the Commercial Crew Program but had not done so as of April 2018  Federal internal control standards state that an agency should design control activities to achieve objectives and respond to risks, which includes segregation of key duties and responsibilities to reduce the risk of error, misuse, or fraud  By overlapping technical authority and programmatic responsibilities, NASA will continue to run the risk of creating an environment of competing interests for the Commercial Crew Program’s safety technical authority  NASA’s Commercial Crew Program is a multibillion dollar effort to facilitate the commercial development of a crew transportation system that can end the United States’ reliance on Russia to maintain an uninterrupted presence on the ISS  Boeing and SpaceX continue to make progress developing a capability to fly to the ISS, but both have continued to experience delays  Program analysis indicates risks of further delays in each contractor’s current schedule, but NASA has not provided that information to Congress in its routine briefings  Without this information, Congress does not know the full extent of potential delays to inform decision making  Additional delays could also disrupt U S  access to the ISS  While NASA is working on potential solutions, there is no contingency plan in place to address this potential gap  Without a viable contingency plan, NASA puts at risk achievement of the U S  goal and objective for the ISS  NASA must balance safety with acceptable risk for human spaceflight  As part of the certification process for each contractor’s spacecraft, NASA has developed one key safety metric, loss of crew  However, the complicated nature of this metric is further muddled by the inconsistent approaches being used across NASA about what inputs to be considered  As a result, there is no clear articulation of what level of risk NASA will accept with respect to this program  In addition, NASA does not have plans to capture lessons learned from how the Commercial Crew Program has used this metric to assess safety and is missing an opportunity to capture this knowledge for future human spaceflight programs  Finally, a space program’s management and oversight approach is an integral part of ensuring that human spaceflight is as safe and successful as possible  Independence of the program management and oversight functions is key to achieving the balance between safety and success  The Commercial Crew Program’s approach, however, burdens the safety technical authority with both programmatic and independent technical authority responsibilities  As a result, NASA has limited assurance that independence can be maintained as part of its institutional process to ensure safety and success  We are making the following five recommendations to NASA: The NASA Associate Administrator for Human Exploration and Operations should direct the Commercial Crew Program to include the results of its schedule risk analysis in its mandatory quarterly reports to Congress  (Recommendation 1) The NASA Administrator should develop and maintain a contingency plan for ensuring a presence on the ISS until a Commercial Crew Program contractor is certified  (Recommendation 2) The NASA Administrator should direct the Chief of Safety and Mission Assurance, the NASA Associate Administrator for Human Exploration and Operations, the Commercial Crew Program Manager, and the Commercial Crew Program Contracting Officer to collectively determine and document before the agency certification review how the agency will determine its risk tolerance level with respect to loss of crew  (Recommendation 3) After completing the agency certification review, NASA’s Chief Engineer and Chief of Safety and Mission Assurance, with support from the NASA Associate Administrator for Human Exploration and Operations and the Commercial Crew Program Manager, should document lessons learned related to loss of crew as a safety threshold for future crewed spaceflight missions, given the complexity of the metric  (Recommendation 4) The NASA Chief of Safety and Mission Assurance should restructure the technical authority within the Commercial Crew Program to ensure that the technical authority for the Office of Safety and Mission Assurance is no longer dual hatted with programmatic and independent technical authority responsibilities  (Recommendation 5) We provided a draft of this report to NASA for review and comment  NASA provided written comments that are reprinted in appendix II  In its response, NASA concurred with three of our recommendations, did not concur with one, and partially concurred with another  NASA concurred with our recommendation to develop and maintain a contingency plan to ensure a U S  presence on the ISS and expects to take action to close this recommendation by the end of December 2018  NASA concurred with our recommendation to document lessons learned related to the loss of crew requirement and expects to take action to close this recommendation by the end of May 2019  NASA concurred with our recommendation to restructure the safety technical authority so that it is no longer dual hatted with programmatic and independent technical authority responsibilities  NASA expects to take action to close this recommendation by the end of August 2018  NASA did not concur with our recommendation that the Commercial Crew Program should include the results of its schedule risk analysis in its quarterly reports to Congress  NASA stated that it uses the contractors’ schedules as a baseline to provide qualitative statements in the NASA summary that accompanies each contractor’s quarterly reports to Congress  NASA believes that this approach is appropriate and is in accordance with the explanatory statement accompanying the Consolidated and Further Continuing Appropriations Act, 2015  NASA also stated that it will be working to ensure that the contractors’ schedules and the program’s internal assessments sync up as the program gets closer to launch  As a result, NASA explained that there will not be a requirement for a detailed NASA assessment, because the contractors’ schedule will either match NASA’s analysis or NASA will discuss its position as it has done in previous reports to Congress  We continue to believe the recommendation is valid because the program’s schedule risk analysis would provide Congress with valuable insight into potential delays, which are likely  Both contractors have repeatedly stated that their schedules are aggressive and that the dates are ambitious  As a result, we found that the contractors frequently delay dates for key events  For example, Boeing has delayed its certification milestone by 17 months and SpaceX by 22 months since the original schedules were established  The program’s recent schedule risk analysis indicates that more delays to certification are likely, but that information is not presented to Congress in NASA’s quarterly reports  Without this information, Congress does not know the full extent of potential delays to inform decision making  NASA partially concurred with our recommendation that the Chief of Safety and Mission Assurance, the NASA Associate Administrator for Human Exploration and Operations, the Commercial Crew Program Manager, and the Commercial Crew Program Contracting Officer should collectively determine and document how the agency will determine its risk tolerance level with respect to loss of crew before the agency certification review  In its response, NASA stated that it documented the agency’s risk tolerance level with respect to loss of crew for the program in its May 2011 safety memo  Further, NASA stated that it documented the requirement to limit risks to the loss of crew in a certification requirements document  NASA stated that ultimately the Commercial Crew Program is accountable for ensuring that the contractors’ systems meet the loss of crew value in this certification requirements document, which is a loss of crew value of 1 in 270  If a contractor’s system cannot meet that loss of crew value, or any other requirement, the program will request a waiver as part of the human rating certification process to ensure transparency  NASA acknowledged in its response that the existence of multiple documents defining residual risk requirements and an agency threshold for loss of crew can be confusing  NASA’s response, however, does not address our finding that it does not have a consistent approach for how to incorporate key inputs, including which debris model should be used or whether to include operational mitigations  NASA stated that it had taken action to address this recommendation; however, NASA did not outline any steps it took to resolve the concern that the risk tolerance for the loss of crew requirement depends on which entity is presenting the results of its analysis  We continue to believe that, before the agency certification review, the key parties must collectively determine how the agency will determine its risk tolerance with respect to loss of crew  We believe this approach will reduce confusion and increase transparency  We are sending copies of this report to NASA Administrator and interested congressional committees  In addition, the report is available at no charge on the GAO website at http://www gao gov  If you or your staff have any questions about this report, please contact me at (202)512-4841 or chaplainc@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report  GAO staff who made major contributions to this report are listed in appendix III  The objectives of our review were to assess (1) the extent to which the contractors have made progress towards meeting the National Aeronautics and Space Administration’s (NASA) certification requirements and NASA’s plans to help ensure continued access to the International Space Station (ISS); and (2) how NASA’s certification process addresses safety of the contractors’ crew transportation systems  To assess the contractors’ progress towards certification, we obtained and reviewed program and contractor documents, including monthly and quarterly updates from April 2017 through May 2018  We interviewed program and contractor officials to discuss the contractors’ recent progress, including upcoming events and any expected delays, and to understand technical risks, potential consequences, and planned mitigation activities  To identify total delays to date, we compared original contract schedules to Boeing and SpaceX’s calendar year 2018 first quarter proposed schedules, which are the most recent  Based on our review of program and contractor documents, we defined the contractors’ key events as: the uncrewed and crewed flight tests, the design certification reviews for each of those flights, and the certification milestone  We selected the two flight tests for each contractor as key events because they are intended to test key system capabilities, including the ability to launch, dock with the ISS, and return safely to Earth  We selected the design certification reviews because they verify the contractors’ crew transportation systems’ capability to safely approach, dock, mate, and depart from the ISS, among other requirements  We selected the certification milestone because it determines whether the crew transportation system meets the Commercial Crew Program’s requirements  To determine the extent to which contractors have delayed these key events over time, we analyzed the contractors’ schedule data from the 13 quarterly progress reports to date, from first quarter 2015 through first quarter calendar year 2018  We also obtained the results of the program’s April 2018 schedule risk analysis  We presented the schedule analysis range from the end of the month of the earliest possible completion date to the end of the month of the latest possible completion date  We reviewed the program’s Congressional requirements to report on cost, schedule, and technical status  Finally, to assess the potential effects of any certification delays on NASA’s access to ISS, we reviewed NASA’s contracts with Boeing and the Russian Federal Space Agency for transportation on the Soyuz vehicle  We interviewed officials from the ISS program and NASA’s Human Exploration and Operations Mission Directorate to determine if the agency had developed contingency plans to mitigate the effects of any certification delays on its access to the ISS  To assess how NASA’s certification process addresses safety of the contractors’ crew transportation systems, we reviewed agency safety policies, program plans, and contract documents to identify what safety assessments were required of the program, which safety factors would be considered in certification reviews, and when certification approval would be granted  We also interviewed program officials and the contractors about their safety policies and procedures as well as about the certification process  We identified the loss of crew requirements for the program and the contractors, and interviewed program and agency officials to determine how loss of crew would be assessed and considered throughout the certification process  To gain a broader understanding of the relative importance of loss of crew, we reviewed NASA safety policies, prior GAO reports, and annual reports from the Aerospace Safety Advisory Panel  To determine how the Orbital Debris Engineering Model (ORDEM) was updated and to assess differences between the former and updated models, we reviewed NASA documentation about the ORDEM 2000 and ORDEM 3 0 models and obtained information about the models from NASA’s Orbital Debris Program Office  To assess the program’s progress in closing requirements, including those requirements specifically related to safety, we reviewed program data for each contractor on contract requirements and closure status  We limited this analysis to requirements included in the CCT-REQ-1130 ISS Crew Transportation and Services Requirements Document because these requirements are verified by the Commercial Crew Program, whereas requirements contained within SSP 50808 ISS to Commercial Orbital Transportation Services Interface Requirements Document are managed by the ISS Transportation and Integration Office  We classified requirements as “safety-specific” when the program’s Office of Safety and Mission Assurance was listed as a verification closure notice signatory (i e , approver)  We then analyzed the program’s data to determine how many verification closure notices had been approved for all requirements and for the subset of safety-specific requirements  We reviewed program and agency documentation, such as organizational charts, program plans, and safety policies as well as interviewed program and agency officials, to determine the role of the safety and mission assurance technical authority in the program  We also reviewed the 2003 Columbia Accident Investigation Board’s Report’s findings and recommendations related to culture and organizational management of human spaceflight programs  We reviewed annual briefings and reports and met with representatives from two organizations that provide NASA with independent assessments of the program, the program’s standing review board and the Aerospace Safety Advisory Panel, to gain their perspectives on the contractor’s progress and how NASA addresses safety in its certification process  We also met with representatives from the National Transportation Safety Board and three experts with background on safety in human spaceflight in order to increase our contextual understanding of the role of safety in human spaceflight missions  We conducted this performance audit from April 2017 to July 2018 in accordance with generally accepted government auditing standards  Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives  We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives  Cristina T  Chaplain, (202) 512-4841 or chaplainc@gao gov  In addition to the contact named above, Molly Traci, Assistant Director; Kazue Chinen; Lorraine Ettaro; Lisa Fisher; Laura Greifner; Kurt Gurka; Miranda Riemer; Juli Steinhouse; Roxanna T  Sun; Hai Tran; Kristin Van Wychen; and Alyssa Weir made significant contributions to this report 

ground_truth: In 2014, NASA awarded two firm-fixed-price contracts to Boeing and SpaceX, worth a combined total of up to $6.8 billion, to develop crew transportation systems and conduct initial missions to the ISS. In February 2017, GAO found that both contractors had made progress, but their schedules were under mounting pressure. The contractors were originally required to provide NASA all the evidence it needed to certify that their systems met its requirements by 2017. A House report accompanying H.R. 5393 included a provision for GAO to review the progress of NASA's human exploration programs. This report examines the Commercial Crew Program, including (1) the extent to which the contractors have made progress towards certification and (2) how NASA's certification process addresses safety of the contractors' crew transportation systems. GAO analyzed contracts, schedules, and other documentation and spoke with officials from NASA, the Commercial Crew Program, Boeing, SpaceX, and two of NASA's independent review bodies that provide oversight. Both of the Commercial Crew Program's contractors, Boeing and Space Exploration Technologies Corporation (SpaceX), are making progress finalizing designs and building hardware for their crew transportation systems, but both contractors continue to delay their certification milestone (see figure). Certification is the process that the National Aeronautics and Space Administration (NASA) will use to ensure that each contractor's system meets its requirements for human spaceflight for the Commercial Crew Program. Further delays are likely as the Commercial Crew Program's schedule risk analysis shows that the certification milestone is likely to slip. The analysis identifies a range for each contractor, with an earliest and latest possible completion date, as well as an average. The average certification date was December 2019 for Boeing and January 2020 for SpaceX, according to the program's April 2018 analysis. Since the Space Shuttle was retired in 2011, the United States has been relying on Russia to carry astronauts to and from the International Space Station (ISS). Additional delays could result in a gap in U.S. access to the space station as NASA has contracted for seats on the Russian Soyuz spacecraft only through November 2019. NASA is considering potential options, but it does not have a contingency plan for ensuring uninterrupted U.S. access. NASA's certification process addresses the safety of the contractors' crew transportation systems through several mechanisms, but there are factors that complicate the process. One of these factors is the loss of crew metric that was put in place to capture the probability of death or permanent disability to an astronaut. NASA has not identified a consistent approach for how to assess loss of crew. As a result, officials across NASA have multiple ways of assessing the metric that may yield different results. Consequently, the risk tolerance level that NASA is accepting with loss of crew varies based upon which entity is presenting the results of its assessment. Federal internal controls state that management should define risk tolerances so they are clear and measurable. Without a consistent approach for assessing the metric, the agency as a whole may not clearly capture or document its risk tolerance with respect to loss of crew. GAO is making five recommendations, including that NASA develop a contingency plan for ensuring a U.S. presence on the ISS and clarify how it will determine its risk tolerance for loss of crew. NASA concurred with three recommendations; partially concurred on the recommendation related to loss of crew; and non-concurred with a recommendation to report its schedule analysis to Congress. GAO believes these recommendations remain valid, as discussed in the report.