You are given two reports by the government agency.  

Report 1:
In the United States, both FRA and FTA regulate rail transportation safety. FRA oversees safety of railroads operating on what is known as the general system, a network of standard gage track over which goods may be transported and passengers may travel. This system includes freight railroads, which typically own their own tracks and locomotives, transporting products among states and regions. FRA also oversees safety of intercity passenger and commuter railroads that operate over tracks owned by freight railroads and other entities. FTA oversees safety of rail transit systems that typically serve individual metropolitan areas, using track not shared with freight and other passenger trains. Rail transit includes a variety of modes, such as heavy and light rail, streetcars, automated guideways, cable cars, and others. Rail transit is an important component of the nation’s transportation network, particularly in large metropolitan areas. Rail transit systems provided over 4.4 billion passenger trips in 2016. “Heavy rail” systems in large cities account for much of the total rail transit activity, including 88 percent of passenger trips in 2016. According to FTA, 61 rail transit systems within 28 states, the District of Columbia, and Puerto Rico are subject to safety oversight by one of the 31 agencies in FTA’s state safety oversight program (see fig. 1). The states have long played a central role in conducting safety oversight of rail transit systems. The Intermodal Surface Transportation Efficiency Act of 1991 required, among other things, that states with rail transit operators designate an agency to oversee the safety of those systems, known as a state safety oversight agency. In overseeing state safety agencies, FTA designed the program as one in which FTA, states, and rail transit operators collaborate to ensure the safety and security of rail transit systems. However, limitations have been identified in the state safety oversight program. In 2006, we reported on some state safety agency challenges in overseeing rail transit safety. Specifically, we found many of the state safety agencies lacked enough qualified staff and adequate levels of training to meet their responsibilities. In a 2009 hearing before the Subcommittee on Highways and Transit of the House Committee on Transportation and Infrastructure, then Secretary Ray LaHood of the Department of Transportation discussed some of the weaknesses under the current state safety oversight program and introduced a public transportation safety legislative proposal. In 2010, various bills were introduced in both houses of Congress that would have provided FTA with various enforcement mechanisms and with the authority to issue safety regulations. Additionally, the bills would have required the Secretary to establish a federal certification program for employees and contractors who carry out a state public transportation safety program. In the 112th Congress, the Senate amended a House bill to include a public transportation safety provision, which eventually became section 20021 of MAP-21, the federal public transportation safety program. MAP-21 enhanced FTA’s authority to oversee the safety of rail transit, potentially addressing some of the weaknesses identified by various stakeholders. Specifically, MAP-21 established a comprehensive Public Transportation Safety Program, which continues to rely on state safety agencies to monitor rail transit systems’ safety operations. MAP-21 required that, within 3 years of the effective date of a final state safety oversight program rule, each eligible state have in place a state safety oversight program certified by FTA. An eligible state must, among other things, establish a state safety agency and determine, in consultation with FTA, an appropriate staffing level for this state agency that is commensurate with the number, size, and complexity of the rail transit systems within the state. Additionally, a state safety agency must be financially and legally independent from any rail transit system it oversees and have investigative and enforcement authority with respect to the safety for its rail transit systems, among other things. Each eligible state has until April 15, 2019, to receive FTA approval of its state safety oversight program, or else FTA will be prohibited from obligating certain federal financial assistance to any entity in the state that is otherwise eligible to receive that federal financial assistance. After that approval, state safety agencies will be evaluated for continued compliance with FTA regulations a minimum of once every 3 years through a triennial review process. According to FTA, these requirements represent a dramatic increase in federal expectations for state safety oversight and for the rail transit industry. MAP-21 also established a state safety oversight grant program, offering federal funding to states for their state safety activities. FTA’s Office of Transit Safety and Oversight administers the state safety oversight program. Freight and passenger railroads have played a transformational role in the development of America and continue to be an important part of the economy. The general railroad system consists of a vast network of operations (see fig. 2). The $60 billion freight rail industry is operated by seven Class I, and hundreds of smaller, railroads. In addition, about 40 railroads move passengers, which carry greater than 670 million passengers per year. The federal government has long provided regulatory oversight of railroad safety, both passenger and freight, that operate on the general system. The Interstate Commerce Commission, the first federal regulatory commission in U.S. history, was established in 1887 to regulate interstate commerce by rail. The Commission’s safety functions were transferred to FRA, which was created by the Department of Transportation Act in 1966. In its role as federal regulator and overseer of railroad safety, FRA prescribes and enforces railroad safety regulations and conducts research and development in support of improved railroad safety and rail transportation policy. FRA utilizes safety inspectors and specialists, primarily covering five safety disciplines, to review and enforce compliance with these regulations. FRA’s safety disciplines are track, signal, and train control; motive power and equipment; operating practices; and hazardous materials. Following several fatal rail accidents between 2002 and 2008, the Rail Safety Improvement Act of 2008 was enacted, the first authorization of FRA’s safety programs since 1994. This act directed FRA to, among other things, issue new safety regulations for different aspects of railroad safety, such as hours of service requirements for passenger railroad workers, positive train control implementation, track inspection rules, and safety at highway-rail grade crossings. FRA’s Office of Railroad Safety administers the agency’s safety program. Rail transportation is a relatively safe way to transport people and products though serious incidents continue to occur on railroads and rail transit. According to an analysis of DOT’s Bureau of Transportation Statistics data by the American Public Transportation Association, travel by rail transit is far safer than automobile travel. From 2000-2014, for instance, there were 6.53 and 0.33 fatalities per billion passenger-miles traveled in cars or light trucks and rail transit, respectively. Within rail travel, the fatality rates on both railroads and rail transit operators have remained similar in recent years. Further, the rate of accidents and incidents—including collisions and derailments—also do not appear to differ substantially between railroads and rail transit in recent years. Nevertheless, serious incidents continue to occur on railroads and rail transit, posing safety risks to passengers, railroad employees, and the public. For example, in June 2009, two WMATA trains collided, resulting in 52 injuries and 9 deaths. A smoke incident on WMATA’s Metrorail system in January 2015 also resulted in the death of 1 person and injured over 90. In a 10-month period from May 2013 to March 2014, the Metro- North commuter railroad, which serves New York and Connecticut, was involved in five accidents that resulted in the death of 6 people and 126 injured. In June 2016, two BNSF Railway freight trains collided near Panhandle, Texas, resulting in the death of three crew members. Incidents such as these have prompted investigations into both the causes and contributing factors of the specific accidents as well as broader rail safety oversight. FRA has a more centralized safety oversight program for railroads, while FTA is implementing changes to the rail transit oversight program, established in federal statute, which relies on states to monitor and enforce safety. Key characteristics of both programs include: (1) the establishment of safety regulations, (2) inspections and other oversight activities, such as audits and investigations, based on those regulations, and (3) enforcement mechanisms to ensure that safety deficiencies are addressed (see fig. 3). FRA has developed extensive railroad safety regulations over decades. FRA’s railroad safety regulations include requirements governing track design and inspection, grade crossings, signal and train control, mechanical equipment including locomotives, and railroad-operating practices including worker protection rules. For example, FRA’s regulations for track and equipment include detailed, prescriptive minimum requirements, such as formulas that determine the maximum allowable speeds on curved track. Many of FRA’s rail safety regulations establish minimum safety requirements, though railroads can apply for waivers. As FRA updates its safety regulations, it has proposed more performance-based regulations in recent years. Many of FRA’s current safety regulations specify the behavior or manner of compliance that railroads must adopt, such as inspecting each locomotive at least every 92 days. Performance-based regulations, however, specify a desired outcome rather than a behavior or manner of compliance. For example, FRA’s recent rulemaking to amend its passenger equipment safety regulations proposes performance-based crashworthiness and occupant protection requirements, rather than explicit targets or tolerances. According to FRA, establishing performance requirements in these areas would allow a more open rail market that incorporates recent technologies. FTA is currently assessing the need for rail transit safety regulations, having been provided the authority to issue safety regulations in 2012. Since MAP-21 was enacted, FTA has finalized regulations implementing the public transportation safety program authorized by statute. These include regulations that establish rules for FTA’s administration of a comprehensive safety program to improve rail transit safety as well as updated regulations governing state safety oversight of rail transit. In addition to its public transportation safety program regulations, FTA also has regulations governing its drug and alcohol testing program. MAP-21 also authorized FTA, for the first time, to issue rail transit safety regulations, which would establish minimum safety performance requirements for rail transit operators, as part of its requirement to develop a National Public Transportation Safety Plan. FTA initiated a regulatory development effort after the passage of MAP-21, which included a compilation and evaluation of existing transit safety standards, guidance, and best practices from the federal government, states, industry, and other sources. After the evaluation, FTA issued a report that concluded there was limited documentation or evidence of the effectiveness of these existing rail transit safety standards. The report included recommendations that are intended to enable FTA to undertake further data-driven, risk-based analysis of rail transit safety performance and the applicability and effectiveness of the identified safety standards. FTA is also currently analyzing specific focus areas to determine any areas that should be addressed by federal safety regulations. For example, FTA is studying the need for regulations related to rail transit vehicle crashworthiness. Since no federal rail transit safety regulations that establish minimum safety performance requirements for rail transit operators currently exist, rail transit operators are subject to different safety standards, depending largely on what voluntary standards they have chosen to adopt, according to American Public Transportation Association officials we spoke with. The American Public Transportation Association, for instance, has issued a variety of rail transit safety standards, addressing various aspects of the industry including operations, training, and inspections. In addition, states vary in the extent to which they have regulations for rail transit operators. For example, officials from the California Public Utilities Commission noted that it has issued a variety of safety regulations applicable to rail transit operators within the state of California to improve safety of rail operations. Both FRA and FTA have mechanisms to gather the input of stakeholders—including rail operators, labor unions, industry associations, and others—when considering development of safety regulations. In developing most of its safety regulations, FRA seeks input from stakeholders through its Railroad Safety Advisory Committee. In 1996, FRA established this committee to develop new regulations through a collaborative process, with the rail community working together to create mutually satisfactory solutions to safety issues. FTA is collaborating with stakeholders as it assesses the need for rail transit safety regulations. More specifically, FTA’s research partner, the Center for Urban Transportation Research, established a working group to collaborate with industry stakeholders to inform the safety regulations development process. FTA also solicited comments from industry stakeholders on its compilation of existing rail transit safety standards. More broadly, FTA also has a Transit Advisory Committee for Safety, which provides information, advice, and recommendations to FTA on safety matters. FRA fulfills its mission, in part, through safety compliance audits and inspections, and investigations. FRA ensures compliance with its safety regulations through inspections, using a staff of railroad safety experts, inspectors, and other professionals assigned to eight regional offices across the nation. For example, to determine a railroad’s compliance with FRA safety regulations, inspectors examine track, equipment, signal devices, employee actions, and procedures and review maintenance and accident records. Additionally, 31 states have rail safety programs that partner with FRA. Under this approach, FRA enters into agreements with states to allow state inspectors to participate in investigative and surveillance activities concerning federal railroad safety laws. State inspectors who participate in this program submit inspection reports to FRA. More broadly, FRA’s inspections are guided by a risk-based model. Under this approach, FRA focuses its inspections on locations that, according to the data-driven model, are likely to have safety problems. Like other operating administrations within DOT, FRA has relatively few resources for overseeing railroads, compared with the size of the general system. The risk-based model is designed to help FRA target the greatest safety risks. FRA has begun utilizing automated inspections as well. In particular, according to FRA, new imaging technologies have the potential to better inspect track for cracks in the rail that could lead to breakage as well as measure the track’s geometry to ensure that rails are positioned to meet standards. To further promote safety in railroad operations, FRA conducts accident investigations. Separate from investigations conducted by NTSB, FRA investigates select railroad accidents to determine root causation, and any contributing factors, so that railroad properties can implement corrective actions to prevent similar incidents in the future. Resources for railroad safety oversight activities have increased in recent years. FRA was appropriated about $218 million in fiscal year 2017, an increase over the approximately $187 million it received in fiscal year 2015, for safety and operations, which funds FRA’s personnel, including inspectors, and safety programs. According to FRA, Congress provided FRA with increased funding in recent years for the purpose of increasing staffing related to specific safety issues, such as trespasser prevention and passenger rail safety. As part of this effort, FRA has hired additional inspectors, going from 347 inspectors in fiscal year 2013 to over 360 currently, out of the nearly 930 total full-time equivalent staff. FRA officials told us, as we have reported in the past, that it can be difficult to recruit, train, and certify qualified inspectors in a timely manner, especially in certain areas of expertise. Further, according to FRA, its inspectors have the ability to inspect less than 1 percent of the general system annually. Though FTA now has more robust inspection authorities, states will continue to conduct front-line rail transit safety oversight activities. MAP- 21 provided FTA with new authorities to inspect, audit, and investigate practices at rail transit agencies, including safety practices, while also preserving the role of state safety agencies to monitor rail transit systems’ safety operations. According to FTA officials, any federal inspections of rail transit operators are intended to supplement a state safety agency’s oversight activities, except where FTA assumes temporary, direct oversight of a rail transit system from an inadequate state safety agency. FTA officials told us that establishing a nationwide safety inspection program at the federal level is inconsistent with the statutory framework of the state safety oversight program and with congressional intent, which contemplates preserving the primary role of state safety agencies in providing direct safety oversight of rail transit systems. The officials also noted that the state-based approach to rail transit safety oversight is valuable because states are generally closer to, and more familiar with, rail transit operators. To date, FTA has utilized its new inspection authorities only on WMATA’s rail system. As part of oversight activities, some state safety agencies have conducted inspections of the rail transit systems they oversee, though they were not required to do so, according to FTA officials we spoke with. To strengthen states’ abilities to conduct oversight activities, FTA has recommended that state safety agencies develop risk-based inspection programs. Further, to ensure the independence of state safety agencies, these agencies cannot receive funding from the rail transit entities they oversee. Resources for FTA’s rail transit safety oversight administrative expenses have remained relatively stable in recent years, though more are needed, according to FTA. Since fiscal year 2012, FTA’s appropriations for administrative expenses, which funds FTA personnel and support activities including the Office of Transit Safety and Oversight, has increased $14 million, to about $113 million in fiscal year 2017, according to FTA. However, for several years, FTA has averaged about 508 total full-time equivalent staff agency-wide, and a little over 30 safety staff in the Office of Transit Safety and Oversight. According to FTA, the Office of Transit Safety and Oversight has been under-resourced since it was established in response to new safety authority provided in MAP-21. For fiscal year 2018, FTA requested in their submission for the President’s Budget proposal funding to hire up to an additional 20 positions for various lines of safety work. FRA’s and FTA’s oversight activities also include regular audits of, and communication with, the rail operators under their oversight. Given finite resources and large rail networks, FRA and FTA audit rail operators’ own inspections rather than conducting comprehensive federal inspections. More specifically, FRA inspectors, and state safety agencies in FTA’s oversight program, regularly examine records of rail operators’ internal inspections to identify safety deficiencies. Officials from FRA, FTA, and five rail stakeholders we spoke with told us that FRA and FTA rail safety oversight programs also rely on collaboration and communication between rail operators and regulators to ensure safety. For example, regular meetings between FRA and railroad staff to discuss safety trends and industry developments are important to ensuring safety, according to officials we spoke with from FRA and the railroads. FRA specialists and inspectors participate, with representatives of railroad labor and management, in the implementation of voluntary safety programs. For example, FRA sponsors the Confidential Close Call Reporting System, a voluntary, confidential program allowing railroads and their employees to report accident and incident “close calls.” According to FRA officials, voluntary programs such as this increase industry awareness of railroad safety and engagement with it. FTA also collaborates with state safety agencies as rail transit safety issues arise, according to FTA officials, using federal oversight and enforcement authorities as a “back-stop” against the oversight of state safety agencies. Additionally, according to officials we spoke with from FTA and two rail transit operators, state safety agency staff meet with rail transit operators regularly, using knowledge of local operating conditions to help ensure safety. FRA uses a variety of tools, including civil penalties, to resolve safety issues. While some safety issues are resolved informally through discussion and collaboration between FRA and railroads, as noted above, some defects identified during inspections are classified as violations and subject to financial penalties. More specifically, when railroads do not resolve issues in a timely manner or identified defects are serious, FRA has the authority to cite violations and assess civil penalties, against either railroads or individuals. Further, as authorized by law, FRA negotiates settlements with railroads and other entities subject to its safety jurisdiction to resolve claims for civil penalties. In fiscal year 2016, FRA assessed over $11.8 million in civil penalties against railroads. According to FRA, fiscal year 2016 was the second year in a row that it took steps to increase penalty amounts paid by railroads, as part of a continued effort to increase consequences for violations that negatively affect safety. To ensure the safety of rail transit systems, states will continue to be the primary enforcers of safety requirements, according to FTA officials, though FTA now has more enforcement tools. MAP-21 preserved the role of state safety agencies as the primary enforcement body for rail transit. FTA has now required that state safety agencies have enforcement authorities sufficient to compel action from rail transit entities to address safety deficiencies. Though no specific authorities are required, FTA has suggested that a variety of mechanisms could be appropriate, such as the ability to remove deficient equipment from service or assess fines. According to FTA, this requirement is designed to overcome a long- standing vulnerability in state safety oversight, which allowed safety deficiencies to remain for long periods of time. MAP-21 and the FAST Act also provided FTA with more options for enforcement when rail transit operators are found to be out of compliance with safety requirements. In particular, FTA can withhold federal funding for rail transit operators or direct a rail transit operator to use federal funding for a specific purpose. Additionally, after FTA assumes temporary direct oversight of an inadequate state safety agency, FTA can withhold federal funds from the state until the state safety oversight program has been certified. To date, FTA has utilized this authority only with the states responsible for safety oversight of WMATA’s rail system. In February 2017, FTA announced that it would withhold 5 percent of fiscal year 2017 urbanized area formula funds from Maryland, Virginia, and the District of Columbia until a new state safety oversight program is certified for WMATA’s rail system. This action built upon FTA’s determination that WMATA’s state safety agency was ineffective at “providing adequate oversight consistent with prevention of substantial risk of death or personal injury.” FRA and FTA also have the authority to directly intervene in rail operations. In particular, both FRA and FTA can suspend the service of rail operators in response to certain safety concerns. Additionally, FTA can assume direct safety oversight of a rail transit operator if FTA determines the state safety oversight program is not adequate, among other things. In response to safety incidents on WMATA’s rail system, FTA assumed temporary and direct safety oversight of WMATA in October 2015, as previously noted. FRA and FTA’s approaches to their rail safety oversight missions each have strengths and limitations, including how the agencies develop safety regulations, conduct inspections, and carry out enforcement. Compared to FRA’s long-standing role in providing safety oversight over railroads, FTA is in the process of implementing significant changes to its program for rail transit safety oversight after being granted new authorities in MAP- 21 and the FAST Act. With respect to regulations, FRA’s extensive and well-established safety regulations are a strength. FTA has made some progress toward developing appropriate safety regulations, such as identifying subjects for potential regulatory action. With respect to inspections, FRA’s use of a risk-based approach to distributing inspection resources is a strength. FTA has sought to address previously identified deficiencies in state safety oversight by recommending that state safety agencies develop risk-based inspection programs. FTA, though, has not provided states guidance for these efforts. With respect to enforcement, FRA’s use of its enforcement authorities is a strength. FTA is also implementing new statutory requirements that state safety agencies have enforcement authorities but does not have a process or methodology to evaluate the effectiveness of these enforcement practices. Extensive and well-established safety regulations are a strength of FRA’s safety oversight program based on studies we reviewed and discussions with rail operators and stakeholder organizations. According to NTSB, FRA’s railroad safety regulations are an important and effective part of its oversight program. Our previous work reported that according to stakeholders, the Railroad Safety Advisory Committee provides a collaborative environment where stakeholders in the rail community work with FRA to identify issues and proposals for safety standards and regulations that improved the quality of railroads’ safety initiatives and fostered a greater level of compliance with safety regulations. This is consistent with views of stakeholders we spoke with, who characterized FRA’s safety regulations as a strength. An industry association told us that FRA’s regulations promote safety by helping to ensure that no operator falls below a minimum threshold for safe operations, while a rail operator told us that federal regulations help to standardize the operating environment and prevent a patchwork of various state regulations. Four stakeholders also characterized the Railroad Safety Advisory Committee, which plays a large role in crafting FRA’s railroad safety regulations, as effective and inclusive. However, based on studies we reviewed and discussions with rail operators’ and stakeholders’ organizations, FRA faces limitations in its efforts to regulate safety across railroad systems that differ from one another and sometimes change more quickly than the federal regulatory process. Five railroad operators and an industry association told us that some of FRA’s safety regulations do not account for differences in railroads or innovation in safety practices, with three railroad operators stating that this approach requires the extensive use of waivers for particular regulations. Further, two railroad operators and a rail transit operator we spoke with stated that additional federal regulations are needed to provide minimum baseline requirements in specific areas of railroad safety such as medical fitness for duty. In 2014, NTSB also found that FRA needs to do more to regulate particular safety issues including medical fitness for duty and signal protection. FRA officials acknowledged that time and resources are two of the primary challenges that the agency faces when developing safety regulations but also noted additional ways in which the agency can require railroads to adopt safety practices. FRA officials described the process of creating or significantly amending a regulation as involving years of work, even before the agency commences with the process of drafting a rule. The officials also noted that the agency has additional tools to compel railroads to adopt safety practices. For example, FRA officials discussed the use of compliance agreements, in which railroads can have fines reduced in exchange for adopting safety measures that go beyond what FRA regulations require. FRA officials are considering the use of performance-based regulations as they update their safety regulations. As noted above, FRA’s proposed regulations regarding passenger equipment safety incorporates performance -based safety requirements, rather than explicit safety targets or tolerances. FRA has promulgated performance-based regulations about the implementation of positive train control, a communications-based system designed to prevent certain types of train accidents, as well as system safety programs that set general safety parameters and thresholds by which successful performance is governed. FRA’s consideration of performance-based regulations is in line with federal guidance. OMB’s Circular A-4 states that performance standards “are generally superior to engineering or design standards because performance standards give the regulated parties the flexibility to achieve regulatory objectives in the most cost-effective way.” Additionally, under Executive Order 12866, agencies should (to the extent permitted by law and where applicable) identify and assess alternative forms of regulation and specify performance objectives, rather than specifying the behavior or manner of compliance that regulated entities must adopt. However, as the language of OMB’s Circular A-4 and Executive Order 12866 suggest that performance-based regulations are not always feasible, studies of performance-based regulations find that as with any other form of regulation, performance-based standards have trade-offs. FRA officials told us that under certain circumstances, performance- based regulations are appropriate for issues regarding design, maintenance, operation, and technology-driven safety requirements. FRA officials we spoke with did not think performance-based standards are appropriate for areas that require standardization. One example is track safety standards, where the need for different operators to use the same equipment precludes a performance-based approach that allows railroads to meet requirements through different means. FRA officials added that a key aspect of the success of performance-based regulations concerns how railroads demonstrate compliance. This concern is consistent with other studies of performance-based regulations, which find that these regulations are most appropriate when regulators have capacity to measure and monitor performance. Though FTA has made progress assessing the state of rail transit safety standards, a limitation of FTA’s rail transit safety oversight program is the lack of federal rail transit safety regulations, which may contribute to inconsistent safety practices across the rail transit industry, according to studies we reviewed and discussions with rail operators and stakeholder organizations. NTSB reported that the structure of FTA’s oversight process leads to inconsistent practices, inadequate standards, and marginal effectiveness. In addition, a 2016 DOT OIG report found that because FTA’s safety standards are voluntary, they are unenforceable. In 2012, FTA gained the authority to issue safety regulations, though it has not done so yet, and NTSB and other stakeholders we spoke with indicated that the lack of such federal safety regulations is a weakness in federal rail transit safety oversight. Despite differences across rail transit systems, there is value in establishing federal rail transit safety regulations, according to stakeholders from all categories of those we interviewed, including a state safety agency, three rail transit operators, a railroad operator, and two industry associations. Some stakeholders identified specific areas that would benefit from federal rail transit regulations. For example, two rail transit agencies called for federal regulations to address operator fatigue. Some of these officials stated that federal rail transit safety regulations could help ensure safety by establishing clear and consistent minimum standards. Officials from a rail transit entity and an industry association stated that voluntary standards are not enough to ensure that transit entities will adopt appropriate safety measures. According to our analysis, a past study, and stakeholders we spoke with, FTA’s ability to develop and implement performance-based regulations is limited by its lack of capacity to collect and analyze rail safety performance data. In 2017, DOT OIG found that data limitations of FTA’s National Transit Database results in limited safety performance criteria in FTA’s National Public Transportation Safety Plan. Further, two rail transit entities as well as a state safety agency we spoke with stated that they face challenges in analyzing data due to either the size of their systems or their capacity. FTA officials told us that they need more data to inform their decisions regarding whether to establish rail transit safety regulations, and also added that a limitation to their ongoing assessment of potential areas for rail transit safety regulation is the concern about public disclosure of safety data provided to FTA and its potential use in private litigation. According to FTA officials, they need more information to do a comprehensive evaluation of efficacy of current safety standards and practices. As required by the FAST Act, FTA has entered into an agreement with the National Academies of Sciences, Engineering, and Medicine, to conduct a study to evaluate whether it is in the public interest to withhold from federal or state court proceedings any information collected by DOT through its public transportation safety program oversight activities. The National Academies of Sciences is expected to complete this study in 2018. FTA is taking positive steps toward developing safety regulations that may address inconsistent safety practices across rail transit operators. FTA officials stated that the agency is considering issuing rail transit safety regulations and also employs additional tools to compel rail transit entities to adopt safety measures. As noted above, FTA is currently studying whether federal regulations are appropriate for specific areas of rail transit safety. Executive Order 12866 and OMB’s Circular A-4 direct federal agencies to consider performance-based regulations when developing regulations. Further, as the Transportation Research Board recently reported, any decision to use performance-based regulations “must take into account the regulator’s own ability to enforce and motivate compliance (through methods such as auditing and field inspections) as well as the capacity of regulated entities to meet their obligations.” FTA officials noted that they are actively engaged with members of the Transportation Research Board in reviewing and discussing these recent findings related to safety regulations for high-hazards industries. In January 2017, FTA issued its National Public Transportation Safety Plan, which FTA officials noted is one component of their transit safety standard development program. According to FTA officials, the plan identifies a list of issue areas that the agency is currently studying to determine whether national regulations are needed. FTA officials also stated that the plan includes “voluntary standards,” which are intended to put the industry “on notice” that federal safety regulations may be proposed in those areas. FTA officials stated that they view the National Public Transportation Safety Plan as iterative and more easily updated compared with official regulations. Additional tools that FTA officials stated the agency employs in its approach to safety oversight include general directives as well as the requirements associated with FTA grants. Based on our assessment and studies we reviewed, a strength of FRA’s safety oversight program is its risk-based approach to distributing inspection resources, which may serve as an example for FTA and state safety agencies. According to NTSB, FRA’s qualified inspectors are a strength of its oversight program. To help target these inspectors to the areas of highest risk, FRA developed the National Inspection Plan, which includes a quantitative model for allocating inspection resources in a way that tries to minimize railroad accidents. This model utilizes data including: (1) accident and incident data that railroads are required to report, (2) data from FRA inspection activity, and (3) information on railroad activities such as train miles and other data. Based on our assessment of FRA’s model, we believe that it can be an appropriate and useful tool for directing its inspection resources based on risk because it relies on statistical methods commonly used to predict the risk of a violation for regulated entities. While we did not review FRA’s entire modeling process, nor did we validate the results it generates, we do believe that FRA’s approach to using these statistical models as a key part of its inspection program is appropriate. However, a potential limitation of FRA’s inspection program is the flexibility granted to individual inspectors and whether the manner and extent to which inspectors implement this discretion may be inconsistent with the risk-based National Inspection Plan. FRA’s National Inspection Plan provides guidance for inspectors about how much time they should spend inspecting individual railroads. According to FRA officials, FRA inspectors have considerable flexibility to deviate from the National Inspection Plan based on their judgment regarding where to more effectively use their resources. FRA officials stated that situations arise that call for deviations in planned inspections. For example, a particular railroad may experience a serious accident and therefore require more oversight from FRA. According to FRA officials, regional offices make these decisions based on their understanding of emerging issues. Inspectors are expected to know their region and decide which locations to go to, and are in part evaluated based on these decisions. When a region’s record of total inspection time spent on a particular railroad differs from the National Inspection Plan by more than 5 percent, the region’s leadership submits an explanation to FRA’s Office of Railroad Safety. This practice, if not monitored, could allow inspectors to deviate from the data-driven model results in ways that undermine the goal of the National Inspection Plan to deploy FRA’s limited resources efficiently and based on risk. However, FRA officials told us that flexibility for individual inspectors is important, and that FRA is continuously monitoring the model’s performance and making changes as appropriate. Further, OECD’s Best Practice Principles for Regulatory Policy note that it is important to ensure “that sufficient flexibility is left to enforcement and inspection officials to adapt their response in proportion to the facts on the ground.” A strength of FTA’s approach to rail transit safety oversight is that it is working to overcome weaknesses in state oversight of rail transit identified in our prior work and stakeholders we spoke with. For example, FTA has noted that in the past some state safety agencies lacked sufficient oversight authorities. To now be certified by FTA, state safety agencies must demonstrate that they have authority to review, approve and oversee the implementation of rail transit operator’s safety plans. Additionally, we have found, and FTA has also noted, that some state safety agencies would benefit from more training and additional staff. To now be certified by FTA, state safety agencies must be capable of directly hiring and developing staff and contract support, as well as have a training plan for certain staff. Though FTA is seeking to implement stronger safety oversight activities, a limitation of its program is that state safety agencies have not received the guidance and support necessary to develop effective inspection programs. FTA does not currently plan to conduct widespread inspections itself and recommends that state safety agencies develop risk-based inspection programs. According to FTA, states have discretion to establish their inspection programs in accordance with their program standards, and are not required to actually conduct inspections as the method of verifying rail transit operators’ compliance with safety rules. However, direct observation, audits, and performance indicator tracking are useful methods for an oversight agency in assessing a regulated entity’s safety culture. Officials we spoke with from selected state safety agencies say that they have received little guidance from FTA on what their risk-based inspection programs should look like. In the materials FTA provided to states, it said that it intends to provide guidance to states on risk-based inspections but did not provide us with a plan or timeline for doing so. Without guidance from FTA, state safety agencies may not develop effective risk-based inspection programs and thus not use their resources efficiently. Effective risk-based inspection programs are particularly important given state safety agencies’ limited resources. We have reported in the past that some state safety agencies lack sufficient resources, including training and staff. Officials from two rail transit operators and all four industry associations we spoke with stated that state safety agencies continue to have limited resources and capacity. Several state safety agencies we spoke with rely on contractors or employees with other responsibilities besides oversight of rail transit to meet their increased oversight responsibilities and achieve certification from FTA. Federal standards for internal control as well as leading practices for regulatory inspections state that agency objectives, including those related to inspections and enforcement, should be clearly communicated. Specifically, federal standards for internal control require that management communicate the necessary quality information to achieve the agency’s objectives. Additionally, the OECD’s Best Practice Principles for Regulatory Policy recommends that governments ensure clarity of rules and processes for enforcement and inspections and clearly articulate rights and obligations of officials. According to OECD, the frequency of inspections and the resources employed should be proportional to the level of risk. A strength of FRA’s safety oversight program is that the agency has and utilizes clear enforcement authority, according to NTSB and stakeholders we spoke with. As previously discussed, FRA has several enforcement tools available when inspectors find that railroads are noncompliant with applicable regulations, including civil penalties, individual liability, compliance orders, and emergency orders. According to NTSB, this array of specific enforcement tools helps ensure safety deficiencies are addressed by railroads. FRA officials also told us that the process of adjudicating civil penalties provides a forum for FRA and railroad officials to meet to discuss safety issues. Four rail operators also told us that FRA’s authority to issue civil penalties is necessary to ensure railroads’ compliance with regulations. However, a potential limitation of FRA’s approach to enforcement is that it is difficult to quantify the effectiveness of FRA’s civil penalties. FRA has reported that it cannot determine whether observable safety improvements are directly attributable to discrete civil penalties or whether the amount of civil penalties has any effect on safety. We have reported in the past about the challenges of determining the effect of penalties on compliance in tax policy, though we also noted that, despite these challenges, some analyses likely would be useful for better understanding the effect of penalties on compliance. FRA also reported, though, that according to the judgments of its inspectors, issuing civil penalties yields observable improvements in safety practices and compliance with the law. Further, according to FRA, though it does not quantify the impact of civil penalties, FRA monitors railroad responses to its enforcement activity and adjusts its oversight as necessary. More broadly, civil penalties are not meant, by themselves, to ensure railroad safety. Instead, FRA reported that it uses its entire regulatory regime as a whole to try and ensure safety. FRA officials also noted that the agency has additional tools, apart from civil penalties, to compel railroads to adopt safety practices. A strength of FTA’s rail transit safety oversight is that it seeks to improve historically weak state safety agency enforcement authorities, as described in our previous report as well stakeholders we spoke with. FTA requires state safety agencies to adopt enforcement authorities that are sufficient to enable states to compel action from rail transit agencies to address identified deficiencies. FTA has also communicated to states that it is focusing its evaluation of each state’s enforcement authorities in two major areas: ensuring that the state can carry out its primary responsibility for rail transit safety in response to (1) an imminent threat to public safety on the rail transit system and (2) a lack of action or non- compliance from the rail transit operator in carrying out certain safety plans. FTA has provided states with examples of the enforcement authorities and policies state safety agencies could establish to address these specific concerns. Authorities to address imminent safety threats may include the authority to suspend rail transit agencies’ operations, inspect and remove deficient equipment or system infrastructure from service, or issue an order requiring the rail transit agency to correct an unsafe condition prior to placing equipment or infrastructure back into passenger service. FTA has also provided state safety agencies with examples of authorities to address a lack of action or cooperation by the rail transit operator, including the authority to withhold or redirect funds, levy civil or criminal fines or penalties, and a formal citation or ticketing program. Though federal law requires that state safety agencies have enforcement authorities over the safety of the rail transit entities they oversee, a limitation of FTA’s approach is that FTA has not developed a method to evaluate the effectiveness of states’ enforcement practices. Certified state safety agencies will be evaluated for continued compliance with FTA regulations every 3 years. This triennial review process (for rail transit safety) seeks to ensure that states are effectively carrying out their responsibilities. While FTA officials told us that they will evaluate state safety agencies’ enforcement during the triennial reviews, FTA has not developed a process or methodology to evaluate whether state enforcement authorities and practices as a whole are effective. Without a method for determining the effectiveness of state safety agencies’ enforcement, FTA may not have the information needed to identify ineffective safety enforcement. As a result, deficiencies may remain for long periods of time, potentially contributing to safety incidents. Federal standards for internal control maintain that agency managers should perform a range of practices that would facilitate the establishment of a system to monitor the effectiveness of agency activities, which in the context of FTA’s mission includes the effectiveness of its rail transit safety oversight. Agency managers should define objectives clearly to enable the identification of risks, establish activities to monitor performance measures and indicators, and externally communicate the necessary quality information to achieve the entity’s objectives. Internal control standards further stipulate that agency managers should perform monitoring activities regarding their internal control system and evaluate the results. To do so, federal standards for internal control state that agency managers should monitor ongoing operations and effectiveness, evaluate the results of this monitoring, and identify any changes that need to be made to achieve improvement in agency operations. The effectiveness of state safety agency enforcement is especially important because questions have been raised about the efficacy of FTA’s own enforcement mechanisms, including its ability to withhold funds from and assume direct control over safety oversight for a rail transit entity. Rail transit operators and industry association representatives we spoke with stated that FTA’s authority to withhold funding from states is overly punitive, and two stakeholders said the FTA needs more precise tools. Officials from an industry association added that withholding funds can be counterproductive, as most state safety agencies are already underfunded and understaffed. FTA officials pointed to examples in which the agency successfully supported state safety agencies in compelling action from rail transit agencies as evidence that the state safety oversight model, in which FTA backs up state safety agencies, is effective. Additionally, officials from numerous state safety agencies and others questioned whether FTA has the capacity to effectively assume direct safety oversight of rail transit operators. FTA has not assumed direct safety oversight of any rail transit operators outside of WMATA, and FTA officials noted that they intend to continue supporting state safety agencies in their oversight wherever possible. The approaches to rail safety oversight utilized by FRA and FTA each have strengths and limitations. However, FTA’s program is currently in transition as the agency implements new authorities and responsibilities provided in federal law. Though FTA has made progress by evaluating existing rail transit safety standards and providing some guidance to states as part of the certification process, limitations in FTA’s approach may still hinder the success of the state-based rail transit safety oversight program. Given the looming 2019 deadline for state safety oversight programs to achieve FTA certification, FTA can improve its efforts to implement its new rail transit safety oversight program. In particular, without guidance from FTA on how to develop and carry out risk-based inspection programs, state safety agencies may not use limited resources efficiently, risking that important safety issues will go undetected. Further, without a method for how it will monitor the effectiveness of states safety agencies’ enforcement, FTA will lack the information needed to identify ineffective state enforcement, which risks allowing safety deficiencies to remain for long periods of time. By providing this additional guidance and direction to the state safety agencies, FTA would help ensure that states are able to effectively identify and resolve rail transit safety issues. We are making the following two recommendations to FTA: The Office of Transit Safety and Oversight should create a plan, with a timeline, for developing guidance for state safety agencies about how to develop and implement a risk-based inspection program. (Recommendation 1) The Office of Transit Safety and Oversight should develop and communicate a method for how it will monitor the effectiveness of the enforcement authorities and practices of state safety agencies. (Recommendation 2) We provided a draft copy of this report to DOT, NTSB, and WMATA for review and comment. In written comments, reproduced in appendix I, DOT agreed with both our recommendations. DOT also provided technical comments, which we incorporated as appropriate. In e-mails, NTSB and WMATA provided technical comments, which we incorporated as appropriate. NTSB noted that we do not discuss the role of system safety initiatives, such as safety management systems, in the FRA and FTA rail safety oversight programs. We agree with NTSB that system safety concepts are increasingly influencing the FRA and FTA approaches to rail safety oversight but, as NTSB also noted, both FRA and FTA lack finalized regulations codifying their approaches to system safety initiatives. Because the extent of rail entities’ implementation of these initiatives varies and is not complete we did not include an assessment of the strengths and limitations of those initiatives in our scope. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of Transportation, Chairman of NTSB, General Manager of WMATA, and the appropriate congressional committees. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or goldsteinm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Individuals who made key contributions to this report are listed in appendix II. In addition to the contact named above, Steve Cohen (Assistant Director); Kyle Browning (Analyst in Charge); Melissa Bodeau; Lacey Coppage; Serena Lo; Sean Miskell; and Josh Ormond made key contributions to this report.

 

 Report 2:
NASA’s Commercial Crew Program is a multi-phased effort that began in 2010. Across the phases, NASA has engaged several companies, using both agreements and contract vehicles to develop and demonstrate crew transportation capabilities. As the program has passed through these phases, NASA has generally narrowed down the number of participants. The early phases of the program were under Space Act agreements, which is what NASA calls the agreements entered into pursuant to its other transaction authority. These types of agreements are generally not subject to the Federal Acquisition Regulation (FAR) and allow the government and its contractors greater flexibility in many areas. Under these Space Act agreements, NASA relied on the commercial companies to propose specifics related to their crew transportation systems, including their design, the capabilities they would provide, and the level of private investment. In these phases, NASA provided technical support and determined whether the contractors met certain technical milestones. In most cases, NASA also provided funding. For the final two phases of the program, NASA awarded FAR-based contracts. By using FAR-based contracts, NASA gained the ability to procure missions to the ISS, while continuing to provide technical expertise and funding to the contractors. NASA levied two sets of requirements on the contractors: the ISS program requirements, which must be met by all spacecraft visiting the ISS whether they carry cargo or crew; and the Commercial Crew Program requirements, which have a focus on system capabilities and safety rather than design. The program also established a verification closure notice process, in which the contractors submit data to NASA to verify they have met all the requirements to be certified. This certification must occur before contractors are allowed to fly initial crewed missions to the ISS. In September 2014, NASA awarded firm-fixed-price contracts to Boeing and SpaceX, valued at up to $4.2 billion and $2.6 billion, respectively, for the Commercial Crew Transportation Capability phase. Under a firm- fixed-price contract, the contractor must perform a specified amount of work for the price negotiated by the contractor and government. This is in contrast to a cost-reimbursement contract, in which the government generally agrees to pay the contractor’s allowable costs regardless of whether work is completed. During this phase, the contractors will complete development of crew transportation systems. Boeing’s spacecraft—CST-100 Starliner—is composed of a crew module and a service module. The crew module will carry the crew and cargo. It also includes communication systems, docking mechanisms, and return systems for Earth landing. The service module provides propulsion on-orbit and in abort scenarios as well as radiators for thermal control. SpaceX’s spacecraft—Dragon 2—is composed of a capsule, which we refer to as the crew module, and a trunk, which we refer to as the support module. The crew module is composed of a pressure section and a service section. This module will carry the crew and cargo. It also includes avionics, docking mechanisms, and return systems for a water landing. The support module includes solar arrays for on-orbit power and guidance fins for escape abort scenarios. Figure 1 shows the spacecraft and launch vehicles for Boeing and SpaceX’s crew transportation systems. The Commercial Crew Transportation Capability phase contracts include three types of services: Contract Line Item 001 encompasses the firm-fixed-price design, development, test, and evaluation work needed to support NASA’s certification of the contractor’s spacecraft, launch vehicle, and ground support systems. Contract Line Item 002 covers any service missions that NASA orders to transport astronauts to and from the ISS. Under this indefinite-delivery, indefinite-quantity line item, NASA has ordered six post-certification missions from each contractor. Each service mission is its own firm-fixed-price task order. NASA must certify the contractors’ systems before they can fly these missions. Contract Line Item 003 is an indefinite-delivery, indefinite-quantity line item for any special studies, tests, and analyses that NASA may request. These tasks do not include any work necessary to accomplish the requirements under contract line item 001 and 002. As of April 2018, NASA had funded studies worth approximately $30 million to Boeing, including approximately $27 million for additional testing of the parachute system. NASA had funded studies worth approximately $44 million to SpaceX, including approximately $34 million for additional testing of the parachute system. For each contractor, the maximum value of this contract line item is $150 million. NASA has made changes to the contracts that have increased their value. While the contracts are fixed-price, their values can increase if NASA adds work or otherwise changes requirements, among other means. As of April 2018, NASA requirement changes had increased the value of contract line item 001 for Boeing by approximately $191 million and for SpaceX by approximately $91 million. NASA divided the certification work under contract line item 001 into two acceptance events: the design certification review and the certification milestone. An acceptance event occurs when NASA approves a contractor’s designs and acknowledges that the contractor’s work is complete and meets the requirements of the contract. The first acceptance event—the design certification review—verifies the contractor’s crew transportation system’s capability to safely approach, dock, mate, and depart from the ISS, among other requirements. After the contractor has successfully completed all of its flight tests, as well as various other activities, the second acceptance event—the certification milestone—determines whether the crew transportation system meets the Commercial Crew Program’s requirements. Following this contract milestone is an agency certification review, which authorizes the use of a contractor’s system to transport NASA crew to and from the ISS. Figure 2 shows a notional path leading up to the agency certification review. The Commercial Crew Program’s certification plan outlines how the program will incrementally review required deliverables leading up to, and supporting, the agency certification review. For each review, the plan describes the information that the contractor and the program will present. At the agency certification review, which is chaired by the Associate Administrator of the Human Exploration and Operations Mission Directorate, the agency will review the program’s formal recommendation to certify the contractor’s crew transportation system. Program officials said that their goal is to develop and review certification evidence incrementally in order to reduce the risk that issues will be identified during the agency certification review. In our February 2017 report, we evaluated the progress made by the two contractors on the Commercial Crew Program and found the following: Both of the Commercial Crew Program’s contractors had made progress developing their crew transportation systems, but both also had aggressive development schedules that were increasingly under pressure. We reported that both Boeing and SpaceX had determined that they would not be able to meet their original 2017 certification dates, and both expected certification to be delayed until 2018. We found that the schedule pressures were amplified by NASA’s need to provide a viable crew transportation option because its contract with Russia’s space agency was to provide crew transportation to the ISS for six astronauts through 2018 with rescue and return through late spring 2019. Purchasing additional seats from Russia involves a contracting process that typically takes 3 years. Without a viable contingency option for ensuring uninterrupted access to the ISS in the event of further Commercial Crew delays, we concluded that NASA was at risk of not being able to maximize the return on its multibillion dollar investment in the space station. The Commercial Crew Program was using mechanisms laid out in its contracts to gain a high level of visibility into the contractors’ crew transportation systems, but maintaining that level of visibility through certification could add schedule pressures. We noted that, for example, due to NASA’s acquisition strategy for this program, its personnel were less involved in the testing, launching, and operation of the crew transportation system. While the program developed productive working relationships with both contractors, obtaining the level of visibility that the program required had also taken more time than the program or contractors had anticipated. Ultimately, we noted that the program had the responsibility for ensuring the safety of U.S. astronauts, and its contracts gave it deference to determine the level of visibility required to do so. We concluded that the program office could face difficult choices moving forward about how to maintain the level of visibility it feels it needs without adding to the program’s schedule pressures. In order to ensure that the United States had continued access to the ISS if the Commercial Crew Program’s contractors experienced additional schedule delays, we recommended in our February 2017 report that the NASA Administrator develop a contingency plan for maintaining a presence on the ISS beyond 2018, including options to purchase additional Russian Soyuz seats, and report to Congress on the results. NASA concurred with this recommendation, and in February 2017, NASA executed a contract modification that purchased two seats and included an option to purchase three additional crewmember seats from Boeing on the Russian Soyuz vehicle. These seats represent a contingency plan for U.S. access to the ISS through 2019. In April 2017, NASA informed the Congress of this action. Boeing and SpaceX continue to make progress developing their crew transportation systems, but both contractors have further delayed the certification milestone to early 2019. These changes have occurred as the contractors continue to work to aggressive schedules, and they have had to delay key events regularly. Further delays are likely as the Commercial Crew Program’s schedule risk analysis shows that the certification milestone is likely to further slip. In addition, as of mid-June 2018, NASA officials told us that these dates may change soon but that both contractors have not yet provided official updates to their schedules to NASA. NASA has not fully shared information with Congress regarding the risks of future schedule delays for the contractors and, as a result, Congress lacks insight into when the contractors will be certified. Also, there may be a gap in access to the ISS if the Commercial Crew Program experiences additional delays. While NASA has begun to discuss potential options, it currently does not have a contingency plan for how to ensure an uninterrupted presence on the ISS beyond 2019. Boeing and SpaceX have continued to make progress finalizing their designs and building hardware as they work toward their certification milestones. The contractors are manufacturing test articles to demonstrate system performance and flight spacecraft to support the uncrewed and crewed flight tests, which are expected to demonstrate the ability to meet contract requirements. As table 1 shows, these test articles and spacecraft vary in levels of completion. Some are built and undergoing testing while others are starting the manufacturing phase. Should any issues arise during integration and test or the flight tests, the contractors may have to complete rework on the spacecraft already under construction. While both contractors are making progress, the Commercial Crew Program is tracking risks that each contractor has to address through testing and other means as they work towards the certification milestone. As we have previously reported, these types of risks are inherent in NASA’s major acquisitions, which are highly complex, specialized, and often pushing the state of the art in space technology, but they could also delay the contractors’ progress if issues arise during testing. The Commercial Crew Program’s top programmatic risks identified for Boeing include challenges related to its abort system performance, parachutes, and launch vehicle. Abort System: Boeing is addressing a risk that its abort system, which it needs for human spaceflight certification, may not meet the program’s requirement to have sufficient control of the vehicle through an abort. In some abort scenarios, Boeing has found that the spacecraft may tumble, which could pose a threat to the crew’s safety. To validate the effectiveness of its abort system, Boeing has conducted extensive wind tunnel testing and plans to complete a pad abort test in July 2018. Parachute System: Boeing is also addressing a risk that during descent, a portion of the spacecraft’s forward heat shield may re- contact the spacecraft after it is jettisoned and damage the parachute system. Boeing’s analysis indicates the risk exists only if one of two parachutes that pull the forward heat shield away from the spacecraft does not deploy as expected, and that potential re-contact is non- detrimental. However, NASA’s independent analysis indicates that this may occur even if both parachutes deploy as expected. If the program determines this risk is unacceptable, Boeing would need to redesign the parachute system, which the program estimates could result in at least a 6-month delay. Launch Vehicle Data: One of the program’s top programmatic and safety concerns is that it may not have enough information from Boeing’s launch vehicle provider, United Launch Alliance, to assess whether the Atlas V launch vehicle prevents or controls cracking that could lead to catastrophic failures. NASA estimates that unfinished work in this area could take Boeing and the United Launch Alliance until the fourth quarter of 2018 to complete. Additionally, the first stage of the Atlas V is powered by the Russian built RD-180 engine, and, according to program and Boeing officials, access to its data is highly restricted by agreements between the U.S. and Russian governments. Since our last report, the Commercial Crew Program has lowered the risk that certification of the launch vehicle might not occur by negotiating steps to access necessary data, but work is still ongoing. The Commercial Crew Program’s top programmatic risks identified for SpaceX are in part related to ongoing design and development efforts related to its launch vehicle design, the Falcon 9 Block 5. Composite Overwrap Pressure Vessel: This Block 5 design includes SpaceX’s redesign of the composite overwrap pressure vessel, which is intended to contain a gas under high pressure. SpaceX officials stated the newly designed vessel aims to eliminate risks identified in the older design, which was involved in an anomaly that caused a mishap in September 2016. SpaceX plans to qualify the updated design for flight prior to the uncrewed flight test design certification review. Engine Turbine Cracking: The Block 5 design also includes design changes to address cracks in the turbine of its engine identified during development testing. NASA program officials told us that they had informed SpaceX that the cracks were an unacceptable risk for human spaceflight. SpaceX officials told us that they have made design changes to this Block 5 upgrade that did not result in any cracking during initial testing. However, this risk will not be closed until SpaceX successfully completes qualification testing in accordance with NASA’s standards without any cracks. As of March 2018, SpaceX had not yet completed this testing. Propellant Loading Procedures: Both the program and a NASA advisory group have raised SpaceX’s plan to fuel the launch vehicle after the astronauts are on board the spacecraft to be a potential safety risk. In the May 2018 meeting minutes, however, the Aerospace Safety Advisory Panel stated that with appropriate controls in place, this approach could be a viable option for the program to consider. SpaceX’s perspective is that this operation may be a lower risk to the crew because it reduces the crew exposure time while the launch vehicle is being loaded with propellant. To better understand the propellant loading procedures, the program and SpaceX agreed to demonstrate the loading process five times from the launch site in the final crew configuration prior to the crewed flight test. The five events include the uncrewed flight test and the in-flight abort test. Therefore, delays to those events would lead to delays to the agreed upon demonstrations, which could in turn delay the crewed flight test and certification milestone. Both contractors have notified NASA that their certification milestones have slipped to January 2019 for Boeing and February 2019 for SpaceX, but the Commercial Crew Program’s schedule risk analysis indicates more delays are likely. This analysis identifies a range for each contractor, with an earliest and latest possible completion date, as well as an average. In April 2018, the program’s schedule risk analysis found there was zero percent chance that either contractor would achieve its current proposed certification milestone. The analysis’s average certification date was December 2019 for Boeing and January 2020 for SpaceX. Figure 3 shows the original Boeing and SpaceX contract schedules and the current proposed schedule for five key events in each contract, as well as NASA’s schedule risk analysis for the certification milestone. Each month, the program updates its schedule risk analysis based on the contractors’ internal schedules as well as program officials’ perspectives and insight into specific technical risks. The Commercial Crew Program manager told us that differences between the contractors’ proposed schedules and the program’s schedule risk analysis include: The contractors are aggressive and use their schedule dates to motivate their teams, while NASA adds additional schedule margin for testing. Both contractors assume an efficiency factor in getting to the crewed flight test that NASA does not factor into its analysis. The program manager also told us that the program meets with each contractor monthly to discuss schedules and everyone agrees to the relationships between events in the schedule even if they disagree on the length of time required to complete events. The program manager added, however, that she relies on her prior experience to estimate schedule time frames as opposed to relying on the contractors’ schedules, which are often optimistic. Our analysis also shows that the contractors often delay their schedules. Both contractors have repeatedly stated that their schedules are aggressive and have set ambitious—rather than realistic—dates, only to frequently delay them. Since the current contracts were awarded in 2014, the Commercial Crew Program has held 13 quarterly reviews for each contractor. For the five key events identified above, Boeing has reported a delay at 7 of those quarterly reviews and SpaceX has reported a delay at 9 of them. In mid-June 2018, NASA officials told us that the dates for these key events may change soon. The information presented in Figure 3 above is based on first quarter calendar year 2018 data. NASA officials stated both contractors have not yet officially communicated new schedule dates to NASA as of the second quarter calendar year 2018. We found that both contractors have updated schedules that indicate delays are forthcoming for at least one key event, but NASA officials told us they lack confidence in those dates until they are officially communicated to NASA by the contractors. As a result, NASA is managing a multibillion dollar program without confidence in its schedule information as it approaches several big events, including uncrewed and crewed flight tests. The risk of future delays in the contractors’ schedules is critical information that NASA has not fully shared with Congress. Moreover, NASA has not yet developed a contingency plan to address the potential gaps that these delays could have on U.S. access to the ISS after 2019. Specifically, in the Explanatory Statement accompanying the fiscal year 2018 Consolidated and Further Continuing Appropriations Act, the House Appropriations Committee stated its expectation that NASA report quarterly to the Senate and House Committees on Appropriations on the status of the Commercial Crew Program contracts. Previously, members of Congress had asked for this information in order to ensure that Congress had adequate insight into this program. While NASA includes both contractors’ proposed schedules in its quarterly report to Congress, NASA does not include the results of its own schedule risk analysis. Given the frequency with which the contractors delay key events in their schedules, the program’s schedule risk assessment provides valuable insight into potential delays that NASA currently is not providing to Congress. In addition, as previously mentioned, NASA executed a contract modification that purchased two seats and included an option to purchase three additional crew member seats through Boeing for an undisclosed value and reported this action to Congress in April 2017. Ultimately, the option was exercised, and NASA purchased a total of five seats on four different Soyuz flights. Boeing obtained these seats through a separate settlement with the Russian firm RSC Energia, which manufactures the Soyuz. These seats were intended to serve as a contingency plan based on schedule information available at that time. However, subsequent delays, as well as the risk of future delays as discussed above, indicate that this contingency plan will likely no longer be sufficient. The earliest and latest possible completion dates for certification in NASA’s April 2018 schedule risk analysis indicate it is possible that neither contractor would be ready before August 2020, leaving a potential gap in access of at least 9 months. We calculated the potential gap based on the contractor certification milestone dates, but there could be some additional time required between that review and the first post-certification service mission to the ISS. As seen in figure 4, if the contractors can maintain their current proposed schedules for their respective certification milestones, a gap in access to the ISS is not expected. However, there would be a gap in access to the ISS if neither contractor has its certification milestone before November 2019, which is when NASA expects the final Russian Soyuz seat for a U.S. astronaut to return. Senior NASA officials told us that sustaining a U.S. presence on the ISS is essential to maintain and operate integral systems, without which the ISS cannot function. Given the importance of maintaining a U.S. presence on the ISS, NASA officials have stated they are working on options to address the potential gap in access. However, officials told us that planning for contingencies is difficult given the extensive international negotiations required for some options. Obtaining additional Soyuz seats seems unlikely, as the process for manufacturing the spacecraft and contracting for those seats typically takes 3 years—meaning additional seats would not be available before 2021. As a result, according to NASA’s Associate Administrator for Human Exploration and Operations, the options NASA is considering include: Refine the remaining Soyuz launch schedule to allow for a return in January 2020, as opposed to November 2019. This would provide 2 additional months of access to the ISS before the commercial crew flights need to start. Use the crewed flight tests as operational flights to transport U.S. astronauts to and from the ISS. In March 2018, NASA modified Boeing’s contract to allow NASA to add a third crew member and extend the length of the flight test, if NASA chooses to do so. This would have limited usefulness, however, in filling a potential gap in access to the ISS if the schedule for Boeing’s crewed flight test slips past the return date for the last Soyuz flight and SpaceX also continues to experience delays. NASA’s Associate Administrator for Human Exploration and Operations stated that he is “brainstorming” other options to ensure access to the ISS but does not have a formal plan. While options are not unlimited and decisions have to be made within the context of the current geopolitical environment, Congress stated in the NASA Authorization Act of 2005 that it is U.S. policy to possess the capability for human access to space on a continuous basis. In 2010, Congress further stated that one of the key objectives of the United States’ human spaceflight policy is to sustain the capability for long-duration presence in low-Earth orbit through full utilization of the ISS. If NASA does not develop options for ensuring access to the ISS in the event of further Commercial Crew delays, it will not be able to ensure that the U.S. policy goal and objective for the ISS will be met. The Commercial Crew Program relies on several contractual mechanisms to assess safety throughout the certification process, and those mechanisms are in varying stages of completion. The program itself, its contractors, and two of NASA’s independent review organizations have raised concerns about the program’s ability to assess and evaluate all of the deliverables in a timely manner. In addition, one of the key safety requirements levied by the program is loss of crew, which captures the probability of death or disability to a crew member. NASA does not have a consistent approach for how to incorporate key inputs to assess this metric, which means the agency as a whole may not clearly capture or document its risk tolerance with respect to loss of crew. Further, the program’s chief safety and mission assurance officer is dual hatted to serve simultaneously in a programmatic position as well as the program’s safety technical authority. This approach creates an environment of competing interests because it relies on the same individual to manage technical and safety aspects on behalf of the program while also serving as the independent oversight of those same areas. The contractors are required to provide several key deliverables to the Commercial Crew Program, which inform the agency certification review and help NASA determine the level of risk it is accepting with respect to safety of each spacecraft. As described below, these deliverables are in varying stages of completion and the program itself, its contractors, and two of NASA’s independent review organizations have raised concerns about the program’s ability to assess and evaluate all of the deliverables in a timely manner. Certification Data Package. Among other things, the certification data package includes a list of seven system safety assessments. For example, the certification data package includes a fault tolerance assessment, which describes the system’s ability to sustain a certain number of undesired events, such as software or operational anomalies. A human error analysis—one of the seven assessments in the data package—evaluates human errors to minimize their negative effects on the system. Boeing held its uncrewed flight test design certification review in December 2017 and submitted its certification data package for NASA approval. Boeing plans three more updates to this data package prior to the final certification milestone. SpaceX has begun to submit data and plans to submit its final certification data package as part of its crewed flight test design certification review, which is scheduled for September 2018. According to the program’s certification review plan, program officials will review and approve the contractors’ certification data packages, which will be used to inform the agency certification review. Phased Safety Review Process. A three-phased safety review process informs the program’s quality assurance activities, and it is intended to ensure that the contractors have identified all safety-critical hazards and implemented associated controls prior to the first crewed flight test. In phase one, the contractors identified risks in their designs and developed reports on potential hazards, the controls they put in place to mitigate them, and explanations for how the controls will mitigate the hazards. In phase two, which is nearing completion, the program reviews and approves the contractors’ hazard reports and develops strategies to verify and validate that the controls are effective. For example, if a control requires that an item be waterproofed, verification and validation strategies could include inspections and tests to confirm that the item is waterproof. As of April 2018, the program had yet to complete this phase, having approved 97 percent of Boeing’s phase two reports and 72 percent of SpaceX’s phase two reports. In phase three, the contractors will conduct the verification activities and submit the hazard reports to the program for approval. The program has begun phase three, including approving 19 percent of Boeing’s phase three reports. Program Requirements. While the program manager told us that all of the requirements contribute to the safety of the commercial systems, safety officials are required to approve a subset of these requirements. Examples of requirements approved by safety officials include the ability to leave the spacecraft in an emergency or to abort a launch. When a contractor is ready for NASA to verify that it has met a requirement, the contractor submits data for NASA to review through a verification closure notice. We define “safety-specific notices” as those requiring safety officials’ approval. As shown in table 2, as of March 2018, the program had approved 2 percent of Boeing’s safety-specific notices and 0 percent of SpaceX’s safety-specific notices. Testing. The program also requires testing to verify and validate the crew transportation system. Agency officials emphasized the importance of testing to safety, stating that testing reduces uncertainty about a system’s performance and can uncover unknown problems. As noted above, both contractors will be conducting an uncrewed and a crewed flight test prior to being certified. While a certain level of risk needs to be accepted to conduct human spaceflight, these flight tests help to mitigate this risk by validating the integrated performance of the hardware and software. Agency and program officials stated that the contractors’ flight tests are critical evidence to support certification of a safe and reliable system. As evidenced by the data above, the program still has a significant amount of work ahead with respect to approving certification packages and closing hazard reports and verification closure notices. We have previously found that the program’s workload was an emerging schedule risk, and the contractors have continued to express concern about program officials’ ability to process and approve certification paperwork in a timely manner. Workload has also been a concern for two of NASA’s independent review organizations. For example, the Aerospace Safety Advisory Panel noted in its January 2018 annual report that the sheer volume of work that remains for the program in terms of closing hazard reports and verification closure notices is significant. In addition, the program’s safety and mission assurance office identified the upcoming bow wave of work in a shrinking time period as a top risk to achieving certification. One mechanism the program put in place to assess the overall safety of each spacecraft—loss of crew—has been a focus of the Aerospace Safety Advisory Panel, Members of Congress, our prior work, and the program itself. Loss of crew captures the probability of death or permanent disability to one or more crew members. It has received a lot of attention, in part, because it has been a top risk for the program since 2015. Specifically, the program has been concerned that neither contractor would be able to meet the contract requirement of a 1 in 270 probability of incurring loss of crew. We identified two key concerns with how NASA is using the loss of crew metric: (1) inconsistent approaches to assess the loss of crew metric and (2) no identified plan to share lessons learned about using the loss of crew metric as a safety threshold. A loss of crew value is generated through a probabilistic safety analysis, which models scenarios that could result in the loss of crew using various inputs. According to the program’s analysis, the probability of on-orbit debris damaging the vehicle has the greatest effect on a loss of crew value. This probability is informed by an orbital debris (debris) model, which was updated in 2014, after the loss of crew requirement was established. The updated debris model makes it harder to meet a loss of crew value, in part, because the modeling environment where the contractors’ systems will operate has changed. For example, the updated model includes a larger span of orbit, greater range of debris sizes, and the addition of material density classifications, which were not included in the former model. Further, the probabilistic safety analysis may include operational mitigations, such as on-orbit inspections that would include using cameras on the ISS to visually survey the spacecraft for damage, which, according to officials, makes it easier to meet a loss of crew value. NASA describes the probabilistic safety analysis as a powerful tool that should be used as part of the overall risk management process to ensure the risk associated with development and operation of a system is understood, evaluated, managed, and mitigated. However, we found differences in the approaches that officials plan to use to assess loss of crew as well as in the loss of crew value being measured that could limit the usefulness of this tool. Agency Certification. The agency certification review for each contractor will include an assessment of whether its crew transportation system meets a loss of crew threshold of 1 in 150 for missions to the ISS, which is based on a May 2011 safety memo from the Office of Safety and Mission Assurance. A loss of crew value with a higher denominator, such as 1 in 270, is harder to meet than with a lower denominator, such as 1 in 150. According to the Chief of the Office of Safety and Mission Assurance, he will assess the 1 in 150 threshold using a probabilistic safety analysis that includes the updated debris model and operational mitigations, such as the on- orbit inspections cited above. Program Office. According to program officials, they will assess whether either contractor meets a 1 in 270 loss of crew value based on a probabilistic safety analysis using the former debris model (not the updated model) and not including operational mitigations. Contracting Officer. According to the contracting officer, each contractor’s loss of crew requirement is 1 in 270 without including operational mitigations. The contracting officer stated that SpaceX’s contract requirement uses the updated debris model in the probabilistic safety analysis, whereas Boeing’s contract requirement uses the former debris model in the probabilistic safety analysis. Program’s Chief Safety and Mission Assurance Officer. According to the program’s chief safety and mission assurance officer, he will conduct a probabilistic safety analysis using the updated debris model and will not include operational mitigations to assess whether each contractor meets a 1 in 200 loss of crew value. This loss of crew value stems from a program update that occurred after the initial contracts were signed. These different approaches are summarized in table 3 below. Agency policy requires human spaceflight programs to set a safety threshold, which NASA did for the Commercial Crew Program when it identified the 1 in 150 loss of crew threshold in the May 2011 safety memo. Subsequently, the program set more rigorous loss of crew values in contract and program documents. NASA also updated the debris model, which we previously noted makes it more difficult to meet a loss of crew value. As a result, NASA does not have a consistent approach for how to incorporate key inputs to the probabilistic safety analysis, including changes to the debris model. Instead, the risk tolerance that NASA is accepting with loss of crew varies based upon which entity is presenting the results of its probabilistic safety analysis. For example, it is possible that the program’s assessment will determine that neither contractor will meet the 1 in 270 contract requirement, but that the agency’s assessment will determine that the contractors meet the 1 in 150 agency certification value because that analysis will include operational mitigations. Or the program’s assessment could determine that Boeing meets the 1 in 270 contractual loss of crew requirement, but the agency’s assessment may determine that Boeing does not meet the 1 in 150 agency certification value because that analysis will use the updated debris model. Federal internal controls state that agency management should define objectives clearly to enable the identification of risks and define risk tolerances. Specifically, management should define risk tolerances in specific and measurable terms, so they are clearly stated and can be measured. In this case, because there will be multiple analyses conducted using different inputs, NASA risks not clearly capturing or documenting, in a coherent manner, its overall risk tolerance with respect to loss of crew before a final decision must be made on whether to certify either crew transportation system. Moreover, capturing the challenges and lessons learned from using the loss of crew metric is critical, particularly because agency officials told us that this is the first time this metric has been used as a safety threshold. Also, there are different viewpoints about the utility of the metric as a safety threshold across the agency. The program manager repeatedly told us that loss of crew is best used as a design tool. For example, program officials told us that both contractors incorporated additional orbital debris shielding into their designs to mitigate the orbital debris risk and improve their loss of crew values. In addition, the Aerospace Safety Advisory Panel reported in 2018 that loss of crew should not be viewed as an absolute measure of actual risk during operations. However, the May 2011 agency safety memo states that a breach of the loss of crew threshold would initiate a termination review of the Commercial Crew Program, which is a more strict application of the loss of crew metric. Both program and safety officials told us that, after the agency certification is complete and lessons learned are available to be compiled, sharing those lessons learned across NASA would be a good idea given the complexities associated with assessing the loss of crew metric. As of April 2018, however, agency officials said they did not have a plan for loss of crew knowledge-sharing. We have previously found that lessons learned provide a powerful method of sharing good ideas for improving work processes, facility or equipment design and operation, quality, safety, and cost-effectiveness. Further, according to NASA’s Knowledge Policy on Program and Projects, which is managed through the Office of the Chief Engineer, a principle of each center and mission directorate’s knowledge strategy is that knowledge is the cornerstone of NASA’s ability to achieve mission success. The policy acknowledges that NASA faces continuous challenges in using what it knows effectively. These challenges include, but are not limited to, enabling the identification and flow of knowledge across organizational boundaries; preserving knowledge at risk of being lost; and providing means for individuals, teams, and the organization to learn from experiences. If NASA does not capture lessons learned from the Commercial Crew Program on using the loss of crew requirement to set a program’s safety threshold and whether it met the agency’s intended goal, future programs will not be able to benefit from the knowledge gained from this multibillion dollar investment. NASA’s governance model prescribes a management structure that employs checks and balances among key organizations to ensure that decisions have the benefit of different points of view and are not made in isolation. As part of this structure, NASA established the technical authority process as a system of checks and balances to provide independent oversight of programs and projects in support of safety and mission success through the selection of specific individuals with delegated levels of authority. The technical authority process has been used in other parts of the government for acquisitions, including the Department of Defense and Department of Homeland Security. The Commercial Crew Program is organizationally connected to three technical authorities within NASA: the Office of the Chief Engineer technical authority, the Office of Chief Health and Medical technical authority, and the Office of Safety and Mission Assurance (safety) technical authority. The safety technical authority is responsible for ensuring from an independent standpoint that the program’s products and processes satisfy NASA’s safety, reliability, and mission assurance policies. The NASA safety technical authority has delegated authority through the Kennedy Space Center Director to the Chief Safety and Mission Assurance Officer for the Commercial Crew Program. We have previously reviewed how NASA has organized its technical authorities for its Exploration Systems Development organization—an organization that oversees the development of the Space Launch System, Orion crew capsule, and associated ground systems that have the goal of extending human presence beyond low-Earth orbit. In October 2017, we found that the Exploration Systems Development organization had established an organizational structure in which the technical authorities for engineering and safety and mission assurance were dual hatted simultaneously in programmatic positions. We found that having the same individual simultaneously fill both a technical authority role and a program role created an environment of competing interests, where the technical authority’s ability to impartially and objectively assess the programs while at the same time acting on behalf of the Exploration Systems Development organization in programmatic capacities may be subject to impairments. We found that this was in contrast to a recommendation from the Columbia Accident Investigation Board report—the result of an in-depth assessment of the technical and organizational causes of the 2003 Space Shuttle Columbia accident—for NASA to establish a technical authority to serve independently of the Space Shuttle program, so that employees would not feel hampered to bring forward safety concerns or disagreements with programmatic decisions. The board’s findings that led to this recommendation included a broken safety culture in which it was difficult for minority and dissenting opinions to percolate up through the hierarchy; dual center and programmatic roles vested in one person that had confused lines of authority, responsibility, and accountability and made the oversight process susceptible to conflicts of interest; and oversight personnel in positions within the program, increasing the risk that these staffs’ perspectives would be hindered by too much familiarity with the programs they were overseeing. In October 2017, we recommended that the division no longer dual hat two individuals who had both programmatic and technical authority responsibilities. As of April 2018, NASA had taken steps to separate the engineering technical authority position from the programmatic position, and NASA’s Chief of Safety and Mission Assurance said he planned to separate the safety position but had not yet completed that action. The Commercial Crew Program employs a similar structure to the Exploration Systems Development organization in that the safety technical authority is dual hatted simultaneously in a programmatic position as the Commercial Crew Program’s Safety and Mission Assurance Manager. According to the program’s safety technical authority, in his programmatic role for the program, he helps set priorities for safety issues, including how staff will be utilized to meet those priorities. In the technical authority role, he provides independent oversight in support of safety and mission success. In his dual-hatted role, this official will be responsible for endorsing the program’s certification recommendations in two different capacities: as the technical authority and as a program authority. As a result, this structure relies on the same individual to completely separate two roles—one to manage the Commercial Crew Program’s safety issues within programmatic cost and schedule constraints, and the other to assess the same issues in an independent oversight role. While the Commercial Crew Program may have an additional level of separation between the safety technical authority and the program’s involvement in the design of commercial systems due to its shared assurance model with the commercial providers, the Commercial Crew Program still maintains a structure where one individual simultaneously serves in both technical authority and programmatic roles. Figure 5 describes some of the conflicting roles and responsibilities of this official in his two different positions. During our review, officials cited several factors in support of a dual- hatted approach: The safety technical authority retains independence because his technical authority reporting path and performance reviews are not under the purview of the Commercial Crew Program chain of command. Due to the Commercial Crew Program’s shared assurance model with commercial providers, the program is operating in a quality assurance role that provides an additional level of separation between the safety technical authority and the program’s involvement in the design of commercial systems. For safety decisions involving cost and schedule where the individual who is dual hatted with both technical authority and programmatic responsibilities may feel conflicted, he stated that he would discuss these matters with his management to validate the logic behind his decision. There are cost and knowledge efficiencies gained from one individual serving in both programmatic and safety technical authority capacities. NASA’s Chief of Safety and Mission Assurance stated that he has great confidence in the individual currently serving in the dual hatted role for the Commercial Crew Program, but acknowledged there is inherent conflict even with the program’s shared assurance model. In December 2017, he stated that, based on our previous work and current discussions, he intends to decouple the programmatic and technical authority responsibilities for the Commercial Crew Program but had not done so as of April 2018. Federal internal control standards state that an agency should design control activities to achieve objectives and respond to risks, which includes segregation of key duties and responsibilities to reduce the risk of error, misuse, or fraud. By overlapping technical authority and programmatic responsibilities, NASA will continue to run the risk of creating an environment of competing interests for the Commercial Crew Program’s safety technical authority. NASA’s Commercial Crew Program is a multibillion dollar effort to facilitate the commercial development of a crew transportation system that can end the United States’ reliance on Russia to maintain an uninterrupted presence on the ISS. Boeing and SpaceX continue to make progress developing a capability to fly to the ISS, but both have continued to experience delays. Program analysis indicates risks of further delays in each contractor’s current schedule, but NASA has not provided that information to Congress in its routine briefings. Without this information, Congress does not know the full extent of potential delays to inform decision making. Additional delays could also disrupt U.S. access to the ISS. While NASA is working on potential solutions, there is no contingency plan in place to address this potential gap. Without a viable contingency plan, NASA puts at risk achievement of the U.S. goal and objective for the ISS. NASA must balance safety with acceptable risk for human spaceflight. As part of the certification process for each contractor’s spacecraft, NASA has developed one key safety metric, loss of crew. However, the complicated nature of this metric is further muddled by the inconsistent approaches being used across NASA about what inputs to be considered. As a result, there is no clear articulation of what level of risk NASA will accept with respect to this program. In addition, NASA does not have plans to capture lessons learned from how the Commercial Crew Program has used this metric to assess safety and is missing an opportunity to capture this knowledge for future human spaceflight programs. Finally, a space program’s management and oversight approach is an integral part of ensuring that human spaceflight is as safe and successful as possible. Independence of the program management and oversight functions is key to achieving the balance between safety and success. The Commercial Crew Program’s approach, however, burdens the safety technical authority with both programmatic and independent technical authority responsibilities. As a result, NASA has limited assurance that independence can be maintained as part of its institutional process to ensure safety and success. We are making the following five recommendations to NASA: The NASA Associate Administrator for Human Exploration and Operations should direct the Commercial Crew Program to include the results of its schedule risk analysis in its mandatory quarterly reports to Congress. (Recommendation 1) The NASA Administrator should develop and maintain a contingency plan for ensuring a presence on the ISS until a Commercial Crew Program contractor is certified. (Recommendation 2) The NASA Administrator should direct the Chief of Safety and Mission Assurance, the NASA Associate Administrator for Human Exploration and Operations, the Commercial Crew Program Manager, and the Commercial Crew Program Contracting Officer to collectively determine and document before the agency certification review how the agency will determine its risk tolerance level with respect to loss of crew. (Recommendation 3) After completing the agency certification review, NASA’s Chief Engineer and Chief of Safety and Mission Assurance, with support from the NASA Associate Administrator for Human Exploration and Operations and the Commercial Crew Program Manager, should document lessons learned related to loss of crew as a safety threshold for future crewed spaceflight missions, given the complexity of the metric. (Recommendation 4) The NASA Chief of Safety and Mission Assurance should restructure the technical authority within the Commercial Crew Program to ensure that the technical authority for the Office of Safety and Mission Assurance is no longer dual hatted with programmatic and independent technical authority responsibilities. (Recommendation 5) We provided a draft of this report to NASA for review and comment. NASA provided written comments that are reprinted in appendix II. In its response, NASA concurred with three of our recommendations, did not concur with one, and partially concurred with another. NASA concurred with our recommendation to develop and maintain a contingency plan to ensure a U.S. presence on the ISS and expects to take action to close this recommendation by the end of December 2018. NASA concurred with our recommendation to document lessons learned related to the loss of crew requirement and expects to take action to close this recommendation by the end of May 2019. NASA concurred with our recommendation to restructure the safety technical authority so that it is no longer dual hatted with programmatic and independent technical authority responsibilities. NASA expects to take action to close this recommendation by the end of August 2018. NASA did not concur with our recommendation that the Commercial Crew Program should include the results of its schedule risk analysis in its quarterly reports to Congress. NASA stated that it uses the contractors’ schedules as a baseline to provide qualitative statements in the NASA summary that accompanies each contractor’s quarterly reports to Congress. NASA believes that this approach is appropriate and is in accordance with the explanatory statement accompanying the Consolidated and Further Continuing Appropriations Act, 2015. NASA also stated that it will be working to ensure that the contractors’ schedules and the program’s internal assessments sync up as the program gets closer to launch. As a result, NASA explained that there will not be a requirement for a detailed NASA assessment, because the contractors’ schedule will either match NASA’s analysis or NASA will discuss its position as it has done in previous reports to Congress. We continue to believe the recommendation is valid because the program’s schedule risk analysis would provide Congress with valuable insight into potential delays, which are likely. Both contractors have repeatedly stated that their schedules are aggressive and that the dates are ambitious. As a result, we found that the contractors frequently delay dates for key events. For example, Boeing has delayed its certification milestone by 17 months and SpaceX by 22 months since the original schedules were established. The program’s recent schedule risk analysis indicates that more delays to certification are likely, but that information is not presented to Congress in NASA’s quarterly reports. Without this information, Congress does not know the full extent of potential delays to inform decision making. NASA partially concurred with our recommendation that the Chief of Safety and Mission Assurance, the NASA Associate Administrator for Human Exploration and Operations, the Commercial Crew Program Manager, and the Commercial Crew Program Contracting Officer should collectively determine and document how the agency will determine its risk tolerance level with respect to loss of crew before the agency certification review. In its response, NASA stated that it documented the agency’s risk tolerance level with respect to loss of crew for the program in its May 2011 safety memo. Further, NASA stated that it documented the requirement to limit risks to the loss of crew in a certification requirements document. NASA stated that ultimately the Commercial Crew Program is accountable for ensuring that the contractors’ systems meet the loss of crew value in this certification requirements document, which is a loss of crew value of 1 in 270. If a contractor’s system cannot meet that loss of crew value, or any other requirement, the program will request a waiver as part of the human rating certification process to ensure transparency. NASA acknowledged in its response that the existence of multiple documents defining residual risk requirements and an agency threshold for loss of crew can be confusing. NASA’s response, however, does not address our finding that it does not have a consistent approach for how to incorporate key inputs, including which debris model should be used or whether to include operational mitigations. NASA stated that it had taken action to address this recommendation; however, NASA did not outline any steps it took to resolve the concern that the risk tolerance for the loss of crew requirement depends on which entity is presenting the results of its analysis. We continue to believe that, before the agency certification review, the key parties must collectively determine how the agency will determine its risk tolerance with respect to loss of crew. We believe this approach will reduce confusion and increase transparency. We are sending copies of this report to NASA Administrator and interested congressional committees. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202)512-4841 or chaplainc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III. The objectives of our review were to assess (1) the extent to which the contractors have made progress towards meeting the National Aeronautics and Space Administration’s (NASA) certification requirements and NASA’s plans to help ensure continued access to the International Space Station (ISS); and (2) how NASA’s certification process addresses safety of the contractors’ crew transportation systems. To assess the contractors’ progress towards certification, we obtained and reviewed program and contractor documents, including monthly and quarterly updates from April 2017 through May 2018. We interviewed program and contractor officials to discuss the contractors’ recent progress, including upcoming events and any expected delays, and to understand technical risks, potential consequences, and planned mitigation activities. To identify total delays to date, we compared original contract schedules to Boeing and SpaceX’s calendar year 2018 first quarter proposed schedules, which are the most recent. Based on our review of program and contractor documents, we defined the contractors’ key events as: the uncrewed and crewed flight tests, the design certification reviews for each of those flights, and the certification milestone. We selected the two flight tests for each contractor as key events because they are intended to test key system capabilities, including the ability to launch, dock with the ISS, and return safely to Earth. We selected the design certification reviews because they verify the contractors’ crew transportation systems’ capability to safely approach, dock, mate, and depart from the ISS, among other requirements. We selected the certification milestone because it determines whether the crew transportation system meets the Commercial Crew Program’s requirements. To determine the extent to which contractors have delayed these key events over time, we analyzed the contractors’ schedule data from the 13 quarterly progress reports to date, from first quarter 2015 through first quarter calendar year 2018. We also obtained the results of the program’s April 2018 schedule risk analysis. We presented the schedule analysis range from the end of the month of the earliest possible completion date to the end of the month of the latest possible completion date. We reviewed the program’s Congressional requirements to report on cost, schedule, and technical status. Finally, to assess the potential effects of any certification delays on NASA’s access to ISS, we reviewed NASA’s contracts with Boeing and the Russian Federal Space Agency for transportation on the Soyuz vehicle. We interviewed officials from the ISS program and NASA’s Human Exploration and Operations Mission Directorate to determine if the agency had developed contingency plans to mitigate the effects of any certification delays on its access to the ISS. To assess how NASA’s certification process addresses safety of the contractors’ crew transportation systems, we reviewed agency safety policies, program plans, and contract documents to identify what safety assessments were required of the program, which safety factors would be considered in certification reviews, and when certification approval would be granted. We also interviewed program officials and the contractors about their safety policies and procedures as well as about the certification process. We identified the loss of crew requirements for the program and the contractors, and interviewed program and agency officials to determine how loss of crew would be assessed and considered throughout the certification process. To gain a broader understanding of the relative importance of loss of crew, we reviewed NASA safety policies, prior GAO reports, and annual reports from the Aerospace Safety Advisory Panel. To determine how the Orbital Debris Engineering Model (ORDEM) was updated and to assess differences between the former and updated models, we reviewed NASA documentation about the ORDEM 2000 and ORDEM 3.0 models and obtained information about the models from NASA’s Orbital Debris Program Office. To assess the program’s progress in closing requirements, including those requirements specifically related to safety, we reviewed program data for each contractor on contract requirements and closure status. We limited this analysis to requirements included in the CCT-REQ-1130 ISS Crew Transportation and Services Requirements Document because these requirements are verified by the Commercial Crew Program, whereas requirements contained within SSP 50808 ISS to Commercial Orbital Transportation Services Interface Requirements Document are managed by the ISS Transportation and Integration Office. We classified requirements as “safety-specific” when the program’s Office of Safety and Mission Assurance was listed as a verification closure notice signatory (i.e., approver). We then analyzed the program’s data to determine how many verification closure notices had been approved for all requirements and for the subset of safety-specific requirements. We reviewed program and agency documentation, such as organizational charts, program plans, and safety policies as well as interviewed program and agency officials, to determine the role of the safety and mission assurance technical authority in the program. We also reviewed the 2003 Columbia Accident Investigation Board’s Report’s findings and recommendations related to culture and organizational management of human spaceflight programs. We reviewed annual briefings and reports and met with representatives from two organizations that provide NASA with independent assessments of the program, the program’s standing review board and the Aerospace Safety Advisory Panel, to gain their perspectives on the contractor’s progress and how NASA addresses safety in its certification process. We also met with representatives from the National Transportation Safety Board and three experts with background on safety in human spaceflight in order to increase our contextual understanding of the role of safety in human spaceflight missions. We conducted this performance audit from April 2017 to July 2018 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Cristina T. Chaplain, (202) 512-4841 or chaplainc@gao.gov. In addition to the contact named above, Molly Traci, Assistant Director; Kazue Chinen; Lorraine Ettaro; Lisa Fisher; Laura Greifner; Kurt Gurka; Miranda Riemer; Juli Steinhouse; Roxanna T. Sun; Hai Tran; Kristin Van Wychen; and Alyssa Weir made significant contributions to this report.

Question: Write a one-page summary of the report 1.

Answer: In 2012 and 2015, DOT was provided with additional authority to oversee the safety of rail transit. Within DOT, FTA is now implementing this authority. The DOT's Office of Inspector General has reported, though, that FTA faces challenges in carrying out its enhanced safety oversight. FRA, also in DOT, has long carried out safety oversight of freight, intercity passenger, and commuter railroads. GAO was asked to review various rail safety and oversight issues, including the differences between FRA's and FTA's rail safety oversight programs. This report examines (1) key characteristics of FRA's and FTA's rail safety oversight programs and (2) strengths and limitations of FRA's and FTA's rail safety oversight programs. GAO assessed FRA's and FTA's information about rail safety oversight activities against guidance from the Office of Management and Budget, leading practices developed by the transit industry, and federal standards for internal control. GAO also interviewed stakeholders, including rail operators chosen based on mode, size, and location. The Department of Transportation's (DOT) Federal Railroad Administration (FRA) and Federal Transit Administration (FTA) carry out different approaches to rail safety oversight. FRA has a more centralized safety oversight program for railroads, while FTA's program for oversight of rail transit safety largely relies on state safety agencies to monitor and enforce rail transit safety, as established in federal statute. Key characteristics of both programs include: (1) safety regulations, (2) inspections and other oversight activities, and (3) enforcement mechanisms to ensure that safety deficiencies are addressed (see figure). There are strengths and limitations to FRA's and FTA's approaches to their safety oversight missions, including how the two agencies develop safety regulations, conduct inspections, and carry out enforcement. The National Transportation Safety Board has reported, and stakeholders GAO spoke with generally agreed, that strengths of FRA's rail safety oversight program include its safety regulations, its risk-based inspection program, and its enforcement authorities. FRA also has potential limitations in its oversight framework, though, such as difficulty evaluating the effectiveness of its enforcement mechanisms. FTA has made some progress implementing changes to the rail transit safety program. However, FTA has not provided all the necessary guidance and support to states' safety agencies to ensure they develop appropriate and effective rail transit safety inspection programs. In particular, FTA has not provided states with guidance on how to develop and implement risk-based inspection programs. Though FTA has said that it will develop such guidance, it does not have a plan or timeline to do so. Without guidance from FTA on how to develop and carry out risk-based inspections, state safety agencies may not allocate their limited resources efficiently, and important safety issues may go undetected. In addition, FTA has not developed a process or methodology to evaluate whether state safety agency enforcement authorities and practices are effective. Without clear evidence that state safety agencies' enforcement is effective, states and FTA may not be able to compel rail transit operators to remedy safety deficiencies. As a result, deficiencies may remain for long periods, potentially contributing to safety incidents. GAO recommends that FTA (1) create a plan, with timeline, for developing risk-based inspection guidance for state safety agencies, and (2) develop and communicate a method for how FTA will monitor whether state safety agencies' enforcement practices are effective. DOT agreed with our recommendations. DOT, NTSB, and WMATA provided technical comments that we incorporated as appropriate.