Highway Safety: More Robust DOT Oversight of Guardrails and Other Roadside Hardware Could Further Enhance Safety
Released: 2016-07-07T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: In 2014, 54 percent of traffic fatalities in the United States occurred as a result of a vehicle's leaving the roadway, according to U.S. Department of Transportation's (DOT) data. Roadside safety hardware, such as guardrails, is meant to reduce the risk of a serious crash when leaving the roadway. But in the last several years, a number of serious injuries and deaths resulted from crashes into roadside safety hardware. GAO was asked to review FHWA's oversight framework for roadside safety hardware.

This report assesses: (1) how FHWA performs oversight of state policies and practices related to roadside safety hardware; (2) the laboratory crash-testing process and FHWA's oversight of this process; and (3) the extent to which information is available on roadside safety hardware's performance once installed. GAO reviewed federal and state policies, surveyed state DOTs and received 44 responses, and reviewed documentation from nine U.S. crash test labs.

section_title: What GAO Found
paragraphs: The Federal Highway Administration (FHWA) oversees and promotes states' installation of crash-tested roadside safety hardware through guidance and policy directives to states and by issuing letters to roadside safety hardware developers that provide states with information on roadside safety hardware that has been crash tested. States that responded to our survey generally stated they require crash testing. However, some inconsistencies across state practices exist, and states' movement to require installation of devices successfully tested to updated, improved crash test standards—in the Manual for Assessing Safety Hardware (MASH)— has been slow. FHWA, in partnership with the American Association of State Highway and Transportation Officials (AASHTO), recently established transition dates to the MASH standards for states, but some challenges exist in developing and approving a sufficient quantity of roadside safety hardware tested to MASH standards. FHWA currently does not have a monitoring plan to report on progress to meeting the established dates; monitoring and reporting would allow FHWA to keep decision makers aware of progress and position FHWA to take corrective actions as needed.

In general, laboratory crash testing appears to be well documented and thorough; however, FHWA's oversight of the process does not address potential threats to independence. GAO found that six of the nine accredited U.S. crash test laboratories evaluate products that were developed by employees of the parent organization—a potential threat to lab independence. FHWA reviews crash tests' results and related documentation, if they are submitted for review, but FHWA relies heavily on the labs to make a pass/fail determination. We found that some other federal agencies in oversight of similar labs' testing settings require third party verification of test results or independent entities to make pass/fail determinations. FHWA does not have a process for formally verifying the testing outcomes and making its own or providing for an independent pass/fail determination. Developing a process for third party verification of roadside safety hardware's lab test results could provide greater assurance that potential threats to independence are fully addressed.

Little is known about the in-service performance of roadside safety hardware because few evaluations of this performance have been done. FHWA and AASHTO recommend that states and others perform in-service performance evaluations (ISPE) of installed roadside safety hardware because crash testing cannot fully capture real-world crash conditions. However, few ISPEs have been done, in part, because of a lack of inventory and crash data. In the summer of 2015 in four states, FHWA began a pilot study that could provide useful information, but according to FWHA officials, the purpose of this phase of the pilot is to determine best practices on data collection rather than assess performance of roadside safety hardware. FHWA officials told us they currently have no plans to include performance findings as part of future phases of this study or in their broader research portfolio. Continuing this study or planning to make ISPEs part of future research could add to the limited ISPE body of knowledge.

section_title: What GAO Recommends
paragraphs: GAO is making recommendations, including that DOT monitor and periodically report on the transition to the MASH crash test standards; develop a process for third party verification of crash test results; and support additional research on roadside safety hardware's in-service performance. DOT concurred with the recommendations and provided technical comments, which were incorporated in the report, as appropriate.

section_title: Letter
paragraphs: Crashes in which a vehicle leaves the roadway are frequently severe and  account for the majority of highway fatalities. In 2014, 17,791 fatalities— 54 percent of traffic fatalities in the United States—occurred as a result of  roadway departures, according to the U.S. Department of Transportation  (DOT)’s data. Roadside safety hardware, such as guardrails and median  barriers, is meant to reduce the risk of a serious crash. Because of the  frequency and severity of these incidents, the standards and procedures  by which roadside safety hardware is tested, installed, and evaluated  after installation are integral to promoting highway safety. In the last  several years, there have been a number of serious injuries and deaths  resulting from crashes into a type of roadside safety hardware, known as  the ET-Plus guardrail end terminal, which is installed at the end of a  guardrail and intended to prevent serious injury if it is struck. These cases  raise questions about the thoroughness of crash testing and oversight of  roadside safety hardware in general. This crash testing and oversight  involves multiple parties including those that develop roadside safety  hardware, laboratories that conduct roadside safety hardware crash  testing, the American Association of State Highway and Transportation  Officials (AASHTO), the Federal Highway Administration (FHWA), and  state departments of transportation.

You asked us to examine the framework for overseeing roadside safety  hardware, including the roles played by FHWA, states, and other parties.  This report addresses: (1) how FHWA performs oversight of state policies  and practices related to roadside safety hardware, (2) the thoroughness  of the laboratory crash-testing process and FHWA’s oversight of this  process; and (3) the extent to which information is available on roadside  safety hardware performance once installed.

To address these topics, we reviewed relevant regulations, policy  documents, and guidance memos from FHWA and selected states on  roadside safety and design. To determine how FHWA performs oversight  of roadside safety hardware, we reviewed roadside safety hardware  standards published by AASHTO and related FHWA guidance, and  evaluated FHWA’s policies and practices for conducting oversight of  roadside safety hardware in comparison with federal internal control  standards for monitoring, designing control activities, and communication  with external stakeholders. To gain perspective on what policies and  procedures states use to oversee roadside safety hardware’s testing,  installation, and performance, we distributed a survey to all 50 states,  plus the District of Columbia and Puerto Rico, and received 44  responses. Our survey findings represent only the states that responded  to our survey.

To assess the thoroughness and sufficiency of the crash-testing process  we requested and reviewed documentation from the nine crash test labs  in the United States that are accredited to international standards and  conduct crash testing for the purposes of FHWA review. We also  reviewed crash-test lab guidance and policy documents. To better  understand the standards and processes labs are expected to have in  place and how international laboratory and roadside safety hardware  testing standards are implemented, we conducted structured interviews  with officials from each of the nine labs and from three international  accrediting bodies that accredit test labs. We also reviewed 10 FHWA  files selected to represent different types of roadside safety hardware for  years between 1993 and 2014 to better understand the processes by  which FHWA oversees the testing process. To determine the state of  knowledge of roadside safety hardware performance, we conducted a  literature review of studies published by state, federal, and academic  sources between 1993 and 2015. We interviewed officials at FHWA’s  Office of Safety, and Office of Safety Research and Development, as well  as state departments of transportation and FHWA’s Division Offices in  five states (Virginia, Ohio, Texas, Maryland, and California), to obtain  non-generalizable examples of how federal and state roadside safety  hardware policies intersect. We selected these states based on the  presence of an accredited crash-testing facility in the state and  recommendations from outside experts regarding the quality of  performance data collection efforts in those states. We visited three of  these states (Virginia, Ohio, and Texas) to visit crash testing labs and  spoke with lab personnel, state DOT officials, FHWA division office  representatives, and other roadside safety hardware stakeholders.  Because of ongoing litigation, we did not review any issues related to the  accidents involving the ET-plus guardrail end terminal. See appendix I for  a full description of our objectives, scope, and methodology.

We conducted this performance audit from April 2015 to June 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe the  evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: If a vehicle leaves the roadway, ideally, the roadside would be clear of all  obstructions and be traversable. However, because there are numerous  roadside areas that cannot be practically cleared of all fixed objects or  that have sharp declines, roadside safety hardware can be used to  reduce the consequences of a departure from the roadway. The goal of  roadside safety hardware is met when the hardware contains, redirects,  or decelerates the vehicle to a safe stop without causing serious injury to  the vehicle’s occupants or other people. General categories for roadside  safety hardware are: 1) longitudinal barriers, which include items such as  guardrails and cable barriers and are intended to reduce the probability of  a vehicle’s striking an object or terrain feature off the roadway that is less  forgiving than the barrier; 2) bridge barriers, which function as longitudinal  barriers but are specific to bridge design; 3) barrier terminals/crash  cushions, which include items like guardrail end terminals that are  intended to absorb or divert the energy of a crash into the end of a  longitudinal barrier; 4) support structures, such as sign supports, which  are designed to break or yield when struck by a vehicle; and 5) work zone  devices, which include a variety of items used in a work zone that are  temporary in nature. See figure 1 below for a depiction of these types of  hardware.

DOT’s primary mission is to ensure the safety of the traveling public. The  strategic goals of the FHWA, within DOT, are to provide safe, reliable,  effective, and sustainable mobility for users of the nation’s highway  system. FHWA distributes about $40 billion to the states each year  through the federal-aid highway program (generally providing 80 to 90  percent of projects’ costs on designated federal-aid highways) for  highway and bridge infrastructure, a portion of which is spent on safety  improvements including roadside safety hardware. FHWA issues  regulations and guidelines, and can perform direct oversight for projects  that use federal funds, including those on the National Highway System  (NHS). The NHS consists of approximately 220,000 miles of the nearly 1- million miles of roadways eligible for federal aid. The NHS includes the  47,000-mile Interstate Highway System as well as other roadways,  connectors important to U.S. strategic defense policy, and connectors to  major intermodal facilities, such as airports or transit hubs.

FHWA administers and oversees the federal-aid highway program  through FHWA’s division offices located in all 50 states, the District of  Columbia and Puerto Rico. As part of FHWA’s risk-based oversight,  division offices and state DOTs have “Stewardship and Oversight  Agreements” that specify the terms under which states assume oversight  responsibility for federally funded projects. Under FHWA’s risk-based  stewardship and oversight program, FHWA is responsible for determining  projects that have an elevated risk or projects where FHWA involvement  can enhance meeting program or project objectives. This involvement  may include conducting oversight of the entire project or a specific phase  or element of the project. For all projects that FHWA does not categorize  as having an elevated risk, responsibility for oversight of design and  construction of projects is generally assumed by the states. For each  federally funded project, FHWA enters into project agreements with the  state in which the state agrees to adhere to all applicable federal laws  and regulations.

Section 109 of Title 23 of the United States Code directs DOT to work in  partnership with the state DOTs to develop standards for the NHS and  other roadway systems. To fulfill this responsibility, FHWA works in  partnership with AASHTO to advance many of its mission areas.  AASHTO is an association representing highway and transportation  departments in the 50 states, the District of Columbia, and Puerto Rico.  AASHTO serves as a liaison between state departments of transportation  and the federal government and develops and maintains design  standards for roadways, bridges, and highway materials. FHWA  incorporates some AASHTO standards into federal regulation, for  example, the Policy on Geometric Design of Highways and Streets  (Green Book), which lists design criteria across a range of roadway types,  from rural roads to freeways. FHWA has an ex-officio, non-voting role on  AASHTO committees. In cooperation, AASHTO and FHWA sponsor  research on common transportation issues through the Transportation  Research Board’s National Cooperative Highway Research Program  (NCHRP), including research on roadside safety hardware. NCHRP  studies are funded by the states from federal-aid highway program funds  apportioned to them.

Roadside safety hardware is developed by manufacturers, states, and  universities and can be crash tested to assess its safety performance.  The nine U.S. crash-testing labs that are accredited and recognized by  FHWA can conduct full-scale crash testing where roadside safety  hardware is hit by a vehicle to determine whether it meets AASHTO- accepted standards for roadside safety hardware. Of the nine labs, three  labs are independently operated; two are owned by companies that also  develop roadside safety hardware; three labs are affiliated with  universities; and the final lab is operated by a state department of  transportation. Representatives from roadside safety hardware  developers, crash test labs, academia, and state and federal  transportation departments participate in Task Force 13, a committee  whose mission is to develop specifications for new materials and  technologies identified for use in highway construction projects. As part  of this mission, Task Force 13 develops, recommends, and promotes  standards and specifications for roadside safety hardware.

The two ways of assessing this performance are lab crash testing and in- service performance evaluations. Crash tests can quantify performance  for specific conditions that represent the “worst practical conditions” in  terms of the speed and angle of the vehicle hitting the hardware. The  performance of hardware is evaluated in terms of risk to the vehicle  occupants and structural adequacy of the hardware, among other items.  AASHTO currently has two sets of crash-testing standards that it  endorses for installing roadside safety hardware: NCHRP Report 350  standards adopted in 1993, which are being phased out, and the  Manual for Assessing Safety Hardware (MASH) adopted in 2009.  AASHTO developed the MASH standards as an update to NCHRP  Report 350, and these standards contain revised criteria for crash tests of  roadside safety hardware. Updates in MASH include: increases in the  size and weight of several test vehicles to better match the current vehicle  fleet, changes to the number of tests and impact conditions, and more  objective evaluation criteria.

AASHTO has also sponsored research on how to assess the  performance of roadside safety hardware once it has been installed, an  assessment that is referred to as in-service performance evaluation  (ISPE). In 2003, NCHRP Report 490: In-Service Performance of Traffic  Barriers, published findings of research and suggested practical  procedures for conducting ISPEs. In-service performance evaluations are  a way of assessing roadside safety hardware’s performance in “real- world” scenarios not captured in a crash-test setting. For example,  performance may be affected by installation factors, such as slope and  grade of roadway and soil type, and maintenance conditions, including  whether the hardware has degraded over time from weather or accidents,  none of which are captured in crash-testing.

section_title: FHWA Encourages States to Install Crash-Tested Roadside Safety Hardware, but the Movement to Adopt Improved Standards Has Been Slow
paragraphs: FHWA oversees and promotes the installation of crash-tested roadside  safety hardware through guidance and policy directives to the states and  by issuing letters to roadside safety hardware developers that submit  crash-test results for review by FHWA. We found that states generally  require crash testing; however, some inconsistencies across state  policies and practices exist, and a movement to adopt the improved  MASH standards has been slow. In 2016, FHWA and AASHTO released  a new Joint Implementation Plan stating that states should transition to  installing only MASH-standard-tested roadside safety hardware in phases  by 2019. However, some concerns have been raised, and FHWA has not  developed a plan to track progress of the states and industry in meeting  the new dates. FHWA has contracted for a full examination of its roadside  safety hardware oversight processes and expects a report with  recommendations for potential changes to these processes in the  summer of 2016.
subsections: section_title: FHWA Encourages States to Install Appropriately Crash-Tested Roadside Safety Hardware
paragraphs: In line with its overall safety mission as well as that of DOT, FHWA  encourages states to install appropriately crash-tested roadside safety  hardware. By law, FHWA is required to ensure that highway projects  designed and constructed with federal funds are safe. FHWA’s Office of  Safety’s specific mission includes advancing the use of scientific methods  and data-driven decisions. Also, according to FHWA’s Office of Safety  website, roadway departure is one of its focus areas. FHWA has issued  policy that roadside safety hardware should demonstrate acceptable  crashworthy performance in order to be used on the NHS and receive  federal-aid reimbursement. To encourage this outcome, FHWA issues  guidance and policy directives to the states and industry. For example, in  2015 FHWA issued a memo that encouraged state agencies to upgrade  their existing installations of guardrail end terminals that had been tested  to standards issued prior to the NCHRP 350 standards, which were  adopted in 1993. Congress directs DOT to work in partnership with the  state DOTs to develop standards for the NHS and other systems.  FHWA works in cooperation with AASHTO to promote state adherence to  crash-testing standards through joint implementation plans. These plans  are voted on and must be approved by a majority of AASHTO’s member  states. FHWA and AASHTO issued joint implementation plans in 2009  and 2016 that provided guidance for states to follow in transitioning to  updated crash test standards.

In addition to providing guidance to states, FHWA also issues federal-aid  reimbursement eligibility letters to roadside safety hardware developers  that submit their product information, crash test results and other  supporting documentation for review. Although it is called a federal-aid  reimbursement eligibility letter, FHWA’s eligibility letter is not required,  and federal-aid reimbursement is not contingent upon receipt of an  eligibility letter. FHWA issues these letters as a service to the states to  provide states with information on the crashworthiness of roadside safety  hardware. FHWA posts the letters on its website creating a central  repository of information for states to know which roadside safety  hardware has been tested. FHWA officials stated that when they receive  a request from a developer for an eligibility letter, the request includes  information on the design of the roadside safety hardware device, the  crash testing report, pictures and videos of the crash testing, and other  information. FHWA officials told us that they follow up with the developer  or test lab if they have questions about any of the data or video evidence.  FHWA also advises developers that if modifications are made to a  roadside safety hardware device that has received an eligibility letter, the  developer must resubmit information to FHWA for review.

Though it is FHWA’s policy that all roadside safety hardware installed on  the NHS should be crash tested, crash testing is not a requirement for  states to receive federal-aid highway program funds because this policy  was never incorporated into regulation or other formal agreements with  the states (such as FHWA’s project agreements with states). According to  FHWA officials, in the absence of a federal statutory or regulatory  requirement for crash testing, FHWA cannot withhold federal funding for  federal-aid highway-program projects’ approvals to a state should the  state choose to install roadside safety hardware that had not been tested  to meet appropriate crash test standards. During our review, we found a  widespread misperception among state DOT and FHWA division office  officials we spoke with that crash testing of roadside safety hardware to  applicable standards and obtaining an FHWA eligibility letter was required  in order to receive federal reimbursement. In 1991, Congress instructed  DOT to issue a final rule regarding revised standards for acceptable  roadside safety hardware. In 1993, FHWA issued a rule that  incorporated crash test standards into regulation by reference as  guidance. FHWA stated at the time that it lacked sufficient knowledge to  be more prescriptive about roadside safety hardware in general and  chose not to make crash testing mandatory through regulation. FHWA  has not issued a proposed rulemaking to require crash test standards  since. FHWA officials told us that they believe encouraging state  compliance is more effective than requiring it through a rulemaking  because the current partnership with AASHTO garners support from  states, and a federal rulemaking can take many years to complete.

section_title: Most States Said They Require Crash Testing, though Some Inconsistencies Exist, and Transition to Improved Standards Has Been Slow
paragraphs: Most states that responded to our survey told us that roadside safety  hardware installed on the NHS is required to be crash tested, and many  of those states said they had processes in place to limit installation of  roadside safety hardware to those that have obtained FHWA eligibility  letters. Nearly all, 43 of the 44 states that responded to our survey, told  us that crash testing to MASH or NCHRP Report 350 standards is  required in their state for major categories of roadside safety hardware.  In addition, 38 of 44 states also responded that they maintain lists of  “approved” or “qualified” products from which contractors can choose  roadside safety hardware for installation. Furthermore, 32 of the 38 states  with these lists responded that all roadside safety hardware on their  qualified or approved product lists have an FHWA eligibility letter.

While our survey results indicate that FHWA’s guidance has been widely  implemented at the state level, they also indicate some inconsistencies in  state policies and some misperception about FHWA policy. First, 10  states responded that that they do not have a specific law, regulation, or  policy document that establishes crash-testing requirements. In follow-up  responses, four of these states told us that they do not have documented  requirements because they believe FHWA requires crash-testing of  roadside safety hardware and that the FHWA requirement governs  roadside safety hardware in their state. If a state’s policy is only to refer  to a federal requirement that does not exist, then effectively no  requirements govern crash testing in that state. Second, while most  states approve installation of only roadside safety hardware that has  received an FHWA eligibility letter, not all states do so. For example, 11  states reported that they have conducted their own crash testing in the  past 10 years, and 6 of the 11 responded that they do not always submit  those roadside safety hardware devices for FHWA review prior to  approving devices for installation. Officials from one state told us they  only submit results for eligibility letter review when they believe the device  is likely to be used by other states.

Federal standards for internal control highlight the need for agencies to  design control activities—policies, procedures, techniques, and  mechanisms—to achieve objectives and address related risks. In June  2012, FHWA issued a memo indicating that division offices should  encourage states to have written policies that incorporate AASHTO’s  guidance on current roadside safety information and operating practices.   However, FHWA has not directed its division offices to help ensure that  states have policies governing crash testing of roadside safety hardware  installed on their roadways as part of this procedural review. Officials in  the five FHWA division offices we interviewed told us they have a  procedure for reviewing states’ standards and design specifications,  which could include states’ standards and requirements for roadside  safety hardware. However, officials in FHWA division offices we  interviewed said that they generally do not examine roadside safety  hardware practices on individual projects as part of FHWA’s risk-based  oversight. Officials in one division office noted that topics like pedestrian  safety would be a higher priority for the division office because officials  said there are more pedestrian deaths than there are deaths from  roadside safety hardware. Division office officials stated that they rely on  the state to ensure that what is incorporated in the project meets state  standards, and officials from four out of five division offices stated that  they do not verify that states are installing state-approved products.  Furthermore, FHWA’s Office of Safety officials told us that they do not  monitor and collect information on state policies with respect to roadside  safety hardware. The absence of written requirements at the state level  and inconsistencies in state practices could, in some cases, result in the  risk of reduced assurance that states are fully implementing appropriate  crash-testing standards.
subsections: section_title: Transition to Improved Standards Has Been Slow
paragraphs: According to FHWA and AASHTO, MASH crash test standards are an  improved set of standards because they better reflect the current vehicle  fleet, which has become heavier and taller over the past 25 years. Two  studies compared the NCHRP Report 350 standard testing to the MASH  test standards. The results of these studies indicated that in some cases  MASH test standards provide a more rigorous evaluation for crash testing  roadside safety hardware. First, in 2010, NCHRP conducted an  evaluation of existing roadside safety hardware devices approved under  NCHRP Report 350. Re-testing these devices and evaluating  performance using the criteria in MASH revealed that 6 of the 21 tests  performed on NCHRP Report 350-compliant roadside safety hardware  devices did not pass. Second, in September 2015, a joint  AASHTO/FHWA review of guardrail end terminals concluded that the  MASH crash test standards incorporate tests relevant for guardrail end  terminals that are not included in NCHRP Report 350 test standards.  Specifically, the study found that NCHRP Report 350 standards do not  fully address performance issues in the areas of side and shallow-angle  impacts. The study recommended fully implementing MASH for new  installations of guardrail end terminals.

States have been slow in transitioning to implement the MASH crash-test  standards. In 2009 AASHTO and FHWA issued a Joint Implementation  Plan adopting MASH as the updated crash-test standards necessary for  an applicant to receive an FHWA eligibility letter for a new roadside safety  hardware device. However, this plan said that states could continue to  install roadside safety hardware tested to the previous NCHRP Report  350 standards. Therefore, manufacturers could continue to produce, and  states could continue to install roadside safety hardware that had already  received an eligibility letter without retesting to MASH crash test  standards. In January 2016, FHWA and AASHTO released a new Joint  Implementation Plan stating that states should transition to installing only  MASH-standard-tested roadside safety hardware. According to the plan,  FHWA will no longer issue eligibility letters for new or modified roadside  safety hardware tested to standards other than the MASH crash-test  standards. The 2016 Joint Implementation Plan calls for states to  complete the transition to the MASH crash-test standards between  December 2017 and December 2019, depending on the type of  hardware. (See table 1 below.) If states comply with the 2016 Joint  Implementation Plan’s dates for transitioning roadside safety hardware  installations to meet the MASH crash-test standards, this transition will be  8 to 10 years after the 2009 Joint Implementation Plan, and states may  continue to install non-MASH-tested hardware on the NHS until  December 2017 at the earliest. FHWA officials noted that roadside safety  hardware often remains on the roads for at least 20 years before being  replaced due to aging, so hardware tested to the older NCHRP Report  350 standards could be on the roads for years to come.

However, at this point it is not clear that states will be able to comply with  the dates set in the plan. In order to meet the transition dates, industry will  have to develop and test products to the MASH standards that have not  previously been tested to these standards, and FHWA will have to review  applications-for-eligibility letters for developers that request them. States  will then have to make changes to either their design and specification  policies or approved lists of products to incorporate only MASH-tested  roadside safety hardware. Industry, to this point, has been slow to move  to develop and test products to the MASH standards. Using eligibility  letters as an indicator, as of March 2016, there are currently only two  guardrail end terminals with eligibility letters that have been tested using  the MASH standards, compared to the13 guardrail end terminals tested to  NCHRP Report 350 with eligibility letters. In the category of longitudinal  barriers, there were only 17 MASH-compliant eligibility letters among the  348 active eligibility letters. In an open letter to AASHTO, the American  Traffic Safety Services Association, an association representing highway  safety industries, expressed concern with the ability for industry to have  enough hardware that meets the MASH crash-test standards by the  transitions dates, as well as the ability of states to approve new hardware  and for FHWA to post new eligibility letters in a timely manner.

FHWA officials told us that states and manufacturers have responded  positively to the new deadlines. However, FHWA officials did express  some concern as to whether states will be able to fully implement MASH  standards by the dates in the 2016 Joint Implementation Plan. Their  concerns included the need for the market to react in a timely manner and  have enough products available to support competition, and to invest in  testing categories of roadside safety hardware that have had little testing  to MASH standards to this point. FHWA officials also told us that as  industry reacts to the dates, FHWA will likely have an influx of requests to  review eligibility letters; FHWA officials told us that they already have a  backlog of eligibility letter applications since FHWA stopped issuing  eligibility letters for modifications to hardware tested to non-MASH  standards at the end of 2015.

Federal standards for internal control highlight the need for agencies to  obtain information needed to achieve their objectives from external  parties, including significant matters related to risks. FHWA officials  stated they will be in a better position in a year to say whether states are  likely to be able to successfully transition to MASH crash-test standards  by the dates specified in the January 2016 Joint Implementation Plan.  However, FHWA has not developed a plan to track progress of the states  and industry in meeting the new dates. Moreover, we found that FHWA  and states currently do not collect information that would assist in  monitoring the transition to MASH standards. For example, as discussed  in the following section, FHWA can interact with developers and crash  test labs during the test process, but FHWA does not collect information  from developers and labs to be informed when hardware that was  previously tested to older standards is re-tested to MASH and fails.  Without this information on test failures, FHWA and states may be  unaware of setbacks to the transition. Also, if states do not have this  information, it may result in the states unknowingly installing failed  hardware during the transition period. In addition, 12 states responded to  our survey that they currently do not require developers to notify them of  modifications made to an approved device. While FHWA requests such  notification, 3 of the 12 states do not have eligibility letters for all  approved devices and could be unaware of design changes. Federal  standards for internal control also highlight the need for agencies to  provide quality information to external parties, including the general public  to help achieve agency objectives. Monitoring and reporting industry  and state progress to the goal dates set in the 2016 Joint Implementation  Plan would allow FHWA to keep decision makers in both DOT and  Congress aware of progress. Such monitoring and reporting of progress  would also position FHWA to take corrective actions as needed to better  assure that states and industry are successfully moving to meeting  improved standards.

section_title: FHWA Is Currently Reviewing Its Oversight Program
paragraphs: FHWA contracted in May 2015 with DOT’s Volpe National Transportation  Systems Center to conduct a full review of its roadside safety hardware  oversight process and expects a report with recommendations for  potential changes to its oversight program in summer 2016. Officials  stated the review will include a full examination of the process by which  roadside safety hardware is developed, evaluated, funded, and assessed,  as well as recommendations for any improvements needed. Specifically,  the report will include: documentation of existing laws, regulations, policies, standards, and  guidelines associated with the roadside safety hardware process; documentation and review of all the steps in FHWA’s current crash- testing evaluation process; and  findings and recommendations to FHWA to improve its oversight.

FHWA officials told us that there may be ways to improve the agency’s  oversight of roadside safety hardware and that everything in the process,  from the partnership relationship with AASHTO to the eligibility letter  process, will be included in the review. During the course of our review,  FHWA implemented some changes to its program, such as clarifying the  need for any modifications to hardware with eligibility letters to be  reevaluated, but FHWA officials stated they were holding off on major  changes to the current oversight program until the Volpe National  Transportation Systems Center’s review is complete.

section_title: Crash Testing Is Well Documented and Thorough, but FHWA’s Oversight Process Does Not Address Potential Threats to Independence
paragraphs: At all nine U.S. labs accredited to conduct crash testing of roadside safety  hardware for FHWA review, laboratory crash testing was well  documented and thorough in terms of consistency in documentation and  test procedures across labs. As part of the crash-testing process, labs  and test sponsors have discretion in making testing decisions in several  important areas. In addition, there is an inherent potential threat to  independence in the testing process because employees in some labs  can test devices that were developed within their parent organization. The  independence requirement in the standards used to accredit labs is  general, and we found varying interpretations and differences in  approaches for mitigating threats to independence across the labs.  FHWA does not require third party verification of crash testing and does  not make its own pass/fail determinations or provide for independent  pass/fail determinations for test results. FHWA also does not provide  additional guidance to labs and accrediting bodies on independence  mitigation measures for crash testing roadside safety hardware. We found  that some other federal agencies with similar testing programs have more  measures than FHWA has to mitigate potential risks to independence.
subsections: section_title: Lab Crash Tests Are Well Documented and Thorough
paragraphs: FHWA requires that crash test labs conducting testing for the purposes of  FHWA eligibility letters be accredited to International Organization for  Standardization (ISO) 17025 standards, which contain management  and technical requirements for labs to be deemed competent to run  laboratory testing. There are nine crash test labs in the United States that  are accredited to these standards and conduct crash testing for the  purposes of FHWA eligibility letters. Our review of the nine accredited  U.S. labs found that individual crash tests were well documented and  thorough because test reports contained documentation that would allow  a third party to understand how the lab conducted the test and how the  test results were interpreted. To evaluate the thoroughness and  documentation for labs’ crash testing, we created both interview  questions and a document request list for all the labs based on  international accreditation requirements as well as the crash-testing  guidelines in MASH. All nine example test reports we reviewed clearly  identified the test standard and the test level the lab used to test the  roadside safety hardware device. The pass/fail criteria being used to  evaluate the roadside safety hardware device was clearly identified, and  all reports described the test results against each of the evaluation  criteria. In addition, all reports described the setup of the device and pre- test procedures, which could include verifying the integrity of the soil,  when applicable, and structural integrity of the test vehicle. Each report  also included between 20 and approximately 100 pictures of the testing  process, along with a description of the results. For more information on  the documents we requested and reviewed, see appendix I.

Labs generally described using requirements specified in test or  accreditation standards as the basis for their procedures. Labs described  sending equipment out to a qualified calibration laboratory, or obtaining  additional expertise and certification to calibrate their own equipment.  Several labs also stated that they keep the test objects on site for a  period of time, in case follow-ups were needed. Specifically, five labs told  us that they kept test documentation on file for at least 2 years, and in  three of these cases, kept records indefinitely. Labs also described going  beyond what standards require in certain instances. For example, five of  the nine labs described using additional cameras or data recording  devices to better capture data that would be useful to industry research or  to the customer.

Accrediting bodies are expected to use the ISO 17025 standards, along  with test standards specific to the industry, such as the MASH crash-test  standards in this case, as the basis for accrediting crash test labs.  Officials from three labs said they had developed documentation  practices specifically in reference to the accreditation process. Each  accreditation body said that it conducts routine audits each year as part of  its accreditation cycle, where accreditation bodies told us they evaluate  such aspects of the testing process as setup, equipment calibration,  competence of personnel, documentation, and record keeping. One lab  reported that its accrediting body assisted it with improved lab procedures  by setting up calibration procedures; two labs reported that accreditation  requirements guide their policies on document retention. In addition,  accreditation standards require labs to collaborate and compare results in  inter-laboratory collaborations, a procedure that labs do via Task Force  13, in order to work toward greater consistency in test procedures and  results interpretation.

section_title: Labs Use Professional Judgment in Some Crash- Test Selection and Interpretation of Test Results
paragraphs: Although individual crash tests are well documented, full-scale crash  testing to evaluate the performance of an individual piece of hardware is a  complex process that requires labs to use professional judgment when  deciding which tests need to be run, and how to interpret the results. Both  NCHRP Report 350 and MASH have a suite of tests in order to cover a  range of crash speeds, angles, and size and weight of vehicles to assess  the performance of the roadside safety hardware device. A majority of  labs (five of nine) reported that they usually recommend to test sponsors  that they run the full suite of tests outlined in the test standard.  However, for modified devices that have previously been tested, there is  some discretion on which tests to run. Because an individual full-scale  crash test can cost about $55,000 (according to a crash test lab we spoke  with) it is advantageous to only run what test sponsors think are the most  critical tests for a given device. For example, in one of the testing  scenarios we reviewed, the lab engineers determined based on prior  testing with a larger vehicle that the MASH test for small cars would not  be necessary for the tested device. Although this decision is documented  in the test report, the reasoning is not detailed, and so it is hard for a  reviewer to evaluate this decision. Of the nine labs we interviewed, four  told us that they frequently consult with FHWA in the test-planning  process, and that these labs generally run the tests in agreement with  FHWA. The other labs told us they either rarely or never consult with  FHWA; these labs encourage test sponsors to communicate directly with  FHWA if they plan to seek an eligibility letter, and in these cases the lab  runs the tests the sponsor requests. As part of the eligibility letter  process, labs can, but are not required to, consult with FHWA for advice  on which tests to run.

Labs have some discretion in interpreting the test results against the  pass/fail criteria of the crash test standard. According to MASH crash test  standards, some interpretation will be necessary for the criteria due to the  “very complex nature of vehicular collisions and the dynamic responses of  an occupant to the collision, as well as human tolerances to impact.”  Eight of the nine labs reported that engineering judgment was necessary  to make a pass/fail decision in at least a small minority of tests, although  one lab reported that up to 30 percent of all NCHRP Report 350 or MASH  crash tests require professional judgement. One lab noted that MASH has  more specific criteria than NCHRP Report 350, but leaves room for  interpretation when it comes to defining failure limits for penetration,  occupant intrusion, and deformation limits, which are all part of the  occupant risk criteria. For example, six of the labs reported that occupant  intrusion standards were the main subjective parameter, because  characteristics such as the amount, type, and location of the intrusion  were important in determining whether the occupant could be harmed.  Lab officials told us that MASH standards, which specify maximum  allowable levels of occupant intrusion, do not always address applied  testing scenarios. For example, officials in one lab described a test on a  post that sliced and made holes in the floor pan of the test vehicle. The  lab officials said they interpreted this to be a failure, although lab officials  suggested that MASH crash test standards do not specify whether holes  in the floor of the vehicle mean the test fails.

section_title: Potential Threats to Lab Independence
paragraphs: In the roadside safety hardware-testing community there is an inherent  potential threat to lab independence because there is often not a formal  separation between design and testing roles within a lab’s parent  organization. Specifically, six of the nine crash test labs we reviewed can  test products that were developed by employees of the same parent  organization. Two manufacturer-owned labs can test products created by  another division of the same company; three university-run labs can test  university employees’ designed products; and a state-based facility tests  products designed by the state department of transportation. In order to  be an accredited lab, the ISO requires labs to identify any conflicts of  interest and have policies to ensure labs are free from undue pressure.  The three accrediting bodies we interviewed told us that the ISO  requirements are usually met by documented conflict-of-interest policies  and by having an organizational structure in which lab employees do not  have conflicting lines of reporting to their parent organization. However,  documentation we obtained and interviews we conducted with labs and  accreditation bodies revealed varying interpretations of the level of  involvement of the device designer in the crash-testing process, and of  the level of involvement of the lab in providing design feedback based on  crash test results, that is appropriate to ensure independence. For  instance, while four labs told us they would offer advice on how to re- design the device if it failed a crash test, the other five labs said they did  not make such recommendations, and one specifically said it interprets  ISO standards to mean that labs should not be involved in making design  recommendations after testing. The ISO standards are intended to  broadly cover testing and calibration laboratories across many industries  to ensure technical competence. Varying interpretations suggest a lack of  specificity in ISO requirements to ensure independence in the testing of  roadside safety hardware.

Of the six labs that test devices developed within their parent  organization, two labs told us they have policies that formally separate the  role of the designer and the tester, although only one had this policy  documented. One of the two labs designates an independent approving  authority to make the final determination of whether the hardware passed  or failed each test and specifies that this person could not have been part  of the design or development of the hardware. The other told us that if a  member of the lab was involved in the design of a device, that person  would not be allowed to make the pass/fail determination. However, the  other four labs do not have a separation that is this clear. Two of these  labs provided us with the general conflict of interest policies of their  parent organizations, and two labs pointed us to conflict-of-interest  policies in their quality manuals, which did not have information about  separating design and testing. The Committee of Sponsoring  Organizations of the Treadway Commission (COSO), a joint initiative of  multiple private-sector-accounting organizations, publishes the Internal  Control-Integrated Framework to help organizations design internal  controls to achieve their objectives. This framework highlights the  importance of the separation of duties within an organization, to reduce  the risk of inappropriate conduct in the pursuit of objectives. The standard  states that when selecting and developing control activities, management  should consider whether duties are divided or segregated among different  people to reduce the risk of error or inappropriate or fraudulent actions.  Labs that do not have this formal separation between design and testing  functions could have threats to the independence of their test analyses.

One of the three accrediting bodies told us that independence can be  difficult to assess because it is not clear what labs that are affiliated with  manufacturers, for instance, must do to mitigate any conflicts of interest.  Officials from two accrediting bodies told us that other federal agencies  provide them with additional guidance on independence and technical  expertise, respectively, and one accrediting body told us that it is  preferable when an agency provides guidance so that the accrediting  body can better apply standards when accrediting labs in a specific  industry. For example, officials from this accrediting body provided an  example of a federal agency that has developed more specific ethics and  integrity requirements than the ISO. Accrediting body officials told us that  this agency requires the accrediting body to assess the labs to these  more specific requirements.

section_title: FHWA Does Not Mitigate Potential Threats to Independence
paragraphs: Federal standards for internal controls state that agencies should  establish policies and procedures to respond to risks as part of their  internal control system. However, apart from the accreditation  requirement, FHWA does not have other mitigation measures in place  with regard to lab independence. FHWA does not provide guidance to  crash test labs or accrediting bodies on mitigating the risks posed by  threats to independence. Providing such guidance could provide greater  assurance that crash-testing is being performed in an independent,  unbiased fashion.

FHWA reviews crash test results as part of its eligibility letter process;  however, FHWA does not have a process for formally verifying the testing  outcomes and making or providing for an independent pass/fail  determination. FHWA relies heavily on the labs to determine whether the  crash test outcome results in a pass or fail determination for roadside  safety hardware. According to ISO standards for accreditation, when a lab  states whether a product complies with requirements, it is offering an  opinion, and it must be marked as such. Officials from one accrediting  body said it would be preferable for labs to provide only the crash-test  result data and have a third party apply criteria in MASH crash test  standards and make the pass/fail determinations. Officials at one lab we  spoke to added that they prefer not to make pass/fail determinations, but  they do so for each test. In the eligibility letter process, FHWA requires  that lab personnel apply the results to relevant crash-test standards and  make a pass/fail determination of the test results. FHWA officials  explained that as part of their eligibility letter-review process, they  examine the crash test lab report, including pictures, videos, and the test  data summary sheets. If FHWA officials have questions, they will contact  the lab or developer. However, eligibility letters state that FHWA is relying  on the assessment of the lab. Moreover, we reviewed 10 case files for  eligibility letters issued between 2005 and 2015 and found that  documentation was not sufficient to determine the rationale behind  FHWA’s decision to issue these letters. For more information on our  review of FHWA’s eligibility letter process, see appendix II.

FHWA officials acknowledged that lab employees’ testing devices that  were developed within their greater parent organization poses the  appearance of an independence threat. In May 2015, FHWA issued a  memo directing the developer and test labs to submit financial conflict-of- interest information in order for a developer to receive an eligibility letter.  FHWA officials told us that this information will not influence a device’s  ability to receive an eligibility letter, but that the information could be  published along with the final eligibility letter for the public to review, in an  effort to increase transparency. FHWA officials also told us that this was  an immediate change they could make but that they are awaiting the  results of the Volpe National Transportation Systems Center’s review  before deciding whether to take additional steps in this area.

section_title: Other Agencies Have More Oversight of Lab Testing
paragraphs: According to federal internal control standards, agencies should ensure  that they communicate quality information to external parties so they can  help the agency achieve its objectives and address related risks.  However, as explained above, there is a potential threat to independence  in the lab crash-testing environment for roadside safety hardware. In  other test settings we found that federal agencies require third party  verification of test results or independent entities to make pass/fail  determinations.

We found that both the Environmental Protection Agency (EPA)—in its  ENERGY STAR Program—and the National Highway Traffic Safety  Administration (NHTSA)—in its testing for Federal Motor Vehicle Safety  Standards and the New Car Assessment Program—have stricter  oversight over the lab-testing process and require third party certification  and/or verification testing. EPA’s ENERGY STAR Program is a voluntary  program to identify and promote energy-efficient products and buildings.  Lab testing of products is conducted to determine whether a product  meets program specifications for efficiency. As we’ve previously found,  the testing requirements for EPA’s ENERGY STAR program have  evolved in response to weaknesses identified in the program by us in  2007 and EPA’s Office of Inspector General in 2008, including a lack of  assurance that tested products met the qualification criteria. In response  to these findings, EPA and the Department of Energy signed a  memorandum of understanding in 2009 to propose several program  enhancements. As part of a review of this program, before these changes  had been implemented, GAO submitted fictitious products for certification  and found that the program was vulnerable to fraud and abuse because  manufacturers could self-certify that their devices met energy standards  without third-party verification. In 2011, we found that EPA had made  considerable progress in addressing these issues by including verification  testing and third party certification in the approval process.

Currently, in order to earn an ENERGY STAR label, products must be  tested by EPA-recognized laboratories, and a subset of products is  verified annually by third-party certification entities. EPA standards  require labs, their accrediting bodies, and third-party certification body  laboratories that verify test results to abide by respective sets of  conditions and criteria in order to be recognized by the ENERGY STAR  Program. EPA also has an application process for all three types of  entities to receive ENERGY STAR Program recognition. Under this  process, EPA requires that labs test products for review by third-party  certification bodies, which determine if the devices meet the program  standards for a product to carry an ENERGY STAR label. EPA’s  standards for ENERGY STAR recognition also require certification bodies  to verify lab test data and make a pass/fail determination, and EPA  officials added that the labs themselves are not supposed to make this  decision. In addition, certification bodies are to conduct verification testing  for a sampling of devices, including off-the-shelf devices, across multiple  categories each year. EPA officials told us they closely oversee the  certification bodies through frequent communication and periodic audits.

We also interviewed officials at NHTSA regarding two forms of vehicle  crash testing that they oversee. First, NHTSA issues federal motor  vehicle safety standards (FMVSSs), with which vehicle manufacturers  must comply, and manufacturers must self-certify that their products meet  these standards. NHTSA then sponsors verification testing of some  vehicle models, where they purchase vehicles from dealer lots and  subject them to crash testing to confirm the manufacturer’s certification.  The verification testing is conducted by labs selected by NHTSA. NHTSA  officials told us that they would not select a lab that has the potential  conflict of interest of being a part of a manufacturer business. Once the  testing is conducted, the raw data is sent to NHTSA and NHTSA officials  make the determination as to whether the vehicle has met the FMVSSs.

Testing for NHTSA’s New Car Assessment Program is similar to that of  the FMVSSs in design, but rather than checking whether vehicles meet  minimum safety standards, NHTSA awards vehicle models with up to 5  stars for their safety performance in crash testing to standards that,  according to NHTSA officials, typically exceed those in the FMVSSs.  Officials stated that because these standards are not federal  requirements, vehicle manufacturers do not have to comply or self-certify  compliance. Officials noted, however, that because the New Car  Assessment Program provides safety ratings information to consumers  the manufacturers have an incentive to receive the highest safety rating  possible. Similar to testing under the FMVSSs, officials said that NHTSA  purchases vehicles from dealer lots and then tests them at selected labs.  The crash-test data is then sent to NHTSA where NHTSA officials  determine the star rating for each test vehicle.

In contrast to these programs, FHWA does not require either third party  certification or verification of crash testing; nor does FHWA provide  additional guidance on independence mitigation measures for crash  testing roadside safety hardware. Establishing a process for third-party  verification of crash test results could provide greater assurance that  threats to independence are fully addressed. FHWA officials told us that  they would favor considering some form of third party review over crash  test results. These officials added that having FHWA conduct the third  party review could be challenging and that FHWA would need to assess  the resources, technical capacity, and legal capacity to perform that role.

section_title: Few In-Service Performance Evaluations of Roadside Safety Hardware Have Been Conducted, in Part, Due to a Lack of Data
paragraphs: According to FHWA and AASHTO-sponsored research, in-service  performance evaluations (ISPE) are recommended for effective roadside  safety hardware oversight because real-world crash conditions, such as  vehicle characteristics, as well as the terrain of the roadway, may vary  widely from those experienced in crash testing. Moreover, crash testing  cannot fully replicate the effects of installation conditions over time on  roadside safety hardware’s performance. In establishing a methodology  for conducting ISPEs, NCHRP Report 490 states that collecting crash  data over multiple years and examining crash sites in real-time can  enable researchers to report more information on roadside safety  hardware’s installation and maintenance issues, the costs associated with  making repairs to damaged hardware, and the severity of injuries  resulting from crashes that involve roadside safety hardware. This can  better equip states to make cost-benefit determinations regarding  roadside safety hardware replacement or new product development.  ISPEs can also inform whether crash-testing standards are appropriately  suited to assessing the effectiveness of roadside safety hardware.

Based on our review of studies published since 1993, when FHWA  recognized NCHRP Report 350 testing standards, few formal ISPEs of  roadside safety hardware have been conducted to fully assess the  performance of roadside safety hardware in actual conditions. After  reviewing government, industry, and academic sources, we found 14  formal ISPEs that were published since 1993. While other studies  included elements of an in-service performance evaluation, 14 studies in  our review combined crash data analysis with real-time visits to crash  sites to document and assess the damage, which is a key characteristic  of a formal ISPE as defined by NCHRP Report 490. Additionally, these  ISPEs tended to focus on longitudinal barriers, such as guardrails and  cable barriers, and barrier terminals, such as guardrail end terminals,  while other types of roadside safety hardware were generally not the  subject of ISPEs.

A key challenge to states conducting ISPEs appears to be the lack of  fully-developed data. As NCHRP Report 490 indicates, having inventory  data on the number of roadside safety hardware devices being studied  and their location is critical to calculating rates of collision with roadside  safety hardware within the study area. In our survey of state DOTs, we  asked officials to describe their data and inventory efforts, and states  reported a general lack of established inventory data. As table 2 shows, a  majority of states indicated that they have inventory data-collection efforts  for barrier terminals/crash cushions, for example, but many of these  efforts are new or are ongoing and therefore are not fully established. For  instance, of the 29 states that reported in response to our survey that they  have inventory data-collection efforts for barrier terminals/crash cushions,  18 said that their efforts are ongoing, and 12 of these said that they had  only been collecting data since 2014. State DOT officials we interviewed  in four states also told us that inventory data they collect may not include  information on condition or location of roadside safety hardware, which as  NCHRP Report 490 notes is necessary for a full understanding of  performance. The other key piece of data is crash data. The current state  of crash data reporting may not facilitate conducting ISPEs of roadside  safety hardware. According to NCHRP Report 490, police will likely not  comment as part of their crash records on factors like soil conditions,  which could influence how guardrail posts, for instance, function in a  crash. Police crash records also do not capture any unreported collisions  and may not consistently document the type of roadside safety hardware  involved in an accident.

According to our survey, only 6 of 44 states that responded said that they  had conducted any formal ISPEs in the last 10 years. State officials we  interviewed also described less formal efforts to evaluate roadside safety  hardware’s performance. For instance, officials in one state told us that  they perform a trial run for any new proprietary roadside safety hardware  device in a sampling of locations and monitor on-site the in-service  performance for 12–18 months prior to approving the device to be used  by contractors across the state. However, state officials told us this effort  is not published in a report. Without published results that document a  methodology that others can repeat, however, such results do not  ultimately add to the broader knowledge base of ISPEs.

Officials in four of the five states we interviewed indicated that they have  cost and/or data constraints related to collecting the necessary data to  conduct formal ISPEs. Officials from the fifth state we interviewed  described a software application they developed to inventory all of the  guardrail end terminals in their state. According to state officials, local  maintenance crews in the state use a custom web application on a mobile  device to record the total number, along with data on the type and  location, of guardrail end terminals in their state. This data is then  uploaded to a central database. State officials said they were planning to  make this a long-term project and apply it to other types of roadside  safety hardware. Officials noted that they are still in the process of adding  the capability for keeping the data up to date. These officials also told us  that the application was relatively inexpensive to develop, and FHWA  officials noted that at least one state was interested in learning more  about the application. State officials told us that as of yet, however, this  technology has not been shared across states.
subsections: section_title: Current FHWA Efforts Aim to Improve Data Collection Practices but Do not Fully Address In-Service Performance Evaluation Information Shortfalls
paragraphs: FHWA has ongoing research to identify best practices for the collection of  data on roadside safety hardware. However, this research is limited to  guardrail end terminals, and the planned scope of work may not be  sufficient to fill the gaps created by the lack of ISPE literature at the state  level. FHWA officials told us that in the summer of 2015, FHWA began a  pilot study on the collection of data on guardrail end terminals’  performance. According to FHWA officials, the first phase of this pilot  study is expected to last through the end of 2016. Officials plan to identify  current challenges to conducting ISPEs as well as recommend best  practices for: 1) the collection of real-time data on crashes involving  roadside safety hardware; 2) interagency communication at the state level  regarding crash reporting; and 3) data management regarding hardware  maintenance and location. FHWA is currently collecting inventory and  crash data in four states (Missouri, Pennsylvania, Massachusetts, and  California) that have agreed to participate in this pilot. FHWA officials  stated that within a selected area of each state, data will be collected by  examining crash sites for six different models of guardrail end terminals.  This data could produce information needed to assess the performance of  the devices with respect to the risk of severe occupant injury if the study  were to be continued. According to FHWA officials, crash specialists from  NHTSA, the agency that collects and reports data on fatal crashes for  DOT, will conduct detailed on-site investigations for fatal and serious  injury crashes, generally within 24 hours of receiving notification of the  crash. Data on crashes resulting in property damage only and other minor  crashes will also be collected. Officials told us that they plan to continue  collecting data through 2016 for this phase of the project.

According to FHWA officials, however, publishing findings on the  effectiveness of guardrail end terminals’ performance is not part of their  current efforts because they first want to provide guidance to states on  best practices for performance data collection. Officials noted that the  time frame for the current phase of the pilot would be insufficient to collect  enough data for statistically significant findings. FHWA officials told us  that they will not determine whether to include performance findings as  part of future phases of the pilot study until this phase is complete at the  end of 2016.

As noted previously, FHWA’s Office of Safety includes in its mission the  need to advance the use of scientific methods and data-driven decisions  in highway policy. The current lack of in-service performance findings and  established inventory data for roadside safety hardware poses challenges  to states making data-driven decisions about highway maintenance.  FHWA officials told us they currently have no plans to include additional  ISPEs for other types of roadside safety hardware as part of their broader  highway-safety research portfolio. Officials cited cost concerns with  gathering data and explained that ISPEs would take on greater relevance  in the future as more MASH-compliant devices are installed on roadways.  However, continuing this study and reporting on the performance of  guardrail end terminals, or planning to make ISPEs part of other future  research, could add to the limited body of knowledge regarding the in- service performance of roadside safety hardware. FHWA officials also  noted that hardware that was installed could still be on the roadways for  20–30 years. ISPEs on current devices can therefore still provide states  with critical information regarding how they might prioritize maintenance  tasks—such as replacing older devices—to best ensure safety for their  motorists. Without robust, ongoing in-service performance evaluations,  less safe hardware may remain in use longer than is necessary.

section_title: Conclusions
paragraphs: FHWA’s cooperation with AASHTO and state DOTs has resulted in states  having policies to install crash-tested roadside safety hardware on the  NHS. However, challenges exist for states, industry and FHWA as the  improved MASH crash-testing standards are phased in over the next few  years. These changes will require cooperation and action from industry,  the states, and FHWA. FHWA has the opportunity to exercise more  robust oversight to ensure greater consistency in the implementation of  improved crash test standards. First, FHWA, through its division offices’  oversight of states’ standards and design specifications, can help ensure  that states have written policies in place that fully reflect the terms of the  2016 state-approved Joint Implementation Plan to address inconsistent  practices across states. Second, monitoring and reporting the states’ and  industry’s progress transitioning to the MASH crash test standards, as  federal standards for internal controls suggest, and making this  information available to Congress and the public would facilitate  transparency and position FHWA to consider midcourse corrections if  required.

FHWA can also take steps to strengthen its role in the assessment of  roadside safety hardware performance—both in the test lab and once  installed on the roadways. Because FHWA’s current oversight process  does not include verification of lab crash-test results and no specific  mitigation measures are in place to address potential threats to  independence, the risks to ensuring the integrity of the crash-testing  process remain unaddressed. Other agencies have introduced policies or  processes into the testing process that mitigate these types of issues;  FHWA could take similar actions. In addition, other agency practices  provide a model for FHWA of closer cooperation with the labs and  accreditation bodies to address the independence issues unique to  roadside safety hardware’s testing. FHWA also has the opportunity to  advance its mission in the scientific evaluation of roadside safety  hardware. FHWA has a pilot project underway that is examining data  collection practices for in-service performance evaluations but currently  has no plans to report on performance findings from either this study or  other research in its portfolio. Continuing this study or planning to make  ISPEs part of future research could add to what is currently a limited body  of knowledge regarding the in-service performance of roadside safety  hardware. FHWA is poised to consider changes to its approach to  roadside safety hardware through a full programmatic review to be  completed in the summer of 2016. Opportunities exist to address all these  issues and to provide states, industry, and the traveling public greater  assurance that FHWA is fulfilling its safety mission and advancing  roadside safety.

section_title: Recommendations
paragraphs: To promote the transition to improved crash test standards, to strengthen  FHWA’s oversight of the roadside safety hardware’s crash-testing  process, and to make more information available to states and industry  on how roadside safety hardware performs in actual conditions, we  recommend that the Secretary of Transportation direct the Administrator  of FHWA to take the following five actions:  1.  Direct FHWA’s division offices to help ensure, through their oversight  of states’ standards and design specifications, that states have written  policies in place to require the installation of appropriately crash- tested roadside safety hardware on the NHS to address inconsistent  practices across states.  2.  Monitor and periodically report to Congress (or report through the  agency’s publicly available website) progress states and the industry  are making in transitioning to the MASH crash-testing standards for  roadside safety hardware.  3.  Provide additional guidance to crash test labs and accreditation  bodies to ensure that labs have a clear separation between device  development and testing in cases where lab employees test devices  that were developed within their parent organization.  4.  Develop a process for third-party verification of results from crash-test  labs.  5.  Support additional research and disseminate results on roadside  safety hardware’s in-service performance, either as part of future  phases of FHWA’s current pilot study on guardrail end terminals’  performance or as part of FHWA’s broader research portfolio.

section_title: Agency Comments
paragraphs: We provided a copy of a draft of this report to the Department of  Transportation for review and comment.  In written comments,  reproduced in appendix III, DOT concurred with GAO’s  recommendations. FHWA also provided technical comments which we  incorporated, as appropriate.

As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to interested congressional  committees and the Secretary of Transportation. In addition, the report  will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-2834 or Flemings@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix IV.

section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report addresses: (1) how FHWA performs oversight of state policies  and practices related to roadside safety hardware; (2) the thoroughness  of the crash-testing process and FHWA’s oversight of this process; and  (3) the extent to which information is available on roadside safety  hardware performance once installed.

To assess FHWA’s role in the oversight of roadside safety hardware and  related state policies, we reviewed FHWA documentation including:  internal memos on roadside safety and implementation of the current  eligibility letter process; guidance memos to the broader roadside safety  hardware community; and policy documents such as the template for  stewardship and oversight agreements agreed to with states. We also  reviewed relevant laws and regulations governing FHWA’s oversight of  roadside safety hardware. We interviewed FHWA officials in headquarters  to understand how policies and practices are carried out. We also applied  federal internal control standards for monitoring, designing control  activities and communication with external stakeholders when reporting  on the agency’s policies and practices for conducting oversight of  roadside safety hardware.

To better understand FHWA’s process in reviewing crash test information  and issuing eligibility letters, we requested and received the FHWA files  for 10 eligibility letters. Two case files were selected by FHWA as  example files; we accepted these files and then selected eight more files  from the roughly 1,000 available. To make our selection, we first limited  the pool to only applications that came to FHWA since 2005. Then we  wanted variation in terms of the following variables: age of application  (variation across those 10 years); new device versus modification to  existing device; type of device; type of standard tested to (MASH or  NCHRP Report 350); and proprietary versus generic. Once the files were  received we reviewed each file to determine whether it had the  information we would expect in order for a third party to understand how  FHWA officials came to the conclusion to issue an eligibility letter.

To better understand states’ roles in the oversight of roadside safety  hardware, we developed and distributed a survey to all 50 states, plus the  District of Columbia and Puerto Rico. Survey questions addressed topics  including policies on crash testing of roadside safety hardware,  procedures for ensuring that only crash-tested hardware is installed on  the national highway system, efforts to collect inventory data on roadside  safety hardware, and what, if any, research states had conducted in the  last 10 years to evaluate the in-service performance of roadside safety  hardware after it is installed. After developing the survey, we conducted  four pre-test interviews with selected states to ensure that the questions  were clear and appropriate for our research objectives. We adjusted the  survey questions as needed in response to feedback prior to survey  distribution.

In October 2015 we distributed the survey to state departments of  transportation representatives from all 50 states, plus the District of  Columbia and Puerto Rico. We followed up and collected responses until  January 2016, at which point we had received responses from 44 out of  52 states and territories. In instances where states did not supply  complete responses to individual questions the answers to those  questions were not included in the survey results, and those states were  removed from the denominator for purposes of summary analysis. For  selected questions, we conducted brief follow-up interviews and solicited  written responses, when appropriate, in order to seek clarification or  elaboration of states’ responses.

To get more information on how states oversee roadside safety hardware,  we selected five states—Maryland, Virginia, Ohio, Texas, and  California—with which to conduct interviews with state departments of  transportation officials and the FHWA’s division offices that oversee the  state departments of transportation. We selected these states based on  the presence of an accredited crash-testing facility in the state and  recommendations from stakeholders regarding the quality of  performance-data collection efforts in those states. In the cases of  Virginia, Ohio, and Texas, interviews with state officials were conducted  on site, and we also conducted interviews with crash-testing lab  personnel and roadside safety hardware developers in the cases of Ohio  and Texas.

To gain information on the thoroughness and independence of the crash- testing process and the extent to which FHWA oversight helps ensure  this, we interviewed the nine domestic crash labs that are accredited, as  required by FHWA, to international crash test lab standards to test  roadside safety hardware for FHWA eligibility letters. To describe how  labs are evaluated against international-testing standards, we reviewed  the international-test lab accreditation standards in ISO 17025 and  interviewed the three accrediting bodies that accredit the nine domestic  crash-testing labs. To evaluate the thoroughness and documentation for  lab crash testing, we reviewed the accreditation requirements in ISO  17025 as well as the crash-testing guidelines in MASH, and analyzed  these documents to create both interview questions and a document  request list for all the labs, in consultation with our technologist. The  questions addressed how labs ensure the quality of the testing  environment, how they interpret test results, how they document each  test, how they comply with conflict of interest requirements in the ISO,  and how communicate with FHWA. We also asked labs to submit their  accreditation reports, quality manual, any relevant conflict-of-interest or  ethics policies, and a sample test report for us to review. We then asked  the nine labs to walk us through a recent example of a product tested for  FHWA compliance, in order to describe how policies and requirements  are implemented in practice. We reviewed the conflict-of-interest policies  to determine the extent to which there was variation in policies across the  labs, and to evaluate whether there were mitigation measures for  potential threats to independence. We also visited four crash test labs  and witnessed two full scale crash tests to gain a better understanding of  the crash testing process. To collect information on how other agencies  oversee lab testing, we reviewed documentation and interviewed officials  from the Environmental Protection Agency’s ENERGY STAR program, as  well as the National Highway Traffic Safety Administration regarding their  vehicle crash testing. Both agencies were referenced in our discussions  with accrediting bodies as examples of other agencies that oversee  accredited testing programs.

To assess the extent of information available on roadside safety hardware  performance once devices are installed, we conducted a literature search  for in-service performance evaluations (ISPE) using government,  academic, and trade publication sources. We also reviewed studies  submitted to us by a highway design and roadside safety hardware  engineering expert. For both sources of studies, we used the National  Cooperative Highway Research Program (NCHRP) Report 490’s  definition of an ISPE to define criteria for determining whether the studies  we reviewed constituted ISPEs for the purposes of our report.  Specifically, we looked for studies that combined analysis of crash data  with real-time site visits. According to NCHRP Report 490, studies that  retroactively or contemporaneously examine crash data are known as  historical studies and collision studies, respectively, whereas an ISPE  adds the element of real-time crash site analysis. NCHRP Report 490  notes that this technique allows researchers to better determine what type  of hardware was struck, whether there were installation techniques or  other site-specific characteristics that contributed to the crash, and  whether the exact device is something a state DOT still uses. We also  stipulated that the study in question involve a specific type of roadside  safety hardware, which we defined according to FHWA’s categories of  hardware for purposes of federal-aid eligibility letters. Moreover, we  restricted our ISPE classification to studies published between 1993 and  2015, when NCHRP Report 350 crash-testing standards were published  and when FHWA first recognized them. As part of our literature search,  we used online search terms that tailored the searches to specific types  of roadside safety hardware as well as key methodological components,  such as site visits.

To inform all of the research questions we also reviewed documentation  and interviewed relevant officials from interested stakeholders. We  reviewed standards and relevant guidance from the American Association  of State Highway and Transportation Officials (AASHTO). To collect  information on how crash test standards are developed and are updated,  we interviewed the AASHTO Technical Committee on Roadside Safety,  as well as officials at the Transportation Research Board’s National  Cooperative Highway Research program. To get more detailed  perspective on how industry, states, and crash-testing facilities  collaborate, we attended a semiannual meeting of Task Force 13, a joint  committee of AASHTO, the Associated General Contractors of America,  and the American Road and Transportation Builders Association, which  develops standards and specifications for bridges and roadside safety  hardware. We also interviewed two roadside safety hardware developers.

We conducted this performance audit from April 2015 to June 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe the  evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

section_title: Appendix II: Review of FHWA Eligibility Letter Files
paragraphs: We reviewed 10 case files for eligibility letters issued between 2005 and  2015 and found that documentation was not sufficient to determine the  rationale behind FHWA’s decision to issue these letters. FHWA officials  explained that as part of their eligibility letter review process, they  examine the crash test lab report, including pictures, videos, and the test- data summary sheets. If FHWA officials have questions they will contact  the lab or developer. However, there is no structured protocol for  documenting the steps FHWA reviewers took or the rationale behind the  decision to issue an eligibility letter. For example, in two cases where  FHWA officials asked questions and received answers from the lab or test  sponsor, it was not possible to trace how FHWA made its determination  to issue an eligibility letter. We also found three instances in which the full  suite of testing was not performed, but no documentation was present  explaining why the lack of testing was acceptable.

We provided this information to FHWA officials who acknowledged that  the basis for those decisions was not documented but stated that in each  case FHWA found the information and reasoning provided by the lab or  test sponsor satisfactory. FHWA officials also told us they made changes  to the eligibility letter review process in 2015 including documenting  communications with developers seeking an eligibility letter, a checklist  for documenting reviews of eligibility letter requests, and updates to the  eligibility letter request form to identify tests that are not critical or not  relevant and the reasons why or why not. FHWA officials told us that this  checklist provides documentation from the submitter concerning why  certain tests were not conducted or why modifications are considered  non-significant to better document this information. We did not evaluate  the impact of these changes since they were made during the course of  our audit work. Officials also characterized these changes as “interim”  because the eligibility letter review process is part of the ongoing  independent Volpe National Transportation Systems Center review of the  program.

section_title: Appendix III: Comments from the Department of Transportation

section_title: Appendix IV: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Steve Cohen (Assistant Director),  Melissa Bodeau, Devin Braun, William Egar, Sarah Farkas, Sarah  Gilliland, Judy Guilliams-Tapia, David Hooper, Leslie Locke, Madhav  Panwar, Malika Rice, Alexandra Squitieri, Jade Winfree, and Elizabeth  Wood made key contributions to this report.


Question-summary annotations: questions: question: What flaw in crash testing laboratories does the FHWA not address?
answer: In general, laboratory crash testing appears to be well documented and thorough; however, FHWA's oversight of the process does not address potential threats to independence.
child_questions: question: How often are there conflicts of interest in these labs?
answer: GAO found that six of the nine accredited U.S. crash test laboratories evaluate products that were developed by employees of the parent organization—a potential threat to lab independence.

question: Why is the independence of crash test labs critical?
answer: FHWA reviews crash tests' results and related documentation, if they are submitted for review, but FHWA relies heavily on the labs to make a pass/fail determination.

question: How common is this issue among other federal agencies?
answer: We found that some other federal agencies in oversight of similar labs' testing settings require third party verification of test results or independent entities to make pass/fail determinations.

question: How would the FHWA’s operations change without these crash test labs?
answer: FHWA does not have a process for formally verifying the testing outcomes and making its own or providing for an independent pass/fail determination.
child_questions: question: How might verification outside of the FHWA, through a third party, be beneficial?
answer: Developing a process for third party verification of roadside safety hardware's lab test results could provide greater assurance that potential threats to independence are fully addressed.

questions: question: How much is known about the in-service performance of roadside safety hardware?
answer: Little is known about the in-service performance of roadside safety hardware because few evaluations of this performance have been done.
child_questions: question: According to FHWA, how should in-service performance be reviewed?
answer: FHWA and AASHTO recommend that states and others perform in-service performance evaluations (ISPE) of installed roadside safety hardware because crash testing cannot fully capture real-world crash conditions.
child_questions: question: To what extent has FHWA's guidance been followed?
answer: However, few ISPEs have been done, in part, because of a lack of inventory and crash data.

question: How is FHWA attempting to review in-service performance?
answer: In the summer of 2015 in four states, FHWA began a pilot study that could provide useful information, but according to FWHA officials, the purpose of this phase of the pilot is to determine best practices on data collection rather than assess performance of roadside safety hardware.

questions: question: To what extent are traffic fatalities caused by vehicles leaving the roadway?
answer: In 2014, 54 percent of traffic fatalities in the United States occurred as a result of a vehicle's leaving the roadway, according to U.S. Department of Transportation's (DOT) data.

question: What hardware is supposed to reduce the risk of a serious crash?
answer: Roadside safety hardware, such as guardrails, is meant to reduce the risk of a serious crash when leaving the roadway.
child_questions: question: How well is this hardware performing?
answer: But in the last several years, a number of serious injuries and deaths resulted from crashes into roadside safety hardware.

question: What aspect of roadside safety is GAO examining?
answer: GAO was asked to review FHWA's oversight framework for roadside safety hardware.