Auto Safety: Status of NHTSA's Redesign of Its Crashworthiness Data System
Published: 2015-03-06T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In 2010, motor vehicle crashes in the United States cost almost 33,000 lives, injured 2.2 million people, and resulted in almost $900 billion in economic costs. As part of its mission to reduce these losses, NHTSA collects and analyzes data on motor vehicle crashes. One NHTSA program that collects crash data is NASS-CDS—a nationally representative sample of police-reported motor-vehicle traffic crashes; however, the NASS-CDS sample was designed in 1988, and subsequent shifts in the population and a declining sample size have necessitated an update of this sample. In 2012, NHTSA started taking steps to redesign NASS-CDS.
Congress mandated GAO to review NHTSA's progress in redesigning NASS-CDS. This report assesses the (1) process NHTSA used to redesign NASS-CDS and (2) the potential for this redesign to improve the NASS-CDS sample. To conduct this review, GAO reviewed relevant information regarding the NASS-CDS redesign and interviewed officials from NHTSA and Westat, the contractor selected to assist NHTSA in redesigning NASS-CDS. Based on comments the public submitted to NHTSA in response to a notice in the Federal Register , GAO also interviewed 21 users of this data and other interested parties regarding the improvements they would like made to NASS-CDS. The Department of Transportation reviewed a draft of this report and provided technical comments, which were incorporated as appropriate.
section_title: What GAO Found
paragraphs: The National Highway Traffic Safety Administration (NHTSA) followed a reasonable process for redesigning the National Automotive Sampling System Crashworthiness Data System (NASS-CDS), which is a nationally representative sample of police-reported motor-vehicle traffic crashes. The Office of Management and Budget (OMB) has standards and guidelines that specify the professional principles and practices that agencies should follow and the level of quality and effort expected when redesigning an existing survey, such as NASS-CDS. NHTSA followed a process consistent with applicable OMB standards and guidelines. For example, NHTSA consulted with NASS-CDS users to identify their requirements and expectations in redesigning NASS-CDS and tasked the contractor, Westat, with developing proposals for a new sample design to meet users' data needs in an effective and efficient manner. As of January 2015, NHTSA planned to replace NASS-CDS with a new sample, called the Crash Investigation Sampling System (CISS). However, NHTSA did not meet a congressional deadline to report on the benefits of increasing the size of the NASS-CDS sample. Specifically, the Moving Ahead for Progress in the 21st Century Act required NHTSA to report, by October 1, 2013, on whether there would be a benefit to increasing the size of the NASS sample as well as to report on the resources necessary to implement NHTSA's recommended sample size, among other things. NHTSA issued its required report in January 2015 as GAO was completing its review. In its report, NHTSA noted that increasing the size of the NASS-CDS sample would help meet the evolving needs of NASS users, but stated there was no precise answer to what an optimal sample size for NASS-CDS would be.
NHTSA expects the new sample it plans to implement as part of this redesign to generate greater statistical precision for key crash-type and injury-severity estimates than that of NASS-CDS using a similarly sized sample. One way NHTSA was able to generate more precise estimates was by selecting new sites at which to collect data. These sites, or “primary sampling units,” better represent the current population and distribution of motor vehicle crashes nationwide, representation that allows NHTSA and others to generate more precise estimates using the data. NHTSA also expects CISS to sample more crashes involving serious injuries and newer vehicles than NASS-CDS currently allows, as users had requested. NHTSA conducted about 4,700 NASS-CDS investigations annually between 1988 and 2013, and while there is no clear optimal sample size, a larger sample size could allow NHTSA to generate estimates that are even more precise or generate estimates for types of crashes that occur infrequently, estimates that could contribute to research that can affect vehicle safety. However, NHTSA's ability to increase the new CISS sample size is limited by its current and expected budget. Additional planned improvements to NASS-CDS include new technologies that allow for safer and more accurate measurements of accident scenes and vehicles involved in crashes. While NHTSA expects these new technologies to also result in some time savings, NHTSA does not expect them to allow for more investigations due to the time-intensive nature of the CISS data-collection effort.

section_title: Letter
paragraphs: In 2010, passenger motor-vehicle crashes in the United States cost  almost 33,000 lives, injured another 2.2-million people, and resulted in  almost $900 billion in economic costs, including intangible costs and  impacts to the victims’ quality of life, according to the National Highway  Traffic Safety Administration (NHTSA). NHTSA was established in 1970  to, among other things, study the causes and impacts of motor vehicle  crashes as part of its mission to save lives, prevent injuries, and reduce  the economic costs that result from crashes. To this end, NHTSA  collects and analyzes data to inform highway safety decision-making at  the federal, state, and local levels. As part of this effort, NHTSA created  the National Automotive Sampling System (NASS). NASS is a data  collection program comprised of two components—the National  Automotive Sampling System Crashworthiness Data System (NASS- CDS) and the National Automotive Sampling System General Estimates  System (NASS-GES)—both of which collect data from nationally  representative samples of police-reported motor-vehicle traffic crashes  and related injuries. NASS-GES consists of data collected from an  annual sample of about 50,000 police accident reports. NASS-CDS, on  the other hand, is a much more in-depth data collection effort for a  smaller sample of crashes—about 4,700 annually on average between  1988 and 2013. For NASS-CDS, NHTSA crash technicians in the field  collect detailed data from crash scenes, vehicles involved in crashes,  and, if possible, interview the individuals involved in the crash. NASS- CDS, which is the focus of this report, is particularly valuable because of  the extensive detail collected from each crash, and data from NASS-CDS,  along with NASS-GES, inform the development of traffic safety  regulations and are used by the automotive industry and others to  develop and analyze vehicle safety features.
The NASS-CDS sample was designed in 1988, and NHTSA and others  have raised concerns that shift in both the population and the number and  type of crashes nationwide necessitate an update to the sample design to  reflect these shifts and remain nationally representative. NASS users  have also raised a concern that the current NASS-CDS sample size is too  small to adequately inform decisions on traffic safety issues. In 2011, the  Congress provided NHTSA with $25 million in funding to begin  redesigning NASS. provision in the Moving Ahead for Progress in the 21st Century Act (MAP- 21) requiring NHTSA to review the quality of data collected as part of  NASS and report on whether there was a benefit to increasing the size of  the NASS-CDS sample.NHTSA to obtain input from interested parties, including automobile  manufacturers, safety advocates, the medical community, and research  organizations. NHTSA began redesigning NASS in 2012 and is expecting  to begin implementing a new sample to replace NASS-CDS in 2015.
Pub. L. No. 112–55, 125 Stat. 552, 658 (2011). The funds provided were from funds for  NHTSA’s Safety Belt Performance Grants program, which was created to encourage  states to enact and enforce seat belt laws.
The Senate Report accompanying the Transportation and Housing and  Urban Development, and Related Agencies Appropriations Bill, 2014,  mandated GAO to review NHTSA’s progress in redesigning NASS-CDS.  This report assesses (1) the process NHTSA has used to redesign  NASS-CDS and (2) the potential for this redesign to improve the NASS- CDS sample.
To assess the process NHTSA used to redesign NASS-CDS, we  reviewed pertinent documents related to the NASS redesign. We also  interviewed NHTSA officials from the National Center for Statistics and  Analysis—a component within NHTSA that oversees the agency’s data  collection efforts, including NASS—and representatives of Westat, the  contractor selected to assist NHTSA in redesigning the NASS-CDS  sample. Based on comments the public submitted to NHTSA in response  to a notice in the Federal Register, we also interviewed 21 NASS users—  including 4 auto manufacturers, 3 automotive suppliers, 4 safety  advocates, 2 members of the medical community, 1 federal agency, and  representatives from 7 research organizations—to understand how they  use NASS-CDS and the improvements they would like to see NHTSA  make to NASS-CDS as part of the redesign. We selected these 21 NASS  users by first contacting all those who submitted comments to NHTSA on  the redesign and then asking these initial contacts what other NASS  users we should interview. The results of our discussions with NASS  users are not generalizable to all NASS users but provide insights into  aspects of NASS-CDS that some users indicated they would like to see  improved. In addition, we visited two of the geographic locations, called  primary sampling units (PSU), where NHTSA collects NASS-CDS data, to  observe NHTSA’s crash technicians conduct their work, and spoke with  NHTSA crash technicians at two others. The PSUs we visited were  Seattle, Washington, and King County, Washington; the PSUs we  contacted were Allegheny County, Pennsylvania, and Muskegon County,  Michigan. We selected these locations to ensure we included each type  of PSU (i.e., urban, county, or group of counties) and to ensure that we  included at least one PSU from each of the two contractors that NHTSA  currently uses to implement the program. The results of our discussions  with PSUs are not generalizable to all PSUs but provide insights into  aspects of the work crash technicians do.
We assessed NHTSA’s efforts to redesign NASS based on government- wide standards and guidelines issued by the Office of Management and  Budget (OMB) that apply to the development and implementation of  statistical surveys such as NASS. OMB’s standards and guidelines  provide a framework for the development of survey concepts, methods,  and design; collecting data; processing data; producing estimates;  analyzing data; reviewing procedures; and disseminating the results. The  standards and guidelines also specify the professional principles and  practices that federal agencies should follow and the level of quality and  effort expected when initiating a new survey or redesigning an existing  survey such as NASS-CDS. Because NHTSA was in the process of  redesigning NASS at the time of our review, we focused our assessment  on reviewing NHTSA’s processes as they relate to the development of  survey concepts, methods, and design.
To assess the potential for the new sample design to improve NASS data,  increase the precision of estimates, and increase the sample size, a team  that included GAO social science analysts with statistical survey expertise  reviewed the sampling methodology for the current NASS-CDS sample  and the design proposed for the new CISS sample. As a part of this  review, we analyzed the proposed changes to the sample design, the  number of PSUs chosen, the overall sample size recommended, and  NHTSA’s available budget resources for the new sample. We compared  the proposed redesign with literature on efficient statistical sample design  to assess the reasonableness of the redesign. We also assessed the  extent to which NHTSA’s proposed design was responsive to user needs,  according to what we learned from our NASS user interviews. Finally, we  interviewed NHTSA and Westat officials on the new sample design.  Appendix I contains additional information on our objectives, scope, and  methodology.
We conducted this performance audit from April 2014 through March  2015 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: NHTSA’s mission is to prevent motor vehicle crashes and reduce injuries,  fatalities, and economic losses associated with these crashes. To carry  out this mission, NHTSA conducts a range of safety-related activities,  including setting vehicle safety standards; investigating possible safety  defects and taking steps to help ensure that products meet safety  standards and are not defective (through recalls if necessary); providing  guidance and other assistance to states to help address traffic safety  issues, such as drunk driving and distracted driving; and collecting and  analyzing data on crashes. In fiscal year 2014, NHTSA’s enacted budget  was $819 million.
NHTSA collects and analyzes crash data for a variety of purposes, such  as to determine the extent of a safety problem and what steps NHTSA  should take to develop countermeasures. NHTSA collects data through  detailed, in-depth investigations as well as to generate national statistics  and nationally representative data, as shown in table 1.
As mentioned previously, this report focuses on NASS-CDS. NHTSA  collects NASS-CDS data through in-depth investigations of a sample of  police-reported motor vehicle crashes that occur in the United States. The  data collected through NASS-CDS are detailed and descriptive and allow  NHTSA and others to assess the crashworthiness of different types of  vehicles, evaluate different vehicle safety systems and designs, and  understand the nature of injuries that people sustain during crashes.  NHTSA uses the data collected as part of NASS-CDS for statistical  analyses in its rulemaking and to estimate the size of the population that  might be affected by its rulemaking. NHTSA also uses NASS-CDS data  for other purposes, such as to identify existing and potential traffic-safety  problems. For example, NHTSA has used NASS-CDS data to investigate  patterns of roof intrusion into a vehicle resulting from real-world rollover  crashes. According to NHTSA, NASS-CDS data showed that the damage  and intrusion that occurred during real-world crashes was greater than  the damage and intrusion that occurred during crash tests, pointing to the  need to revisit NHTSA’s standards for the strength requirements for a  vehicle’s roof. Others, including other federal agencies, universities,  research institutions, and the automobile and insurance industries, use  NASS-CDS data to understand the nature and consequences of real  world crashes. For example, the National Transportation Safety Board, an  independent federal agency, uses NASS-CDS data for research purposes  as well as for conducting its accident investigations, whereas automobile  manufacturers may use NASS-CDS data to study crash patterns and how  those patterns have changed over time in order to prioritize their own  research on vehicle designs and safety features.
NHTSA collects NASS-CDS data and information using stratified  sampling—a statistical method of sampling in which a population is  divided into two or more parts (called strata) and a sample is selected  from each part (or stratum). NASS-CDS, specifically, is a stratified, three- stage probability sample, as illustrated in figure 1.
The first stage of the NASS-CDS sample was the selection of PSUs—the  geographic locations where NHTSA collects data. NHTSA defined the  PSUs so that their minimum population was approximately 50,000, and  each PSU consisted of a central city, a county, a group of counties, or a  portion of a large county excluding a central city. The PSUs were grouped  into 12 strata based on geographic region (i.e., Northeast, South,  Midwest, and West) and urbanization type (i.e., large central cities, large  suburban areas, all others). The PSUs to be sampled were allocated to  each stratum roughly proportional to the number of crashes in each  stratum, and at least two PSUs were then selected from each stratum. As  of 2014, a total of 24 PSUs comprised the NASS-CDS sample, as shown  in figure 2.
The second stage was the selection of police jurisdictions within the  sampled PSUs. About 170 police jurisdictions across the United States  are part of the NASS-CDS sample and the number of jurisdictions per  PSU varies (e.g., the Seattle, Washington PSU has two police  jurisdictions in its sample, whereas the King County, Washington PSU  has seven). Each police jurisdiction was assigned a “measure of size”  that reflects the number, severity, and type of crashes in each jurisdiction.  A sample of police jurisdictions was then selected from each sampled  PSU, and those jurisdictions having a larger measure of size were  oversampled.
The third and final stage is the ongoing selection of the actual police  accident reports that are filled out by a police officer at the scene of a  motor vehicle crash. Each week, the sampled police jurisdictions are  contacted and all police accident reports that have accumulated since the  previous week are reviewed and classified into a stratum based on the  types of vehicles involved, most severe police-reported injury, disposition  of the injured, tow status, and model year of the vehicles.
To be eligible for inclusion in NASS-CDS, a motor vehicle crash must (1)  be police-reported, (2) involve a harmful event resulting from the crash  (such as property damage or personal injury), and (3) involve at least one  passenger car, light truck, van, or sport utility vehicle in transport on a  traffic-way that was towed from the scene due to damage. The gross  vehicle weight rating should be less than 10,000 pounds. Crashes are  selected so that a larger percentage of higher severity crashes are  selected than lower severity crashes, but every motor vehicle crash that  occurs within one of the PSUs where NASS-CDS data are collected and  that meets these conditions has a chance of being selected for  investigation. NHTSA selected the first stage of the current NASS-CDS  sample in 1988 and the second stage in 1995 but selects the police  accident reports weekly so that the evidence from the motor vehicle  crashes that might be investigated is still intact and the memory of the  individuals involved is still fresh.
NHTSA contracts with two companies that use small teams of crash  technicians located across the country to collect NASS-CDS data. These  teams typically include a team leader, one or two crash technicians, and  an assistant, and each team reports to one of two contractor-led control  centers, called zone centers. NHTSA’s crash technicians collect over 600  data elements during their investigations, including information on the  damage vehicles sustained, the crash forces involved, injuries to victims,  and factors that caused those injuries. Those investigations generally  involve inspecting the scene of a crash and the vehicles involved;  interviewing the drivers and occupants involved, if possible; reviewing  official medical reports detailing any injuries sustained; and  reconstructing what happened during the crash, as shown in figure 3. The  crash technicians coordinate with law enforcement agencies, hospitals,  tow yard operators, repair garages, and the drivers and occupants  involved in the crashes while performing their work, and the information  they collect is subject to review by NHTSA. That information, in turn, is then made available to NHTSA and the public for research purposes.
The number of NASS-CDS investigations NHTSA conducts each year  varies, as shown in figure 4. Between 1988 and 2013, NHTSA conducted  an average of about 4,700 NASS-CDS investigations each year.  However, since 2009, the number of NASS-CDS investigations  conducted has steadily decreased, and in 2013, only about 3,400 NASS- CDS investigations were conducted. According to NHTSA, factors that  have contributed to this decline include the budget for NASS-CDS and  rising costs. For example, funding for NASS-CDS has been flat-lined  since 2010, whereas costs—including costs for labor, information  technology, leases and fuel—have risen.
NHTSA’s effort to redesign NASS-CDS is part of NHTSA’s larger Data  Modernization Project, begun in 2012, which also affects NASS-GES and  FARS. Specific to NASS-CDS, NHTSA’s Data Modernization Project  involves the following:  redesigning the NASS-CDS sample by reviewing the data elements  that comprise the sample and the statistical methodology behind  selecting the sample; upgrading the equipment and information technology that supports  NASS-CDS to reduce redundancy, improve data quality, and enhance  the experience of NASS users; and  implementing a new sample to replace NASS-CDS.
section_title: NHTSA Followed a Reasonable Process for Redesigning NASS-CDS
subsections: section_title: NHTSA Redesigned NASS-CDS in Accordance with Applicable OMB Standards and Guidelines
paragraphs: We found that the process NHTSA followed to redesign NASS-CDS is  consistent with applicable government-wide standards and guidelines  issued by OMB that apply to the development of survey concepts,  methods, and design. OMB’s standards and guidelines specify the  professional principles and practices that federal agencies are required to  adhere to and the level of quality and effort expected when initiating a  new survey or redesigning an existing survey. In the case of redesigning  NASS-CDS, the OMB standards and guidelines that apply include  recommended practices for the development of survey concepts,  methods and design. As such, they highlight the importance of  consulting with potential users to identify their requirements and  expectations,  including design elements in a sample to meet stated objectives, and  testing a survey’s components prior to full-scale implementation.
OMB’s standards and guidelines are not intended to substitute for the  extensive existing literature on statistical and survey theory, methods, and  operations. Further, these standards and guidelines specify that agencies  should engage knowledgeable and experienced survey practitioners to  effectively achieve the goals of OMB’s standards. While the process  NHTSA has followed is consistent with applicable OMB standards and  guidelines, NHTSA has not yet started implementing its new sample  design. Accordingly, we were not able to assess its implementation  efforts.
To redesign NASS-CDS, NHTSA awarded a contract to Westat in May  2012 to assist the agency in redesigning the NASS-CDS sample. Westat  provides services relating to survey planning, design, development, and  administration and analysis, and Westat researchers are known to be  experts in the field of survey sampling. Westat’s tasks included  reviewing the data elements NASS-CDS collects as well as the statistical  methodology behind NASS-CDS. NHTSA, in conjunction with Westat,  solicited comments from NASS users through the Federal Register and  held a public listening session with NASS users—steps that are  consistent with OMB’s recommended practice to consult with potential  users to identify their requirements and expectations. Through the  Federal Register, in June 2012, NHTSA solicited and subsequently  received comments from 25 individuals and organizations regarding the  redesign and their data needs. NHTSA also held a public listening  session with NASS users in July 2013. During this listening session,  NHTSA officials provided users with an update on its progress in  redesigning NASS as well as an opportunity to provide additional  comments, and eight NASS users provided comments at that listening  session. According to NHTSA officials, the comments received indicated  that users generally wanted NHTSA to increase the NASS-CDS sample  size, collect additional data during NASS-CDS investigations, and  improve the quality of the data collected. This includes collecting  additional data from event data recorders and on the use of crash- avoidance technologies, as well as more detailed diagrams of the scenes  of crashes.
Consistent with OMB’s recommended practice to include design elements  to meet stated objectives, NHTSA tasked Westat with (1) identifying data  elements that are responsive to the current and future needs of both  NHTSA and the public and (2) developing recommendations for a new  sample design that met users’ data needs in an effective and efficient  manner while still maintaining national representativeness. As part of its  review, Westat reviewed the comments NASS users submitted and also  assembled a team of experts in crash investigation, transportation safety  research, and injury control to review NASS’s data elements, identify  areas of research that should be better addressed in the future, and make  recommendations. Both NHTSA and Westat considered the feasibility of  suggestions users made to fundamentally change how NASS-CDS data  are collected. For example, some users commented that police officers  who fill out accident reports at crash scenes could do more to assist  NHTSA’s data collection efforts, such as by photographing the crash  scene and the vehicles involved at the time of the crash. However,  according to NHTSA officials, such suggestions were deemed not to be  practical because they would require resources from NHTSA to provide  equipment and training to law enforcement officials to implement. NHTSA  officials also noted that police officers on the scene might not be willing to  cooperate with additional data collection duties when they are responding  to a crash, and police jurisdictions have varying technological capacities  to handle the storage and dissemination of photos or other additional  data.
Westat officials also told us they analyzed the NASS-CDS sample design  to identify its limitations. This examination included reviewing the sample  size, stratification and sampling allocation, and weighting procedures.
Westat and NHTSA considered various alternative design options for the  new sample design, and NHTSA chose a probability-based approach to  meet its objective of maintaining national representativeness. To assess  whether Westat used the appropriate statistical survey design principles  and methodology to ensure that its objectives would be met, NHTSA had  early drafts of Westat’s work reviewed by three independent  consultants. We have previously reported that such reviews can  improve the technical quality of a project and enhance the credibility of  the decision-making process, and as a result of these independent  reviews, NHTSA officials said they felt confident moving forward with  Westat’s proposals for the new sample design. Thus, in May 2014,  NHTSA announced that it planned to replace NASS-CDS with a new  system called the Crash Investigation Sampling System (CISS)—which  we discuss in detail in objective 2.
Consistent with OMB’s recommended practice to test a survey’s  components prior to full-scale implementation, NHTSA plans to  implement the new CISS PSUs in phases. As of January 2015, NHTSA’s  plans for the new sample call for initially implementing 24 new PSUs as a  first phase and up to 73 PSUs in the future, if its budget allows. Prior to  implementing all 24 new PSUs that comprise phase 1, NHTSA plans to  first implement 5 of the PSUs, which, among other things, will allow  NHTSA to test out the sample design and new equipment, such as  electronic distance-measuring equipment that will support its data  collection, among other things, prior to implementing the remaining  PSUs. Figure 5 shows the location of the new CISS PSUs.
To pay for the Data Modernization Project, the Congress provided  NHTSA with $25 million in 2011 and another $3.5 million through its fiscal  year 2014 appropriation. Of this available funding, NHTSA officials said  they allocated $2,500,000 (9 percent) to redesign the NASS-CDS and  NASS-GES samples; $16,500,000 (58 percent) for information- technology infrastructure upgrades and new equipment, such as  electronic distance-measuring equipment; and $9,500,000 (33 percent) to  implement the new samples. Because NHTSA has not yet started  implementing the new samples or obligated all of the funding for new  equipment, about $12 million of the $28.5 million provided was still  available as of the time of our review. Table 2 shows the funding  Congress provided and NHTSA’s reported obligations, as of December 1,  2014.
However, as of the time of our review, NHTSA officials told us their time  frames to begin implementing the new CISS PSUs were uncertain due to  a government-wide cap on travel spending currently in place. Specifically,  in 2012 OMB issued a memorandum, entitled Promoting Efficient  Spending to Support Agency Operations, which directed agencies to  spend at least 30 percent less on travel expenses than in fiscal year 2010  and to maintain that level of spending through fiscal year 2016.According to NHTSA, this cap on travel spending could delay its plans  because implementing the new CISS PSUs requires that NHTSA staff  travel to train the new crash technicians as well as to gain the cooperation  of police jurisdictions, tow yards, and others. If no increases are provided,  NHTSA stated that it would try to mitigate some of this limitation by, for  example, training the new crash technicians at a local facility. NHTSA  officials said they are currently working to obtain relief from this cap and  hope to start implementing the new CISS PSUs beginning in 2015.  According to NHTSA officials, failure to obtain relief from this cap could  result in delays or additional costs in implementing the new PSUs.
section_title: NHTSA Was Late in Reporting on the Benefits of a Larger Sample
paragraphs: While we found NHTSA’s approach to redesigning NASS-CDS has been  reasonable, we note that NHTSA was not timely in responding to  Congress’ direction to provide information on the size of the NASS-CDS  sample. Specifically, MAP-21 required NHTSA to conduct a  comprehensive review of the data elements collected as part of NASS  and report on whether there was a benefit to increasing the size of the  NASS sample. For example, the act required NHTSA to provide  Congress with information on the types of analyses that can be  conducted and the conclusions that can be drawn under the current  sample size and an expanded sample size, the number of investigations  that NHTSA should conduct as part of the sample that would allow for  optimal data analysis, NHTSA’s recommendations for improvements, and  the resources necessary to implement NHTSA’s recommendations. The  act also required that NHTSA obtain input from interested parties,  including automobile manufacturers, safety advocates, the medical  community, and research organizations. The act required NHTSA to  report to Congress on the results of its review, including the benefits of a  larger sample size, no later than October 1, 2013. NHTSA missed this  deadline and issued its report in January 2015, as we were completing  our review.
In its report, NHTSA stated that meeting the needs of all NASS users is a  challenge and that there is no precise answer to what the optimal sample  size for NASS-CDS would be. However, NHTSA also noted that  increasing the size of the NASS-CDS sample would help meet the  evolving needs of NASS-CDS users. We agree with NHTSA that there is  no precise answer to what the optimal sample size for NASS-CDS is, and  discuss this in more detail in objective 2.
section_title: The CISS May Not Substantially Increase the Sample Size but May Improve Precision and Data Quality
subsections: section_title: Although Its Sample Size May Not Substantially Increase, NHTSA Still Expects the CISS to Generate More Precise Crash and Injury Estimates
paragraphs: One means of determining the extent the Data Modernization Project  redesign will improve the NASS-CDS sample is to assess the potential for  CISS to meet a main technical objective of the Data Modernization  Project: achieving similar or greater levels of statistical precision for seven  important crash and injury estimates. Four of these measures are for  crash types—rear-end crashes, head-on crashes, angle crashes, and  rollovers and three are for injury-severity—incapacitating, non- incapacitating, and fatal.
The statistical precision of an estimate provides a measure of how close  the estimate is expected to be to the population value it is attempting to  describe. Improving the statistical precision allows for more accurate  estimates and, in turn, informs the language NHTSA uses to make  projections from the sample that apply to the whole population.  Comparing the precision of the estimates NASS-CDS generates to the  expected precision of the new CISS estimates is a method of determining  whether the sample design has improved.
The precision of a sample’s estimates can be increased by selecting a  larger sample, using a more efficient sample design, or both. When a  more efficient sample design is used, it is possible to generate estimates  with similar or greater levels of precision with a smaller sample size.  While NASS users indicated that they wanted to see an increase in the  size of the sample as part of the redesign, NHTSA officials stated that  expanding the sample size would increase the cost of collecting the data  for an extensive data collection effort like NASS-CDS, beyond expected  budgetary resources, which we will discuss in more detail later in this  report. Decisions made in the process of designing a sample must  balance available resources and the ability of the sample to meet the  stated objectives within the defined precision requirements.
NHTSA expects the new CISS design to achieve similar or greater levels  of precision for NHTSA’s 7 key estimates by using a more efficient  sample design, not by substantially increasing the sample size from the  historical average for NASS-CDS. Westat developed several proposed  sample designs and made design recommendations to NHTSA. NHTSA  then modified Westat’s recommended design to produce similar or more  precise results for the 7 key estimates using a sample of 24 PSUs, which  would result in between 4,000 and 4,500 investigations annually. By way  of comparison, in recent years, NASS-CDS has produced a sample size  of about 3,500 investigations annually; however, between 1988 and 2013,  has produced about 4,700 investigations annually. According to NHTSA,  even though the expected sample size for CISS is comparable to the  historical average for the NASS-CDS sample, the end result is that the  new design that NHTSA is pursuing for CISS should be as precise if not  more so than the current NASS-CDS design for the key estimates  NHTSA indentified. Table 3 below summarizes the differences between  NASS-CDS and CISS.
There are many ways to design a sample to generate more statistically  precise estimates. One way NHTSA improved the expected statistical  precision of estimates from CISS was by selecting new PSUs that better  represent the current population and the number and types of crashes  nationwide. According to statistical literature, in a statistical sample that  uses the same PSUs for a number of years such as NASS-CDS, PSUs  should be re-selected periodically in order to ensure that the sample  reflects the total population the sample is attempting to describe.  However, the current PSUs for NASS-CDS were selected in 1988. Since  NHTSA selected the current CDS PSU sample based on population and  crash counts from more than 30 years ago, the CDS PSU sample has  gradually become less representative of the population and crashes in the  United States, and as a result, CDS estimates have become less  statistically precise. By reselecting new PSUs, CISS data are expected to  better represent the population and the areas in which the highest  number of crashes with serious injuries occur, according to NHTSA.  Consequently, the selection of PSUs is expected to allow for more  precise estimation of crashes involving serious injuries. The improvement  in the representativeness of the selected PSUs contributes to the  improvement in statistical precision without increasing the sample size,  making the new sample design more statistically efficient.
Moreover, NHTSA expects the new sample design for CISS to contain  more crashes with serious injuries and crashes involving newer vehicles  than NASS-CDS currently contains, which also should make estimates of  these crashes more statistically precise. For example, NHTSA designed  CISS so that 10 percent of the police accident reports selected for CISS  investigations will contain a newer vehicle and an incapacitating injury, up  from 6.9 percent in NASS-CDS. The higher sampling rate for newer  vehicles and serious injury crashes is expected to increase the number of  these crashes selected. This step will improve the precision of estimates  and address some users need to have more of these types of crashes in  the sample. NASS users who provided NHTSA with comments about the  NASS redesign indicated they wanted both of these changes.
NHTSA’s determination of the sample size was dependent upon available  resources, and NHTSA emphasized this in its January 2015 report to  Congress. According to both NHTSA and Westat officials, budget  constraints were the key factor driving both the new sample design and  the decision not to increase the sample size. The budget for NASS-CDS  has remained at $12,500,000 per year since 2010, and NHTSA officials  also told us that the future budget for CISS remains uncertain. Because of  the budget constraints, Westat recommended a design for the new  sample with the fewest number of PSUs, police jurisdictions, and police  accident reports that would meet NHTSA’s precision requirements and  that NHTSA could realistically afford given its budget. In addition, there  are limitations to how many investigations NHTSA’s crash technicians  can conduct. Specifically, according to NHTSA, crash technicians can  only currently conduct about 3 investigations every 2 weeks, to ensure  that their investigations are high quality and thorough. The design that  NHTSA is pursuing is expected to cost about $13.5 million annually.  According to NHTSA officials, the expected cost for CISS is higher than  the current NASS budget. NHTSA can afford to implement 24 PSUs at  this time, because, according to NHTSA officials, the amount  appropriated for Highway Safety Research and Operations in fiscal year  2014 included a $5 million increase supporting the operating budget for  crash data collection that can be used to supplement the NASS-CDS  budget. NHTSA officials said that because this funding was added to  NHTSA’s base budget, they expect the funding will be available in future  years. However, according to NHTSA, this funding would need to keep  pace with inflation to help offset expected increases in operational costs.
section_title: The New Design Will Allow NHTSA to Increase or Decrease the Sample Size If Future Budgets Change
paragraphs: Another improvement as a result of the Data Modernization Project is the  flexibility of the new sample design. Although NHTSA did not pursue a  larger sample compared to historic levels due to budget constraints, the  new sample design will allow NHTSA to add or subtract PSUs or police  jurisdictions to increase or decrease the sample size in the future if its  budget changes. Adding PSUs to increase the sample size is more  efficient than simply adding more investigations within the selected 24  PSUs. According to statistical literature, greater increases in precision are  achieved by increasing the number of PSUs rather than the number of  investigations that are conducted per PSU. Additionally, according to  NHTSA, because of limits in the number of crashes involving serious  injuries or crashes involving newer vehicles within a particular geographic  area, adding an additional PSU provides a new pool of crashes to sample  from. NHTSA built this flexibility into the sample design to address the  uncertainty of the future budget and to allow for sample size expansion if  future budgets allow. To build this flexibility into the sample design,  NHTSA identified 73 PSUs, which represents NHTSA’s estimation of its  preferred sample size for CISS, that it can bring online one at a time as  resources become available. However, adding PSUs to increase  precision and increase the sample size is more expensive than adding  more investigations within the selected 24 PSUs, as described above.
The sample size of 24 PSUs can also be reduced if budgets are further  constrained, but that could jeopardize the gains in statistical precision  achieved with the new sample design. NHTSA developed projections to  illustrate what size of a sample the agency could potentially implement  given future CISS budgets. For example, according to NHTSA, if the  budget for CISS was reduced to $11 million, even with a higher caseload  than crash technicians currently conduct, the most investigations NHTSA  could conduct annually is just under 2,600. If the budget for CISS was  increased to $20 million, NHTSA could conduct over 5,000 investigations  a year. The smaller sample for $11 million would be expected to produce  less precise estimates than the larger sample for $20 million.
Even though NHTSA expects its new design to meet its precision  requirements for the seven key crash and injury estimates it identified,  NHTSA officials said the design may not meet precision requirements for  other estimates or include a sufficient number of specific crash-types that  occur infrequently (rare crash populations). According to NHTSA, the  optimal sample size for CISS is impossible to determine without first  defining which estimates should meet precision requirements or which  rare crash populations are required to meet other analytic needs. For  example, NHTSA officials said that 73 PSUs selecting about 15,000  investigations annually would be a reasonable sample size not only for  attempting to meet precision requirements for additional estimates but  also for obtaining estimates for rare populations. A rare population can  be crashes such as a side impact crash involving a child, which despite  resulting in the death or injury of about 6,500 children under age 15, only  accounted for about 0.1 percent of all crashes and 0.5 percent of serious  injury crashes in 2011. From this small of a percentage of the total crash  population, CISS as designed with 24 PSUs can be expected to select  around 4 crashes per year for investigation. Increasing the sample size to  73 PSUs and 15,000 investigations could increase the number of  selected serious injury side impact crashes involving an injured child to  about 20 per year, according to NHTSA analysis.
A sample of this size would allow NHTSA and external CISS users to  better study these relatively rare crash populations and generalize their  findings for these crashes to the population of all side impact crashes  involving injured children. However, operating those 73 PSUs could cost  at least three times the $13.5 million currently planned for CISS, or  approximately $41 million annually, according to NHTSA officials. A  smaller increase in sample size would also increase the number of  selected side-impact crashes, but to a lesser extent. For example,  according to NHTSA, a sample of about 7,500 investigations could  increase the number of selected serious injury side-impact crashes  involving an injured child to about 10 per year. According to NHTSA, this  would require operating about 40 PSUs and cost about twice what is  currently planned for CISS. This smaller increase would allow users to  better study these populations but would require more time to accumulate  enough cases to generalize their findings. However, NHTSA also noted  that it does not like to combine more than 5 years of crash data to shed  light on a problem that depends on the ever-changing crash environment,  and that 5 years of data should produce between15 and 20 cases for  even very rare crash populations (0.1 percent of serious injury crashes in  a given year). Three users we interviewed estimated that a sample  somewhere around 10,000 investigations per year would make the data  considerably more useful to them.
While the larger sample size would allow NHTSA to produce more  estimates that are precise and more investigations for rare populations, it  is not possible to quantify the benefit of this increase in precision. It is  also not possible to determine the sample size that would result in the  highest value for society in terms of reducing the human life and  economic costs of motor vehicle crashes because the causal link  between the data collected and the potential benefits, if any, is not  possible to establish.
NHTSA could implement a sample even larger than 73 PSUs if resources  allowed, but the sample sizes required to produce estimates for certain  sub-groups that some NASS users had requested, such as at the make  and model level would be impractical. According to NHTSA officials, such  an estimate would require a sample size that is both not possible to  determine for all vehicle types and would exceed any reasonable  expectation of resources available for CISS. There are many types of  vehicles, some of which are more common (such as the Ford F-150 pick- up truck) than others (such as the Tesla Model S). For any make and  model vehicle, only a small percentage are involved in a crash that would  be eligible for inclusion in the CISS sample. Whether a sample is large  enough to yield an adequate number of crash types involving a particular  make and model vehicle depends in large part on the number of crashes  involving that vehicle type. According to GAO analysis, similar to the  above example of side impact crashes resulting in the death or injury of a  child, in order to identify 20 side impact crashes resulting in death or  injury involving a particular make and model vehicle, there would have to  be about 6,500 side impact crashes resulting in death or injury nationwide  involving that make and model vehicle in one year, which is highly  unlikely. As the crash-type of interest approaches very small percentages  of the total number of crashes, it becomes less probable that the sample  will adequately capture these crashes. Since NHTSA only investigates  several thousand crashes each year, the percentage chance of even one  of these investigations being selected for a NASS-CDS investigation is  very small. Officials noted that NHTSA’s Special Crash Investigations  (SCI) program conducts investigations into issues that arise from specific  agency special needs, and those investigations could include make- and  model-level defects and other issues. NHTSA currently has three SCI  teams that travel to investigate crashes according to agency priorities and  recalls, separate from current NASS-CDS sampled investigations.
In addition to the ability to scale the sample size up or down, the new  sample design also allows NHTSA to substitute a PSU, police jurisdiction,  or police accident reports, according to NHTSA officials. A PSU or police  jurisdiction can be replaced if there are cooperation or information sharing  challenges. During implementation, substituting a PSU or police  jurisdiction would be less challenging than after the CISS is fully  implemented. Substituting a PSU after implementation would require  hiring and training new crash technicians. A police accident report can be  replaced if it is incomplete or cannot be thoroughly researched, but  replacing a police accident report was purposefully made difficult to avoid  the potential for bias from crash technicians who choose to replace a  police accident report for their own reasons. For example, if a police  accident report included a vehicle that was impossible to locate for  inspection, the technician could allow the sampling algorithm to select a  replacement police accident report.
Finally, the new design allows NHTSA to implement separate modules to  study crashes involving large trucks, motorcyclists, bicyclists, and  pedestrians—as NASS users had requested. Westat developed initial  plans for each of these subsets as additional modules that could be  conducted as separate studies utilizing the CISS sites.
section_title: Data Accuracy Is Also Expected to Improve Because of New Technology, but the New Technology Will Not Increase the Number of Investigations
paragraphs: As part of the Data Modernization Project, NHTSA also plans to equip its  crash technicians with new technology to help improve the efficiency and  accuracy of the data they collect. Improving the accuracy of NASS-CDS  data with more electronic data collection methods was one aspect of  NASS that users indicated they hoped NHTSA would address as part of  the redesign. For example, in the comments NHTSA received in response  to its Federal Register notice, NASS users indicated they wanted scalable  diagrams of crash scenes. Currently, crash technicians collect NASS- CDS data using paper forms and have to enter the roadway to manually  measure a crash scene using measuring wheels and tape measures.  Afterward, they have to manually enter their measurements into a  computer program, which creates an electronic image of the crash scene,  which in turn is made available to NASS users. However, according to  NHTSA, those drawings are not ideal when attempting to conduct  detailed research of a crash scene because the diagrams provided are  not scalable. The new equipment NHTSA plans to provide its crash  technicians include tablet computers, which will allow crash technicians to  electronically collect and transmit data remotely from the field; new  accident reconstruction software, which will automatically create scalable  diagrams of crash scenes; and new electronic distance measuring  equipment, which is expected to improve the accuracy and efficiency of  scene and vehicle inspections while also allowing crash technicians to  take their scene measurements safely from the roadside. Figure 6 shows  NHTSA crash technicians collecting crash scene data using tape  measurements in the street and using new technology from the side of  the road.
While NHTSA expects new equipment will help its crash technicians  collect more accurate data, it does not expect the new equipment will  considerably reduce the time it takes to conduct an investigation or allow  its crash technicians to conduct more investigations. NHTSA officials said  the new equipment they plan to provide should help reduce the time it  currently takes to conduct scene and vehicle inspections. However, this  represents only a portion of the time NHTSA’s crash technicians spend  each week performing scene and vehicle inspections. Further, as part of  the NASS redesign NHTSA is also increasing the amount of information  that its crash technicians collect, which, in turn, will require more time to  collect. This includes data on the use of crash-avoidance technologies in  newer vehicles as well as additional data for older vehicles, as some  users had requested. Thus, the new equipment will not substantially  decrease the amount of time technicians spend overall collecting data for  crash investigations or the cost of collecting this data.
Collecting data this detailed is expensive and time consuming. For  example, according to a study NHTSA conducted in 2012, an average  NASS-CDS investigation takes about 25 hours to perform, and NHTSA’s  crash technicians spend, on average, about 10 percent of that time  inspecting crash scenes and about 25 percent of their time inspecting  vehicles. In contrast, NHTSA’s crash technicians spend 13 percent of  their time sampling police accident reports for investigations. Figure 7  below provides information on the percentage of hours per week on  average NHTSA’s crash technicians spend performing various aspects of  NASS-CDS investigations. Because NHTSA has not yet started collecting  data for CISS, it is not possible to determine the number of hours per  week crash technicians will spend on various aspects of the sampling and  data collection for CISS crash investigations.
section_title: Concluding Observations
paragraphs: NASS-CDS provides NHTSA and others with an important source of data  to understand the real-world nature and consequences of motor-vehicle  traffic crashes. In redesigning NASS-CDS, NHTSA has followed a  process that is consistent with applicable government-wide standards and  guidance for redesigning statistical surveys. NHTSA has also taken steps  to improve upon the original design for NASS-CDS in developing CISS— the system that will replace NASS-CDS—such as by making the sample  more precise as well as by making the sample design more flexible to  adapt to future budgets. While the proposed sample size will be sufficient  to meet NHTSA’s requirements for the program, NHTSA does not plan to  substantially increase the sample size. By increasing the size of the CISS  sample, NHTSA and others could likely do more to study motor-vehicle  traffic crashes in an effort to save lives and reduce the economic costs of  crashes. Sampling sufficient cases to conduct analyses of rare  populations requires a significantly larger sample. However, NHTSA’s  ability to increase the size of this sample is dependent on its available  resources, and according to NHTSA, increasing the size of the sample to  such an extent would require a budget several times its current size. The  specific benefits of the larger sample are impossible to determine, leaving  the Congress with less information than would be desirable to help  determine the appropriate level of funding for this program. However,  should the Congress decide that it would be appropriate to enable  NHTSA and other users to conduct additional analyses of crashes that,  while they may occur rarely, can still result in significant loss of life and  economic cost, this report provides information on the potential for  different sample sizes to meet that need.
section_title: Agency Comments
paragraphs: We provided a draft of this report to the Department of Transportation for  review and comment. The Department of Transportation provided  technical comments, which we incorporated as appropriate.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of Transportation, and the Acting Chairman of  the National Transportation Safety Board. This report will also be  available at no charge on the GAO website http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-2834 or flemings@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix II.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: This report assesses (1) the process the National Highway Traffic Safety  Administration (NHTSA) used to redesign the National Automotive  Sampling System Crashworthiness Data System (NASS-CDS) and (2)  the potential for this redesign to improve the NASS-CDS sample. We  limited our scope to assessing NHTSA’s redesign of the NASS-CDS  component of the National Automotive Sampling System (NASS). As a  result, this report does not discuss the National Automotive Sampling  System General Estimates System (NASS-GES) or other NHTSA data  collection programs.
To assess the process NHTSA used to redesign NASS-CDS, we  reviewed pertinent documents related to the NASS redesign and  interviewed knowledgeable NHTSA officials from the National Center for  Statistics and Analysis—a NHTSA component that oversees the agency’s  data collection efforts, including NASS—and representatives of Westat,  the contractor selected to redesign the NASS-CDS sample. We also  interviewed 21 NASS users or other interested parties, including  automobile manufacturers, suppliers, safety advocates, members of the  medical community, and representatives from research organizations, to  understand how they use NASS-CDS and the improvements they would  like to see NHTSA make to NASS-CDS as part of the redesign. We  selected these 21 NASS users by first contacting those that submitted  comments to NHTSA on the redesign and then asking these initial  contacts who else we should interview. Specific NASS users we  interviewed or received comments from are listed in table 4.
The results of our discussions with NASS users are not generalizable to  all NASS users but provide insights into aspects of NASS-CDS that some  users indicated they would like to see improved. In addition, we visited  two of the geographic locations, called primary sampling units (PSU),  where NHTSA collects NASS-CDS data, to observe NHTSA’s crash  technicians conduct their work, and spoke with NHTSA crash technicians  at two others. The PSUs we visited were Seattle, Washington, and King  County, Washington; the PSUs we contacted were Allegheny County,  Pennsylvania, and Muskegon County, Michigan. We selected these  locations to ensure we included each type of PSU (i.e., urban, county, or  group of counties) and to ensure that we included at least one PSU from  each of the two contractors that NHTSA uses to implement the program.
The results of our discussions with PSUs are not generalizable to all  PSUs but provide insights into aspects of the work crash technicians do.  We assessed NHTSA’s efforts to redesign NASS based on government- wide standards and guidelines issued by the Office of Management and  Budget (OMB) that apply to the development and implementation of  statistical surveys such as NASS. OMB’s standards and guidelines  provide a framework for the development of survey concepts, methods,  and design; collecting data; processing data; producing estimates;  analyzing data; reviewing procedures; and disseminating the results.  These OMB documents also specify the professional principles and  practices that federal agencies should follow and the level of quality and  effort expected when initiating a new survey or redesigning an existing  survey such as NASS-CDS. Because NHTSA was in the process of  redesigning NASS at the time of our review, we focused our assessment  on reviewing NHTSA’s processes as they relate to the development of  survey concepts, methods, and design.
To assess the potential for the new sample design to improve NASS-CDS  data, increase precision of estimates, and increase the sample size, a  team that included GAO social science analysts with statistical survey  expertise reviewed the sampling methodology for the current NASS-CDS  sample and the design proposed for the new Crash Investigation  Sampling System (CISS) sample. As a part of this review, we analyzed  the proposed changes to the sample design, the number of PSUs  chosen, the overall sample size recommended, and NHTSA’s budgetary  constraints for the new sample. We compared the proposed redesign,  including the sample selection process and sample size, with literature on  efficient statistical sample design to assess the reasonableness of the  redesign. We also assessed the extent to which NHTSA’s proposed  design was responsive to user needs, according to what we learned from  our NASS user interviews. Finally, we interviewed NHTSA and Westat  officials on the new sample design.
We conducted this performance audit from April 2014 through March  2015 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments:
paragraphs: In addition to the contact named above, the following individuals made  important contributions to this report: Andrew Von Ah, Assistant Director;  James Ashley; Lorraine Ettaro; David Hooper; Wesley A. Johnson; Sarah  Jones; Joshua Ormond; and Amy Rosewarne.