Auto Safety: Status of NHTSA's Redesign of Its Crashworthiness Data System
Released: 2015-03-06T00:00:00

Letter

Background

NHTSA’s mission is to prevent motor vehicle crashes and reduce injuries, fatalities, and economic losses associated with these crashes. To carry out this mission, NHTSA conducts a range of safety-related activities, including setting vehicle safety standards; investigating possible safety defects and taking steps to help ensure that products meet safety standards and are not defective (through recalls if necessary); providing guidance and other assistance to states to help address traffic safety issues, such as drunk driving and distracted driving; and collecting and analyzing data on crashes. In fiscal year 2014, NHTSA’s enacted budget was $819 million.

NHTSA collects and analyzes crash data for a variety of purposes, such as to determine the extent of a safety problem and what steps NHTSA should take to develop countermeasures. NHTSA collects data through detailed, in-depth investigations as well as to generate national statistics and nationally representative data, as shown in table 1.

As mentioned previously, this report focuses on NASS-CDS. NHTSA collects NASS-CDS data through in-depth investigations of a sample of police-reported motor vehicle crashes that occur in the United States. The data collected through NASS-CDS are detailed and descriptive and allow NHTSA and others to assess the crashworthiness of different types of vehicles, evaluate different vehicle safety systems and designs, and understand the nature of injuries that people sustain during crashes. NHTSA uses the data collected as part of NASS-CDS for statistical analyses in its rulemaking and to estimate the size of the population that might be affected by its rulemaking. NHTSA also uses NASS-CDS data for other purposes, such as to identify existing and potential traffic-safety problems. For example, NHTSA has used NASS-CDS data to investigate patterns of roof intrusion into a vehicle resulting from real-world rollover crashes. According to NHTSA, NASS-CDS data showed that the damage and intrusion that occurred during real-world crashes was greater than the damage and intrusion that occurred during crash tests, pointing to the need to revisit NHTSA’s standards for the strength requirements for a vehicle’s roof. Others, including other federal agencies, universities, research institutions, and the automobile and insurance industries, use NASS-CDS data to understand the nature and consequences of real world crashes. For example, the National Transportation Safety Board, an independent federal agency, uses NASS-CDS data for research purposes as well as for conducting its accident investigations, whereas automobile manufacturers may use NASS-CDS data to study crash patterns and how those patterns have changed over time in order to prioritize their own research on vehicle designs and safety features.

NHTSA collects NASS-CDS data and information using stratified sampling—a statistical method of sampling in which a population is divided into two or more parts (called strata) and a sample is selected from each part (or stratum). NASS-CDS, specifically, is a stratified, three- stage probability sample, as illustrated in figure 1.

The first stage of the NASS-CDS sample was the selection of PSUs—the geographic locations where NHTSA collects data. NHTSA defined the PSUs so that their minimum population was approximately 50,000, and each PSU consisted of a central city, a county, a group of counties, or a portion of a large county excluding a central city. The PSUs were grouped into 12 strata based on geographic region (i.e., Northeast, South, Midwest, and West) and urbanization type (i.e., large central cities, large suburban areas, all others). The PSUs to be sampled were allocated to each stratum roughly proportional to the number of crashes in each stratum, and at least two PSUs were then selected from each stratum. As of 2014, a total of 24 PSUs comprised the NASS-CDS sample, as shown in figure 2.

The second stage was the selection of police jurisdictions within the sampled PSUs. About 170 police jurisdictions across the United States are part of the NASS-CDS sample and the number of jurisdictions per PSU varies (e.g., the Seattle, Washington PSU has two police jurisdictions in its sample, whereas the King County, Washington PSU has seven). Each police jurisdiction was assigned a “measure of size” that reflects the number, severity, and type of crashes in each jurisdiction. A sample of police jurisdictions was then selected from each sampled PSU, and those jurisdictions having a larger measure of size were oversampled.

The third and final stage is the ongoing selection of the actual police accident reports that are filled out by a police officer at the scene of a motor vehicle crash. Each week, the sampled police jurisdictions are contacted and all police accident reports that have accumulated since the previous week are reviewed and classified into a stratum based on the types of vehicles involved, most severe police-reported injury, disposition of the injured, tow status, and model year of the vehicles.

To be eligible for inclusion in NASS-CDS, a motor vehicle crash must (1) be police-reported, (2) involve a harmful event resulting from the crash (such as property damage or personal injury), and (3) involve at least one passenger car, light truck, van, or sport utility vehicle in transport on a traffic-way that was towed from the scene due to damage. The gross vehicle weight rating should be less than 10,000 pounds. Crashes are selected so that a larger percentage of higher severity crashes are selected than lower severity crashes, but every motor vehicle crash that occurs within one of the PSUs where NASS-CDS data are collected and that meets these conditions has a chance of being selected for investigation. NHTSA selected the first stage of the current NASS-CDS sample in 1988 and the second stage in 1995 but selects the police accident reports weekly so that the evidence from the motor vehicle crashes that might be investigated is still intact and the memory of the individuals involved is still fresh.

NHTSA contracts with two companies that use small teams of crash technicians located across the country to collect NASS-CDS data. These teams typically include a team leader, one or two crash technicians, and an assistant, and each team reports to one of two contractor-led control centers, called zone centers. NHTSA’s crash technicians collect over 600 data elements during their investigations, including information on the damage vehicles sustained, the crash forces involved, injuries to victims, and factors that caused those injuries. Those investigations generally involve inspecting the scene of a crash and the vehicles involved; interviewing the drivers and occupants involved, if possible; reviewing official medical reports detailing any injuries sustained; and reconstructing what happened during the crash, as shown in figure 3. The crash technicians coordinate with law enforcement agencies, hospitals, tow yard operators, repair garages, and the drivers and occupants involved in the crashes while performing their work, and the information they collect is subject to review by NHTSA. That information, in turn, is then made available to NHTSA and the public for research purposes.

The number of NASS-CDS investigations NHTSA conducts each year varies, as shown in figure 4. Between 1988 and 2013, NHTSA conducted an average of about 4,700 NASS-CDS investigations each year. However, since 2009, the number of NASS-CDS investigations conducted has steadily decreased, and in 2013, only about 3,400 NASS- CDS investigations were conducted. According to NHTSA, factors that have contributed to this decline include the budget for NASS-CDS and rising costs. For example, funding for NASS-CDS has been flat-lined since 2010, whereas costs—including costs for labor, information technology, leases and fuel—have risen.

NHTSA’s effort to redesign NASS-CDS is part of NHTSA’s larger Data Modernization Project, begun in 2012, which also affects NASS-GES and FARS. Specific to NASS-CDS, NHTSA’s Data Modernization Project involves the following: redesigning the NASS-CDS sample by reviewing the data elements that comprise the sample and the statistical methodology behind selecting the sample; upgrading the equipment and information technology that supports NASS-CDS to reduce redundancy, improve data quality, and enhance the experience of NASS users; and implementing a new sample to replace NASS-CDS.

NHTSA Followed a Reasonable Process for Redesigning NASS-CDS

NHTSA Redesigned NASS-CDS in Accordance with Applicable OMB Standards and Guidelines

We found that the process NHTSA followed to redesign NASS-CDS is consistent with applicable government-wide standards and guidelines issued by OMB that apply to the development of survey concepts, methods, and design. OMB’s standards and guidelines specify the professional principles and practices that federal agencies are required to adhere to and the level of quality and effort expected when initiating a new survey or redesigning an existing survey. In the case of redesigning NASS-CDS, the OMB standards and guidelines that apply include recommended practices for the development of survey concepts, methods and design. As such, they highlight the importance of consulting with potential users to identify their requirements and expectations, including design elements in a sample to meet stated objectives, and testing a survey’s components prior to full-scale implementation.

OMB’s standards and guidelines are not intended to substitute for the extensive existing literature on statistical and survey theory, methods, and operations. Further, these standards and guidelines specify that agencies should engage knowledgeable and experienced survey practitioners to effectively achieve the goals of OMB’s standards. While the process NHTSA has followed is consistent with applicable OMB standards and guidelines, NHTSA has not yet started implementing its new sample design. Accordingly, we were not able to assess its implementation efforts.

To redesign NASS-CDS, NHTSA awarded a contract to Westat in May 2012 to assist the agency in redesigning the NASS-CDS sample. Westat provides services relating to survey planning, design, development, and administration and analysis, and Westat researchers are known to be experts in the field of survey sampling. Westat’s tasks included reviewing the data elements NASS-CDS collects as well as the statistical methodology behind NASS-CDS. NHTSA, in conjunction with Westat, solicited comments from NASS users through the Federal Register and held a public listening session with NASS users—steps that are consistent with OMB’s recommended practice to consult with potential users to identify their requirements and expectations. Through the Federal Register, in June 2012, NHTSA solicited and subsequently received comments from 25 individuals and organizations regarding the redesign and their data needs. NHTSA also held a public listening session with NASS users in July 2013. During this listening session, NHTSA officials provided users with an update on its progress in redesigning NASS as well as an opportunity to provide additional comments, and eight NASS users provided comments at that listening session. According to NHTSA officials, the comments received indicated that users generally wanted NHTSA to increase the NASS-CDS sample size, collect additional data during NASS-CDS investigations, and improve the quality of the data collected. This includes collecting additional data from event data recorders and on the use of crash- avoidance technologies, as well as more detailed diagrams of the scenes of crashes.

Consistent with OMB’s recommended practice to include design elements to meet stated objectives, NHTSA tasked Westat with (1) identifying data elements that are responsive to the current and future needs of both NHTSA and the public and (2) developing recommendations for a new sample design that met users’ data needs in an effective and efficient manner while still maintaining national representativeness. As part of its review, Westat reviewed the comments NASS users submitted and also assembled a team of experts in crash investigation, transportation safety research, and injury control to review NASS’s data elements, identify areas of research that should be better addressed in the future, and make recommendations. Both NHTSA and Westat considered the feasibility of suggestions users made to fundamentally change how NASS-CDS data are collected. For example, some users commented that police officers who fill out accident reports at crash scenes could do more to assist NHTSA’s data collection efforts, such as by photographing the crash scene and the vehicles involved at the time of the crash. However, according to NHTSA officials, such suggestions were deemed not to be practical because they would require resources from NHTSA to provide equipment and training to law enforcement officials to implement. NHTSA officials also noted that police officers on the scene might not be willing to cooperate with additional data collection duties when they are responding to a crash, and police jurisdictions have varying technological capacities to handle the storage and dissemination of photos or other additional data.

Westat officials also told us they analyzed the NASS-CDS sample design to identify its limitations. This examination included reviewing the sample size, stratification and sampling allocation, and weighting procedures.

Westat and NHTSA considered various alternative design options for the new sample design, and NHTSA chose a probability-based approach to meet its objective of maintaining national representativeness. To assess whether Westat used the appropriate statistical survey design principles and methodology to ensure that its objectives would be met, NHTSA had early drafts of Westat’s work reviewed by three independent consultants. We have previously reported that such reviews can improve the technical quality of a project and enhance the credibility of the decision-making process, and as a result of these independent reviews, NHTSA officials said they felt confident moving forward with Westat’s proposals for the new sample design. Thus, in May 2014, NHTSA announced that it planned to replace NASS-CDS with a new system called the Crash Investigation Sampling System (CISS)—which we discuss in detail in objective 2.

Consistent with OMB’s recommended practice to test a survey’s components prior to full-scale implementation, NHTSA plans to implement the new CISS PSUs in phases. As of January 2015, NHTSA’s plans for the new sample call for initially implementing 24 new PSUs as a first phase and up to 73 PSUs in the future, if its budget allows. Prior to implementing all 24 new PSUs that comprise phase 1, NHTSA plans to first implement 5 of the PSUs, which, among other things, will allow NHTSA to test out the sample design and new equipment, such as electronic distance-measuring equipment that will support its data collection, among other things, prior to implementing the remaining PSUs. Figure 5 shows the location of the new CISS PSUs.

To pay for the Data Modernization Project, the Congress provided NHTSA with $25 million in 2011 and another $3.5 million through its fiscal year 2014 appropriation. Of this available funding, NHTSA officials said they allocated $2,500,000 (9 percent) to redesign the NASS-CDS and NASS-GES samples; $16,500,000 (58 percent) for information- technology infrastructure upgrades and new equipment, such as electronic distance-measuring equipment; and $9,500,000 (33 percent) to implement the new samples. Because NHTSA has not yet started implementing the new samples or obligated all of the funding for new equipment, about $12 million of the $28.5 million provided was still available as of the time of our review. Table 2 shows the funding Congress provided and NHTSA’s reported obligations, as of December 1, 2014.

However, as of the time of our review, NHTSA officials told us their time frames to begin implementing the new CISS PSUs were uncertain due to a government-wide cap on travel spending currently in place. Specifically, in 2012 OMB issued a memorandum, entitled Promoting Efficient Spending to Support Agency Operations, which directed agencies to spend at least 30 percent less on travel expenses than in fiscal year 2010 and to maintain that level of spending through fiscal year 2016.According to NHTSA, this cap on travel spending could delay its plans because implementing the new CISS PSUs requires that NHTSA staff travel to train the new crash technicians as well as to gain the cooperation of police jurisdictions, tow yards, and others. If no increases are provided, NHTSA stated that it would try to mitigate some of this limitation by, for example, training the new crash technicians at a local facility. NHTSA officials said they are currently working to obtain relief from this cap and hope to start implementing the new CISS PSUs beginning in 2015. According to NHTSA officials, failure to obtain relief from this cap could result in delays or additional costs in implementing the new PSUs.

NHTSA Was Late in Reporting on the Benefits of a Larger Sample

While we found NHTSA’s approach to redesigning NASS-CDS has been reasonable, we note that NHTSA was not timely in responding to Congress’ direction to provide information on the size of the NASS-CDS sample. Specifically, MAP-21 required NHTSA to conduct a comprehensive review of the data elements collected as part of NASS and report on whether there was a benefit to increasing the size of the NASS sample. For example, the act required NHTSA to provide Congress with information on the types of analyses that can be conducted and the conclusions that can be drawn under the current sample size and an expanded sample size, the number of investigations that NHTSA should conduct as part of the sample that would allow for optimal data analysis, NHTSA’s recommendations for improvements, and the resources necessary to implement NHTSA’s recommendations. The act also required that NHTSA obtain input from interested parties, including automobile manufacturers, safety advocates, the medical community, and research organizations. The act required NHTSA to report to Congress on the results of its review, including the benefits of a larger sample size, no later than October 1, 2013. NHTSA missed this deadline and issued its report in January 2015, as we were completing our review.

In its report, NHTSA stated that meeting the needs of all NASS users is a challenge and that there is no precise answer to what the optimal sample size for NASS-CDS would be. However, NHTSA also noted that increasing the size of the NASS-CDS sample would help meet the evolving needs of NASS-CDS users. We agree with NHTSA that there is no precise answer to what the optimal sample size for NASS-CDS is, and discuss this in more detail in objective 2.

The CISS May Not Substantially Increase the Sample Size but May Improve Precision and Data Quality

Although Its Sample Size May Not Substantially Increase, NHTSA Still Expects the CISS to Generate More Precise Crash and Injury Estimates

One means of determining the extent the Data Modernization Project redesign will improve the NASS-CDS sample is to assess the potential for CISS to meet a main technical objective of the Data Modernization Project: achieving similar or greater levels of statistical precision for seven important crash and injury estimates. Four of these measures are for crash types—rear-end crashes, head-on crashes, angle crashes, and rollovers and three are for injury-severity—incapacitating, non- incapacitating, and fatal.

The statistical precision of an estimate provides a measure of how close the estimate is expected to be to the population value it is attempting to describe. Improving the statistical precision allows for more accurate estimates and, in turn, informs the language NHTSA uses to make projections from the sample that apply to the whole population. Comparing the precision of the estimates NASS-CDS generates to the expected precision of the new CISS estimates is a method of determining whether the sample design has improved.

The precision of a sample’s estimates can be increased by selecting a larger sample, using a more efficient sample design, or both. When a more efficient sample design is used, it is possible to generate estimates with similar or greater levels of precision with a smaller sample size. While NASS users indicated that they wanted to see an increase in the size of the sample as part of the redesign, NHTSA officials stated that expanding the sample size would increase the cost of collecting the data for an extensive data collection effort like NASS-CDS, beyond expected budgetary resources, which we will discuss in more detail later in this report. Decisions made in the process of designing a sample must balance available resources and the ability of the sample to meet the stated objectives within the defined precision requirements.

NHTSA expects the new CISS design to achieve similar or greater levels of precision for NHTSA’s 7 key estimates by using a more efficient sample design, not by substantially increasing the sample size from the historical average for NASS-CDS. Westat developed several proposed sample designs and made design recommendations to NHTSA. NHTSA then modified Westat’s recommended design to produce similar or more precise results for the 7 key estimates using a sample of 24 PSUs, which would result in between 4,000 and 4,500 investigations annually. By way of comparison, in recent years, NASS-CDS has produced a sample size of about 3,500 investigations annually; however, between 1988 and 2013, has produced about 4,700 investigations annually. According to NHTSA, even though the expected sample size for CISS is comparable to the historical average for the NASS-CDS sample, the end result is that the new design that NHTSA is pursuing for CISS should be as precise if not more so than the current NASS-CDS design for the key estimates NHTSA indentified. Table 3 below summarizes the differences between NASS-CDS and CISS.

There are many ways to design a sample to generate more statistically precise estimates. One way NHTSA improved the expected statistical precision of estimates from CISS was by selecting new PSUs that better represent the current population and the number and types of crashes nationwide. According to statistical literature, in a statistical sample that uses the same PSUs for a number of years such as NASS-CDS, PSUs should be re-selected periodically in order to ensure that the sample reflects the total population the sample is attempting to describe. However, the current PSUs for NASS-CDS were selected in 1988. Since NHTSA selected the current CDS PSU sample based on population and crash counts from more than 30 years ago, the CDS PSU sample has gradually become less representative of the population and crashes in the United States, and as a result, CDS estimates have become less statistically precise. By reselecting new PSUs, CISS data are expected to better represent the population and the areas in which the highest number of crashes with serious injuries occur, according to NHTSA. Consequently, the selection of PSUs is expected to allow for more precise estimation of crashes involving serious injuries. The improvement in the representativeness of the selected PSUs contributes to the improvement in statistical precision without increasing the sample size, making the new sample design more statistically efficient.

Moreover, NHTSA expects the new sample design for CISS to contain more crashes with serious injuries and crashes involving newer vehicles than NASS-CDS currently contains, which also should make estimates of these crashes more statistically precise. For example, NHTSA designed CISS so that 10 percent of the police accident reports selected for CISS investigations will contain a newer vehicle and an incapacitating injury, up from 6.9 percent in NASS-CDS. The higher sampling rate for newer vehicles and serious injury crashes is expected to increase the number of these crashes selected. This step will improve the precision of estimates and address some users need to have more of these types of crashes in the sample. NASS users who provided NHTSA with comments about the NASS redesign indicated they wanted both of these changes.

NHTSA’s determination of the sample size was dependent upon available resources, and NHTSA emphasized this in its January 2015 report to Congress. According to both NHTSA and Westat officials, budget constraints were the key factor driving both the new sample design and the decision not to increase the sample size. The budget for NASS-CDS has remained at $12,500,000 per year since 2010, and NHTSA officials also told us that the future budget for CISS remains uncertain. Because of the budget constraints, Westat recommended a design for the new sample with the fewest number of PSUs, police jurisdictions, and police accident reports that would meet NHTSA’s precision requirements and that NHTSA could realistically afford given its budget. In addition, there are limitations to how many investigations NHTSA’s crash technicians can conduct. Specifically, according to NHTSA, crash technicians can only currently conduct about 3 investigations every 2 weeks, to ensure that their investigations are high quality and thorough. The design that NHTSA is pursuing is expected to cost about $13.5 million annually. According to NHTSA officials, the expected cost for CISS is higher than the current NASS budget. NHTSA can afford to implement 24 PSUs at this time, because, according to NHTSA officials, the amount appropriated for Highway Safety Research and Operations in fiscal year 2014 included a $5 million increase supporting the operating budget for crash data collection that can be used to supplement the NASS-CDS budget. NHTSA officials said that because this funding was added to NHTSA’s base budget, they expect the funding will be available in future years. However, according to NHTSA, this funding would need to keep pace with inflation to help offset expected increases in operational costs.

The New Design Will Allow NHTSA to Increase or Decrease the Sample Size If Future Budgets Change

Another improvement as a result of the Data Modernization Project is the flexibility of the new sample design. Although NHTSA did not pursue a larger sample compared to historic levels due to budget constraints, the new sample design will allow NHTSA to add or subtract PSUs or police jurisdictions to increase or decrease the sample size in the future if its budget changes. Adding PSUs to increase the sample size is more efficient than simply adding more investigations within the selected 24 PSUs. According to statistical literature, greater increases in precision are achieved by increasing the number of PSUs rather than the number of investigations that are conducted per PSU. Additionally, according to NHTSA, because of limits in the number of crashes involving serious injuries or crashes involving newer vehicles within a particular geographic area, adding an additional PSU provides a new pool of crashes to sample from. NHTSA built this flexibility into the sample design to address the uncertainty of the future budget and to allow for sample size expansion if future budgets allow. To build this flexibility into the sample design, NHTSA identified 73 PSUs, which represents NHTSA’s estimation of its preferred sample size for CISS, that it can bring online one at a time as resources become available. However, adding PSUs to increase precision and increase the sample size is more expensive than adding more investigations within the selected 24 PSUs, as described above.

The sample size of 24 PSUs can also be reduced if budgets are further constrained, but that could jeopardize the gains in statistical precision achieved with the new sample design. NHTSA developed projections to illustrate what size of a sample the agency could potentially implement given future CISS budgets. For example, according to NHTSA, if the budget for CISS was reduced to $11 million, even with a higher caseload than crash technicians currently conduct, the most investigations NHTSA could conduct annually is just under 2,600. If the budget for CISS was increased to $20 million, NHTSA could conduct over 5,000 investigations a year. The smaller sample for $11 million would be expected to produce less precise estimates than the larger sample for $20 million.

Even though NHTSA expects its new design to meet its precision requirements for the seven key crash and injury estimates it identified, NHTSA officials said the design may not meet precision requirements for other estimates or include a sufficient number of specific crash-types that occur infrequently (rare crash populations). According to NHTSA, the optimal sample size for CISS is impossible to determine without first defining which estimates should meet precision requirements or which rare crash populations are required to meet other analytic needs. For example, NHTSA officials said that 73 PSUs selecting about 15,000 investigations annually would be a reasonable sample size not only for attempting to meet precision requirements for additional estimates but also for obtaining estimates for rare populations. A rare population can be crashes such as a side impact crash involving a child, which despite resulting in the death or injury of about 6,500 children under age 15, only accounted for about 0.1 percent of all crashes and 0.5 percent of serious injury crashes in 2011. From this small of a percentage of the total crash population, CISS as designed with 24 PSUs can be expected to select around 4 crashes per year for investigation. Increasing the sample size to 73 PSUs and 15,000 investigations could increase the number of selected serious injury side impact crashes involving an injured child to about 20 per year, according to NHTSA analysis.

A sample of this size would allow NHTSA and external CISS users to better study these relatively rare crash populations and generalize their findings for these crashes to the population of all side impact crashes involving injured children. However, operating those 73 PSUs could cost at least three times the $13.5 million currently planned for CISS, or approximately $41 million annually, according to NHTSA officials. A smaller increase in sample size would also increase the number of selected side-impact crashes, but to a lesser extent. For example, according to NHTSA, a sample of about 7,500 investigations could increase the number of selected serious injury side-impact crashes involving an injured child to about 10 per year. According to NHTSA, this would require operating about 40 PSUs and cost about twice what is currently planned for CISS. This smaller increase would allow users to better study these populations but would require more time to accumulate enough cases to generalize their findings. However, NHTSA also noted that it does not like to combine more than 5 years of crash data to shed light on a problem that depends on the ever-changing crash environment, and that 5 years of data should produce between15 and 20 cases for even very rare crash populations (0.1 percent of serious injury crashes in a given year). Three users we interviewed estimated that a sample somewhere around 10,000 investigations per year would make the data considerably more useful to them.

While the larger sample size would allow NHTSA to produce more estimates that are precise and more investigations for rare populations, it is not possible to quantify the benefit of this increase in precision. It is also not possible to determine the sample size that would result in the highest value for society in terms of reducing the human life and economic costs of motor vehicle crashes because the causal link between the data collected and the potential benefits, if any, is not possible to establish.

NHTSA could implement a sample even larger than 73 PSUs if resources allowed, but the sample sizes required to produce estimates for certain sub-groups that some NASS users had requested, such as at the make and model level would be impractical. According to NHTSA officials, such an estimate would require a sample size that is both not possible to determine for all vehicle types and would exceed any reasonable expectation of resources available for CISS. There are many types of vehicles, some of which are more common (such as the Ford F-150 pick- up truck) than others (such as the Tesla Model S). For any make and model vehicle, only a small percentage are involved in a crash that would be eligible for inclusion in the CISS sample. Whether a sample is large enough to yield an adequate number of crash types involving a particular make and model vehicle depends in large part on the number of crashes involving that vehicle type. According to GAO analysis, similar to the above example of side impact crashes resulting in the death or injury of a child, in order to identify 20 side impact crashes resulting in death or injury involving a particular make and model vehicle, there would have to be about 6,500 side impact crashes resulting in death or injury nationwide involving that make and model vehicle in one year, which is highly unlikely. As the crash-type of interest approaches very small percentages of the total number of crashes, it becomes less probable that the sample will adequately capture these crashes. Since NHTSA only investigates several thousand crashes each year, the percentage chance of even one of these investigations being selected for a NASS-CDS investigation is very small. Officials noted that NHTSA’s Special Crash Investigations (SCI) program conducts investigations into issues that arise from specific agency special needs, and those investigations could include make- and model-level defects and other issues. NHTSA currently has three SCI teams that travel to investigate crashes according to agency priorities and recalls, separate from current NASS-CDS sampled investigations.

In addition to the ability to scale the sample size up or down, the new sample design also allows NHTSA to substitute a PSU, police jurisdiction, or police accident reports, according to NHTSA officials. A PSU or police jurisdiction can be replaced if there are cooperation or information sharing challenges. During implementation, substituting a PSU or police jurisdiction would be less challenging than after the CISS is fully implemented. Substituting a PSU after implementation would require hiring and training new crash technicians. A police accident report can be replaced if it is incomplete or cannot be thoroughly researched, but replacing a police accident report was purposefully made difficult to avoid the potential for bias from crash technicians who choose to replace a police accident report for their own reasons. For example, if a police accident report included a vehicle that was impossible to locate for inspection, the technician could allow the sampling algorithm to select a replacement police accident report.

Finally, the new design allows NHTSA to implement separate modules to study crashes involving large trucks, motorcyclists, bicyclists, and pedestrians—as NASS users had requested. Westat developed initial plans for each of these subsets as additional modules that could be conducted as separate studies utilizing the CISS sites.

Data Accuracy Is Also Expected to Improve Because of New Technology, but the New Technology Will Not Increase the Number of Investigations

As part of the Data Modernization Project, NHTSA also plans to equip its crash technicians with new technology to help improve the efficiency and accuracy of the data they collect. Improving the accuracy of NASS-CDS data with more electronic data collection methods was one aspect of NASS that users indicated they hoped NHTSA would address as part of the redesign. For example, in the comments NHTSA received in response to its Federal Register notice, NASS users indicated they wanted scalable diagrams of crash scenes. Currently, crash technicians collect NASS- CDS data using paper forms and have to enter the roadway to manually measure a crash scene using measuring wheels and tape measures. Afterward, they have to manually enter their measurements into a computer program, which creates an electronic image of the crash scene, which in turn is made available to NASS users. However, according to NHTSA, those drawings are not ideal when attempting to conduct detailed research of a crash scene because the diagrams provided are not scalable. The new equipment NHTSA plans to provide its crash technicians include tablet computers, which will allow crash technicians to electronically collect and transmit data remotely from the field; new accident reconstruction software, which will automatically create scalable diagrams of crash scenes; and new electronic distance measuring equipment, which is expected to improve the accuracy and efficiency of scene and vehicle inspections while also allowing crash technicians to take their scene measurements safely from the roadside. Figure 6 shows NHTSA crash technicians collecting crash scene data using tape measurements in the street and using new technology from the side of the road.

While NHTSA expects new equipment will help its crash technicians collect more accurate data, it does not expect the new equipment will considerably reduce the time it takes to conduct an investigation or allow its crash technicians to conduct more investigations. NHTSA officials said the new equipment they plan to provide should help reduce the time it currently takes to conduct scene and vehicle inspections. However, this represents only a portion of the time NHTSA’s crash technicians spend each week performing scene and vehicle inspections. Further, as part of the NASS redesign NHTSA is also increasing the amount of information that its crash technicians collect, which, in turn, will require more time to collect. This includes data on the use of crash-avoidance technologies in newer vehicles as well as additional data for older vehicles, as some users had requested. Thus, the new equipment will not substantially decrease the amount of time technicians spend overall collecting data for crash investigations or the cost of collecting this data.

Collecting data this detailed is expensive and time consuming. For example, according to a study NHTSA conducted in 2012, an average NASS-CDS investigation takes about 25 hours to perform, and NHTSA’s crash technicians spend, on average, about 10 percent of that time inspecting crash scenes and about 25 percent of their time inspecting vehicles. In contrast, NHTSA’s crash technicians spend 13 percent of their time sampling police accident reports for investigations. Figure 7 below provides information on the percentage of hours per week on average NHTSA’s crash technicians spend performing various aspects of NASS-CDS investigations. Because NHTSA has not yet started collecting data for CISS, it is not possible to determine the number of hours per week crash technicians will spend on various aspects of the sampling and data collection for CISS crash investigations.

Concluding Observations

NASS-CDS provides NHTSA and others with an important source of data to understand the real-world nature and consequences of motor-vehicle traffic crashes. In redesigning NASS-CDS, NHTSA has followed a process that is consistent with applicable government-wide standards and guidance for redesigning statistical surveys. NHTSA has also taken steps to improve upon the original design for NASS-CDS in developing CISS— the system that will replace NASS-CDS—such as by making the sample more precise as well as by making the sample design more flexible to adapt to future budgets. While the proposed sample size will be sufficient to meet NHTSA’s requirements for the program, NHTSA does not plan to substantially increase the sample size. By increasing the size of the CISS sample, NHTSA and others could likely do more to study motor-vehicle traffic crashes in an effort to save lives and reduce the economic costs of crashes. Sampling sufficient cases to conduct analyses of rare populations requires a significantly larger sample. However, NHTSA’s ability to increase the size of this sample is dependent on its available resources, and according to NHTSA, increasing the size of the sample to such an extent would require a budget several times its current size. The specific benefits of the larger sample are impossible to determine, leaving the Congress with less information than would be desirable to help determine the appropriate level of funding for this program. However, should the Congress decide that it would be appropriate to enable NHTSA and other users to conduct additional analyses of crashes that, while they may occur rarely, can still result in significant loss of life and economic cost, this report provides information on the potential for different sample sizes to meet that need.

Agency Comments

We provided a draft of this report to the Department of Transportation for review and comment. The Department of Transportation provided technical comments, which we incorporated as appropriate.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Transportation, and the Acting Chairman of the National Transportation Safety Board. This report will also be available at no charge on the GAO website http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-2834 or flemings@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II.

Appendix I: Objectives, Scope, and Methodology

This report assesses (1) the process the National Highway Traffic Safety Administration (NHTSA) used to redesign the National Automotive Sampling System Crashworthiness Data System (NASS-CDS) and (2) the potential for this redesign to improve the NASS-CDS sample. We limited our scope to assessing NHTSA’s redesign of the NASS-CDS component of the National Automotive Sampling System (NASS). As a result, this report does not discuss the National Automotive Sampling System General Estimates System (NASS-GES) or other NHTSA data collection programs.

To assess the process NHTSA used to redesign NASS-CDS, we reviewed pertinent documents related to the NASS redesign and interviewed knowledgeable NHTSA officials from the National Center for Statistics and Analysis—a NHTSA component that oversees the agency’s data collection efforts, including NASS—and representatives of Westat, the contractor selected to redesign the NASS-CDS sample. We also interviewed 21 NASS users or other interested parties, including automobile manufacturers, suppliers, safety advocates, members of the medical community, and representatives from research organizations, to understand how they use NASS-CDS and the improvements they would like to see NHTSA make to NASS-CDS as part of the redesign. We selected these 21 NASS users by first contacting those that submitted comments to NHTSA on the redesign and then asking these initial contacts who else we should interview. Specific NASS users we interviewed or received comments from are listed in table 4.

The results of our discussions with NASS users are not generalizable to all NASS users but provide insights into aspects of NASS-CDS that some users indicated they would like to see improved. In addition, we visited two of the geographic locations, called primary sampling units (PSU), where NHTSA collects NASS-CDS data, to observe NHTSA’s crash technicians conduct their work, and spoke with NHTSA crash technicians at two others. The PSUs we visited were Seattle, Washington, and King County, Washington; the PSUs we contacted were Allegheny County, Pennsylvania, and Muskegon County, Michigan. We selected these locations to ensure we included each type of PSU (i.e., urban, county, or group of counties) and to ensure that we included at least one PSU from each of the two contractors that NHTSA uses to implement the program.

The results of our discussions with PSUs are not generalizable to all PSUs but provide insights into aspects of the work crash technicians do. We assessed NHTSA’s efforts to redesign NASS based on government- wide standards and guidelines issued by the Office of Management and Budget (OMB) that apply to the development and implementation of statistical surveys such as NASS. OMB’s standards and guidelines provide a framework for the development of survey concepts, methods, and design; collecting data; processing data; producing estimates; analyzing data; reviewing procedures; and disseminating the results. These OMB documents also specify the professional principles and practices that federal agencies should follow and the level of quality and effort expected when initiating a new survey or redesigning an existing survey such as NASS-CDS. Because NHTSA was in the process of redesigning NASS at the time of our review, we focused our assessment on reviewing NHTSA’s processes as they relate to the development of survey concepts, methods, and design.

To assess the potential for the new sample design to improve NASS-CDS data, increase precision of estimates, and increase the sample size, a team that included GAO social science analysts with statistical survey expertise reviewed the sampling methodology for the current NASS-CDS sample and the design proposed for the new Crash Investigation Sampling System (CISS) sample. As a part of this review, we analyzed the proposed changes to the sample design, the number of PSUs chosen, the overall sample size recommended, and NHTSA’s budgetary constraints for the new sample. We compared the proposed redesign, including the sample selection process and sample size, with literature on efficient statistical sample design to assess the reasonableness of the redesign. We also assessed the extent to which NHTSA’s proposed design was responsive to user needs, according to what we learned from our NASS user interviews. Finally, we interviewed NHTSA and Westat officials on the new sample design.

We conducted this performance audit from April 2014 through March 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: GAO Contacts and Staff Acknowledgments

GAO Contact

Staff Acknowledgments:

In addition to the contact named above, the following individuals made important contributions to this report: Andrew Von Ah, Assistant Director; James Ashley; Lorraine Ettaro; David Hooper; Wesley A. Johnson; Sarah Jones; Joshua Ormond; and Amy Rosewarne.