Auto Recalls: NHTSA Should Take Steps to Further Improve the Usability of Its Website
Released: 2017-12-04T00:00:00

Letter

Background

Safety defect vehicle recalls (auto recalls) are initiated when a defect in a vehicle or vehicle equipment creates an unreasonable safety risk, as determined by NHTSA or a manufacturer. After a recall is initiated, manufacturers are required to provide written notification to vehicle owners via First-Class Mail within 60 days and remedy the defect. Franchised dealers—which sell or lease an auto manufacturer’s new vehicles—perform the recall remedy.

Before manufacturers send recall notification letters to affected vehicle owners, NHTSA reviews draft letters and envelopes to ensure they include required information about the safety defect. Required information includes, among other things, a clear description of the safety defect, an evaluation of the risk to vehicle safety, and a statement that the manufacturer will remedy the defect without charge. See appendix II for an example of a notification letter.

The number of vehicles affected by safety defect vehicle recalls has increased dramatically since 2011 (see fig. 1). The increase reflects, in part, several large-scale recalls. For example, in 2014, General Motors initiated a recall of over 8 million vehicles with faulty ignition switches. Similarly, according to NHTSA in 2014 and 2015, some passenger vehicle manufacturers began recalling Takata air bag inflators, recalls that have grown to include approximately 34 million vehicles and 19 auto manufacturers. For the Takata recall, NHTSA issued various orders and established a Coordinated Remedy Program under which the agency oversees the supply of remedy parts and risk-based prioritization of vehicles for repair, and manages related recalls with the assistance of an Independent Monitor. The Independent Monitor assesses compliance with the applicable orders issued by NHTSA and makes recommendations aimed at enhancing the remedy program.

According to NHTSA’s Strategic Plan 2016–2020, this unprecedented recall activity encouraged the agency to improve its system for identifying and addressing defective vehicles. For example, the plan states that NHTSA’s “vision is to achieve a 100-percent completion rate for every recall by improving communication at every level, at every step of the way.” Thus, according to the plan, NHTSA and the auto industry have committed to identifying and implementing effective strategies to inform consumers of safety defects and envision that their coordination will bolster recall efforts to improve completion rates.

NHTSA reported that annual completion rates for passenger vehicle recalls have remained relatively flat, ranging from 63 to 67 percent between calendar year 2011 and calendar year 2014. See appendix III for completion rates by vehicle component and vehicle type. In part, to improve communication and encourage consumers to complete repairs, NHTSA and manufacturers provide auto recall information to the public on their websites. For example, certain motor vehicle manufacturers are required to allow consumers to search a vehicle’s recall remedy status on the Internet using the vehicle identification number (VIN). NHTSA also provides publicly available auto recall information on its website, including examples of recall notification letters.

In December 2016, NHTSA began consolidating its websites into NHTSA.gov to provide a single access point for its auto recall content. One of these websites, safercar.gov, was once NHTSA’s primary method of communicating auto recall information to consumers; however, the agency is in the process of moving this information to NHTSA.gov. NHTSA’s Strategic Plan 2016–2020 states that the agency wants NHTSA.gov to be a comprehensive user-friendly platform that serves as the premier source of vehicle safety information by, for example, improving the website’s search capabilities. NHTSA also aims to encourage consumers to use its website’s auto recall information through its communications program.

NHTSA’s Office of Communications and Consumer Information (OCCI) is the primary office responsible for implementing the agency’s public communication efforts. OCCI intends to increase public engagement with the agency’s information through its social media channels, such as Instagram, Twitter, and Facebook. The amount OCCI obligated to support the agency’s auto recall efforts has increased from nearly $.5 million in fiscal year 2011 to about $2.5 million in fiscal year 2016. According to NHTSA officials, these obligations supported various efforts, including public awareness campaigns, an auto recall hotline, advertising agencies, exhibits at auto shows, and NHTSA’s mobile application.

Auto Recall Information Use Varies, and Most Consumers in Focus Groups Preferred Electronic Recall Notifications in Addition to Mail

Consumers in Our Focus Groups Primarily Considered Safety Risk and Convenience when Using Auto Recall Information to Make Repair Decisions

As part of our focus group discussion sessions, consumers selected safety risk and convenience as the two most influential factors they considered when using auto recall information to decide whether to complete repairs.

All factors considered: During each session, we first asked consumers to describe all the factors they considered. Across the sessions, consumers shared a wide variety of factors including availability of a loaner vehicle, time to schedule and complete the repair, safety risk, and other factors. For example, some consumers had not yet repaired their vehicles because they were “just waiting” for parts to become available. Other consumers considered their previous customer service experiences at the franchised dealership or the distance they would need to travel to complete the repair. For example, one consumer at our rural focus group location told us it would take roughly 2 hours to reach the dealership’s repair shop.

Most influential factors considered: After the discussion of all factors, we then asked each consumer to select the single most influential factor they considered. Consumers in the sessions overwhelmingly selected safety risk and convenience as the two most influential factors (see table 1).

More than half of consumers in our focus group discussion sessions selected safety risk as the most influential factor they considered when making repair decisions. They told us that their perception of the risk influenced whether or not they repaired their vehicle. For instance, some consumers stated that they completed repairs immediately, because the risks “sounded serious” or that they considered the defect a “safety concern.” Conversely, some consumers said they did not complete the repairs because the defect “didn’t sound very urgent.”

While each recall notification letter is required to include an evaluation of the risk to vehicle safety reasonably related to the defect, consumers in our focus group sessions shared mixed opinions about the quality and clarity of safety risk information included in the notification letter they received. For example, some consumers told us the letter’s safety risk information seemed vague. For instance, one consumer told us the letter’s description of the safety defect did not clearly state the chances of an increased risk of injury and so he “had to figure out on his own.” In addition, some consumers commented that the safety risk information could be more prominent in the notification letter, that the letter could emphasize the severity of the risk, or that the letter could describe the risk in simpler language. However, other consumers stated the notification letter they received adequately described the recall’s safety risk. In June 2011, we recommended that NHTSA modify the requirements for defect notification letters to include additional information to obtain readers’ attention. In 2013, NHTSA responded to our recommendation by requiring manufacturers to include the statement “IMPORTANT SAFETY RECALL” at the top of auto recall notification letters.

Focus Group Participant’s Comment “I don't want to be without a car for half the day or stay with my kids all day.”

Consumers in our focus group discussion sessions selected convenience as the second most influential factor they considered in making repair decisions. While some consumers described the “hassle” of the repair and being “too busy” to schedule and fix the defect, other consumers told us they repaired their vehicles more easily because, for example, they could take advantage of previously scheduled service appointments to also repair the defect. Also, some consumers in our sessions stated that the letter or notification they received could better address the inconvenience of the recall, for example by including better estimates of how long repairs might take. In addition, some consumers recommended the letter include options for scheduling needed repairs.

As we discuss later in the report, NHTSA officials told us they continue to work with auto manufacturers to identify ways to encourage consumers to complete needed repairs, while representatives from some manufacturers we met with described specific steps they have taken to address some of the inconveniences consumers may experience in completing repairs. For example, one manufacturer facilitated a pilot program for a third-party service provider in conjunction with dealers to repair vehicles at the owner’s home or place of work, while another manufacturer told us they work with individual dealers to hold events specifically for recall repairs when consumers can come in to have repairs performed after normal business hours.

Industry Stakeholders’ Use of Publicly Available Auto Recall Information Varies

Industry stakeholders’ use of auto recall information varies because these stakeholders play different roles in the auto recall process. Auto manufacturers are primarily responsible for providing auto recall information to the public and others, including NHTSA and auto dealers. Franchised dealers are responsible for performing the recall remedy for manufacturers and therefore use manufacturer-provided information for that purpose. Specifically, all of the franchised dealers we interviewed told us they identify recalls on new vehicles in their inventory primarily by accessing auto recall information through internal manufacturer databases. These franchised dealers may also use information from third-party providers or publicly available auto recall information on NHTSA’s website to identify recalls affecting used vehicles.

Independent dealers—which are not generally authorized by manufacturers to perform recall remedies—may use publicly available auto recall information to identify open recalls. Specifically, 2 of the 3 independent dealers we met with told us they use NHTSA’s VIN look-up tool to search for open recalls affecting vehicles in their inventory before selling them to consumers. However, these dealers told us that the current design of the tool takes too much time to use, because it requires users to search each VIN individually. For example, one dealer told us each search took about 15 seconds to perform, resulting in significant time and cost because the dealership has tens of thousands of vehicles in its inventory. These dealers told us being able to search multiple VINs in a single search (i.e., VIN-batch search) could save them time or money.

Representatives from the Alliance of Automobile Manufacturers stated they—in coordination with other industry stakeholders—are working with a third-party provider to develop a search tool that would address this concern by enabling VIN-batch searches for use by government agencies, such as state departments of motor vehicles, and commercial entities. The group anticipates the tool will be available in the first half of 2018.

Most Consumers in Our Focus Groups Prefer to Receive Recall Notification by Electronic Means in Addition to Mail

Although the vast majority of consumers who participated in our focus group discussion sessions reported a preference to receive auto recall notification by mail, most preferred to receive notifications by at least one additional electronic means such as e-mail, phone calls, and text messages. Eighty of the 94 consumers in our sessions reported a preference for receiving notification by mail, and all but 4 reported actually receiving mailed notification (see fig. 2). However, 69 of the 94 consumers in our sessions also reported a preference for receiving recall notification by electronic means, but only 7 reported actually receiving at least one type of electronic notification. This result suggests a gap between industry recall notification practices and notification preferences for most consumers in our focus groups, especially for younger consumers who were more likely to report a preference for notification by electronic means. For complete results of the questionnaire we administered to consumers for the discussion session, see appendix IV. As we discuss later in this report, in September 2016, NHTSA issued a Notice of Proposed Rulemaking (NPRM) that proposes to require auto manufacturers to notify consumers about auto recalls by electronic means in addition to First-Class Mail. NHTSA officials told us the agency is working with the administration on NHTSA’s regulatory portfolio and priorities, including this rulemaking.

Some manufacturers told us they use additional methods to reach consumers, including notifying consumers by electronic means and translating recall information into Spanish. For example, representatives from one manufacturer told us they always notify consumers by e-mail before sending out the required First-Class Mail letter notification. These representatives told us using multiple recall notification means resulted in higher recall completion rates. In addition, eight of the remaining nine manufacturers told us they use supplemental electronic means notification on a case-by-case basis—generally using additional means to improve recall completion rates—while four manufacturers stated they consider safety risk severity when deciding when or how to use additional notification means for individual recalls. Also, representatives from 3 of the 10 manufacturers we spoke with told us they translate the entire mailed notification letter into Spanish.

Most Consumers in Focus Groups Found the Auto Recall Areas of NHTSA.gov Generally Easy to Use, but Some Experienced Difficulties

Usability Testing with Consumers Found the Auto Recall Areas of NHTSA’s Website Generally Easy to Use

In late 2016, NHTSA launched its redesigned NHTSA.gov website, including the auto recall areas consumers assessed during our testing sessions. According to responses to a questionnaire we administered during our testing sessions, 78 of the 94 consumers found the auto recall areas of NHTSA.gov either “somewhat” or “very easy” to use (see fig. 3). See appendix V for complete participant responses to the questionnaire we administered to each consumer.

To inform the development of the redesigned website, NHTSA worked with a contractor to conduct a usability study in 2015 to evaluate users’ reactions to the agency’s websites, including NHTSA.gov. According to agency officials, NHTSA implemented several changes based on the findings from the usability study, including: the creation of a dedicated “recalls” area of NHTSA.gov, and the ability for users to access the VIN look-up tool in three different ways—on the homepage, in the “recalls” area, and through direct links either in a NHTSA e-mail for subscribers or from an external website.

In addition, NHTSA officials told us that Department of Transportation (DOT) and NHTSA staff meet as needed to discuss the website and consider improvements. For example, the officials said they monitor user searches for the relevance and accuracy of results and adjust the search software to better assist users in finding auto recall information. Officials also told us the agency collects a variety of other information about how visitors use NHTSA.gov, including how visitors access the website, and makes adjustments accordingly. For instance, NHTSA incorporated responsive web design as part of the agency’s ongoing consolidation effort—meaning the site is optimized for viewing on desktop, tablet, and mobile devices. In addition to monitoring searches and how visitors access NHTSA.gov, NHTSA officials told us they collect and consider online survey data to make website improvements and use web-analytic software to monitor, for example, where visitors choose to exit the website. Officials stated that such monitoring activities have allowed NHTSA to identify and correct problems with NHTSA.gov.

We did not directly evaluate the accessibility of the auto recall areas of NHTSA.gov to ensure the ability of people with physical or mental disabilities to use the website. However, NHTSA officials provided us with an overview of several steps the agency takes to ensure NHTSA.gov complies with website accessibility requirements. For example, according to officials, NHTSA subscribes to a service that provides monthly accessibility scans of the agency’s websites.

Consumers in Our Focus Groups Identified Opportunities to Improve the Usability of Certain Auto Recall Tasks on NHTSA’s Website

While most consumers in our usability testing sessions generally found the auto recall areas of NHTSA’s website easy to use, some consumers experienced difficulties completing tasks we asked them to perform (see table 2). Specifically, during each testing session we asked participants to perform tasks using the primary means NHTSA.gov provides for consumers to access information about auto recalls affecting their vehicles: searching for auto recalls using their vehicle’s VIN; searching for auto recalls using their vehicle’s year, make, and model; and locating NHTSA’s auto recall notification e-mail subscription service.

In addition, an evaluation we requested to corroborate the results of our consumer usability testing, identified similar issues. As discussed below, consumers experienced these difficulties because the auto recall areas of NHTSA.gov do not always reflect federal and industry key website usability practices, which describe standards and guidelines for making websites easy to use. Following such practices can assist agencies in creating quality websites while providing the flexibility necessary to meet organizational needs. Website usability is particularly important for agencies, such as NHTSA, that are responsible for conveying safety information to the public. Federal standards for internal control state that agencies should communicate quality information externally and select appropriate methods for communicating with the public.

While most consumers in our usability testing sessions found searching for recalls by VIN somewhat or very easy, some consumers found the search results did not provide the information they were seeking. When we asked consumers to perform VIN searches, they generally found the VIN look-up tool easy to use—88 of 94 consumers found searching with a VIN either somewhat or very easy. But some consumers experienced difficulties performing this task. Specifically, some consumers who had had their vehicles repaired expected to find the completed recall on the search results page. However, they were confused because the page is designed to display only open (i.e., unrepaired) recalls, not completed (i.e., repaired) recalls—leading these consumers to question the accuracy of the results. In addition, the evaluation conducted by website usability professionals found that, when an error occurred during a VIN search, the error message was too difficult to locate on the search results page. The evaluation recommended the error message have greater weight and more prominence on the page.

Federal key website usability practices state that agencies should ensure that results of user searches provide the precise information being sought, and in a format matching users’ expectations. When users are confused by search results, or do not immediately find what they are searching for, they become frustrated and may abandon the search or the website entirely. Since NHTSA launched the VIN look-up tool in August 2014, the number of VIN searches performed has increased (see fig. 4). According to NHTSA officials, major increases occurred in mid-2015— when the Takata air bag inflator recalls were announced—and in early 2017, when NHTSA made the VIN look-up tool search function available on NHTSA.gov and displayed it prominently on the website. Ensuring the usability of NHTSA’s VIN look-up tool is particularly important because it is the only way on NHTSA.gov for a consumer to determine whether their specific vehicle has an open safety recall.

Recall Search Using Vehicle Year, Make, and Model Some consumers’ vehicle year, make, and model searches were hampered by the information required to conduct an accurate search, as the content on the website is not always in plain language. We asked consumers to perform a recall search using their vehicles’ year, make, and model, and 78 of 94 consumers found the task to be either somewhat or very easy. However, some consumers found that they did not know enough information about their specific vehicles to feel confident that they were searching for the correct vehicle. For example, a year, make, and model search for a 2009 Toyota Tacoma may ask the consumer to choose among vehicle options, including “2009 TOYOTA TACOMA REGULAR CAB W/SAB RWD/AWD.” Acronyms such as “W/SAB”— which stands for “with side air bags”—may be confusing to consumers.

Federal key website usability practices state that federal agencies should write website content using plain language, so website visitors can easily find and use what they need.

Focus Group Participant’s Comment “I think [the Recall Notification E-mail System Sign-Up is] poorly placed. I had to scroll to find it. I had to search for it. You want at the top .”

Recall Notification E-mail System Sign-Up Some consumers suggested improvements to make the Recall Notification E-mail System Sign-Up easier to locate on the homepage. NHTSA first made its Recall Notification E-mail System Sign-Up available in March 2008. Of the 94 consumers in our testing sessions, 66 found it either “somewhat” or “very easy” to find the Recall Notification E-mail System Sign-Up—making this the least easy of the three tasks we asked consumers to perform. Specifically, several consumers said the Recall Notification E-mail System Sign-Up should include a clearer description, be easier to find, and be located at the top of the homepage (see fig. 5). These improvements are particularly important because some consumers in our focus group sessions told us that the ability to sign up for auto recall e-mail notifications was the most useful part of the auto recall areas of NHTSA.gov.

The website evaluation conducted by website usability professionals recommended that NHTSA streamline its homepage with more of a focus on primary website tasks. The evaluation also found that users must move through too many pages to sign up for recall e-mails.

Federal key website design and usability practices state that agencies should put important items closer to the top of the page, where users can better locate the information. Key practices also state that agencies should design their websites so users can successfully complete the most common tasks in the fewest number of steps.

The website usability difficulties that consumers in our focus groups experienced may be due to the fact that NHTSA has not studied the website’s usability since the agency redesigned NHTSA.gov in late 2016 and, therefore, may have been unaware of these difficulties prior to our review. NHTSA plans to conduct a website usability study with consumers after the consolidation effort, discussed above, is complete. However, NHTSA could not provide a general time frame for conducting the study because it has not yet determined when the consolidation effort will be complete. We have previously reported that it is essential for organizations to effectively guide their information technology efforts by establishing timelines to complete them, among other strategic planning best practices. Without establishing a completion date for its website consolidation effort, the website usability difficulties we identified may persist and limit the effectiveness of NHTSA’s primary means of providing consumers with safety recall information about their vehicles on NHTSA.gov.

NHTSA Has Initiated Activities to Raise Consumer Awareness about Recalls, but It Is Too Early to Evaluate the Agency’s Efforts Public Awareness Campaign

In January 2016, NHTSA launched a national advertising campaign encouraging consumers to check for open recalls using the agency’s VIN and year, make, and model look-up tools. Through March 2017, NHTSA spent about $1 million on its Safe Cars Save Lives campaign, which sponsors advertisements on Google, Facebook, and other media platforms. For example, Google might place NHTSA’s advertisement above other search results, when a consumer typed certain keywords— such as “recall,” “airbag recalls,” or “safercar.gov”—into the search.

NHTSA evaluated the campaign’s effectiveness by monitoring website traffic performance reports to determine how frequently consumers clicked on NHTSA-sponsored advertisements and ultimately searched for open recalls using the agency’s look-up tools. NHTSA also compared results across media platforms and adjusted the campaign’s strategy to improve performance. For example, NHTSA optimized advertisements on mobile devices, since mobile-device users performed more recall searches than other users. According to NHTSA data, the awareness campaign resulted in consumers performing 1.1 million recall searches through March 2017—a cost of about $0.90 per search. Agency data indicate that this cost generally decreased as NHTSA improved the campaign’s strategy. Agency officials told us NHTSA plans to spend another approximately $1.8 million on Safe Cars Save Lives from September 2017 through September 2018 due to the campaign’s effectiveness in raising the public’s awareness about auto recalls.

Pilot Program with States

NHTSA began implementing a mandated 2-year pilot grant program intended to evaluate the feasibility and effectiveness of informing consumers about open auto recalls during state vehicle registration. In September 2016, NHTSA solicited applications to participate in the program, wherein selected states would inform consumers—at no charge—about open recalls using all means that permit consumers to register vehicles within the state (e.g., in person, Internet, and mail).

According to NHTSA, only one state applied for the grant. In September 2017, NHTSA awarded the sole applicant $223,000.

Under the program, the grantee needs to collect and report program performance data, including the extent to which open recalls have been identified and repaired. In addition, the grantee must report whether certain notification means were more effective than others and what could be done to improve the program. Upon completion of the pilot program, NHTSA is required to evaluate the extent to which open recalls identified have been remedied. Auto manufacturers we met with were generally supportive of the program. Specifically, representatives from 9 of the 10 manufacturers told us that notifying consumers about open recalls during vehicle registration can raise consumer awareness or improve recall completion rates.

Proposed Rulemaking

In September 2016, NHTSA issued a Notice of Proposed Rulemaking (NPRM), which proposes to require auto manufacturers to notify consumers about open recalls by electronic means—such as e-mails, phone calls, and text messages, in addition to First-Class Mail. As we described earlier, auto manufacturers are currently required to notify consumers about safety recalls affecting their vehicles via First-Class Mail. According to NHTSA, the NPRM aims to aid in efficiently and effectively improving recall completion rates, by proposing that manufacturers provide notification using electronic means in addition to First-Class Mail.

Consumers in our focus groups as well as auto manufacturers and consumer associations we interviewed generally supported additional notification using electronic means.

Consumers in our focus groups: As we discussed earlier, 69 of the consumers in our focus group discussion sessions reported they would prefer to receive additional notification by at least one type of electronic means. However, only 7 consumers actually received such notifications—suggesting a gap between industry notification practices and notification preferences for these consumers.

Auto manufacturers: Representatives from 9 of 10 manufacturers we interviewed told us they generally support providing notification using electronic means. Although the NPRM proposes a broad definition of electronic means to give manufacturers flexibility to determine the most effective means, these representatives also shared implementation concerns. For example, representatives from 1 of the 9 manufacturers told us that—although the company collects e- mail addresses from some customers for other purposes—not all customers provide e-mail addresses, and those collected are not always accurate. As we discussed previously, most manufacturers we met with currently use supplemental electronic means notification on a case-by-case basis.

Consumer associations: Similarly, both consumer associations we interviewed told us additional electronic notification can help reach consumers who do not complete repairs after receiving initial mailed notification.

NHTSA’s proposal would maintain manufacturer reporting requirements, though it may result in additional reporting. This additional information could help the agency evaluate the effectiveness of various means of consumer notification. We previously found that NHTSA may be able to use manufacturers’ data to identify what factors make some recalls more or less successful than others. We recommended that NHTSA use the recall data it collects to analyze particular patterns or trends that may characterize successful recalls and determine whether these factors represent best practices.

If the NPRM is finalized, manufacturers would provide NHTSA with representative copies of the newly required electronic notifications, in addition to mailed notifications, and would specify the electronic means used, such as e-mail or text message. According to NHTSA officials, this information could allow the agency to track and evaluate the effectiveness of various notification means used by manufacturers by, for example, comparing completion rates across means—a key step in identifying best practices that could encourage consumers to complete repairs. However, it is too early for NHTSA to conduct such an evaluation, because the agency has not issued a final rule. NHTSA officials told us the agency is working with the administration on NHTSA’s regulatory portfolio and priorities, including this rulemaking.

Collaboration with Stakeholders

NHTSA has also taken steps to collaborate with industry stakeholders and explore consumer education best practices. For example, in April 2015 NHTSA hosted a day-long workshop that brought together auto industry stakeholders to examine public education of the recall process. During the workshop, participants identified current barriers to the public’s awareness of auto recalls and discussed potential solutions to address them, such as using text messages and social media to communicate with younger consumers and using different delivery methods for recall notices. Similarly, in January 2016 NHTSA and 18 auto manufacturers adopted a set of Proactive Safety Principles to explore and employ new ways to increase safety recall participation rates. For example, NHTSA and auto manufacturers agreed to share industry best practices and policies based on lessons learned from ongoing safety recalls.

The Independent Monitor of Takata in conjunction with NHTSA has also issued a set of coordinated communications recommendations based on consumer research, best practices observed during the Takata recall, and discussions with manufacturers. For example, the recommendations encourage manufacturers to: pursue a “multi-touch” communications strategy that employs non- traditional means, such as e-mail and text messages; convey risk in clear, accurate and urgent terms; and include a clear “call to action” designed to facilitate prompt and efficient scheduling of repairs.

According to NHTSA officials, the agency relies on auto manufacturers to evaluate the effectiveness of these efforts. However, agency officials told us NHTSA reviews manufacturers communication plans as part of the Takata recall’s Coordinated Remedy Program and provides ongoing recommendations on manufacturers’ communication language, approach, and strategies.

Conclusions

With the recent steep increase in safety defect vehicle recalls and continued low recall completion rates, it is vital for consumers to be able to easily access and use publicly available auto recall information.

NHTSA has taken important steps to improve its website—which provides safety recall information to consumers—resulting in most consumers in our focus groups finding the website easy to use. However, the difficulties some experienced in attempting to complete essential auto recall tasks demonstrated that NHTSA.gov does not always reflect key website usability practices for website design. Although NHTSA plans to conduct a website usability study with consumers after consolidating its websites, it has not determined a completion date for this effort—an essential step for organizations to effectively guide their information technology efforts. Without such a date, the website usability difficulties may persist and limit the effectiveness of NHTSA.gov in providing consumers with recall information about their vehicles. By addressing these difficulties in the interim, NHTSA can better assure that consumers obtain this information, which can be vital to their safety.

Recommendations for Executive Action

We are making the following two recommendations to NHTSA: The Administrator of NHTSA should determine a completion date for the agency’s website consolidation effort. (Recommendation 1)

The Administrator of NHTSA should, while the agency continues its website consolidation effort, take interim steps to improve the usability of the auto recall areas of NHTSA.gov by addressing the website usability difficulties we identified. (Recommendation 2)

Agency Comments

We provided a draft of this report to DOT for review and comment. In its written comments, reproduced in appendix VI, DOT stated that it concurred with our recommendations. The department also provided technical comments, which we incorporated as appropriate.

We are sending copies of this report to relevant congressional committees, the Secretary of Transportation, and the Administrator of NHTSA. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-2834 or flemings@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VII.

Appendix I: Objectives, Scope, and Methodology

This report examines the use of publicly available auto recall information for safety defects affecting passenger vehicles. The report addresses the following objectives: (1) How do consumers and industry stakeholders use publicly available auto recall information? (2) How easy or difficult to use do consumers find the auto recall areas of NHTSA.gov? (3) What steps, if any, has the National Highway Traffic Safety Administration (NHTSA) taken to raise consumer awareness about auto recalls and how has NHTSA evaluated the effectiveness of these steps?

We define publicly available auto recall information to include information on the auto recall areas of NHTSA.gov, such as examples of notification letters that manufacturers mail to consumers. This report focuses on safety defect vehicle recalls affecting passenger vehicles that are initiated when a defect in a vehicle or vehicle equipment creates an unreasonable safety risk, as determined by NHTSA or a manufacturer.

To determine how consumers use publicly available auto recall information, we conducted and analyzed transcripts and questionnaires from 12 consumer focus groups we conducted with used and new vehicle owners who had experienced an auto recall in the last 24 months. Each focus group was split into two sessions: (1) a discussion session to explore participants’ thoughts, experiences, and preferences about auto recall information and (2) a website usability testing session. Also, we administered a questionnaire as part of each of these sessions. For the discussion session, we asked consumers about the recall notification process and how they used the recall information, and for the website usability testing session, we asked consumers to fill in a questionnaire during the session itself as they assessed the usability of the auto recall areas of NHTSA’s website. We conducted the 12 focus groups at six locations across the country, with each group including 7 or 8 consumers for a total of 94 participants. Half of the focus groups were comprised of consumers who had completed the repair and the remaining half included consumers who had not completed the repair.

We selected the six focus group locations to provide population and geographic dispersion. To ensure geographic dispersion, we selected at least one location in each U.S. Census region (see table 3). To ensure population dispersion, we selected Metropolitan Statistical Areas representing a range of population sizes based on 2015 U.S. Census estimates. To ensure our selection included the perspectives of vehicle owners in geographically distant or isolated communities, we also selected a rural location, which we defined as a city or town that has a population of less than 50,000 inhabitants and is not an urbanized area contiguous and adjacent to a city or town that has a population of greater than 50,000 inhabitants.

Using information provided by the participants, we selected focus group participants based on age, income, gender, education level, race, and ethnicity to ensure we collected a range of perspectives on auto recall information use. However, since we did not select a representative sample of participants, focus group results are not generalizable to all vehicle owners.

During focus group discussion sessions, we asked participants to discuss factors they considered when deciding whether to repair their recalled vehicle and then to select the single most influential factor. Each of the 12 focus group sessions was audio recorded and transcriptions were created; transcripts served as the record for each group. We then evaluated those transcripts using systematic content analysis to identify the factors consumers considered when deciding whether to complete repairs and any suggested improvements to the auto recall communication process. The analysis was conducted in three steps. First, two analysts independently developed a code framework and then worked together to resolve any discrepancies. Second, each transcript was coded independently by analysts using the framework and any discrepancies were resolved by both analysts agreeing on the coding of the associated statement by a participant. Third, if needed, another analyst adjudicated any continued disagreement between coders.

Because the transcripts did not include a unique identifier for each focus group participant, we conducted our analysis of focus group session discussions at the group level (i.e., of the 12 focus groups we conducted). We also administered and analyzed a questionnaire as part of each discussion session to quantify responses regarding consumers’ use of auto recall information, including how they received and preferred to receive auto recall notifications. Our analysis of the questionnaire responses was conducted at the individual consumer level (i.e., of the 94 consumers who participated).

These focus group sessions were structured, guided by a moderator who used a standardized list of questions to encourage participants to share their thoughts, experiences, and preferences. We also conducted two pretest focus groups at our headquarters and made some revisions to the focus group guide prior to beginning the sessions with consumers. Methodologically, focus groups are not designed to demonstrate the extent of a problem or to generalize results to a larger population or provide statistically representative samples or reliable quantitative estimates. Instead, they are intended to generate in-depth information about the reasons for the focus group participants’ thoughts, experiences, and preferences on specific topics.

The projectability of the information produced by our focus group sessions is limited. For example, the information includes only the responses from the vehicle owners from the 12 selected groups and their individual responses to questions we asked. The experiences and preferences expressed may not reflect other vehicle owners’ thoughts and preferences. In addition, while the composition of the groups was designed to ensure a range of age and education levels, among the other criteria mentioned previously, the groups were not constructed using a random sampling method.

To determine how industry stakeholders use auto recall information, we interviewed selected auto manufacturers, selected franchised and independent auto dealerships, NHTSA program officials, and other industry stakeholders. Specifically, we interviewed representatives from the following 10 auto manufacturers, selected based on each manufacturer’s sales market share (small, medium, and large), place of ownership (foreign and domestic), and experience with auto recalls (lower to higher based on the average annual number of auto recall campaigns and average market share of each manufacturer from 2010 to 2014) to collect a range of perspectives on how manufacturers use auto recall information:

Tesla Motors, Inc.

To understand the perspective of auto dealers, we interviewed four franchised dealerships, one in each of the four U.S. Census regions where we conducted focus groups with consumers. We also interviewed three independent auto dealerships in two U.S. Census regions. The results of these interviews are not generalizable to all auto manufacturers and dealerships, but provide insights about how some industry stakeholders use auto recall information.

We conducted interviews with NHTSA program officials to understand NHTSA’s role in the auto recall process. In addition, we interviewed other stakeholders, including the Independent Monitor of Takata, which assists NHTSA in overseeing the Takata recall, as well as officials from consumer associations and other industry groups (see table 4).

To evaluate how easy or difficult consumers find the auto recall areas of NHTSA.gov to use, we reviewed various website usability resources to understand federal and industry key website usability practices for making websites easy to use, such as focusing on design and how easily users can find information. In addition, we reviewed federal standards for internal control related to communicating quality information externally. During our usability testing sessions, we asked consumers to attempt to complete auto recall tasks—the primary means NHTSA.gov provides for consumers to access information about auto recalls affecting their vehicles—and discuss their experiences. We then compared consumers’ experiences with the usability of the website against these practices.

To identify key website usability practices, we analyzed guidance documents from NHTSA and other federal agencies. For example, we analyzed the General Services Administration’s (GSA) and the Department of Health and Human Services’ Research-Based Web Design & Usability Guidelines, which includes quantified, peer-reviewed guidelines intended to help federal agencies improve the design and usability of their information-based websites. We also analyzed GSA’s Requirements for Federal Websites and Digital Services, and the U.S. Digital Services Playbook to identify key practices for making websites easy to use. Identified key practices are: (1) design and content— focusing on the layout, headers, and design; (2) navigation—how easily users can find information; (3) clarity—the ability to read and digest content; (4) identity and purpose—whether the site clearly presents its purpose; and (5) accessibility—the ability of people with physical or mental disabilities to use the site.

To analyze the results of focus group website testing sessions, we performed a systematic content analysis of the session transcripts using the same content analysis methods described above and an analysis of the questionnaire we administered to each participant during the website usability sessions. Specifically, we analyzed the transcripts from the website usability testing sessions to account for consumers’ experiences, including their initial impressions of the website and any suggested usability improvements. We also analyzed the results of the questionnaire that each participant completed where participants were asked to mark responses regarding their experience including an assessment of the usability of the auto recall areas of NHTSA.gov. Our analysis of the results from the questionnaire responses was conducted at the individual consumer level (i.e., of the 94 consumers who participated) while our analysis of focus group session discussions was conducted at the group level (i.e., of the 12 focus groups we conducted).

To corroborate the results of usability testing sessions we conducted with the consumers in our focus groups, we requested that five website usability professionals from GSA’s Federal User Experience Community conduct an independent evaluation of the auto recall areas of NHTSA.gov against federal and industry key website usability practices (described above). The website usability professionals developed a website usability evaluation form, which they used to individually evaluate the auto recall areas of NHTSA’s website. The website usability professionals then met to form a consensus and provided us with one final group evaluation of the website usability of the auto recall areas of NHTSA.gov. Also, although neither our usability testing nor the website usability evaluation conducted by website usability professionals directly addressed accessibility, we interviewed responsible agency officials about how the agency assesses the accessibility of NHTSA.gov.

We also requested and analyzed website data provided by NHTSA to understand how consumers access and use NHTSA.gov. Requested data included the number of subscribers to NHTSA’s Recall Notification E-mail System Sign-up; the number of weekly vehicle identification number (VIN) searches performed on NHTSA.gov from August 2014 through May 2017; and NHTSA.gov usage data by device (i.e., usage by mobile devices, tablets, and desktop computers). We assessed the reliability of these data by reviewing any supporting documents provided by the agency and interviewing responsible NHTSA officials, and concluded the data were sufficiently reliable for our reporting purposes.

While we did not independently review the usability of auto manufacturers’ auto recall websites, we requested and reviewed the results of any audits that NHTSA performed of these websites, including whether the websites met statutory and regulatory requirements for providing auto recall information to the public. We then corroborated any audit findings by reviewing the auto recall websites of the selected auto manufacturers that we interviewed.

To determine any steps NHTSA has taken to raise consumer awareness about auto recalls and how NHTSA evaluates the effectiveness of any steps, we reviewed relevant statutes, regulations, and proposed rules, including the Fixing America’s Surface Transportation Act and a Notice of Proposed Rulemaking related to recall notification methods. We also reviewed agency and other documents that describe or evaluate NHTSA’s public awareness activities. For example, we analyzed NHTSA’s strategic planning documents—such as NHTSA’s Strategic Plan 2016–2020—to identify ongoing public awareness activities along with their related goals, objectives, or performance metrics. Similarly, we requested and analyzed any documents NHTSA uses to evaluate the effectiveness of its public awareness activities, including performance reports for NHTSA’s ongoing Safe Cars Save Lives campaign.

To assess the reliability of data included in these performance reports— such as VIN searches performed—we reviewed agency documentation and interviewed agency officials about the reliability, accuracy, and completeness of the data and determined the data were sufficiently reliable for our reporting purposes. We reviewed performance management practices as provided in the Government Performance and Results Act of 1993 (GPRA), the GPRA Modernization Act of 2010, and standards for internal control in the federal government to identify any opportunities for improvement. We also performed a literature review to identify any related published articles and research studies.

To understand how NHTSA implements and evaluates any public awareness activities, we also interviewed responsible agency officials from NHTSA’s Office of Communications and Consumer Information and other offices. In addition, we discussed NHTSA’s public awareness efforts during interviews with industry stakeholders, including selected auto manufacturers, selected franchised and independent auto dealerships, and other industry stakeholders. We analyzed the results of these interviews along with the focus group discussions we conducted with consumers (discussed above) to identify perspectives on the effectiveness of NHTSA’s public awareness steps.

We conducted this performance audit from October 2016 to December 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Example of an Auto Manufacturer’s Recall Notification Letter

Appendix III: Annual Recall Completion Rates by Vehicle Component and Vehicle Type

The National Highway Traffic Safety Administration (NHTSA) is required to conduct a biennial analysis of vehicle safety recall completion rates and submit the results of its analysis in a report to certain congressional committees. The report must include, among other things, the annual recall completion rate by vehicle type and vehicle component (such as brakes, fuel systems, and air bags) for each of the 5 years preceding the year the report is submitted. According to NHTSA’s May 2017 report, completion rates for all vehicles combined ranged between 63 percent and 67 percent between calendar year 2011 and calendar year 2014 (see table 5). However, NHTSA reported wider variation when the recall completion rates are broken down by vehicle type. Similarly, the report found that completion rates for most component categories fall within a range of 60 percent to 75 percent (see table 6).

The annual completion rate is a volume-based, weighted metric, such that the more vehicles affected by the recall, the more weight or influence it has on the computed rate.

Appendix IV: Focus Group Participants’ Responses to Recall Notification Questionnaire

Focus group participants responded to a questionnaire we administered to collect information on consumers’ auto recall notification preferences during our discussion sessions. Table 7 shows participants’ responses to the administered questionnaire, by age group. We present these responses by age group, because consumers’ notification preferences may vary according to their ages.

Appendix V: Focus Group Participants’ Responses to Website Usability Questionnaire

Focus group participants responded to a questionnaire we administered to collect information on the usability of NHTSA.gov during our usability testing sessions. Table 8 shows focus group participants’ responses to the administered questionnaire, by age group. We present these responses by age group, because consumers’ website usability needs or preferences may vary according to their ages.

Appendix VI: Comments from the Department of Transportation

Appendix VII: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, H. Brandon Haller (Assistant Director); Katherine Blair; Jason Blake; Melissa Bodeau; Alicia Cackley; William Colwell (Analyst in Charge); Lacey Coppage; Elizabeth Dretsch; Jaci Evans; Marcia Fernandez; Sarah Kaczmarek; Malika Rice; Todd Schartung; and Andrew Stavisky made key contributions to this report.