Consumer Product Safety Commission: Awareness, Use, and Usefulness of SaferProducts.gov
Published: 2013-03-11T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In the wake of increased product recalls in 2007-2008, Congress passed the Consumer Product Safety Improvement Act of 2008 (CPSIA). Among other things, CPSIA required CPSC to establish a database on the safety of consumer products that is publicly available, searchable, and accessible through the CPSC website. In response, CPSC launched SaferProducts.gov (http:// www.saferproducts.gov ) in March 2011, which has two main functions--to provide (1) a mechanism for online reporting of product safety issues and (2) the ability to search for these issues or others, such as recalls. CPSIA also required GAO to study the general utility of the website. This report examines (1) CPSC's efforts to inform the public about SaferProducts.gov, (2) who is using the website and to what extent, and (3) the extent to which consumers have found the website to be useful. To do this, GAO analyzed agency documents and data from 2011 to 2012; interviewed CPSC officials, researchers, and consumer and industry groups; reviewed federal standards, guidance, and best practices for website usability; and conducted website usability tests with 37 consumers in three locations.
section_title: What GAO Found
paragraphs: The Consumer Product Safety Commission (CPSC) has used various approaches to inform the public about SaferProducts.gov, including using social media, public service announcements, and printed materials, and promoting the site during speeches and events. CPSC's efforts to inform the public about SaferProducts.gov have been part of a larger effort to raise awareness about the agency as a whole. While CPSC has employed many key practices for consumer education planning, it has not established metrics for measuring the success of its efforts. Without such metrics, the agency cannot determine which efforts have had the most impact on increasing awareness and use of the site.
While CPSC collects some data on the category of persons, such as consumers or health care professionals, who submit reports (one of the main functions of the site), it does not collect data about who is using the site to search for information (the other main function). In addition, to minimize the reporting burden on users, CPSC has not asked for demographic data about the users (such as their age, gender, or income level). Therefore, it was difficult for GAO to assess, as mandated by Congress, whether a broad range of the public has used the site. Moreover, without such data, CPSC has been limited in its ability to target its marketing and outreach efforts to increase use of the site.
Many consumers in GAO's usability tests thought the site generally was easy to use and had helpful information, but identified areas for improvement. The consumers generally could perform basic searches and follow instructions to report an unsafe product, and although none were aware of the site before the tests, most said they would use the site again. However, some of the search functions posed challenges. In addition, some consumers expressed concern about registering with the site and said this might prevent them from completing a report. Other consumers were not clear about the site's purpose, thinking it would focus on safe rather than unsafe products. By addressing the usability challenges GAO identified, CPSC could help users take full advantage of all the available features of SaferProducts.gov. Furthermore, cost-effective federal resources exist across the government to help agencies improve the usefulness of their sites.
section_title: What GAO Recommends
paragraphs: CPSC should (1) establish and incorporate metrics to assess efforts to increase awareness and use of SaferProducts.gov, (2) look for cost-effective ways of gathering additional data about site use, and (3) implement cost-effective usability improvements to the site. CPSC supported these recommendations.

section_title: Letter
paragraphs: A large number of children’s product recalls in 2007 and 2008 led to  heightened scrutiny of the Consumer Product Safety Commission’s  (CPSC) regulation of consumer products and public access to product  safety information. In response to these concerns, Congress enacted the  Consumer Product Safety Improvement Act (CPSIA) on August 14, 2008,  to strengthen CPSC’s authority to enforce safety standards and provide  greater public access to product safety information. CPSIA required  CPSC to establish a publicly available, searchable database of consumer  and other products and substances the Commission regulates that are  reported to be unsafe.
On March 11, 2011, CPSC launched the mandated database at  SaferProducts.gov (http://www.saferproducts.gov). The website has two  portals—one for consumers and one for businesses. For the consumer  portal, website users can perform two main functions on the site. First,  consumers and members of other statutorily defined groups can submit   Second, consumers  reports of harm or the risk of harm from products.and others can search for information on products reported to be unsafe  that they own or may wish to purchase. SaferProducts.gov also shows  manufacturers’ comments alongside incident reports if the manufacturer  requests their comments be published. The act also requires that GAO  analyze the utility of the website, including an assessment of its use by  consumers and efforts by CPSC to inform the public about the database.  In this report, we examine (1) CPSC’s efforts to inform the public about  SaferProducts.gov, (2) who has used SaferProducts.gov and to what  extent, and (3) the extent to which consumers have found  SaferProducts.gov to be useful.
To address the first objective, we reviewed CPSC marketing materials  and budget, evaluation, and planning documents, and interviewed CPSC  officials to determine the status of the agency’s efforts for  SaferProducts.gov. This included reviewing the targets of CPSC’s  outreach, how the agency evaluated the outcomes of these efforts, and  any future plans. We compared CPSC’s efforts with key practices for  consumer education planning identified in a prior report and interviewed  other entities (such as consumer groups) to determine what additional  steps, if any, CPSC could take to better inform the public about  SaferProducts.gov. For context, we reviewed documentation and interviewed officials about CPSC’s strategies to increase overall public  awareness of the agency. To address the second objective, we analyzed  data for 2011 through 2012 on users who filed incident reports through  various methods, including SaferProducts.gov, phone, e-mail, postal mail,  and fax, and analyzed other data CPSC maintains to measure the  amount of website use. To assess data reliability, we drew on our prior  work assessing SaferProducts.gov and reviewed relevant documentation,  performed manual tests to look for errors, and interviewed CPSC officials  responsible for the data. We determined that, for the purposes of this  report, the data were sufficiently reliable. We also interviewed CPSC  officials about efforts to collect any demographic data—such as age,  gender, and income—about users, and reviewed existing studies,  including those by consumer groups and a law firm, related to use of  SaferProducts.gov.
To address the third objective, we conducted website usability tests with  37 consumers in three locations—Washington, D.C., Dallas, Texas, and  San Francisco, California—to obtain their views on how easy it was to  use SaferProducts.gov and how helpful the information on the website  was. We chose these locations for geographic dispersion and ease of  testing. We followed the protocols and used the Washington, D.C.,  facilities of the General Services Administration (GSA) for our testing  conducted through the First Fridays Usability Testing Program. At this  location, we conducted three one-on-one tests with consumers whom  GSA recruited. We asked consumers to complete usability tasks we  developed and discuss their general experiences using the site. In the  other two locations, we worked with a contractor to recruit prospective  website testers having a mix of characteristics in terms of age, gender,  education, and ethnicity. We conducted two separate focus groups (of 8-9  consumers) in each of the other locations, asking them to complete  specified usability tasks and discuss their experiences using the website.  While the results of the tests are not generalizable to all U.S. consumers,  they provided us with in-depth interactive feedback and detailed  perspectives from a range of consumers about the usability challenges  associated with SaferProducts.gov. We also analyzed other usability  resources to determine key practices on how to make websites easy to  use and helpful. Appendix I contains a more detailed description of our  objectives, scope, and methodology. Appendix II contains the script used  in our website usability tests.
We conducted this performance audit from July 2012 to March 2013 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: CPSC was created in 1972 under the Consumer Product Safety Act to  regulate certain consumer products and address those that pose an  unreasonable risk of injury; assist consumers in evaluating the  comparative safety of consumer products; and promote research and  investigation into the causes and prevention of product-related deaths,  injuries, and illnesses. CPSC’s jurisdiction is broad, covering thousands  of types of manufacturers and consumer products used in and around the  home and in sports, recreation, and schools. CPSC does not have  jurisdiction over some categories of products, including automobiles and  other on-road vehicles, tires, boats, alcohol, tobacco, firearms, food,  drugs, cosmetics, medical devices, and pesticides. Other federal  agencies—including the National Highway Traffic Safety Administration;  Coast Guard; Bureau of Alcohol, Firearms, Tobacco, and Explosives;  Department of Agriculture; Food and Drug Administration; and  Environmental Protection Agency—have jurisdiction over these products.
Consumers and others previously were able to report safety problems or concerns about  consumer products through CPSC’s toll-free hotline; the U.S. mail, or a form on CPSC’s  website submitted through e-mail and they can continue to use these methods in lieu of  submitting reports through SaferProducts.gov. CPSIA and CPSC define “harm” as injury,  illness, or death or risk of injury, illness or death. 15 U.S.C. § 2055a(g); 16 C.F.R. §  1102.6(b)(4). be published. As required by statute, CPSC disclaims any responsibility  to guarantee the accuracy of a report.
The submitter of an incident report on SaferProducts.gov must fit into one  of five categories: (1) consumers; (2) local, state, and federal government  agencies; (3) health care professionals; (4) child service providers; and  (5) public safety entities. CPSC regulations specify that “consumers”  include, but not be limited to, users of consumer products, family  members, relatives, parents, guardians, friends, attorneys, investigators,  professional engineers, agents of a user of a consumer product, and  observers of the consumer products being used.
CPSIA requires the following information when submitting a report of harm: (1)  description of the consumer product sufficient to distinguish the product as a product or  component part regulated by CPSC; (2) identity of the manufacturer or private labeler by  name; (3) description of the harm related to use of the consumer product; (4) approximate  or actual date of the incident; (5) category of submitter; (6) submitter’s contact information;  (7) submitter’s verification that the information contained therein is true and accurate; and  (8) consent to publication of the report of harm. 15 U.S.C. § 2055a(b)(2)(B); C.F.R. §  1102.10(d). Subject to §§ 1102.24 and 1102.26, CPSC will publish reports of harm  containing all the required information. required to contact the submitters for further information.transmits a copy to manufacturers, importers, and private labelers  identified in the reports, to provide them with the opportunity to comment.  Qualifying reports and manufacturer comments submitted for publication  are then available on SaferProducts.gov (see fig. 1).
section_title: CPSC Has Used Various Approaches to Inform the Public about SaferProducts.gov, but Has Not Established Metrics to Assess Its Efforts
paragraphs: CPSC’s efforts to promote SaferProducts.gov formed part of a larger  effort to increase the public’s awareness of the agency. CPSC has taken  a variety of approaches to inform the public about SaferProducts.gov,  many of which are consistent with key practices for consumer education  planning. However, CPSC has not established metrics for its efforts. As a  result, the agency does not know which of its efforts have had the most  impact on increasing awareness and use of SaferProducts.gov.
subsections: section_title: Outreach about SaferProducts.gov Has Been Part of an Effort to Increase Awareness of CPSC
paragraphs: CPSC’s efforts to inform the public about SaferProducts.gov have been  part of a larger effort to increase the public’s awareness of the agency.  According to CPSC officials, certain segments of the public may not be  aware of the agency or its mission in product safety, much less be aware  of SaferProducts.gov. Likewise, roughly one-third of the 37 consumers  who participated in our website usability tests were aware of CPSC or its  mission. To promote awareness of CPSC, officials have conducted public  information campaigns related to various product safety hazards such as  fire hazards and those involving children’s products, issued press  releases about product recalls, and used social media. Officials said  that media stories promoting the use of SaferProducts.gov have had the  benefit of promoting CPSC as a resource not only for information about  product recalls (for which the agency is most commonly known), but also  as a place where consumers can raise concerns about the safety of  consumer products.
In addition to the outreach efforts noted above, CPSC has planned  initiatives to assess the public’s awareness of the agency as a whole. In  fiscal year 2011, CPSC’s Office of Communications received funds to  award a contract to plan and conduct field surveys to assess consumer  awareness of the agency. CPSC and a contractor are developing the  survey tool. These surveys are to cover such areas as the public’s  knowledge and awareness of the safety issues for which CPSC is  responsible, how the agency’s work affects consumers, and how the  public responds to product recalls and other safety hazards that CPSC  communicates. CPSC officials told us that they plan to administer the  survey in 2013, but have been awaiting approval of the survey from the  Office of Management and Budget. CPSC also recently redesigned its  main website, CPSC.gov, based on feedback from the public. According  to CPSC officials, this redesign allowed the agency to provide a more  visible link to SaferProducts.gov.
section_title: CPSC Has Used a Variety of Approaches to Inform the Public about SaferProducts.gov
paragraphs: As it has for publicizing the agency, CPSC has used a variety of  approaches to inform the public about SaferProducts.gov, including the  use of social and other media. Before launching SaferProducts.gov,  CPSC hosted a web conference on January 11, 2011, to inform  interested stakeholders such as consumer groups and the public about  the site’s search function and the information required to submit an  incident report.2011, CPSC promoted the new website through print and other media.  According to CPSC officials, the agency’s promotional strategy  emphasized both the public’s ability to search SaferProducts.gov for  reports and submit such reports. In addition, near the 1-year anniversary  of SaferProducts.gov, CPSC launched three public service   Around the time it launched SaferProducts.gov in March  announcements (PSAs) about SaferProducts.gov, sending these PSAs to  local and national media and making them available on online media  channels, such as YouTube (http://www.youtube.com). According to  CPSC officials, the agency has a contract with a video production  company to produce and distribute the videos. CPSC officials said that  the PSAs have been among the 10 most-viewed videos on CPSC’s  YouTube channel. They added that it was difficult to attract extensive  television coverage or the best airtime slots given CPSC’s PSA budget of  about $50,000 for fiscal year 2012. Further, the officials said that PSAs  can cost from $700,000 to $1 million to produce, distribute, and air during  prime viewing or listening times.
The agency also has distributed informational materials to target  audiences at conferences and community events; referenced the site in  speeches and presentations by the Chairman, Commissioners, and staff;  and held press interviews to promote the site, according to CPSC  officials. For example, CPSC developed a series of brochures, including  some tailored to specific professional sectors, such as health care, child  care, public safety, and government. CPSC officials noted that they have  mentioned the site at conferences, particularly those aimed at minority  populations and professional groups. The agency also has made a data  feed of the incident reports available to third-party software developers to  create mobile applications and provided information for developers in a  frequently asked questions page on SaferProducts.gov.
In conducting its public information efforts, CPSC has employed a  number of strategies consistent with key practices for consumer  education planning that we identified in a prior report. For example,  CPSC has worked with stakeholders such as consumer groups (a key  practice) to promote SaferProducts.gov, and used a variety of media  (another key practice) to promote the site. CPSC also has identified  “messengers” such as consumer groups and state attorneys general to  assist with publicity, and identified the resources needed for publicity  (other key practices).
Most of the consumer product safety experts we interviewed from nine  groups representing consumers, researchers, and various industries  stated that CPSC has been taking appropriate measures to promote the  site. However, some also suggested that CPSC could conduct more  targeted outreach to other professional groups, such as those in health  care, and other populations, such as parents.
section_title: CPSC Has Not Established Metrics to Evaluate Its Outreach on SaferProducts.gov
paragraphs: While CPSC has employed many of the key practices for consumer  education planning as described previously, it has not employed one of  the key practices that could further improve the efficacy of its outreach for  SaferProducts.gov. Specifically, CPSC has not established metrics, such  as process and outcome metrics, to measure the success of its outreach   In its 2013 performance budget request, as part of an effort to  efforts.increase awareness of the agency, CPSC has a goal for the number of  visits to CPSC.gov. However, CPSC does not have a similar goal for the  number of visits to SaferProducts.gov, although it collects such data (as  discussed in the next section of this report). Similarly, CPSC has not  determined whether its efforts to publicize SaferProducts.gov at  conferences or through PSAs have led to increased use of  SaferProducts.gov after the events. CPSC also has not incorporated tools  or features on the site (such as a drop-down menu on the homepage that  would ask users to select an option such as “conferences,” “PSAs,”  “printed materials,” or “media”) to identify how the user learned about and  arrived at the site. The information generated by such tools also may  provide CPSC with ideas for additional metrics to measure awareness  and use of the site.
CPSC has not established metrics to evaluate its outreach efforts for  SaferProducts.gov because the agency has been focused on increasing  awareness of CPSC and improving the functionality of CPSC.gov. CPSC  officials said that in comparison with SaferProducts.gov, CPSC.gov  received almost 10 times as many visits each month. Officials have said  they may focus on evaluating outreach efforts for SaferProducts.gov in  the future. However, without current metrics to assess the efficacy of its  outreach for SaferProducts.gov, CPSC will not know which of its efforts— for instance, promoting the site at conferences and using PSAs—have  had the most impact on increasing awareness and use of  SaferProducts.gov, or be able to best target its limited resources to  increase use of the site.
section_title: CPSC Collects Limited Data about the Use of SaferProducts.gov
paragraphs: CPSC collects limited data about the use of SaferProducts.gov. To track  use, CPSC collects data on the number of visitors, most frequently visited  pages, and number of reports received, among other metrics. CPSC also  collects some data about the category of person who is submitting a  report. However, CPSC does not collect any data about who is using the  site to search for information. In particular, CPSC has not sought to  collect demographic data, such as age, gender, or income. In mandating  this report, Congress required us to assess whether a broad range of the  public uses the site. However, CPSC’s limited data collection related to  use of the site made it difficult to conduct such an assessment.
subsections: section_title: Website Use Remained Relatively Constant, with Searches and Submission of Reports Constituting Frequent Uses
paragraphs: According to CPSC officials, the agency’s primary measure of the extent  of use of SaferProducts.gov is the number of visitors each month. CPSC  collects these data through web analytics software. According to CPSC’s  data, visits to SaferProducts.gov exceeded 100,000 each month since  June 2011, a few months after the launch of the site (see fig. 2), peaking  at about 238,000 in November 2012. CPSC officials have not been able  to identify the reasons for the increase in visits.
CPSC also collects data on the most frequently visited pages each month  (see fig. 3). These data show that users frequently used the site to search  for information—for example, to search for recalled products or incident  reports submitted by other users of the site.
CPSC also collects data on the number of reports received each month  through SaferProducts.gov, as well as by phone, e-mail, postal mail, and  fax. These data show that users submitted more than 1,000 reports from  all sources each month from March 2011 through December 2012 (see  fig. 4).
CPSC collects some data about the categories of persons using  SaferProducts.gov to submit incident reports but does not collect  additional data such as age, gender, or income level of the submitters or  others who use the site to search for information. When completing a  report, CPSC requires submitters to state whether they are consumers,  represent a government agency, or are health care or other professionals,  among other categories of user. As shown in table 1, our analysis of more  than 12,000 reports posted on SaferProducts.gov as of January 2013  found that most report submitters—about 97 percent—identified  themselves as consumers, results consistent with our prior reporting. As  stated previously, “consumers” include, but are not limited to, users of  consumer products, family members, relatives, friends, attorneys,  investigators, and others. Representatives of government agencies and  public safety entities, as well as health care professionals, child service  providers, and medical examiners and coroners also submitted reports.
CPSC also asks report submitters to state their relationship to the victim  of the incident (such as self, parent, or spouse). As shown in table 2, of  those who identified themselves as consumers, most identified  themselves as the victims of an incident. However, many submitters did  not specify a relationship.
Of those who did specify a relationship to the victim, 4,463, or 60 percent,  reported that they were the victims, and 1,867, or 25 percent, reported  that their child was the victim (see table 3).
CPSC asks that submitters specify the location of the reported incident,  including the country and state. Most submitters providing this  information—about 90 percent—reported that the incident took place in  the United States (see table 4). Submitters also reported that incidents  took place in other countries or did not specify where the incident took  place.
In addition, states with the highest population—such as California, Texas,  New York, Florida, and Illinois—had the most reported incidents (see fig.  5).
Beyond these data, CPSC does not request or obtain additional details  about the users of SaferProducts.gov. According to CPSC officials, the  agency also cannot distinguish new users from returning users because  CPSC’s web analytics software has not been configured with “cookies” to  capture these data. Officials have cited resource and privacy concerns  as reasons for not collecting these data, although they said they have  been considering using cookies in the future. In addition, CPSC does not  collect more specific demographic information such as age, gender, or  income level from the submitters of reports or other site users, citing an  interest in minimizing the reporting burden on users. As an example,  CPSC has not requested that site users voluntarily provide this  information during the report submission process or after submitting a  report. Congress required us to assess whether a broad range of the  public uses SaferProducts.gov, but CPSC’s limited data collection made it  difficult to conduct such an assessment. In addition, standards for internal  control in the federal government state that agencies should have timely,  relevant information for management decision-making purposes.result of its limited data collection about users of the site, CPSC has been  limited in its ability to target its marketing and outreach efforts on specific  groups, populations, or areas to achieve the goal of increasing use of the  site.
section_title: Consumer Testing Showed SaferProducts.gov to Be Generally Useful, but Identified Areas for Improvement
paragraphs: As discussed earlier in this report, our website usability tests focused on  asking consumers in our testing sessions to judge if SaferProducts.gov  was easy to use. We had the consumers perform various tasks (such as  searching for recalled products and submitting mock incident reports) and  asked for opinions about the site’s usefulness. A moderator facilitated the  sessions and we elicited feedback from participants. In addition, a GSA  official with expertise in website usability assessed SaferProducts.gov,  and another GSA official reviewed the site for website accessibility.
subsections: section_title: Consumers Said the Site Was Easy to Use, but Noted a Few Search Functions and a Registration Page as Problematic
paragraphs: Many consumers in our testing sessions generally found  SaferProducts.gov easy to use, but they encountered difficulties with  certain aspects of the two main functions: searching for information and   Of the 37 consumers who participated in our  submitting incident reports.testing sessions, 20 found SaferProducts.gov easy to use as indicated by  their responses to a questionnaire we administered following each test  session. For example, almost all the consumers were easily able to  determine what initial steps to take to search for or report a product that  may be unsafe. In addition, the expert evaluator reviewing the site at our  request described the site as clean and easy to navigate.
In conducting the search tasks, consumers generally were able to find  recalled products using basic key word searches. But some search  functions, including those that required more complicated searches such  as use of an advanced search function to narrow results, posed  challenges. For example, in one testing session, no consumers were able  to complete a task that required them to narrow their search by injury,  time period, and location. In another session, the calendar function, which  filters the results by time period, posed particular challenges. Five of the  eight consumers in that session experienced difficulties in having to enter  and, when seeking to make one change, re-enter all the dates to focus  their search on products recalled within a particular time period. The  expert evaluator from GSA experienced similar challenges in using the  calendar function.
In addition, when asked to search for and compare safety information for  two products—one for which there were search results and one for which  there were none—almost all the testers had difficulty interpreting the lack  of search results for the latter product. For example, while some testers  assumed that a search for a product that produced no results indicated  that the product was safe, others did not make this presumption. In our  testing sessions, most consumers were not sure which product to  purchase based on their searches and roughly a quarter indicated that  they would leave SaferProducts.gov to search other websites if they  found no results on SaferProducts.gov. In contrast to SaferProducts.gov,  other websites inform users of a possibly incorrect search term, such as a  typographical error, which helps users interpret the results of their  searches and identify potential errors.
During the usability tests, consumers experienced fewer challenges using  the reporting function than the search function. To submit an incident  report, consumers must enter information on a series of pages that  include a combination of required and optional fields. During our testing,  consumers found the instructions for submitting a report to be generally  clear. For example, almost all the testers thought the instructions for  submitting information about the incident, product, and victims were clear.
However, 15 of the 37 consumers in our test sessions expressed concern  about apparently needing to register before submitting a report and  generally did not notice that they could continue without registering (see  fig. 6). By registering on SaferProducts.gov, site users can save their  reports to complete at a later time and receive updates on the status of  reports. When reaching the registration page, over a third of the  consumers in our focus group sessions said that they would not be  inclined to register. In one session, seven of nine consumers said they  would not be inclined to register and thought that having to register was a  deterrent to completing a report. In another session, none of the  participants noticed the option to skip registration. Some of those who  noticed that they could skip registration emphasized that the option  should be more prominent—for example, placed alongside the  registration box rather than below it where it might not be immediately  visible. Likewise, as an issue of website usability, the expert evaluator  from GSA reviewing the site on our behalf noted that the “continue  without registering” option was not prominent enough and stated that  registration may deter users from continuing the report submission  process.
In addition, some consumers in our testing sessions said that the  reporting pages contained too many questions and described the  submission process as cumbersome, particularly for busy individuals  such as parents. To address this, one consumer suggested grouping all  of the required fields on one page. The expert evaluator from GSA also  suggested that all the questions in the reporting process should be  reviewed to determine if each was necessary.
When CPSC first developed SaferProducts.gov, the agency conducted  three focus groups—one with consumers and one with professionals—to  test the site and assess users’ experience with it.testing only addressed the incident reporting function, not the search  function, and focused on (1) awareness of how and where to submit a  safety complaint and (2) general reactions to the site. CPSC has not  conducted additional usability testing since launching SaferProducts.gov  in March 2011. As mentioned previously, CPSC officials have said that  issues such as assessing the level of awareness of CPSC and  redesigning CPSC.gov were higher priorities than assessing and  improving SaferProducts.gov.
CPSC’s focus group  A number of resources across the federal government are available to  help agencies in making their websites more usable. For example, as  cited previously, GSA’s First Fridays Usability Testing Program is  designed to teach agency officials how to find and fix usability problems  at no cost to the agency. The program’s services are (1) formal tests, (2)  quick tests, (3) mobile tests, (4) observation, and (5) expert evaluation.GSA also offers DigitalGov University, which includes courses in web  design and usability best practices. In addition, the Department of Health  and Human Services (HHS) operates two usability labs, both of which are  free of charge to other federal agencies, to evaluate websites to ensure  that they are easy-to-use and useful. Furthermore, GSA and HHS  maintain HowTo.gov and Usability.gov, respectively, to provide guidance  and resources to help agencies create websites that are usable, useful,  and accessible.
Because of the usability issues in the areas we identified, consumers may  not take advantage of all the features of SaferProducts.gov, and  consumers may be dissuaded from completing and submitting incident  reports. As a result, CPSC may not be obtaining all possible information  from consumers that can help inform its safety assessments and other  regulatory efforts.
section_title: Consumers Initially Were Not Aware of SaferProducts.gov and Thought the Site’s Purpose Was Unclear, but Found the Site Generally Useful after Using It
paragraphs: None of the consumers in our test sessions previously had heard of  SaferProducts.gov, although a few were familiar with CPSC as an agency  involved in recalling certain products. In addition, 5 of the 37 consumers  who participated in our tests said that the purpose of SaferProducts.gov  was not clear based on its name and the initial information on the home  page. In our testing sessions, roughly a third of the consumers  commented that the name of the website—SaferProducts.gov—and the  home page did not accurately convey what consumers could and could  not do on the site. For example, when asked about their impressions of  SaferProducts.gov, over a quarter of the testers thought that they would  find information about safe products, such as a list of products that meet  certain standards or a rating of products. These consumers did not  appear to notice information on the home page indicating that they would  only find information on unsafe products (see fig. 7). In our testing  sessions, several consumers commented that the website would be more  aptly named UnsafeProducts.gov. In addition, during our one-on-one  testing sessions in Washington, D.C., two of the testers had difficulty  distinguishing between recall notices and incident reports, which serve   Likewise, although the expert evaluator from GSA  different purposes.was able to obtain a general sense of the purpose of the site, he noted  that a tagline (brief text that gives users an immediate idea of what the  site does) would help reinforce the site’s purpose.
In one testing session, a few consumers also said that it was not apparent  from looking at the home page that CPSC did not regulate certain  categories of products, such as automobiles and medications, although  more than half of the consumers in our testing sessions said at the outset  that they routinely searched online for safety information on particular  products. Similarly, none of the consumers in our testing sessions noticed  that they could be directed to the agency’s main site, CPSC.gov, by  clicking on certain links in SaferProducts.gov. Only the expert evaluator  noticed that by opening a recall notice, the website user would leave  SaferProducts.gov and go to CPSC.gov.
However, as consumers completed the various tasks in the testing  sessions, they better understood the website’s features and functions. In  responding to our closing questions about their overall experiences in  using the site, most said that they would use SaferProducts.gov again  now that they were aware of it. For example, some consumers found  information about product recalls the most useful component of the site  and said they would give more weight to this information. In our testing  sessions, about one-quarter of the consumers also found value in the  incident reports, noting that they helped website users understand  whether products that had not yet been recalled had safety issues. Two of  the consumers commented that they found the content of the reports to  be more credible than other websites that provide a forum for consumer  complaints. In addition, a few consumers pointed to the amount of detail  in the reports, such as the incident description, location, and date as  particularly helpful.
Nevertheless, because of the usability issues in the areas we identified  (for example, not having a clear and “up-front” statement of what the site  contains and how it can be used), consumers may not use all of the site’s  available features and be dissuaded from completing and submitting  reports. CPSC officials also acknowledged that awareness of the agency  could be heightened if consumers were informed about CPSC while using  or searching SaferProducts.gov.
section_title: Conclusions
paragraphs: CPSC has used many approaches to inform the public about  SaferProducts.gov, employing many key practices of consumer education  planning in the process. Incorporating the promotion of SaferProducts.gov  into its broader effort to increase awareness of the agency has  represented a logical approach that has prevented duplication. However,  our work confirms CPSC’s perception that public awareness of  SaferProducts.gov is likely low. For example, none of the participants in  our usability tests had heard of SaferProducts.gov prior to the testing.  Although CPSC has employed many of the key practices for consumer  education planning, it has not established metrics to measure the success  of its efforts. By establishing such metrics, the agency would be better  able to determine which of its outreach efforts had the most impact on  increasing awareness and use of the site and thus could more effectively  target its limited resources to increase use of the site.
In addition to establishing and using metrics, more data about the use of  SaferProducts.gov could help CPSC target its marketing and outreach.  Currently, CPSC collects limited data about the use of SaferProducts.gov.  For example, it collects data on the number of visitors, but not whether  they are using the site to search for information—one of the main  functions of the site. It also does not collect demographic data about the  users’ age, gender, or income. These types of data could help CPSC  identify groups, populations, or areas on which to focus to further  increase use of the site.
Our usability testing with consumers identified other ways in which CPSC  may increase the use of SaferProducts.gov. Although our testing revealed  that many consumers found the site generally easy to use, it also  revealed that certain search functions, site registration, and lack of a clear  statement of purpose posed challenges for some users. By improving the  site in these areas, CPSC could help ensure that consumers take  advantage of all the features of the site and are able to search for and  report information in an easy and convenient manner. Making these  improvements also may provide CPSC with additional reports from  consumers to inform its safety assessments and other regulatory efforts.
section_title: Recommendations for Executive Action
paragraphs: To improve the awareness, use, and usefulness of SaferProducts.gov,  CPSC should take the following three actions: establish and incorporate metrics to assess efforts to increase  awareness and use of SaferProducts.gov,  look for cost-effective ways of gathering additional data about the  users and their use of SaferProducts.gov, and  implement cost-effective usability improvements to  SaferProducts.gov, taking into account the results of any existing  usability testing or any new testing CPSC may choose to conduct.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to CPSC for review and comment. In  commenting on the draft report, the Chairman and Commissioners stated  that they support the report’s recommendations. Specifically, they stated  that CPSC staff will look for cost-effective ways to improve awareness of  SaferProducts.gov, improve the usability of the site based on research on  best practices in web design, and gather additional metrics about users.  The Chairman and Commissioners’ comments are reprinted in appendix  III. CPSC also provided technical comments that we incorporated in the  report as appropriate.
We are sending copies of this report to interested congressional  committees and to the Chairman and Commissioners of CPSC. In  addition, the report will be available at no charge on GAO’s website at  http://www.gao.gov.
If you or your staff members have any questions about this report, please  contact me at (202) 512-8678 or cackleya@gao.gov. Contact points for  our Offices of Congressional Relations and Public Affairs may be found  on the last page of this report. GAO staff who made major contributions to  this report are listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The objectives of our report were to examine (1) the Consumer Product  Safety Commission’s (CPSC) efforts to inform the public about  SaferProducts.gov, (2) who has been using the website and to what  extent, and (3) the extent to which consumers have found the website to  be useful.
For the first objective, we reviewed CPSC marketing, budget, evaluation,  and planning documents to determine the status of the agency’s public  information efforts related to SaferProducts.gov. We sought to determine  whom CPSC had been targeting; how, if at all, the agency was evaluating  the outcomes of its efforts; and any future plans to promote awareness  and use of the site. We compared CPSC’s efforts with criteria on key  practices for consumer education planning. We interviewed consumer  product safety experts from nine groups representing consumers,  researchers, and various industries to determine what additional steps, if  any, CPSC could take to better inform the public about  SaferProducts.gov. We identified these experts through our prior work or  based on recommendations from those we interviewed. For context, we  interviewed an official in CPSC’s Office of Communications and reviewed  CPSC’s strategies to increase awareness of the agency as a whole.
See GAO-12-30. reliability and determined that, for the purposes of this report, the data  were sufficiently reliable. We did not review specific incident descriptions  that individuals filed and do not attest to the reliability of that information.
For the third objective, we conducted website usability tests with 37  consumers—who represented a mix of demographic characteristics in  terms of age, gender, and educational level—to obtain their views on how  easy it was to use SaferProducts.gov and how useful they found the  website. We conducted the tests in Washington, D.C., Dallas, Texas, and  San Francisco, California. We chose these locations for geographic  dispersion and ease of testing. We followed the protocols and used the  Washington, D.C. facilities of the General Services Administration (GSA)  for our testing conducted through the First Fridays Usability Testing  Program. At this location, GSA recruited three volunteer testers on our  behalf. Consistent with the GSA program protocols, a moderator  facilitated the testers’ execution of various website tasks, such as  searching for recalled products and submitting mock incident reports. We  followed similar protocols in San Francisco and Dallas. To identify the  participants in San Francisco and Dallas, we worked with a contractor to  recruit prospective testers who had a mix of demographic characteristics.  We held two focus groups in each location, for a total of four groups. Two  groups had eight participants per group and the other two groups had  nine participants per group. In all four groups, a moderator facilitated the  testers’ execution of various tasks, as was done in Washington, D.C.  Although the results of our usability tests are not generalizable to all U.S.  consumers, they provided us with in-depth, interactive feedback and  detailed perspectives from a range of website users about the usability  challenges associated with SaferProducts.gov.
To supplement our approach, we requested and reviewed an expert  evaluation conducted by the First Fridays program manager. The GSA  official evaluated SaferProducts.gov based on the following criteria: (1)  accessibility—the ability of people with physical or mental disabilities to  use the site; (2) identity and purpose—whether the site clearly presents  its purpose, including what the site offers and what a user can do on it; (3)  clarity—the ability to read and digest content; (4) navigation—how easily  users can find information; and (5) design and content—focusing on the  layout, headers, and design. Another GSA official provided a more in- depth accessibility review of SaferProducts.gov to identify issues that  users with disabilities might encounter when navigating the site.  According to GSA, although an expert evaluation can be a useful starting  point for determining a website’s usability strengths and weaknesses, the  expert evaluation emphasizes the importance of the user experience. In  addition, we reviewed various other website usability resources and  criteria, including Usability.gov, to understand the key practices for  making websites easy to use and helpful.
We conducted this performance audit from July 2012 to March 2013 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: GAO Script, Including Questions and Tasks, for Usability Testing with Consumers
section_title: Appendix III: Comments from the U.S. Consumer Product Safety Commission
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Debra Johnson, Assistant  Director; Meghana Acharya; Mark Bird; William Carrigg; Jeremy Cluchey;  Meredith Graves; Ronald Ito; Sarah Kaczmarek; May Lee; Marc Molino;  Patricia Moye; Barbara Roesmann; Andrew Stavisky; and Julie Trinder  made key contributions to this report.