Broadband Performance: Additional Actions Could Help FCC Evaluate Its Efforts to Inform Consumers
Published: 2015-04-17T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Broadband is increasingly seen as an essential communications service, with applications in education, medicine, public safety, and entertainment. However, consumers can experience broadband performance problems. FCC, which has primary responsibility for regulating broadband, has taken steps to measure broadband performance and to require that ISPs give consumers information about the performance of their services.
GAO was asked to review issues related to broadband performance information. This report examines (1) broadband performance information available to consumers and its limitations, if any, and (2) FCC's actions to evaluate its efforts to provide consumers with broadband performance information.
To address these objectives, GAO reviewed FCC proceedings; conducted a literature review; analyzed comments filed with FCC regarding broadband performance information; and interviewed FCC officials and various stakeholders from industry and public interest groups.
section_title: What GAO Found
paragraphs: Consumers can access broadband performance information from several sources, including Internet service providers (ISP), online speed tests, and the Federal Communications Commission (FCC); however, the information has some limitations. For example:
ISP information: FCC's transparency rule requires ISPs to disclose broadband performance information, but ISPs' disclosures vary, and some stakeholders said that the lack of standardization of disclosures can make it difficult for consumers to compare broadband services. Some ISPs, however, question the need and benefit of standardized disclosures. FCC recently adopted enhancements to the transparency rule, such as specifying what information ISPs must disclose, and FCC is considering potential disclosure formats.
Speed tests: Consumers can use information from these tests to verify their broadband speeds. However, speed tests can be affected by many factors and may not detect congestion affecting a specific website. Thus, it can be difficult for consumers to identify the cause of their broadband performance problems.
FCC reports: Through the Measuring Broadband America (MBA) program, FCC tests ISPs' networks and compares their actual and advertised speeds in an annual report . However, the report is not targeted toward consumers, and stakeholders stated that consumers may not be aware of the report.
FCC has taken steps to evaluate its efforts to provide consumers with broadband performance information, such as its transparency rule and MBA program; however, FCC's ability to evaluate its efforts is limited by a lack of useful performance information and relevant performance goals and measures. For instance, while FCC obtains information about the effectiveness of its efforts from stakeholders' comments and consumers' complaints, FCC has not sought performance information from more objective sources, such as consumer research—as has been done by other government entities. Further, although consumer complaints can provide FCC with valuable insights on topics of interest to the commission, FCC has acknowledged that complaints may not provide a complete picture of consumer's information needs and some industry stakeholders have questioned FCC's reliance on complaints as a basis for making decisions. In addition, although FCC's strategic plan includes strategic objectives related to informing consumers about broadband networks, FCC lacks performance goals and measures to monitor the impact and effectiveness of its efforts to provide consumers with broadband performance information. GAO has previously reported that critical elements of effective performance management include information that is complete and consistent, with performance measures linked to goals. Without such information and measures, FCC cannot be assured that its efforts to provide consumers with broadband performance information are effective and meeting consumers' needs.
section_title: What GAO Recommends
paragraphs: FCC should take additional steps to evaluate its efforts to provide consumers with broadband performance information. This should include: (1) conducting or commissioning research on the effectiveness of its efforts and making the results publicly available, and (2) establishing performance goals and measures that allow FCC to monitor and report on these efforts. FCC concurred with GAO's recommendations.

section_title: Letter
paragraphs: Broadband is increasingly seen as an essential communications service,  with applications in education, medicine, public safety, and entertainment.  Broadband allows people to share large amounts of information at  increasingly faster speeds and provides access to information otherwise  unavailable, such as online courses not offered in their local area. Press  reports indicate that consumers spent over $100 billion on Internet access  in 2013, and the Federal Communications Commission (FCC) has noted  that the number of fixed and mobile residential broadband connections in  the United States grew from 48 million in December 2010 to nearly 180  million in December 2013. Consumers’ use of broadband has changed  over the years as well, with more people using broadband to access  video and data-intensive applications, and this trend is expected to  continue. For example, Cisco (a provider of network infrastructure)  projects that consumer Internet traffic in North America will grow at an  average annual rate of 21.5 percent between 2013 and 2018. However,  FCC has previously expressed concerns regarding consumers’ access to  accurate and useful information about their broadband service  performance. In addition, GAO has previously identified limitations in the  broadband performance information available to consumers, and recently  noted the need to ensure consumers understand issues that can affect  their use of broadband, such as usage-based pricing.
Preserving the Open Internet, Report and Order, 25 FCC Rcd. 17905 (2010). interview based on our review of comments filed in FCC’s 2014 Open  Internet and related proceedings, as well as based on recommendations  from other organizations we interviewed.
To determine how FCC evaluates its efforts to provide consumers with  broadband performance information, we reviewed relevant FCC  documents, such as FCC’s strategic and performance plans. We also  reviewed FCC’s requests for comment on consumers’ information needs  and the effectiveness of FCC’s efforts to provide consumers with  broadband performance information, as well as selected comments filed  in response to these proceedings (as described above). We also  reviewed prior GAO reports on FCC and results-oriented management  and compared FCC’s use of performance information and measures to  performance management best practices outlined in previous GAO work.  To further inform our assessment, we interviewed representatives from  the stakeholder groups identified above, as well as officials from the  Federal Trade Commission and National Telecommunications and  Information Administration regarding their efforts to inform consumers  about broadband issues and to evaluate the effectiveness of those  efforts. In addition, we reviewed reports and studies on consumer  disclosures conducted by FCC’s Transparency Working Group; foreign  telecommunications regulatory bodies, including those in the United  Kingdom and European Union; and other federal agencies, such as the  Consumer Financial Protection Bureau. See appendix I for more  information about our scope and methodology.
We conducted this performance audit from July 2014 to April 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: The term “broadband” commonly refers to Internet access that is high  speed and provides an “always-on” connection, so users do not have to  reestablish a connection each time they access the Internet. Broadband  speeds are described in download and upload capabilities, and are  defined as the maximum rates at which data are delivered over the  Internet. A higher speed indicates a faster information delivery rate. For  example, a 10 megabits per second (Mbps) service should deliver ten  times as much data as a 1 Mbps service in the same period of time.
Internet service providers (ISP) have marketed and competed with one  another on the speed of their various broadband plans. Advances in  technology, such as the use of fiber (described below) and new wireless  technologies, have allowed ISPs to offer increasingly faster speeds that  support new services and applications, such as streaming video. FCC  takes those advances into account as part of its required inquiry into  whether “advanced telecommunications capability” is being deployed to  all Americans in a reasonable and timely fashion. Pursuant to this  inquiry, FCC recently updated the minimum speed benchmark it uses to  measure the availability of advanced telecommunications capability. In  addition to speed, a number of factors can be used to evaluate  broadband performance. See table 1.
Consumers generally subscribe to broadband in two ways:
Fixed: In-home fixed Internet plans are often sold as a monthly  subscription by cable television or telephone companies. Service from  cable television companies is generally provided through the same  coaxial cables that deliver television programming. Service from  telephone companies is generally provided through traditional copper  telephone lines—commonly referred to as digital subscriber line (DSL)  service—or fiber-optic lines, which convert electrical signals carrying  data into light and send the light through glass fibers. These network  technologies generally have higher speeds than mobile networks.  Consumers can connect a variety of devices to in-home fixed  networks through a wired connection or wireless Wi-Fi connection  (see fig. 1). Consumers are increasingly using the Internet to  supplement or replace their use of traditional services, such as  traditional telephone and cable TV service.
Mobile: Traditionally, mobile providers sold access to the Internet as  an add-on to mobile telephone service plans that may or may not  include a multiyear contract. Mobile service is provided through cell  tower coverage with data sharing through radio spectrum. A number  of devices may connect to mobile broadband networks, such as smart  phones, tablets, and mobile devices that enable laptops to connect to  mobile networks (see fig. 2).
The 2014 American Customer Satisfaction Index (ACSI)with respect to fixed broadband service, consumer satisfaction regarding  data speeds and video streaming increased from 2013 to 2014. However,  ACSI also found that during this time period, customer satisfaction related  to service interruptions and outages, speed and service reliability, and  peak evening-hour performance declined. With respect to mobile  broadband providers, ACSI reported that customer satisfaction related to  speed for downloading data and streaming content increased from 2013   reported that  to 2014; however, ACSI also reported that the customer satisfaction  scores indicated that improvements were still needed in speed and  reliability.
Broadband performance problems can manifest in a number of ways,  such as slow loading of content, lost connections, and pauses in  streaming video and audio. FCC, ISPs, content providers, and  researchers have noted that a variety of factors can affect broadband  performance, such as accessing the Internet during peak usage times,  outdated equipment in the home, or multiple users in the same household  accessing the Internet simultaneously. In addition, mobile broadband  performance will vary based on the location of the phone or device.  Running multiple applications at once or having multiple users in a  household may require greater speeds. Different types of Internet  applications use varying amounts of data and require different minimum  speeds to operate properly (see table 2). For example, email and web- browsing use small amounts of data, while watching movies uses larger  amounts of data. Use of data-intensive broadband applications is  projected to grow. FCC has noted that consumers are increasingly  using broadband applications that require large amounts of data, such as  streaming video content, which can require more consistency of service  and be more visibly affected by technical problems.
In addition, congestion at various points on the network can affect a  user’s broadband performance. In order to connect their customers to the  Internet, ISPs link with other networks, including “backbone providers”  that move Internet traffic long distances and interconnect with other  networks. Companies may use a number of options to deliver their  content across the Internet, such as connecting with a backbone provider  (directly, or through an ISP), connecting directly to the consumer’s ISP, or  paying a content delivery network to deliver their content to ISPs. See  figure 3 for a simplified illustration of how content moves from websites  and applications to consumers. Congestion occurring within the content  provider’s network, at points of interconnection between the networks, or  within the ISP’s network can all affect a consumer’s broadband  performance. ISPs’ traffic management policies can also affect a  consumer’s broadband performance. This issue is discussed later in the  report.
FCC has primary responsibility for regulating broadband. In response to a  requirement in the American Recovery and Reinvestment Act of 2009,  FCC staff developed The National Broadband Plan, which includes a  series of recommendations aimed at ensuring all Americans have access  to broadband. In addition, section 706 of the Telecommunications Act of  1996, as amended, directs FCC “to encourage the deployment on a  reasonable and timely basis of advanced telecommunications capability  to all Americans” through a variety of measures, including measures that  promote competition in the local telecommunications market. FCC has  repeatedly noted the connection between informed consumers and  increased competition, noting that the latter is an important part of  promoting broadband deployment.
Preserving the Open Internet, Report and Order, 25 FCC Rcd. 17905 (2010), aff’d in  part, vacated and remanded in part sub nom, Verizon v. FCC, 740 F.3d. 623 (D.C. Cir.  2014). incentive and ability to act in ways that limit Internet openness (e.g., by  blocking or discriminating against unaffiliated applications and services).
section_title: Although Consumers Can Access a Number of Sources of Information on Broadband Performance, the Information Is Limited
paragraphs: Consumers can access information on broadband performance from a  number of sources, including their ISPs, third-party websites, and FCC  reports. FCC adopted a transparency rule requiring ISPs to provide  consumers with information on their services’ performance  characteristics. However, the information available to consumers is not  standardized, and some stakeholders stated that this lack of  standardization makes it difficult for consumers to compare broadband  services. In addition, the information available through speed tests may  not accurately reflect the end-user’s actual experience. FCC has taken  steps to measure and report on the extent to which ISPs are providing the  speeds they advertise. However, the reports are not targeted toward  consumers, and stakeholders stated that consumers might not be aware  of the information.
subsections: section_title: Performance Information from ISPs and Third Parties Is Not Standardized and May Not Accurately Reflect the End-User’s Actual Experience
paragraphs: Consumers interested in purchasing broadband service can go to  providers’ websites to get information on performance, including speed  and latency, as well as information on availability and price. ISPs may  also provide tools for selecting the correct package based on use and  information on factors that affect broadband performance. ISPs told us  that consumers may also call and speak to a representative to determine  the best package for their needs, and existing customers may call for help  troubleshooting problems with their broadband service.
FCC’s 2010 Open Internet Order included a transparency rule intended to  provide consumers with more information on broadband performance.  The rule requires that fixed and mobile providers “publicly disclose  accurate information regarding the network management practices,  performance, and commercial terms of its broadband Internet access  services sufficient for consumers to make informed choices regarding use  of such services…” In response to concerns from ISPs regarding the burden associated with specific requirements, FCC did not require ISPs  to disclose specific information in a standardized format. Instead, the  2010 Open Internet Order stated that FCC expected disclosures to  include some or all of certain types of information, such as actual speed,  to be timely and prominently disclosed in plain language accessible to  consumers and to, at a minimum, be provided through a publicly  available, easily accessible website. As part of its 2014 Open Internet  NPRM, FCC proposed enhancing the transparency rule and requested  comment on a variety of options, some of which are discussed later in this  report. In the resulting February 2015 Open Internet Order, FCC adopted  enhancements to the transparency rule, and stated that ISPs are required  to disclose certain information under the transparency rule. Specifically,  FCC now states that disclosures of network performance should include  actual speed, latency, and packet loss, and disclosures should also  include network management practices applied to specific users or user  groups (for example, users in a particular service plan or geographic  location). FCC also requires ISPs to develop a mechanism for directly  notifying consumers if their individual use of a network will trigger a  network practice that is likely to have a significant impact on the  consumer’s use of the service.
Currently, ISPs’ disclosures vary with respect to length, content, and  where they are placed on ISPs’ websites. In addition, according to public  interest groups we spoke with, the complexity of this information and its  lack of standardization across ISPs can make it difficult for consumers to  find and use the information to compare broadband products and  services. Some groups also raised concerns that consumers have  difficulty understanding the information. However, some of the ISP and  trade associations we spoke with questioned the rationale for more  specific disclosure requirements, stating that consumers have access to  the information they need or that flexibility is important, given the  difficulties in explaining technical performance information to consumers.  In addition, some ISPs and a trade association argued in written  comments filed with the FCC that a standardized disclosure requiring  specific information from ISPs could overlook features that may be of  greater importance to consumers (such as nationwide wireless access) or  confuse consumers by providing highly technical information. However,  other ISPs were open to adopting a standardized label (as discussed  below), with one ISP stating in written comments that it might help  consumers make informed choices.
FCC’s Open Internet Advisory Committee’s Transparency Working Group  studied how ISPs’ present performance and pricing information and  issued a report in which it recommended that FCC promote a voluntary  labeling program to help consumers more easily compare and select  broadband service offerings. The working group suggested the program  be voluntary and result in a label that would include information on an  ISP’s upload and download speed, price, and usage restrictions.  However, the report also noted a number of complexities associated with  implementing a standardized label program, such as accounting for  variability in broadband speeds, as well as for factors beyond the ISP’s  control, and the potential for consumers to purchase more expensive  packages due to a lack of consumer education on what speeds they  need. FCC officials told us that they have considered standardized labels,  but developing a label that is easy to use, useful for consumers with  differing broadband needs, and relevant in a changing environment is  complicated. Researchers have noted that standardized performance  information could help third parties develop tools and information that  allow consumers to compare multiple broadband services. FCC  requested comment on the working group’s label proposal as part of its  2014 Open Internet NPRM. In its 2015 Open Internet Order, FCC  established a voluntary safe harbor for the format and nature of the  required disclosure to consumers. To take advantage of the safe harbor,  a broadband provider must provide a consumer-focused, stand-alone  disclosure. FCC, however, did not mandate the exact format of such  disclosure at this time. Instead, it directed its Consumer Advisory  Committee to develop and submit to the commission a voluntary  disclosure format that ISPs could use to ensure compliance with the  transparency rule no later than October 31, 2015.
Online third-party speed tests allow consumers to test the performance of  their broadband service by sending and receiving data between the  consumer’s device (such as a laptop or smart phone) and a test server.  For example, www.speedtest.net measures users’ broadband speed, and  www.pingtest.net tests other parameters and provides an overall grade  that includes information on which applications may be affected by the  user’s broadband performance characteristics. FCC also released an  application that allows users to test the speed, latency, and packet loss of  their mobile broadband.to help consumers understand their broadband performance. In some  cases, third parties have created sites that rank broadband providers  based on the results of speed-test data. For example, OOKLA’s  www.netindex.com uses data from its speed test and consumer surveys  to create country, state, and city-level rankings of fixed and mobile  broadband providers in terms of upload and download speed, as well as  other performance measures. This allows consumers to compare  providers based on their performance in the consumer’s local area.  RootMetrics.com provides similar data on mobile broadband, including  assessments of the mobile ISPs’ speed and reliability.
In addition, ISPs may provide speed test tools  However, the information from speed tests may not provide consumers  with the information they need to understand what factors are affecting  their broadband performance. Studies we reviewed and stakeholders we  spoke with noted that speed tests can be affected by a variety of factors,  which makes it difficult for consumers to identify what specific problem is  affecting their broadband performance. For example, researchers have  noted that online speed tests can be affected by the home equipment and  the number of users accessing the Internet in the home; however, these  tests may not enable consumers to discern that these factors are  affecting their speeds. In addition, speed tests can be limited in  reflecting the end-user’s actual experience because speed tests vary  methodologically, and different tests can provide differing results,  depending on the location of the speed test server. If a speed test server  is located outside the consumer’s ISP network, then the results could be  affected by congestion occurring on networks outside of the ISP’s control.  For example, when using third-party speed tests, a consumer’s traffic  may travel over networks that are not controlled by their ISP, and  congestion on those networks may have an effect on the speed test. On  the other hand, speed tests hosted by an ISP may only test the  performance of the ISP’s network, and thus do not capture congestion  that may be affecting a consumer’s ability to access content outside of  their ISP’s network.
Stakeholders we interviewed noted that consumers may experience  performance problems due to disputes between ISPs and content  providers regarding interconnection (interconnection disputes) that lead to  network congestion when consumers are accessing specific sites, and  these problems may not be reflected in speed tests. Congestion at  interconnection points has been identified as a problem contributing to  poor broadband performance, particularly with respect to streaming video  content. Such congestion may not be adequately captured by third-party  or ISP speed tests, since it may be limited to a specific interconnection  point not used by the speed test server. Some content providers, such as  Google, have provided alerts to consumers when Internet congestion is  affecting their broadband performance. However, information regarding  interconnection agreements can be subject to non-disclosure  agreements, and there have been several disputes regarding who is  responsible for addressing interconnection congestion. In June 2014,  FCC released a statement asserting that these interconnection disputes  can affect consumers’ broadband performance and that in the interest of  providing consumers with more transparency, FCC was obtaining copies  of interconnection agreements to further its understanding of  interconnection disputes. Separately, in the 2015 Open Internet Order,  FCC stated that it would hear interconnection disputes on a case-by-case  basis.
ISPs’ traffic management policies can also affect the results of speed  tests, making comparisons across providers difficult. In August 2014, a  public interest group announced it was filing a complaint against T- Mobile. According to the group, T-Mobile was slowing down the  broadband speeds provided to mobile broadband subscribers that had   however, T-Mobile did not slow down traffic  reached their usage limits; coming from speed test applications. Thus, if a T-Mobile customer whose  broadband was being slowed down accessed a speed test to try and  determine their broadband speed, their speed test would show a higher  speed than the customer was actually receiving. The public interest group  stated that this was a violation of the transparency rule, because  consumers were not receiving accurate information regarding network  performance. In November, FCC issued a press release stating that T- Mobile agreed to take several actions to ensure consumers were able to  obtain accurate measures of their speed, including providing a link to a  speed test that would reflect the speed reduction. There have also been  press reports that backbone networks’ traffic management policies have  affected speed test results, and researchers have noted that ISPs can  design their networks to provide better speed test results.
section_title: FCC Reports on Broadband Performance, but the Reports Are Not Targeted toward Consumers
paragraphs: Since 2011, FCC has published annual reports with broadband  performance results from its Measuring Broadband America (MBA)   The MBA program compares fixed ISPs’ average delivered  effort.download and upload speeds against their advertised speeds (for a 24- hour period, as well as for the peak usage time of 7 PM to 11 PM on  weeknights), and provides additional information on measurements of  latency and website loading time. FCC is working to expand its MBA  effort to include mobile broadband providers and, as part of that process,  has released an app for consumers to download and test their mobile  broadband speeds.
MBA developed out of a recommendation in the 2010 National  Broadband Plan to improve the availability of information for consumers  about their broadband service. At the time, there were concerns that ISPs  were not delivering their advertised speeds, and data were not available  to determine whether this was the case. FCC officials stated that MBA  allowed them to verify the accuracy of ISPs’ advertising claims. FCC  collaborated with industry and public interest stakeholders when  designing the MBA program. The sample population is drawn from  subscribers of 14 of the largest ISPs (which serve over 80 percent of the  residential marketplace) and consists of about 6,000 volunteers. Unlike  online speed tests that consumers may access from a variety of websites,  MBA only tests elements under the direct or indirect control of a  consumer’s ISP, from the consumer’s gateway—the modem or router  used by the consumer to access the Internet—to a nearby major Internet  gateway point. Thus, the results are not affected by congestion occurring  outside of an ISP’s network.
In 2014, FCC found that the ISPs included in the report were, on average,  delivering 101 percent of advertised download speeds during the peak  usage hours. FCC, ISPs, and public interest group representatives  stated that the transparency provided through MBA spurred competition  among providers and led to improved speeds, which benefit consumers.
In addition, ISPs that participate in the MBA effort can provide consumers  with their MBA results in order to comply with the transparency rule’s  disclosure requirements. Researchers have used raw data from MBA to  conduct in-depth analyses of broadband performance. FCC officials also  told us that it uses the information from the MBA report to inform its  rulemakings on broadband, such as raising the broadband speed for  providers to be eligible for the Connect America Fund, which supports  telephone and broadband service for rural residents.
FCC is also in the process of considering next steps for the program. As  noted above, in order to provide comparable results across ISPs, the  MBA initially only measured the ISP’s network. Thus, delays incurred in  the consumer’s home or in segments of the Internet outside an ISP’s  network were excluded from the results. For example, in the 2014 MBA  report, FCC noted that it excluded data from the report after discovering  some testing servers (and thus, any measurements taken using these  servers) were affected by congestion at interconnection points where  ISPs exchange traffic with Internet backbone providers. FCC acknowledged that consumers accessing services and content over  the affected paths would likely see a significant degradation in their  service, but FCC stated that it did not include the results from these  servers in its report because it aims to provide an analysis of average  network performance. Two public interest groups we interviewed raised  concerns that the MBA results were not reflecting the experience of  consumers whose broadband performance was negatively affected by  interconnection disputes. In the 2014 MBA report, FCC noted that while  MBA was initially focused on measuring broadband performance from the  consumer to the end of the service provider’s network, Internet services  and applications rely on a complex and variable arrangement of  interconnected networks to reach consumers. FCC stated that it is  exploring options to leverage its MBA effort to gain more insights into the  evolving performance of the Internet and provide better information to  consumers. For example, FCC officials told us that it is testing an  expansion of the MBA program to include measuring broadband  performance when a consumer is using streaming video services.
According to FCC, this congestion was caused by business disputes between ISPs and  a backbone provider. FCC did collect results from the affected servers and released the  data for use by academics and others examining such congestion issues.
Although the program initially was developed to provide consumers  access to broadband performance information, many of the various  stakeholders we interviewed, as well as FCC, generally agreed that most  consumers are unaware of the MBA report and do not rely on it to obtain  broadband performance information. MBA reports are highly technical  and include detailed appendices and raw data. FCC and some of the  ISPs and public interest groups we interviewed noted that information  collected through the MBA program may reach consumers through  secondary sources, such as ISPs’ advertising and transparency rule  disclosures, news reports, and tech blogs. In addition, as noted above,  FCC and many of the ISPs and public interest groups stated that the  program has spurred competition among providers and led to improved  speeds. However, ISPs’ offerings and performance can vary  geographically, and in the Open Internet comments we reviewed, two  stakeholders raised concerns that the MBA reports do not provide  consumers with information about broadband performance at the local  level. The National Broadband Plan recommended that FCC’s broadband  performance reporting effort should include “detailed information about  the actual performance of the country’s top broadband service providers  in different geographic markets (e.g., by county, city or [Metropolitan  Statistical Area]).” In the 2014 MBA report, FCC stated that the data are  not collected in a way that permits meaningful conclusions about  broadband performance at the local level and that this limitation was a  result of cost issues and the finite number of measurement devices that  could be deployed over the course of the project. A report on the MBA  effort that was commissioned by FCC and conducted by the North  Carolina State University’s Institute for Advanced Analytics primarily  reviewed MBA’s methodology, but also made some suggestions for  making the data more consumer-friendly, including the use of a tool  developed by the group to help consumers compare ISPs’ performance in  their states (when enough data were available to allow for statistically  valid comparisons).
section_title: FCC Has Taken Steps to Evaluate Its Efforts to Inform Consumers, but Could Enhance Its Performance Information and Measures
paragraphs: FCC has taken steps to determine whether its efforts to provide  consumers with broadband performance information—such as the  transparency rule and MBA program—are effective and meeting  consumers’ needs. Specifically, FCC has reviewed consumer complaints  and requested stakeholder comments on this issue in several recent  rulemaking proceedings. However, FCC’s ability to evaluate its efforts is  limited by a lack of useful performance information. In addition, FCC  established strategic objectives related to informing consumers in its  strategic plan, but lacks performance goals and measures to monitor the  effectiveness of its efforts to provide consumers with broadband  performance information. Additional actions—such as conducting or  commissioning objective consumer research and establishing  performance goals and measures—could help FCC evaluate whether its  efforts to provide consumers with broadband performance information are  effective and meeting consumers’ needs.
subsections: section_title: FCC Has Requested Stakeholder Comments in Several Recent Rulemakings and Reviewed Consumer Complaints
paragraphs: FCC has requested stakeholder comments in rulemaking proceedings  and reviewed consumer complaints filed with the commission to  determine whether its efforts to provide consumers with broadband  performance information—such as its transparency rule and MBA  program—are effective and meeting consumers’ needs. According to  FCC officials, FCC relies heavily on stakeholder comments to inform and  explain decisions underlying the commission’s orders; however, FCC may  also request stakeholders’ comments to develop a body of knowledge on  issues of interest to the commission. In several recent rulemaking  proceedings, FCC has requested comments from stakeholders on (1)  consumers’ broadband performance information needs, (2) best practices  for presenting and displaying performance information to consumers, and  (3) potential improvements to FCC’s efforts to provide consumers with  broadband performance information, among other things. For instance,  FCC officials told us that stakeholder comments were used as part of  FCC’s determination to enhance its transparency rule. Table 3  summarizes some of the consumer-related issues and questions FCC  has raised and requested stakeholder comments on in recent years.
According to FCC officials, FCC also relies on consumer complaints filed  with the commission to determine whether its efforts to provide  consumers with broadband performance information are effective and  meeting consumers’ needs. For example, in the 2014 Open Internet  NPRM, FCC stated that it had received and analyzed hundreds of  consumer complaints related to ISPs’ disclosures under the transparency  rule and that these complaints were part of the agency’s rationale for  tentatively concluding the rule should be strengthened. In addition,  officials told us that consumer complaints have indicated that service  reliability is now as important to consumers as speed due to the  popularity of video-streaming services, such as Netflix, and that as a  result, they are exploring ways to further incorporate service reliability  information into the MBA program. They also told us that based on  consumer complaints, they knew that consumers could not easily  access—and in some cases were not provided—certain performance  information (e.g., ISPs’ network management practices, data limits, etc.),  and FCC tried to address this issue through the transparency rule.
section_title: FCC’s Ability to Evaluate Its Efforts Is Limited by a Lack of Useful Performance Information and Relevant Performance Goals and Measures
subsections: section_title: FCC Obtains Inconsistent and Incomplete Performance Information from Stakeholder Comments and Consumer Complaints
paragraphs: We have previously found that leading organizations that have  progressed toward results-oriented management use performance  information as a basis for decision-making and that the usefulness of  performance information can be affected by its completeness, accuracy,  consistency, validity, and credibility, among other things. For example,  performance information can focus on various dimensions of  performance, such as outcomes, outputs, quality, or customer  satisfaction, and can help agencies monitor their progress in meeting  programmatic and operational goals. Although stakeholder comments  and consumer complaints can provide FCC with valuable insights on  topics of interest to the commission, the information FCC obtains from  these sources lacks some of the characteristics of useful performance  information, such as consistency and completeness.
Our review of selected stakeholders’ Open Internet comments found that  they generally provided inconsistent views regarding the effectiveness of  FCC’s efforts to provide consumers with broadband performance  information. For instance, some public interest groups and content  providers argued that FCC’s transparency rule was not providing  consumers the information they need to make informed decisions  regarding broadband services. In their view, FCC should have required  ISPs to provide additional technical information—such as detailed  information on network congestion—under the transparency rule.  However, ISPs countered that this level of technical detail would be  confusing to consumers and that the transparency rule had been effective  in providing consumers with the information they need. For example, Cox  Communications, an ISP, stated that, “most of the expanded disclosure  proposals included in the NPRM would impose burdensome obligations  that are highly unlikely to prove useful to consumers and would only  degrade the overall effectiveness of the transparency rule.” FCC has  acknowledged that it has not received consistent information in response  to its requests for comments. For instance, in its 2010 Open Internet  Order, FCC states that despite broad agreement that broadband  providers should disclose information sufficient to enable end users and  content providers to understand the capabilities of broadband services,  commenters disagree about the appropriate level of detail required to  achieve this goal. As a result, FCC decided to continue to allow ISPs  flexibility in the implementation of the transparency rule while providing  guidance regarding acceptable disclosure methods. Similarly, in its 2015  Open Internet Order, FCC noted that despite its efforts to seek comment  on this issue, the record was lacking with respect to specific details on  how such a disclosure should be formatted. Thus, as previously  mentioned, FCC asked its Consumer Advisory Committee to propose a  disclosure format.
As noted above, in the 2014 Open Internet NPRM, FCC cited consumer  complaints as a basis for tentatively concluding that the transparency rule  should be strengthened. However, in the NPRM FCC also acknowledged  that its analysis of consumer complaints provided limited insight into the  issues that consumers were experiencing. Specifically, FCC stated that  since the transparency rule took effect, it had received a “significant  number” of consumer complaints about provider speeds, charges, and  other practices that the rule was designed to disclose. However, FCC  also stated that in some cases, it is difficult to discern from the complaints  “whether the consumer’s frustration is with slow speeds or high prices  generally, or instead with how the service as actually provided differs from  what the provider has advertised.” Moreover, because consumer  complaints reflect the perspective of a non-representative sample of  consumers—i.e., those who have experienced broadband issues and  have chosen to report those issues to the FCC—they do not fully capture  the effectiveness of FCC’s efforts for a broader range of consumers.  Consequently, several Open Internet comments we reviewed filed by  ISPs and industry associations questioned FCC’s reliance on complaints  as a basis for decision-making.
In addition, we have previously noted concerns with FCC’s complaint  process. In 2009, we found that many wireless phone service  consumers were not aware that they could file complaints with FCC when  they encountered problems. We also found that FCC did not complete in- depth analyses of consumer complaints and lacked goals and measures  that clearly identified the intended outcomes of its complaint-processing  efforts. As a result, we noted that FCC may not be aware (1) of emerging  trends in consumer problems, (2) if specific rules are being violated, or (3)  if additional rules are needed to protect consumers. We made a number  of recommendations to help address these issues, and FCC has  indicated that it is in the process of reforming the agency’s complaint  process. For example, in January 2015, FCC launched a new online  consumer-help center. According to FCC, the online center will make it  easier for consumers to file complaints with FCC, improve  communications between consumers and FCC representatives, and help  streamline FCC’s process for synthesizing and analyzing trends in  consumer complaints. While these steps may enhance FCC’s ability to  assess the effectiveness of its efforts among the subset of consumers  who file complaints, they will not provide FCC information regarding the  effectiveness of its efforts for a broader range of consumers.
Without consistent and complete information regarding the effectiveness  of its efforts to provide consumers with broadband performance  information, FCC’s ability to evaluate its efforts and make performance- based decisions about these efforts is limited. In addition, FCC may find it  difficult to convince stakeholders that enhancements to its transparency  rule and MBA program are warranted. As we reviewed industry groups’  Open Internet comments, we found many stating that FCC did not provide  adequate evidence that ISPs’ disclosures under the transparency rule  were ineffective and that enhancements or changes to the rule were  needed. For example, Cisco, an equipment provider, stated that FCC’s  concerns regarding the accuracy of information some consumers  received under the transparency rule do not imply that additional  disclosures are called for, but rather that the commission may need to  enforce the requirements instead.
In contrast to the approaches FCC typically relies on to evaluate the  effectiveness of its efforts to provide consumers with broadband  performance information, other government entities have undertaken  extensive, qualitative, and quantitative consumer research to evaluate the  effectiveness of their efforts to inform consumers and ensure that  consumers’ information needs are being met. For example:
Ofcom—the United Kingdom’s (UK) communications regulatory  agency—has conducted qualitative and quantitative research to  determine consumers’ use and understanding of ISPs’ traffic  management information and the effectiveness of a standardized form  that all major UK ISPs voluntarily use to publish their traffic  management information. For instance, in 2013, Ofcom  commissioned a study that involved qualitative workshops with 135  participants and a representative survey of UK broadband consumers.  The study found that consumers had problems understanding some  aspects of the standardized form and suggested several potential  improvements that could make the information more understandable  to consumers, such as adding a summary to the form explaining the  relevance of traffic management policies to broadband products.
The U.S. Consumer Financial Protection Bureau (CFPB) conducted  qualitative and quantitative consumer research when revamping  mortgage disclosure forms. The CFPB tested the prototype of the  revised disclosure form through 10 rounds of qualitative testing in nine  cities. After issuing the revised forms the CFPB also conducted a  large-scale quantitative validation test—that included 858 consumers  in 20 locations—to compare the revised forms with the disclosures  that were currently in use.
Some stakeholders we spoke with and whose comments we reviewed  also suggested that FCC should undertake consumer research to inform  and evaluate its efforts to provide consumers with broadband  performance information. In 2009, the Federal Trade Commission filed  comments with FCC stating that research has shown that well-designed  standardized disclosures can improve consumer understanding and  facilitate competition; however, to be most effective, these disclosures  should be developed and revised based on controlled, quantitative,  objective tests of consumer understanding. In addition, representatives  from one industry association we spoke with stated that FCC’s analytical  framework, which largely consists of requesting and reviewing  stakeholder comments, is not consumer-centric and may not be the most  productive process for obtaining an accurate representation of  consumers’ information needs. These representatives noted that given  the time, effort, and money ISPs expend to collect broadband  performance data for FCC’s MBA program, it is important to make sure  that the data are presented in a way that is meaningful to consumers. In  their view, FCC should also be conducting consumer research, such as  focus groups, to make sure that its MBA effort produces a report that  consumers can understand. According to FCC officials, FCC’s goal is to  improve the accessibility, accuracy, and relevance of the broadband  performance information available to consumers. Because consumers’  decision-making processes are complex and can vary, FCC has not  attempted to formally evaluate whether its efforts to provide consumers  with broadband performance information are meeting consumers’ needs.  Officials also told us that they do not believe this type of research has  been conducted outside of FCC.
section_title: FCC Lacks Performance Goals and Measures to Monitor Its Efforts to Inform Consumers
paragraphs: FCC’s current strategic plan includes strategic objectives related to  informing consumers about broadband networks—reflecting its position  that informed consumers help facilitate competition in the broadband  market. Specifically, FCC includes “ensure effective policies are in place  to promote and protect competition” and “take action where competition is  not sufficient to protect the public interest, including ensuring that  consumers remain informed” as two of its 17 strategic objectives and  states that it will continue to engage consumers though its outreach and  education initiatives to facilitate informed choice in the competitive  communications marketplace. However, FCC does not include any  performance goals under these strategic objectives that define desired  outcomes for its efforts to provide consumers with broadband  performance information, such as its transparency rule and MBA  program. Consequently, FCC also lacks relevant performance measures  under these strategic objectives that would allow it to monitor the impact  and effectiveness of its efforts. We have previously reported that effective  performance goals and measures possess certain characteristics. These  include the following: 1) performance goals should clearly define the  organization’s desired outcomes, and 2) each performance goal should  have a few performance measures that clearly tie back to the goal, cover  key performance dimensions, and take different priorities into account.Without performance goals and measures that allow FCC to monitor its  progress on its goals related to providing consumers with broadband  performance information, FCC has limited assurance that its efforts to  provide consumers with broadband performance information are effective  and meeting consumers’ needs. In addition, FCC cannot demonstrate  that it is achieving its objectives to ensure consumers are informed as  envisioned in its strategic plan.
section_title: Conclusions
paragraphs: In recent years, the broadband market has undergone significant  changes. More Americans are accessing broadband and relying on their  broadband service for an increasingly diverse range of uses, including  education, medicine, public safety, and entertainment. As broadband  traffic grows, consumers’ broadband performance issues—such as  slower than expected speeds and disruptions to video-streaming  services—may simultaneously become more prevalent and more critical  to resolve. However, broadband performance can be affected by a  number of complicated, technical factors that are not readily apparent to  most consumers.
FCC has maintained that informed consumers play an important role in  facilitating competition in the broadband market and has undertaken  efforts to provide consumers with broadband performance information,  such as the transparency rule and the MBA program. However, FCC  lacks useful performance information and relevant performance goals and  measures to evaluate the effectiveness of these efforts and monitor their  impact. As a result, FCC cannot be assured that its efforts to provide  consumers with broadband performance information are effective and  meeting consumers’ needs, and its ability to make performance-based  decisions will remain limited. Moreover, FCC may find it difficult to  convince stakeholders that its enhancements to the transparency rule are  warranted and address their concerns that FCC has not provided  adequate evidence to support these enhancements.
section_title: Recommendations for Executive Action
paragraphs: To help FCC determine whether its efforts to provide consumers with  broadband performance information are effective and meeting  consumers’ needs, and whether additional efforts—such as a  standardized label suggested by FCC’s transparency working group— could benefit consumers, FCC should take the following two actions:  1.  conduct or commission research on the effectiveness of FCC’s efforts  to provide consumers with broadband performance information and  make the results of this research publicly available, and  2.  establish performance goals and measures under the agency’s  relevant strategic objectives that allow it to monitor and report on the  impact and effectiveness of its efforts.
section_title: Agency Comments
paragraphs: We provided a draft of this report to FCC for review and comment. In its  written comments, reproduced in appendix II, FCC generally concurred  with our recommendations. FCC described its various consumer outreach  and education efforts and agreed that extensive research would be  helpful in evaluating the effectiveness of its efforts to inform consumers.  FCC stated that it will evaluate consumer research on broadband  performance and move expeditiously to take action. FCC also stated that  it would use consumer research, consumer comments, and in-person  feedback at consumer-related events to establish performance measures  that will allow it to monitor and report on its progress. FCC also provided  an appendix with details about its outreach efforts as well as technical  comments that were incorporated as appropriate.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the Chairman of the  Federal Communications Commission and the appropriate congressional  committees. In addition, this report will be available at no charge on  GAO’s website at http://www.gao.gov.
If you or your staff has any questions about this report, please contact me  at (202) 512-2834 or goldsteinm@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last   page of this report. GAO staff who made key contributions to this report  are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The objectives of this report were to examine (1) what information is  currently available to consumers regarding broadband performance and  the limitations of this information, if any, and (2) how does the Federal  Communications Commission (FCC) evaluate its efforts to provide  consumers with broadband performance information?
To assess what information is currently available to consumers regarding  broadband performance and any limitations of this information, we  reviewed prior GAO reports on this issue, relevant FCC rulemaking  proceedings, including its 2014 Open Internet proceeding, FCC’s  transparency rule and Measuring Broadband America reports, Internet  service providers’ (ISP) websites, information provided by companies,  including Netflix and Google, and third-party sites that provide broadband  performance information to consumers. We also conducted a literature  review to identify articles and studies relevant to broadband performance  measurement and disclosure of broadband performance information to  consumers. We used key-word searches to identify peer-reviewed articles  and other work in relevant databases and reviewed the websites of  researchers identified by FCC as conducting work relevant to our review.  We also interviewed representatives from FCC, public-interest and  consumer-advocacy groups, industry associations, content or equipment  providers, and broadband providers. We identified stakeholders to  interview based on our review of comments filed in FCC’s 2014 Open  Internet and related proceedings, as well as based on recommendations  from other organizations we interviewed. Finally, we reviewed a  judgmental sample of comments filed in FCC’s 2014 Open Internet  proceeding. In this proceeding, FCC requested comment on the  effectiveness of its efforts to provide broadband performance information  to consumers, including its transparency rule and its Measuring  Broadband America program, and its proposed enhancements to these  efforts. We ran key-word searches that included terms related to  transparency and disclosure for consumers to identify relevant comments,  and selected comments from a variety of stakeholders, including public  interest groups, ISPs, content providers, trade associations, and  consumer representatives. We reviewed these comments to identify  statements regarding the effectiveness of FCC’s efforts to provide  broadband performance information to consumers, including comments  supporting and opposing FCC’s proposed enhancements to these efforts.
To determine how FCC evaluates its efforts to provide consumers with  broadband performance information, we interviewed FCC officials  regarding their efforts to provide consumers with broadband performance  information and evaluate and monitor the effectiveness of these efforts  and reviewed relevant FCC documents, including its strategic and  performance plans, consumer surveys, and resources available to  consumers on FCC’s website. We also reviewed FCC’s requests for  comment on consumers’ information needs and the effectiveness of  FCC’s efforts to provide consumers with broadband performance  information, as well as selected comments filed in response to these  proceedings (as described above). To determine what additional actions,  if any, FCC could take to evaluate its efforts to provide consumers with  broadband performance information, we interviewed representatives from  the stakeholder groups identified above, as well as officials from the  Federal Trade Commission and National Telecommunications and  Information Administration regarding their efforts to inform consumers  about broadband issues and to evaluate the effectiveness of those  efforts. In addition, we reviewed reports and studies on consumer  disclosures conducted by FCC’s Transparency Working Group; foreign  telecommunications-regulatory bodies, including those in the United  Kingdom and European Union; and other federal agencies, such as the  Consumer Financial Protection Bureau. We also reviewed prior GAO  reports on FCC and results-oriented management and compared FCC’s  use of performance information and measures to performance  management best practices outlined in previous GAO work.
We conducted this performance audit from July 2014 to April 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Federal Communications Commission
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
paragraphs: Mark L. Goldstein, (202) 512-2834 or goldsteinm@gao.gov.
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Teresa Anderson, Assistant  Director; Steve Brown; Crystal Huggins; Bert Japikse; Sara Ann  Moessbauer; Jaclyn Nidoh; Josh Ormond; Cheryl Peterson; Amy  Rosewarne; and Hai Tran made key contributions to this report.