Broadband Performance: Additional Actions Could Help FCC Evaluate Its Efforts to Inform Consumers
Released: 2015-05-15T00:00:00

Letter

Background

The term “broadband” commonly refers to Internet access that is high speed and provides an “always-on” connection, so users do not have to reestablish a connection each time they access the Internet. Broadband speeds are described in download and upload capabilities, and are defined as the maximum rates at which data are delivered over the Internet. A higher speed indicates a faster information delivery rate. For example, a 10 megabits per second (Mbps) service should deliver ten times as much data as a 1 Mbps service in the same period of time.

Internet service providers (ISP) have marketed and competed with one another on the speed of their various broadband plans. Advances in technology, such as the use of fiber (described below) and new wireless technologies, have allowed ISPs to offer increasingly faster speeds that support new services and applications, such as streaming video. FCC takes those advances into account as part of its required inquiry into whether “advanced telecommunications capability” is being deployed to all Americans in a reasonable and timely fashion. Pursuant to this inquiry, FCC recently updated the minimum speed benchmark it uses to measure the availability of advanced telecommunications capability. In addition to speed, a number of factors can be used to evaluate broadband performance. See table 1.

Consumers generally subscribe to broadband in two ways:

Fixed: In-home fixed Internet plans are often sold as a monthly subscription by cable television or telephone companies. Service from cable television companies is generally provided through the same coaxial cables that deliver television programming. Service from telephone companies is generally provided through traditional copper telephone lines—commonly referred to as digital subscriber line (DSL) service—or fiber-optic lines, which convert electrical signals carrying data into light and send the light through glass fibers. These network technologies generally have higher speeds than mobile networks. Consumers can connect a variety of devices to in-home fixed networks through a wired connection or wireless Wi-Fi connection (see fig. 1). Consumers are increasingly using the Internet to supplement or replace their use of traditional services, such as traditional telephone and cable TV service.

Mobile: Traditionally, mobile providers sold access to the Internet as an add-on to mobile telephone service plans that may or may not include a multiyear contract. Mobile service is provided through cell tower coverage with data sharing through radio spectrum. A number of devices may connect to mobile broadband networks, such as smart phones, tablets, and mobile devices that enable laptops to connect to mobile networks (see fig. 2).

The 2014 American Customer Satisfaction Index (ACSI)with respect to fixed broadband service, consumer satisfaction regarding data speeds and video streaming increased from 2013 to 2014. However, ACSI also found that during this time period, customer satisfaction related to service interruptions and outages, speed and service reliability, and peak evening-hour performance declined. With respect to mobile broadband providers, ACSI reported that customer satisfaction related to speed for downloading data and streaming content increased from 2013 reported that to 2014; however, ACSI also reported that the customer satisfaction scores indicated that improvements were still needed in speed and reliability.

Broadband performance problems can manifest in a number of ways, such as slow loading of content, lost connections, and pauses in streaming video and audio. FCC, ISPs, content providers, and researchers have noted that a variety of factors can affect broadband performance, such as accessing the Internet during peak usage times, outdated equipment in the home, or multiple users in the same household accessing the Internet simultaneously. In addition, mobile broadband performance will vary based on the location of the phone or device. Running multiple applications at once or having multiple users in a household may require greater speeds. Different types of Internet applications use varying amounts of data and require different minimum speeds to operate properly (see table 2). For example, email and web- browsing use small amounts of data, while watching movies uses larger amounts of data. Use of data-intensive broadband applications is projected to grow. FCC has noted that consumers are increasingly using broadband applications that require large amounts of data, such as streaming video content, which can require more consistency of service and be more visibly affected by technical problems.

In addition, congestion at various points on the network can affect a user’s broadband performance. In order to connect their customers to the Internet, ISPs link with other networks, including “backbone providers” that move Internet traffic long distances and interconnect with other networks. Companies may use a number of options to deliver their content across the Internet, such as connecting with a backbone provider (directly, or through an ISP), connecting directly to the consumer’s ISP, or paying a content delivery network to deliver their content to ISPs. See figure 3 for a simplified illustration of how content moves from websites and applications to consumers. Congestion occurring within the content provider’s network, at points of interconnection between the networks, or within the ISP’s network can all affect a consumer’s broadband performance. ISPs’ traffic management policies can also affect a consumer’s broadband performance. This issue is discussed later in the report.

FCC has primary responsibility for regulating broadband. In response to a requirement in the American Recovery and Reinvestment Act of 2009, FCC staff developed The National Broadband Plan, which includes a series of recommendations aimed at ensuring all Americans have access to broadband. In addition, section 706 of the Telecommunications Act of 1996, as amended, directs FCC “to encourage the deployment on a reasonable and timely basis of advanced telecommunications capability to all Americans” through a variety of measures, including measures that promote competition in the local telecommunications market. FCC has repeatedly noted the connection between informed consumers and increased competition, noting that the latter is an important part of promoting broadband deployment.

Preserving the Open Internet, Report and Order, 25 FCC Rcd. 17905 (2010), aff’d in part, vacated and remanded in part sub nom, Verizon v. FCC, 740 F.3d. 623 (D.C. Cir. 2014). incentive and ability to act in ways that limit Internet openness (e.g., by blocking or discriminating against unaffiliated applications and services).

Although Consumers Can Access a Number of Sources of Information on Broadband Performance, the Information Is Limited

Consumers can access information on broadband performance from a number of sources, including their ISPs, third-party websites, and FCC reports. FCC adopted a transparency rule requiring ISPs to provide consumers with information on their services’ performance characteristics. However, the information available to consumers is not standardized, and some stakeholders stated that this lack of standardization makes it difficult for consumers to compare broadband services. In addition, the information available through speed tests may not accurately reflect the end-user’s actual experience. FCC has taken steps to measure and report on the extent to which ISPs are providing the speeds they advertise. However, the reports are not targeted toward consumers, and stakeholders stated that consumers might not be aware of the information.

Performance Information from ISPs and Third Parties Is Not Standardized and May Not Accurately Reflect the End-User’s Actual Experience

Consumers interested in purchasing broadband service can go to providers’ websites to get information on performance, including speed and latency, as well as information on availability and price. ISPs may also provide tools for selecting the correct package based on use and information on factors that affect broadband performance. ISPs told us that consumers may also call and speak to a representative to determine the best package for their needs, and existing customers may call for help troubleshooting problems with their broadband service.

FCC’s 2010 Open Internet Order included a transparency rule intended to provide consumers with more information on broadband performance. The rule requires that fixed and mobile providers “publicly disclose accurate information regarding the network management practices, performance, and commercial terms of its broadband Internet access services sufficient for consumers to make informed choices regarding use of such services…” In response to concerns from ISPs regarding the burden associated with specific requirements, FCC did not require ISPs to disclose specific information in a standardized format. Instead, the 2010 Open Internet Order stated that FCC expected disclosures to include some or all of certain types of information, such as actual speed, to be timely and prominently disclosed in plain language accessible to consumers and to, at a minimum, be provided through a publicly available, easily accessible website. As part of its 2014 Open Internet NPRM, FCC proposed enhancing the transparency rule and requested comment on a variety of options, some of which are discussed later in this report. In the resulting February 2015 Open Internet Order, FCC adopted enhancements to the transparency rule, and stated that ISPs are required to disclose certain information under the transparency rule. Specifically, FCC now states that disclosures of network performance should include actual speed, latency, and packet loss, and disclosures should also include network management practices applied to specific users or user groups (for example, users in a particular service plan or geographic location). FCC also requires ISPs to develop a mechanism for directly notifying consumers if their individual use of a network will trigger a network practice that is likely to have a significant impact on the consumer’s use of the service.

Currently, ISPs’ disclosures vary with respect to length, content, and where they are placed on ISPs’ websites. In addition, according to public interest groups we spoke with, the complexity of this information and its lack of standardization across ISPs can make it difficult for consumers to find and use the information to compare broadband products and services. Some groups also raised concerns that consumers have difficulty understanding the information. However, some of the ISP and trade associations we spoke with questioned the rationale for more specific disclosure requirements, stating that consumers have access to the information they need or that flexibility is important, given the difficulties in explaining technical performance information to consumers. In addition, some ISPs and a trade association argued in written comments filed with the FCC that a standardized disclosure requiring specific information from ISPs could overlook features that may be of greater importance to consumers (such as nationwide wireless access) or confuse consumers by providing highly technical information. However, other ISPs were open to adopting a standardized label (as discussed below), with one ISP stating in written comments that it might help consumers make informed choices.

FCC’s Open Internet Advisory Committee’s Transparency Working Group studied how ISPs’ present performance and pricing information and issued a report in which it recommended that FCC promote a voluntary labeling program to help consumers more easily compare and select broadband service offerings. The working group suggested the program be voluntary and result in a label that would include information on an ISP’s upload and download speed, price, and usage restrictions. However, the report also noted a number of complexities associated with implementing a standardized label program, such as accounting for variability in broadband speeds, as well as for factors beyond the ISP’s control, and the potential for consumers to purchase more expensive packages due to a lack of consumer education on what speeds they need. FCC officials told us that they have considered standardized labels, but developing a label that is easy to use, useful for consumers with differing broadband needs, and relevant in a changing environment is complicated. Researchers have noted that standardized performance information could help third parties develop tools and information that allow consumers to compare multiple broadband services. FCC requested comment on the working group’s label proposal as part of its 2014 Open Internet NPRM. In its 2015 Open Internet Order, FCC established a voluntary safe harbor for the format and nature of the required disclosure to consumers. To take advantage of the safe harbor, a broadband provider must provide a consumer-focused, stand-alone disclosure. FCC, however, did not mandate the exact format of such disclosure at this time. Instead, it directed its Consumer Advisory Committee to develop and submit to the commission a voluntary disclosure format that ISPs could use to ensure compliance with the transparency rule no later than October 31, 2015.

Online third-party speed tests allow consumers to test the performance of their broadband service by sending and receiving data between the consumer’s device (such as a laptop or smart phone) and a test server. For example, www.speedtest.net measures users’ broadband speed, and www.pingtest.net tests other parameters and provides an overall grade that includes information on which applications may be affected by the user’s broadband performance characteristics. FCC also released an application that allows users to test the speed, latency, and packet loss of their mobile broadband.to help consumers understand their broadband performance. In some cases, third parties have created sites that rank broadband providers based on the results of speed-test data. For example, OOKLA’s www.netindex.com uses data from its speed test and consumer surveys to create country, state, and city-level rankings of fixed and mobile broadband providers in terms of upload and download speed, as well as other performance measures. This allows consumers to compare providers based on their performance in the consumer’s local area. RootMetrics.com provides similar data on mobile broadband, including assessments of the mobile ISPs’ speed and reliability.

In addition, ISPs may provide speed test tools However, the information from speed tests may not provide consumers with the information they need to understand what factors are affecting their broadband performance. Studies we reviewed and stakeholders we spoke with noted that speed tests can be affected by a variety of factors, which makes it difficult for consumers to identify what specific problem is affecting their broadband performance. For example, researchers have noted that online speed tests can be affected by the home equipment and the number of users accessing the Internet in the home; however, these tests may not enable consumers to discern that these factors are affecting their speeds. In addition, speed tests can be limited in reflecting the end-user’s actual experience because speed tests vary methodologically, and different tests can provide differing results, depending on the location of the speed test server. If a speed test server is located outside the consumer’s ISP network, then the results could be affected by congestion occurring on networks outside of the ISP’s control. For example, when using third-party speed tests, a consumer’s traffic may travel over networks that are not controlled by their ISP, and congestion on those networks may have an effect on the speed test. On the other hand, speed tests hosted by an ISP may only test the performance of the ISP’s network, and thus do not capture congestion that may be affecting a consumer’s ability to access content outside of their ISP’s network.

Stakeholders we interviewed noted that consumers may experience performance problems due to disputes between ISPs and content providers regarding interconnection (interconnection disputes) that lead to network congestion when consumers are accessing specific sites, and these problems may not be reflected in speed tests. Congestion at interconnection points has been identified as a problem contributing to poor broadband performance, particularly with respect to streaming video content. Such congestion may not be adequately captured by third-party or ISP speed tests, since it may be limited to a specific interconnection point not used by the speed test server. Some content providers, such as Google, have provided alerts to consumers when Internet congestion is affecting their broadband performance. However, information regarding interconnection agreements can be subject to non-disclosure agreements, and there have been several disputes regarding who is responsible for addressing interconnection congestion. In June 2014, FCC released a statement asserting that these interconnection disputes can affect consumers’ broadband performance and that in the interest of providing consumers with more transparency, FCC was obtaining copies of interconnection agreements to further its understanding of interconnection disputes. Separately, in the 2015 Open Internet Order, FCC stated that it would hear interconnection disputes on a case-by-case basis.

ISPs’ traffic management policies can also affect the results of speed tests, making comparisons across providers difficult. In August 2014, a public interest group announced it was filing a complaint against T- Mobile. According to the group, T-Mobile was slowing down the broadband speeds provided to mobile broadband subscribers that had however, T-Mobile did not slow down traffic reached their usage limits; coming from speed test applications. Thus, if a T-Mobile customer whose broadband was being slowed down accessed a speed test to try and determine their broadband speed, their speed test would show a higher speed than the customer was actually receiving. The public interest group stated that this was a violation of the transparency rule, because consumers were not receiving accurate information regarding network performance. In November, FCC issued a press release stating that T- Mobile agreed to take several actions to ensure consumers were able to obtain accurate measures of their speed, including providing a link to a speed test that would reflect the speed reduction. There have also been press reports that backbone networks’ traffic management policies have affected speed test results, and researchers have noted that ISPs can design their networks to provide better speed test results.

FCC Reports on Broadband Performance, but the Reports Are Not Targeted toward Consumers

Since 2011, FCC has published annual reports with broadband performance results from its Measuring Broadband America (MBA) The MBA program compares fixed ISPs’ average delivered effort.download and upload speeds against their advertised speeds (for a 24- hour period, as well as for the peak usage time of 7 PM to 11 PM on weeknights), and provides additional information on measurements of latency and website loading time. FCC is working to expand its MBA effort to include mobile broadband providers and, as part of that process, has released an app for consumers to download and test their mobile broadband speeds.

MBA developed out of a recommendation in the 2010 National Broadband Plan to improve the availability of information for consumers about their broadband service. At the time, there were concerns that ISPs were not delivering their advertised speeds, and data were not available to determine whether this was the case. FCC officials stated that MBA allowed them to verify the accuracy of ISPs’ advertising claims. FCC collaborated with industry and public interest stakeholders when designing the MBA program. The sample population is drawn from subscribers of 14 of the largest ISPs (which serve over 80 percent of the residential marketplace) and consists of about 6,000 volunteers. Unlike online speed tests that consumers may access from a variety of websites, MBA only tests elements under the direct or indirect control of a consumer’s ISP, from the consumer’s gateway—the modem or router used by the consumer to access the Internet—to a nearby major Internet gateway point. Thus, the results are not affected by congestion occurring outside of an ISP’s network.

In 2014, FCC found that the ISPs included in the report were, on average, delivering 101 percent of advertised download speeds during the peak usage hours. FCC, ISPs, and public interest group representatives stated that the transparency provided through MBA spurred competition among providers and led to improved speeds, which benefit consumers.

In addition, ISPs that participate in the MBA effort can provide consumers with their MBA results in order to comply with the transparency rule’s disclosure requirements. Researchers have used raw data from MBA to conduct in-depth analyses of broadband performance. FCC officials also told us that it uses the information from the MBA report to inform its rulemakings on broadband, such as raising the broadband speed for providers to be eligible for the Connect America Fund, which supports telephone and broadband service for rural residents.

FCC is also in the process of considering next steps for the program. As noted above, in order to provide comparable results across ISPs, the MBA initially only measured the ISP’s network. Thus, delays incurred in the consumer’s home or in segments of the Internet outside an ISP’s network were excluded from the results. For example, in the 2014 MBA report, FCC noted that it excluded data from the report after discovering some testing servers (and thus, any measurements taken using these servers) were affected by congestion at interconnection points where ISPs exchange traffic with Internet backbone providers. FCC acknowledged that consumers accessing services and content over the affected paths would likely see a significant degradation in their service, but FCC stated that it did not include the results from these servers in its report because it aims to provide an analysis of average network performance. Two public interest groups we interviewed raised concerns that the MBA results were not reflecting the experience of consumers whose broadband performance was negatively affected by interconnection disputes. In the 2014 MBA report, FCC noted that while MBA was initially focused on measuring broadband performance from the consumer to the end of the service provider’s network, Internet services and applications rely on a complex and variable arrangement of interconnected networks to reach consumers. FCC stated that it is exploring options to leverage its MBA effort to gain more insights into the evolving performance of the Internet and provide better information to consumers. For example, FCC officials told us that it is testing an expansion of the MBA program to include measuring broadband performance when a consumer is using streaming video services.

According to FCC, this congestion was caused by business disputes between ISPs and a backbone provider. FCC did collect results from the affected servers and released the data for use by academics and others examining such congestion issues.

Although the program initially was developed to provide consumers access to broadband performance information, many of the various stakeholders we interviewed, as well as FCC, generally agreed that most consumers are unaware of the MBA report and do not rely on it to obtain broadband performance information. MBA reports are highly technical and include detailed appendices and raw data. FCC and some of the ISPs and public interest groups we interviewed noted that information collected through the MBA program may reach consumers through secondary sources, such as ISPs’ advertising and transparency rule disclosures, news reports, and tech blogs. In addition, as noted above, FCC and many of the ISPs and public interest groups stated that the program has spurred competition among providers and led to improved speeds. However, ISPs’ offerings and performance can vary geographically, and in the Open Internet comments we reviewed, two stakeholders raised concerns that the MBA reports do not provide consumers with information about broadband performance at the local level. The National Broadband Plan recommended that FCC’s broadband performance reporting effort should include “detailed information about the actual performance of the country’s top broadband service providers in different geographic markets (e.g., by county, city or [Metropolitan Statistical Area]).” In the 2014 MBA report, FCC stated that the data are not collected in a way that permits meaningful conclusions about broadband performance at the local level and that this limitation was a result of cost issues and the finite number of measurement devices that could be deployed over the course of the project. A report on the MBA effort that was commissioned by FCC and conducted by the North Carolina State University’s Institute for Advanced Analytics primarily reviewed MBA’s methodology, but also made some suggestions for making the data more consumer-friendly, including the use of a tool developed by the group to help consumers compare ISPs’ performance in their states (when enough data were available to allow for statistically valid comparisons).

FCC Has Taken Steps to Evaluate Its Efforts to Inform Consumers, but Could Enhance Its Performance Information and Measures

FCC has taken steps to determine whether its efforts to provide consumers with broadband performance information—such as the transparency rule and MBA program—are effective and meeting consumers’ needs. Specifically, FCC has reviewed consumer complaints and requested stakeholder comments on this issue in several recent rulemaking proceedings. However, FCC’s ability to evaluate its efforts is limited by a lack of useful performance information. In addition, FCC established strategic objectives related to informing consumers in its strategic plan, but lacks performance goals and measures to monitor the effectiveness of its efforts to provide consumers with broadband performance information. Additional actions—such as conducting or commissioning objective consumer research and establishing performance goals and measures—could help FCC evaluate whether its efforts to provide consumers with broadband performance information are effective and meeting consumers’ needs.

FCC Has Requested Stakeholder Comments in Several Recent Rulemakings and Reviewed Consumer Complaints

FCC has requested stakeholder comments in rulemaking proceedings and reviewed consumer complaints filed with the commission to determine whether its efforts to provide consumers with broadband performance information—such as its transparency rule and MBA program—are effective and meeting consumers’ needs. According to FCC officials, FCC relies heavily on stakeholder comments to inform and explain decisions underlying the commission’s orders; however, FCC may also request stakeholders’ comments to develop a body of knowledge on issues of interest to the commission. In several recent rulemaking proceedings, FCC has requested comments from stakeholders on (1) consumers’ broadband performance information needs, (2) best practices for presenting and displaying performance information to consumers, and (3) potential improvements to FCC’s efforts to provide consumers with broadband performance information, among other things. For instance, FCC officials told us that stakeholder comments were used as part of FCC’s determination to enhance its transparency rule. Table 3 summarizes some of the consumer-related issues and questions FCC has raised and requested stakeholder comments on in recent years.

According to FCC officials, FCC also relies on consumer complaints filed with the commission to determine whether its efforts to provide consumers with broadband performance information are effective and meeting consumers’ needs. For example, in the 2014 Open Internet NPRM, FCC stated that it had received and analyzed hundreds of consumer complaints related to ISPs’ disclosures under the transparency rule and that these complaints were part of the agency’s rationale for tentatively concluding the rule should be strengthened. In addition, officials told us that consumer complaints have indicated that service reliability is now as important to consumers as speed due to the popularity of video-streaming services, such as Netflix, and that as a result, they are exploring ways to further incorporate service reliability information into the MBA program. They also told us that based on consumer complaints, they knew that consumers could not easily access—and in some cases were not provided—certain performance information (e.g., ISPs’ network management practices, data limits, etc.), and FCC tried to address this issue through the transparency rule.

FCC’s Ability to Evaluate Its Efforts Is Limited by a Lack of Useful Performance Information and Relevant Performance Goals and Measures

FCC Obtains Inconsistent and Incomplete Performance Information from Stakeholder Comments and Consumer Complaints

We have previously found that leading organizations that have progressed toward results-oriented management use performance information as a basis for decision-making and that the usefulness of performance information can be affected by its completeness, accuracy, consistency, validity, and credibility, among other things. For example, performance information can focus on various dimensions of performance, such as outcomes, outputs, quality, or customer satisfaction, and can help agencies monitor their progress in meeting programmatic and operational goals. Although stakeholder comments and consumer complaints can provide FCC with valuable insights on topics of interest to the commission, the information FCC obtains from these sources lacks some of the characteristics of useful performance information, such as consistency and completeness.

Our review of selected stakeholders’ Open Internet comments found that they generally provided inconsistent views regarding the effectiveness of FCC’s efforts to provide consumers with broadband performance information. For instance, some public interest groups and content providers argued that FCC’s transparency rule was not providing consumers the information they need to make informed decisions regarding broadband services. In their view, FCC should have required ISPs to provide additional technical information—such as detailed information on network congestion—under the transparency rule. However, ISPs countered that this level of technical detail would be confusing to consumers and that the transparency rule had been effective in providing consumers with the information they need. For example, Cox Communications, an ISP, stated that, “most of the expanded disclosure proposals included in the NPRM would impose burdensome obligations that are highly unlikely to prove useful to consumers and would only degrade the overall effectiveness of the transparency rule.” FCC has acknowledged that it has not received consistent information in response to its requests for comments. For instance, in its 2010 Open Internet Order, FCC states that despite broad agreement that broadband providers should disclose information sufficient to enable end users and content providers to understand the capabilities of broadband services, commenters disagree about the appropriate level of detail required to achieve this goal. As a result, FCC decided to continue to allow ISPs flexibility in the implementation of the transparency rule while providing guidance regarding acceptable disclosure methods. Similarly, in its 2015 Open Internet Order, FCC noted that despite its efforts to seek comment on this issue, the record was lacking with respect to specific details on how such a disclosure should be formatted. Thus, as previously mentioned, FCC asked its Consumer Advisory Committee to propose a disclosure format.

As noted above, in the 2014 Open Internet NPRM, FCC cited consumer complaints as a basis for tentatively concluding that the transparency rule should be strengthened. However, in the NPRM FCC also acknowledged that its analysis of consumer complaints provided limited insight into the issues that consumers were experiencing. Specifically, FCC stated that since the transparency rule took effect, it had received a “significant number” of consumer complaints about provider speeds, charges, and other practices that the rule was designed to disclose. However, FCC also stated that in some cases, it is difficult to discern from the complaints “whether the consumer’s frustration is with slow speeds or high prices generally, or instead with how the service as actually provided differs from what the provider has advertised.” Moreover, because consumer complaints reflect the perspective of a non-representative sample of consumers—i.e., those who have experienced broadband issues and have chosen to report those issues to the FCC—they do not fully capture the effectiveness of FCC’s efforts for a broader range of consumers. Consequently, several Open Internet comments we reviewed filed by ISPs and industry associations questioned FCC’s reliance on complaints as a basis for decision-making.

In addition, we have previously noted concerns with FCC’s complaint process. In 2009, we found that many wireless phone service consumers were not aware that they could file complaints with FCC when they encountered problems. We also found that FCC did not complete in- depth analyses of consumer complaints and lacked goals and measures that clearly identified the intended outcomes of its complaint-processing efforts. As a result, we noted that FCC may not be aware (1) of emerging trends in consumer problems, (2) if specific rules are being violated, or (3) if additional rules are needed to protect consumers. We made a number of recommendations to help address these issues, and FCC has indicated that it is in the process of reforming the agency’s complaint process. For example, in January 2015, FCC launched a new online consumer-help center. According to FCC, the online center will make it easier for consumers to file complaints with FCC, improve communications between consumers and FCC representatives, and help streamline FCC’s process for synthesizing and analyzing trends in consumer complaints. While these steps may enhance FCC’s ability to assess the effectiveness of its efforts among the subset of consumers who file complaints, they will not provide FCC information regarding the effectiveness of its efforts for a broader range of consumers.

Without consistent and complete information regarding the effectiveness of its efforts to provide consumers with broadband performance information, FCC’s ability to evaluate its efforts and make performance- based decisions about these efforts is limited. In addition, FCC may find it difficult to convince stakeholders that enhancements to its transparency rule and MBA program are warranted. As we reviewed industry groups’ Open Internet comments, we found many stating that FCC did not provide adequate evidence that ISPs’ disclosures under the transparency rule were ineffective and that enhancements or changes to the rule were needed. For example, Cisco, an equipment provider, stated that FCC’s concerns regarding the accuracy of information some consumers received under the transparency rule do not imply that additional disclosures are called for, but rather that the commission may need to enforce the requirements instead.

In contrast to the approaches FCC typically relies on to evaluate the effectiveness of its efforts to provide consumers with broadband performance information, other government entities have undertaken extensive, qualitative, and quantitative consumer research to evaluate the effectiveness of their efforts to inform consumers and ensure that consumers’ information needs are being met. For example:

Ofcom—the United Kingdom’s (UK) communications regulatory agency—has conducted qualitative and quantitative research to determine consumers’ use and understanding of ISPs’ traffic management information and the effectiveness of a standardized form that all major UK ISPs voluntarily use to publish their traffic management information. For instance, in 2013, Ofcom commissioned a study that involved qualitative workshops with 135 participants and a representative survey of UK broadband consumers. The study found that consumers had problems understanding some aspects of the standardized form and suggested several potential improvements that could make the information more understandable to consumers, such as adding a summary to the form explaining the relevance of traffic management policies to broadband products.

The U.S. Consumer Financial Protection Bureau (CFPB) conducted qualitative and quantitative consumer research when revamping mortgage disclosure forms. The CFPB tested the prototype of the revised disclosure form through 10 rounds of qualitative testing in nine cities. After issuing the revised forms the CFPB also conducted a large-scale quantitative validation test—that included 858 consumers in 20 locations—to compare the revised forms with the disclosures that were currently in use.

Some stakeholders we spoke with and whose comments we reviewed also suggested that FCC should undertake consumer research to inform and evaluate its efforts to provide consumers with broadband performance information. In 2009, the Federal Trade Commission filed comments with FCC stating that research has shown that well-designed standardized disclosures can improve consumer understanding and facilitate competition; however, to be most effective, these disclosures should be developed and revised based on controlled, quantitative, objective tests of consumer understanding. In addition, representatives from one industry association we spoke with stated that FCC’s analytical framework, which largely consists of requesting and reviewing stakeholder comments, is not consumer-centric and may not be the most productive process for obtaining an accurate representation of consumers’ information needs. These representatives noted that given the time, effort, and money ISPs expend to collect broadband performance data for FCC’s MBA program, it is important to make sure that the data are presented in a way that is meaningful to consumers. In their view, FCC should also be conducting consumer research, such as focus groups, to make sure that its MBA effort produces a report that consumers can understand. According to FCC officials, FCC’s goal is to improve the accessibility, accuracy, and relevance of the broadband performance information available to consumers. Because consumers’ decision-making processes are complex and can vary, FCC has not attempted to formally evaluate whether its efforts to provide consumers with broadband performance information are meeting consumers’ needs. Officials also told us that they do not believe this type of research has been conducted outside of FCC.

FCC Lacks Performance Goals and Measures to Monitor Its Efforts to Inform Consumers

FCC’s current strategic plan includes strategic objectives related to informing consumers about broadband networks—reflecting its position that informed consumers help facilitate competition in the broadband market. Specifically, FCC includes “ensure effective policies are in place to promote and protect competition” and “take action where competition is not sufficient to protect the public interest, including ensuring that consumers remain informed” as two of its 17 strategic objectives and states that it will continue to engage consumers though its outreach and education initiatives to facilitate informed choice in the competitive communications marketplace. However, FCC does not include any performance goals under these strategic objectives that define desired outcomes for its efforts to provide consumers with broadband performance information, such as its transparency rule and MBA program. Consequently, FCC also lacks relevant performance measures under these strategic objectives that would allow it to monitor the impact and effectiveness of its efforts. We have previously reported that effective performance goals and measures possess certain characteristics. These include the following: 1) performance goals should clearly define the organization’s desired outcomes, and 2) each performance goal should have a few performance measures that clearly tie back to the goal, cover key performance dimensions, and take different priorities into account.Without performance goals and measures that allow FCC to monitor its progress on its goals related to providing consumers with broadband performance information, FCC has limited assurance that its efforts to provide consumers with broadband performance information are effective and meeting consumers’ needs. In addition, FCC cannot demonstrate that it is achieving its objectives to ensure consumers are informed as envisioned in its strategic plan.

Conclusions

In recent years, the broadband market has undergone significant changes. More Americans are accessing broadband and relying on their broadband service for an increasingly diverse range of uses, including education, medicine, public safety, and entertainment. As broadband traffic grows, consumers’ broadband performance issues—such as slower than expected speeds and disruptions to video-streaming services—may simultaneously become more prevalent and more critical to resolve. However, broadband performance can be affected by a number of complicated, technical factors that are not readily apparent to most consumers.

FCC has maintained that informed consumers play an important role in facilitating competition in the broadband market and has undertaken efforts to provide consumers with broadband performance information, such as the transparency rule and the MBA program. However, FCC lacks useful performance information and relevant performance goals and measures to evaluate the effectiveness of these efforts and monitor their impact. As a result, FCC cannot be assured that its efforts to provide consumers with broadband performance information are effective and meeting consumers’ needs, and its ability to make performance-based decisions will remain limited. Moreover, FCC may find it difficult to convince stakeholders that its enhancements to the transparency rule are warranted and address their concerns that FCC has not provided adequate evidence to support these enhancements.

Recommendations for Executive Action

To help FCC determine whether its efforts to provide consumers with broadband performance information are effective and meeting consumers’ needs, and whether additional efforts—such as a standardized label suggested by FCC’s transparency working group— could benefit consumers, FCC should take the following two actions: 1. conduct or commission research on the effectiveness of FCC’s efforts to provide consumers with broadband performance information and make the results of this research publicly available, and 2. establish performance goals and measures under the agency’s relevant strategic objectives that allow it to monitor and report on the impact and effectiveness of its efforts.

Agency Comments

We provided a draft of this report to FCC for review and comment. In its written comments, reproduced in appendix II, FCC generally concurred with our recommendations. FCC described its various consumer outreach and education efforts and agreed that extensive research would be helpful in evaluating the effectiveness of its efforts to inform consumers. FCC stated that it will evaluate consumer research on broadband performance and move expeditiously to take action. FCC also stated that it would use consumer research, consumer comments, and in-person feedback at consumer-related events to establish performance measures that will allow it to monitor and report on its progress. FCC also provided an appendix with details about its outreach efforts as well as technical comments that were incorporated as appropriate.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Chairman of the Federal Communications Commission and the appropriate congressional committees. In addition, this report will be available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff has any questions about this report, please contact me at (202) 512-2834 or goldsteinm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

The objectives of this report were to examine (1) what information is currently available to consumers regarding broadband performance and the limitations of this information, if any, and (2) how does the Federal Communications Commission (FCC) evaluate its efforts to provide consumers with broadband performance information?

To assess what information is currently available to consumers regarding broadband performance and any limitations of this information, we reviewed prior GAO reports on this issue, relevant FCC rulemaking proceedings, including its 2014 Open Internet proceeding, FCC’s transparency rule and Measuring Broadband America reports, Internet service providers’ (ISP) websites, information provided by companies, including Netflix and Google, and third-party sites that provide broadband performance information to consumers. We also conducted a literature review to identify articles and studies relevant to broadband performance measurement and disclosure of broadband performance information to consumers. We used key-word searches to identify peer-reviewed articles and other work in relevant databases and reviewed the websites of researchers identified by FCC as conducting work relevant to our review. We also interviewed representatives from FCC, public-interest and consumer-advocacy groups, industry associations, content or equipment providers, and broadband providers. We identified stakeholders to interview based on our review of comments filed in FCC’s 2014 Open Internet and related proceedings, as well as based on recommendations from other organizations we interviewed. Finally, we reviewed a judgmental sample of comments filed in FCC’s 2014 Open Internet proceeding. In this proceeding, FCC requested comment on the effectiveness of its efforts to provide broadband performance information to consumers, including its transparency rule and its Measuring Broadband America program, and its proposed enhancements to these efforts. We ran key-word searches that included terms related to transparency and disclosure for consumers to identify relevant comments, and selected comments from a variety of stakeholders, including public interest groups, ISPs, content providers, trade associations, and consumer representatives. We reviewed these comments to identify statements regarding the effectiveness of FCC’s efforts to provide broadband performance information to consumers, including comments supporting and opposing FCC’s proposed enhancements to these efforts.

To determine how FCC evaluates its efforts to provide consumers with broadband performance information, we interviewed FCC officials regarding their efforts to provide consumers with broadband performance information and evaluate and monitor the effectiveness of these efforts and reviewed relevant FCC documents, including its strategic and performance plans, consumer surveys, and resources available to consumers on FCC’s website. We also reviewed FCC’s requests for comment on consumers’ information needs and the effectiveness of FCC’s efforts to provide consumers with broadband performance information, as well as selected comments filed in response to these proceedings (as described above). To determine what additional actions, if any, FCC could take to evaluate its efforts to provide consumers with broadband performance information, we interviewed representatives from the stakeholder groups identified above, as well as officials from the Federal Trade Commission and National Telecommunications and Information Administration regarding their efforts to inform consumers about broadband issues and to evaluate the effectiveness of those efforts. In addition, we reviewed reports and studies on consumer disclosures conducted by FCC’s Transparency Working Group; foreign telecommunications-regulatory bodies, including those in the United Kingdom and European Union; and other federal agencies, such as the Consumer Financial Protection Bureau. We also reviewed prior GAO reports on FCC and results-oriented management and compared FCC’s use of performance information and measures to performance management best practices outlined in previous GAO work.

We conducted this performance audit from July 2014 to April 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Federal Communications Commission

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Mark L. Goldstein, (202) 512-2834 or goldsteinm@gao.gov.

Staff Acknowledgments

In addition to the contact named above, Teresa Anderson, Assistant Director; Steve Brown; Crystal Huggins; Bert Japikse; Sara Ann Moessbauer; Jaclyn Nidoh; Josh Ormond; Cheryl Peterson; Amy Rosewarne; and Hai Tran made key contributions to this report.