Telecommunications: Current Broadband Measures Have Limitations and New Measures Are Promising but Need Improvement
Released: 2009-10-09T00:00:00

Letter

Background

The Internet is a vast network of interconnected networks that is used by governments, businesses, research institutions, and individuals around the world to communicate, engage in commerce, perform research, educate, and entertain. The Internet became widely accessible to U.S. households by the mid-1990s. Early on, the primary means to access the Internet was a dial-up connection, in which a standard telephone line is used to make an Internet connection. A dial-up connection offers data transmission speeds of up to 56 kilobits, or 1,000 bits per second (Kbps). Broadband access to the Internet became available by the late 1990s. Broadband differs from a dial-up connection in certain important ways. First, broadband connections offer a higher-speed Internet connection than dial-up. For example, some broadband connections offer speeds exceeding 1 million bits per second (Mbps) both upstream (data transferred from the consumer to the Internet service provider) and downstream (data transferred from the Internet service provider to the consumer). These higher speeds enable consumers to receive information much faster and thus enable certain applications to be used and content to be accessed that might not be possible with a dial-up connection. The higher transmission speeds that broadband offers cost more than dial-up, and some broadband users pay a premium to obtain very-high-speed service. Second, broadband provides an “always on” connection to the Internet, so users do not need to establish a connection to the Internet service provider each time they want to go online. Although broadband often is referred to as a singular service, it is available in a wide variety of data speeds—ranging from 768 Kbps to greater than 100 Mbps. FCC’s current categories for collecting data on the number of broadband subscribers by advertised download and upload speeds range from greater than 200 Kbps but less than 768 Kbps to equal to or greater than 100 Mbps. On August 20, 2009, as part of the proceeding to develop a National Broadband Plan, FCC posted a public request for comment on defining “broadband.”

Consumers can receive a broadband connection to the Internet through a variety of technologies that offer varying speeds of service, including, but not limited to, the following: Cable modem. Cable television companies first began providing broadband service in the late 1990s over their cable networks. When provided by a cable company, broadband service is referred to as cable modem service. Cable modem service is primarily available in residential areas. Cable modem service enables cable operators to deliver broadband service by using the same coaxial cables that deliver pictures and sound to television sets. Most cable modems are external devices that have two connections, one to the cable wall outlet and the other to a computer or router. Although the speed of service varies with many factors, download speeds of up to 6 Mbps are typical. Cable providers are developing even higher-speed services.

DSL. Local telephone companies provide digital subscriber line (DSL) service, another form of broadband service, over their telephone networks on capacity unused by traditional voice service. To provide DSL service, telephone companies must install equipment in their facilities and install or provide DSL modems and other equipment at customers’ premises and remove devices on phone lines that may cause interference. Most residential customers receive older, asymmetric DSL (ADSL) service with download speeds of 1.5 Mbps to 3 Mbps. ADSL technology can achieve speeds of up to 8 Mbps over short distances. Newer DSL technologies can support services with much higher download speeds.

Fiber. This technology, also known as fiber optic, converts electrical signals carrying data to light and sends the light through transparent glass fibers smaller than the diameter of a human hair. Fiber optic systems can transmit data at speeds far exceeding current DSL or cable modem speeds, typically by tens of gigabits per second. Fiber optic technology may be provided in several ways, including fiber to a customer’s home or business or to a location somewhere between the provider’s facilities and the customer. In the latter case, the last part of the connection to the customer’s premises may be provided over cable, copper loop, or radio technology. Such hybrid arrangements may be less costly than providing fiber all the way to the customer’s premises, but they generally cannot achieve the high transmission speed of a full fiber-to-the-premises connection.

Satellite. Three providers currently offer broadband service via satellite in the United States. These providers use geostationary satellites that orbit in a fixed position above the equator and wirelessly transmit and receive data directly to and from subscribers. Satellite companies provide transmission from the Internet to the user’s computer and from the user’s computer to the Internet, eliminating the need for a telephone or cable connection. Typically a consumer can expect to receive (download) at a speed of about 1 Mbps and send (upload) at a speed of about 200 Kbps. Transmission of data via satellite causes a slight lag in transmission, typically one-quarter to three-fourths of a second, thus rendering this service less suitable for certain Internet applications, such as videoconferencing. While satellite broadcast service may be available throughout the country, it generally costs more than most other broadband modes and its use requires a clear line of sight between the customer’s antenna and the southern sky. Both the equipment necessary for service and recurring monthly fees are generally higher for satellite broadband service, compared with most other broadband transmission modes.

Wireless. Land-based, or terrestrial, wireless broadband connects a home or business to the Internet using a radio link. Some wireless services are provided over unlicensed radio spectrum and others over spectrum that has been licensed to particular companies. In licensed bands, some companies are offering fixed wireless broadband throughout cities. Also, mobile telephone carriers—such as the large companies that provide traditional cell phone service—have begun offering broadband mobile wireless Internet service over licensed spectrum—a service that allows subscribers to access the Internet with their mobile phones or laptops in areas throughout cities where their provider supports the service. A variety of broadband access technologies and services also are provided on unlicensed spectrum—that is, spectrum that is not specifically under license for a particular provider’s network. For example, wireless Internet service providers may offer broadband access in particular areas by establishing a network of subscriber stations, each with its own antenna that relays signals throughout a neighborhood and has a common interface to the Internet. Subscribers place necessary reception equipment outside their homes that transmits and receives signals from the nearest antenna. Also, wireless fidelity (Wi-Fi) networks—which provide broadband service in so-called hot spots, or areas within a radius of up to 300 feet—can be found in cafes, hotels, airports, and offices. Such netowrks generally use a short-range technology that provides speeds up to 54 Mbps. Some technologies, such as Worldwide Interoperability for Microwave Access (known as WiMAX), can operate on either licensed or unlicensed bands, and can provide broadband service up to approximately 30 miles.

FCC has primary responsibility for regulating broadband. Section 706 of the Telecommunications Act of 1996 directs FCC to encourage the deployment of advanced telecommunications capability, which includes broadband, to all Americans. Under this authority, FCC has to date established a minimal regulatory environment for broadband Internet access services. In the past, FCC has stated that less regulation has encouraged providers to invest in broadband infrastructure. The Communications Act, as amended, allows FCC to classify services as telecommunications services or information services, the latter being subject to fewer regulatory restrictions. FCC, through a number of proceedings, classified broadband Internet access (regardless of the platform) as an information service. FCC does not have explicit statutory authority to regulate the provision of information services; however, FCC has the authority to impose regulations under what is termed its ancillary jurisdiction to regulate services that are reasonably related to its existing statutory authority. FCC also has the authority to adopt broadband regulations to ensure that broadband providers are capable of providing authorized surveillance to law enforcement agencies.

As part of its responsibilities, FCC has periodically issued a report to Congress on the status of advanced telecommunications capability in the United States, including the quality of broadband data. To assist in the preparation of this report, in 2000, FCC implemented the previously described broadband reporting form, a semiannual reporting requirement for facilities-based broadband Internet service providers. In November 2004, FCC modified its rules on filing this information, and the revised rules went into effect for the companies’ second filing in 2005. Specifically, FCC removed existing reporting thresholds, and all companies were required to report their total state subscribership by technology. In 2006, we reported that the approach FCC then used to collect data on broadband deployment, which counted broadband service providers with subscribers at the ZIP code level, resulted in inadequate information about broadband deployment. Subsequent to our recommendation, in March 2008, FCC acted to increase the precision and quality of its broadband data by revising its methodology and requiring that broadband providers report the number of broadband connections in service by census tract.

In addition to FCC’s data collection effort using its broadband reporting form, the Broadband Data Improvement Act calls for additional actions to improve the quality of data available on broadband deployment. Among other things, the act directs FCC to (1) periodically survey consumers to collect information on the types of technologies used by consumers to access the Internet, the applications or devices used in conjunction with broadband service, and the actual connection speeds of users; (2) collect information on reasons why consumers have not subscribed to broadband services; (3) determine certain demographic data for geographical areas not served by any provider of advanced telecommunications capability (i.e., areas where broadband has not yet been deployed); and (4) provide information on the extent of broadband service capability, including the speed and price of broadband service in a total of 75 communities in at least 25 countries.

FTC also has regulatory jurisdiction over broadband services with respect to competition and consumer protection issues. FTC’s jurisdiction over broadband services comes chiefly from its statutory mandate to prevent “unfair methods of competition” and “unfair or deceptive acts or practices in or affecting commerce” under FTC’s enabling legislation, the FTC Act. Although this authority is very broad, certain limited market sectors are expressly excluded from FTC’s enforcement authority. In particular, FTC’s enforcement authority does not reach “common carriers subject to the Communications Act of 1934, as amended. However, since most broadband Internet services are not provided on a common carrier basis, they are generally part of the larger economy subject to FTC’s general competition and consumer protection authority with regard to methods, acts, or practices in or affecting commerce. FTC has, where appropriate, investigated and brought enforcement actions in matters involving access to content via broadband and other Internet access services. Additionally, FTC has brought a variety of cases against Internet service providers that have engaged in allegedly deceptive marketing and billing practices.

Two other federal agencies have responsibility for telecommunications policies. The Office of Science and Technology Policy (OSTP) within the Executive Office of the President has a broad mandate to advise the President and the federal government on the effects of science and technology on domestic and international affairs and has led interagency efforts to develop science and technology policies and budgets. NTIA is the President’s principal telecommunications and information adviser and works with other executive branch agencies to develop the Administration’s telecommunications policies.

Multiple Measures Are Available for Consumers, Industry, Government, and Others to Assess Broadband Performance

Consumers Can Generally Access Measures of Availability, Price, Advertised Speed, and Actual Delivered Speed from Broadband Providers and Third Parties

Although there are limitations that we discuss later, consumers interested in broadband service can generally contact providers or search provider Web sites to determine the availability of service, advertised price, and advertised speed of broadband service in their area. For example, consumers can go to att.com or timewarnercable.com and enter their street address to learn about the availability of broadband service at their address, including price and advertised speeds. Each Web site also provides a phone number that consumers can use to reach a customer service representative to obtain information on availability, price, and advertised speeds of service. Consumers can then make their own comparisons of these prices and advertised speeds. In addition, third parties provide consumer Web sites, such as dslreports.com, that assemble this information for consumers to review.

However, actual delivered speeds depend on multiple factors, such as the equipment of the consumer, the applications in use, and Internet traffic, and may not always match advertised speeds or the theoretical maximum speeds stated by the provider. Consequently, there are tools available to consumers to measure actual delivered speed. Consumers with broadband service have access to their actual delivered speeds through speed tests from broadband provider Web sites and third parties. Speed tests generally measure the “last mile” speed (download and upload) of the consumer’s connection. Some third-party Web sites also provide information on actual delivered speeds of service and allow consumers to compare speeds. For example, speedtest.net allows individuals to compare their speed with that of other consumers by provider or in a set geographic region.

Some states have also completed broadband mapping efforts that provide consumers with information on broadband performance, including availability and advertised speed. We previously reported that 12 states had mapped broadband deployment, and 2 of these states, California and Massachusetts, had mapped both the speed and availability of broadband in their state and placed the information on their state’s Web site. In its 2008 report, California also provided information on average delivered upload and download speeds aggregated throughout the state and advertised residential speeds by price.

Industry and Government Have Some Broadband Performance Measures to Make Comparisons across Various Segments of the United States

The stakeholders we interviewed told us that FCC’s broadband data, collected through its broadband reporting form, constitute the primary data source generally used to measure performance and make comparisons across various segments of the United States, although there are limitations, which we discuss later. The Commission has tracked broadband subscribership and deployment since 2000 through its broadband reporting form. In 2006, we reported that the approach FCC then used to collect data on broadband deployment, which counted broadband service providers with subscribers at the ZIP code level, resulted in inadequate information about broadband deployment. To improve this information, in 2008, the Commission revised the semiannual reporting requirements of the broadband reporting form. The Commission now requires most broadband providers to file subscribership information by census tract, including the number of subscribers by technology, speed tier, and business/residential connection. In addition, mobile wireless service providers are now required to report the number of connections in (1) individual states, (2) the census tracts that best represent their broadband service footprint, and (3) in a separate category, the number of subscribers whose device and subscription permit them access to the lawful Internet content of their choice. These changes are expected to result in data that are more detailed than what was previously collected. The first round of data filings under the new requirements was due on March 16, 2009. As of September 2009, FCC staff was still in the process of analyzing the information.

Stakeholders also identified the Pew Internet & American Life Project’s reports on home broadband adoption as a source for measuring adoption and making comparisons across the United States. The results in the reports are based on data from approximately 2,300 telephone interviews (on both landline and cellular telephones) conducted by Princeton Survey Research International over the course of a month. The 2009 report included the following information that can be used to compare rural and nonrural areas: broadband adoption, broadband connection type, and, when applicable, reason for not having broadband access or Internet access.

Industry, Government, and Other Stakeholders Have Many Performance Measures Available for Making International Comparisons

Through our literature review and interviews with stakeholders, we focused on 10 performance measures often used by industry, government, and other stakeholders to make international comparisons of broadband service, as summarized below (limitations of these measures are discussed later). These measures fall into two general categories: (1) broadband- specific measures and (2) more general measures that cover a wide array of information and communications technology (ICT).

The broadband-specific rankings measure a nation’s broadband performance by focusing on the availability, penetration (or adoption), and quality of broadband in each country, and include those listed below (see table 1 for the U.S. ranking for each.)

Broadband Adoption Index. The Phoenix Center for Advanced Legal and Economic Public Policy Studies recently developed the Broadband Adoption Index (BAI), which proposes to compare the actual value that a society derives from broadband usage with that country’s target level for adopting various broadband technologies based on maximizing societal well-being. These targets vary by technology, demographic group, and country. The index does not include an overall ranking of countries based on broadband performance, because each country has its own unique set of adoption targets.

Broadband Quality Score. The Oxford Saïd Business School in Oxford, United Kingdom (UK), in conjunction with the University of Oviedo in Oviedo, Spain, and Cisco Systems, Inc., created the Broadband Quality Score (BQS) in September 2008 to highlight each representative nation’s ability to benefit from next-generation Web applications and services. According to the study, to establish broadband leadership, countries must focus on broadband availability, penetration, and quality.

Broadband Subscribers per 100 Inhabitants. OECD produces many broadband-related measures annually on its online broadband Web site. According to FCC and many of the stakeholders we interviewed, one of the most widely reported figures on broadband performance is OECD’s count of broadband subscribers per 100 inhabitants by technology. OECD also collects comparative data from its 30 member countries on multiple broadband measures such as penetration, usage, coverage, prices, services and speeds, and choice and competition. However, unlike other stakeholders, OECD does not aggregate its data into a composite indicator of national broadband performance.

Broadband Performance Index. The European Commission recently implemented the Broadband Performance Index (BPI), which measures and benchmarks the overall broadband performance of European Union member states based on a range of factors, which could include speeds, rural coverage, affordability, innovation, and other socioeconomic dimensions. In particular, the BPI ranks the EU-27 countries plus Norway in terms of supply and demand factors that affect the penetration and use of broadband.

ITIF broadband rankings. For its broadband rankings, the Information Technology and Innovation Foundation (ITIF) measures three primary broadband indicators, household penetration (rather than subscriber), average speed, and price, to rank the broadband performance of OECD nations. ITIF notes the importance of non-policy factors on a nation’s broadband performance, including demographic, economic, and broadband supply variables.

In contrast to broadband-specific rankings, the other performance measures we identified were based on each country’s development and use of ICT. These rankings are more general, focusing on the larger picture of how ICT usage, infrastructure, and skills can affect a country’s economic growth. According to an official at the Technology and Policy Institute (TPI), broadband is but one component in the makeup of a country’s ICT landscape, as ICT encompasses Internet usage along with other forms of telecommunications. According to FCC, these various measures demonstrate the value of understanding the broader context when making comparisons regarding broadband deployment and adoption. Examples of ICT-specific rankings include the following: Connectivity Scorecard. The Dean at the Haskayne School of Business at the University of Calgary in Calgary, Canada, worked in collaboration with Nokia Siemens Networks and LECG (a global services and consulting firm) to release the first version of the Connectivity Scorecard in 2008. The scorecard measures the impact of ICT on economic growth in three key areas of society—the consumer sector, the business sector, and the government sector. The report presents separate sets of rankings for “innovation-driven economies” and “resource- and efficiency-driven economies” while specifically focusing on each country’s ICT infrastructure and usage.

E-readiness Ranking. The Economist Intelligence Unit (EIU) is the business information arm of The Economist Group, publisher of The Economist magazine. The EIU produces an annual E-readiness Ranking, which measures the quality of a country’s ICT infrastructure as well as the ability of its consumers, businesses, and government to use ICT to their benefit. The EIU makes this assessment by specifically measuring a country’s connectivity and technology infrastructure, business environment, social and cultural environment, legal environment, government policy and vision, and consumer and business adoption. Overall, more than 100 separate qualitative and quantitative criteria are considered.

Networked Readiness Index: The World Economic Forum, in cooperation with INSEAD international business school’s eLab research center and Cisco Systems, Inc., produced the Networked Readiness Index (NRI). The NRI is used to assess the extent to which different economies benefit from the latest ICT advances based on their ICT environment, readiness, and usage while taking into account the key roles played by individuals, businesses, and governments. The NRI covers 134 economies worldwide and accounts for nearly 70 factors.

International Communications Market. In its 2008 report, Ofcom, the regulator for the UK communications industry, described developments in international communications markets, including information on broadband availability and usage. In the report, Ofcom aimed to provide statistically driven international comparative data for the UK communications sector by examining trend data from 2002 to 2007 on how various countries’ industries, consumers, and regulatory landscapes affect their communication markets.

ICT Development Index. The International Telecommunication Union (ITU), a United Nations agency, developed the ICT Development Index (IDI), which measures the development of ICT, the level of advancement of ICT, and the development potential of ICT in more than 150 countries worldwide, comparing their progress between 2002 and 2007. The purpose of the index is to track the global digital divide and to measure each country’s progress toward becoming an “information society.” The primary index measures ICT infrastructure/access, use, and skills, while a separate index was created to capture the price of ICT relative to a country’s income.

Eight of the 10 performance measures listed above are “composite indexes,” i.e., combinations of measures that are generally used to try to account for and normalize a variety of factors such as demographic, economic, and geographic differences among countries, which according to many of the stakeholders we spoke with can affect broadband deployment and penetration. Several of the stakeholders identified advantages to using composite indexes in making international comparisons. Officials from the European Commission reported that composite indexes are a useful tool to summarize the multidimensional issues, such as the socioeconomic differences among countries, which cannot be captured by a single indicator. According to ITU, compared with single indicators, composite indexes allow grouping several key performance indicators into one figure that captures a variety of information and provides a more comprehensive picture. While the various indexes differ on which demographic, economic, and geographic factors play a greater role in the supply and demand of broadband, income, age, education, population density, gross domestic product (GDP) per capita, and intermodal competition are generally considered important. According to ITIF, nonpolicy factors, such as demographic, economic, and broadband supply variables explain about three-quarters of the differences among nations’ broadband performance in international rankings. The determination of which factors to include or exclude in a composite index can greatly affect a nation’s ranking in a report, as demonstrated by the fact that the United States’ broadband and ICT rankings vary greatly by study, as shown in table 1.

Stakeholders Reported Limitations with Current Broadband Performance Measures, but Views Are Mixed on Potential Alternatives and Ongoing Efforts Need Improvement

Price, Actual Delivered Speed, and Service Reliability Measures for Consumers Have Limitations, but There Are Differing Views on Alternatives

Even though consumers have access to measures, stakeholders told us that measures of price, actual delivered speed, and service reliability have limitations that may affect their usefulness for consumers: Price. Stakeholders told us the available pricing measures for consumers are limited. For example, officials from the Consumer Federation of America and Pew Internet & American Life Project told us the lack of a comprehensive and consistent measure from the government for consumers to compare prices from providers was a limitation. They added that improved measures for prices would help consumers make more informed decisions about broadband services. Although FCC has open proceedings on requiring providers to include measures of price in the broadband reporting form, it currently does not collect this information.

Actual delivered speed. Stakeholders also identified limitations regarding the speed tests for consumers to measure actual delivered speeds. A representative from Akamai, a company that handles approximately 15 to 20 percent of all Internet traffic worldwide through its global server network, said one problem with speed tests is that the result can be significantly affected by the location of the server that is used to test the speed; the farther away the server, the less accurate the result. Many other factors can also affect a user’s speed of service, such as congestion on the network, time of day, and other applications that the user may have open on the computer when testing. NTIA officials told us that the speed tests are not able to determine the Internet traffic congestion points, if any, along the chain of networks. An official from the Pew Internet & American Life Project told us the results of the speed tests are not verified by other parties. He also explained that some third-party Web sites that attempt to compare actual delivered speeds have limited numbers of respondents and do not have an independent party verify the results, a fact that decreases the utility of the information for making comparisons. Finally, an official from the Information Technology and Innovation Foundation said the lack of comprehensive data for consumers to compare actual delivered speeds from providers was a limitation for consumers in comparing service options and policy makers in monitoring broadband. Actual delivered speed can be an important measure for consumers because it can determine whether or not a connection can be used to originate and receive high-quality voice, data, graphics, and video. FCC has open proceedings on requiring providers to report actual delivered speeds on the broadband reporting form, but it currently does not collect this information.

While broadband connection speeds that customers experience are generally not identical to the advertised speeds or theoretical maximums offered by the broadband provider, there is some evidence that consumers are not focused on this issue. Despite access to the tools to measure actual speed, one study found that few people actually know the speed of their broadband connections. In its report titled “Home Broadband Adoption 2006,” the Pew Internet & American Life Project reported that 81 percent of broadband users did not know their home connection speed. In addition, the federal government has received relatively few complaints regarding broadband speed. From February 1, 2008, through May 12, 2009, FCC reported receiving about 624,000 informal complaints, of which only 157 were related to broadband speed. Further, FTC reported receiving approximately 147 complaints that could be related to broadband speeds from January 2005 through June 19, 2009. According to some stakeholders, such as the Information Technology and Innovation Foundation, consumers appear more concerned with their end user experience, such as the ability to complete transactions or use their applications.

Service reliability. Some stakeholders we contacted, including BroadbandCensus.com, IEEE (previously known as the Institute of Electrical and Electronics Engineers), the Internet Engineering Task Force, Akamai, an economist from the Massachusetts Institute of Technology (MIT), NTIA, and Wireless Internet Service Provider Association (WISPA) are concerned that there is no measure for consumers that addresses service reliability. A service reliability measure would provide information to consumers on factors such as transmission quality, which affects perceived speed and could be useful to consumers in comparing the reliability of broadband services. According to an official from Akamai, service quality is the most difficult performance measure to define, measure, and relay to a consumer.

While consumers have measures of price, advertised speed, and actual delivered speed to make decisions regarding broadband service, some stakeholders suggested improved measures of price and actual delivered speeds for consumers, as shown in table 2.

As shown in table 3, stakeholders identified arguments for and against the proposed measures.

It should be noted that while federal and state agencies and public/private partnerships, academicians and think tanks, consumer advocacy groups, and trade and industry groups identified arguments for and against the proposed measures, broadband providers generally only provided arguments against the proposed measures. Thus, while stakeholders identified multiple alternatives, they differed on the need for FCC to develop additional reporting requirements to measure price, average actual delivered speed, and service reliability as follows: Consumer advocacy groups and academicians and representatives from think tanks generally believed there was a need for improved information on price and actual delivered speeds to make comparisons and good decisions about service. These stakeholders preferred that FCC require broadband providers to report price per megabit per second and the averaged actual delivered speed of last-mile connections (from the home to the first provider node or aggregation point) to provide more consistent measures for consumers to make comparisons. These stakeholders generally believed that calculating price per megabit should be done using the published, stand-alone nonpromotional, noncontractual price. Some suggested providing an average price by speed tier, while others suggested providing the lowest and highest prices by speed tier. Finally, some consumer advocacy groups and academicians and representatives from think tanks also favored a measure on service reliability to provide consumers with information on the quality of their connections.

In contrast, broadband providers and trade and industry groups generally did not perceive a need for additional broadband measures because, in their opinion, price and speed information is readily available from providers and third-party sources. According to these stakeholders, additional reporting requirements would be an intrusion into a market that is working, as evidenced by falling prices for increased speeds. They added that additional reporting requirements would be an impediment to investment in infrastructure, as more resources would need to be devoted to data collection. These stakeholders also reported that price per megabit and the average actual delivered speed are difficult to measure (as previously shown in table 3), and that FCC is not likely to report the information in a timely fashion. For example, in the past, it has taken FCC close to a year to report the data from the broadband reporting form once it has been submitted by broadband providers.

While officials at federal and state agencies and public-private partnerships generally said more information is good, there were mixed opinions on the need for FCC to require additional broadband measures. None of the federal agencies we interviewed provided an opinion; an official with the California Public Utilities Commission was uncertain if additional requirements were needed because similar information is already available to the public; and of the two interviewed, one public/private partnership was for additional broadband measures and one was against.

Finally, all stakeholder groups generally noted that FCC’s efforts to develop periodic surveys, per the Broadband Data Improvement Act, and a voluntary registry for consumers to report information about their broadband service, could be used to collect and disseminate price and speed information for consumer use. However, stakeholders also cautioned that periodic consumer surveys and a voluntary registry may not provide reliable information because consumers are not informed enough about the price and speed of their broadband service to report accurate information, and they believe that this should be taken into consideration when reviewing the results. Additionally, consumers may not take the time to enter their information in a registry, as current voluntary registries for broadband data sponsored by third parties are sparsely populated.

Current Data Sources for Industry and Government to Compare Broadband across Various Segments of the United States Have Limitations, and Stakeholders Generally Support Ongoing Efforts for Improvement

Despite FCC’s efforts to improve the data collected through its broadband reporting form, comparisons of broadband service across various segments of the country still have the following limitations that diminish their usefulness in informing policy and investment decisions: While FCC requires most broadband providers to report broadband subscribership on the broadband reporting form, it does not have a reporting requirement for these providers to report broadband availability. Additionally, although the majority of those we interviewed cited the change from reporting by ZIP codes to census tract as an improvement, some said the data still do not provide enough granularity to track subscribership in tribal lands or rural areas. In fact, according to FCC’s report Bringing Broadband to Rural America: Report on a Rural Broadband Strategy, there are no accurate data on broadband deployment in rural America, including where broadband facilities are deployed, prices, speeds, and the number of subscribers.

FCC also does not require broadband providers to report price information for broadband services on its broadband reporting form, so it is difficult to measure how price varies across various segments of the country. The Commission has open proceedings concerning whether and how the Commission could collect price information for broadband services. For example, the Commission sought comment on requiring providers to report, for each state or each census tract in which they offer service, the monthly price the provider charges for stand-alone broadband service in each of the speed tiers used for the broadband reporting form, potential alternatives, and whether and in what form the Commission should use the reported service price information.

Similarly, FCC does not require broadband providers to include information on actual broadband connection speeds experienced by consumers, although the data from the revised broadband reporting form will provide information on the number of connections by advertised speed. As previously mentioned, actual delivered speed can determine the applications that can be run by consumers and could be useful in comparing broadband service across various segments of the country. The Commission also has open proceedings concerning how the Commission might require broadband service providers to report actual broadband connection speeds, and any alternative means, in addition to or other than requiring such service provider reporting.

Some stakeholders noted that FCC may overestimate the number of wireless broadband users. FCC’s reporting requirement for mobile wireless broadband service providers collects data on the number of terrestrial mobile wireless subscribers whose subscription and device allow them to access the Internet content of their choice, not the number of consumers actually using broadband on the device. According to a Vice President and Senior Fellow at the Technology Policy Institute, it is unlikely that all persons whose subscription and device allow them to access the Internet actually use the service. As a result, counts of the number of terrestrial mobile wireless subscribers whose subscription and device allow them to access the Internet content of their choice may overestimate the number of wireless broadband users. However, other stakeholders, such as an official with the Rural Utilities Services, thought the reporting standard would produce accurate results, as they thought most consumers that paid for the service would use it. Stakeholders we spoke with generally characterized mobile wireless as a complement to and not a substitute for fixed wireline service. They added that this may change as the technology improves over time. Stakeholders also generally agreed that the mobile wireless counts should be kept separate from fixed wireline counts when determining deployment and availability.

Stakeholders also identified limitations with the Pew Internet & American Life Project data. While the survey collects information on cost, speed, availability, and usage, the data are limited because the sample size lacks the granularity needed for making comparisons at the state or regional level.

Despite the concerns about FCC’s data collected through the broadband reporting form, several stakeholders said they found the data useful. According to one academic expert, FCC’s broadband data are the best publicly available data on the geographic dispersion of broadband services across the United States. In addition, an official with a consumer advocacy organization said FCC’s changes to the broadband data collection struck the right balance between the need for detailed subscribership data and the burden to providers of gathering such information by choosing the census tract as the geographic unit for data collection.

To address the limitations in broadband data, recently enacted legislation requires the Secretary of Commerce to obtain more complete data on broadband availability. The Broadband Data Improvement Act requires the Secretary of Commerce to establish a grant program for multiple purposes, including collection of state-level broadband data. The American Recovery and Reinvestment Act of 2009 requires NTIA to establish a comprehensive nationwide inventory map of existing broadband service capability and availability in the United States that depicts the geographic extent to which broadband service is deployed and available from a commercial provider or public provider throughout each state. By February 17, 2011, NTIA must make the national inventory map available online to the public in a form that is interactive and searchable. The Recovery Act provides up to $350 million, pursuant to the Broadband Data Improvement Act, for developing and maintaining the national broadband inventory map.

NTIA has used the grant-making authority provided under the Broadband Data Improvement Act to establish the State Broadband Data and Development Grant Program. Through this program, NTIA has solicited grant applications from states for projects designed to collect data, develop state maps, conduct state planning efforts, and deliver data to NTIA for the purposes of developing the national broadband map. As of September 9, 2009, NTIA had received applications representing all 50 states, 5 territories, and the District of Columbia. NTIA is currently reviewing the applications and plans to announce funding decisions beginning in early fall 2009. Applicants must demonstrate that they have the ability to provide a substantially complete set of all broadband mapping data on or before February 1, 2010, and to complete such data collection by March 1, 2010. NTIA officials told us they are working closely with FCC regarding the development of the map. As part of its efforts, NTIA is requiring awardees under the State Broadband Data and Development Grant Program to provide, among other things, the following information: for each facilities-based provider of broadband service, a list of all census blocks of 2 square miles or smaller in which broadband service is available in the provider’s service area; for census blocks of greater than 2 square miles, for each facilities-based provider of broadband service, a list of all street segments in the census block in which broadband service is available in such provider’s service area; for wireless providers, geographical information system compatible polygonal shape files depicting areas in which broadband service is available; technology type of service provided by census block, street segment, or shape file area, as applicable; maximum advertised speed available across each service area or local franchise area, by metropolitan or rural statistical area; actual delivered speed that can be consistently achieved during expected periods of heavy network usage by census block, or street segment, as applicable; and middle-mile connection points.

Though the program does not require it, awardees may satisfy program requirements by providing address-level data. Awardees may also provide last-mile connection points, if available. Identification of a provider’s name and its availability/speed at a particular address is considered confidential. However, identification of a service provider’s specific service area, or “footprint,” at the census block or street segment level is not considered confidential and will be displayed on the national broadband map. The initial period of performance for awards under the program was 5 years from the date of the award. However, on September 10, 2009, NTIA announced that it will fund the mapping and data collection efforts for 2 years from the date of the award and will assess lessons learned, determine best practices, and investigate opportunities for improved data collection prior to obligating funding for subsequent years.

In the notice of funds availability for the State Broadband Data and Development Grant Program, NTIA noted that it reserved the right to request that FCC exercise its authority to compel any service provider subject to its jurisdiction to provide data. NTIA also explained that, to the extent possible, the service areas of individual providers will be aggregated with those of other providers of the same technology type. According to NTIA officials, this determination was based on its review of the comments, an examination of mapping methodologies employed at the state level, and consultation with FCC.

Stakeholders generally agreed that the national broadband inventory map would help supplement gaps in FCC’s broadband data by providing detailed data on availability and subscribership across the country. For example, a Pew Internet & American Life Project official told us that broadband mapping has the most potential for providing the granular and accurate information required to make comparisons across the country. Several stakeholders also explained that in order for the national broadband map to be effective, NTIA needs to develop data collection standards to help ensure that the data collected by each state are comparable across states. Some stakeholders also stressed the need for collecting information regarding demand side data (desire for service or usage).

Despite the consensus among stakeholders regarding the potential benefits of broadband mapping, there are some concerns about the effort.

We found NTIA did not provide guidance on how to calculate actual delivered speed that can be consistently achieved during expected periods of heavy network usage at the address. For example there is no guidance on the number of speed measurements that must be taken or a definition of heavy network usage. We have previously reported that consistency— the extent to which data are collected using the same procedures—is a key dimension of data quality and a key attribute of a successful performance measure. NTIA officials told us they chose not to provide this guidance because each provider may have a different method for measuring speed, and they did not want to prescribe a standard method, given the multiple technologies used. However, this could result in inconsistent measurements across grantees, limiting the effectiveness of the mapping effort in making comparisons across the country.

While NTIA required applicants to provide a description of the methods the applicant intends to employ to verify data accuracy, it did not set out specific standards on how to do so. NTIA’s notice of funds availability did provide the following example: “A project should propose to collect availability data by address . . . and should cross-check that data for accuracy by using at least one other metric.” We have previously reported that both verification and validation of performance data are important for reducing the risk of producing inaccurate data; this additional information helps to place the credibility of an agency’s reported performance data in context for decision makers. NTIA offic told us they chose not to specify how grantees should verify data becauser they did not want to be too prescriptive, as allowing states to develop thei own data verification processes may yield best practices that can be used going forward. While it is too early to determine the effect, if any, of the limited guidance, the lack of specific standards for data verification could result in inconsistent data across states, limiting the effectiveness o data in making comparisons acros s the country.

The broadband providers we spoke with were generally concerned about the cost and burden of complying with any additional reporting requirements. For example, officials from Time Warner told us that some providers do not store data in an address-by-address format and would have to revise their existing data collection procedures, taking time and resources away from network upgrades. According to FCC, broadband providers already average 337 staff hours to complete the reporting requirements for the broadband reporting form. Other stakeholders, such as Connected Nation, Consumers Union, and the Organization for the Promotion and Advancement of Small Telecommunications Companies (OPASTCO), also acknowledged that additional reporting requirements can be particularly burdensome to small broadband providers in rural areas that do not have the staff and resources of larger broadband providers. In addition, the NTIA requirement to provide data on availability may overlap with FCC’s requirement for broadband providers to report subscribership information through the broadband reporting form, because subscribership is a subset of availability. Service must be available for a consumer to be a subscriber. To ease the potential burden on broadband providers, NTIA has timed its future data collection efforts to coincide with FCC’s broadband data collection.

Finally, some stakeholders, including the Pew Internet & American Life Project, Consumer Federation of America, and Consumers Union, were concerned that some data underlying the state maps would not be publicly available for review. They explained that public-private partnerships often agree to nondisclosure agreements with broadband providers to facilitate data collection by easing provider concerns regarding what the providers consider to be the proprietary nature of the data. However, according to these stakeholders, this reduces the transparency of the maps and prevents other interested parties from analyzing the information.

Again, stakeholders generally noted that FCC’s efforts to develop periodic surveys (per the Broadband Data Improvement Act) and a voluntary registry could be used to collect and disseminate price and speed information to make comparisons of broadband service across the country. But they cautioned that information gleaned from these efforts is limited and therefore should be a supplement to other data collection efforts, because, as previously mentioned, consumers may not be well informed about the price and speed of their Internet service.

International Broadband Comparisons Have Limitations for a Variety of Reasons, but Stakeholders Generally Support FCC’s Efforts to Develop Additional International Comparisons

As previously discussed, stakeholders reported that socioeconomic differences among countries can limit the efficacy of international comparisons. For example, OECD and ITU report broadband subscribers per 100 inhabitants rather than as a percentage of households. According to a senior official at the Technology Policy Institute, household size alone explains most of the differences in the broadband rankings of countries, since countries with larger households are likely to have lower per capita residential connections. As the Phoenix Center demonstrated, even if every home and business in every OECD country were wired with a broadband connection, the United States’ per capita rank would fall from 15th to 20th because the United States has a larger average household size than countries, such as Sweden and Iceland, that rank above it. According to FTC staff, because the socioeconomic status of individual countries and the historical nature of their Internet access markets can vary widely, simple comparisons of individual indicators such as broadband deployment and adoption rates across countries may not be meaningful.

In contrast to OECD’s use of subscriber data, the composite indexes we previously described attempt to take into account the socioeconomic differences and other variables among countries when comparing broadband performance. However, according to stakeholders, even composite indexes provide limited analysis because of their complex nature and the number of variables they seek to measure. For example, one of the authors of the Connectivity Scorecard noted that composite indexes are “ultimately based on subjective decisions about which indicators to include or exclude and how to weight these indicators.” The more factors or variables considered in a composite index, the more data must be collected, normalized, and weighted for comparative purposes. A spokesperson for the EIU’s E-readiness ranking stated that more variables increase the room for error. Multiple variables also make it difficult to determine a causal relationship for policy-making purposes between the variable and its measured impact on the result, according to officials with the European Commission. For example, the EIU included nearly 100 quantitative and qualitative variables in its E-readiness Ranking Report in an attempt to measure the impact of a country’s social, political, economic, and technological developments on its ICT usage and infrastructure. A representative of the EIU told us that there are limitations to this approach, and that some of the unit’s data must be estimated because of the sheer number of variables the EIU attempts to consider for the 70 countries in the E-readiness Ranking Report.

Stakeholders also reported that the necessary data to improve international comparisons of broadband deployment and penetration are not available. OECD and others have noted that while supply-side data from broadband providers are both readily available and easily quantifiable, demand-side data from consumers for measuring broadband penetration are limited. Some stakeholders, such as officials with ITU, TPI, and ITIF, have noted the importance of collecting demand-side data through household surveys to more accurately reflect how consumers use their personal broadband service for economic or social gain. Governments are also increasingly recognizing the importance of collecting better demand-side data. For example, EU member countries are now required to collect household survey data on ICT usage.

In addition, stakeholders reported a lack of uniformity and reliability with the data used to make international broadband comparisons, whether by composite index or single indicator. For example, although most of the countries that participate in international broadband ranking systems recognize broadband to be Internet service above 256 Kbps, there is no internationally agreed upon definition for broadband, which affects the comparability of the data collected. OECD and ITU have recommended uniform reporting standards among their member countries, but the standards are neither enforceable nor applicable to countries outside their membership. In addition, some of the organizations that develop international comparisons rely on participating countries to provide the needed data rather than independently gather the data directly from providers or in the form of household surveys, a fact that leads some to question its reliability. The officials we interviewed from the organizations that develop international comparisons told us they have limited ability to corroborate the data received from participating governments, outside of questioning and confirming a figure when a number appears out of line with trend data. Estimates are also made when the data are simply lacking for a particular country.

Currently, discussions are also taking place on how to collect and differentiate among wireline, wireless, and mobile wireless broadband counts. According to OECD, wireless Internet connections at broadband speeds are increasingly available and particularly important in underserved areas around the world. Similarly Internet access via mobile cellular networks has grown rapidly with the increasing availability of third-generation (3G) networks and enabled devices that allow users to access the Internet over mobile cellular networks using a laptop, cell phone, or alternative mobile device. A representative from the Economist Intelligence Unit stated that mobile wireless Internet access is particularly important for individuals in developing countries, such as in Africa, where mobile access may be their primary Internet source. However, stakeholders noted that it is important to differentiate between 3G subscribers whose plan may allow them to access the Internet on their mobile device and those who actually take advantage of the service; current data usually do not differentiate and are therefore potentially misleading. OECD is in discussions with member countries to develop a common methodology to improve the collection of mobile wireless data.

Despite the concerns raised about the limitations of the measures used for international comparisons, several stakeholders found the comparisons useful. As previously mentioned, OECD’s count of broadband subscribers per 100 inhabitants by technology is one of the most reported figures. Representatives from the Consumer Federation of America, Free Press, and the Pew Internet & American Life Project said the OECD broadband comparisons provide valuable information to policy makers. In its guidance on developing composite indicators, OECD noted that composite indexes used by other organizations in making international broadband comparisons are recognized as a useful tool in policy analysis and public communication. The indexes serve the important purpose of raising awareness among policy makers and the public of areas that deserve particular attention in future policy decisions.

FCC has noted that a more fully developed picture of broadband markets would provide more accurate and useful international comparisons. The recent Broadband Data Improvement Act mandated that FCC include in future 706 reports information that compares the extent of broadband service capability in a total of 75 communities in at least 25 countries abroad for each of the data benchmarks for broadband service under FCC’s current speed tiers. The Commission was directed to choose international communities for the comparison that will offer a population size, density, topography, and demographic profile that are comparable to those of various communities within the United States. In May 2009, FCC officials informed us that they had already assembled a cross-bureau team of economists and attorneys to perform this international comparison. FCC staff is currently in the process of identifying and reaching out to a number of countries believed to have the relevant broadband data necessary to make such comparisons. According to the officials, they have sent letters to 37 countries to request data. They are working under the assumption that the mandate will require them to communicate the results of their comparisons in the next Section 706 report, which is to be released in February of 2010. In addition, on March 31, 2009, FCC posted a public request for comment on the international comparisons component of the act. The majority of stakeholders we spoke with generally support FCC’s efforts to develop an additional international comparison on broadband performance. Although the term “community” was not defined in the act and had yet to be defined by FCC, this level of analysis could be more granular and therefore more comparable than what is generally provided in current international comparison reports. Representatives from organizations such as Connected Nation and Free Press support data that are collected and analyzed at a more granular local level rather than at a national level, because they believe that such data make the comparisons more relevant.

Conclusions

A wide range of measures to assess broadband performance is generally available to consumers, industry, and government. However, many stakeholders told us that the measures used by consumers and those used to make comparisons across the United States and among other countries have limitations. Reaching a compromise among broadband providers, consumer advocates, and others on improved broadband measures in the United States has proven to be difficult because they do not agree on alternatives for improvement. Nevertheless, all stakeholders are generally supportive of NTIA’s State Broadband Data and Development Grant Program and its effort to create a national broadband inventory map, which could help fill some current gaps in data. NTIA has made progress in (1) implementing its State Broadband Data and Development Grant Program and (2) requiring grantees to collect data that have important implications for consumers, policy makers, and industry in measuring broadband performance. NTIA will begin receiving data by March 2010 as part of its new grant initiative to collect state-level broadband data and establish a national broadband inventory map. However, NTIA lacks specific guidance for grantees on calculating actual delivered speeds. Without such guidance, it will be difficult to ensure the consistency, and therefore the quality, of the data, limiting the effectiveness of the mapping effort in making comparisons across the country. In addition, while NTIA provided potential grantees with an example of how to verify data accuracy, it did not provide specific standards to verify data accuracy. Consequently, NTIA will need to determine whether the data provided in the initial submission are accurate, and if additional guidance is needed. Developing procedures to help ensure consistent and accurate data is critical, as NTIA begins to distribute funds to grantees and they begin their data collection. More importantly, this effort has the potential to provide consumers, policy makers, and industry with accurate and reliable information such as broadband availability, type, and advertised and actual delivered speed by census block, information that could be used by each in their decision-making process and help guide broadband investment in unserved or underserved populations.

Recommendation for Executive Action

To increase the data quality and subsequent results from the State Broadband Data and Development Grant Program, including a searchable nationwide inventory map of existing broadband service capability and availability in the United States, we recommend the Secretary of Commerce examine the first round of data collection and determine whether to develop specific guidance for grantees to improve the consistency and accuracy of the data collected under the program.

Agency Comments

We provided a draft of this report to the Department of Commerce and FCC for their review and comment. The Department of Commerce provided written comments, which are reprinted in appendix II. In its written comments, the Department of Commerce generally agreed with our recommendation and stated that it had already begun taking actions to address the recommendation. More specifically, the Department of Commerce stated that immediately following the awarding of grant funds, it will investigate opportunities for improved data collection methods including qualitative and quantitative analyses of data collection and verification methods, as well as an assessment of which methods are cost- efficient and accurate. FCC responded that it did not have any comments on the draft report.

We are sending copies of this report to the Secretary of Commerce and the Chairman of the Federal Communications Commission. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about his report, please contact me at (202) 512-2834 or goldsteinm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Contact information and major contributors to this report are listed on appendix III.

Appendix I: Scope and Methodology

To gather information related to both objectives, we reviewed related documentation and laws, including the Broadband Data Improvement Act, enacted in 2008; the legislative history of the act; the American Recovery and Reinvestment Act of 2009 and its legislative history; various Federal Communications Commission (FCC) proceedings; and reports from the Congressional Research Service (CRS). We also conducted a literature review to identify broadband performance measures, including international broadband comparisons.

To identify the broadband performance measures available to consumers, industry, government, and other stakeholders, we interviewed officials and representatives from several stakeholder groups. On the basis of the requirements of the mandate, the literature review, the judgment of our staff with expertise in broadband and telecommunications issues, and suggestions from the initial interviews held, we determined to include the following stakeholder groups in our analysis to ensure a variety of perspectives and views on broadband performance measures: academicians and think tanks, broadband providers, consumer advocacy groups, federal and state agencies and public/private partnerships, international organizations, and trade and industry groups. We used the same process to identify potential stakeholders for interviews. Table 4 contains a detailed list of the stakeholders included in our study: To evaluate the limitations, if any, of the measures, and how the measures could be supplemented or improved, we interviewed and reviewed related documentation from the stakeholders previously mentioned to obtain their opinions and analysis on the strengths and limitations of the measures and any potential options identified. We also asked the stakeholders to discuss the validity and reliability of the measures and any potential improvements.

Although representatives from the think tanks and academicians we interviewed identified limitations with the data that are used to make international comparisons, stakeholders generally use the same sources, thought the data were adequate, and support current efforts being made to improve the quality of the data.

We conducted this performance audit from February 2009 through October 2009 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Commerce

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, David Sausville (Assistant Director), Eli Albagli, Derrick Collins, Amy Rosewarne, Andrew Stavisky, Hai Tran, Amy Ward-Meier, and Mindi Weisenbloom made key contributions to this report.