Measuring Mobile Broadband Performance: Metrics and Policy Challenges

Mobile broadband is increasingly central to virtually all discussions of broadband Internet policy, industry structure, and network management. Performance measurement is needed to monitor and regulate service quality, to make informed purchase decisions, and to assess market/network trends/behavior and diagnose faults. In recognition of this, policymakers, industry participants, researchers, and standards bodies have initiated multiple mobile broadband measurement projects. In a number of cases, the mobile measurement efforts evolved from earlier fixed broadband measurement initiatives or are closely associated because of shared infrastructure, metrics, or sponsors. Because of the obvious close connection between mobile and fixed broadband as potential substitutes and/or complements, the goals, audience, and use of the measurements will be similar. Consequently, users will naturally wish to compare and integrate fixed and mobile results. While there are obvious commonalities between fixed and mobile broadband performance measurement, there are important differences that impact program/metric design and the interpretation and use of results arising as a byproduct of mobility, wireless media, and differences in usage contexts. In spite of these complexities, the public availability of measurement data for mobile broadband presents a valuable opportunity to expand our collective understanding of the broadband ecosystem, but also intensifies the potential for tussle over the appropriate interpretation of the soon-to-be more abundant sources of comparative broadband performance data. Consumers and content/application providers may be interested to learn when and where cellular LTE service might offer a better user experience than public WiFi or a home fixed connection. With mobile applications, form factor, battery usage, and other factors arguably make assessing (or managing) the user's quality of experience inherently more challenging than for fixed broadband services – and consequently, imply a more complex metrics challenge. The new data offers opportunities to investigate this. Those concerned with broadband management policy (aka, Network Neutrality) will also have a vested interest in how this data is used. The goal of this paper is to assess the current status of new mobile broadband measurement initiatives and identify the opportunities and challenges these pose for the broadband Internet ecosystem. We frame our discussion by first reviewing some of the key initiatives we are aware of and by reviewing the challenges of mobile broadband measurement as distinct but related to the earlier work on fixed broadband measurement. We then consider some of the specific technical challenges of metric design and measurement implementation confronted by these initiatives. These include issues like defining an appropriate averaging (over coverage/time) strategy to adequately capture the quality of experience that matters for a mobile service; crowd-sourcing sampling issues; and other such details. While consensus is emerging on how best to address some of these issues, mobile broadband measurement is very much a work-in-progress and a status check on this is timely. We also review recent research literature discussing the design or making use of the new datasets and measurement platforms to highlight new insights and suggest directions for further research. To highlight some of the policy/market implications of the mobile broadband measurement platforms, we examine two distinct questions (a) how might efforts to measure mobile broadband performance best complement and leverage developments in spectrum management policy; (b) whether there should be distinct mobile and fixed broadband minimum performance standards.