Roaming performance study (Smart 2018/0011) : executive summary : an assessment of technical performance of mobile networks for roaming in the EU.
Date: 2021-03-18
Institutions: Joint Research Centre
Subjects: EU Member State, benchmarking, data collection, mobile phone, research report, roaming, transmission network

Roaming Performance Study  
(Smart 2018/0011) Final Report 
Executive Summary 

An assessment of technical 
performance of mobile 
networks for Roaming in the 
EU 

Chawdhry P., Folloni G., Lumachi S.,  

Luzardi S. 

2021 

EUR 30602 EN 
XX 

 
 
  
This publication is a Technical report by the Joint Research Centre (JRC), the European Commission’s science and knowledge service. 
It aims to provide evidence-based scientific support to the European policymaking process. The scientific output expressed does not 
imply a policy position of the European Commission. Neither the European Commission nor any person acting on behalf of the 
Commission is responsible for the use that might be made of this publication. For information on the methodology and quality 
underlying the data used in this publication for which the source is neither Eurostat nor other Commission services, users should 
contact the referenced source. The designations employed and the presentation of material on the maps do not imply the 
expression of any opinion whatsoever on the part of the European Union concerning the legal status of any country, territory, city or 
area or of its authorities, or concerning the delimitation of its frontiers or boundaries. 

Contact information  
Name: Pravir Chawdhry 
Address: TP723, via E. Fermi 2749, Ispra (VA), I-21027 Italy 
Email: Pravir.Chawdhry@ec.europa.eu 
Tel.: +39 0332 785823 

EU Science Hub 
https://ec.europa.eu/jrc 

JRC124253 

EUR 30602 EN 

PDF 

ISBN 978-92-76-30838-6 

ISSN 1831-9424 

doi:10.2760/26543 

Luxembourg: Publications Office of the European Union, 2021 

© European Union, 2021 

The reuse policy of the European Commission is implemented by the Commission Decision 2011/833/EU of 12 December 2011 on 
the reuse of Commission documents (OJ L 330, 14.12.2011, p. 39). Except otherwise noted, the reuse of this document is authorised 
under the Creative Commons Attribution 4.0 International (CC BY 4.0) licence (https://creativecommons.org/licenses/by/4.0/). This 
means that reuse is allowed provided appropriate credit is given and any changes are indicated. For any use or reproduction of 
photos or other material that is not owned by the EU, permission must be sought directly from the copyright holders. 

All content © European Union, 2021 
(unless otherwise specified) 
Android® is a registered trademark of Google LLC 
Apple® is a registered trademark of Apple Inc. 
IOS® is a registered trademark of CISCO Inc. 

How  to  cite  this  report:  Chawdhry  P.,  Folloni  G.,  Lumachi  S.,  Luzardi  S., Roaming  Performance  Study  (SMART  2018/0011)  Final 
Report  Executive  Summary,  An  assessment  of  technical  performance  of  mobile  networks  for  Roaming  in  the  EU, EUR  30602  EN, 
Publications Office of the European Union, Luxembourg, 2021, ISBN 978-92-76-30838-6, doi:10.2760/26543, JRC124253. 

 
 
 
 
 
 
 
 
  
  
  
  
 
 
 
 
 
 
 
 
 
 
 
Contents 

Abstract .............................................................................................................................................................................................2 

Executive Summary ......................................................................................................................................................................3 

1 

 
 
 
 
 
 
 
 
 
 
 
 
 
Abstract 

In the context of the review of the EU Roaming Regulation, this document presents results of a study 
on  roaming  performance  assessment  by  field  measurements  on  mobile  broadband  involving  40 
mobile  networks  in  13  EU  countries.  JRC’s  mobile  app  netBravo  was  used  to  carry  out  the 
measurements and analysis of data. Download speed, upload speed and latency were measured for all 
roaming tests and results were analysed. The study found mixed results on the quality of service (QoS) 
in  roaming.  Customers  had  better  as  well  as  worse  QoS  than  at  home.  However,  customers  of  21 
mobile  networks  from  11  countries  at  least  once  had  worse  QoS  in  roaming  compared  to  at  home 
even when technical conditions were available for better quality. Such cases accounted for 25% of all 
roaming instances in the tests. 

2 

 
 
 
Executive Summary  

Regulation (EU) 531/2012 of the European Parliament and the Council of 13 June 2012 on roaming on 
public mobile communications networks within the Union, amended in 2015 and 2017 (hereinafter the 
'Roaming  Regulation'),  mandates  the  Commission  to  conduct  a  review  by  December  2019  of  the 
roaming  rules.  One  of  the  review  requirements  is  to  provide  an  assessment  of  the  availability  and 
quality of roaming services. The Commission’s Review report1 was adopted on 29 November 2019, and 
is followed by a legal proposal. 

Following  a  request  by  Directorate-General  for  Communications  Networks,  Content  and  Technology 
(DG  CNECT)  for  the  above  review  purpose,  the  Joint  Research  Centre  (JRC)  undertook  preliminary 
work, to assess the technical performance and quality of service (QoS) of EU roaming in a small sub-set 
of EU Member States (MSs) during the first year of the RLAH rules taking effect (i.e. between October 
2017 and August 2018). This work served as a scoping exercise for a systematic approach developed 
later on in the present study for further tests carried out between October 2018 and October 2019.  

In the present project, assessment of the performance of roaming has been based on real data from 
extensive  field  measurements  on  mobile  networks  in  a  subset  of  EU  MSs.  The  JRC  mobile  app 
netBravo2 was used to measure mobile network performance. The data thus collected was analysed to 
help answer relevant policy questions posed by DG CNECT. Results of the analysis are used to further 
inform the Impact Assessment for the review and prolongation of the Roaming Regulation3.  

Results from the present study by the JRC have contributed in terms of evidence-based assessment of 
roaming performance to the roaming review led by DG CNECT in 2020. Additional tests measuring the 
performance of MVNOs SIM cards when roaming were foreseen  during the first half of 2020. Due to 
the COVID pandemic and travel restrictions those measurements had to be cancelled.  

The present project complements the work carried out by the JRC.I1 unit on the economic aspects of 
the roaming markets under the projects SMART 2018/0010 and SMART 2019/0004. 

What were the research questions? 

(1)  Whether  roaming  customers  have  different  quality  of  service  when  roaming  compared  to  the 
performance on their own home network. 

(2) Whether quality of service on visited networks differs between the customers of the visited network 
and the roaming users visiting the network.  

(3)  Whether  roaming  customers  have  different  quality  of  service  on  a  visited  network  compared  to 
other visiting roaming customers. 

(4) How often did  customers have worse quality of  service  in roaming than at home, even  when the 
visited network was technically and practically able to provide better quality? 

1 Report on the review of the roaming market, COM(2019)616 final, and SWD(2019)416 available here. 

2 For further information and how to download netBravo, see the Annex to this Report and netbravo.jrc.ec.europa.eu.  

3 This Initiative is included in the 2020 Commission Work Programme addressing the specific objective “Digital for consumers” 
and has to be seen in the broader political context of creating a Europe Fit for the Digital Age. The Initiative contributes to 
the  ambition  to  make  the  most  out  of  the  digital  transition  to  enhance  opportunities  to  connect,  communicate,  solve 
societal issues and do business. 

3 

 
 
 
 
                                           
What was tested? 

In the context of the EU Roaming Regulation, the JRC carried out a series of field measurements on 37 
mobile networks in 13 EU countries4  to assess the performance of roaming. Performance of 29 visiting 
SIMs from 12 EU countries was evaluated in relation to 1) their own home network, 2) in relation to the 
domestic SIM of the visited network, and 3) in relation to other visiting SIMs on the visited network. 

Where were the tests done? 

Altogether,  SIMs  of 40 different networks  were used  in field  measurements.  Three of these  networks 
did not happen to be visited by any visiting SIM therefore number of visited networks was 37. 

Number of visited countries: 

Number of visited networks: 

13 

37 

Number of visiting countries:                  12 

Number of visiting SIMs:  

29 

How were the tests done? 

Tests were carried out in a controlled manner using the JRC’s mobile app netBravo, which is designed 
to measure the coverage and quality of mobile networks and the performance of mobile broadband. 
All test data was stored and processed on the netBravo server with software adapted to the needs of 
the project. The app is available for free for smartphones running on Android® and Apple® iOS®. 

The  measurement  equipment  used  was  a  set  of  smartphones  of  the  same  model  (i.e.  the  same 
technical  specifications)  with  netBravo  app  installed.  The  handsets  used  were  suitable  for  a  4G+ 
network capability.  

Tests  were  done  in  key  city  locations  with  high  network  availability.  All  visiting  SIMs  on  the  same 
visited  network  were  tested  repeatedly  in  a  common  time  frame  to  ensure  the  network  traffic 
conditions were similar.  

The  measurement  data  on  download  speed,  upload  speed  and  latency  for  various  roaming  SIMs  in 
visited  countries,  as  well  as  at  home,  was  collected  and  analysed.  Comparisons  were  made  on  the 
performance of visiting SIMs vs home SIMs (of the visited network) and vs other visiting SIMs on the 
same visited network and in the same situations. The average values of download speed and upload 
speed observed in several test samples were used in comparing performance.  

4 At the time of the tests, the Roaming Regulation was still applicable in the UK. 

4 

 
 
 
 
 
 
 
 
                                           
Three types of analysis were performed on the measurement data set obtained in field tests: 

(a)   Customer-level analysis of roaming performance  
(b)   Network-level analysis of roaming performance 
(c)   Cross-correlation analysis between the above sets. 

What was found? 

Q1  –  Do  roaming  customers  have  different  quality  of  service  when  roaming  compared  to  the 
performance on their own home network? 

Looking at the overall data of 177 roaming instances between 37 visited networks and 29 visiting SIMs, 
the download speed for customer SIMs was worse in roaming than in their respective home network 
on 39% occasions. The upload speed was worse in roaming than at home on 59% occasions. Latency in 
roaming was found worse than at home in 62% cases. 

Q2 – Does the quality of service on visited networks differ between the customers of the visited 
network and the roaming users visiting the network? 

Across  the  37  networks  visited  by  29  visiting  SIMs,  results  from  field  tests  show  that  the  download 
speed  for  visiting SIMs (roaming customers) was worse than the home SIM (home  customer) of the 
visited  network  in  50%  of  the  roaming  instances.  The  upload  speed  for  the  visiting  SIMs  was  worse 
than the home SIM in 72% of the roaming instances. Latency for visiting SIMs was found worse than 
the home SIM in 73% cases. 

Q3  –  Does  the  roaming  customer  have  different  quality  of  service  on  a  visited  network 
compared to other visiting roaming customers? 

In relation to the roaming peers (other roaming users in the same visited network, at the same location 
and  at  the  same  time),  there  was  even  distribution  of  instances  of  roaming  performance  with  below 
average  and  above  average  download  speeds  for  the  visited  networks:  38%  had  better  performance 
than the average of all roaming users5, 40% had worse performance and 22% had about the average 
level  of  performance.  The  same  was  observed  for  the  distribution  of  upload  performance:  35%  had 
better  than  average  performance,  39%  had  worse  than  average  and  26%  had  average  level  of 
performance.  For  latency  the  distribution  was  36%  above  average,  32%  below  average  and  32% 
average. 

The  figures  were  similar  when  considering  only  the  cases  where  a  network  was  visited  by  at  least  5 
visiting SIMs. It shows that the test results were overall statistically consistent. 

However,  it  was  found  that  some  visiting  customer  SIMs  were  generally  more  likely  to  have  below 
average performance than others, pointing towards possibly QoS in roaming being offered to them at 
an unfavourable level. 

Q4  –  How  often  did  customers  have  worse  quality  of  service  in  roaming  than  at  home,  even 
when the visited network was technically and practically able to provide better quality? 

On further analysing the results for Q1-Q3, it was found that 20% roaming customers (6 out of the 29 
visiting  SIMs)  had  generally  worse  download  performance  in  roaming  than  at  home  and  also  below 
average download performance than the other roaming customers in the same visited network.  

Detailed  analysis  showed  that  21  customers  from  11  countries  had  worse  download  speed 
performance than at home and also worse than what was technically possible on at least one visited 
network. Altogether, such cases accounted for 25% of all 177 roaming instances. 

5  For  a  given  visited  network,  the  average  level  of  download  speed  was  calculated  based  on  the  download  speed  for  all  the 

visiting SIMSs for that visited network.  

5 

 
                                           
Conclusions 

Objective assessment of roaming performance is a complex exercise. There are several variables such 
as the QoS of the home network, QoS of the visited network, and contractual terms on the QoS both 
on wholesale level between the mobile operators as well as for a given customer. Moreover, there are 
temporal  and  spatial  variables  such  as  the  traffic  density  and  user  density  of  the  network  that  cause 
the  network  performance  to  fluctuate  over  time  and  across  different  locations.  These  result  in  the 
roaming performance to vary considerably for the visiting SIMs.  

For a single customer, home v. roaming performance can be anecdotal or episodic. The same SIM can 
have better performance than at home for one visited  network and worse than at home  for another 
visited network depending on the test conditions (time, place, traffic) as well as inherent performance 
specifications  of  each  network  and  the  QoS  criteria  and  constraints  that  may  be  applied  by  the 
network operators.  

Measuring a visiting SIM’s performance in relation to the performance of the home SIM of the visited 
network was found to be a useful yardstick to neutralize the effect of some of the above temporal and 
spatial variables. Doing so with a number of visiting SIMs added statistical diversity in the data  set as 
well as a broader view of the complexity involved in analysing cross-border roaming. 

Measuring the performance of a visiting SIM with respect to other visiting SIMs was yet another way to 
benchmark relative performance across several spatial locations.  

This study is quite unique of its kind in terms of the scope, evidence-based methodology and scale of 
field tests. The tests had involved nearly half of the EU countries and about a third of all EU networks, 
taking into  account the distribution between inbounder  and outbounder roaming  countries6, making 
them representative and to show a genuine picture of the diversity in roaming performance across the 
EU. 

6 An outbounder operator has a customer base which consumes more mobile services abroad (i.e. on the networks of partner 
operators in other EU/EEA countries), than those consumed by the partner operators’ customer base on its own network 
(i.e. when acting as a visited network). Conversely, an inbounder operator has a customer base which consumes less mobile 
services abroad than those consumed by the partner operators' customer base on its own network. 

6 

 
 
 
 
                                           
The relative performance of downloading data in roaming for most of the visiting SIMs was a mix of 
better/worse/same  in  relation  to  the  benchmarks  devised.  However,  the  upload  performance  and 
latency were found overwhelming in favour of the home SIMs.  

The  study  identified  44  instances  of  download  performance  in  roaming  (specific  visiting  SIMs  on 
specific  visited  networks)  that  cannot  be  explained  through  variables  such  as  temporal,  spatial  or 
technical  characteristics  alone.  For  such  cases,  accounting  for  25%  of  all  roaming  instances,  more 
transparent  information  on  the  QoS  applied  on  a  visiting  SIM  could  help  to  explain  the  causes  for 
poorer QoS in roaming than at home.  

7 

 
 
  
 
 
 
 
8 

 
 
 
GETTING IN TOUCH WITH THE EU 

In person 

All over the European Union there are hundreds of Europe Direct information centres. You can find the address of the 
centre nearest you at: https://europa.eu/european-union/contact_en 

On the phone or by email 

Europe Direct is a service that answers your questions about the European Union. You can contact this service: 

- by freephone: 00 800 6 7 8 9 10 11 (certain operators may charge for these calls), 

- at the following standard number: +32 22999696, or 

- by electronic mail via: https://europa.eu/european-union/contact_en 

FINDING INFORMATION ABOUT THE EU 

Online 

Information about the European Union in all the official languages of the EU is available on the Europa website at: 
https://europa.eu/european-union/index_en 

EU publications 
You can download or order free and priced EU publications from EU Bookshop at: 
https://publications.europa.eu/en/publications. Multiple copies of free publications may be obtained by contacting 
Europe Direct or your local information centre (see https://europa.eu/european-union/contact_en). 

 
 
K
J
-
N
B
-
3
0
6
0
2
-
E
N
-
N

doi:10.2760/26543 

ISBN 978-92-76-30838-6