Roaming performance study (Smart 2018/0011) : final report : an assessment of technical performance of mobile networks for roaming in the EU.
Date: 2021-03-05
Institutions: Joint Research Centre
Subjects: EU Member State, benchmarking, data collection, mobile phone, research report, roaming, transmission network

Roaming Performance Study  
(Smart 2018/0011) Final Report  

An assessment of technical 
performance of mobile 
networks for Roaming in the 
EU 

Chawdhry P., Folloni G., Lumachi S.,  

Luzardi S. 

2021 

EUR 30602 EN 

 
 
 
 
  
This publication is a Technical report by the Joint Research Centre (JRC), the European Commission’s science and knowledge service. 
It aims to provide evidence-based scientific support to the European policymaking process. The scientific output expressed does not 
imply a policy position of the European Commission. Neither the European Commission nor any person acting on behalf of the 
Commission is responsible for the use that might be made of this publication. For information on the methodology and quality 
underlying the data used in this publication for which the source is neither Eurostat nor other Commission services, users should 
contact the referenced source. The designations employed and the presentation of material on the maps do not imply the 
expression of any opinion whatsoever on the part of the European Union concerning the legal status of any country, territory, city or 
area or of its authorities, or concerning the delimitation of its frontiers or boundaries. 

Contact information 
Name: Pravir Chawdhry 
Address: TP723, via E. Fermi 2749, Ispra (VA), I-21027 Italy 
Email: Pravir.Chawdhry@ec.europa.eu 
Tel.: +39 0332 785823 

EU Science Hub 
https://ec.europa.eu/jrc 

JRC124115 

EUR 30602 EN 

PDF 

ISBN 978-92-76-30476-0 

ISSN 1831-9424 

doi:10.2760/682 

Luxembourg: Publications Office of the European Union, 2021  

© European Union, 2021 

The reuse policy of the European Commission is implemented by the Commission Decision 2011/833/EU of 12 December 2011 on 
the reuse of Commission documents (OJ L 330, 14.12.2011, p. 39). Except otherwise noted, the reuse of this document is authorised 
under the Creative Commons Attribution 4.0 International (CC BY 4.0) licence (https://creativecommons.org/licenses/by/4.0/). This 
means that reuse is allowed provided appropriate credit is given and any changes are indicated. For any use or reproduction of 
photos or other material that is not owned by the EU, permission must be sought directly from the copyright holders. 

All content © European Union 2021 
(unless otherwise specified) 
Android® and GooglePlay® are registered trademarks of Google LLC 
Apple® and AppStore® are registered trademarks of Apple Inc. 
Bluetooth® is a registered trademark of Bluetooth Special Interest Group Inc. 
IOS® is a registered trademark of CISCO Inc. 
Samsung S4® and Samsung S9® are registered trademarks of Samsung Electronics Co. Ltd. 
Wi-Fi® is a registered trademark of Wi-Fi Alliance® 

How to cite this report: Chawdhry P., Folloni G., Lumachi S., Luzardi S., Roaming Performance Study (SMART 2018/0011) Final 
Report, An assessment of technical performance of mobile networks for Roaming in the EU, EUR 30602 EN, Publications Office of 
the European Union, Luxembourg, 2021, ISBN 978-92-76-30476-0, doi:10.2760/682, JRC124115. 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Contents 

Acknowledgements ......................................................................................................................................................................1 

Abstract .............................................................................................................................................................................................2 

Glossary .............................................................................................................................................................................................3 

List of figures ..................................................................................................................................................................................5 

List of tables ....................................................................................................................................................................................7 

Executive Summary ......................................................................................................................................................................8 

1 

Introduction ........................................................................................................................................................................... 13 

1.1  Roaming in the EU .................................................................................................................................................... 13 

1.2  Regulatory context .................................................................................................................................................... 14 

1.3  Research questions ................................................................................................................................................... 14 

1.4  Research Methodology ........................................................................................................................................... 15 

2  Work Description ................................................................................................................................................................. 16 

2.1  WP1: Data Collection ............................................................................................................................................... 16 

2.2  WP2 : Data Analysis .................................................................................................................................................. 17 

2.3  WP3: Report ................................................................................................................................................................. 18 

3  Test Methodology ............................................................................................................................................................... 19 

3.1  Definition of the scope of tests ............................................................................................................................ 19 

3.2  Development of a detailed test method .......................................................................................................... 20 

4  Data collection ...................................................................................................................................................................... 30 

5  Analysis and Findings ........................................................................................................................................................ 31 

5.1  Data analysis ................................................................................................................................................................ 31 

5.1.1  Q1. Do roaming customers have different quality of service when roaming compared to 
the performance on their own home network? ................................................................................................ 31 

(A) 

(B) 

(C) 

Download performance ................................................................................................................... 31 

Upload performance ......................................................................................................................... 33 

Latency performance......................................................................................................................... 35 

5.1.2  Q2. Does the quality of service on visited networks differ between the customers of the 
visited network and the roaming users visiting the network? .................................................................... 36 

(A) 

(B) 

(C) 

Download Performance ................................................................................................................... 37 

Upload performance ......................................................................................................................... 39 

Latency performance......................................................................................................................... 41 

5.1.3  Q3. Does the roaming customer have different quality of service on a visited network 
compared to other visiting roaming customers? ............................................................................................. 42 

(A) 

(B) 

Download performance ................................................................................................................... 43 

Upload performance ......................................................................................................................... 45 

i 

(C) 

Latency .................................................................................................................................................... 47 

5.1.4  Q4. How often did customers have worse quality of service in roaming than at home, 
even when the visited network was technically and practically able to provide better quality? .. 51 

6  Conclusions ............................................................................................................................................................................ 53 

Annexes .......................................................................................................................................................................................... 54 

Annex 1. netBravo ............................................................................................................................................................... 54 

ii 

 
 
 
 
 
 
 
 
 
 
 
 
 
Acknowledgements 

The  authors  would  like  to  thank  Emina  Mujic,  Erika  Busechian,  Minas  Karatzoglou  and  Matthieu 
Delescluse from DG CNECT for fruitful collaboration in this project. Authors would like to thank several 
colleagues  and  friends  for  their  timely  assistance  with  the  procurement  and  topping  up  of  national 
SIMs  from  time  to  time  in  the  course  of  the  project:  Antriksh  Awasthi,  Karen  Boniface,  Herschel 
Chawdhry,  Chrysanthi  Chountala,  Jaime  Ferragut,  Joaquim  Fortuny,  Sergio  Mota,  Georg  Peter,  Bent 
Pedersen,  Georgios  Theodoridis.  Without  their  generous  and  prompt  help  with  the  logistics  of  pre-
paid SIMs, the extent of field tests would have been severely restricted. 

Authors 

Chawdhry P., Folloni G., Lumachi S., Luzardi S. 

1 

 
 
Abstract 

In the context of the review of the EU Roaming Regulation, this document presents results of a study 
on  roaming  performance  assessment  by  field  measurements  on  mobile  broadband  involving  40 
mobile  networks  in  13  EU  countries.  JRC’s  mobile  app  netBravo  was  used  to  carry  out  the 
measurements and analysis of data. Download speed, upload speed and latency were measured for all 
roaming tests and results were analysed. The study found mixed results on the quality of service (QoS) 
in  roaming.  Customers  had  better  as  well  as  worse  QoS  than  at  home.  However,  customers  of  21 
mobile  networks  from  11  countries  at  least  once  had  worse  QoS  in  roaming  compared  to  at  home 
even when technical conditions were available for better quality. Such cases accounted for 25% of all 
roaming instances in the tests.  

2 

 
 
Glossary  

Bandwidth (of a network) 

The  capacity  of  a  network  communications  link  to  transmit  the  maximum  amount  of  data 
from  one  point  to  another  in  a  given  amount  of  time.  Available  bandwidth  are  set  by  the 
MNO separately for the downlink traffic and uplink traffic to optimise resource allocation 
for the overall quality of service in mobile broadband 

Customer SIM 

The SIM card issued by an operator to a specific customer under a contract 

Downlink  

The  network  communications  link  to  carry  the  traffic  from  an  end-point  towards  the  end-
user (the customer) 

Download (DL) by an end-user 

The  customer  activity  (generated  data  traffic),  which  requires  the  network  to  deliver  data 
from an end-point (an online service, database, website etc.) to the customer’s mobile device 

Domestic operator 

Home (network) operator 

EU 

European Union 

Home (network) operator 

The mobile network operator with which the end-user has a service contract. It provides the 
user with a SIM to enjoy the electronic communications services specified in the contract. 

Home  customer  /  Home  user 
(of a network) 

The end-user who has the service contract with a network operator (= MNO customer) 

Home  user  of  the  visited 
network 

A  SIM  used  in  its  own  country  and  own  network,  while  the  network  is  being  visited  by 
roaming users from other countries 

Home country 

The country where the end-user’s home operator is established and issues a network SIM for 
its electronic communications services 

International roaming 

Roaming by an end-user on a mobile network in another country 

Jitter 

Latency  

Mbps 

MNO 

Jitter is the irregular speed of different packets travelling in a telecom network between two 
end-points. It is important in real-time applications such as voice communications where it 
can result in packet arriving late and out of sequence, which can cause garbled sounds. 

The time it takes for a data packet to travel from its source to its destination and back to the 
source in an electronic communications network. Also known as the round trip time 

Megabits  per  second:  a  measure  of  data  transfer  rate  in  an  electronic  communications 
networks 

Mobile  Network  Operator,  an  operator  that  builds  its  own  network  and  has  full  control  of 
domestic usage on its network 

MNO customer 

The end-user who has the service contract with a mobile network operator  

MVNO 

Mobile Virtual Network Operator, an operator that rents access to a piece of the domestic 
operator’s network and does not build its own access infrastructure. 

3 

ms 

MS 

Millisecond, a measure of time used in electronic communications networks  

Member State 

Perceived quality 

The  quality  of  service  experienced  by  an  end-user  while  using  a  mobile  service,  without 
measuring it technically 

QoS 

QoE 

RLAH  

Quality of Service  

Quality  of  Experience  of  an  internet  access  service  involving  end-to-end  customer 
experience. Also known as QOS-3. 

Roam  like  at  home,  the  EU  initiative  ending  additional  charges  on  roaming  across  the  EU 
countries 

Roaming 

The ability of an end-user to connect to, and use services of, a mobile network other than 
(the one of) its own home operator’s 

Roaming customer  

A  customer  of  a  roaming  provider  of  regulated  roaming  services,  whose  contract  or 
arrangement permits Union-wide roaming 

Roaming peers  

Two or more roaming customers who happen to be visiting the same MNO abroad in the 
same location and at the same time  

SIM (card) 

Subscriber  Identification  Module, the  chip  which  provides  the  end-user  with  authenticated 
access to a mobile network when it is inserted in the end-user’s mobile telephone  

Visited (network) operator 

The operator that supplies services to the roaming end-user in a visited Member State 

Visiting SIM (card)  

The SIM card of a roaming user when it is connected to a visited network 

Uplink 

The  network  communications  link  to  carry  the  traffic  from  the  end-user  (the  customer) 
towards another end-point  

Upload (UL) by an end-user 

The  customer  activity  (generated  data  traffic),  which  requires  the  network  to  transfer  data 
from  the  customer’s  mobile  device  to  another  end-point  (an  online  service,  database, 
website etc.)  

User experience 

The  QoS  of  a  network  as  experienced  by  an  end-user  in  the  use  of  an  electronic 
communications service 

4 

 
 
 
List of figures 

Figure 1. Overall research methodology used to carry out roaming performance assessment 

Figure 2. Quality of Service in an end-to-end telecommunication network (Source: BEREC) 

Figure 3. Detailed test procedure adopted for roaming performance assessment 

Figure 4. Sets of Visited and Visiting Networks 

Figure 5. Checklist for test site selection 

Figure 6. Mapping the visiting SIMs on visited networks in a test location 

Figure 7. Mapping table between visiting SIMs and visited networks in a test location 

Figure 8. Test Round on a visited network, made of 10 Test Cycles 

Figure 9.  Map of roaming test sites 

Figure 10. For each customer SIM, its download performance on a visited network was assessed in 
relation to that on its home network 

Figure 11. MNO customer SIMs’ roaming performance compared to at home – Download speed 

Figure 12. Overall distribution of download performance of visiting SIMs in relation to their 
performance at home 

Figure 13. For each customer SIM, its upload performance on a visited network was assessed in 
relation to that on its home network 

Figure 14. MNO customer SIMs’ roaming performance compared to at home – Upload speed 

15 

19 

20 

23 

25 

26 

26 

28 

30 

31 

32 

32 

33 

34 

Figure 15. Overall distribution of upload performance of visiting SIMs in relation to their performance 
at home 

34 

Figure 16. For each customer SIM, its latency performance on a visited network was assessed in 
relation to that on its home network 

Figure 17. MNO customer SIMs’ roaming performance compared to at home - Latency 

35 

35 

Figure 18. Overall distribution of latency performance of visiting SIMs in relation to their performance 
at home 

36 

Figure 19. For each visited network, download performance of visiting SIMs was assessed in relation to 
38 
that of the own SIM of that visited network 

Figure 20. Download performance of visiting SIMs compared to the own SIM of the visited network  38 

Figure 21. Overall distribution of download performance of visiting SIMs in relation to visited network’s 
39 
own SIM 

Figure 22. For each visited network, upload performance of visiting SIMs was assessed in relation to 
that of the own SIM of that visited network 

Figure 23. Upload performance of visiting SIMs compared to the own SIM of the visited network 

39 

40 

Figure 24. Overall distribution of upload performance of visiting SIMs in relation to visited network’s 
own SIM 

40 

Figure 25. For each visited network, latency for visiting SIMs was assessed in relation to that for the 
own SIM of that visited network 

Figure 26. Latency for visiting SIMs compared to the own SIM of the visited network 

41 

41 

5 

 
Figure 27. Overall distribution of latency for visiting SIMs in relation to visited network’s own SIM  42 

Figure 28. Process of assessing the relative download performance of a visiting SIM in a visited 
network Vx in relation to the download performance of other visiting SIMs in that network 

43 

Figure 29. Download performance in roaming of customer SIMs of different MNOs in visited networks 
44 
in relation to other visiting SIMs in those visited networks 

Figure 30. Overall distribution of download performance of visiting SIMs in relation to other roaming 
SIMs in the same visited network 

45 

Figure 31. Process of assessing the relative upload performance of a visiting SIM in a visited network 
Vx in relation to the upload performance of other visiting SIMs in that network 

46 

Figure 32. Upload performance in roaming of customer SIMs of different MNOs in visited networks in 
relation to other visiting SIMs in those visited networks 

46 

Figure 33. Overall distribution of upload performance of visiting SIMs in relation to other roaming SIMs 
47 
in the same visited network 

Figure 34. Process of assessing the relative latency performance of a visiting SIM in a visited network 
Vx in relation to the latency of other visiting SIMs in that network 

48 

Figure 35. Latency in roaming of customer SIMs of different MNOs in visited networks in relation to 
other visiting SIMs in those visited networks 

48 

Figure 36. Overall distribution of latency for visiting SIMs in relation to other roaming SIMs in the same 
49 
visited network 

Figure 37. Customer SIMs with generally worse speed performance in roaming 

50 

Figure 38. Download performance in roaming of customer SIMs of different MNOs in relation to the 
own customers of the visited networks 

51 

Figure 39. Overall distribution of download performance of visiting SIMs in relation to visited network’s 
51 
own SIM 

Figure 40.  Problem cases identified in roaming tests 

52 

6 

 
List of tables 

Table 1. Mobile Networks and SIMs included in the tests .....................................................21 

7 

 
 
 
Executive Summary  

Regulation (EU) 531/2012 of the European Parliament and the Council of 13 June 2012 on roaming on 
public mobile communications networks within the Union, amended in 2015 and 2017 (hereinafter the 
'Roaming  Regulation'),  mandates  the  Commission  to  conduct  a  review  by  December  2019  of  the 
roaming  rules.  One  of  the  review  requirements  is  to  provide  an  assessment  of  the  availability  and 
quality of roaming services. The Commission’s Review report1 was adopted on 29 November 2019, and 
is followed by a legal proposal. 

Following  a  request  by  Directorate-General  for  Communications  Networks,  Content  and  Technology 
(DG  CNECT)  for  the  above  review  purpose,  the  Joint  Research  Centre  (JRC)  undertook  preliminary 
work, to assess the technical performance and quality of service (QoS) of EU roaming in a small sub-set 
of EU Member States (MSs) during the first year of the RLAH rules taking effect (i.e. between October 
2017 and August 2018). This work served as a scoping exercise for a systematic approach developed 
later on in the present study for further tests carried out between October 2018 and October 2019.  

In the present project, assessment of the performance of roaming has been based on real data from 
extensive  field  measurements  on  mobile  networks  in  a  subset  of  EU  MSs.  The  JRC  mobile  app 
netBravo2 was used to measure mobile network performance. The data thus collected was analysed to 
help answer relevant policy questions posed by DG CNECT. Results of the analysis are used to further 
inform the Impact Assessment for the review and prolongation of the Roaming Regulation3.  

Results from the present study by the JRC have contributed in terms of evidence-based assessment of 
roaming performance to the roaming review led by DG CNECT in 2020. Additional tests measuring the 
performance of MVNOs SIM cards when roaming were foreseen during the first half of 2020. Due to 
the COVID pandemic and travel restrictions those measurements had to be cancelled.  

The present project complements the work carried out by the JRC.I1 unit on the economic aspects of 
the roaming markets under the projects SMART 2018/0010 and SMART 2019/0004. 

What were the research questions? 

(1)  Whether  roaming  customers  have  different  quality  of  service  when  roaming  compared  to  the 
performance on their own home network. 

(2) Whether quality of service on visited networks differs between the customers of the visited network 
and the roaming users visiting the network.  

(3)  Whether  roaming  customers  have  different  quality  of  service  on  a  visited  network  compared  to 
other visiting roaming customers. 

(4) How often did customers have worse quality of  service  in roaming than at home, even  when the 
visited network was technically and practically able to provide better quality? 

1 Report on the review of the roaming market, COM(2019)616 final, and SWD(2019)416 available here. 

2 For further information and how to download netBravo, see the Annex to this Report and netbravo.jrc.ec.europa.eu.  

3 This Initiative is included in the 2020 Commission Work Programme addressing the specific objective “Digital for consumers” 
and has to be seen in the broader political context of creating a Europe Fit for the Digital Age. The Initiative contributes to the 
ambition to make the most out of the digital transition to enhance opportunities to connect, communicate, solve societal issues 
and do business. 

8 

 
 
 
 
                                           
What was tested? 

In the context of the EU Roaming Regulation, the JRC carried out a series of field measurements on 37 
mobile networks in 13 EU countries4  to assess the performance of roaming. Performance of 29 visiting 
SIMs from 12 EU countries was evaluated in relation to 1) their own home network, 2) in relation to the 
domestic SIM of the visited network, and 3) in relation to other visiting SIMs on the visited network. 

Where were the tests done? 

Altogether,  SIMs  of 40 different networks  were used  in field  measurements.  Three of these  networks 
did not happen to be visited by any visiting SIM therefore number of visited networks was 37. 

Number of visited countries: 

Number of visited networks: 

13 

37 

Number of visiting countries:                   12 

Number of visiting SIMs:  

29 

How were the tests done? 

Tests were carried out in a controlled manner using the JRC’s mobile app netBravo, which is designed 
to measure the coverage and quality of mobile networks and the performance of mobile broadband. 
All test data was stored and processed on the netBravo server with software adapted to the needs of 
the project. The app is available for free for smartphones running on Android® and Apple® iOS®. 

The  measurement  equipment  used  was  a  set  of  smartphones  of  the  same  model  (i.e.  the  same 
technical  specifications)  with  netBravo  app  installed.  The  handsets  used  were  suitable  for  a  4G+ 
network capability.  

Tests  were  done  in  key  city  locations  with  high  network  availability.  All  visiting  SIMs  on  the  same 
visited  network  were  tested  repeatedly  in  a  common  time  frame  to  ensure  the  network  traffic 
conditions were similar.  

The  measurement  data  on  download  speed,  upload  speed  and  latency  for  various  roaming  SIMs  in 
visited  countries,  as  well  as  at  home,  was  collected  and  analysed.  Comparisons  were  made  on  the 
performance of visiting SIMs vs home SIMs (of the visited network) and vs other visiting SIMs on the 
same visited network and in the same situations. The average values of download speed and upload 
speed observed in several test samples were used in comparing performance.  

4 At the time of the tests, the Roaming Regulation was still applicable in the UK. 

9 

 
 
 
 
 
 
 
 
                                           
Three types of analysis were performed on the measurement data set obtained in field tests: 

(a)   Customer-level analysis of roaming performance  
(b)   Network-level analysis of roaming performance 
(c)   Cross-correlation analysis between the above sets. 

What was found? 

Q1  –  Do  roaming  customers  have  different  quality  of  service  when  roaming  compared  to  the 
performance on their own home network? 

Looking at the overall data of 177 roaming instances between 37 visited networks and 29 visiting SIMs, 
the download speed for customer SIMs was worse in roaming than in their respective home network 
on 39% occasions. The upload speed was worse in roaming than at home on 59% occasions. Latency in 
roaming was found worse than at home in 62% cases. 

Q2 – Does the quality of service on visited networks differ between the customers of the visited 
network and the roaming users visiting the network? 

Across  the  37  networks  visited  by  29  visiting  SIMs,  results  from  field  tests  show  that  the  download 
speed  for  visiting SIMs (roaming customers) was worse than the home SIM (home  customer) of the 
visited  network  in  50%  of  the  roaming  instances.  The  upload  speed  for  the  visiting  SIMs  was  worse 
than the home SIM in 72% of the roaming instances. Latency for visiting SIMs was found worse than 
the home SIM in 73% cases. 

Q3  –  Does  the  roaming  customer  have  different  quality  of  service  on  a  visited  network 
compared to other visiting roaming customers? 

In relation to the roaming peers (other roaming users in the same visited network, at the same location 
and  at  the  same  time),  there  was  even  distribution  of  instances  of  roaming  performance  with  below 
average  and  above  average  download  speeds  for  the  visited  networks:  38%  had  better  performance 
than the average of all roaming users5, 40% had worse performance and 22% had about the average 
level  of  performance.  The  same  was  observed  for  the  distribution  of  upload  performance:  35%  had 
better  than  average  performance,  39%  had  worse  than  average  and  26%  had  average  level  of 
performance.  For  latency  the  distribution  was  36%  above  average,  32%  below  average  and  32% 
average. 

The  figures  were  similar  when  considering  only  the  cases  where  a  network  was  visited  by  at  least  5 
visiting SIMs. It shows that the test results were overall statistically consistent. 

However,  it  was  found  that  some  visiting  customer  SIMs  were  generally  more  likely  to  have  below 
average performance than others, pointing towards possibly QoS in roaming being offered to them at 
an unfavourable level. 

Q4  –  How  often  did  customers  have  worse  quality  of  service  in  roaming  than  at  home,  even 
when the visited network was technically and practically able to provide better quality? 

On further analysing the results for Q1-Q3, it was found that 20% roaming customers (6 out of the 29 
visiting  SIMs)  had  generally  worse  download  performance  in  roaming  than  at  home  and  also  below 
average download performance than the other roaming customers in the same visited network.  

Detailed  analysis  showed  that  21  customers  from  11  countries  had  worse  download  speed 
performance than at home and also worse than what was technically possible on at  least one visited 
network. Altogether, such cases accounted for 25% of all 177 roaming instances. 

5  For  a  given  visited  network,  the  average  level  of  download  speed  was  calculated  based  on  the  download  speed  for  all  the 
visiting SIMSs for that visited network.  

10 

 
                                           
Conclusions 

Objective assessment of roaming performance is a complex exercise. There are several variables such 
as the QoS of the home network, QoS of the visited network, and contractual terms on the QoS both 
on wholesale level between the mobile operators as well as for a given customer. Moreover, there are 
temporal  and  spatial  variables  such  as  the  traffic  density  and  user  density  of  the  network  that  cause 
the  network  performance  to  fluctuate  over  time  and  across  different  locations.  These  result  in  the 
roaming performance to vary considerably for the visiting SIMs.  

For a single customer, home v. roaming performance can be anecdotal or episodic. The same SIM can 
have better performance than at home for one visited  network and worse than at home  for another 
visited network depending on the test conditions (time, place, traffic) as well as inherent performance 
specifications  of  each  network  and  the  QoS  criteria  and  constraints  that  may  be  applied  by  the 
network operators.  

Measuring a visiting SIM’s performance in relation to the performance of the home SIM of the visited 
network was found to be a useful yardstick to neutralize the effect of some of the above temporal and 
spatial variables. Doing so with a number of visiting SIMs added statistical diversity in the data  set as 
well as a broader view of the complexity involved in analysing cross-border roaming. 

Measuring the performance of a visiting SIM with respect to other visiting SIMs was yet another way to 
benchmark relative performance across several spatial locations.  

This study is quite unique of its kind in terms of the scope, evidence-based methodology and scale of 
field tests. The tests had involved nearly half of the EU countries and about a third of all EU networks, 
taking into  account the distribution between inbounder  and outbounder roaming  countries6, making 
them representative and to show a genuine picture of the diversity in roaming performance across the 
EU. 

6 An outbounder operator has a customer base which consumes more mobile services abroad (i.e. on the networks of partner 
operators  in  other  EU/EEA  countries),  than those  consumed  by  the partner  operators’  customer  base  on  its  own  network  (i.e. 
when acting as a visited network). Conversely, an inbounder operator has a customer base which consumes less mobile services 
abroad than those consumed by the partner operators' customer base on its own network. 

11 

 
 
 
                                           
The relative performance of downloading data in roaming for most of the visiting SIMs was a mix of 
better/worse/same  in  relation  to  the  benchmarks  devised.  However,  the  upload  performance  and 
latency were found overwhelming in favour of the home SIMs.  

The  study  identified  44  instances  of  download  performance  in  roaming  (specific  visiting  SIMs  on 
specific  visited  networks)  that  cannot  be  explained  through  variables  such  as  temporal,  spatial  or 
technical  characteristics  alone.  For  such  cases,  accounting  for  25%  of  all  roaming  instances,  more 
transparent  information  on  the  QoS  applied  on  a  visiting  SIM  could  help  to  explain  the  causes  for 
poorer QoS in roaming than at home.  

12 

 
 
 
1 

Introduction 

Since  the  introduction  of  the  first  generation  of  public  mobile  networks  in  the  late  1980s,  mobile 
electronic  communications  today  span  the  globe  with  all  192  countries  covered  by  one  or  more 
national  mobile  networks.  Each  national  mobile  network  is  operated  by  a  mobile  network  operator 
(MNO) licensed to provide mobile services to its customers within the national territory.  

Mobile networks are interconnected to allow a customer in one network to place calls to a customer in 
another  mobile  network  in  the  same  country  or  in  another  country.  This  provides  a  global  reach  for 
customers of a national mobile network operator. 

When  a  mobile  customer  is  located  outside  the  coverage  area  of  its  own  MNO,  the  mechanism  of 
roaming allows that customer to use services of another MNO, provided that an arrangement exists for 
inter-network  mobility  between  the  two  network  operators.  In  popular  terms,  it  is  called  roaming 
between mobile networks. 

International  roaming  across  national  borders  is  a  functional  necessity  to  provide  customers  with 
access to mobile services when they travel abroad. 

With  increased  affordability  of  mobile  telecom  services  through  subscriptions  or  pre-paid  contracts 
and lower cost of mobile telephone handsets, the last 20 years have seen a rapid growth in the uptake 
of  mobile  services.  Consequently,  the  demand  for  international  roaming  has  also  soared  with 
consumers travelling abroad on business and holidays.  

Since  the  early  years,  the  cost  of  international  roaming  has  been  generally  exorbitant  leading  to 
unexpected bills for many customers. Today with greater competition and regulatory intervention, the 
cost of  Union-wide roaming has become  more predictable especially in the context of  the European 
Single Market.  

Beyond  the  issue  of  the  cost  of  roaming,  a  major  question  is  about  the  varying  level  of  quality  of 
service  for  consumers  in  roaming.  This  question  is  particularly  relevant  in  the  context  of  the  EU 
Roaming Regulation. The present project has focused on the assessment of roaming performance  for 
consumers in a number of mobile networks across a number of EU Member States.  

1.1  Roaming in the EU  

Roaming,  as  defined  by  the  Roaming  Regulation,  is  a  service  that  allows  a  customer  (consumer  or 
business)  of  a  public  Mobile  (Virtual)  Network  Operator  (M(V)NO)  in  one  EU  country  (country  A)  to 
have  access  to  mobile  services  (voice,  SMS  or  data)  from  a  Mobile  Network  Operator  (MNO)  when 
travelling in another EU country (country B).  

The Operator A ensures that its customers remain connected to  a mobile network of the Operator B 
when travelling abroad while using  the same  mobile  handset  (or possibly  laptop  or  tablet in  case of 
data  roaming)  and  the  same  phone  number.  Operator  A,  that  wants  to  offer  roaming  services  to  its 
customers  ("retail  roaming  services")  in  country  B,  has  to  buy  these  services  from  a  Mobile  Network 
Operator (MNO) (Operator B) located in the visited country B through commercial wholesale roaming 
agreements ("wholesale roaming services").  

In practice, when a customer of Operator A places a call or uses mobile data while roaming abroad in 
country B, that service is provided by an Operator B in the visited country B. The roaming customer’s 
home  Operator  A  has  to  pay  the  visited  Operator  B  for  that  service.  (“Wholesale  roaming  charges”). 
The level of wholesale roaming charges is capped by the Roaming Regulation (for data the price caps 
is decreasing each year, since there is a glide path). 

13 

 
1.2  Regulatory context 

In  October  2015,  the  European  Parliament  and  the  Council  adopted  Regulation  (EU)  2015/2120, 
amending  the  Roaming  Regulation  (EU)  531/2012,  and  mandating  the  end  of  retail  roaming 
surcharges in the EU from 15 June 2017, subject to fair use policy and sustainability derogation (also 
known as "Roam-Like-At-Home" or RLAH rules).  

Regulation  531/2012  of  13  June  2012  of  the  European  Parliament  and  the  Council  on  roaming  on 
public mobile communications networks within the Union, amended in 2015 and 2017 (hereinafter the 
'Roaming  Regulation'),  mandates  the  Commission  to  conduct  a  review  by  December  2019  of  the 
Roaming Regulation.  

The  Roaming  Regulation  mandates  the  Commission  to  submit  biennial  reports  to  the  European 
Parliament  and  the  Council,  accompanied,  if  appropriate,  by  a  legislative  proposal.  Moreover,  the 
Roaming  Regulation  specifies  the  list  of  elements  to  be  assessed  by  the  Commission  in  this  report. 
One of them is the availability and quality of roaming services in the EU. 

The  Commission’s  Review  report7  was  adopted  on  29  November  2019,  and  is  followed  by  a  legal 
proposal. 

The  present  project  contributed  to  the  review  of  the  quality  of  roaming  services  and  to  inform  the 
Impact  Assessment  on  the  review  and  prolongation  of  the  Roaming  Regulation8.  In  view  of  the 
roaming review requirements, a number of tasks for scientific and technical support were identified by 
DG CNECT where the JRC agreed to contribute and bring its expertise in evidence-based performance 
evaluation and data analysis. 

1.3  Research questions 

The two main research questions addressed in the present study were:  

1. 

2. 

Whether roaming customers have different quality of service when roaming compared 
to the performance on their own home network.  
Whether quality of service on visited networks differs between the customers of the 
visited network and the roaming users visiting the network.  

As part of the research methodology, the following two questions were added: 

3. 

4. 

Whether roaming customers have different quality of service on a visited network 
compared to other visiting roaming customers.  
How often did customers have worse quality of service in roaming than at home, even 
when the visited network was technically and practically able to provide better quality? 

In  a  related  experimental  work9,  network  configuration  for  roaming  was  assessed  for  11  mobile 
networks  in  six  countries.  Authors  investigated  the  roaming  performance  in  voice-over-IP  (VoIP) 
networks as well as content access policy of the operators. Latency penalty of c. 60ms was observed for 
roaming. However statistical comparison of quality of service in roaming vs home for downloads and 
uploads speeds was not reported.  

7 Report on the review of the roaming market, COM(2019)616 final, and SWD(2019)416 available here. 

8 See the Inception Impact Assessment on the roaming review initiative for more information, here.  

9 Experience: Implications of roaming in Europe, Anna Maria Mandalari*, Andra Lutu, Ana Custura, Ali Safari Khatouni, Özgü Alay, 
Marcelo Bagnulo, Vaibhav Bajpai, Anna Brunstrom, Jörg Ott, Marco Mellia, Gorry Fairhurst, MobiCom18 Proceedings of the 24th 
Annual International Conference on Mobile Computing and Networking, pp 179-189, Oct 2018.  

14 

 
 
                                           
1.4  Research Methodology 

An overall research methodology, as shown in Figure 1, was adopted to answer the above questions. It 
was  based  on  the  collection  of  evidence  of  roaming  performance  from  field  tests  on  a  number  of 
networks, creation of relevant data sets, analysis of the data and finally to draw conclusions. It included 
development of detailed methodologies for measurements and analysis, described in later sections of 
the report. 

Field tests of roaming 
in several mobile 
networks in the EU 

Create data sets 
relevant to the 
research questions 

Analyse data sets 

Draw conclusions 

Selection of visited 
networks 

Selection on visiting 
SIMs 

Performance of 
several visiting SIMs 
and Home SIM in a 
visited network 

Performance of a SIM 
at Home and in 
several visited 
networks 

Analytical method for 
Q1 

Summarise 
observations from 
the analysis of 
questions Q1-Q4 

Analytical method for 
Q2 

QoS parameters 

Cross-correlation 
between above 
datasets 

Analytical method for 
Q3 

Analytical method for 
Q4 

Measurement tool 

Measurement 
protocol 

Mobile equipment 

Figure 1. Overall research methodology used to carry out roaming performance assessment 

15 

 
2  Work Description 

2.1 

 WP1: Data Collection 

Objective: To carry out field tests on mobile broadband  in order to collect data to compare roaming 
performance in several EU MSs. 

Description: 

The  JRC  undertook  field  test  campaigns  in  the  following  13  EU  Member  States:  Spain,  Italy,  Latvia, 
Belgium, Greece, France, Germany, Poland, Romania, Finland, Netherlands, Sweden, UK10. 

The JRC procured subscriber identification modules (SIMs) for 40 mobile network operators (MNOs) in 
the following 13 EU MSs (the number of SIMs for each MS are shown in parenthesis)11: 

Spain (3), Italy (4), Latvia (2), Belgium (2), Greece (3), France (3), Germany (3), Poland (4), Romania (4), 
Finland (2), Netherlands (3), Sweden (3), UK (4). 

Ten  advance  generation  mobile  handsets  (Samsung  S9®)  were  procured  to  run  the  tests.  The  JRC’s 
mobile broadband test application netBravo was installed on these handsets to carry out roaming test 
campaigns on mobile networks in those countries. 

netBravo  was  used  to  measure  the  performance  of  visited  networks  for  the  visiting  SIMs  as  well  as 
home SIMs of the visited networks. 

These test campaigns were organised to collect adequate data at suitable locations to allow objective 
analysis  of  the  relative  performance  of  mobile  networks  and  their  compliance  with  the  roaming 
regulation. 

JRC  elaborated  appropriate  test  and  measurement  procedure  for  structured  field  test  campaigns, 
according  to  project  objectives  and  regulatory  requirements  for  robust  and  comprehensive  roaming 
performance assessment.  

Results of test campaigns were shared with DG CNECT on an on-going basis for which JRC adapted its 
web application for netBravo for interactive visualization of roaming test data. Security mechanism was 
implemented limiting access to the roaming test data for authorised users only.  

Activities carried out:   

1.  Adapting JRC’s mobile application netBravo for roaming performance tests; 

2.  Acquisition and pre-paid subscription management of SIMs for various mobile networks in the 

13 visited countries, 

3.  Definition of the exact measurement protocol for field tests; 

4.  Organisation of  test campaigns and  collection of mobile broadband performance data using 
netBravo  mobile  application  for  SIMs  of  various  national  mobile  networks  in  home  and 
roaming mode; 

5.  Rendering  of  collected  roaming  data  on  the  JRC’s  netBravo  web  application  in  tables  and 

graphs. 

10 At the time of the tests, the Roaming Regulation was still applicable in the UK. The roaming traffic in the UK was therefore still 
treated as Union-wide roaming traffic.   

11 Of the 40 SIMs, only 29 were used as the visiting SIMs in roaming; the others were used only as home SIMs of visited networks 

16 

 
 
                                           
2.2  WP2 : Data Analysis 

Objective:  Processing  and  analysis  of  individual  and  relative  performance  of  mobile  broadband  in 
various mobile networks for their home and roaming users. 

Description: 

JRC carried out this work in consultation with DG CNECT to formulate and answer analytical questions, 
based on field evidence collected in WP1, on roaming performance assessment. 

The work under this work package consisted of building a database from the data collected from tests 
on individual mobile network operators (MNOs) during field tests. DG CNECT advised on the analytical 
criteria used to assess both individual and relative performance of mobile networks.    

The two main initial research questions addressed in this analysis were:  

(a)  Whether roaming customers have different quality of service when roaming compared 

to the performance on their own home network.  

(b)  Whether quality of service on visited networks differs between the customers of the 

visited network and the roaming users visiting the network.  

As part of the research & analysis methodology, the following two questions were added: 

(c)  Whether roaming customers have different quality of service on a visited network 

compared to other visiting roaming customers.  

(d)  Did customers have worse quality of service in roaming than at home, even though the 

visited network was technically and practically able to provide better quality? 

The  JRC  adapted  its  netBravo  database  and  website  to  perform  data  processing,  analysis  and 
visualization in order to provide a fair assessment of the mobile broadband regime in roaming across 
the  EU,  based  on  the  statistical  evidence  collected  during  field  tests.  For  this  purpose,  the  analysis 
framework applied to the data was jointly agreed between DG CNECT and the JRC.  

Results  were  produced  in  appropriate  tables,  charts,  and  where  possible  maps,  to  appropriately 
visualise the data and analytical results from a policy perspective. 

The relevant results, analyses, and the associated tables, charts and maps, were used by DG CNECT in 
the  Impact  Assessment  Report  accompanying  the  legal  proposal  on  roaming  on  public  mobile 
communications networks within the Union. 

Activities and Deliverables: 

1.  Adapting netBravo website to create a database of roaming performance tests; 
2.  Statistical analysis of test results to answer the review questions on the performance of mobile 

broadband roaming in the EU; 

3.  Production of tables, graphs and accompanying explanation along with online visualization of 

the analytical results. 

17 

 
 
 
2.3  WP3: Report 

Objective: Drafting a report describing the methodology,  analysis and conclusions of the work done 
by JRC.E2 

Description: 

The  present  report  was  prepared  describing  the  test  methodology,  analytical  results  and  technical 
conclusions of mobile broadband roaming performance. Relevant parts of the report were used by DG 
CNECT in the impact assessment report accompanying the roaming legislative proposal. JRC produced 
tables  and  graphs  on  an  ad-hoc  basis  to  be  included  in  relevant  policy  documents.  JRC  has  also 
contributed with explanatory text within the scope of this project, to the drafting of such documents 
by the Commission services. 

Activities carried out: 

1.  Methodological and analytical Report;  

2.  Tables,  charts,  and  explanatory  text  for  the  Commission's  reports  related  to  the  roaming 

review and impact assessment; 

3.  The present Final Report of the project. 

18 

 
3  Test Methodology 

A  systematic  process  was  followed  to  develop  and  apply  a  test  method  to  carry  out  roaming 
performance assessment in the project. 

3.1  Definition of the scope of tests 

The  scope  of  roaming  tests  was  to  measure  quality  of  service  of  mobile  broadband  for  roaming 
customers on a visited network and to compare it with that for the home users of the visited network.  

In  particular,  EECC12  Article  104  deals  with  the  quality  of  service  related  to  internet  access  services; 
EECC  Annex  X  sets  out  the  QoS  parameters  for  internet  services  as  defined  in  ITU  standard  ITU-T 
Y.2617. BEREC guidelines13 contribute to a consistent application of Article 104(2) and Annex X.  

Figure 2. Quality of Service in an end-to-end telecommunication network (Source: BEREC) 

In  the  present  study,  the  focus  was  on  mobile  broadband  roaming  rather  than  on  voice 
communication. Hence data transfer speed for download and upload activities were measured. Latency 
was also measured to highlight the effect of traffic steering often used by operators for roaming which 
would introduce additional end-to-end delay. Jitter was not measured for mobile broadband tests as it 
is more relevant for voice communication rather than for data communication. 

In  addition,  net  neutrality  tests14  were  carried  out  however  those  tests  are  beyond  the  scope  of  the 
present  report.  This  report  therefore  presents  the  speed  results  and  latency.  During  the  study  it  was 
found that apart from the volume of data transfer allowed, speed was usually the only QoS parameter 
publicised in retail offers to pre-paid consumers.  

12 DIRECTIVE (EU) 2018/1972 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 11 December 2018 establishing the 
European 
https://eur-lex.europa.eu/legal-
Electronic 
content/EN/TXT/PDF/?uri=CELEX:32018L1972&from=EN 

Communications 

Code 

BEREC 

13 
https://berec.europa.eu/eng/document_register/subject_matter/berec/download/0/9043-berec-guidelines-detailing-quality-of-
se_0.pdf  

Guidelines 

detailing 

Quality 

Service 

Parameters 

of 

14 Net neutrality refers to the principle of equal treatment of data traffic being transmitted over the internet, i.e. that the  ‘best 
efforts’  are  made  to  carry  data,  no  matter  what  it  contains,  which  application  transmits  the  data  (“application-agnosticism”), 
where 
guidelines 
it 
https://berec.europa.eu/files/document_register_store/2016/8/NN%20Factsheet.pdf) 

comes 

BEREC 

About 

where 

goes. 

from 

(see 

or 

it 

19 

 
 
                                           
3.2  Development of a detailed test method 

A  detailed  test  method  was  developed  and  implemented  in  two  phases  involving  in-field 
measurements, as follows: 

(a) Feasibility tests 

The goal of the feasibility phase was to develop and refine the test methodology in both technical and 
logistical sense and validate it in the light of in-field performance and preliminary data analysis. 

(b) Detailed tests 

After validation of the test methodology, a more comprehensive set of roaming tests was undertaken.  

Figure  3  shows  the  various  steps  that  were  followed  to  implement  and  validate  the  testing 
methodology leading to a comprehensive set of test data that were finally analysed for the assessment 
of QoS of mobile broadband in roaming. These steps are outlined below.  

1.Selection of a 
measurement tool 
for QoS 

8. Calibration of 
handsets 

9.Establish ideal 
test conditions 
and test locations 

2. Selection of 
handsets 

7. Update of 
handsets 

3. Selection of 
countries 

6. Preliminary 
tests 

10. Mapping of 
roaming SIMs 
onto host network 

11. Field tests by 
home country and 
host network 

4. Identification of 
network operators 

5. Purchase of 
operator SIMs 

12. Upload and 
review of data 

Figure 3. Detailed test procedure adopted for roaming performance assessment 

1)  Selection of a measurement tool for QoS 

The QoS measurement tool forms the most important part of data collection. The project decided to 
use netBravo15, an in-house android app developed at JRC. It allows the test engineer to query current 
technical parameters of the host mobile network, carry out broadband speed test, visualize and store 
the test data and upload test results to the netBravo server for data aggregation and further analysis. 

2)  Selection of handsets 

Mobile  handsets  (smart  phone)  differ  in  terms  of  their  sensitivity  to  the  network  signal  and 
performance. Therefore it was decided to select a set of identical smartphones that could operate up 
to the maximum rated speed of the host network. Initially, a number of Samsung S4® handsets were 
selected  due  to  their  ready  availability  in  the  project.  During  the  feasibility  phase,  it  was  found  that 
some host networks were able to deliver speeds above that of Samsung S4® (70 mbps). As a result, 
handsets  of  higher  specifications  were  tested.  Eventually  ten  identical  Samsung  S9®  handsets  were 

15 netbravo.jrc.ec.europa.eu 

20 

 
 
                                           
used  for  field  tests  and  data  collection  as  these  could  operate  even  on  the  so  called  4G+,  the 
enhanced version of 4G networks.  

3)  Selection of MSs 

For a Single Market, at that time consisting of 28 Member States, it was considered sufficient to carry 
out  tests  in  roughly  half  as  many  countries  in  different  geographic  regions,  taking  into  account  the 
distribution  of  inbounder  and  outbounder  roaming  countries,  to  get  a  general  picture  of  roaming 
performance. For the sake of scientific neutrality, the paper refers to them through country codes. 

For a Single Market, at that time consisting of 28 Member States, it was considered sufficient to carry 
out  tests  in  roughly  half  as  many  countries  in  different  geographic  regions,  taking  into  account  the 
distribution  of  inbounder  and  outbounder  roaming  countries,  to  get  a  general  picture  of  roaming 
performance. For the sake of scientific neutrality, the paper refers to them through country codes. 

Thirteen EU Member States as shown in Table 1 were selected for tests: 

Number of visited countries:  

13 

Number of visiting countries:  

12 

Table 1. Mobile Networks and SIMs included in the tests 

Country  

Belgium 

Finland 

France 

Germany 

Greece 

Italy 

MNO  Anon. 
Code 

Used  as  Visited 
Network 

Used  as  Visiting 
SIM 

Pre-paid 
roaming 

with 

BE_A 

BE_B 

FI_A 

FI_B 

FR_B 

FR_C 

FR_A 

DE_A 

DE_B 

DE_C 

EL_A 

EL_B 

EL_C 

IT_B 

IT_C 

IT_D 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

No 

Yes 

No 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

No 

Yes 

No 

Yes 

Yes 

Yes 

Yes 

Yes 

No 

No 

Yes 

Yes 

Yes 

21 

 
 
 
 
 
 
 
 
 
 
 
Latvia 

IT_A 

LV_A 

LV_B 

Netherlands 

NL_A 

Poland 

Romania 

Spain 

Sweden 

UK 

NL_B 

NL_C 

PL_A 

PL_B 

PL_C 

PL_D 

RO_A 

RO_B 

RO_C 

RO_D 

ES_A 

ES_B 

ES_C 

SE_C 

SE_B 

SE_A 

UK_A 

UK_B 

UK_C 

No16 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

No16 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

No16 

UK_D 

Yes 

16 None of the visiting SIMs selected this network in roaming 

Yes 

Yes 

No 

Yes 

Yes 

Yes 

No 

No 

No 

No 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

No 

No 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

No 

Yes 

Yes 

Yes 

No 

No 

No 

No 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

No 

No 

Yes 

Yes 

Yes 

Yes 

Yes 

22 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
                                           
4)  Selection of network operators 

There were typically between two  and four  mobile network operators (MNO) per country. Subject to 
the availability of their SIMs, MNOs were selected to test QoS of visiting SIMs and home users.  

Altogether, 40 SIMs were used over various tests, but not all of them were used  on all the locations. 
This was due to gradual procurement of SIMs as each country was visited one after another. Of these, 
37 SIMs corresponded to the visited networks and 29 to the visiting networks. 26 SIMs corresponded 
to both the categories. Three networks acted as visiting only as no visiting SIM chose them as a visited 
network.  Similarly,  eleven  networks  acted  as  visited  only  as  they  did  not  have  roaming  options  but 
were used by visiting SIMs for their roaming performance.  

Total number of Network SIMs used: 40 

Number of visited networks: 37 

Number of visiting networks (visiting SIMs): 29  

Number of networks which acted as both visiting & visited: 26 

The study does not include tests for MVNOs. Additional tests measuring the QoS of MVNOs SIM cards 
when roaming were foreseen during the first half of 2020. Due to the COVID-19 pandemic and travel 
restrictions those measurements had to be cancelled.  

37  MNOs  which 
visited 
were 
networks 

29  MNOs  which 
were 
Visiting 
SIMs providers 

26  MNOs  which 
visited 
were 
networks  as  well 
as  visiting  SIMs 
providers 

Figure 4. Sets of Visited and Visiting Networks 

5)  Subscriber identification modules (SIMs) 

The  project  used  pre-paid  SIMs  wherever  they  were  available  for  the  MNOs  within  the  selected 
countries. Each such SIM represented a unique user belonging to a given MNO. When a SIM was used 
in  its  own  country  and  own  network,  it  acted  like  a  ‘home  user  for  the  visited  network’  whereas 
when it was used in a country other than its home, it acted like a ‘roaming user’. Altogether 40 SIMs 
issued by different MNOs were deployed in roaming tests. These are listed in Table 1 above. 

23 

 
 
 
 
 
6)  Preliminary tests – identification of logistic and technical issues 

In  preliminary  tests,  several  logistic  and  technical  challenges  were  identified.  Among  these  were  the 
issues such as pre-paid contract period, manual or auto-renewal, costs and data traffic allowance, top-
up procedures, language of the MNO web site and app, etc. There were important constraints such as 
maximum data rate allowed in a pre-paid contract, the data allowance and roaming restrictions. 

In most countries, an EU passport or identity card was sufficient to buy a pre-paid SIM. The pre-paid 
contract period was typically one month after which the contract terms and top-up procedure varied 
considerably  from  automatic  renewal  to  manual  top-up  and  selection  of  a  data  package.  The  data 
allowance varied across the SIM operators from 500 MB to several GB per month. In some cases data-
only  packages  were  available,  in  others  data  plus  voice  calls  plus  SMS.  In  some  cases,  prep-paid 
packages were available for only national traffic (home use) but not in roaming. In some cases, only a 
credit card issued in the home country could be used for top up. Such issues presented a continuous 
challenge to top up SIMs and keep them active during the course of project. 

7)  Update of handsets 

For  the  upgrade  of  test  equipment,  advanced  handsets  (Samsung  S9®)  were  selected,  capable  of 
speeds up to 300 Mbps on a 4G+ network. 

Ten identical  Samsung S9® units were purchased  to be  able to  test operating  network performance 
for up to ten SIMs in a single test session, consisting of one home SIM and 9 visiting SIMs, all hosted 
on the same operating network.  

The reason to choose identical handsets was to eliminate variations of performance between different 
models of smartphone. 

8)  Calibration and software updates of handsets 

Even  though  the  test  handsets  were  of  the  same  type,  to  eliminate  variations  between  them,  all 
handsets were tested in a benchmarking and calibration process. 

All  handsets  were  tested  on  the  same  network  and  using  the  same  SIM  in  a  session.  Each  handset 
performed the speed test ten times and results were compared for minimum, maximum and average 
speeds registered by each handset. 

The tests were carried out in a commercial 4G network cell within the JRC on a Saturday morning. At 
this time, the occupancy of the site was very low and background network cell traffic was expected to 
be low. 

Furthermore,  any  updates  to  the  system  software  on  handsets  were  checked  for  availability  and 
downloaded before each test campaign. 

9)  SIM top up, verify test conditions and test locations 

Before start of a test campaign, all the SIMs were topped up for the expected amount of data traffic 
consumed by netBravo. On a 4G network it is typically 50+ MB per speed test requiring a minimum of 
500 MB for a test campaign consisting of 10 cycles of speed tests.  

To make a reliable comparison between the QoS of a home user’s SIM and roaming users’ SIMs  – all 
accessing the same network – network conditions play an important role. First of all, the test location 
needs to have good network coverage with high bandwidth available. That would usually mean a  city 
centre  (outdoor)  or  airport  location  where  cells  are  provisioned  with  high  capacity  and  nearby  base 
stations by most of the network providers. 

Test location: In case of poor quality network signal, indicating poor network coverage, the speed of 
networks would be sporadic and poor for everyone. A comparison of such ‘roaming performance’ with 
‘home performance’ of a visiting SIM would be meaningless. Any difference in performance in such a 
situation would be statistically insignificant and results will not be technically representative.  

24 

Timing of tests: Similarly, even in case of good quality network signal, the network traffic tends to be 
sporadic  during  busy  hours  i.e.  it  varies  substantially  from  one  moment  to  the  next,  as  well  as  the 
network speed being generally a lot slower than the rated speed. For example, at an airport or in a city 
centre  location,  high  user  density  during  peak  hours  would  mean  the  network  conditions  to  be  far 
from ideal to make meaningful comparative tests between roaming and home user SIMs. On the other 
hand,  tests  during  peak  hours  could  provide  a  clue  towards  equitable  treatment  of  the  visiting  SIMs 
and  home  network  SIMs.  However,  the  variation  of  results  from  one  speed  test  to  the  next  was  so 
much that this option was not considered very useful. 

Above all, the location should have 4G (or 4G+) network available with a high capacity for broadband 
traffic.  

By experience, the ideal test  environment to  test a commercial network in a reproducible way would 
satisfy the following conditions: 

-  Availability of 4G or 4G+ signals (to allow any 3G vs 4G roaming policy) 
- 
- 
- 
- 

good signal quality: -85 dBm or better  
high network capacity: e.g. city centre, airport arrivals/departure hall; shopping mall 
low traffic congestion in the network cell: e.g. late evening or early morning 
logistical feasibility to set up test equipment without disturbance for one hour or more per 
visited network.  

Does the location 
have a 4G or 4G+ 
network available? 

Is the signal strength 
good  

(c. -85 dBm)? 

Does the network 
traffic seem low and 
stable at this time?  

(low variability over 
several samples of 
speed test) 

Does this location 
have high network 
capacity? 

(Home SIM able to 
deliver speeds c. 100 
Mbps) 

Right logistics to do 
tests over a few 
hours? 

(work space, privacy, 
duration of use)   

Figure 5. Checklist for test site selection 

10)  Mapping of roaming SIMs onto host networks 

The home vs roaming tests required, first of all, to identify the preferred visited network for each of the 
visiting  SIMs.  After  the  visiting  SIM  was  placed  into  the  test  phone,  the  phone  was  restarted,  as 
required by Android. In most cases, the visited network was selected automatically. In some cases, the 
visited network was set quickly on restarting the test phones. In some cases, it took a few minutes. In 
some  cases,  automatic  selection  did  not  work  and  the  visited  network  had  to  be  found  manually  by 
trial and error.  

The  mapping  process  allowed  a  common  round  of  testing  all  the  visiting  SIMs  preferring  the  same 
visited  network.  That  meant  keeping  a  common  time  window  (and  common  network  conditions)  for 
comparative  tests  and  avoiding  statistical  variations  due  to  changes  in  network  conditions.  It  also 
allowed real time observations on the performance of various SIMs in the same visiting network.  

25 

 
 
All visiting SIMs were mapped in this way to a specific visited network. A mapping table was prepared 
in this way per visited network before commencing a test session.  

Place the 
roaming SIM in 
a test phone 

•Start the 

mapping process 
for each roaming 
SIM  

Restart the 
phone 

•Required by 
Android  

Wait for visited 
network to be 
selected 

•Automatic 
selection 
•Else, manual 
selection  

Enable  
roaming data 

• Mobile data 

icon shows on 
the screen 

Verify network 
conditions 

•Check network 
parameters in 
netBravo 

Figure 6. Mapping the visiting SIMs on visited networks in a test location 

Set 1 

Set 2 

Set 3 

Visited Country X 

Visited Country X 

Visited Country X 

Operating Network  NetX1 

Operating Network NetX2 

Operating Network NetX3 

•Roaming SIM 1 
•Roaming SIM 2 
•Roaming SIM 5 
•... 
•Home SIM Op Net X1 

•Roaming SIM 3 
•Roaming SIM 6 
•Roaming SIM 7 
•... 
•Home SIM Op Net X2 

•Roaming SIM 4  
•Roaming SIM 8 
•Roaming SIM 9 
•... 
•Home SIM Op Net X3 

Figure 7. Mapping table between visiting SIMs and visited networks in a test location 

26 

 
 
 
 
11)  Preparation of phones for field tests in a visited host country   

Once  all  visiting  SIMs  have  been  mapped  to  the  visited  networks  (one  of  the  operating  networks 
(MNOs) in the visited country), each set of SIMs per visited network is prepared for testing in a single 
batch. In the figure above, assuming there being three MNOs in the visited country, three sets of SIMs 
were identified – one for each visited network. Numbers of visiting SIMs for each visited network were 
not the same; it depended on the roaming agreements between cross-border MNOs. 

There were ten test phones available therefore a set of up to 10 SIMs for a given visited network (one 
home SIM of the visited  network and  the rest visiting SIMs on the same network) was prepared at a 
time as follows.  

1) 
2) 
3) 

4) 
5) 
6) 

7) 

8) 

Off 
Off 
On 

Insert one SIM per test phone (ten identical test phones were available). 
Power on (or restart) all the phones with SIMs inserted 
Check settings on all phones:  
a.  Flight mode Off 
b.  Mobile data On 
c.  Wi-Fi®  
d.  Bluetooth®  
e.  Data roaming  
f.  Network mode: auto connect (4G LTE/3G/2G) 
g.  Status icon: ‘Roaming flag’ R visible for visiting SIMs 
h.  Status icon:  ‘Mobile data’ icon visible  
Start test app netBravo on each phone 
Check network Actual cell details on netBravo on each phone 
Note the following information per phone  
a.  SIM operator name and code 
b.  Network operator name and code  
c.  Network type (4G/3G) 
d.  Cell id 
e.  Data state 
f. 
g.  Take screenshot or photo of ‘Actual cell detail’ 
Set all test phones in an array (side-by-side in one or two rows), along with their SIM 
card logo 
Take photo of the test set  

(it should be: ‘Connected’) 

Field strength  

12)  Field tests on visited networks 

At a time, one visited network was selected.  For the selected network, the visiting SIMs (identified in 
the mapping process above) as well as the home SIM of the visited network were installed on the test 
phones.  With  10  identical  test  devices  available,  one  home  SIM  plus  up  to  9  visiting  SIMs  could  be 
tested in one round. In some cases the testing had to continue into another round when the number 
of mapped SIMs on a visited network was more than 9.  

During each round of tests, Wi-Fi® was set to OFF on all test devices. netBravo was started on all test 
devices. First of all the network parameters on each phone were checked and recorded in the netBravo 
network configuration screen, showing information such as SIM operator code, network operator code, 
type of network (3G or 4G), cell id and signal quality parameters.  

Speed test with netBravo was run sequentially on the test devices, one device at a time in a test cycle. 
The test cycle was repeated ten times.  

At the end of 10 test cycles the test round for a visited network was concluded. 

27 

 
All  SIMs  operating  on 
the 
visited 
network 

same 

Test 
SIM.n 

Test 
SIM.1 

Test 
SIM4 

Test 
SIM.2 

Test 
SIM.3 

Repeat  this  Test 
Cycle  10  times  to 
complete  a  Test 
Round 
a 
Visited Network 

for 

Figure 8. Test Round on a visited network, made of 10 Test Cycles 

The procedure of a test round per visited network is described below. 

On each phone in the test set up, set netBravo on Test dashboard: 

a)  Zoom-in on Start Test screen on all test phones 

b)  Click on ‘Start Test’ button on one test phone 

i. 

ii. 
iii. 
iv. 

v. 

The test starts with basic network tests (Net name, Net type, Net IP. Server, Ping, 
ICMP, SIP) 
Then, occurs Download Test (DL) 
Then, occurs Upload Test (UL) 
At the end, the button ‘Start Test’ reappears. The test results are shown on the 
screen. 
Note the following in the log book: Location, Date, Time of test, phone Id, DL, UL, 
ping 

c)  Repeat Step b) on the rest of the test phones, one at a time 
d)  That completes one test cycle of speed test in a round-robin fashion 
e)  As a visual record, take a sample photo of the test set up showing Speed Test results 
f)  Note any anomalies observed during the test cycle (specific to a SIM or generally in the test 

conditions) 

g)  Repeat Steps a) to f) for the next test cycle 
h)  Repeat step g) ten times 
i)  The above procedure completes one Test Round for a visited network (i.e. the visited 

network’s SIM and all the visiting SIMs in the same visited network) 

Repeat Steps a) to i) above for each visited network in the visited country. 

13)  Upload and review of data 

The data from the tests is stored locally on each phone by the netBravo app automatically. At the end 
of the process, the data is uploaded from each test phone’s netBravo to the JRC server.  

28 

 
 
 
The  uploaded  data  is  reviewed  and  compared  with  that  in  the  Log  Book,  recorded  during  the  tests. 
This is part of the verification process to ensure the uploaded data is correctly received on the server.  

The netBravo application on the server website allows visual and graphic inspection of test results, in a 
set of analytical graphs and tables. 

29 

4  Data collection 

In the context of the EU Roaming Regulation, the JRC carried out a series of field measurements on 37 
mobile  networks  in  13  EU  countries17    to  assess  the  performance  of  roaming.  Performance  of  29 
visiting  SIMs  from  12  EU  countries  was  evaluated  in  relation  to  1)  their  own  home  network,  2)  in 
relation  to  the  domestic  SIM  of  the  visited  network,  and  3)  in  relation  to  other  visiting  SIMs  on  the 
visited network. 

The  map  below  shows  the  various  locations  where  tests  were  carried  out  to  collect  roaming 
performance data. 

Number of visited countries:  

Number of visiting countries: 

Number of visited networks:  

Number of visiting SIMs:  

Total number of SIMs used:  

13 

12 

37  

29 

40  

Figure 9.  Map of roaming test sites 

17 At the time of the tests, the Roaming Regulation was still applicable in the UK. 

30 

 
 
 
 
 
 
 
                                           
5  Analysis and Findings 

5.1  Data analysis 

5.1.1  Q1.  Do  roaming  customers  have  different  quality  of  service  when  roaming 

compared to the performance on their own home network? 

Scenario: 

A  travelling  sales  woman  from  Member  State  X  has  a  network  SIM  of  Operator  X1.  For  seamless 
broadband connectivity, she uses her mobile phone for mobile broadband access wherever in the EU 
she  visits.  In  her  visited  countries,  she  is  able  to  roam  like  at  home  without  surcharges  and  finds  it 
convenient  and  cost  effective.  However  the  quality  of  her  mobile  broadband  service  varies 
considerably and is often worse than what she is used to getting in her mobile network back home.  

Does her home network operator X1 have any policies on QoS in roaming? To figure out she decides 
to  run  speed  tests  on  visited  networks  in  various  countries  that  she  visits.  She  runs  speed  tests  also 
when she gets back home from her business trip.  

To address the above question, speed test measurements carried out by netBravo were analysed for 29 
customer SIMs for which speed test data was available both at home and in roaming.  

Speed tests with netBravo consisted of three parts: Download of a large file, upload of a large file and 
a test on network latency. The analytical procedure and results of the analysis are summarized below. 

(A) 

Download performance 

Download a file (DL): 10 speed tests 

H 

SIM on customer’s 
home network 

Av. DL speed = N Mbps 

A customer 
SIM 

R 

R 

R 

SIM in roaming on 
visited network #1 

Av. DL speed = N Mbps 
=> same performance 

SIM in roaming on 
visited network #2 

Av. DL speed < N Mbps 
=> worse performance 

SIM in roaming on 
visited network #n 

Av. DL speed > N Mbps 
=> better performance 

Figure 10. For each customer SIM, its download performance on a visited network was assessed in relation to that 
on its home network  

31 

 
 
 
 
Figure 11. MNO customer SIMs’ roaming performance compared to at home – Download speed 

X-axis: Customer SIM (Home network operator) 

Y-axis: Number of visited networks 

For four of the 29 SIMs, the download performance was always better in roaming and for four SIMs, it 
was always worse in roaming than in their respective home network. For the other 21 of the 29 SIMs, 
there  was  a  mixed  picture  of  download  performance  in  roaming  as  compared  to  in  their  respective 
home  network:  On  some  occasions  they  had  worse  download  performance  in  roaming  than  in  their 
respective  home  network  and  on  some  occasions  similar  or  even  better  download  performance  in 
roaming than in their home network.  

Figure 12. Overall distribution of download performance of visiting SIMs in relation to their performance at home 

Looking at the overall data for all the 29 SIMs going through 177 unique roaming instances between 
them, on 53% occasions the average download speed of a SIM was found to be better in roaming than 
in its home network; on 8% occasions the download performance of a SIM was similar in roaming as in 
its  home  network;  and  on  39%  occasions  (69  instances  out  of  177),  a  SIM  had  worse  download 
performance in roaming than in its home network.  

The number of visited networks on which a SIM was tested varied between 1 and 12. This was because 
the  total  number  of  operational  pre-paid  visiting  SIMs  grew  over  the  course  of  the  project  and  also 

32 

 
 
 
because  sometimes  the  top-up  process  from  abroad  for  a  pre-paid  SIM  did  not  work  for  logistic 
reasons.  As  a  result,  in  case  of  two  particular  SIMs,  the  number  of  visited  networks  was  only  one 
whereas  for  five  SIMs,  the  number  of  visited  networks  was  ten  or  more.  For  the  29  SIMs,  the  total 
number of unique pairs of roaming instances (visiting SIM, visited network) was 177, with the average 
number of visited networks per visiting SIM being 6.  

When considering only those 19 SIMs for which the number of visited networks was five or more with 
a total of 150 unique roaming instances, on 53% occasions SIMs had better download performance in 
roaming than at home, on 8% occasions they had the same performance and on 39% occasions they 
had worse  performance in roaming than in their respective home network. The  distribution was thus 
similar to the pattern for the overall set of 29 visiting SIMs. 

Overall,  out  of  the  29  visiting  SIMs,  25  SIMs  had,  at  least  once,  worse  download  performance  in 
roaming than at home. These accounted for 69 of the 177 roaming instances. 

(B) 

Upload performance 

The process of analysing the data for upload performance was similar to the one used for download 
performance. It is shown in the figure below. 

Upload a file (UL): 10 speed tests 

H 

SIM on customer’s 
home network 

Av. UL speed = P Mbps 

A customer 
SIM 

R 

R 

R 

SIM in roaming on 
visited network #1 

Av. UL speed = P Mbps 
=> same performance 

SIM in roaming on 
visited network #2 

Av. UL speed < P Mbps 
=> worse performance 

SIM in roaming on 
visited network #n 

Av. UL speed > P Mbps 
=> better performance 

Figure 13. For each customer SIM, its upload performance on a visited network was assessed in relation to that on 
its home network  

Out  of  the  29  SIMs,  for  six  SIMs  the  upload  performance  was  always  better  or  same  in  roaming  as 
compared to in their respective home network and for 10 SIMs, it was always worse in roaming than in 
their respective home network.  

For 13 of the 29 SIMs, there was a mixed picture of upload performance in roaming as compared to in 
their respective home network. In other words, for 45% of the SIMs, on some occasions they had worse 
upload performance in roaming than in their respective home network and on some occasions similar 
or even better upload performance in roaming than in their home network.  

33 

 
 
 
Figure 14. MNO customer SIMs’ roaming performance compared to at home – Upload speed 

X-axis: Customer SIM (Home network operator) 

Y-axis: Number of visited networks 

Figure 15. Overall distribution of upload performance of visiting SIMs in relation to their performance at home 

Looking at the overall data across all the 29 SIMs going through 177 collective roaming performances 
between  them,  on  31%  occasions  the  average  upload  speed  of  a  SIM  was  found  to  be  better  in 
roaming than in its home network; on 10% occasions the upload performance of a SIM was similar in 
roaming  as  in  its  home  network;  and  on  59%  occasions,  a  SIM  had  worse  upload  performance  in 
roaming than in its home network.   

The number of visited networks on which a SIM was tested varied. This was because the total number 
of operational pre-paid visiting SIMs grew over the course of the project and also because sometimes 
the top-up  process from abroad for a pre-paid SIM did not work for logistic reasons.  As a result, in 
case of  two  particular  SIMs, the number of visited  networks was only one whereas for five  SIMs, the 
number of visited networks was ten or more. For the 29 SIMs, the average number of visited networks 
was 6.  

When considering only those 19 SIMs for which the number of visited networks was five or more with 
a total of 150 unique roaming instances, the distribution was similar to that observed for the whole set 
of 29 visiting SIMs.  

34 

 
 
 
 
(C) 

Latency performance  

Recalling that latency in a network signifies the time delay in round-trip information flow, the shorter 
the latency the better it is for the internet customer’s experience of quality of service. 

The  process  used  to  analyse  the  data  for  latency  was  similar  to  the  one  used  for  download 
performance. It is shown in the figure below. 

Network latency: 10 latency tests ‘Ping’ 

SIM on customer’s 
home network 

H 

Av. latency = L ms 

A customer 
SIM 

R 

R 

R 

SIM in roaming on 
visited network #1 

Av. latency = L ms   
=> same performance 

SIM in roaming on 
visited network #2 

Av. latency < L ms  
=> better performance  

SIM in roaming on 
visited network #n 

Av. latency > L ms  
=> worse performance 

Figure 16. For each customer SIM, its latency performance on a visited network was assessed in relation to that on 
its home network 

Figure 17. MNO customer SIMs’ roaming performance compared to at home - Latency 

X-axis: Customer SIM (Home network operator) 

Y-axis: Number of visited networks 

35 

 
 
 
 
Out  of  the  29  visiting  SIMs,  for  16  (55%)  SIMs  the  latency  was  always  or  nearly  always  worse  in 
roaming than in their respective home network. However, for five SIMs the latency was generally better 
or  same  in  roaming  as  compared  to  in  their  respective  home.  It  suggests  that  latency  as  a  QoS 
parameter could depend on the SIM operator’s roaming policy or the technical implementation such 
as traffic steering.  

Looking at the overall data for all the 29 SIMs with 177 collective roaming instances between them, on 
25% occasions the latency for a SIM was found to be better in roaming than in its home network; 13% 
occasions  it was  similar  in  roaming as in its home network; and on 62%  occasions, a  SIM had worse 
latency in roaming than in its home network.   

When considering only those 19 SIMs for which the number of visited networks was five or more with 
a  total  of  150  unique  roaming  instances,  on  28%  occasions  SIMs  had  better  latency  performance  in 
roaming than at home, on 12% occasions they had the same performance and on 60% occasions they 
had worse  performance in roaming than in their respective home network. The  distribution was thus 
similar to that observed for the whole set of 29 visiting SIMs.  

Figure 18. Overall distribution of latency performance of visiting SIMs in relation to their performance at home 

To summarise the findings in regard to Q1, the download performance of roaming customer SIMs was 
worse than in their respective home network on 39% occasions. The upload performance was worse in 
roaming  than  at  home  on  59%  occasions.  The  latency  performance  was  worse  in  roaming  than  at 
home on 62% occasions. 

Overall,  of  the  29  visiting  SIMs,  25  SIMs  had  at  least  once  worse  download  performance  in  roaming 
than at home. These accounted for 69 of the 177 roaming instances. 

5.1.2  Q2. Does the quality of service on visited networks differ between the customers 

of the visited network and the roaming users visiting the network? 

Scenario: 

A  number  of  business  people  from  different  EU  countries  are  participating  in  an  annual  Technology 
Conference  in  City  X.  For  network  security  reason,  they  prefer  to  use  mobile  network  for  broadband 
access  rather  than  the  free  ‘Guest  Wi-Fi®’  network  of  the  conference  centre.  There  are  two  MNOs 
operating  in  City  X.  Some  visiting  delegates  are  automatically  routed  on  MNO  X1  and  some  are 
automatically routed on MNO X2. Several visiting users of MNO X1 seem to be getting generally slow 
network performance so they ask a conference organizer, also a customer of MNO X1, if there are any 

36 

 
 
 
 
problems with MNO X1. Some of them, including the conference organiser, decide to run speeds tests 
in front of the conference  centre premises to  compare their broadband  performance on MNO X1 to 
figure  out  the  problem.  Does  MNO  X1  offer  better  QoS  to  its  home  customers  as  compared  to  the 
visiting (roaming) users?  

In order to answer Q2 speed test measurements carried out by netBravo were analysed for 37 visited 
networks. For each of these visited networks, to compare relative performance, a home customer SIM 
had to be available and there had to be at least one visiting SIM that had registered on it as a visiting 
SIM.  

There were 40 SIMs deployed in all. However, not all of them could be used in every visited country for 
logistics reasons as explained below. Out of the visiting SIMs used for tests in a visited country, some 
visiting  SIMs  would  automatically  choose  to  roam  on  one  visited  network  while  some  others  on 
another  visited  network  and  so  on.  For  this  reason,  the  number  of  visiting  SIMs  tested  in  a  visited 
network varied from one network to another.  

The  number  of  existing  MNOs  per  visited  country  varied  between  2  and  4.  Out  of  the  40  MNOs,  in 
three  cases  a  network  in  the  visited  country  was  selected  by  none  of  the  visiting  SIMs.  Therefore  in 
practice, there were only 37 visited networks, with a minimum 1 and maximum 4 visited networks per 
visited country. 

The total number of visiting SIMs over the course of the project was 29. The number of visiting SIMs 
tested  on  a  visited  network  varied  between  1  and  15.  This  was  (a)  because  the  total  number  of 
operational pre-paid visiting SIMs available at the time of roaming test in a given network varied over 
the  course  of  the  project;  (b)  due  to  the  automatic  selection  of  the  preferred  visited  network  during 
roaming by each visiting SIM.  As a result, as noted before, three of the operating networks were not 
chosen by any of the visiting SIMs. In case of six visited networks, the number of visiting SIMs was only 
one whereas for another six visited networks, the number of visiting SIMs was nine or more. For the 37 
visited networks, the average number of visiting SIMs per visited network was about 5.  

Analytical method 

1.  The netBravo test data was analysed for each of the visited network.  
2.  For each visited network, download and upload speeds were compared for its visiting SIMs 
(roaming customers) vis a vis those of the visited network’s own SIM (home customer). 

3.  A visiting SIM’s speed performance on a given visited network was classed as better / the same 

/ worse depending on the above comparison. This counted as one instance of roaming 
performance.  

4.  Numbers of instances of better / the same / worse performance were counted for each of the 

visited networks. 

5.  The relative performance is plotted in a graphical form (bar chart), for all visited networks. Each 
single bar corresponding to a visited network, shows the distribution of the three types of 
performance on it by visited SIMs. 

6.  Total numbers of instances of better / the same / worse performance were summed for all of 

the visited networks. 

7.  The overall performance of the visiting SIMs for all visited networks is shown in a pie chart. 
8.  Steps 5 and 7 above, allowed analytical observations and conclusions to be drawn based on 

field tests on mobile networks. 

(A) 

Download Performance 

Speed tests with netBravo consisted of three parts: Download of a large file, upload of a 
large file and a test on network latency. Results for these tests are analysed below. 

37 

 
For most of the visited networks, there was no clear cut case of uniformly worse or better performance 
by  the  visiting  SIMs  as  compared  to  their  respective  home  SIM.  In  other  words,  for  a  typical  visited 
network, some of the visiting SIMs had worse performance than that by its home SIM, some visiting 
SIMs  had  the  same  performance  as  the  home  SIM  and  some  visiting  SIMs  had  even  better 
performance than the home SIM of the visited network. 

Downloading a file (DL): 10 speed tests 

Visited network’s SIM 

Av. DL speed = N Mbps 

Visiting SIM #1 

Visiting SIM #2 

Visiting SIM #n 

Av. DL speed = N Mbps 
=> same performance 

Av. DL speed < N Mbps 
=> worse performance 

Av. DL speed > N Mbps 
=> better performance 

Visited Network 

Figure 19. For each visited network, download performance of visiting SIMs was assessed in relation to that of the 
own SIM of that visited network 

Figure 20. Download performance of visiting SIMs compared to the own SIM of the visited network 

X-axis: Visited network 

Y-axis: the number of visiting SIMs that had better/same/worse download speed than the visited network 
operator’s own SIM 

Looking at the overall data across all the 37 visited networks in 13 countries, with varying number of 
visiting SIMs between 1 and 15 per visited network, there were 177 unique roaming instances of pairs 
(visiting SIM, visited network). The average number of SIMs per visited network was about 5.  

For the download, in 50% cases (88 instances) the  download speed for a visiting SIM was found to 
be worse than that for the respective home SIM; in 9% of the cases, the performance for a visiting SIM 

38 

 
 
 
was similar to the respective home SIM; and in 41% cases, a visiting SIM had better performance than 
the home SIM.  

When considering only those 19 visited networks where the number of visiting SIMs was five or more, 
out of 141 unique roaming instances, in 45% cases visiting SIMs had better download performance, in 
8%  cases  they  had  the  same  performance  and  in  47%  cases  they  had  worse  performance  than  the 
visited network’s home SIM.  

Figure 21. Overall distribution of download performance of visiting SIMs in relation to visited network’s own SIM  

(B) 

Upload performance 

The process of analysing the data for upload  performance was similar to the one used for download 
performance. It is shown in the figure below. 

Uploading a file (UL): 10 speed tests 

Visited network’s SIM 

Av. UL speed = P Mbps 

Visiting SIM #1 

Visiting SIM #2 

Visiting SIM #n 

Av. UL speed = P Mbps 
=> same performance 

Av. UL speed < P Mbps 
=> worse performance 

Av. UL speed > P Mbps 
=> better performance 

Visited Network 

Figure 22. For each visited network, upload performance of visiting SIMs was assessed in relation to that of the 
own SIM of that visited network 

As the graph below shows, there was a clearer picture for the upload performance of the visiting SIMs 
in relation to the home SIM of respective visited network.  

39 

 
 
 
 
 
On  19  of  the  37  visited  networks,  the  upload  performance  of  visiting  SIMs  in  relation  to  the  home 
SIM  of  respective  visited  network  was  always  worse  whereas  on  five  visited  networks,  it  was  always 
better. For the remaining visited networks, it was a mixed result where some of the visiting SIMs had 
worse upload performance than that of the home SIM, some had the same performance as the home 
SIM  and  some  visiting  SIMs  had  still  better  upload  performance  than  the  home  SIM  of  the  visited 
network. 

Figure 23. Upload performance of visiting SIMs compared to the own SIM of the visited network 

X-axis: Visited network 
Y-axis: the number of visiting SIMs who had better/same/worse upload speed than the visited network 
operator’s own SIM 

Figure 24. Overall distribution of upload performance of visiting SIMs in relation to visited network’s own SIM  

Looking at the overall data across all the 37 visited networks with 177 unique instances of (visiting SIM, 
visited network), in 72% cases the upload speed for a visiting SIM was found to be worse than that for 
the respective home SIM; in 10% of the cases, the upload performance of a visiting SIM was similar to 
the  respective  home  SIM;  and  in  18%  cases,  a  visiting  SIM  had  better  upload  performance  than  the 
home SIM.  

When considering only those 19 visited networks where the number of visiting SIMs was five or more, 
out of the 141 unique roaming instances, in 17% cases visiting SIMs had better upload performance, in 

40 

 
 
 
 
11% cases they had the same performance and in 72% cases they had worse upload performance than 
that of the visited network’s own SIM. 

(C) 

Latency performance 

Recalling that latency in a network signifies the time delay in round-trip information flow, the shorter 
the latency the better it is for the internet customer’s experience of quality of service. 

The  process  of  analysing  the  data  for  latency  was  similar  to  the  one  used  to  assess  download 
performance. It is shown in the figure below. 

Network latency: 10 latency tests (‘Ping’) 

Visited network’s SIM 

Av. latency = L ms   

Visiting SIM #1 

Visiting SIM #2 

Visiting SIM #n 

Av. latency = L ms   
=> same performance 

Av. latency < L ms   
=> better performance 

Av. latency > L ms   
=> worse performance 

Visited Network 

Figure 25. For each visited network, latency for visiting SIMs was assessed in relation to that for the own SIM of 
that visited network 

Figure 26. Latency for visiting SIMs compared to the own SIM of the visited network 

X-axis: Visited network 
Y-axis: the number of visiting SIMs who had better/same/worse latency than the visited network 
operator’s own SIM 

As the graph above shows, there was a clear picture of the latency for the visiting SIMs in relation to 
the home SIM of respective visited network.  

41 

 
 
  
 
On  16  of  the  37  visited  networks,  the  latency  for  visiting  SIMs  in  relation  to  the  home  SIM  of 
respective visited network was always worse whereas on three visited networks, it was always better or 
the same. On the rest, it was generally a mixed result between better, worse or the same as that for the 
home SIM of the visited network. 

Figure 27. Overall distribution of latency for visiting SIMs in relation to visited network’s own SIM  

Looking  at  the overall data  for  all the 37 visited networks with  177  unique  instances of  (visiting  SIM, 
visited network), in  73% cases the  latency for a visiting  SIM was found to  be worse than  that for the 
respective home SIM; in 12% of the cases, the latency for a visiting SIM was similar to the respective 
home SIM; and in 15% cases, a visiting SIM had better latency than the home SIM.  

When considering only those 19 visited networks where the number of visiting SIMs was five or more, 
out of the 141 unique roaming instances in 15% cases visiting SIMs had better latency, in 11% cases 
they had the same latency and in 74% cases they had worse latency than that of the visited network’s 
own SIM. 

In  summary,  on  Q2,  results  from  field  tests  show  that  the  download  performance  of  visiting  SIMs 
(roaming  customers)  was  worse  than  the  home  SIM  (home  customer)  of  the  visited  network  in  50% 
cases (88 of the 177 roaming instances). The upload performance of the visiting SIMs was worse than 
the home SIM in 72% of the roaming  instances. The  latency of the visiting SIMs was worse  than  the 
home SIM in 73% of the roaming instances. 

It  is  worth  noting  that  the  download  performance  is  crucial  to  the  internet  activities  which  involve 
receiving  large  files  from  others  (e.g.  images  or  documents  in  emails  and  the  cloud)  and  streaming 
videos. On the other hand, upload performance plays an important role for the user in sending large 
quantities  of  information  to  others  such  as  sending  documents,  photos  or  videos.  For  real  time 
interactions  such  as  in  a  videoconference,  latency  as  well  as  both  types  of  transmission  activity  are 
important.  

5.1.3  Q3.  Does  the  roaming  customer  have  different  quality  of  service  on  a  visited 

network compared to other visiting roaming customers? 

Scenario:  

Imagine  a  number  of  young  persons  from  different  EU  countries  meeting  at  a  Youth  Hostel  in  a 
popular tourist city. As it happens, they are all roaming on the same visited network, shown on their 
smart  phones.  They  decide  to  watch  on  their  phones,  an  online  video  guide  about  local  museums 
before  visiting  them  the  next  day.  Some  of  them  seem  to  be  getting  an  uninterrupted  video 
experience while others sitting next to them are finding it frustrating as their video keeps pausing. The 
frustration shows on some of the faces as they look at each  other as if to say ‘What’s wrong with my 

42 

 
 
mobile’? Clearly, the poor experience could not be due to poor network coverage or traffic congestion 
since  network  conditions  are  the  same  for  all  these  friends.  Perhaps  there  is  a  different  quality  of 
service in the contracts of these customers or the wholesale agreement between their respective home 
and visited networks differ in terms of the QoS. 

The following analysis attempts to provide possible evidence for such a discrepancy.  

The performance of visiting SIMs in various visited networks could not be compared in absolute terms 
vis-à-vis their  performance at home, for  reasons  such as traffic  congestion, signal quality on the test 
site, the planned network capacity and the QoS parameters set by the operators. For this reason, the 
direct comparison between roaming performance in one network and the home network is necessary 
but  not  sufficient  for  the  purpose  of  full  analysis  of  roaming  performance.  An  additional  perspective 
was needed.  

For  this reason, this analytical step  was devised  to  benchmark the roaming  performance of a visiting 
SIM in the context of the visited network. Thus if, for  example, a visited network had inherently poor 
performance  conditions,  one  would  expect  all  visiting  SIMs  to  have  low  performance.  Conversely,  in 
high performance conditions of a network, one would expect all visiting SIMs in that network to  have 
high performance. Hence the performance of a specific visiting SIM in relation to other visiting SIMs in 
the same visited network at the same location and at the same time would be an additional objective 
assessment of roaming performance in the given context.  

(A) 

Download performance 

The assessment of relative download performance of a visiting SIM in the context of a visited network 
involved  the  steps  shown  in  Figure  28  below.  The  assessment  process  was  repeated  for  all  177 
instances of (visiting SIM, visited network) pairs between 29 visiting SIMs and 37 visited networks. 

Download a file (DL): 10 speed tests 

Customer visiting 
SIM S1 in a visited 
network 

Another visiting SIM 
S2 in the same 
visited network 

Another visiting SIM 
S3 in the same 
visited network 

Another visiting SIM 
Sn in the same 
visited network 

DL speed 
N1 Mbps 

DL speed 
N2 Mbps 

DL speed 
N3 Mbps 

s
i
h
t

n

i

s
r
e
m
a
o
r

f
o

d
e
e
p
s

l

d
a
o
n
w
o
d

e
g
a
r
e
v

DL speed 
Nn Mbps  A

n

/

)
n
N
+

.
.
.

3
N
+
2
N
+
1
N
(
=
v
a
N

,
k
r
o
w
t
e
n

Relative download 
performance of 
Customer SIM S1 
in Vx 

N1 > Nav 
Above 
average 

N1 = Nav 

Average 

N1 < Nav 
Below 
average 

R 

Visited 
network 
Vx 

Figure 28. Process of assessing the relative download performance of a visiting SIM in a visited network Vx in 
relation to the download performance of other visiting SIMs in that network 

As the figure shows, the download  performance of the visiting SIM S1 in the visited network Vx was 
compared with the average level of download performance of all the visiting SIMs in the network Vx. 

43 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
The result was classified  as  above average / average / below average, based on this comparison. This 
counted as a single unique instance of relative performance (S1, Vx). The step was repeated for all the 
visited networks for that Customer visiting SIM. 

The numbers of counts for each of above average/average/below average were used to represent the 
overall relative performance of the visiting SIM S1 in all its visited networks. It is represented in Figure 
29 by the coloured bar denoted by its home network operator on the X-axis. 

Procedure: 

Suppose a customer SIM S1 visits several networks (Vx, Vw, … Vz) in roaming. Let’s take one such 
network Vx. 

For the Customer SIM S1 visiting network Vx,  

(a)  The  download  speeds  for  all  ‘n’  visiting  SIMs  S1,  S2,  …  Sn  in  the  visited  network  Vx  were 

considered;   

(b)  These speeds are represented by N1, N2, N3, … Nn respectively for the SIMs S1, S2, … Sn; 

(c)  The average of these download speeds was calculated as Nav = {(N1+N2+ … +Nn) / n} 

(d)  Compare the download speed N1 of Customer SIM S1 with the average value Nav 

     If N1 > Nav   =>  

above average 

     If N1 = Nav   =>  

average 

     If N1 < Nav   =>  

below average. 

(e)  Repeat steps (a) – (d) for all networks visited by S1 (Vw, Vy, … Vz).  

(f)  Count the number of times S1 finds itself ‘above average’ | ‘average’ | ‘below average’. 

(g)  Plot the bar for  the SIM  S1 in Figure 29, represented by the name of its home  network. The 
length  of  the  bar  is  equal  to  the  number  of  visited  networks  in  which  customer  SIM  S1  was 
tested.  

Figure 29. Download performance in roaming of customer SIMs of different MNOs in visited networks in relation 
to other visiting SIMs in those visited networks  

X-axis = the customer SIMs designated by their respective (home) network  
Y-axis = the number of visited networks in which the customer SIM of a designated network had above 
average / below average / average download performance relative to other visiting SIMs tested side-by-
side in the same visited network 

Looking  at  the  relative  performance  of  download  in  roaming  as  per  the  above  criteria,  it  can  be 
observed in Figure 29 that for some roaming customers (visiting SIMs), the download performance was 

44 

  
  
 
 
 
more  often  above  average  (longer  GREEN  bars)  whereas  for  some  it  was  generally  below  average 
(longer RED bars).  

The  implication  of  this  finding  is  that  those  customers  with  higher  above  average  count  often  had 
superior  QoS  in  download  in  comparison  to  the  average  level  of  quality  enjoyed  by  all  roaming 
customers in the same visited network at the same time and the same location. For example, among its 
12 visited networks, the IT_D customer had superior download performance while roaming in 6 visited 
networks, below average in four and average performance in two. In contrast, the DE_A customer had 
below average download performance seven of the eight networks that it visited. 

Figure 30. Overall distribution of download performance of visiting SIMs in relation to other roaming SIMs in the 
same visited network 

Looking  at  the  pie  chart  representing  the  performance  of  download  for  all  roaming  instances  in 
relation to  other roamers,  the distribution was nearly even between above average (38%)  and below 
average performance (40%), with 22% classed at the average level. It indicates that the average levels 
were generally also close to the statistical median. Considering only the roaming instances of 19 SIMs 
which visited 5 or more networks, the distribution was similar for the three categories: above average 
(39%) below average (38%), and average (23%). 

However,  for  the  customer  SIMs  whose  download  roaming  performance  was  always  or  often  below 
average  in  their  visited  networks  (visited  countries),  the  graph  suggests  that  their  contractual  QoS 
conditions  for  roaming  (and/or  respective  wholesale  roaming  agreements)  could  be  less  favourable 
than those customers who had more often above average performance.  

(B) 

Upload performance 

The  assessment  of  relative  upload  performance  of  a  visiting  SIM  in  the  context  of  a  visited  network 
involved  the  steps  shown  in  Figure  31  below.  Like  for  download  testing,  the  assessment  process  for 
upload was repeated for every visiting SIM in every visited network. 

The upload performance of the customer visiting SIM in that visited network was compared  with the 
average level of upload performance of all the visiting SIMs in that network. The benchmarking result 
was classified as above average / average / below average, based on this comparison. 

The step was repeated for all the visited networks for that Customer visiting SIM. 

45 

 
 
 
The numbers of counts for each of above average/average/below average were used to represent the 
overall relative performance of the Customer visiting SIM in all its visited networks. It is represented by 
coloured bars in Figure 32. 

Upload a file (UL): 10 speed tests 

Customer visiting 
SIM S1 on a visited 
network 

Another visiting SIM 
#2 in on the same 
visited network 

Another visiting SIM 
#3 in on the same 
visited network 

Another visiting SIM 
#n in on the same 
visited network 

UL speed 
P1 Mbps 

UL speed 
P2 Mbps 

UL speed 
P3 Mbps 

UL speed 
Pn Mbps 

s
i
h
t

n

i

s
r
e
m
a
o
r

f
o

d
e
e
p
s

d
a
o
p
u

l

e
g
a
r
e
v
A

n

/

)
n
P
+

.
.
.

3
P
+
2
P
+
1
P
(
=
v
a
P

,
k
r
o
w
t
e
n

Relative upload 
performance of 
SIM S1 in Vx 

P1 > Pav 
Above 
average 

P1 = Pav 

Average 

P1 < Pav 
Below 
average 

R 

Visited 
network 
Vx 

Figure 31. Process of assessing the relative upload performance of a visiting SIM in a visited network Vx in relation 
to the upload performance of other visiting SIMs in that network 

Figure 32. Upload performance in roaming of customer SIMs of different MNOs in visited networks in relation to 
other visiting SIMs in those visited networks 

X-axis represents the customer SIMs designated by their respective (home) network 

Y-axis represents the number of visited networks in which the customer SIM of a designated network 
had above average / below average / average upload performance relative to other visiting SIMs tested 
side-by-side 

46 

 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Looking at the relative performance of upload in roaming as per the above criteria, it can be observed 
in Figure 32 that for some roaming customers (visiting SIMs) the upload performance was more often 
above average whereas for some it was generally below average.  

The implication of this finding is that those customers with high  above average count frequently had 
superior QoS in upload in comparison to the average level of quality enjoyed by all roaming customers 
at the same time and the same location. For example, a DE_C customer, among its 9 visited networks, 
had  superior  roaming  performance  for  upload  in  all  its  visited  networks  compared  to  the  average 
upload performance of all roamers in those networks. In contrast, the IT_B customer had in eight of its 
ten visited networks, a below average upload performance and on two occasions average performance 
for upload. 

Figure 33. Overall distribution of upload performance of visiting SIMs in relation to other roaming SIMs in the 
same visited network 

Looking at the pie chart in Figure 33, representing the upload performance in all roaming instances in 
relation to  other roamers,  the distribution was nearly even between above average (35%)  and below 
average (39%) with 26% clustered around the average. It indicates that the average levels for upload 
speed were generally also close to the statistical median. 

Considering only the roaming instances for 19 SIMs which visited 5 or more networks, the distribution 
was similar:  above average (39%), below average (34%) and average (27%). 

For the customer SIMs whose upload roaming performance was always or often below average in their 
visited  networks  (visited  countries),  the  graph  suggests  that  their  contractual  QoS  conditions  for 
roaming  (and/or  respective  wholesale  roaming  agreements)  could  be  less  favourable  than  those 
customers who had more often above average performance.  

(C) 

Latency  

Recalling that latency in a network signifies the time delay in round-trip information flow, the shorter 
the latency the better it is for the internet customer’s experience of quality of service. 

The  assessment  of  relative  latency  of  a  visiting  SIM  in  the  context  of  a  visited  network  involved  the 
steps  shown  in  Figure  34  below.  Like  for  download  testing,  the  assessment  process  for  latency  was 
repeated for every visiting SIM in every visited network. 

The latency of the customer visiting SIM in that visited network was compared with the average level 
of  latency  of  all  the  visiting  SIMs  in  that  network.  The  benchmarking  result  was  classified  as  above 
average / average / below average, based on this comparison. 

The step was repeated for all the visited networks for that customer visiting SIM. 

47 

 
 
The numbers of counts for each of above average/average/below average were used to represent the 
overall relative  performance of  a visiting SIM in  all its visited  networks. It is represented  by  coloured 
bars in Figure 35. 

Network latency: 10 latency tests (‘Ping’) 

Customer visiting 
SIM S1 on a  
visited network 

Another visiting SIM 
#2 in on the same 
visited network 

Another visiting SIM 
#3 in on the same 
visited network 

Another visiting SIM 
#n in on the same 
visited network 

Latency 
L1 ms 

Latency 
L2 ms 

Latency 
L3 ms 

Latency 
Ln ms 

s
i
h
t

n

i

s
r
e
m
a
o
r

r
o
f

y
c
n
e
t
a

l

e
g
a
r
e
v
A

n

/

)
n
L
+

.
.
.

3
L
+
2
L
+
1
L
(
=
v
a
L

,
k
r
o
w
t
e
n

Relative latency 
performance of 
SIM S1 in Vx 

L1 < Lav 
Below 
average 

L1 = Lav 

Average 

L1 > Lav 
Above 
average 

R 

Visited 
network 
Vx 

Figure 34. Process of assessing the relative latency performance of a visiting SIM in a visited network Vx in relation 
to the latency of other visiting SIMs in that network 

Figure 35. Latency in roaming of customer SIMs of different MNOs in visited networks in relation to other visiting 
SIMs in those visited networks 

X-axis represents the customer SIMs designated by their respective (home) network 

Y-axis represents the number of visited networks in which the customer SIM of a designated network 
had above average / below average / average latency relative to other visiting SIMs tested side-by-side 

48 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Looking at the relative latency in roaming as per Looking at the relative latency in roaming as per the 
above  criteria,  it  can  be  observed  in  Figure  35  that  for  some  roaming  customers  (visiting  SIMs),  the 
latency was more often more than average whereas for some it was often average or less than average.  

The  implication  of  this  finding  is  that  those  customers  with  high  count  of  below  average  latency 
frequently  had  superior  QoS  in  comparison  to  the  average  level  of  quality  enjoyed  by  all  roaming 
customers  at  the  same  time  and  the  same  location.  For  example,  the  DE_B  customer,  among  its  7 
visited networks, had superior latency on five occasions compared to the average latency by roamers 
in  those  networks.  In  contrast,  the  SE_A  customer  had  in  five  of  its  six  visited  networks,  a  below 
average latency and only once the average. 

Looking  at  the  pie  chart  in  Figure  36,  representing  relative  roaming  performances  for  latency  in  all 
roaming  instances,  the  distribution  was  nearly  even  between  more  than  average  (36%)  and  less  than 
average latency (32%) with 26% clustered around the average level. It indicates that the average levels 
for latency were generally also close to the statistical median. 

Considering  only  the  roaming  instances  of  the  19  SIMs  which  visited  5  or  more  networks,  the 
distribution was:  less than average (40%), more than average (27%) and average (33%). 

For  the  customers  who  often  had  more  than  average  latency  in  their  visited  networks,  the  graph 
suggests  that  their  contractual  QoS  conditions  for  roaming  (and/or  respective  wholesale  roaming 
agreements)  could  be  less  favourable  than  those  customers  who  more  often  had  less  than  average 
latency.  

Figure 36. Overall distribution of latency for visiting SIMs in relation to other roaming SIMs in the same visited 
network 

To summarise the analysis on Q3, in relation to the roaming peers (other roaming users in the same 
visited network, at the same location and at the same time), there was even distribution of instances of 
roaming performance with below average and above average download speed. The same was true for 
the  distributions  of  upload  performance  and  the  latency.  It  shows  that  the  test  results  were  overall 
statistically valid. However, it was found that some customer SIMs were generally more likely to have 
below  average  performance  than  others,  pointing  towards  possibility  of  the  QoS  in  roaming  being 
offered to them at an unfavourable level. 

49 

 
 
 
 
Cases with problems 

To  identify  the  cases  with  problems  in  QoS  during  roaming,  the  following  cross-correlation  analysis 
was performed, looking at the download speed performance. 

(a)  Which  roaming  customers  had  often  worse  download  speed  in  roaming  than  in  their  home 
network?  

13 such customer SIMs were identified from Figure 11 in Q1. 

(b) Which roaming customers had often worse download speed than their roaming peers?  

15 such customer SIMs were identified from Figure 29 in Q3. 

(c) Which customers fared poorly in both cases (a) and (b)? 

Six such customer SIMs were identified  

The following Venn diagram shows the result of this analysis: 

Figure 37. Customer SIMs with generally worse speed performance in roaming 

The cross-correlation analysis was extended to answer the fourth question. 

50 

 
 
 
 
 
5.1.4  Q4.  How  often  did  customers  have  worse  quality  of  service  in  roaming  than  at 
home,  even  when  the  visited  network  was  technically  and  practically  able  to 
provide better quality? 

In the analysis for Q1, it was already observed from Figures 11 and 12, that out of the 29 visiting SIMs, 
25 SIMs had, at least once, worse download performance in roaming than at home. It accounted for 69 
of the 177 roaming instances.  

To  answer  the  above  question,  data  was  further  analysed  for  all  177  roaming  instances  to  find  out 
instances where a visited network had better technical conditions available than the QoS received by 
its visiting SIMs. Therefore the comparisons made in Q2 were analysed and re-mapped on per visiting 
SIM basis.  

Results of this analysis are shown in Figure 38. 

Figure 38. Download performance in roaming of customer SIMs of different MNOs in relation to the own 
customers of the visited networks 

X-axis represents the customer SIMs designated by their respective (home) network 

Y-axis represents the number of visited networks in which the customer SIM of a designated network 
had better / worse /  the same download performance relative to the own SIM of the visited network 

Figure 39. Overall distribution of download performance of visiting SIMs in relation to visited network’s own SIM  

51 

 
 
 
 
It  can  be  observed  in  Figure  38  that  27  Customer  SIMs  had,  at  least  once,  worse  download 
performance than that of the visited network’s own SIM.   

Figure 39 presents an overview of all the instances in Figure 38, showing that in 50% cases (88 of the 
177 instances)  a visiting SIM had worse download  performance  in a visited  network than that of the 
visited network’s own SIM.  

Doing intersection between the set of 88 ‘red’ cases of Figures 38 & 39 and the set of 69 ‘red’ cases of 
Figures 11 & 12, a common set of 44 cases was found which represented all those roaming instances 
where  a  visiting  SIM  had,  at  least  once,  worse  download  roaming  performance  than  at  home  and  at 
the same time it was worse than what was technically possible18 in the visited network. 

The above results are summarised below:  

(a) Of the 29 visiting SIMs, 25 SIMs had, at least once, worse download performance in 

roaming than at home. It accounted for 69 of the 177 roaming instances.  

(b) Of the 29 visiting SIMs, 27 SIMs had, at least once, worse download performance in 
roaming than what was technically possible in the visited network. It accounted for 
88 of the 177 roaming instances. 

(c) 44 of the 177 roaming instances were common to (a) and (b), when a visiting SIM 
had worse download performance in roaming than at home and at the same time 
worse than what was technically possible in the visited network.  

(d) These 44 instances occurred for 21 SIMs from 11 countries. 

Figure 40.  Problem cases identified in roaming tests 

In summary on Q4:  The analysis  for download speeds  showed that  21 customers  from 11 countries 
had  worse    download  speed  performance  in  roaming  than  at  home  even  when  better  quality  was 
technically possible on the visited networks.  

18  By  ‘technically  possible’,  it  means  that  a  higher  speed  on  a  visited  network  was  achieved  by  the  home 

customer of the visited network during the same test round. 

52 

 
                                           
6  Conclusions 

This study concerns assessment of the quality of service in roaming in the context of the EU Roaming 
Regulation. The study addressed the following questions: 

(1)  Whether roaming customers had different quality of service when roaming compared to the 

performance on their own home network.  

(2)  Whether quality of service on visited networks differs between the customers of the visited 

network and the roaming users visiting the network.  

(3)  Whether roaming customers had different quality of service on a visited network compared to 

other visiting roaming customers.  

(4)  How often did customers have worse quality of service in roaming than at home, even when 

the visited network was technically and practically able to provide better quality? 

The  methodology  of  the  study  was  based  on  field  measurements  by  29  SIMs  from  12  countries, 
roaming in 13 countries, on 37 mobile networks. Altogether SIMs from 40 MNOs were involved. The 
tests  produced  177  roaming  instances  of  (visiting  SIM,  visited  network)  pairs.  The  field  tests  were 
carried  out  during  October  2017  –  October  2019.  The  JRC  mobile  app  netBravo  designed  for  the 
measurement  of  quality  in  mobile  and  broadband  networks  was  used  to  measure  and  record  the 
mobile  network  performance  (download  speed,  upload  speed  and  latency)  by  various  SIMs  both  in 
their home network and in visited networks. 

The following are the main findings of the study: 

(1)  Out of the 177 roaming instances, the download performance of customer SIMs was worse 
than in their respective home network on 39% occasions (69 instances) whereas the upload 
performance was worse in roaming than at home on 59% occasions. The latency was worse in 
roaming than at home on 62% occasions.  

(2)  The download performance of visiting SIMs (roaming customers) was worse than the home 
SIM (home customer) of the visited network in 50% cases (88 of the 177 roaming instances). 
The upload performance of the visiting SIMs was worse than the home SIM in 72% of the 
roaming instances. The latency of the visiting SIMs was worse than the home SIM in 73% of 
the roaming instances. 

(3)  In relation to the roaming peers (other roaming users in the same visited network, at the same 
location and at the same time), there was even distribution of instances of roaming with below 
average and above average download performance. The same was true for the distribution of 
upload performance and latency. It shows that the test results were overall statistically valid. 
However, it was found that some customer SIMs were generally more likely to have below 
average performance than others, pointing towards possibly QoS in roaming being offered to 
them at unfavourable level. 

(4)  Six Customer SIMs (20%) had generally worse speed in roaming than at home and also in 

relation to other roaming customers in the visited networks.  

(5)  In 25% cases (44 instances out of 177 roaming instances), a visiting SIM had, at least once, 

worse download performance in roaming than in its home network even when better quality 
was technically possible in the visited network. These cases occurred for 21 Customer SIMs 
from 11 countries. 

The study shows that taking into account the number of analysed  networks and MSs in this analysis, 
roaming  customers (in 25% of all roaming instances) had worse download speed in roaming than at 
home even when technical conditions on the visited network existed for them to achieve better quality 
of service. 

53 

 
Annexes 

Annex 1. netBravo 

NETBRAVO - AN APP TO MAP MOBILE NETWORK COVERAGE AND MEASURE 
QUALITY OF SERVICE 

http://netbravo.eu 
http://netbravo.jrc.ec.europa.eu  

54 

 
 
 
 
 
 
What is netBravo? 

netBravo  is  a  mobile  app  developed  by  the  Joint  Research  Centre  of  the  European  Commission  for 
smart  phones  and  tablets  to  help  measure  the  quality  of  mobile  and  broadband  connections  as 
experienced by users anywhere in the world.  

netBravo 

Mobile 
User  

PC 
User 

Mobile 
Network 
Coverage 

Wi-Fi 
Network 
Quality 

Fixed 
Broadband 
Quality 

Mobile 
Broadband 
Quality 

Why should I use netBravo?         

By using netBravo, you get real evidence of the quality of your internet connection and the quality of 
signals of your mobile network in your area. By sharing your mobile network measurements with other 
users, in other areas and even other networks, together you all get a big picture to help you choose a 
network based on cost, quality and coverage.  

55 

 
 
          
 
 
 
 
 
 
 
 
 
How do I use netBravo?   

As a  mobile user, you  need a  smartphone or  tablet running Android® or  iOS®. These  cover  a large 
majority  of  smartphone  users  worldwide.  You  will  need  to  download  the  netBravo  app  from 
AppStore®  or  GooglePlay®.  Once  you  have  installed  the  app,  you  can  start  looking  at  the  signal 
quality of your local Wi-Fi® Access Points and run speed test and quality of your internet connection 
whether connected through Wi-Fi® or a mobile data network (3G/4G).  

As a PC user, you can access netBravo directly from your web browser by accessing http://netbravo.eu 
where  you  can  browse  the  quality  data  on  the  map.  You  can  also  run  a  speed  test  on  your  internet 
connection.  

How does it work? 

netBravo works on the principle of crowd sourcing. It collects measurements made voluntarily by end 
users  on  their  mobiles,  providing  aggregate  data  for  locations  covered  by  any  network.  Whereas 
netBravo  continuously  samples  the  signal  strength  of  mobile  signals  and  Wi-Fi®  connections  at  its 
user’s location, it also allows the user to run speed and quality tests on their internet connection. All 
data collected by netBravo is anonymous, respecting  the principles of privacy. The measurements are 
uploaded to the netBravo server and aggregated with other measurements to  become visible on the 
netBravo site on an interactive map. 

User Smart 
phone  or 
PC 

Mobile 
User  

PC 
User 

Test your 
internet 
connection 

Upload 
data to 
Server 

View map 
on 
Website 

netBravo 
website 

Current Status of use 

netBravo  is  already  being  used  widely  in  Europe,  as  evident  from  the  map  data.  It  has  been 
downloaded by users worldwide in more than 70 countries. As on 1 January 2021, there were over 7 
million measurements on the quality of cellular networks and over 19 million on Wi-Fi® networks.  

56 

 
 
 
 
 
 
  
   
 
 
 
 
How does netBravo help policy makers? 

netBravo  is  one  of  the  few  independent  apps  that  contribute  data  to  the  European  Broadband 
Mapping Portal. The portal maps the progress in MSs on the broadband infrastructure and the quality 
of  service  of  high-speed  broadband  in  the  context  of  Europe2020  goals.    netBravo  also  tests  and 
validates the emerging rules of Open Internet (net neutrality) and their technical implementation. This 
allows  you  to  check  if  an  internet  service  provider  routinely  blocks  or  slows  down  some  internet 
applications (such as internet telephony or VOIP). 

https://www.broadbandmapping.eu/ 

57 

 
 
 
 
How can netBravo help non-EU countries? 

Any country in the world can benefit from mapping the coverage  and quality of ICT infrastructure in 
various regions – cities, rural areas, mountains, tourist spots etc.  

The  evidence  from  netBravo  on  the  quality  of  service  can  help  them  in  refining  their  policies  for 
communications  infrastructure,  strategic  investment  and  further  development:  education,  tourism, 
businesses and e-Government services.  

netBravo  

QoE 

Current 
infrastructure, 
QoS 

Development 
Goals 

Future 
priorities for 
infrastructure 
investments 

How can a country use netBravo to map its infrastructure? 

Pilot phase – Getting Started 

On  the  user  side,  the  administration  would  need  to  promote  the  use  of  netBravo  among  the 
population.  

Because netBravo is a crowd-source application, its  impact depends on the depth and breadth of its 
use by the general public. The more people use it, the more shared information will become available 
about the coverage and quality of mobile and broadband networks in their community and country.  

On the policy side, the administration will need access to the aggregated data collected by netBravo. 
This is already provided to Open Data Portal and on the netBravo website. 

Adoption phase - Lots of users, lots of data 

Once the policy makers have ascertained the value of QoS data to promote their development goals, 
they  can  move  towards  full  adoption  of  netBravo  as  a  system.  This  means  owning  or  co-sharing  the 
host infrastructure and service in their own environment. For this, one or more netBravo servers could 
be  installed,  nearer  the  users,  in  each  country  to  allow  acquisition  of  network  quality  data  and  its 
processing better tailored to their needs.  

Such servers can be installed in one or more large cities. The options are: 

58 

 
 
 
 
 
  at a  national university with good access to the internet backbone and back-up power;  

 

in a government ministry or agency with good access to the internet backbone and back-up 
power; 

  at an internet service provider with web hosting service. 

The  economics  and  options  for  installing  local  servers  would  vary  in  each  country.  An  economic 
analysis can be carried out on a case-by-case basis. 

How much does netBravo cost the end user? 

The netBravo app and website access are free of charge.  

The  automatic sampling of signal quality of  mobile  and Wi-Fi® networks by your  smart phone does 
not cost you anything. It consumes a little amount of battery power when the app is running. 

Whenever  you  decide  to  run  a  speed  test  on  your  internet  connection,  netBravo  will  do  some  data 
transfer (both upload and download) to test the quality of connection. In this case, your internet data 
plan will be used (i.e. your active Wi-Fi® or mobile connection at the time of running the test).  

When you are on a Wi-Fi® connection, netBravo app automatically uploads sampled measurements to 
the netBravo server which will merge your samples with those from all the other netBravo users. 

Can a user see the quality measurements on the map? 

Users  can  see  measurements  only  as  part  of  all  the  other  samples  by  all  the  other  users.  All  the 
measurements are anonymous so it is not possible to tell who took which measurements. We are not 
interested  to  know  your  SIM  number  or  your  mobile  phone  number  since  these  have  no  link  to  the 
quality of your internet connection. Instead, we collect the name of the network service provider and 
the type of mobile device you use because these affect your quality of service 

In which languages is the netBravo app available? 

To involve a maximum breadth of citizens, netBravo has been designed in a multi-lingual framework. 
Currently,  it is available in  English, French, Italian,  Portuguese  and Arabic.  To add a new language to 
the user interface is a simple process. 

59 

 
 
 
 
 
 
 
 
 
60 

 
 
GETTING IN TOUCH WITH THE EU 

In person 

All over the European Union there are hundreds of Europe Direct information centres. You can find the address of the 
centre nearest you at: https://europa.eu/european-union/contact_en 

On the phone or by email 

Europe Direct is a service that answers your questions about the European Union. You can contact this service: 

- by freephone: 00 800 6 7 8 9 10 11 (certain operators may charge for these calls), 

- at the following standard number: +32 22999696, or 

- by electronic mail via: https://europa.eu/european-union/contact_en 

FINDING INFORMATION ABOUT THE EU 

Online 

Information about the European Union in all the official languages of the EU is available on the Europa website at: 
https://europa.eu/european-union/index_en 

EU publications 
You can download or order free and priced EU publications from EU Bookshop at: 
https://publications.europa.eu/en/publications. Multiple copies of free publications may be obtained by contacting 
Europe Direct or your local information centre (see https://europa.eu/european-union/contact_en). 

 
 
 
 
K
J
-
N
A
-
3
0
6
0
2
-
E
N
-
N

doi:10.2760/682 

ISBN 978-92-76-30476-0