A framework for monitoring the impact of 5G public private partnership and the associated key performance indicators (KPIs) : final report.
Date: 2017-05-31
Institutions: Directorate-General for Communications Networks, Content and Technology
Subjects: EU financing, Internet, impact of information technology, information technology, new technology, public-private partnership, research and development, single market, telecommunications, telecommunications industry

A Framework for monitoring 
the impact of 5G Public 
Private Partnership and the 
associated Key Performance 
Indicators (KPIs)   

 SMART 2015/0013 

FINAL REPORT 

A study prepared for the European Commission 

DG Communications Networks, Content & Technology by:  

Digital 
Single 
Market 

1 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
This study was carried out for the European Commission by: 

Tech4i2 

CONNECT, Trinity College Dublin 

Internal Identification: 

Contract Number: 30-CE-0768169/00-66 

SMART Number: 2015/0013 

DISCLAIMER  

By the European Commission, Directorate-General of Communications Networks, Content 
and Technology. 

The information and views set out in this publication are those of the author(s) and do not 
necessarily reflect the official opinion of the Commission. The Commission does not 
guarantee the accuracy of the data included in this study. Neither the Commission nor any 
person acting on the Commission’s behalf may be held responsible for the use which may be 
made of the information contained therein.  

ISBN 978-92-79-69011-2 

Doi:10.2759/79440 

© European Union, 2017. All rights reserved. Certain parts are licensed under 
conditions to the EU.  

2 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Contents 

Abstract .......................................................................................................... 5 

Executive Summary ....................................................................................... 6 

1. 

Introduction ........................................................................................... 12 
1.1.  The study .................................................................................................. 12 
1.2.  The Final Report ....................................................................................... 13 

2.  Overview ................................................................................................ 14 
2.1. 
Introduction .............................................................................................. 14 
2.2.  Key Performance Indicators: An introduction ....................................... 15 
2.3.  5G PPP Performance Indicators ............................................................. 16 
2.4.  The framework, chronology and KPI question methods ....................... 21 
2.5.  5G PPP KPI and evaluation discussions ................................................ 26 

3.  Shortlisting the KPIs ............................................................................. 27 
3.1. 
Introduction .............................................................................................. 27 
3.2.  From long-list to medium-list .................................................................. 27 
3.3.  Operating KPIs ......................................................................................... 28 
3.4.  Performance KPIs .................................................................................... 30 
3.5.  Societal KPIs ............................................................................................ 31 
3.6.  Gap analysis ............................................................................................. 32 
3.7.  Conclusion ............................................................................................... 34 

4.  Shortlisted KPIs and consultation ....................................................... 35 
4.1. 
Introduction .............................................................................................. 35 
4.2.  Refining the medium list of KPIs ............................................................ 35 
4.3.  Consultation and validation .................................................................... 43 
4.4.  Operationalising the KPIs ....................................................................... 45 
4.5.  Monitoring or progress evaluation against targets ............................... 55 

5.  Recommendations for the monitoring process .................................. 60 
Introduction .............................................................................................. 60 
5.1. 
5.2.  Developing an innovative approach to 5G monitoring .......................... 60 
5.3.  Governance .............................................................................................. 69 
5.4.  Conclusion ............................................................................................... 70 

6.  Annex 1  KPI briefing documents distributed to stakeholders ......... 72 
6.1.  Operating KPIs ......................................................................................... 72 
6.2.  Performance KPIs .................................................................................... 74 
6.3.  Societal KPIs ............................................................................................ 79 

7.  Annex 2  Phase 1 5G PPP projects ...................................................... 82 

8.  Annex 3  Glossary ............................................................................... 101 

3 

 
 
 
 
 
 
 
 
 
 
4 

 
 
 
Abstract 
This study provides recommendations for the development of an innovative approach 
to monitoring and evaluation by creating a 5G Observatory that will gather 5G PPP 
monitoring results and information from other studies and use this as catalyst for 
debate and interaction between the 5G PPP projects and those who will utilise the 
5G innovations and technologies developed by projects. 

The study developed a robust methodological framework and short listing process to 
identify 29 KPIs to monitor 5G PPP projects receiving €700 million support, plus EU 
industry inputs, from the Horizon 2020 programme.  KPIs were categorised into three 
groups:  i) Operating KPIs likely to be achieved within two-years;  ii) Performance 
KPIs likely to come to fruition at the end of 5G PPP projects; and iii) Societal KPIs 
that examine longer-term outcomes and impacts for consumers, businesses and 
different environments (cities, workplaces, homes etc.). 

The KPI shortlisting process included extensive stakeholder consultation.  
Stakeholder groups included:  i) The 19 5G PPP projects;  ii) Key stakeholders in the 
supply and value chains; and iii) Policymakers and representatives from Regulatory 
Authorities.  The study also utilised an online 5G expert group (developed by the 
study team) with more than 250 5G expert members for online consultation.  

5 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Executive Summary 
This report provides details of research and recommendations for an evaluation 
methodology to develop Key Performance Indicators to monitor the €700 million 
support, allocated as part of the Horizon 2020 programme alongside EU industry, for 
5G Public Private Partnership (5G PPP) projects.  The funding and 19 first round 
projects aim to accelerate research developments in 5G technology. The emergence 
of 5G is an important milestone in the pursuit of ubiquitous broadband access in 
Europe, a vital part of the Digital Agenda for Europe and the key enabler of a Digital 
Single Market. 

The study undertook shortlisting to move from a long list of potential KPIs to a short 
list of 29 KPIs.   The methodological framework for the study categorised KPIs into 
three groups:   

  Operating KPIs (e.g. leverage and SME participation) likely to be 

achieved within the two-year life span of most 5G PPP projects;  

  Performance KPIs examine technical capability improvements (e.g. 
1,000 times higher wireless capacity and zero perceived downtime) 
likely to come to fruition at the end of 5G PPP projects; 

  Societal KPIs examine longer-term outcomes and impacts (e.g. 

advanced user controlled privacy and lower energy consumption) that 
will arise from the utilisation of 5G technical capabilities (developed by 
5G PPP projects and brought to market) by consumers, businesses, 
government and others.   

Key performance indicators measure the quality of an organisation’s performance 
and assist in monitoring performance and the achievement of wider goals. In 
essence KPIs define success and track progress in meeting strategic goals. This 
focus on strategic or long-term goals is what distinguishes KPIs from the wider array 
of ‘performance indicators’.  KPIs assist an organisation(s) to measure that it is ‘on 
track’ and is working towards and attaining a beneficial outcome or improvement.   

Previous studies of PPP projects1 and the Seventh Framework Programme2 have 
highlighted shortcomings in achieving some of the strategic objectives and goals 
desired by the initiatives.  Most notably the need to promote higher-value-chain 
involvement   and/or industrial relevance, to make project results better known to 
business and citizens, better co-ordination and collaboration between projects and 
the need for projects to have greater impact on societal and economic goals.  
Thorough development of KPIs and monitoring can help to address these 
shortcomings.  The KPIs provide projects with clear criteria against which they will be 
evaluated.  Regular monitoring measures performance and reinforces the need for 
projects to perform well against targets or in relation to their peer group.   

The foundations for the methodological framework for the study build on European 
Commission Impact Assessment Guidelines, Better Regulation Toolbox components 
and international best practices.    

The proposed monitoring and measurement activities are based on logic chains that 
provide controllable inputs and outputs.  The 5G PPP projects control the outputs 
that they achieve.  These outputs, innovations and technologies created by 5G PPP 
projects will produce outcomes.  An example of an outcome could be the use of 5G 

1 EC.  2013.  Final assessment of the research PPPs in the European Economic Recovery Plan p54 to 

56.  The study considered three specific sectors.  But many of the comments were generic with 
relevance to all PPP evaluations. 

2 Tech4i2 and Deloitte.  2010. Impact assessment of EC i2010. SMART 2007/0035. 

6 

 
 
                                                
capabilities in vehicles.  These outcomes will lead to longer-term impacts.  To 
continue the automotive example; impacts of the use of 5G capabilities in vehicles 
(developed by 5G PPP projects) could include a reduction in congestion, hydro-
carbons or pollution.  These knock-on or trickle down impacts will only arise when 5G 
is more widely used in vehicles.   

There is a temporal dimension to inputs, outputs, outcomes and impacts. Inputs in 
the form of Horizon 2020 and industry resources will support the activities of 5G PPP 
projects; these are happening now in first round 5G PPP projects. Outputs are likely 
to be achieved towards the end of the activities of projects. Outcomes will probably 
only start to take place after 5G PPP projects have concluded and the innovations 
and technologies developed have been recognised as beneficial, this is likely to be 
three to five years away. Significant impacts are likely to be eight or more years away 
when the 5G PPP project innovations and technologies are utilised in products and 
services used by citizens and businesses.   

An overview of the three categories of indicators and the likely chronology for 
achievement of the objectives is provided below. 

Operating KPIs relate to the activities being undertaken now and over the duration 
of the 5G PPP contractual period by the organisations involved in the 5G PPP 
projects.  Many of the proposed KPIs are factual (SME participation, leverage and 
skills development etc.).  Others relate to KPIs that will be achieved over the lifetime 
of the study (market share, influence on standards, creation of IPR etc.), for most this 
is a three-year duration.   

Performance KPIs largely relate to the technological capability developments or 
outputs from 5G PPP projects.  These are only likely to come to fruition towards the 
end of the 36 month (maximum) period for which most projects run. 

Societal KPIs concern the outcomes and impacts that will arise from the utilisation 
of 5G technical capabilities (developed by 5G PPP projects and subsequently 
brought to market) by consumers, businesses, government and other organisations.  
They will arise in verticals (industries) and different environments (cities, semi-urban, 
rural, workplaces, homes, public buildings and other locations).  These benefits are 
very unlikely to arise while 5G PPP projects are operating.  Some early benefits may 

7 

 
 
 
start appearing as early as 2018, but most observers have suggested 5G capabilities 
(and thus the resulting outcomes and impacts) will not become widely commercially 
available until after 2020. 

A two stage shortlisting process was undertaken to refine and ensure the suitability 
of the KPIs identified by desk research at the start of the study.  These 65 KPIs were 
mapped for relevance against the 19 first round 5G PPP projects. 1,235 
assessments (65 x 19) were made of the relevance of KPIs to first round projects.  
The study team carefully read details of all 5G PPP projects and talked to those 
involved with projects before making an assessment of relevance.   

Alongside the cross-tabulation exercise the study team also undertook gap analysis 
to identify KPIs not identified in contracts and other relevant 5G documentation.  This 
was important because 5G is a large and moving target and consequently 
Performance KPIs, which were set out at contract stage, may not fully align with or 
capture the current trajectory of 5G as it evolves and matures towards 
standardisation. For example, the increased focus on vertical markets and the 
accelerated maturation of new Internet of Things solutions such as the LPWAN 
offerings has served to both clarify and expand the diverse and sometimes divergent 
capabilities that 5G must support.  A useful example of an important area identified in 
gap analysis that was not addressed by the KPIs identified in desk research was 
support for mobile edge computing (MEC).  MEC has been rapidly gaining traction 
within the 5G community as an expression of the use of NFV and SDN at the edge of 
the network.  

Gap analysis also considered innovations and technologies not addressed by first 
round 5G PPP projects, which might be supported in later rounds of funding.  Most 
notable was the need for KPIs related to Data centre and Core network optical 
networking. 

Gap analysis led to three additional KPIs being included in the medium list of KPIs to 
address the gaps identified.   

This first stage of the shortlisting process reduced the long list of 65 KPIs to a 
medium list of 30 KPIs.  The number of Operating KPIs was reduced from 13 (found 
in desk research) to 11.  Thirteen Performance KPIs were selected (from 43) and six 
Societal KPIs were identified (from nine identified in desk research).   

The second stage of the shortlisting process commenced with a reconsideration 
of the rationale for including each KPI.  The review process included the 
examination of European Parliament and other Commission requirements regarding 
the goals and outputs expected from Horizon 2020 (and historically from FP7).  It 
also investigated lessons learned from previous evaluation studies of FP7 and PPP 
initiatives.  Analysis also investigated expectations regarding 5G capabilities found in 
the 5G Action Plan and other documents produced by leading 5G organisations. 

In essence the review process ensured that the KPIs met European objectives, were 
aligned with the expectations of leading 5G organisations and that they avoided 
pitfalls and adhered to best practices identified in previous monitoring and evaluation 
exercises. 

The second stage included a stakeholder consultation exercise to ensure the 
shortlisted KPIs are ‘fit for purpose’.  Short briefing papers (of two to four pages of 
A4) describing each of the three categories of KPIs were developed.  The briefing 
documents provided a background to the rationale for the selection of the KPIs.  The 
purpose for collecting information and the focus of the KPIs were also described.   

8 

 
 
Consultation and engagement focused on three stakeholder groups. Wider online 
engagement with any interested individual or organisation was also undertaken.  The 
three key stakeholder groups included: 

The 19 5G PPP projects:  The objective of contacting projects was to 
seek the views of these organisations about the relevance of the KPIs to 
their activities; 

Key stakeholders in the supply and value chains:  Briefing documents 
were sent to representatives in relevant verticals.  Key stakeholders were 
asked about the importance of the different KPIs to their activities.  
Respondents were also asked to provide details of other KPIs that might 
be relevant to their activities; 

Policymakers:  Briefing documents were sent to policymakers and 
representatives from Regulatory Authorities.  Policymakers were asked 
about the importance of KPIs to their Members.  Respondents were also 
asked to provide details of other KPIs that might be relevant to their 
activities. 

The study utilised an online LinkedIn 5G expert group (developed by the study team) 
for online consultation.  The group has more than 250 expert 5G members.   This 
more extensive online approach maximised the opportunity for input from a large 
group with expert knowledge of 5G. 

The second stage shortlisting process identified 29 KPIs3.  These included seven 
Operating KPIs, twelve Performance indicators and ten Societal KPIs.  The KPIs, the 
rationale for their inclusion and operational metrics to measure the indicators are 
provided in the three Annex 1 briefing documents shared with stakeholders at the 
end of the study.  

The penultimate task in the study was to develop detailed metrics to measure the 
indicators.  As well as providing clear measurement methods for each KPI it was also 
necessary to consider the baseline situation for each KPI, relevant targets and how 
progress should be reported.  Key considerations included: 

Baseline situation:  A baseline assessment provides a critical reference 
point for assessing changes and impact.  It establishes a basis for 
comparing the situation before and after an intervention.  Baseline data 
enables inferences to be made about the effectiveness of an intervention.  
Several of the KPIs make reference to a baseline situation in 2010.  
Baselines are less important for Operating and Societal KPIs because 
projects are ‘standalone’ initiatives that have no precedents. 

Targets:  An interesting contradiction exists between the traditional 
purposes of using KPIs in monitoring and evaluation and the manner in 
which they have frequently been used in monitoring European Commission 
projects.  KPI literature highlights that targets create performance standards 
and motivate continuous improvement.  Most European Commission 
monitoring studies do not establish targets.  Interestingly only seven of the 
KPIs from first stage analysis of 5G contracts had explicit targets.  

The study team recommend that wherever possible, targets for KPIs should 
be established so that projects have clear goals and objectives.  If targets 
are not established it is possible to monitor performance and improvements, 

3 The second stage process involved omitting some of the 30 KPIs found in the first stage of the 

shortlisting process.  Those removed were replaced by five ‘new’ KPIs. 

9 

 
 
                                                
but the magnitude of these improvements becomes somewhat nebulous.  
Some projects could perceive that there is little incentive to perform at 
anything above minimum levels perceived as satisfactory.   

Obviously setting targets is difficult.  But we assert that this issue can easily 
be addressed for many of the KPIs associated with 5G projects and others 
supported by Horizon 2020.  For example, no clear targets are set for KPIs 
associated with dissemination or support for open innovation ecosystems.  
Numerous previous initiatives similar to the 5G PPP have been undertaken 
in the past.  Project monitoring reports from relevant comparable projects 
could be analysed to derive the average number of research papers, 
patents, demonstrators etc. and these could be set as minimum levels for 
the KPIs. 

The study concludes by recommending that an independent 5G Observatory should 
undertake 5G PPP monitoring.  It is recommended that a Strategy Board of no more 
than six people should oversee the Observatory.  The Board should be comprised of 
personnel from DG CONNECT and diverse stakeholders with a mix of evaluation, 
technical and technology impact expertise.  At least two board members should be 
experts in supporting technology exploitation. 

This study recommends the development of a new and innovative approach to 
monitoring and evaluation by creating an independent 5G Observatory that will 
gather 5G PPP monitoring results and information from other studies and use this as 
catalyst for debate and interaction between the 5G PPP projects and those who will 
utilise the 5G innovations and technologies developed by projects.   

The Observatory will achieve two key impacts.  It will: 

i)  Maximise outputs and the exploitation of 5G PPP project results by  

ii) Catalysing 5G PPP project representatives and other European and 
global stakeholder engagement with potential 5G users and other 
stakeholders.   

Importantly it will actively support projects to commercialise their technologies and IP 
by providing training and support and inviting potential investors to workshops.  This 
will help to ensure Horizon 2020 support has a commercial impact. 

The innovative approach adopted recommended for the observatory will overcome 
shortcomings reported in earlier PPP initiatives that lacked industrial relevance and 
impact.  It will also address the monitoring deficiencies highlighted in previous 
studies.   

Innovative elements of the recommended approach to evaluation include:- 

  Clarity and transparency in robustly defining ‘fit for purpose’ KPIs to 
monitor and promote the activities (including technologies and 
capabilities) being developed by 5G projects;  

 

Identification of clear rationale and measurement criteria for KPIs; 

  Development, with 5G PPP projects and 5G users, of clear targets for all 
KPIs.  These targets will stretch projects and ensure outputs are more 
aligned with the needs of 5G users; 

  Catalyse engagement between three groups – i) 5G PPP projects, ii) 

potential users of 5G capabilities, and iii) investors and others that can 
support the exploitation of 5G inventions; 

  Support for 5G PPP projects in providing the hard and soft skills and 
strategic market intelligence to support projects in promoting their 
inventions to those that can exploit their technologies. 

10 

 
 
The Observatory will monitor 5G market and technology forecasts to provide projects 
and policymakers throughout EU28 Member States with up-to-date insights to 5G 
market requirements, market potential and new business models, and insights to the 
economic and societal impacts of 5G. 

11 

 
 
 
 
 
 
1. Introduction    

1.1.  The study  

In December 2013, the EC signed a landmark agreement with the ‘5G Infrastructure 
Association’ to establish a Public Private Partnership on 5G (5G-PPP) with the aim of 
accelerating research developments in 5G technology. €700 million has been 
allocated as part of the Horizon 2020 programme alongside EU industry which is 
expected to match the investment fivefold.  The emergence of 5G is an important 
milestone in the pursuit of ubiquitous broadband access in Europe, a vital part of the 
Digital Agenda for Europe and the key enabler of a Digital Single Market.  

The European Commission selected Tech4i2 and Trinity College Dublin to undertake 
research to propose an evaluation methodology and to develop Key Performance 
Indicators to ensure that 5G-PPP projects meet quantitative and qualitative target 
performance levels.   

The study undertook shortlisting to move firstly from a long list of potential KPIs; to a 
short list of 29 KPIs.  

KPIs were categorised by three classifications;  Operating KPIs (e.g. leverage and 
SME participation) likely to be achieved within the two-year life span of most 5G-PPP 
projects.  Performance KPIs examine technical capability improvements (e.g. 1,000 
times higher wireless capacity and zero perceived downtime) likely to come to fruition 
at the end of 5G-PPP projects.  Societal KPIs examine longer-term outcomes and 
impacts (e.g. advanced user controlled privacy and lower energy consumption) that 
will arise from the utilisation of 5G technical capabilities (developed by 5G PPP 
projects and brought to market) by consumers, businesses, government and others.   

12 

Figure 1  Methodology for the Study 

 
 
 
1.2.  The Final Report 

This Final Report for the study on “A framework for monitoring impact of 5G Public 
Private Partnership and the associated Key Performance Indicators (KPIs)” (SMART 
2015/0013) is provided after the final meeting which took place on Tuesday 18th 
October 2016.  

This Final Report is provided on time and in line with the agreed schedule for the 
project.  

The next chapter provides an introduction to the role and purposes of Key 
Performance Indicators in guiding and monitoring the performance of 5G PPP 
projects.  It provides a framework for the study and for the development of KPIs.  The 
chapter also considers how this study complements KPI research undertaken by the 
5G Infrastructure Association.   

Chapter 3 provides an overview of the initial short-listing process that reduced a long 
list of 69 potential indicators to a ‘medium’ list of approximately 30 KPIs.  These KPIs 
were cross-tabulated against the focus and activities of the 19 5G-PPP projects 
funded by the first phase of funding.  More than 1,235 assessments of KPIs projects 
were undertaken.  

The penultimate chapter describes how the medium list of KPIs was further refined to 
produce evidence based rationale to produce a short list of 29 ‘fit for purpose’ KPIs.  
Consultation and engagement was then undertaken to obtain stakeholder views 
regarding the KPIs.  Following consultation, the recommended KPIs were then 
operationalised by carefully defining how they should be measured.   

The final chapter concludes with recommendations concerning how the monitoring 
and evaluation process should be undertaken.  It reviews and examines how data 
collection, validation, analysis and dissemination processes can be developed to 
support the activities of 5G PPP projects, other researchers, stakeholders and the 
sectors and businesses most likely to be involved in the deployment and utilisation of 
5G.  The study recommends that an independent 5G Observatory should be 
developed to undertake 5G PPP monitoring.  Governance structures for the 
Observatory are suggested.   

13 

 
 
 
 
 
2. Overview  

2.1. 

Introduction 

The overall objective of the study described in the Tender Specifications was to 
provide a framework of realistic and detailed metrics to monitor the progress of the 
5G-PPP towards its objectives, and a methodology to assess the impact of its 
activities through the contractual KPIs.  

Operational objectives are: 

A.  Monitor and assess the Operating KPIs:  The study is required to 
deepen the definition of each of the operating KPIs taking into account 
the fact that 5G systems will reshape the telecom industry by providing 
new tools and capabilities for the infrastructure supply and manufacturing 
sectors, and as well reshape industries around a wider core of 
applications/services than today (in particular due to the blurring between 
the telecom, IT, and microelectronics) and modify the value chains and 
the role that existing suppliers will play within these future value chains. 

B.  Monitor and assess the Performance KPIs: The Tender 
Specifications highlight that the performance KPIs are based on the 
scientific and technical developments and indicators that relate to a wide 
range of areas including wireless and optical networks, convergence 
capabilities beyond the access last mile, network management and 
automation functions (including availability, robustness and security 
aspects), information centric networks, network functions virtualisation 
(NFV), software defined networks (SDN), networks of cloud and 
hardware implementations. 

C.  Monitor and assess the Societal KPIs:  The Tender Specifications 
emphasise that few other domains will likely have such a potentially 
transforming effect on society since 5G is a critical infrastructure in the 
emergence of a "hyper-connected" reality and digital society, where the 
distinction between offline and online is itself increasingly blurring, and 
the human-centric communication gets intertwined with a machine-type of 
communication. 

D.  Overall framework and assessment methodology:  It is noted that 
an important dimension of monitoring the 5G-PPP projects is to assess 
methodically and regularly the overall impact of the 5G PPP activities and 
projects in a monitoring framework where the operational, performance-
related and societal KPI's can be evaluated in an integrated way. 

The next section of this chapter provides an introduction to Key Performance 
Indicators and the role they can have in guiding and monitoring the performance of 
5G PPP projects. 

Section 2.3 provides an overview of indicators documented in 5G PPP contracts.  
These provided a starting point for developing operating, performance and societal 
KPIs. 

Section 2.4 utilises Better Regulation Toolbox methods and evaluation methods to 
develop a framework for the study and the development of KPIs.  The framework is 
based on key impact assessment components - input, output, outcomes and impact.  

14 

 
 
It provides a clear insight to the timeline for the achievement of the three groups of 
KPIs.   

Section 2.5 describes discussions and alignment with 12 KPIs developed by the 5G 
Infrastructure Association.   

2.2.  Key Performance Indicators: An introduction 

Key performance indicators measure the quality of an organisation’s performance 
and assist in monitoring performance and the achievement of wider goals.  In many 
cases, key performance indicators are posed as questions that need to be answered, 
and the answers provide the indications of organisational performance and strategic 
development.   

Carefully selecting and monitoring KPIs helps those working towards the KPIs to 
better understand objectives and monitor their performance4.  In essence KPIs define 
success and track progress in meeting strategic goals. This focus on strategic or 
long-term goals is what distinguishes KPIs from the wider array of ‘performance 
indicators’.  KPIs assist an agency to measure that it is ‘on track’ – most often, that it 
is working towards and attaining a beneficial outcome or improvement5.  In many 
cases, KPIs are used in projects and to measure service delivery6. 

Previous studies of PPP projects7 and the Seventh Framework Programme8 have 
highlighted shortcomings in achieving some of the strategic objectives and goals 
desire by the initiatives.  Most notably the need to promote higher-value-chain 
involvement   and/or industrial relevance, to make project results better known to 
business and citizens, better co-ordination and collaboration between projects and 
the need for projects to have greater impact on societal and economic goals. 

Thorough development of KPIs and monitoring can help to address the shortcomings 
identified in previous studies.  The KPIs provide projects with clear criteria against 
which they will be evaluated.  Regular monitoring measures performance and 
reinforces the need for projects to perform well against targets or in relation to their 
peer group.   

Well-designed KPIs help stakeholders and those working in projects to do a number 
of things9, including: 

  Establish baseline information (i.e., the current state of performance); 

  Set performance standards and targets to motivate continuous improvement; 

  Measure and report improvements over time; 

  Compare performance across geographic locations; 

  Benchmark performance against regional and international peers or norms; 

4 Lorette K.  2015.  How do I develop Key Performance Indicators?  

http://smallbusiness.chron.com/develop-key-performance-indicators-4676.html 

5 USAid.  2013.  Developing Key Performance Indicators.  https://www.hfgproject.org/developing-key-

performance-indicators-toolkit-health-sector-managers 

6 Public Records Office Victoria.  2015.   Key Performance Indicators.  http://prov.vic.gov.au/wp-

content/uploads/2011/05/1010g3.pdf 

7 EC.  2013.  Final assessment of the research PPPs in the European Economic Recovery Plan p54 to 

56.  The study considered three specific sectors.  But many of the comments were generic with 
relevance to all PPP evaluations. 

8 Tech4i2 and Deloitte.  2010. Impact assessment of  EC i2010. SMART 2007/0035. 
9 USAid.  2013.  Developing Key Performance Indicators.  https://www.hfgproject.org/developing-key-

performance-indicators-toolkit-health-sector-managers/ 

15 

 
 
                                                
  Allow stakeholders to independently judge sector performance. 

George Washington University provide a further list of operational benefits from the 
development of good KPIs10.  These include:- 

  Focus employees' attention on what matters most to success; 

  Allow measurement of accomplishments, not just of the work that is 

performed; 

  Provide a common language for communication; 

  Help reduce intangible uncertainty; 

  Are verifiable, to ensure data collection accuracy. 

This study describes how KPIs were developed to provide 5G PPP projects with 
clear criteria against which they will be evaluated.  It also provides details of how 
regular monitoring can be undertaken to reinforce the need for projects to perform 
well against targets or in relation to their peer group.   

A further important question is addressed in section 4.5 Should targets be 
established for KPIs?  Traditional use of KPIs sets performance standards and 
targets.  As the next section demonstrates many 5G PPP contracts have identified 
indicators, but few have established targets. 

2.3.  5G PPP Performance Indicators 

A relatively large number of potential performance indicators are documented in the 
5G PPP contracts and the Tender Specifications for SMART 2015/0013.  The 5G 
PPP contract provides clear and well-documented higher level indicators.  This 
section briefly considers the long list of potential KPIs that were considered suitable 
for the study.  

Six high-level KPIs are designated in Section 1.6 of the 5G PPP contract.  These 
include: 

  Providing 1,000 times higher wireless area capacity and more varied 

service capabilities compared to 2010. 

  Saving 90 per cent of energy per service provided. The main focus is in 

mobile communication networks where the dominating energy 
consumption comes from the radio access network. 

  Reducing the service creation time cycle from 90 hours to 90 minutes. 

  Creating a secure, reliable and dependable Internet with zero perceived 

downtime for services. 

  Facilitating very dense deployments of wireless communication links for 

over 7 trillion wireless devices serving over 7 billion people. 

  Ensuring User controlled privacy. 

59 additional lower level KPIs11 are documented on pages 20 to 46.  Table 1 
provides an overview of the high and lower level KPIs listed in the 5G PPP contract 
and the Tender Specifications. 

10 http://balancedscorecard.org/Resources/Performance-Measures-KPIs 
11 Seven low level Wireless network KPIs, 15 for optical networks, four for network management and 

16 

origination, two for implementing convergence beyond the access last mile, six for information centric 
networks, four for network function virtualization, three for software defined networking, two for 

 
 
                                                
The 5G PPP contract (page 61) suggests that the lower level are provided for 
different areas and they will be used as “a baseline for monitoring progress and 
industrial sector commitments towards the realisation of the PPP objectives”. 

Table 1 provides an overview of 65 potential KPIs that are considered in the 
proposed study.  The table divides the potential KPIs into the three categories of 
KPIs designated for the three tasks in the proposed study.  There are 13 Operating 
KPIs, 43 Performance indicators and nine Societal indicators.  

Table 1  Long list of KPIs for the study 

 Key Performance Indicators  Studied 

Source 

Operating KPIs 
Leverage effect of EU research and innovation funding 
in terms of private investment in R&D for 5G systems  

Tender Specifications and 5G-PPP 

contract 

Target SME participation commensurate with an 

Tender Specifications and 5G-PPP 

allocation of 20 per cent of the total public funding  
Reach a global market share for 5G equipment and 
services delivered by European headquartered ICT 
companies at, or above, the reported 2011 level of 43 
per cent global market share in communication 
infrastructure  

contract 

Tender Specifications and 5G-PPP 

contract 

Reduced OPEX in terms of energy consumption and 

5G-PPP contract (Automated 

complexity of human operator tasks 

Network Organisation) 

Minimised CAPEX and OPEX  

5G-PPP contract (Information Centric 

Networks) 

Minimise signalling and traffic overheads 

5G-PPP contract (Information Centric 

Open innovation ecosystem  

CAPEX and OPEX reduction 

Networks) 

5G-PPP contract (Network Function 

Virtualisation) 

5G-PPP contract (Network Function 

Virtualisation) 

Availability of standards on SLA negotiation  

5G-PPP contract (Networks of 

Percentage of industry (target > 50 per cent) including 
significant SME participation (target > 20 per cent) 
participation.  

Clouds) 

5G-PPP contract (Wireless networks) 

PPP roadmap refreshed every two years at least  

5G-PPP contract (Wireless networks) 

Number of peer reviewed scientific publications/results   5G-PPP contract (Wireless networks) 

Number of standards influenced by the research, and 

5G-PPP contract (Wireless networks) 

generated patents  

Performance KPIs 

 Key Performance Indicators  Studied 

Source 

Providing 1000 times higher wireless area capacity 

Tender Specifications and 5G-PPP 

network of clouds, two for ensuring availability and seven for ensuring efficient hardware 
implementation  

17 

 
 
 
 
 
 
 
 
 
                                                                                                                                      
and more varied service capabilities compared to 
2010  

contract 

Reducing the average service creation time cycle from 

Tender Specifications and 5G-PPP 

90 hours to 90 minutes (as compared to the 
equivalent time cycle in 2010  

contract 

Very dense deployments to connect over 7 trillion 
wireless devices serving over 7 billion people  

Tender Specifications and 5G-PPP 

contract 

Secure, reliable and dependable Internet with a "zero 

Tender Specifications and 5G-PPP 

perceived" downtime for services provision  

contract 

Improved network reliability and resilience  

Scalability of the mechanisms improved regarding 
number of coordinated elements and number of 
coordinated control loops 

Improvements in collecting and processing user data 

Quality of user behaviour characterisation  

5G-PPP contract (Automated 

Network Organisation) 

5G-PPP contract (Automated 

Network Organisation) 

5G-PPP contract (Availability, 
Robustness and Security) 
5G-PPP contract (Availability, 
Robustness and Security) 

Hardware performance improvements  

5G-PPP contract (Efficient Hardware 

Implementation) 

RF solutions such as improved battery life and 

5G-PPP contract (Efficient Hardware 

ensuring autonomy of up to 10-20 years; Reduce gap 
between μW range from possible harvested energy 
and mW range for electronics and RF 
communication; RF front-end receiver power 
consumption below 1 mW; Ultra-miniaturisation of 
antennas while maintaining high efficiency  

Implementation) 

Millimetre wave communications  

5G-PPP contract (Efficient Hardware 

Implementation) 

Digital architectures and components for transceivers 

5G-PPP contract (Efficient Hardware 

and micro-servers  

Implementation) 

Energy efficiency by using heterogeneous hardware in 

5G-PPP contract (Efficient Hardware 

combining parallel processor cores  

Implementation) 

Number of converged networks 

The factor of total increase in capacity  

5G-PPP contract (Convergence 

Beyond the Last Mile) 

5G-PPP contract (Convergence 

Beyond the Last Mile) 

Improvements in response times 

5G-PPP contract (Information Centric 

Networks) 

Demonstration and test network platforms  

5G-PPP contract (Information Centric 

Real deployments  

Networks) 

5G-PPP contract (Network Function 

Virtualisation) 

Demonstrations of interoperability  

5G-PPP contract (Network Function 

Core capacity increased by >10x 

Core reach increased up to 2500km 

Virtualisation) 

5G-PPP contract (Optical networks) 

5G-PPP contract (Optical networks) 

Core equipment density/bit increased by a factor of 10 

5G-PPP contract (Optical networks) 

Core energy reduction/bit increased by a factor of 10 

5G-PPP contract (Optical networks) 

Core automation and control improved (30 per cent 
network efficiency increase > 30 per cent OPEX 
savings) 

5G-PPP contract (Optical networks) 

Access and aggregation capacity increased by >10x 

5G-PPP contract (Optical networks) 

18 

Access and aggregation reach increased up to 100km 

5G-PPP contract (Optical networks) 

 
 
Access and aggregation equipment density/bit 

5G-PPP contract (Optical networks) 

increased by a factor of 10 

Access and aggregation energy reduction/bit increased 

5G-PPP contract (Optical networks) 

by a factor of 10 

Access and aggregation automation and control 

5G-PPP contract (Optical networks) 

improved (30 per cent network efficiency increase > 
30  per cent OPEX savings) 

Data centre capacity increased by >10x 

5G-PPP contract (Optical networks) 

Data centre reach increased up to 2kn (intra-DC) or 

5G-PPP contract (Optical networks) 

100km (inter-DC) 

Data centre equipment density/bit increased by a 

5G-PPP contract (Optical networks) 

factor of 10 

Data centre energy reduction/bit increased by a factor 

5G-PPP contract (Optical networks) 

of 10 

Data centre automation and control improved (30 per 
cent network efficiency increase > 30 per cent OPEX 
savings) 

5G-PPP contract (Optical networks) 

Reference implementations of a NetworkOS  

5G-PPP contract (Software Defined 

Networking) 

Availability of standards on NetworkOS concepts  

5G-PPP contract (Software Defined 

Networking) 

Greater number of SDN controlled network resources  

5G-PPP contract (Software Defined 

Backward compatibility  

Networking) 

5G-PPP contract (Software Defined 

Networking) 

Mobile data volume supported per area  

5G-PPP contract (Wireless networks) 

Amount of spectrum made available via new intelligent 
spectrum management techniques, target of doubling 
as a minimum 

5G-PPP contract (Wireless networks) 

Number of connected devices increased by 10x to 

5G-PPP contract (Wireless networks) 

100x 

Increased battery life for low power devices (10x) 

5G-PPP contract (Wireless networks) 

End-to-end latency reduced by a factor of 5 

5G-PPP contract (Wireless networks) 

19 

 
 
   
 
 
 
Societal KPIs 

 Key Performance Indicators  Studied 

Source 

Enabling advanced user controlled privacy  

Tender Specifications and 5G-PPP 

contract 

Reduction of energy consumption per service up to 90 

Tender Specifications and 5G-PPP 

per cent (as compared to 2010)  

contract 

European availability of a competitive industrial offer 

Tender Specifications and 5G-PPP 

for 5G systems and technologies  

contract 

New economically-viable services of high societal 

Tender Specifications and 5G-PPP 

value like U-HDTV and M2M applications  

contract 

Establishment and availability of 5G skills development 
curricula in partnership with the European Institute of 
Innovation and Technology (EIT)  

Tender Specifications and 5G-PPP 

contract 

QoE/QoS - increasing customer satisfaction in terms of 

5G-PPP contract (Automated 

throughput, handover reliability and call drop rate 
Improved data access and discovery in terms of QoS 

Network Organisation) 

5G-PPP contract (Information Centric 

and QoE  

Networks) 

QoS requirements in the cloud environment 

5G-PPP contract (Networks of 

Clouds) 

Lower energy consumption  

5G-PPP contract (Wireless networks) 

Figure 2 presents an overview of the three categories of indicators (presented above 
in Table 1) and the likely chronology for achievement of the objectives. 

Source: Study team conceptualization aligning tasks with key components in EC 2009 Impact Assessment 
Guidelines – inputs, outputs, outcomes and impacts. 

Figure 2 Timespans for achievement of the three groups of KPIs 

20 

 
 
 
 
 
 
 
It is evident that the three groups of KPIs will be achieved over different timescales, 
see Figure 2.   

2.4.  The framework, chronology and KPI question methods 

2.4.1.  Framework 

Preceding sections have focused on the purpose of utilising KPIs and provided an 
initial insight to three groups of indicators presented in 5G PPP contracts and the 
likely timescales over which the objectives for each indicator will be achieved.  This 
section provides an overview of the framework and logic for the study 

Figure 2 provided a clear overview of the three categories of KPIs to be evaluated 
and the time span over which each KPI category will be achieved.  In general: 

  Operating KPIs will be achieved now or in the near future; 

  Performance KPIs are only likely to be achieved towards the end of 5G 

PPP projects; 

  Societal KPIs are only likely to be achieved when 5G technologies are 

distributed in four or more years time. 

This examination of KPIs arising from 5G PPP projects is well aligned with traditional 
evaluation and monitoring activities.  In order to illustrate the traditional approach we 
use the figure below, which we elaborate and improve from the 2004 European 
Commission guidelines and Better Regulation Toolbox12, see Figure 3.  The 
Commission’s evaluation and impact assessment system has been assessed by 
numerous independent bodies and found to be on a par with international standards - 
transparently assessing legislative and non-legislative options by comparing both 
potential benefits and costs in economic, social and environmental terms and by 
subjecting the assessments to rigorous scrutiny.  

Source: Study team elaboration of EC 2004 Impact Assessment Guidelines  

Figure 3 The logic chain for evaluation 

12 EC.  2015.  Better Regulation Toolbox.  Tools #41 and #56.  And EC.  2004.  Evaluating EU activities: 
A practical guide for the Commission services.  These guidelines were later supported by the 2009 
EC Impact Assessment Guidelines, these guidelines were further refined during a 2014 revision – EC. 
2014.  2014 revision of the European impact assessment guidelines: Public consultation document. 

21 

 
 
 
                                                
The inputs, outputs, outcomes and impact structure presented in Figure 3 is 
common across a number of monitoring and evaluation approaches adopted by most 
international organisations13. 

Monitoring and measurement activities usually focus on logic chains that are 
controllable (inputs and outputs).  In the 5G PPP study the Commission controls and 
provides the inputs or resources allocated to 5G PPP projects.   

The 19 projects control the outputs that they can achieve.  The selected projects 
should already have in place (or possess the means to acquire it via EC funding) the 
financial, human capital, technological, knowledge and other resources to provide the 
outputs or 5G technical developments that their project aims to achieve.   

The 5G PPP projects cannot control how the knowledge and prototypes they will 
create will be used to produce outcomes.  An example of an outcome could be the 
use of 5G capabilities in vehicles.  Exogenous factors likely to affect this outcome are 
the views of vehicle manufacturers and their consumers about the benefits of 5G 
capabilities in vehicles.  Both of these factors are beyond the control of the 5G PPP 
projects.   

Additional non-controllable exogenous factors will arise when one tries to induce or 
analyse longer-term impacts.  Desirable impacts of the use of 5G capabilities 
developed by 5G PPP projects could include a reduction in hydro-carbons or 
pollution.  These knock-on or trickle down impacts will only arise when 5G is more 
widely used in vehicles (the outcome in the previous paragraph) and vehicle 
manufacturers, consumers, policymakers and other third parties allow data 
generated by 5G enabled by vehicles to be better utilised to regulate or support a 
reduction congestion.  Thus for impacts the exogenous factors are generally more 
numerous and wider ranging.  Common and more wide ranging exogenous factors 
include economic conditions, social factors and cultural differences between groups. 

2.4.2.  Chronology 

The preceding short overview demonstrates how there is a temporal dimension to 
inputs, outputs, outcomes and impacts. As noted in the bullet point list above - Inputs 
or the activities of 5G PPP projects will be happening now. Outputs are likely to be 
achieved towards the end of the activities of projects. Outcomes will probably only 
start to take place after 5G PPP projects have concluded and the value of the 
capabilities developed have been recognised as beneficial, this is likely to be three to 
five years away. Significant impacts are likely to be eight or more years away.  It is 
useful to provide a brief note on the terminology used for measurement and 
evaluation in Figure 3, before going on to relate these to the three groups of KPIs: 

 

Inputs provide initiatives with resources to undertake activities;  

  Outputs are the final product of initiatives.  The production of these is 
generally within the control of those implementing them. These will be 
aspirational at the start of initiatives and known or measurable at the 
conclusion of initiatives; 

  Outcomes are the direct and intermediate changes produced for specific 

constituencies as a result of initiatives.  Their occurrence generally depends 
also on some intervening or exogenous variables; 

22 

13 For example the OECD. 2013.  Evaluating development activities.  United Nations. 2011.  Results-

based management handbook.  ILO. 2013. ILO evaluation manager handbook.   

 
 
                                                
  The term Impacts is used to indicate broader and longer-term changes for 
economy and society as a whole. A larger number and/or wider range of 
intervening or exogenous variables generally influence the achievement of 
impacts. 

In the study the concept of inputs is extended beyond the usual terminology that 
focuses on resources provided to an initiative.  In our study it also includes 
consideration and monitoring of the activities of 5G PPP projects. This slight 
extension is required to more easily accommodate Task A  Operating KPIs.  This 
extension can be tautologically justified by highlighting that the outputs (development 
of 5G capabilities) is not just achieved by the provision of resources by the 
Commission.  These resources have to be used to undertake a variety of activities 
(research, pro-type development, market testing etc.) to achieve the outputs.  Thus in 
the augmented overview of the study framework the term ‘inputs’ is substituted with 
‘activities’ to better capture the range of activities that 5G PPP projects will 
undertake, see Figure 4. 

Source: Study team elaboration of EC 2009 Impact Assessment Guidelines  

Figure 4 The framework for the proposed study 

The preceding overview has highlighted that the underlying framework for the study 
is grounded in robust and well established evaluation methodologies. 

23 

 
 
 
 
 
2.4.3.  KPI reliability over time 

It is important to highlight that the reliability of information provided by 5G PPP 
project respondents and other stakeholders is likely to become increasingly 
unreliable the further into the future that questions/KPIs examine.  Respondents 
should be able to unequivocally answer questions and provide evidence of what they 
are doing now in relation to inputs/activities, for example when examining SME 
participation.  Some activities may not be completed until the end of the 5G project, 
for example the total amount of leverage.  For these questions it will be wise to ask 
the current level of progress and anticipated achievements at the end of the project.  
Subsequent rounds of questions (perhaps six monthly) for 5G PPP projects can then 
update achievements and re-examine progress against initial perceived 
achievements.  This will provide a useful check on over-optimism in early rounds of 
KPI data collection from projects. 

KPIs about outputs will have to adopt a similar approach to the Task A KPIs 
extending over the duration of the 5G PPP project (leverage was used as an 
example).  KPIs about outputs and the development of 5G technical capabilities will 
have to focus in the first instance about asking questions about anticipated 
achievements at the end of the project.  Subsequent rounds of questions (perhaps 
six monthly) can then update achievements and re-examine progress against initial 
perceived outputs and investigate ‘slippage’ or degradation in 5G technical 
capabilities that projects will achieve. 

The team for the proposed research have recently completed the study on the 
‘Identification and quantification of key socio-economic data for the strategic planning 
of 5G introduction in Europe’ (SMART 2014/0008).  Input from a large number of 
experts and those attending workshops has shown that few people have the 
forecasting skills, underlying research information and job requirements to consider 
the longer term outcomes and impacts that might arise from improved 5G 
capabilities four more years hence.  It is likely that mainstream commercial utilisation 
of 5G capabilities will not take place for six to eight years.   

2.4.4.  The characteristics of the KPIs 

The remainder of this section continues the consideration of the nature and purpose 
of KPIs and related questions by examining the three categories of KPIs to be 
examined in the proposed study. 

Operating KPIs (Task A) largely relate to the activities being undertaken now and 
over the duration of the 5G PPP contractual period by the organisations involved in 
the 5G PPP projects.  Many of the proposed Task A KPIs are factual (SME 
participation, leverage and skills development etc.).  Others relate to KPIs that will be 
achieved over the lifetime of the study (market share, influence on standards, 
creation of IPR etc.), for most this is a three-year duration.  This later group of 
proposed KPIs will be further reviewed to ensure they align with the study framework.  
It might be more consistent to re-allocate some (such as influence on global 
standardisation) as outputs rather and inputs/activities.   

Put simply (utilising the framework described above) the Task A KPIs examine the 
current activities of 5G PPP projects and the activities they will undertake before the 
end of their EC funding period.  The projects are providing the inputs/activities to 
develop the 5G technological capabilities outlined in Task B.  These technological 
capabilities will be the outputs from the 5G PPP projects. 

24 

 
 
 
 
The nature of these KPIs – providing inputs now to achieve outputs will determine 
the nature of questions and manner in which they can be asked.  Respondents will 
know what they are doing now they should have factual evidence to support their 
answers 

In the case of KPIs to be achieved in the near future, for example influence on global 
standardisation and creation of IPR, SMART14 KPIs will need to be developed to 
provide a clear insight to desired activities.  Requests for supporting information to 
‘evidence’ responses to these KPIs could include details of the number and/or 
importance of meetings to be attended and lobbying methods for influence, key 
areas where IPR is being achieved and identification of how this will be recognised 
(patents, licences, creative commons licences).  If these do not currently exist, the 
study should act as a catalyst to prompt respondents to provide clear goals and start 
collecting required ‘evidence’. 

Performance KPIs  (Task B) largely relate to the technological capability 
developments or outputs from 5G PPP projects.  These are only likely to come to 
fruition towards the end of the 36 month (maximum) period for which most projects 
run15, see section 2.4. 

As noted above in the early months/years of these projects it will probably only be 
possible to ask about the expected eventual 5G capability and progress towards 
achieving the performance KPIs and the magnitude of outputs (for example number 
of times wireless capacity [900 times, 1,000 times, 1,100 times etc.], reduction in 
service creation time [80 minutes, 90 minutes 100 minutes etc.]) and the likely 
timespan to achieve these outputs. 

As projects progress it should be possible to obtain more precise information about 
the magnitude of KPIs together with evidence (documents) that the objective has 
been achieved.  

Societal KPIs (Task C) This final group largely relate to the outcomes and impacts 
that will arise from the utilisation of 5G technical capabilities (developed by 5G PPP 
projects and subsequently brought to market) by consumers, businesses, 
government and other organisations.  They will arise in verticals (industries) and 
different environments (cities, semi-urban, rural, workplaces, homes, public buildings 
and other locations). 

As noted above these benefits are very unlikely to arise during the duration of the 5G 
PPP projects.  While some early benefits may start appearing as early as 2018, 
many observers have suggested 5G capabilities will not become widely commercially 
available until after 202016.  Investigation of these benefits will therefore always be 
speculative and reliant on the perceptions of respondents about outcomes and 
impacts that will arise from their projects.   

Whilst it will be possible, and is probably politically desirable to introduce Task C 
KPIs about societal impacts, we suggest that reliance on views suggested by 
respondents about longer-term KPIs are treated with caution.  As noted previously 
the reliability of information and/or perceptions provided by respondents is likely to 
become increasingly unreliable the further into the future KPI questions are 
investigated.  We therefore suggest that Task C societal KPIs are limited in number 

14 Specific, measurable, achievable, realistic and time-dependent questions or parameters 
15 The 5G PPP web site reports that four projects are due to conclude in July 2017, one in September 
2017, one in October 2017, nine in December 2017, one in March 2018 and three in July 2018. See 
https://5g-ppp.eu/5g-ppp-phase-1-projects 

16 Telecoms Intelligence.  2016.  Road to 5G Outlook 2016.  http://telecoms.com/intelligence/download-

the-road-to-5g-outlook-2016-now/ 

25 

 
 
                                                
and carefully chosen to provide a clear focus on issues about which project 
respondents and/or others in the value chain might have some knowledge.  
Nonetheless, next section briefly describes how the study framework can be further 
developed in the proposed study to categorise and record key economic and societal 
KPIs. 

2.5.  5G PPP KPI and evaluation discussions 

Our team has held discussions with Jean Sebastien Bedo and Didier Bourse on a 
number of occasions during the study.  They are key personnel within the 5G 
Infrastructure Association also working on the development of KPIs. 

The 5G Infrastructure Association provided our team with details of 12 KPIs, see 
Table 2.  The rationale for selecting the 12 KPIs were not provided.  They suggest 
three operating KPIs, five performance and four societal KPIs.  They are discussing 
these with technology boards and technology experts from the 5G PPP projects.  
These 12 5G Infrastructure Association KPIS have been cross referenced with KPIs 
recommended in this study.  Details are provided in the table and see section 4.2. 

Table 2 Twelve KPIs developed by the 5G Infrastructure Association 

Group 

 Operating KPIs 

KPIs 

B1 

Leverage effect of EU research and innovation funding in terms of private investment 

in R&D for 5G systems in the order of 5 to 10 times  - Section 4.2 KPI B1 

B2 

Target SME participation under this initiative commensurate with an allocation of 20 

per cent of the total public funding  - Section 4.2 KPI B3 

B3 

Reach a global market share for 5G equipment and services delivered by European 
headquartered ICT companies at, or above, the reported 2011 level of 43 per cent 
global market share in communication  - Section 4.2 KPI S7 

Performance KPIs 

P1 

Providing 1000 times higher wireless area capacity and more varied service 

capabilities compared to 2010  - Section 4.2 KPI P1 

P2 

Reducing the average service creation time cycle from 90 hours to 90 minutes  - 

Section 4.2 KPI P2 

P3 

Facilitating very dense deployments of wireless communication links to connect over 

7 trillion wireless devices serving over 7 billion people  - Section 4.2 KPI P7 

P4 

Creating a secure, reliable and dependable internet with a "zero perceived" downtime 

for services provision  - Section 4.2 KPI S4 

P5 

Reduction of energy consumption per service up to 90 per cent (as compared to 

2010)  - Section 4.2 KPI P4 

Societal KPIs 

S1 

S2 

Enabling advanced user controlled privacy  - Section 4.2 KPI S2 

European availability of a competitive industrial offer for 5G systems and 

technologies  - Section 4.2 KPI S6, S8, S9, S10 

S3 

Stimulation of new economically-viable services of high societal value like U-HDTV 

and M2M applications  - Section 4.2 KPI S1 

S4 

Establishment and availability of 5G skills development curricula   -  Section 4.2 S5 

26 

 
 
 
 
 
3. Shortlisting the KPIs 

3.1. 

Introduction 

This chapter provides an overview of the first short-listing process that derived a 
‘medium’ list of approximately 30 KPIs.  These activities comprised Stages 1, 2 and 3 
of the study presented in Figure 1. 

The next section provides an overview of the relevance of KPIs identified in the 
previous chapter to the 19 5G PPP projects funded by the first phase of funding.  
More than 1,235 assessments of KPIs and projects were undertaken. 

Analysis identifies a medium list of 11 Operating KPIs in section 3.3, 13 Performance 
KPIs in section 3.4 and six Societal KPIs in section 3.5.   

Section 3.6 presents the results of gap analysis to identify areas not adequately 
covered by the medium list of KPIs.   

3.2.  From long-list to medium-list 

The 65 KPIs identified in Table 1 were mapped for relevance against the 19 first 
round 5G PPP projects.  Thus, in total, 1,235 assessments were made of KPIs and 
projects.   

Our team carefully read details of all 5G PPP projects17 and talked to those involved 
with them.  It must however be accepted that the cross-tabulation process was 
subjective, despite being based on the best knowledge available about projects in 
the public domain. 

Due to a shortage of public information in relation to KPIs on 5G-PPP projects, some 
projects were more difficult than others to map against KPIs.  Many projects had not 
yet made their initial deliverables publicly available when this activity was undertaken 
in May 2016.  5G-PPP projects such as Superfluidity and Flex5Gware provided more 
detailed information and relevant KPIs. This helped our team to map KPIs. However, 
others have more limited information available about their goals and objectives. 

The results presented in this section and Table 3 were therefore based upon the 
best information available in May 2016.   
Table 3 uses the Likert scale14 to display information about operating, performance 
and societal KPIs cross-tabulated with the 19 5G PPP projects.  The study team 
undertook the assessment of the relevance of KPIs to 5G-PPP projects in Table 3.  
The table shows the study team’s perception of the relevance of the long list of 65 
potential KPIs presented in Table 1 to the 5G-PPP projects.  

This table shows that 26 per cent of all KPIs had a Likert18 score indicating of “high 
relevance” (four) or “very high relevance” (five) in our analysis.  

17 Evidence for Likert scoring is primarily sourced from the 5G-PPP official project websites and 

discussions with members of projects. More specifically, information is mainly cross-referenced with 
areas of these websites which list the 5G-PPP goals, aims and objectives. 

18 Likert scales are one of the most widely used approaches for scaling responses, particularly in survey 

research.  When responding to a Likert question respondents specify their level of agreement or 
disagreement on a symmetric agree-disagree scale (usually with five or seven categories) for a series 
of statements.  Thus, the range captures the intensity of respondents’ feelings for a given item.  A 
scale can be created as the simple sum of questionnaire responses over the full range of the scale.  
In so doing, Likert scaling assumes that distances on each item are equal. 

27 

 
 
                                                
This is in contrast to 74 per cent of KPIs that have a rating of “very low”, “low” or 
“average” relevance in our analysis.  The majority of cross tabulations (56 per cent 
with a Likert scale of one) had little relevance to 5G PPP projects. 

Table 3  Overview of Likert Scale analysis 

Likert Score 

KPI Group 

Operating KPIs 

Performance KPIs 

Societal KPIs 

Very 
Low 

43 

529 

58 

Low 

Med. 

High 

Very 
High 

Total 

15 

60 

0 

43 

66 

20 

21 

10 

16 

93 

92 

57 

215 

757 

151 

Total (percentage) 

630 (56) 

75   
(7%) 

129 
(11%) 

47   
(4%) 

242 
(22%) 

112319 

The remainder of this chapter focuses on the KPIs that had a score of 4 or 5 in the 
cross-tabulation process, thus indicating “high relevance” (four) or “very high 
relevance” (five) to the 5G PPP projects.   

3.3. 

 Operating KPIs 

Table 1 identified 13 Operating KPIs.  Figure 5 provides an overview of the KPIs that 
had relevance to eight or more of the 5G PPP projects.  In total these included 11 of 
the 13 Operating KPIs.   

19 The 65 KPIs mapped against the 19 5G PPP projects should have produced 1,235 assessments.  In 
the event some assessments were so irrelevant or hard to analyse (but definitely not relevant) that no 
judgement was made.  Thus the total in Table 2 only relates to 1,123 assessments where a score 
could more accurately be provided.  The remaining 112 cross-tabulations (1,235 - 1,123) were not 
relevant and no score was provided. 

28 

 
 
 
 
 
                                                
Figure 5  The medium list of operating KPIs 

The four most relevant KPIs across all 5G-PPP projects for KPIs are “Leverage effect 
of EU research and innovation funding”, “Target SME participation commensurate 
with an allocation of 20 per cent of the total public funding”, “Reach a global market 
share for 5G equipment and services” and “Percentage of industry (target > 50 per 
cent) including significant SME participation (target > 20 per cent) participation”. 

As indicated in Figure 5, there are 11 operating KPIs shortlisted in total.  It was 
recommended that eleven KPIs should be included in the medium list of KPIs for 
further consideration.  Two KPIs were rejected because they did not have sufficient 
relevance to 5G PPP projects.  However, it is important to highlight that rejection of 
KPIs was not just a numerical exercise.  All rejected KPIs were carefully considered 
by our team to determine whether they might still signify relevant metrics even if not 

29 

 
 
 
 
widely applicable.  Nonetheless, a review of the two rejected variables decided that 
they were not thought to be important KPIs20. 

Fulsome analysis of all Operating KPIs studied can be found in the Interim Report. 

3.4.  Performance KPIs  

Table 1 identified 43 Performance KPIs. Figure 6 provides an overview of the 13 
KPIs that had relevance to five or more of the 5G PPP projects.   

Figure 6  The medium list of Performance KPIs 

The three most relevant KPIs across all 5G-PPP projects for Performance KPIs are 
“Providing 1000 times higher wireless area capacity and more varied service 
capabilities compared to 2010” (which was relevant to nine projects), “Reducing the 
average service creation time cycle from 90 hours to 90 minutes (as compared to the 
equivalent time cycle in 2010)” and “Secure, reliable and dependable Internet with a 

30 

20 The two were:- PPP roadmap refreshed every two years at least and Number of peer reviewed 

scientific publications/results. 

 
 
 
                                                
"zero perceived" downtime for services provision”. The latter two were only relevant 
to eight projects. 

The 13 Performance related KPIs shortlisted in Figure 6 were recommended for 
inclusion in the medium list of KPIs for further research in the study.  

Fulsome analysis of all Performance KPIs studied can be found in the annex in 
section 7.2. This provides an overview of the 30 KPIs that were long-listed but not 
recommended for inclusion in the medium list.  None were thought to be relevant to 
the evaluation process. 

3.5.  Societal KPIs  

Table 1 identified nine societal KPIs. Figure 7 provides an overview of the six KPIs 
that had relevance to seven or more of the 5G PPP projects.   

Figure 7  The medium list of Societal KPIs 

The three most relevant KPIs across all 5G-PPP projects for Societal KPIs are 
“enabling advanced user controlled privacy”, “Reduction of energy consumption per 
service up to 90 per cent (as compared to 2010)” and “European availability of a 
competitive industrial offer for 5G systems and technologies”.  

The six Performance related KPIs shortlisted in Figure 7 were recommended for 
inclusion in the medium list of KPIs for further research in the study.  

31 

 
 
  
 
Fulsome analysis of all Societal KPIs studied can be found in the annex in section 
7.3. Careful consideration of the three rejected KPIs confirmed that they were not 
very important for the overall evaluation exercise21. 

3.6.  Gap analysis 

This section provides the results of gap analysis to identify KPIs not identified in the 
5G PPP contracts and other relevant 5G documents. As with the shortlisting process 
in the previous section this gap analysis refers only to analysis of work underway in 
the Phase 1 projects.  A similar process could be engaged to update the KPIs when 
the next phase of projects is underway.  

It is noteworthy that 5G is a large and moving target and consequently Performance 
KPIs which were set out at contract stage may not fully align with or capture the 
current trajectory of 5G as it evolves and matures towards standardisation. For 
example, the increased focus on vertical markets22 and the accelerated maturation of 
new Internet of Things solutions such as the LPWAN offerings has served to both 
clarify and expand the diverse and sometimes divergent capabilities that 5G must 
support23.   

A significant piece of work published at the end of 2015 by the ITU-R Focus Group 
on IMT-2020 identifies a number of gaps in the current standardisation efforts for 
5G24. That report has played a significant role in helping the study team evaluate how 
the existing set of down-selected KPIs could be improved to capture the work of the 
current and future 5G PPP projects phases. 

Softwarisation of network functionality forms a large part of the objectives for 5G. 
This is captured to some extent by the existing Performance KPIs, mainly those 
indicated by the 5G-PPP contracts. These KPIs generally address issues of network 
function virtualisation, software defined networking and the development of a 
Network OS for 5G systems. In evaluating the existing 19 projects we took a liberal 
approach to classifying the contribution of projects to meeting these KPIs. In 
particular, while the actual term Network OS was not used in the project material 
available to us, the descriptions of work we could access indicated that some efforts 
aligned with this group of KPIs was being undertaken in this area. This is reflected in 
two Network OS KPIs being selected for the medium list. 

However, one area of work within this group of KPIs that is not picked up by the 
existing softwarisation KPIs is in the area of support for mobile edge computing 
(MEC). MEC has been rapidly gaining traction within the 5G community as an 
expression of the use of NFV, SDN etc at the edge of the network25. MEC presents a 

21 The three were QoE/QoS - increasing customer satisfaction in terms of throughput, handover 

reliability and call drop rate.  New economically-viable services of high societal value like U-HDTV and 
M2M applications and Establishment and availability of 5G skills development curricula in partnership 
with the European Institute of Innovation and Technology 

22 Vertical markets white papers produced by the 5G Infrastructure Association https://5g-ppp.eu/white-

papers/ 

23 The rapid standardization by the 3GPPP of an NB-IoT solution (approved in June 2016 

http://www.3gpp.org/news-events/3gpp-news/1785-nb_iot_complete) comparable to non-3GPPP 
LPWAN offerings such as SigFox and LoRa has brought some focus to the challenges for 5G in 
accommodating RATs that differ significantly from those in the existing 4G portfolio.  

24 Focus Group IMT-2020: Report on Standards Gap Analysis, December 2015.  

http://www.itu.int/en/ITU-T/focusgroups/imt-2020/Documents/T13-SG13-151130-TD-PLEN-
0208%21%21MSW-E.docx 

25 ETSI White Paper No. 11 “Mobile Edge Computing A key technology towards 5G”, September 2015 
http://www.etsi.org/images/files/ETSIWhitePapers/etsi_wp11_mec_a_key_technology_towards_5g.pd
f 2015  

32 

 
 
                                                
new and distinct set of standardisation challenges that are not captured by the 
existing KPIs and which are targeted more to the 5G core and to 5G systems as a 
whole, i.e. they do not address the unique challenges posed by the MEC concept of 
placing core-like functionality at the edge in support of applications that require 
proximity between the application server and the end user or require the ability for 
the application server to directly interact with the RAN. As described in the ITU Focus 
Group report, MEC presents particularly high priority challenges in terms of; lifecycle 
management of MEC applications at the edge, support for authorised third-parties at 
the edge, inter-edge mobility for applications moving between edge devices and 
support for traffic routing among MEC applications, among others.  

As a distinct KPI, QoS is only addressed by the Societal KPIs. The medium list 
contains one QoS KPI -  QoS requirements in the cloud environment (Source: 5G-
PPP contract (Networks of Clouds)). While other Performance KPIs may implicitly 
capture QoS, this is not obvious and projects that are addressing this important area 
cannot be identified. 

The ITU Focus Group highlights end-to-end QoS as being a distinct gap in the 
current standardisation efforts of 5G. In particular it notes that the definition of end-to-
end needs to be comprehensively revised for 5G where end-to-end paths will 
encompass a variety of new routes, e.g. device-to-device, device-to-edge, etc. And in 
support of new performance targets, such as ultra low latency, and new types of 
traffic and services, such as automation and augmented reality, the existing QoS 
methods related to human-to-human and human-to-machine connectivity do not 
suffice.  

The growing importance for 5G to address the needs of vertical market sectors 
means that 5G systems should have the ability to support domain-specific service 
level agreements (SLAs). The ability to measure and monitor domain-related QoS 
metrics on an end-to-end basis across diverse sets of operators, tenants, services, 
RATs and terminals will be key to support the diverse 5G ecosystems and SLAs that 
may be entered into. As such, a Performance KPI related to QoS would capture 
developments within the 5G PPP.  

Finally, it is apparent from the evaluation of the Performance KPIs against the 19 5G-
PPPs that there is a mismatch between the KPIs identified in the 5G-PPP contracts 
(Optical Networks) and the work related to optical networking being undertaken in the 
funded projects. None of these KPIs make the cut in the initial down-selection to a 
medium list, as they are not currently fit to capture the work being undertaken in the 
projects.  

In particular it can be seen that the KPIs related to Data centre and Core network 
optical networking are not addressed by the Phase 1 5G PPP projects. While the 
projects do address some of the KPIs under the Access and Aggregation heading of 
the Optical Networks KPIs, the KPIs themselves do not lend themselves to capture 
work in the projects and consequently do not qualify for the medium list. 

The ITU-R Focus Group paper notes that optical networking issues relating to mobile 
front haul and back haul networking for 5G are seen as a gap. It is apparent that 
some of this work is being addressed in the current projects, e.g. the work of 5G-
Crosshaul and 5G-XHaul, and will be a focus for Phase II. In the ITU-R Focus Group 
document, the objectives are cast differently to the KPIs that were set out in the 
original 5G PPP contracts. In addition to addressing the need for new CPRI protocols 
in this part of the network and the development of virtualisation of PON services, the 
Focus Group highlighted the challenges brought about by the diversity of 5G as 
being particularly important to address. That includes front haul/back haul support for 
the diversity of terminals (ranging from IoT gateways to MEC servers, etc.), diversity 
of traffic types (implying the ability to increase and decrease capacity and QoS), 

33 

 
 
diversity of network operator types and diversity of RATs. These issues are being 
addressed in the current 5G PPP projects and would be better captured by a 
reformulated optical networking KPI. 

Based on the above, three additional KPIs are suggested for inclusion in the medium 
list of KPIs to address the gaps identified. These KPIs are Support for Mobile Edge 
Computing, QoS for new types of services and new end-to-end paths and Front 
haul/back haul support for diverse types of terminals, traffic, network operators and 
RANs. 

3.7.  Conclusion 
This chapter has presented the methodology by which the long list of 65 KPIs was 
reduced to a medium list of 30 KPIs. 

The next chapter provides an overview of how the medium list was further refined to 
provide a definitive set ‘fit for purpose’ KPIs. 

34 

 
 
 
 
4. Shortlisted KPIs and consultation   

4.1. 

Introduction 

The previous chapter presented details of the medium listing process to generate 30 
KPIs.   

This chapter provides an overview of stage 4 to 6 of the methodology presented in 
the introduction; see Figure 1.  It describes how the medium list of KPIs was further 
refined to ensure the KPIs were ‘fit for purpose’. 

Section 4.2 describes how previous research, best practice guidelines and 
consideration of users needs were combined to provide an overview of the rationale 
for KPIs and to refine their focus. 

Section 4.3 describes the consultation and engagement process undertaken to 
obtain stakeholder views about KPIs.  Following consultation the recommended KPIs 
were then operationalised by carefully defining how they should be measured in 
section 4.4.  

Section 4.5 considers the merits of establishing targets for KPIs.  Previous relevant 
initiatives were analysed to provide an insight to relevant targets that might be 
appropriate for the 5G PPP KPIs. 

The chapter concludes with recommendations regarding how the monitoring and 
evaluation process should be undertaken.  This focuses on issues concerning 
governance, monitoring activities and dissemination of 5G PPP KPI progress. 

4.2.  Refining the medium list of KPIs 

In deriving the medium list of KPIs, careful research was undertaken for each KPI to 
ensure that KPIs were placed in the appropriate groups26 and to reduce duplication27. 

Most importantly the rationale for including KPIs was reconsidered.  This review 
process included the examination of European Parliament and other Commission 
requirements regarding the goals and outputs expected from Horizon 2020 (and 
historically from FP7).  It also investigated lessons learned from previous evaluation 
studies of FP7 and PPP initiatives.  Analysis also investigated expectations regarding 
5G capabilities found in the 5G Action Plan and other documents produced by 
leading 5G organisations. 

In essence the review process ensured that the KPIs met European objectives, were 
aligned with the expectations of leading 5G organisations and they avoided pitfalls 
and adhered to best practices identified in previous monitoring and evaluation 
exercises. 

To provide further insights to the nature and comprehensiveness of the KPIs simple 
graphic visualisations were developed to highlight the key areas identified by KPIs.   

26 For example the medium list included lower energy consumption as a societal KPIS.  This KPIS was 

transferred to the performance KPI group.   

27 Several KPIs had overlapping characteristics.  For example two KPIs suggested the requirement for 

CAPEX and OPEX reduction. 

35 

 
 
 
                                                
The remainder of this section describes the key themes for each group of KPIs, It 
provides a graphic visualisation and provides clear rationale for the inclusion of the 
KPIs. 

4.2.1.   Operating KPIs  
The seven operating KPIs focus on three key areas:- 

  The operational aspects of projects (leverage and make-up of partnerships); 

  Contributions to the development of standards; 

  Dissemination of output and ideas. 

A simple graphic below provides a representation of the KPIs in relation to the three 
key areas of focus.  The graphic is followed by an overview of the KPIs and the 
rationale for seeking information.   

Figure 8  Graphical representation of the Operating KPIs 

An overview of the KPIs and the rationale for seeking information are provided 
below.  For each KPI the right hand column provides an ‘evidence’ base for why the 
indicator should be included.  Some of these rationales relate to requirements 
articulated by the European Commission.  Others are related to expectations 
regarding 5G capabilities found in the 5G Action Plan28 and other documents 
produced by leading 5G organisations.  

36 

28 EC.  2016.  5G for Europe: An Action Plan.  http://ec.europa.eu/newsroom/dae/document.cfm? 

doc_id= 17131 

 
 
 
 
 
 
                                                
Table 4  Rationale for the Operating KPIs 

KPI  

Rationale  

Leverage arising from 
5G PPP funding O1* 

Industry participation in 
5G PPP projects O2 

The 5G PPP contract targets leverage from private investment in R&D for 

5G systems in the order of five to ten times.  

There is a need to move the innovation chain closer to the market. 

Research PPPs should link with downstream activities   in the whole 
value.  PPPs should promote higher-value-chain involvement   or 
industrial relevance29.  The 5G Action Plan highlights the need for industry 
consultation and input30 

SME participation in 5G 

SMEs have faster innovation cycles and the ability to open niche markets 

PPP projects O3* 

and to bring disruptive technologies to the market and grow 
employment31.   

Contribution to 5G 

Standards 
negotiations O4 

Contribution to Service 
Level Agreement 
negotiations O5 

Research papers, 

patents and 
demonstrators arising 
from 5G PPP projects 
O6 

PPPs should play a greater role in standardisation activities and utilise 
project results and their expertise to focus on standardisation actions 
benefit European industry32. 

PPPs should play a role in Service Level Agreement (SLA) activities and 

utilise project results and their expertise to focus on SLA actions to benefit 
European industry33. 

The PPP community needs to make project results better known to the 

wider public by sharing solutions to common problems, exchanging best 
practices, joint dissemination and promotion of results and 
demonstrators.34   

Support for open 

innovation 
ecosystems O7 

Within PPP projects higher degrees of co-ordination and collaboration 
between projects has been a key element of research and innovation 
success35.  

The asterisk in this table indicate KPIs that correspond with one of the 12 KPIs suggested by the 5G 
Infrastructure association, see Table 2. 

29 EC.  2013.  Final assessment of the research PPPs in the European Economic Recovery Plan p54 to 

56.  The study considered three specific sectors.  But many of the comments were generic with 
relevance to all PPP evaluations.  

30 EC.  2016.  5G for Europe: An Action Plan.  Ibid.  
31 EC.  2013.  ibid. p20 
32 EC.  2016.  Ibid p3, 7 and 8.  For example Action 5 focuses on “initial global 5G standards by the end 
of 2019 at the latest, to enable a timely commercial launch of 5G, and paving the way for a wide range 
of future connectivity scenarios” 

33 EC.  2013. ibid. p55 
34 EC.  2013. ibid. p52 
35 EC.  2013. Ibid.  p20 and 49 

37 

 
 
 
 
                                                
4.2.2.  Performance KPIs  
Of the 43 performance related KPIs on the long list many were too tightly focused on 
narrow areas requiring specific solutions, i.e. they are low-level component specific 
KPIs.  In some cases this fine grained level of specificity could result in progress 
within the PPPs not being picked up by these KPIs and consequently 5G-related 
progress within the PPPs could go unmonitored. To counter this, we have 
amalgamated some of the low-level focussed KPIs into medium level ones that target 
a single technology rather than very specific outcomes.  

Furthermore, where the long list for KPIs targeted objectives such as energy 
efficiency or security or automation in a number of different low-level component-
specific KPIs, we have included these objectives in KPIs that can monitor progress 
wherever it occurs within the 5G network.  

In addition gap analysis was undertaken to identify potential deficiencies in the scope 
of the proposed KPIs. Gap analysis identified 3 additional KPIs related to mobile 
edge computing and data centres; satellite and front haul/back haul solutions. The 
latter two KPIs in particular, captured by P9 and P10 in Table 5 can be better defined 
and refined when the Phase II projects begin.  

The suggested KPIs have been mapped against the Horizon 2020 5G Infrastructure 
model used to conceptualise the different Phase II 5G PPP project areas (the original 
drawing is shown in Figure 9 for clarity, the mapping is shown in Figure 10). 

Figure 9 Conceptualisation of Horizon 2020 5G Infrastructure components 

38 

 
 
 
 
 
 
 
Figure 10 Graphical representation of the Performance  KPIs 

An overview of the KPIs and the rationale for seeking information are provided 
below.  

Table 5 Rationale for the Performance KPIs 

KPI 

Rationale  

Providing 1000 times higher 

wireless area capacity and more 
varied service capabilities 
compared to 2010   P1* 

Reducing the average service 

creation time cycle from 90 hours 
to 90 minutes (as compared to 
the equivalent time cycle in 2010)  
P2* 

Maturation of Network OS and NFV 
concepts, increased number of 
SDN controllable resources  P3 

The 5G PPP contract indicates high-level performance targets 
for 5G. This KPI lends itself to monitoring progress towards 
headline objectives of achieving Enhanced Mobile Broadband 
services, including broadcast and pervasive video, and the 
ability to cater for new services and end-users, e.g. the 
verticals. 

A key challenge for 5G networks will be their ability to support 
more agile and dynamic creation of services for end-users 
and third parties. This existing 5G PPP contract high level KPI 
targets this objective. 

The maturation of network softwarisation concepts is a major 
goal for the PPPs. The 5G PPP contract identifies a number of 
specific low-level KPIs related to hardening these 
demonstrators, standards and real deployments, amalgamated 
here into overarching medium-level softwarisation maturation 
KPI. 

39 

 
 
 
 
Reducing energy consumption to 10 

per cent of 2010 levels   P4* 

Substantially increasing energy efficiency is a key objective for 
the PPPs, and the European Commission more generally36. 
This 5G PPP contract high level KPI particularly targets 
efficiencies in the RAN, low-level KPIs in the contract also 
target energy efficiencies in optical networks. 

Improvements in automated 
network control, reduced 
signalling and reduced network 
OPEX   P5 

This KPI targets the development of solutions that enable more 

complex, diverse 5G networks to operate with reduced 
human intervention and reduced OPEX. This medium-level 
KPI is a system objective applicable to every 5G network 
element, amalgamating many low-level KPIs in the 5G PPP 
contract.  

Integration of Networking, 
Computing and Storage 
resources in one unified 
programmable infrastructure  P6 

This KPI monitors the integration of, and support for, non-telco 
and third-party Cloud Computing, Data centre, Mobile Edge 
Computing entitles in 5G networks, from the edge to the 
Core. An ITU Focus Group report37 highlights issues around 
third-party authorisations for MEC in the RAN as particular 
challenges.  

Very dense deployments to connect 
over 7 trillion wireless devices 
serving over 7 billion people  P7* 

The emergence and growth of the Internet of Things creates a 

large opportunity for 5G networks. This high level KPI 
monitors the development of solutions supporting Massive 
Machine Type Communication services. 

End-to-end latency reduced by a 

factor of five   P8 

Integration of satellite-based 
solutions to improve reach, 
resilience and range of services 
of networks  P9 

Improved front haul/back haul and 
architectural support for diverse 
types of terminals, traffic, network 
operators and RANs  P10 

The reduction of end-to-end latency (where end-to-end may be 
defined to mean a path at the network edge, e.g. by use of 
MEC, D2D, V2X concepts, etc.) is a key objective for 5G 
enabling a new set of services such as Tactile Internet.  

Satellite will feature as a component of 5G networks. It is not 

present in the 2013 5G PPP contract long list. This initial KPI 
monitors any progress to integrate satellite solutions, e.g. new 
air interfaces, signalling etc3.  

This KPI monitors front haul/back haul support for the diversity 
of terminals (ranging from IoT gateways to MEC servers, etc), 
diversity of traffic types (implying the ability to increase and 
decrease capacity and QoS), diversity of network operator 
types and diversity of RATs in 5G networks. While not 
elaborated in the 5G PPP contract, the ITU FG report 
identifies this challenge as requiring attention to fulfil the 
objectives of 5G38.  

Increasing spectral availability and 
efficiency, developing efficient 
hardware to support mmWave 
communications  P11 

This KPI amalgamates low-level 5G PPP Contract KPIs to 

monitor any solutions that improve the availability of 
spectrum, i.e. release more spectrum for 5G use, increase 
the efficiency with which it is used and also to monitor 
mmWave technologies. 

Improved optical performance in 

Improvements in optical networking throughout the 5G network 

36 5G-PPP, European Commission, “5G Vision Brochure v1”, 2015. 
37 Focus Group IMT-2020: Report on Standards Gap Analysis, December 2015.  

40 

http://www.itu.int/en/ITU-T/focusgroups/imt-2020/Documents/T13-SG13-151130-TD-PLEN-
0208%21%21MSW-E.docx  p. 86. 
38 ibid FG IMT-2020 2015 ibid. p. 30. 

 
 
 
 
 
 
                                                
Access and Aggregation 
networks, Core networks and 
Data Centre networks  P12 

are identified through low-level KPIs focussing on increased 
reach and capacity, reduced energy consumption/bit, reduced 
equipment density/bit ratios among others. This medium-level 
KPI monitors progress in optical networking that supports any 
of 5G PPP contract low-level optical networking objectives. 

The asterisk in this table indicate KPIs that correspond with one of the 12 KPIs suggested by the 5G 
Infrastructure association, see Table 2. 

4.2.3.  Societal KPIs  
The ten societal KPIs are spread across two key dimensions - societal issues and 
competitiveness concerns. 

Figure 11 provides a representation of the KPIs in relation to societal (in the orange 
box) and competitive (blue box) objectives shown below. The five KPIs in the central 
green box, where societal and competitive objectives overlap, are important in 
providing both objectives (e.g. security, privacy, QoS, skills and healthcare39).  

Figure 11 A graphical representation of the ten Societal KPIs 

An overview of the KPIs and the rationale for seeking information are provided 
overpage.  For each KPI the right hand column provides an ‘evidence’ base for why 
the indicator should be included.  Some of these rationales relate to requirements 
articulated by the European Commission and Parliament.  Others are related to 
expectations regarding 5G capabilities found in the 5G Action Plan and other 
documents produced by leading 5G organisations.  

39 Energy is a sector where 5G is expected to have considerable benefits to the healthcare sector and 

the citizens who use the service. 

41 

 
 
 
 
 
 
                                                
Key Performance Indicator 

Rationale for inclusion 

Develop new economically-

viable services of high societal 
value   S1* 

Provide data and system 
security for citizens, 
businesses and devices   S2* 

The European Parliament emphasised the need for investment in 
RDI to tackle growing societal challenges effectively40. 5G-PPP 
contracts highlight the need for services of high-societal value41.  
The 5G for Europe Action Plan emphasises the need for 5G to 
‘deliver concrete benefits to society’42. 
Data protection and security laws apply in a big data context43.  

Security of data and 5G systems will be fundamental in adhering 
to forthcoming European General Data Protection Regulations44.  

Provide user controlled privacy   

5G-PPP contracts noted the need to develop advanced user 

S3 

controlled privacy. 

Reliable and dependable Quality 

of Service and Quality of 
Experience with "zero 
perceived" downtime   S4* 

Development of 5G skills 
curricula and education/ 
training providing skills that are 
competitive at world level   S5* 

Support remote monitoring of 
health and wellness in the 
health sector   S6* 

This KPI focuses on all uses and users of 5G networks. But some 
vertical sectors, e.g. utilities, automotive, healthcare, and other 
enterprise users, will demand higher assurances of service 
resiliency, QoS/E and network security than before45.  

The European Parliament reaffirmed the need for investment in 

RDI to be complemented by education, training and skills 
development that are competitive at world level46. 

The 5G for Europe Action Plan emphasises the need for 5G to 
support healthcare47. Healthcare is a sector where 5G is expected 
to have considerable impact. Telecare and telehealth 
developments, generally facilitated by 5G monitoring and 
information sharing, offer considerable benefits for those that are 
ill and well48. 

40 European Parliament.  2011.  Ibid. p7 
41 5G PPP contracts mentioned UHDTV as an objective with societal value.  The societal challenges 
identified by the European Commission are very diverse (including health, agriculture, energy, 
climate, inclusion and security.  See https://ec.europa.eu/programmes/horizon2020/en/h2020-
section/societal-challenges).  In the absence of precise objectives it is suggested that specific 
applications or opportunities for 5G are not defined.  Nonetheless, as outlined in a study undertaken 
by the study team, socio-economics of 5G can have considerable social benefits.  These include 
addressing the digital divide, reduction in road accidents, enhanced healthcare monitoring, reductions 
in emissions and the wide variety of benefits arising from 5G enabled IoT, see 
https://ec.europa.eu/digital-single-market/en/news/study-identification-and-quantification-key-socio-
economic-data-strategic-planning-5g 

42 EC..  2016.  5G for Europe: An Action Plan 

http://ec.europa.eu/newsroom/dae/document.cfm?doc_id=17131 

43 ICO Report on Big data and data protection; see https://ico.org.uk/media/for-

organisations/documents/1541/big-data-and-data-protection.pdf 

44 Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the 

protection of natural persons with regard to the processing of personal data and on the free 
movement of such data, and repealing Directive 95/46/EC; http://eur-lex.europa.eu/legal-
content/EN/TXT/?uri=uriserv:OJ.L_.2016.119.01.0001.01.ENG&toc=OJ:L:2016:119:TOC  

45 Vertical markets white papers produced by the 5G Infrastructure Association https://5g-ppp.eu/white-

papers/ 

46 European Parliament.  2011.  Ibid. p5 and 8 
47 EC..  2016.  5G for Europe: An Action Plan.  Staff Working Document 
http://ec.europa.eu/newsroom/dae/ document.cfm?doc_id=17132 

42 

48 Tech4i2, Trinity college et al.  2016 forthcoming.  Identification and Quantification of Key socio-

economic data for the strategic planning of 5G introduction in Europe.  https://ec.europa.eu/digital-
single-market/en/news/study-identification-and-quantification-key-socio-economic-data-strategic-

 
 
 
                                                
European 5G systems and 
technologies should be 
globally competitive   S7* 

PPP initiatives are launched to boost industrial competitiveness 
and job creation49.  5G-PPP contracts noted the need for a 
competitive industrial offer. 

Develop new economically-

viable services in key strategic 
value chains   S8* 

The European Commission review of PPP projects highlighted that 

support needs to advance industrial competitiveness in key 
strategic value chains50.  

Support machine type 

communication in the energy 
industry   S9* 

The 5G for Europe Action Plan emphasises the need for 5G to 
support the energy sector51. Energy is a sector where 5G has 
high business potential through 5G support for energy grid 
protection and control, and the massive volume of machine type 
communication for emerging smart metering developments52.  

Support the development of 

telematics and data 
communications in connected 
and automated vehicles   S10* 

The 5G for Europe Action Plan emphasises the need for 5G to 
support the automotive sector53. Automotive is the area where 5G 
is expected to have the greatest impact; through sharing real-time 
data about vehicle locations and through new business models 
arising form enhanced telemetry data54.  

The asterisk in this table indicate KPIs that correspond with one of the 12 KPIs suggested by the 5G 
Infrastructure association, see Table 2. 

4.3.  Consultation and validation 
This section describes the consultation and engagement process to obtain 
stakeholder views about KPIs.  Engagement activities will be concluded at the public 
presentation taking place in Brussels on Wednesday 19th October.   

The final part of the study was a consultation exercise to ensure the shortlisted KPIs 
are ‘fit for purpose’.  Short briefing papers (of two to four pages of A4) describing 
each of the three categories of KPIs were developed for distribution during the 
engagement process.  The briefing documents provided a background to the 
rationale for the selection of the KPIs.  The purpose for collecting information and the 
focus of the KPIs were also described.   

planning-5g. and 5G Infrastructure Association.  2015.  5G and eHealth. https://5g-ppp.eu/wp-
content/uploads/2014/02/5G-PPP-White-Paper-on-eHealth-Vertical-Sector.pdf 

49 EC.  2013.  Ibid.  pages 55 and 71 
50 EC.  2013.  Ibid.  p55.  Four key sectors have been identified as being impacted by 5G in previous 

White Papers (Automotive, Energy, eHealth, Media and entertainment).  

51 EC..  2016.  5G for Europe: An Action Plan.  Staff Working Document 
http://ec.europa.eu/newsroom/dae/ document.cfm?doc_id=17132 

52 5G Infrastructure Association.  2015.  5G and energy.  https://5g-ppp.eu/wp-

content/uploads/2014/02/5G-PPP-White_Paper-on-Energy-Vertical-Sector.pdf 

53 EC..  2016.  5G for Europe: An Action Plan.  Staff Working Document 
http://ec.europa.eu/newsroom/dae/ document.cfm?doc_id=17132 

54 KPI S6 is a catchall objective to measure 5G impact in key value chains.  One value chain where 

considerable impact has consistently been forecast is automotive.  See Tech4i2, Trinity college et al.  
2016 forthcoming.  Identification and Quantification of Key socio-economic data for the strategic 
planning of 5G introduction in Europe.  https://ec.europa.eu/digital-single-market/en/news/study-
identification-and-quantification-key-socio-economic-data-strategic-planning-5g 

43 

 
 
 
 
                                                                                                                                      
Figure 12 The first page of the performance KPI briefing paper 

Consultation and engagement focused on three stakeholder groups. Wider online 
engagement with any interested individual or organisation was also undertaken. 

The three key stakeholder groups included: 

The 19 5G PPP projects:  The objective of contacting projects was to 
seek the views of these organisations about the relevance of the KPIs to 
their activities; 

Key stakeholders in the supply and value chains:  Our team sent 
briefing documents to representatives in relevant verticals.  Key 
stakeholders were asked about the importance of the different KPIs to 
their activities.  Respondents were also asked to provide details of other 
KPIs that might be relevant to their activities; 

44 

 
 
 
 
Policymakers:  Our team sent briefing documents to policymakers and 
representatives from Regulatory Authorities.  Policymakers were asked 
about the importance of KPIs to their Members.  Respondents were also 
asked to provide details of other KPIs that might be relevant to their 
activities. 

The study utilise an online LinkedIn 5G expert group for online consultation, see 
Figure 13.  The group has more than 250 expert 5G members.   This more extensive 
online approach maximised the opportunity for input from a large group with expert 
knowledge of 5G. 

At the start of the project details of the study and the three briefing papers were 
placed on the 5G expert group developed by Tech4i2 for the 5G socio-economic 
impact study55.  Towards the end of the project the final selection of KPIs and 
explanations for the inclusion of each variable were also shared online.   

Figure 13 Screenshot of the request for project input on the 5G Expert Forum 

4.4.  Operationalising the KPIs 

4.4.1.  Introduction 

The preceding KPI descriptions provided clear details of what should be examined, 
but they did not describe detailed metrics for KPIs.  This level of detail is important to 
ensure that all those collecting the information are perfectly clear about the required 
information.   

The objective of selecting operational KPIs is to select those measures that have 
certain inherent qualities that deliver the most value as a tool for analysis, program 

55 SMART 2014/0008 

45 

 
 
 
 
 
                                                
monitoring and evaluation, performance improvement and the communication of 
results. 

The acronym “SMART” is often used to refer to the characteristics of good 
performance indicators. Each letter of the acronym represents an important 
characteristic56. These include:- 

  Specific: Does the indicator convey at a glance what it is measuring, and 
how the measurement is derived? KPIs should communicate clearly to the 
stakeholders what a project is doing. 

  Measurable: Can the measurement be expressed as an objective value (e.g., 
the number of persons assisted, percentage of population connected)? Do 
reliable data exist? Can they be easily collected? Better yet, are they already 
being collected? 

  Achievable: Does the indicator measure something within the program or 

activity’s manageable control? 

  Relevant: Does the indicator measure the most important result of the 

activity? 

  Time-bound: Is there a deadline for achieving the performance indicator? 
Are data reported at sufficiently regular intervals to support tracking and 
management decision-making? 

Different studies offer different definitions or characteristics for the “SMART” 
acronym57, because in reality, there are more than five issues to consider when 
evaluating the merits of one performance measure over another.  Nonetheless, the 
SMART acronym provides clear guidance when developing KPIs. 

The Tables below provide operational measures for the KPIs presented in section 3.  
The tables highlight that relatively simple nominal, ordinal or ratio data can be 
collected from projects.  By ensuring all 5G PPP projects collect exactly the same 
information for monitoring it will be possible to develop comparisons between 
projects for most of the KPIs.   

As well as providing clear parameters for the KPIs the following sections consider 
three key elements:- 

Baseline situation:  A baseline assessment provides a critical reference 
point for assessing changes and impact.  It establishes a basis for 
comparing the situation before and after an intervention.  Baseline data 
enables inferences to be made about the effectiveness of an intervention. 

Baseline analysis is particularly relevant to performance KPIs in this study.  
Several of the KPIs (enhanced from 5G PPP contracts) make reference to a 
baseline situation in 2010.  Baselines are less important for Operating KPIs 
because projects are ‘standalone’ initiatives that have no precedents.  In a 
similar manner Societal KPIs will rarely have baselines since the impact of 
5G will create new opportunities and new benefits. 

56 USAid. 2013. ibid 
57 For example CLEAR - Collaborative (Goals should encourage employees to work together 

collaboratively and in teams); Limited (Goals should be limited in both scope and duration); Emotional 
(Goals should make an emotional connection to employees, tapping into their energy and passion); 
Appreciable (Large goals should be broken down into smaller goals so they can be accomplished 
more quickly and easily for long-term gain); Refinable (Set goals with a headstrong and steadfast 
objective, but as new situations or information arise, give yourself permission to refine and modify 
your goals). http://www.inc.com/peter-economy/forget-smart-goals-try-clear-goals-instead.html 

46 

 
 
                                                
Targets:  One of the elements of the SMART acronym is ‘achievable’.  This 
element largely relates to targets embodied in KPIs.  Target indicators 
should be linked, as directly as possible, to interventions, minimising the 
effect of external factors. 

As the next section highlights only seven of the 29 suggested KPIs, 
developed from 5G contracts and gap analysis, had explicit targets.  This 
issue is addressed in this section by discussing methods of setting targets.  
The next section provides an explanation for the need for targets and 
potential methods for generating evidence based achievable targets.   

Reporting:  Reporting involves collecting (and validating) evidence of 
progress in relation to a KPI that a project claims has been achieved.  This 
information can be collected directly from a 5G PPP project – according to 
the parameters defined in the remainder of this section.  This is a form of 
primary data collection.   

Secondary data sources such as reviews of published peer-reviewed 
papers, technical reports and patents can also be reviewed to investigate 
and validate the progress of projects against KPIs.  Information from 
secondary sources are particularly relevant to some of the  Operating and 
performance KPIs.  Forecasts made in journal articles and other 
publications will also be beneficial in monitoring and predicting societal 
KPIs.   

In the remainder of this section reporting obligations require evidence 
documents to be held locally for audit and validation purposes.  Resources 
available for monitoring and evaluation will determine the level of auditing.   

4.4.2.  Operational metrics 
This section provides parameters for measuring Operating KPIs.  Baseline analysis 
is not relevant to operating KPIs because 5G PPP projects are ‘standalone’ initiatives 
that have no precedents.   

Only two of the KPIs have targets B1 (Five to ten times leverage) and B3 (20 per 
cent SME participation).  However, the next section provides suggestions about how 
targets could be established by considering the performance of similar FP7 and 
Horizon 2020 projects. 

Most of the information required for reporting will be collected locally by projects.  
For nearly all the KPIs these ‘evidence documents’ should be held locally for audit 
purposes.   

Indicator 

Leverage arising 
from 5G PPP 
funding O1 

Table 6  Metrics for operating KPIs 

Operational metric 

The resources provided by project partners alongside 5G PPP funding to 

support the development of a project.  These resources should be calculated 
annually over the funding period, preferably in financial terms.  For example 
the € value of investments in equipment or other resources and the monetary 
value (including on-costs) of human resources providing input to the project. 
Details for both categories should be provided separately.   

Target:  5G contracts suggest leverage should be between five and ten times 

the value of funding. 

Industry participation  This KPI should record annually the number of partners receiving financial 

47 

 
 
in 5G PPP projects 
O2 

support in a project, the size and industry (NACE Rev. 2 classification) should 
be recorded.    

SME participation in 
5G PPP projects 
O3 

Contribution to 5G 

Standards 
negotiations O4 

Contribution to 
Service Level 
Agreement 
negotiations O5 

The number of partners receiving financial support in a project with fewer than 
250 persons employed should be recorded annually.  Ideally SMEs should be 
categorised by industry (NACE Rev. 2 classification) and the size band (micro 
enterprises: fewer than 10 persons employed; small enterprises: 10 to 49 
persons employed; medium-sized enterprises: 50 to 249 persons employed). 

Target:  5G contracts suggest partnerships should consist of more than 20 per 

cent SMEs. 

This KPI should record annually the input of projects (or project partners on 
behalf of partnerships) to standardisation activities.  The nature of inputs 
should be recorded as the number of:-  a) minor communications 
(conversations or emails), b) substantial communications (formal submissions 
or recommendations, relevant documents should be held for audit purposes), 
c) Major contributions (attendance at official standards meetings, details 
should be held for audit purposes).  

This KPI should record annually the input of projects (or project partners on 

behalf of partnerships) to Service Level Agreement negotiations.  The nature 
of inputs should be recorded as the number of:-  a) minor communications 
(conversations or emails), b) substantial communications (formal submissions 
or recommendations, relevant documents should be held for audit purposes), 
c) Major contributions (attendance at official Service Level Agreement 
meetings, details should be held for audit purposes). 

Research papers, 

patents and 
demonstrators 
arising from 5G 
PPP projects O6 

This KPI should record annually the research outputs, publications and patents 
from 5G PPP projects.  All research papers should be submitted as soon as 
possible (and at the latest on publication) to OpenAire58.  All patent 
applications should ideally be submitted to Member State Patent Offices or 
the European Patent Office59.  Details of demonstrator or prototypes should 
be provided; further details should be held for audit purposes.  All relevant 
data should be submitted to the Open Research Data Pilot60 

Support for open 

innovation 
ecosystems O7 

61The European research and innovation e-infrastructure landscape is very 
fragmented62.  Consequently this KPI should record annually contributions to 
open innovation ecosystems in the form of contributions to conferences and 
symposia.  Meetings details should be held for audit purposes. 

58 Details of information to deposit can be found at https://www.openaire.eu/open-access-in-horizon-
2020.  Submission to OpenAire will provide sufficient information for contributions to be audited on 
bibliographic databases, such as Scopus. 

59 Registration details for patent should be provided so that they can be audited using PATSTAT. 

European Patent Office (EPO) have developed PATSTAT is comprised of three individual software 
products. These are EPO Worldwide Patent Statistical Database, EPO Worldwide Legal Status 
Database for PATSTAT and European Patent Register for PATSTAT.  

60 https://www.openaire.eu/opendatapilot 
61 Submission of relevant outputs to OpenAire and the Open Research Data Pilot represent two forms of 

open innovation.  But as an EC communication highlights more needs to be done 
http://ec.europa.eu/research/science-society/document_library/pdf_06/era-communication-towards-
better-access-to-scientific-information_en.pdf 

62 http://ec.europa.eu/research/participants/portal/desktop/en/opportunities/h2020/topics/einfra-12-

2017.html 

48 

 
 
 
                                                
4.4.3.  Performance KPIs 
Some of the KPIs have clear objectives set down in the source material from which 
they have been developed. Generally these KPIs relate to objectives that examine 
the improvement of concepts, processes, technologies and practices that are 
recognisable and well understood within the existing cellular world. Other KPIs are 
less well defined. Generally these KPIs relate to areas in which 5G is attempting to 
support a much more diverse ecosystem. In these cases it is likely that there will 
need to be a process of defining and refining a set of agreed descriptions of the KPIs 
with the interested stakeholders as the research matures. 

In order to measure improvement it will be necessary to establish well-understood, 
well-defined baselines. These baselines will serve as benchmarks on which to 
measure progress by the projects in a clear and equitable manner. The 5G PPP 
Contract uses 2010 as a baseline, e.g. the headline KPI of x1000 capacity is with 
respect to 2010 capacity. This is not a satisfactory definition of the capacity baseline. 
This baseline, along with others, will need to be set in terms of actual technologies – 
most likely an LTE release. If LTE release 12 or 13 is used then the target of x1000 
would need to be reset taking account of improvements already made between the 
‘2010’ capacity and the capacity achievable under LTE release 12 or 13. 

Where the KPI measures something that is not purely technical or that cannot simply 
be defined with reference to a standard then alternative authoritative and agreed 
sources should be used. For example, in the case where the aim is to reduce the 
OPEX of certain operator processes it will be necessary to establish a clear definition 
and understanding of those processes by engaging the relevant sources. 

Targets have been suggested in the 5G PPP Contact for several of the KPIs. These 
targets vary from absolute targets, e.g. the Reach targets for optical networks are 
defined in terms of certain distances, to relative targets, i.e. percentage 
improvements on an assumed baseline. Whether these targets are realistic, too 
ambitious or not ambitious enough is a matter for discussion. In nearly all cases 
there is no rationale given for the targets. An example of this is the 7 trillion device 
target for KPI P7 which targets ‘very dense deployments’. While the 3GPP have 
given some reasoning for the density targets they have used as the design criteria for 
the recently standardised NB-IoT, it is far from clear where the 7 trillion figure has 
been arrived from.  

Additionally, it may be difficult initially to agree KPI targets for objectives for which 
there is no baseline data or for which there is a lack of clarity in terms of what the 
dimensions of the ultimate deployment scenario will look like, e.g. the definition of the 
latency KPI P8 will be reliant on agreed end-to-end paths in emerging use cases 
such as the Automotive vertical. 

In terms of aiding the operationalization of these KPIs, it may be useful to use 
sources such as the NGNM63, non-European sources such as NTT DoCoMo64 and 
the newly formed 5G Automotive Association65, among others.  

63 NGNM, “Recommendations for NGMN KPIs and Requirements for 5G”, June 2016, 

https://www.ngmn.org/uploads/media/160603_Annex_-
_NGMN_Liaison_to_3GPP_RAN__72_v1_0.pdf, 

64 NTT DOCOMO, “NTT DOCOMO 5g Activities – Toward 2020 Launch for 5G Services”, NTT 

Technical Journal Volume 17 No 4, 2016. 
https://www.nttdocomo.co.jp/english/binary/pdf/corporate/technology/rd/technical_journal/bn/vol17_4/v
ol17_4_002en.pdf 

65 https://www.ericsson.com/news/160927-telecommunications-and-automotive-players_244039854_c 

49 

 
 
                                                
For most of the KPIs reporting will generally take the form of published papers, 
project deliverables and other technical reports.  Variants from this form of reporting 
are provided in Table 7. 

Table 7 Operational metrics for Performance KPIs 

Key Performance 
Indicator 

Providing 1000 times 
higher wireless area 
capacity and more 
varied service 
capabilities compared 
to 2010   P1 

Reducing the average 
service creation time 
cycle from 90 hours to 
90 minutes (as 
compared to the 
equivalent time cycle in 
2010)  P2 

Maturation of Network 

OS and NFV concepts, 
increased number of 
SDN controllable 
resources  P3 

Operational metric 

This KPI monitors a well-understood and relatively well-defined objective.  
Baseline: while the contract indicates that the x1000 increase is with regard 
to 2010 this benchmark lacks specificity. It may be preferable to choose a 
baseline based on existing 3GPP standards, e.g. LTE release 12 or 
release 13. Alternatively, a representative organisation such as the GSMA 
could indicate what the baseline capacity levels for cells are understood to 
be in 2016.  

Target: The target is a relative one. However, as noted it is not useful 

unless a baseline or baselines are established. Different baselines may be 
required to address capacity availability in different deployment scenarios. 

This KPI, while having a clear objective, is lacking in definition. The 

clarification of this KPI goes hand in hand with the establishment of a 
baseline. 

Baseline: the purported 90 hours service time should be clarified – each of 
the component steps or processes involved in creating a service should be 
elaborated. Where possible, the time involved in each process of service 
creation should be noted. This is likely outside the scope of a standards 
body but an operators’ and/or vendors’ representative organisation could 
agree on a definition or description of current best practice.  

Target: the target is clear for this KPI – 90 minutes. However, as noted 

above if new types of services are required for sectors that would not be 
served by the current approach then it may be necessary to define 
additional targets. 

This KPI is based on the 5G PPP Contract that provides sufficient detail to 

readily operationalize it. 

Baseline: This KPI is more qualitative than quantitative in form. The 

Contract calls for the maturation of Network OS and NFV concepts to be 
measured in terms of real deployments, reference implementations, 
standards, demonstrators, open ecosystems etc. As such, a simple 
baseline noting the existing number of such deployments, etc., should 
suffice to measure progress over the course of the 5G PPP projects. With 
respect to the ‘increased number of SDN controllable resources’, a 
baseline understanding of the existing number of resources, where they 
are well defined, should be agreed.  

Target: There are no defined targets for this KPI. After the baseline has 

been established a target can be set based on an assessment of how thin 
or shallow the current availability of deployments, etc., is.  

Reducing energy 

consumption to 10 per 
cent of 2010 levels   P4 

This KPI is relatively straightforward. There is a call for general reduction of 
energy consumption of 90 per cent based on 2010 levels. This applies to 
all parts of the 5G systems apart from the radio access elements where an 
80 per cent reduction is called for.  

Baseline: As with other KPIs the baseline needs to be clearly established. 
That should take the form of agreed baseline technologies and agreed 
energy consumption figures on a per component basis, i.e. radio, optical, 
etc.  

Target: 10 per cent of 2010 baseline.  

50 

 
 
 
Improvements in 

automated network 
control, reduced 
signalling and reduced 
network OPEX   P5 

Integration of 

Networking, Computing 
and Storage resources 
in one unified 
programmable 
infrastructure  P6 

Very dense 

deployments to 
connect over 7 trillion 
wireless devices 
serving over 7 billion 
people  P7 

End-to-end latency 

reduced by a factor of 
five   P8 

The KPI lacks some specificity that will require it to be refined before being 

made operational.  

Baseline: The 5G PPP contract indicates that ‘reduced network OPEX’ may 
be measured with regard to current human operator tasks.  The process of 
baselining this KPI may involve the contribution operators’ and/or vendors’ 
representative organisations to agree a number of representative samples 
of human operated tasks and the effort they require. For ‘reduced 
signalling’ a baseline should be established with regard to existing 
QoS/QoE for ease of measurement and reporting. This may be done by 
reference to existing standards.  

Target: The target is a 30 per cent reduction on the baseline. Where a new 
class of service or experience is defined during the development of 5G, 
e.g. to serve a new market or vertical need, then new targets will need to 
be defined and agreed.  

This KPI is far from being operational. This KPI should monitor the 

integration of, and support for, non-telco and third-party Cloud Computing, 
Data centre, Mobile Edge Computing entitles in 5G networks, from the 
edge to the Core. An ITU Focus Group report66 highlights issues around 
third-party authorisations for MEC in the RAN as particular challenges.   

To operationalise this KPI it will be necessary to expand on the existing 

definition and understanding of a ‘unified programmable infrastructure’ as 
laid out by the ITU FG report.   

This KPI will need to be refined before it can be made operational. The KPI 
is aimed at the Internet of Things or Internet of Everything area. As 5G 
seeks to service both Massive IoT and Critical IoT segments this KPI may 
need to be refined to target different classes of IoT devices. Furthermore, 
the reference to 7 billion people in the KPI may not be necessary; there is 
not necessarily a direct mapping between populations and the consequent 
number of devices. As mentioned in the introduction above the headline 
target for this KPI lacks scenario-based motivation or support. 

Baseline: Given how relatively immature this area is within the sphere of 
cellular networks it may be premature to create a baseline. There are a 
number of existing standards, which serve different segments of the IoT 
space, e.g. LTE-M, NB-IOT. A selection of these might serve as a baseline 
for comparison.  

Target: The headline target is 7 trillion devices globally. This target is not 

meaningful and should be refined. 

The crux of operationalising this KPI is in the definition of ‘end-to-end’.  
Baseline: Commonly understood end-to-end paths within a baseline 

version of the existing standards should be specified. The existing detailed 
latency figures for these well-defined paths should be agreed. However, 
the baseline may need to be refined as 5G develops. The ITU Focus 
Group report on Standards Gap Analysis called for the identification, 
clarification and definition of new end-to-end paths that may arise under 
new 5G architectures. For example, new end-to-end path definitions may 
be created to serve the needs of the Tactile Internet.  

Target: The 5G PPP contract identifies 2 clear targets; <1ms and a general 

x5 reduction.  

Reporting: When the issues raised above are clarified then it should be 
possible to report on this KPI using published peer-reviewed papers, 
deliverables and technical reports to support claims. 

66 Focus Group IMT-2020: Report on Standards Gap Analysis, December 2015.  

http://www.itu.int/en/ITU-T/focusgroups/imt-2020/Documents/T13-SG13-151130-TD-PLEN-
0208%21%21MSW-E.docx  p. 86. 

51 

 
 
                                                
 
Integration of satellite-
based solutions to 
improve reach, 
resilience and range of 
services of networks  
P9 

Improved front 

haul/back haul and 
architectural support 
for diverse types of 
terminals, traffic, 
network operators and 
RANs  P10 

Increasing spectral 
availability and 
efficiency, developing 
efficient hardware to 
support mmWave 
communications  P11 

As with P6 this KPI is far from being operational at present. Initially this KPI 

monitors any progress to integrate satellite solutions, e.g. new air 
interfaces, signalling etc.6768.  This KPI is not present in the 5G PPP 
Contract and is more related to Phase II 5G PPP support. 

Baseline: The baseline for this KPI could be informed or defined by existing 

techniques by which satellite and LTE address these aims.   

Targets: Neither the 5G PPP Contract nor the Networld 2020 document set 
out specific targets for satellite. Networld 2020 sees satellite as essentially 
enabling the other headline KPIs to be realised in more places with better 
reliability. As such it will be necessary to develop explicit targets that 
define reach and resiliency. Additionally, and likely with reference to P1, it 
will be necessary to agree on targets around the improvement in the range 
of services enabled by satellite. 

While not elaborated in the 5G PPP contract, the ITU FG report identifies 
this KPI as requiring attention to fulfil the objectives of 5G69.  This KPI is 
more related to Phase II 5G PPP support.  A challenge for operationalising 
this KPI will be the definition or characterisation of newly developed 
diversity within the 5G ecosystem. For example, the embrace of IoT, the 
embrace of different vertical sectors and the support of new types of 
applications such as augmented and virtual reality will influence this 
diversity. A further significant challenge will be around defining the types 
for improvements required in supporting front haul/ back haul networks 
and developing targets for these. 

Baseline: There may not be an existing baseline for this KPI as it looks to 

address performance improvements for traffic types, terminals and 
network operators that do not currently exist or are only in development. 
Consequently it may be difficult to benchmark existing backhaul/fronthaul 
solutions. Initial work should focus on agreeing on the characteristics of 
the diversity to be supported and on agreeing on the types of 
improvements which are required within the hauling networks.  

Target: There are no current hard targets.  
Reporting: Initial reporting could focus on the hardening of definitions for 
new traffic models, operator models etc., and the definition of agreed 
targets. Then the focus can shift to assessing how backhaul/fronthaul 
networks support these diverse demands. 

This KPI is broad, focussing on the radio access network. The targets for 

this KPI are not well defined and require work before it can become 
operational. 

Baseline: In terms of evaluating spectral availability it may be easier to 

assess this in terms broad bands of spectrum, e.g.  <1Ghz, 1Ghz to 6GHz 
and > 6Ghz bands. It should be relatively straightforward to assess the 
current (or 2010) availability of spectrum in these bands. Improvements in 
availability can be measured against this baseline. A baseline for hardware 
supporting mmWave communication should be established. 

Target: The targets specified in the 5G PPP contract specify a doubling of 
spectrum availability against some baseline, at a minimum. This target 
may be refined when an assessment of the baseline is conducted. The 
contract simply calls for improvements in the RF front-end using CMOS 
technology.  

Improved optical 

performance in Access 

The KPI objectives are very clearly set out in the 5G PPP contract. For each 
of the three optical networks listed, the contract identifies 5 areas requiring 

67 Networld 2020,  “The role of satellites in 5G (v5)”,  2014. 
68 ibid 5G-PPP,EC 2015. 
69 ibid FG IMT-2020 2015 ibid. p. 30. 

52 

 
 
 
                                                
and Aggregation 
networks, Core 
networks and Data 
Centre networks  P12 

improvements: Reach, Capacity, Equipment density/Bit, Energy 
Reduction/Bit, Automation and Control. 

Baseline: Aside from Reach a baseline will be required for each of the 

other 4 areas. Clarity on the definitions of these 4 should also be agreed or 
stated at the outset. 

Target: The reach targets for each optical network are defined in terms of 

kilometres. For each of Capacity, Equipment density/Bit, Energy 
Reduction/Bit, Automation and Control, the targets are specified relative to 
the baseline. 

4.4.4.  Societal KPIs 
This section provides parameters for measuring Societal KPIs.  Baseline analysis is 
generally not relevant to Societal KPIs because 5G will introduce new technologies 
and capabilities that will lead to step change in progress and/or new opportunities 
and benefits will result form 5G utilisation.   

Only one of the KPIs has a target S4 (zero perceived downtime).  Presenting precise 
targets for societal KPIs could be foolhardy and stifle innovation and flexibility to 
utilise 5G capabilities.  Policy makers have presented broad strategic goals – for 
example the European Parliament emphasised the need for investment in RDI to 
tackle growing societal challenges effectively70. 5G-PPP contracts highlight the need 
for services of high-societal value71.  But policymakers have not defined precise 
directions or goals for research.   

5G PPP contracts mentioned UHDTV as an objective with societal value.  But in 
relation to the broad European Commission definition of societal challenges 
(including health, agriculture, energy, climate, inclusion and security) a focus on 
UHDTV seems somewhat whimsical. If adopted as a goal it would demonstrate how 
a clear target (UHDTV) could deflect 5G PPP research away from other societal 
challenges 

In the absence of precise objectives it is suggested that specific applications or 
opportunities for 5G are not defined.  Nonetheless, to address criticisms that some 
previous research has not been applied significantly to improving society or 
enhancing particular key sectors identified by the Commission the KPIs do highlight 
the need for 5G PPP projects to focus on 5G utilisation72 to:- 

  Address societal challenges; 

  Enhance health and wellness in the Health Sector; 

  Support machine type communication in the Energy industry; 

  Develop telematics and data communications in connected and 

automated vehicles. 

As noted in section 2.4 most of the societal impacts of 5G are unlikely to arise before 
commercial deployment of 5G after 2020.  As a result, reporting will largely consist 

70 European Parliament.  2011.  Ibid. p7 
71 See https://ec.europa.eu/programmes/horizon2020/en/h2020-section/societal-challenges).   
72 The first paragraph in 5G Action Plan highlights these goals.  EC.  2016.  5G for Europe: An Action 

Plan.  http://ec.europa.eu/newsroom/dae/document.cfm? doc_id= 17131.  They are also highlighted in 
the 5G Manifesto for timely deployment of 5G in Europe produced by senior representatives of 
leading EU businesses.  http://telecoms.com/wp-content/blogs.dir/1/files/2016/07/ 
5GManifestofortimelydeploymentof5GinEurope.pdf 

53 

 
 
 
                                                
of monitoring forecasts and potential impacts of 5G and attempting to estimate the 
input to these impacts that 5G PPP projects might have.   

By better communicating details of these forecasts to projects it should be possible 
for those working in projects to better understand how their technological 
development might be utilised in the key societal, health, energy and automotive 
sectors.  Thus while not reporting progress, the reporting process could be beneficial 
in influencing 5G projects. 

Nonetheless for some of the KPIs, such as security (S2); privacy (S3); zero 
downtime (S4) and skills (S5), it should be possible to record progress amongst the 
projects in developing systems and processes that accommodate these 
requirements. 

Table 8 Operational metrics for Societal KPIs 

Key Performance Indicator 

Operational metric 

Develop new economically-

viable services of high societal 
value   S1* 

Advances that support societal challenges (including health, 

agriculture, energy, climate, inclusion and security) should be 
reported.   

Provide data and system 
security for citizens, 
businesses and devices   S2* 

Advances that develop the security of data and protect 5G systems 

should be reported.  One focus for developments should be 
support for European General Data Protection Regulations73.  

Provide user controlled privacy   

Advances that develop user-controlled privacy should be reported. 

S3 

Reliable and dependable Quality 

of Service and Quality of 
Experience with "zero 
perceived" downtime   S4 

Development of 5G skills 
curricula and education/ 
training providing skills that are 
competitive at world level   S5* 

Developments that contribute to ensuring ‘zero downtime’ and 
enhancing quality of services should be reported.  These can 
probably categorised by the reduction or elimination of 
equipment failure, high bit error rates, power failures, overloads 
due to exceeding the channel capacity, cascading failures, 
software bugs and human error. 

Developments in skills development and training should focus on 

technical elements related to 5G and wider competences74.  The 
competences, curricula and materials to be developed will 
include 5G technical competences, 5G sectoral competences, 
industry wide technical competences, workplace competences, 
academic competences and personal effectiveness 
competences. 

Support remote monitoring of 
health and wellness in the 

Advances that support health and well-being should be reported.  

Telecare and telehealth developments are impacts highlighted by 
policymakers and studies 75.  

73 Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the 

protection of natural persons with regard to the processing of personal data and on the free 
movement of such data, and repealing Directive 95/46/EC; http://eur-lex.europa.eu/legal-
content/EN/TXT/?uri=uriserv:OJ.L_.2016.119.01.0001.01.ENG&toc=OJ:L:2016:119:TOC  
74 OECD.  2015.  The G20 skills strategy for developing and using skills for the 21st Century.  
https://www.oecd.org/g20/topics/employment-and-social-policy/The-G20-Skills-Strategy-for-
Developing-and-Using-Skills-for-the-21st-Century.pdf 

54 

75 Tech4i2, Trinity college et al.  2016 forthcoming.  Identification and Quantification of Key socio-

economic data for the strategic planning of 5G introduction in Europe.  https://ec.europa.eu/digital-
single-market/en/news/study-identification-and-quantification-key-socio-economic-data-strategic-

 
 
 
                                                
health sector   S6 

European 5G systems and 
technologies should be 
globally competitive   S7* 

This KPI focuses on the global market share for 5G equipment and 
services delivered by European headquartered ICT companies. 
Baseline: The baseline for this KPI is 2011.  This is reported as 
being about 40 per cent by observers7677. 
Target: After clearly defining the baseline value target, at a 
minimum, should maintain the market share of European 
headquartered ICT companies. 

Develop new economically-

viable services in key strategic 
value chains   S8 

Three key sectors have been identified for this group of KPIs 

(health, energy and automated vehicles). Advances in other key 
strategic value chains should also be reported.    

Support machine type 

communication in the energy 
industry   S9 

Advances that support the energy sector should be reported.  

Energy grid protection and control, and support for 
communication for emerging smart metering developments are 
impacts highlighted by policymakers and studies 78  

Support the development of 

Advances that support the development of telematics and data 

telematics and data 
communications in connected 
and automated vehicles   S10 

communications in connected and automated vehicles should be 
reported.  Development should be categorised by the NHTSA five 
levels for autonomous vehicles79 

Annex 1 provides an overview of short briefing documents that were distributed 
through an online discussion group to more than 250 KPI experts.  The briefing 
documents provided details of the KPI, the rationale for including the KPI and the 
operational metrics for the KPI, including targets. 

4.5.  Monitoring or progress evaluation against targets 

An interesting contradiction exists between the traditional purposes of using KPIs in 
monitoring and evaluation and the manner in which they have frequently been used 
in monitoring European Commission projects.   

Section 2.2 highlighted that KPIs:- 

  Provide projects with clear criteria against which they will be evaluated; 

  Measure and report improvements over time. 

planning-5g. and 5G Infrastructure Association.  2015.  5G and eHealth. https://5g-ppp.eu/wp-
content/uploads/2014/02/5G-PPP-White-Paper-on-eHealth-Vertical-Sector.pdf 

76 Kleiner T.  2015. 5G research in Horizon 2020.  

ec.europa.eu/information_society/newsroom/cf/dae/document.cfm?doc_id=5204 

77 5G Infrastructure Public Private Partnership.  2015.  Advanced 5G Network Infrastructure for the 

Future Internet Public Private Partnership in Horizon 2020: Creating a Smart Ubiquitous Network for 
the Future Internet. P10 https://5g-ppp.eu/wp-content/uploads/2014/02/Advanced-5G-Network-
Infrastructure-PPP-in-H2020_Final_November-2013.pdf 

78 5G Infrastructure Association.  2015.  5G and energy.  https://5g-ppp.eu/wp-

content/uploads/2014/02/5G-PPP-White_Paper-on-Energy-Vertical-Sector.pdf 

79 NHTSA.  2013.  U.S. Department of Transportation Releases Policy on Automated Vehicle 
Development.   http://www.nhtsa.gov/About-NHTSA/Press-Releases/U.S.-Department-of-
Transportation-Releases-Policy-on-Automated-Vehicle-Development 

55 

 
 
 
 
 
                                                                                                                                      
These activities focus on providing clarity in monitoring.  The section also went on to 
highlight that KPIs:- 

  Help to better understand objectives; 

  Set performance standards and targets to motivate continuous 

improvement; 

  Set targets for implementing growth strategies. 

These criteria suggest that clear performance targets should be established. 

Interestingly only seven of the 29 suggested KPIs, developed from 5G contracts and 
gap analysis, had explicit targets.  These were:- 

  Five to ten times leverage  B1 

  20 per cent SME participation  B3 

  1,000 times higher wireless area capacity  P1 

  Reducing the average service creation time cycle from 90 hours to 90 

minutes  P2 

  Reducing energy consumption to 10 per cent of 2010 levels  P4 

  Connect over 7 trillion wireless devices serving over 7 billion people  P7 

  Zero downtime  S4 

Three of the KPIs (P1, P2 and P4) have a requirement to improve performance 
from 2010 levels.  Our team have found it difficult to find suitable information to 
define what the benchmark reference levels were in 2010.  Thus, even the 
KPIs with targets related to 2010 remain rather obscure. 

The remaining 22 KPIs do not have targets.  Clearly setting targets could 
restrict the flexibility of some projects, it is also possible that incorrect or 
inappropriate targets might initially be established.  Those developing KPIs 
must have had a rationale for inclusion and some idea of potential goals but 
clear targets have not been recorded.  Evaluation experts have highlighted that 
‘all other things being equal, the broader the target, the greater the flexibility’80.  
Targets can promote the a focus on a single goal (similar to the worst excesses 
of a command economy) and stifle innovation.   

Nonetheless to ensure clear goals, that can still be described and measured in 
broad terms, the study team recommend that wherever possible, targets should 
be established so that projects have clear goals and objectives.  If targets are 
not established it is possible to monitor performance and improvements, but 
the magnitude of these improvements becomes somewhat nebulous. Some 
projects could perceive that there is little incentive to perform at anything above 
minimum levels perceived as satisfactory.  

Obviously setting targets is difficult.  But we assert that this issue can easily be 
addressed for many of the KPIs associated with 5G projects and others 
supported by Horizon 2020. 

For example, no clear targets are set for dissemination or support for open 
innovation ecosystems.  Numerous previous initiatives similar to the 5G PPP 
have been undertaken in the past.  Project monitoring reports from these 

56 

80 Kamieniecki S. and Kraft M.  2013.  The Oxford Handbook of U.S. Environmental Policy.  

https://books.google.co.uk/books?isbn=019974467X 

 
 
                                                
projects could be analysed to derive the average number of research papers, 
patents, demonstrators, contributions to conferences, workshops and open 
innovation ecosystems.  Indeed, OpenAire81 has already collected sufficient 
information for our team to investigate the performance of leading projects and 
the average for all projects in producing publications and patents. However it 
must be noted this is an historic list. 

Figure 14 and Figure 15 show the top 20 FP7 projects (provided as acronyms) for 
the number of publications and patents.  

HiPEAC is noteworthy for the amount of publications it produced – 1,373.  When we 
examined the OpenAire data this equated to 7.6 per cent of all publications on the 
database.  The next largest project (EGI-InSpire) only produced about a quarter (27 
per cent) of the number of publications produced by HiPEAC.  The average number 
of publications per project is 16. 

Figure 14  Top 20 project acronyms by number of publications 

Patents are more evenly distributed than publications.  The maximum number of 
patents produced by projects is eight (Ultra, athenis and reservoir).  The average 
number of patents per project is 2.1. 

Interestingly, there was very little correlation between the most productive projects 
for publications and those for patents. 

81 https://www.openaire.eu/ 

57 

 
 
 
 
                                                
Figure 15  Top 20 project acronyms by number of patents 

We believe the preceding section provided an insight into some areas of the 
OpenAire publications database that had not previously been investigated.  These 
complemented previous research such as the overview of publications in a 2016 
Commission Staff Working Paper82.  The working paper provided a useful high-level 
overview of projects by thematic area, see Table 9.  It also demonstrates that 
bibliographic analysis can move beyond simple quantitative measures and move into 
more qualitative measures concerning the quality of publications in high impact 
journals. 

The thematic areas in Table 9 do not align precisely with the focus of 5G projects but 
they do provide an insight into the number and quality of publications that might be 
expected from 5G PPP projects focussing on energy (an average of 6.4 publications 
per project), Transport (1.9) and security (3.7, with only 13.6 per cent in high impact 
journals).  It is notable how relatively pitiful these levels of publications are in relation 
to the average of 16 per project presented in Figure 14. 

We recommend that wherever possible for 5G PPP projects and other Horizon 
2020 initiatives that more evaluation should be concentrated on the previous 
performance of projects.  For commonly adopted KPIs (industry participation, 
number of SMEs, research papers, patents, open innovation, input to 
standards negotiations etc.) previous averages (or even results in line with 
projects in the upper quartile of the peer group) should be adopted as minima 
for KPI targets. 

Table 9  Overview of FP7 publications by thematic area 

58 

82 EC.  2016.  Ex-Post Evaluation of the Seventh Framework Programme: Commission Staff Working 

Document COM(2016) 5 final SWD(2016) 2 final 

 
 
 
                                                
Source: SESAM RESPIR 

An added advantage of establishing targets is that the progress of projects 
towards achieving these goals can be more easily understood and presented to 
stakeholders.  Single figures about research reports mean little to the layperson 
and most policymakers.  It is outside their sphere of reference.   

For example they have little idea of whether a total output in year two of a three 
year project of 12 research reports is good or bad.  But if the average output 
goal of 16 reports from FP7 project analysis was adopted and known to 
observers (and the project team) they would know that the project was 75 per 
cent of the way to achieving the target.  This reference point or goal also 
makes it easier to present the status of projects in relation to performance.  
Simple traffic lights systems or percentage data visualisations can be easily 
understood.  

59 

 
 
 
 
 
 
5.  Recommendations for the monitoring process  

Introduction 

5.1. 
Preceding sections have focused on the data and information that will need to be 
collected from 5G PPP projects.  This section provides recommendations about how 
the data collection, validation, analysis and dissemination process can be developed 
to support the activities of 5G PPP projects, other researchers, stakeholders and the 
sectors and businesses most likely to be involved in the deployment and utilisation of 
5G.   

5.2.  Developing an innovative approach to 5G monitoring 
We believe that it is possible to develop a new approach to monitoring and 
evaluation by creating an independent 5G Observatory that should gather 5G PPP 
monitoring results and information from other studies and use this as a catalyst for 
debate and interaction between the 5G PPP projects and those who will utilise 5G 
capabilities83.  The proposed monitoring methods for the observatory will achieve two 
key impacts.  It will maximise outputs and the exploitation of 5G PPP project 
results by catalysing European and global stakeholder engagement with 
potential 5G users and other stakeholders.  Importantly the observatory should 
actively support projects to commercialise their technologies and IP by providing 
training and support and inviting potential investors to workshops. This should help to 
ensure Horizon 2020 support has a commercial impact. 

The innovative approach proposed for the observatory will overcome shortcomings 
reported in earlier PPP about initiatives lacking industrial relevance and impact.  

Innovative elements of the new independent approach to evaluation adopted by the 
observatory should include:- 

  Clarity and transparency in robustly defining ‘fit for purpose’ KPIs to monitor 
and promote the activities (including technologies and capabilities) being 
developed by 5G projects;  
Identification of clear rationale and measurement criteria for KPIs; 

 

  Development, with 5G PPP projects and 5G users, of clear targets for all 

KPIs.  These targets will stretch projects and ensure outputs are more aligned 
with the needs of 5G users; 

  Catalyse engagement between three groups – a) 5G PPP projects, b) 
potential users of 5G capabilities, and c) investors and others that can 
support the exploitation of 5G inventions; 

  Support for 5G PPP projects in providing the hard and soft skills and strategic 
market intelligence to support projects in promoting their inventions to those 
that can exploit their technologies. 

83 The term ‘capability is used throughout this report to describe technical improvements and inventions. 
They focus at a more granular level than the broader KPIs.  For example the broader performance 
goal of dynamically creating your own network will rely on a number of underlying technology 
capability improvements across wireless, optical, SDN and NFV technologies.  A longer description of 
‘capabilities’ is provided in the description for micro-objective 3A in section 1.1. 

60 

 
 
 
                                                
The Observatory should monitor 5G market and technology forecasts to provide 
5G-PPP projects and policymakers throughout EU28 Member States with up-to-date 
insights to 5G market requirements, market potential and new business models, and 
insights to the economic and societal impacts of 5G. 

The observatory will enable the mandatory requirement for monitoring and evaluation 
of the 5G PPP programme and projects (across all three phases) to be met. It will 
also address the three previous widespread concerns about evaluation, 
shortcomings in the relevance of research, and the need for research to support 
businesses and achieve societal goals. The aim of monitoring led by the observatory 
should therefore be to: 

PROJECT AIM:  Use 5G PPP monitoring results and information 
from other studies as a catalyst for debate and interaction between 
the 5G PPP projects developing technologies, those who will utilise 
5G capabilities, potential investors and citizens. 

The 5G Observatory will robustly monitor and evaluate 5G PPP projects in 
accordance with EC monitoring and evaluation guidelines84 and impact 
assessments85. Monitoring should develop a new innovative approach that meets the 
requirement expressed by the Commission “for a new system with an independent 
review of project quality”86.   

In addition to collecting, validating and disseminating monitoring information, the 
Observatory should also monitor global 5G studies and publications, particularly 
those that relate to market and user requirements. Overviews of these studies should 
be shared by the Observatory with all 5G PPP projects to make them aware of users’ 
technology requirements, new business models and forecasts of the impact of 5G on 
the economy and in addressing societal challenges87. The Observatory will also 
constitute a resource to assist the European Commission in further developing 5G 
policies, particularly the 5G Action Plan88.   

In addition to providing 5G PPP projects with information about market requirements, 
the Observatory should also host events for 5G PPP project representatives that 
introduce and develop relationships between projects and the organisations that will 
utilise 5G technologies and research.  Investors and others that might exploit and 
commercialise 5G project’s technologies should also be invited to attend workshops 
from year two onwards. Training and support should be provided by the Observatory 
to enhance project representatives knowledge of commercialisation methods and 
strategies. 

The Observatory should also meet the goal, expressed by the European 
Parliament89, of engaging citizens and civil society in public outreach activities to 
generate and sustain public support for Horizon 2020.  The Observatory should 
provide an online resource for citizens and all other stakeholder groups to learn 

84 EC. 2014.  Guidance document on monitoring and evaluation for the programme period 2014 to 2020.  

http://ec.europa.eu/regional_policy/sources/docoffic/2014/working/wd_2014_en.pdf 

85 EC.  2011.  Impact assessment accompanying the Communication from the Commission 'Horizon 

2020 - The Framework Programme for Research and Innovation'.  Commission Staff Working Paper.  
SEC(2011) 1427 final 
86 EC.  2011.  Ibid page 47.   
87 https://ec.europa.eu/programmes/horizon2020/en/h2020-section/societal-challenges 
88 EC.  2016.  5G for Europe: An action plan.  https://ec.europa.eu/digital-single-

market/en/news/communication-5g-europe-action-plan-and-accompanying-staff-working-document 

89 Official journal of the European Union. 2013.  Regulation (EU) No 1291/2013 of the European 

Parliament establishing Horizon 2020 - The Framework Programme for Research and Innovation 
(2014-2020) and repealing Decision No 1982/2006/EC. Para 22. http://eur-lex.europa.eu/legal-
content/EN/TXT/PDF/?uri=CELEX:32013R1291&from=EN 

61 

 
 
                                                
about 5G and to understand the potential impacts of 5G forecasted by the proposed 
project and other global studies. 

FP7 and Horizon 2020 evaluation reports have traditionally been passive and audit 
focused, generally reporting progress to the Commission to meet laudable 
accountability requirements. In addition to achieving these traditional objectives the 
Observatory should adopt an innovative approach and use the results from 
evaluation studies as (annual) catalysts to promote engagement between 5G PPP 
projects and the users of their research and technologies. This should help to ensure 
future 5G technology users understand what is being developed (by 5G PPP projects 
and other initiatives) and, importantly, that 5G PPP projects understand market 
requirements and are better able to develop their projects to meet market needs to 
better support the achievement of economic and societal goals.  The outcome from 
the monitoring undertaken by the Observatory will therefore be:-  

PROJECT OUTCOME:  To ensure that 5G PPP project research is 
more widely utilised and more readily meets the needs of potential 
users and citizens. 

The overall aim of the funding for 5G and the above benefits for 5G PPP projects and 
users will be achieved by the Observatory through four macro objectives that 
incrementally build on each other to achieve project outcomes, see Figure 16. Some 
of these macro objectives are supported by additional micro objectives. 

Figure 16  Four key objectives for the 5G Observatory 

5.2.1.  Operationalising objective 1: Define ‘fit for purpose’ KPIs to 
monitor progress, activities and impact of 5G PPP projects 

This study has defined 29 KPIs to monitor – a) Operating KPIs, b) Performance 
KPIs, and c) Societal KPIs.  These KPIs are sound and robust but it has already 
been identified that more detailed technological discussions are required and the 
importance of KPIs may change over time.   The Observatory should undertake 
these further developments.  

The seven operating KPIs are well defined; these relate to accounting and 
transparency objectives expected of most Horizon 2020 studies.  Twelve 
performance KPIs in this study have been broadly defined. But, as agreed with the 

62 

 
 
 
 
5G Technology Board90, some elements require considerable refinement to more 
closely delineate complex technical aspects.  Refinement of these aspects should be 
undertaken by the Observatory with representatives from 5G PPP projects and 
organisations utilising 5G capabilities. This should ensure ‘fitness for purpose’ 
amongst projects and the potential users of their research and innovation. These 
KPIs should include wireless capacity, device connections and downtime. 

Ten societal KPIs developed in this study concern impacts that are likely to arise 
when 5G is widely and commercially available; this should probably be after 2020 
when most of the 5G PPP projects have been completed. Tightly defining these KPIs 
might constrict innovation or the research activities of 5G PPP projects (see section 
4.5). Instead it is suggested that forecasts predicting the potential impact of 5G 
capabilities, being monitored by the Observatory, should be used to provide an 
insight for 5G PPP projects and European entrepreneurs and businesses about the 
likely uses and impacts of 5G research and innovations that 5G PPP projects are 
developing.  The societal KPIs examine 5G impacts in automotive, energy and 
healthcare sectors and the impact of 5G on other key strategic value chains.  In 
addition societal challenges and indicators related to security and user privacy are 
also included. 

It is important to highlight that KPIs developed and defined now may not be suitable 
in the future. Requirements perceived as important in 2016 or 2017 might be 
superseded and/or new capabilities might arise in the future. The KPIs intended to 
monitor 5G PPP projects need to be constantly monitored by the Observatory to 
ensure they are fit for purpose throughout the duration of the programme (and into 
the future in the case of the longer term impacts of societal objectives). To ensure 
that the KPIs are ‘fit for purpose’ they might need to be revised and or added to as 
time progresses.   

Other countries are developing KPIs for 5G development. Compatibility with 
international KPIs should be important to benchmark EU progress and to maintain 
the worldwide ‘standing’ of EU28 research. 

It is also evident that some larger businesses that will utilise 5G capabilities are 
developing their own KPIs. It is important that the KPIs developed in the proposed 
monitoring exercise are aligned with those of greatest interest for the business 
community and others that might utilise 5G capabilities. 

Regular monitoring of KPIs developed in other countries and by organisations should 
ensure that the KPIs in the proposed evaluation exercise are aligned with 
international measures and that they meet the needs of industry and other 
organisations. Providing 5G PPP projects with details about these KPIs and the 
areas of greatest interest in other countries and by industry should be a catalyst to 
raise awareness regarding the issues of greatest concern to the potential users of 
their research and innovations. 

5.2.2.  Objective 2: Establish performance standards and targets for KPIs 
Traditionally key performance indicators measure the quality of an organisation’s 
performance and assist in monitoring performance and the achievement of wider 
goals.  Carefully selecting and monitoring KPIs helps those working towards the KPIs 
to better understand objectives and monitor their performance91. In essence KPIs 
define success and track progress in meeting strategic goals.  

90 Presentation and discussion to the 5G Technology Board.  Brussels. 19th October 2016 
91 Lorette K.  2015.  How do I develop Key Performance Indicators?  

http://smallbusiness.chron.com/develop-key-performance-indicators-4676.html 

63 

 
 
                                                
To define success and track progress, two key pieces of information are required. 
Firstly, clear measures or metrics need to be defined. This will be achieved through 
the first objective. The second key requirement of traditional KPI monitoring is to 
establish targets. Only when targets have been set can clear objectives for a KPI be 
understood and progress towards achieving the objectives monitored. Realistic 
targets motivate continuous improvement and provide projects with clear criteria 
against which they will be evaluated. 

Defining a KPI and metrics for measurement makes it possible to monitor, but the 
magnitude of these improvements becomes somewhat nebulous. Without targets 
there is no incentive for projects to perform at anything above minimum levels 
perceived as satisfactory.  

A unique and innovative element of the proposed monitoring exercise, in comparison 
with other, FP7 and Horizon 2020 projects will be the development of clear targets 
and/or performance standards for each KPI.   

A huge number of previous FP7 and Horizon 2020 studies have been undertaken 
that have used the seven operating KPIs proposed for monitoring 5G PPP projects. 
Analysis of previous levels of leverage, SME participation, publications and patents 
in previous studies; similar to 5G PPP projects, will provide an insight to previous 
averages.  These averages could be used as minimum levels of achievement for 5G 
PPP projects that could be agreed or more challenging targets such as achieving 
outputs in line with the upper quartile of previous studies, which could be adopted. 

Discussion between our team and the 5G Technology Board in Brussels in October 
established that clear metrics for the 12 performance KPIs could be identified. The 
notion of developing targets with 5G PPP projects and potential users of 5G 
capabilities was also regarded as beneficial.  

As noted earlier (see objective 1) it would be unwise to establish precise KPIs for the 
ten societal KPIs since this might constrict innovation and constrain flexibility. The 
notion of setting targets for these KPIs is equally unwise. Nonetheless, societal KPIs 
are important because previous studies of PPP projects92 and the Seventh 
Framework Programme93 have highlighted shortcomings in achieving some of the 
strategic objectives and goals desired by the initiatives, most notably the need to 
promote higher-value-chain involvement   and/or industrial relevance and to have a 
greater impact on the economy and societal challenges.   

Section 4.5 emphasised that only seven of the 29 suggested KPIs, developed from 
5G contracts and gap analysis, had explicit targets. However, four of these targets 
related to comparison with a baseline situation at previous time - a.) Reducing 
energy consumption to ten per cent of 2010 levels, b.) Reducing average service 
creation time cycle from 90 hours to 90 minutes compared to the equivalent time 
cycle in 2010, c.) Providing 1000 times higher wireless area capacity compared to 
2010, d.) Maintaining the global market share for 5G equipment and services to the 
same level achieved in 2011.   

For all four of these KPIs the situations in 2010 and 2011 data is available; but 
research has not undertaken provide robust quantitative data94.  It will therefore be 
necessary for the Observatory to establish the baseline situation for KPIs that 

92 EC.  2013.  Final assessment of the research PPPs in the European Economic Recovery Plan p54 to 

56.  The study considered three specific sectors.  But many of the comments were generic with 
relevance to all PPP 

93 Tech4i2 and Deloitte.  2010. Impact assessment of  EC i2010. SMART 2007/0035. 
94 For example the market share of equipment in 2011 is generally reported as 40 per cent by 

observers. 

64 

 
 
                                                
reference previous time periods. 

5.2.3.  Objective 3: Implementing a methodology to measure, validate 
and disseminate information about the performance of 5G PPP 
projects 

Work undertaken by this study will lay a solid foundation of metrics and baseline data 
to guide the Observatory in the collection of monitoring information about the 
performance of 5G PPP projects. Evidence of achievements for each KPI should be 
sought and progress recorded and validated by the Observatory.  Results should 
obviously be passed on to EC project managers and the external assessors for 5G 
PPP projects. 

Members of the Observatory should regularly travel to 5G PPP project team 
meetings to better understand what projects are doing and to promote liaison with 
organisations utilising 5G capabilities.   

Data about operating and performance KPIs should be collected from projects. The 
Observatory should collect information about progress towards societal KPIs that will 
measure longer-term impacts of 5G capabilities on the economy and society. 
Observatory desk based research should monitor forecasts about the uptake and 
impact of 5G capabilities to assess societal and economic impacts of 5G PPP 
projects beyond the lifetime of the programme. 

After validation and/or the approval of data collected from projects, results should be 
published on the Observatory web site.  

This study has identified that verticals require more granular information (than 
relatively wide ranging KPIs) about the capabilities and technologies being 
developed by 5G PPP projects.  The current performance KPIs measure broad 
improvements in technical components of the 5G system and are quite agnostic of 
how and where those technical innovations are consumed.  The capabilities on the 
other hand attempt to describe what those technical improvements, when taken 
alone or in combination, will mean for the end consumer, whether that is the 
individual human consumer or an industrial consumer. For instance, the capability of 
dynamically creating your own network will rely on a number of underlying 
technology improvements across wireless, optical, SDN and NFV technologies. 
A previous EC study95 on the socio-economic impacts of 5G provided insights into 
the individual needs of verticals with respect to the 5G capabilities on offer. It is 
apparent that different industries interpret the capabilities on offer within the context 
of their own domain and existing operational technologies.  Each key industry will call 
on combinations of various capabilities in different ways and to different degrees. 
Indeed, each industry could call on the multiple combinations of the capabilities to 
address different service needs and deployment scenarios.  

The promise of enabling ultra reliable and low latency services to end users becomes 
more meaningful when described in terms of scenarios envisaged by industry. The 
needs of the automotive industry to address platooning of heavy goods vehicles on 
remote highways will likely be different to the needs of the automotive industry in 
urban environments where connected-cars may be wirelessly tethered to roadside 

95 Tech4i2, Trinity College Dublin et. al.  2016.  Identification and quantification of key socio-economic 

data for the Strategic Planning of 5G introduction in Europe. SMART 2014/0008. 
https://ec.europa.eu/digital-single-market/news/identification-and-quantification-key-socio-economic-
data-strategic-planning-5g-introduction 

65 

 
 
 
                                                
basestations for control purposes. The platooning scenario might require ultra low 
latency (as defined in milliseconds over an inter-vehicle distance) and ultra high 
reliability (since it may not have the assistance of macro-cellular coverage in remote 
areas).  But the urban scenario of connected cars may be able to relax its demand 
on those capabilities; to very low latency (as defined in milliseconds over a vehicle-
to-roadside distance) and very high reliability (as defined perhaps as a certain 
service uptime). 

In essence, for each capability to address the actual needs of the industries they are 
targeting, they need to be defined in more granular terms.  A latency capability then 
becomes  a  series  of  sub-capabilities  addressing  specific  consumption  scenarios 
such  as  ultra  low  V2V  latency,  very  low  V2X  latency,  etc.    This  teasing  out  of  the 
what latency means in terms of the end-to-end paths, and whether one or both of the 
end  points  is  mobile,  presents  a  more  crisp  and  precise  5G  proposition  to  be 
addressed by 5G PPP projects. 

To  facilitate  this  engagement  the  Observatory  should  develop  a  more  granular 
overview  of  the  technologies  and  capabilities  being  developed  by  projects.  See 
Figure 17 for an indicative overview of 5G energy, latency and wireless capabilities. 

Figure 17  An indicative matrix to map capabilities against end user 
requirements 

In  conjunction  with  the  key  industries  targeted  by  KPIs  (automotive,  energy, 
healthcare) the Observatory should map the capabilities being developed by 5G PPP 
projects against their requirements.  

In this respect it is useful that the first 5G vertical group (the 5G Automotive Alliance) 
became operational on 12th December 2016.  The Observatory should seek to work 
with (and support) this group and other sectoral 5G organisations. 

Capability  mapping  undertaken  by  the  Observatory  should  be  useful  to  the  EC,  5G 
PPP  projects  and  those utilising  5G  in a  number  of  ways.    Firstly,  it  will  enable  the 
relative  importance  of  5G  capabilities  to  be  identified  and  their  importance  for 
different sectors to be investigated for the first time. Secondly, it will be an important 
form of gap analysis to highlight gaps where Phase I and II projects are not actively 
developing  capabilities  required  by  industry.  Finally,  the  mapping  exercise  will 

66 

 
 
 
provide another focus to enable 5G projects and 5G capability users to more easily 
understand  each  others  ‘fine-grained’  capability  needs.  Capability  mapping  will 
provide  another  important  catalyst  for  engagement  between  5G  PPP  project 
personnel and the businesses and organisations that will utilise 5G. 

5.2.4.  Objective 4: Use KPI performance information and impact 
forecasts to catalyse engagement between 5G PPP project 
personnel, the businesses and organisations that will utilise 5G 
and investors that might help to exploit 5G PPP technologies 
Preceding objectives have incorporated some elements of liaison and engagement in 
developing  targets  for  KPIs  with  organisations  that  will  utilise  5G  capabilities.  An 
important objective for the proposed monitoring exercise is to remove the traditional 
repressive nature of project evaluation for audit purposes and instead use evaluation 
as  a  catalyst  to  promote  liaison  between  5G  PPP  project  personnel  and  the 
businesses and organisations that will utilise 5G capabilities. 

Exceptional data visualisation tools and excellent web site usability will be important 
in  drawing  users  to  the  project  web  site.  Results  of  KPI  monitoring  should  be 
presented  on  the  web  site  alongside  information  about  the  5G  capabilities  that 
projects  are  developing.  In  addition,  forecasts  of  growth  in  5G  markets  and  the 
potential impact of 5G on industries, the economy and society will also be provided.  
This information should be useful to 5G PPP projects and organisations utilising 5G. 

The web site will be an important element in promoting globally the progress of the 
EU  and  the  5G  PPP  projects  in  5G  research  and  innovation.  It  will  also  act  as  a 
catalyst to enable businesses to understand the impact of 5G and get in touch with 
5G PPP projects. 

5.2.5.  Monitoring objectives and activities 
Figure 18 provides an overview of the key objectives outlines and the start of this 
proposal and the work packages required to achieve each objective. As noted above 
objectives 1 and 2 and their associated works packages address short comings in 
previous research by introducing new and innovative methods to develop KPIs, 
establish metrics and define performance targets. Objectives 3 and 4 largely relate to 
the implementation of the methodology once the foundations for the monitoring 
exercise have been established.   

67 

 
 
 
Figure 18  Monitoring objectives and associated work packages 

5.2.6.  Operational activities: A conclusion 
Figure 19 provides an overview of objectives and activities and (following KPI 
protocols) it maps the above impacts and targets against the project objectives. This 
demonstrates that all objectives have clearly defined impact criteria with targets to be 
achieved by the end of the monitoring exercise. 

68 

 
 
 
 
Figure 19  Impact targets for the key objectives of the proposed project 

5.3.  Governance  
Governance of the evaluation process is required at two levels – strategic and 
operational. 

Strategic governance:  Strategic governance establishes the mission, vision 
and values for the evaluation process96.  These elements are then articulated 
as strategic goals that are defined in a business plan.  This will also usually 
verify or stipulate the Key Performance Measures.   

A number of the initiatives that undertake monitoring or observatory activities had 
executive or director boards with governance and oversight roles.  For example, the 
Cluster Observatory97 had an ‘Advisory Board’, Transparency International98 had a 
Board of Directors with 12 members, and their detailed responsibilities were set out 
in a Charter.  Larger organisations have established Committees ‘beneath’ the main 
board to oversee particular activities. 

96 OECD.  2014.  Risk Management and Corporate Governance.  http://www.oecd.org/daf/ca/risk-

management-corporate-governance.pdf 

97 The European Cluster Observatory is a single access point for statistical information, analysis and 
mapping of clusters and cluster policy in Europe that is foremost aimed at European, national, 
regional and local policy-makers as well as cluster managers and representatives of SME 
intermediaries.  http://ec.europa.eu/growth/smes/cluster/observatory 

98 Transparency International is a global movement sharing one vision: a world in which government 

politics, business, civil society and the daily lives of people are free of corruption.  
http://www.transparency.org/ 

69 

 
 
 
                                                
Recommendation:  5G PPP monitoring activities should be overseen by a board of 
personnel from DG CONNECT and, if appropriate other relevant stakeholders within 
the European Commission or from industry or citizen organisations. The Board, 
which should probably not be more than six people, should be comprised of people 
with a mix of evaluation, technical and technology impact expertise.  At least two 
board members should be experts in supporting technology exploitation. 

Operational governance:  Operational governance concerns the execution of 
the business plan and the achievement of other objectives99 stipulated by the 
strategic governance organisation or board.   

The European Commission could undertake monitoring, analytical and dissemination 
activities.  However, it is understood that Phase II 5G PPP funding is available to 
undertake support actions to enhance the activities of 5G PPP projects.  The 
particular mix of skills, industry and stakeholder contacts and breadth of knowledge 
and expertise is unlikely to be found immediately within the European Commission.   

Another important condition for the organisation(s) undertaking monitoring and 
evaluation activities is that they are reputable and independent of the main actors 
within 5G PPP projects.  This is important to overcome any allegations of bias or 
partisanship that might arise from stakeholders and/or projects. 

Recommendation:  5G PPP monitoring, analysis and dissemination activities, 
overseen by a strategic board, should be undertaken by organisation(s) with the 
demonstrable mix of appropriate skills (evaluation, monitoring, validation, analysis, 
web site creation, dissemination, engagement and consultation), experience and 
network contacts.  The organisation(s) should be independent of involvement in 5G 
PPP projects, but knowledgeable of the key issues regarding evaluation, 5G 
technical development and impacts. 

5.4.  Conclusion 

This chapter has provided robust rationale for the selection of 26 operating (seven 
KPIs), performance (12 KPIs) and societal (seven) KPIs.  Section 4.4 provided clear 
details of how each KPI should be measured.  The chapter concluded with 
recommendations regarding how the monitoring and evaluation process should be 
undertaken. 

The recommended innovative 5G Observatory will develop and implement an 
exemplar scalable and replicable approach to monitoring 5G PPP projects. The 
proposed study will ensure 5G PPP projects are better informed about their relative 
performance, the progress of other projects and the requirements of those who will 
use their research and innovations.   

This study recommends the development of a new approach to monitoring and 
evaluation by creating an independent 5G Observatory that will gather 5G PPP 
monitoring results and information from other studies and use this as catalyst for 
debate and interaction between the 5G PPP projects and those who will utilise 5G 
capabilities100.   

99 Vitria.  2008.  Formalizing Operational Governance.  http://www.vitria.com/pdf/WP-Formalizing-

Operational-Governance.pdf 

70 

100 The term ‘capability is used throughout this report to describe technical improvements and 

inventions. They focus at a more granular level than the broader KPIs.  For example the broader 
performance goal of dynamically creating your own network will rely on a number of underlying 

 
 
                                                
The Observatory will achieve two key impacts.  It will: 

i)  Maximise outputs and the exploitation of 5G PPP project results by  

ii) Catalysing 5G PPP project representatives and other European and 
global stakeholder engagement with potential 5G users and other 
stakeholders.   

Importantly it will actively support projects to commercialise their technologies and IP 
by providing training and support and inviting potential investors to workshops. This 
will help to ensure Horizon 2020 support has a commercial impact. 

The innovative approach adopted recommended for the observatory will overcome 
shortcomings reported in earlier PPP initiatives that lacked industrial relevance and 
impact.  It will also address the monitoring deficiencies highlighted in previous 
studies.   

Innovative elements of the recommended approach to evaluation include:- 

  Clarity and transparency in robustly defining ‘fit for purpose’ KPIs to 
monitor and promote the activities (including technologies and 
capabilities) being developed by 5G projects;  

Identification of clear rationale and measurement criteria for KPIs; 

 
  Development, with 5G PPP projects and 5G users, of clear targets for all 
KPIs.  These targets will stretch projects and ensure outputs are more 
aligned with the needs of 5G users; 

  Catalyse engagement between three groups – i) 5G PPP projects, ii) 

potential users of 5G capabilities, and iii) investors and others that can 
support the exploitation of 5G inventions; 

  Support for 5G PPP projects in providing the hard and soft skills and 
strategic market intelligence to support projects in promoting their 
inventions to those that can exploit their technologies. 

The Observatory will monitor 5G market and technology forecasts to provide projects 
and policymakers throughout EU28 Member States with up-to-date insights to 5G 
market requirements, market potential and new business models, and insights to the 
economic and societal impacts of 5G. 

technology capability improvements across wireless, optical, SDN and NFV technologies.  A longer 
description of ‘capabilities’ is provided in the description for micro-objective 3A in section 1.1. 

71 

 
 
 
 
                                                                                                                                      
6.  Annex 1  KPI briefing documents distributed to 

stakeholders 

6.1.  Operating KPIs 

72 

 
 
 
 
 
 
 
 
73 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6.2.  Performance KPIs  

74 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
75 

 
 
 
 
 
 
 
76 

 
 
 
 
 
 
 
 
 
 
77 

 
 
 
 
 
 
78 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6.3.  Societal KPIs 

79 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
80 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
81 

 
 
 
 
 
 
 
 
7. Annex 2  Phase 1 5G PPP projects 

A total of 19 projects were supported from the 83 proposals received by the 
Commission in response to the first call of the 5G PPP, of which all but two (5G-
Ensure and 5GEx) had a planned start date of the 1st July 2015. A brief overview of 
the 19 projects, together with start and finish dates, is provided below. This overview 
will enable those less familiar with the 5G PPP projects to gain a better 
understanding of the activities of the 19 projects. 

Euro 5G - https://5g-ppp.eu/euro-5g-deliverables-list/  

Planned start date:   July 2015 
Planned end date: 

September 2017 

The primary objective of the Euro-5G project is to facilitate 
effective and efficient co-operation and integration between all 
projects of the 5G-PPP, the European Commission, The 5G-Infrastructure 
Association, Networld2020 ETP, related projects from EUREKA, and related national 
initiatives to maximise the European momentum towards, and benefits from, the 
future 5G integrated, ubiquitous and ultra-high capacity networks. 

The metrics for evaluating the success of 5G PPP will be based on the KPIs included 
in the 5G PPP Contractual Arrangement signed by the 5G Infrastructure Association 
on behalf of the European ICT Sector and the European Commission in December 
2013. 

This project is closely linked with the 5G Infrastructure Association and will strive to 
ensure there is a seamless integration of the European industrial policies, as 
generated by the Association, with the work plans of the projects under this program 
so the results will be as useful and relevant as is possible. 

The Euro-5G project will drive the 5G-PPP goal to maintain and enhance the 
competitiveness of the European ICT industry in the 5G domain. 

The Euro-5G project will ensure that European society can enjoy the economic and 
societal benefits these future networks through promotion of uptake and simulating 
adoption and use. 

The Euro-5G project will ensure effective and efficient co-operation and integration 
between all projects of the 5G-PPP, 

The Euro-5G project will extend the project and programme level collaboration to 
other European Commission projects in H2020, the CELTIC Plus Eureka cluster, and 
related national initiatives to maximize the European momentum towards, and 
benefits from, the future 5G integrated, ubiquitous and ultra-high capacity networks. 

The Euro-5G project will monitor and analyse international 5G activities and will 
facilitate respective activities (e.g. meetings, workshops etc.) and work together with 
the 5G-Infrastructure Association and the European Commission to create good 
international relations with these global initiatives, ensure recognition of the 
European developments and promote global interoperability of 5G solutions. 

The Euro-5G project will, as a core activity, launch and maintain a comprehensive 
communications and dissemination program emanating from a lively and 
continuously updated 5G-PPP.eu website and including various social media tools as 
appropriate. This program level communications plan will be shared with all the 5G-
PPP projects and will include annual reports on the program performance. 

82 

 
 
 
 
Flex5Gware - http://www.flex5gware.eu/  

Planned start date:   July 2015 
June 2017 
Planned end date: 

 Flex5Gware will perform research, development and 
prototyping on key building blocks of 5G network elements and 
devices both in the hardware (HW) and software (SW) domains. These research, 
development and prototyping activities entail many system design challenges that 
will be solved through disruptive approaches and resulting technologies. Precisely, in 
Flex5Gware, design and development of analogue components to enable massive 
MIMO (Multiple Input Multiple Output) in mmWave (millimeter wave) spectrum bands 
will be carried out. In the mixed signal and conversion stages domain, important 
research and results will be obtained related to crucial 5G components like full 
duplex communications (simultaneous transmission and reception), high-speed 
broadband converters, etc. In the digital domain, drastic progress in the area of 
building HW components will be achieved for important features like FBMC (Filter 
Bank Multi-Carrier) transceivers, LDPC (low-density parity check) codes, etc. 
Moreover, a sophisticated, HW-agnostic, SW platform will be developed, capable of 
deciding the optimal splitting of functionality between HW and SW. This will yield 
powerful HW/SW systems, with interface abstractions, for flexible control and 
management, across heterogeneous wireless devices and access networks.  

The overall objective of Flex5Gware is to deliver highly reconfigurable HW platforms 
together with HW-agnostic SW environments, targeting both network elements and 
devices, and taking into account the need for increased capacity, reduced energy 
footprint, as well as scalability and modularity for enabling a smooth transition from 
4G mobile wireless systems to the 5G era. 

In order to fulfill the above overall objective, Flex5Gware will address the following 
specific objectives: 

 

Increase the HW versatility and reconfigurability:  

  Designing and developing HW-agnostic, flexible and cost-effective SW 

platforms 

 

Increasing the overall achievable capacity provided by 5G communication 
platforms 

  Decreasing the overall energy consumed by 5G communication platforms 

 

Identifying and prototyping key HW and SW building blocks and providing a 
proof of concept for all developments in Flex5Gware 

83 

 
 
 
 
 
 
 
 
5G NORMA - https://5gnorma.5g-ppp.eu/  

Planned start date:   July 2015 
Planned end date:  December 2017  

With 5G NORMA, leading players in the mobile ecosystem aim to 
underpin Europe’s leadership position in 5G. 

The key objective of the project is to develop a conceptually novel, adaptive and 
future-proof 5G mobile network architecture. 

The architecture will enable unprecedented levels of network customisability, 
ensuring stringent performance, security, cost and energy requirements to be met; as 
well as providing an API driven architectural openness, fuelling economic growth 
through over-the-top innovation. 

To show the value of the architecture both to the wireless industry and to the users in 
society, the project will conduct a socio-economic analysis –closely interacting with 
the use case analysis– to evaluate and quantify the benefits of 5G NORMA 
innovations. 

Furthermore, the 5G NORMA consortium will also demonstrate the feasibility of the 
key innovations developed in the project, in particular the service and context 
dependent adaptation of network function as well as the software defined mobile 
network control. 

The key objective is achieved through innovations like “multi-service and context-
aware adaptation of network functions” and “mobile network multi-tenancy” enabled 
by novel concepts of “adaptive (de)composition and allocation of mobile network 
functions”, “software-defined mobile network control”, as well as “joint optimization of 
mobile access and core network functions.” 

The objectives of 5G NORMA have technical, economic, social and energy-efficiency 
dimensions. They are supported by the technical objectives of 5G NORMA: 

  Multi-service- and context-aware adaptation of network functions 

  Mobile network multi-tenancy 

  Adaptive (de)composition and allocation of mobile network functions 

  Software-Defined Mobile network Control (SDMC) 

  Joint optimisation of mobile access and core network functions 

84 

 
 
 
 
 
METIS-II - https://metis-ii.5g-ppp.eu/  

Planned start date:   July 2015 
June 2017 
Planned end date: 

The METIS-II builds on the successful METIS project and aims 
to develop the overall 5G radio access network design and to 
provide the technical enablers needed for an efficient integration 
and use of the various 5G technologies and components currently developed. 
METIS-II will provide the 5G collaboration framework within 5G-PPP for a common 
evaluation of 5G radio access network concepts  

METIS-II will provide an overall 5G RAN design, describing an overall protocol stack 
architecture with all the functionalities and interfaces needed to fulfill the 5G vision. 
The overall 5G RAN design will be built upon the following key innovation pillars 
developed in METIS-II: 

  Holistic spectrum management architecture. 

  Holistic air interface harmonisation framework. 

  Agile Resource Management (RM) framework. 

  Cross-layer and cross-air-interface access and mobility framework. 

  Common control and user plane framework. 

METIS-II pursues the following key objectives: 

  Develop the overall 5G RAN design, focusing particularly on designing the 
technology for an efficient integration of legacy and novel radio access 
network (RAN) concepts into one holistic 5G system. 

  Provide the 5G collaboration framework within 5G PPP for the 5G RAN 

design and a common evaluation of 5G RAN concepts from both 
performance and technoeconomical perspective. More specifically, METIS-II 
will further refine 5G scenarios, requirements and key performance indicators 
(KPIs), develop a performance and techno-economical evaluation framework, 
and an open-source 5G evaluation and visualisation tool for illustrating the 
key 5G use cases and RAN design solutions. For all mentioned aspects, as 
well as spectrum and overall 5G RAN design aspects, METIS-II strives to 
facilitate consensus building within 5G PPP, for instance through the 
organization of a series of 5G PPP crossproject workshops. 

  Prepare concerted action towards regulatory and standardisation bodies for 
an efficient standardisation, development and economically attractive rollout 
of 5G. 

85 

 
 
 
 
5G-Xhaul - http://www.5g-xhaul-project.eu/ 

Planned start date:   July 2015 
June 2018 
Planned end date: 

 5G-XHaul proposes a converged optical and wireless transport solution able to 
flexibly connect Small Cells to the core network. Exploiting user mobility, our solution 
enables the dynamic allocation of network resources to hotspots. To support these 
novel concepts, the main technical and research challenges are the development of: 

  Dynamically programmable, high capacity, low latency, point-to-multipoint 

mm-Wave transceivers, cooperating with sub-6-GHz radios. 

  Time-shared optical network offering elastic bandwidth allocation, cooperating 

with advanced passive optical networks. 

  Software-defined cognitive control plane, able to forecast traffic demand in 

time and space, and accordingly, reconfigure network components. 

Small Cells, Cloud-Radio Access Networks (C-RAN), Software Defined Networks 
(SDN) and Network Function Virtualization (NVF) are key enablers to address the 
demand for broadband connectivity with low cost and flexible implementations. 
However, Small Cells, in conjunction with C-RAN, SDN, NVF pose very stringent 
requirements on the transport network. 

5G-XHaul will develop wireless solutions for dynamic backhaul and fronthaul 
architectures alongside very high capacity optical interconnects. 

5G-XHaul aims at building up an ambitious converged optical and wireless network 
solution that relies on a flexible infrastructure able to support of the backhaul and 
fronthaul networks required to tackle the future challenges that RAN imposes. 5G-
XHaul will provide an efficient, reconfigurable, modular and highly scalable platform 
to support RAN processing, depending on different architectural solutions, network 
elements and devices. The future 5G backhaul requires higher flexibility to unlock the 
potential of increased, more efficient and more flexible spectrum usage. The high 
deployment cost of optical fibre will be alleviated in 5G-XHaul with the use of mm-
Wave communications for both backhaul and fronthaul purposes (opex and capex). 

86 

 
 
 
 
Selfnet - https://selfnet-5g.eu/  

Planned start date:   July 2015 
June 2018 
Planned end date: 

The SELFNET project aims to design and implement an autonomic network 
management framework to achieve self-organizing capabilities in managing network 
infrastructures by automatically detecting and mitigating a range of common network 
problems that are currently still being manually addressed by network operators, 
thereby significantly reducing operational costs and improving user experience. 

Technical and Research Challenges that Selfnet faces are: 

  Enabling and optimising the holistic use of Software Defined Networking 

(SDN), Network Function Virtualisation (NFV), cloud computing, AI and other 
related technologies to achieve novel cost-effective realtime autonomous 5G-
network management. 

  Designing highly usable HoN metrics that can precisely reflect the current 

network and service operation conditions in relation to the 5G Key 
Performance Indicator (KPI) requirements. 

  Devising innovative, efficient and scalable proactive algorithms to resolve or 

mitigate the predicted potential network management issues. 

The main objectives of Selfnet are: 

  To realise self-monitoring and detection facilities to enable timely status 

awareness of the 5G network infrastructure in terms of a customisable and 
extensible set of high-level Health of Network (HoN) metrics. 

  To realise an extensive and distributed 5G Self-Organising Network (SON) 
autonomic management engine based on new Artificial Intelligence (AI) and 
related technologies. 

  To realise 5G SON orchestration and virtual infrastructure management. 

  To apply the SELFNET framework in a range of essential 5G SON use cases 
designed to address major problems in the current network management 
faced by the network operators. 

87 

 
 
 
 
FANTASTIC 5G - http://fantastic5g.eu/  

Planned start date:   July 2015 
Planned end date:  December 2017 

The main challenge for FANTASTIC-5G is to develop a 
modular air interface which is able to support all the 
anticipated use-cases with highest efficiency and scalability without being overly 
complex on the network side. To this end, the project will develop the technical AI 
components (e.g. flexible waveform and frame design, scalable multiple access 
procedures, adaptive retransmission schemes, enhanced multi-antenna schemes 
with/without cooperation, advanced multi-user detection, interference coordination, 
support for ultra-dense cell layouts, multi-cell radio resource management, device-to-
device) and integrate them into an overall AI framework where adaptation to the high 
degree of heterogeneity 5G will face will be accomplished. Our work will also 
comprise intense validation and system level simulations. The ambition is to 
outperform a system involving dedicated air interfaces. 

FANTASTIC-5G objectives are: 

  To develop a flexible and scalable multi-service air interface. 

  With ubiquitous coverage and high capacity where and when needed. 

  Being highly efficient in terms of energy and resource consumption. 

  Being future proof and allowing for sustainable delivery of wireless services 

far beyond 2020. 

  To evaluate and validate the developed concepts. 

  Build up consensus on reasonable options for the standardization of 5G. 

The project deals exclusively with lower carrier frequencies (< 6 GHz). 

88 

 
 
 
 
 
 
5G-Crosshaul - http://5g-crosshaul.eu/  

Planned start date:   July 2015 
June 2017 
Planned end date: 

Xhaul will design the new 5G-backhauling/fronthauling network for 5G and will solve 
the fundamental cost, network efficiency and scalability issues of 5G transport 
network in the following aspects: 

 

 

It will design converged networking devices (XFE) running a unified data 
plane capable of transporting all kinds of Xhaul traffic. This will significantly 
reduce the network cost leveraging the integrated network design and will 
improve network utilization. 

It will facilitate network densification. In Xhaul, new physical layer 
technologies and solutions (leveraging optical fiber, mmWave, free-space 
optics and low-cost copper infrastructure, etc.) will be explored to significantly 
reduce the deployment and installation cost. 

 

It will design a centralized control plane following the SDN paradigm providing 
high flexibility and scalability (XCI).  

Xhaul aims at developing an adaptive, sharable, cost-efficient 5G-transport network 
solution integrating the fronthaul and backhaul segments of the network. This 
transport network will flexibly interconnect distributed 5G radio access and core 
network functions, hosted on in-network cloud nodes, through the implementation of 
two novel building blocks: i) a control infrastructure using a unified, abstract network 
model for control plane integration (Xhaul Control Infrastructure, XCI); ii) a unified 
data plane encompassing innovative high-capacity transmission technologies and 
novel deterministic latency switch architectures (Xhaul Packet Forwarding Element, 
XFE). Xhaul will greatly simplify network operations despite growing technological 
diversity. It will hence enable system-wide optimization of Quality of Service (QoS) 
and energy usage as well as network aware application development. 

89 

 
 
 
 
 
CogNet - https://5g-ppp.eu/cognet/  

Planned start date:   July 2015 
Planned end date:  December 2017 

The Cognet proposal focuses on applying Machine Learning research to enable the 
level of Network Management technology required fulfilling the 5G vision. 

Technical and research challenges that CogNet faces are: 

  Collect and process “Big Data” from the 5G network in real time. 

  Develop new algorithms utilizing machine learning to learn from the collected 

data and apply this to the network management. 

 

Improve the scalability, resilience and security of 5G network. 

  Make measurable improvements to the networks as recognized through KPIs. 

CogNet aims to research and develop a real-time network management platform with 
the capability to scale to address the requirements of the future 5G network. 

More specifically: 

  To collect and preprocess big data from the 5G network.  

  Develop, a system for self-management of network nodes while supporting 

federated network management. 

  Application of Machine Learning algorithms to address: 

o  Demand prediction and provisioning allowing the network to resize 

using virtualization. 

o  Network resilience issues including identifying network errors, faults or 

conditions such as congestion or performance degradation. 

 

Identifying serious security issues such as unauthorized intrusion or 
compromised network components and liaises with autonomic network 
management to formulate and take appropriate action. 

90 

 
 
 
 
 
Coherent - http://www.ict-coherent.eu/  

Planned start date:   July 2015 
Planned end date:  December 2017 

 The exponential growth of mobile traffic, drastically 
increasing of network complexity, and the strong need for inter-network coordination 
of wireless network resources call for breakthroughs in control, coordination and 
flexible spectrum management in 5G networks.  

The COHERENT project deals with challenges in inter-network coordination of 5G 
heterogeneous radio access networks, by introducing software-defined networking 
design principles into radio access networks. It will develop common control 
interfaces and software-development kits to enable programmable control and 
coordination in heterogeneous mobile networks. The programmable control in 5G 
radio access networks will provide mobile operators a flexible and cost efficient way 
to implement new low layer control functions, provision wireless resources, manage 
different types of radio access networks, and thus to support open innovation in 5G 
mobile networks.  

The COHERENT project aims to research, develop and showcase a unified 
programmable control framework for 5G heterogeneous radio access networks. The 
project will focus on three innovations in control and coordination of 5G networks: 

  Software defined networking for radio access networks, to enable a scalable 

and flexible control and coordination framework for complex resource 
coordination and spectrum management in 5G networks. 

  Efficient radio resource modeling and management in programmable radio 

access networks, with well-defined control interfaces and protocols to greatly 
simplify the management of heterogeneous mobile networks, to be verified by 
various 5G use cases. 

  Flexible spectrum management, based on the awareness of spectrum usage 
through the COHERENT control framework, to support different spectrum 
access schemes, including but not limited to, Licensed Shared Access, 
License Assisted Access, and flexible duplex in 5G radio access networks. 

91 

 
 
 
 
mmMAGIC - https://5g-mmmagic.eu/  

Planned start date:   July 2015 
June 2017 
Planned end date: 

 The use of very high frequencies for mobile communications is challenging but 
necessary for supporting 5G’s extreme mobile broadband service which require very 
high (up to 10 Gbps) data rates, and in some scenarios, also very low end-to-end 
latencies (less than 5 ms). 

In the project, waveform, frame structure and numerology will be developed and 
designed as well as novel adaptive and cooperative beam-forming and tracking 
techniques to address the specific challenges of millimetre wave mobile 
communications. It will undertake extensive radio channel measurements in the 6-
100 GHz range and develop advanced channel models for rigorous validation and 
feasibility analysis of the proposed concepts, as well as usage by regulatory and 
standards fora. Seamless and flexible integration with other 5G and legacy radio 
interfaces will be realized through design and validation of novel internetworking 
functions and architecture components. 

The main objective of the mmMAGIC project is to develop concepts and key 
components for a new 5G mobile radio access technology which is expected to 
operate in a range of frequency bands between 6 and 100 GHz, here referred to as 
millimeter wave frequencies. Frequency regions suitable to support the identified 5G 
use cases will be identified and assessed. This new RAT is envisaged as a key 
component in the overall 5G multi-RAT ecosystem. The project also aims to 
accelerate standardisation of mm-wave technologies for 5G so that industry and 
citizens will benefit from commercialisation around 2020. 

92 

 
 
 
 
 
CHARISMA - http://www.charisma5g.eu/  

Planned start date:   July 2015 
Planned end date:  December 2017 

 CHARISMA brings together 10G-wireless (via mm-
wave/60GHz & free-space optics, FSO) access and 100G fixed optical (OFDM-PON) 
solutions through an intelligent cloud radioaccess-network (C-RAN) and intelligent 
radio remote head (RRH) platform with IPv6 Trust Node routing featuring very low 
latency traffic management. Low-cost Ethernet is used across the front- and 
backhaul, with virtualised end-user equipment (vCPE), intelligence distributed across 
the back-, front-hauls and perimetric data transports. 

Ad-hoc mobile device interconnectivities (D2D, D2I, C2C etc.) content delivery 
network (CDN) and mobile distributed caching (MDC) offer an energy-efficient (better 
than x20 improvement possible) information-centric network (ICN) architecture. 
Furthermore, caching provides efficient utilization of scarce resources by lowest 
common aggregation of data and/or localized execution of communications. 

Also, CHARISMA will exploit network programmability and virtualisation technologies 
to achieve multi-tenancy and enable rapid adoption of emerging network 
applications. 

CHARISMA proposes an intelligent hierarchical routing and paravirtualised 
architecture that combines two important concepts: devolved offload with shortest 
path nearest to end-users and end-to-end security service chain via virtualized open 
access physical layer security (PLS). 

The architecture of CHARISMA focuses on two goals: low-latency (<1ms) and 
security that are key for the development of future converged wireless/wireline 
advanced 5G networks. CHARISMA will design and implement a cloud infrastructure 
solution with increased spectral, energy efficiency and enhanced performance. 

93 

 
 
 
 
VirtuWind - http://www.virtuwind.eu/  

Planned start date:   July 2015 
June 2018 
Planned end date: 

The main aim of VirtuWind is to develop a SDN and NFV 
ecosystem for industrial domains, based on open, modular, and secure 
communication framework, leading to a prototype demonstration for intra-domain and 
inter-domain scenarios in real wind parks as a representative use case of industrial 
networks, and quantify the economic benefits of the solution. 

Translating VirtuWind’s objectives and applying the concepts developed in the 
project to the domain of wind energy will address the following technical and 
research challenges: 

  Programmable industrial networks via SDN. 

  Multi-tenancy support via NFV as multiple stakeholders need different access 

profile. 

  Techno-economic analysis for foreseen OPEX and CAPEX reduction. 

 

Increase in service provisioning velocity. 

The main aim of VirtuWind is to develop a SDN & NFV ecosystem for industrial 
domains, based on open, modular, and secure communication framework, leading to 
a prototype demonstration for intra-domain and inter-domain scenarios in real wind 
parks as a representative use case of industrial networks, and quantify the economic 
benefits of the solution. Following are the 5 key objectives of VirtuWind: 

  Realize industrial-grade QoS for intra-domain SDN solution. 

  Guarantee inter-domain QoS for SDN based multioperator ecosystem. 

  Reduce time and cost for service provisioning and network maintenance. 

  Assure security-by-design for the SDN and NFV ecosystem. 

  Field trial of intra- and inter-domain SDN and NFV prototype. 

94 

 
 
 
 
 
 
SONATA - http://www.sonata-nfv.eu/  

Planned start date:   July 2015 
Planned end date:  December 2017 

software 
SONATA targets both the flexible programmability of 
networks and the optimization of their deployments. As such they will help address 
the significant challenges associated with both the development and deployment of 
the complex services envisioned for 5G networks and empowered by these 
technologies. 

Software Defined Networking (SDN) and Network Function Virtualization (NFV) are 
emerging as major transformational technologies, evolving the telecom sector with 
new network capabilities and business opportunities. SONATA addresses the 
significant challenges associated with both the development and deployment of the 
complex services envisioned for 5G networks and empowered by these 
technologies: 

  Models, methods and tools for programming virtualized services. 

  Methods, enablers and algorithms for uniform and integrated orchestration of 

network services, on top of an infrastructure providing connectivity, 
computation, and storage resources. 

  Tools for efficient and reliable deployment and management of developed 

services in a dynamic and scalable manner. 

  Efficient integration of development and operations (DevOps) of services 

between multiple stakeholders. 

SONATA targets the flexible programmability of software networks, supporting 
network function chaining and orchestration. Innovations will make service platforms 
modular and easier to customize to the needs of different service providers, and 
introduce a specialised DevOps model for supporting developers. 

Specific objectives include: 

  Reduce time-to-market of networked services, by a) streamlining 

development with abstract programming models and SDK; and b) applying a 
DevOps model that integrates operators, manufacturers and third-party 
developers.  

  Optimise resource utilisation via orchestration and reduce costs of service 
deployment & operation, by a) mapping complex services to connectivity, 
computing and storage resources; and b) automatically reconfiguring 
running/competing services. 

  Accelerate the industry adoption of software networks, by a) supporting the 
full service lifecycle: development, testing, orchestration, deployment, 
management and operations; and b) defining a roadmap for uptake of its 
results towards the larger transition to SDN/NFV. 

95 

 
 
 
 
 
 
 
 
 
 
SPEED-5G - https://speed-5g.eu/  

Planned start date:   July 2015 
Planned end date:  December 2017 

On the research front, SPEED-5G will investigate and develop different sets of 
mechanisms and techniques for dynamic capacity and coverage expansion, 
including: 

  Combining radio technologies, spectrum and transmit energy. 

  Coexistence and interference control mechanisms using emerging 

multicarrier waveforms. 

  Operation in dirty/grey spectrum. 

  Management mechanisms in support of spectrum microtrading in lightly 

licensed bands, which are currently not utilized. 

  Techniques in support of centralized / distributed smart resource 

management. 

SPEED-5G will research and develop the appropriate functionalities for addressing 
the mentioned challenges and a testbed to illustrate the gains that can be achieved. 
SPEED-5G will undertake a holistic experimentation by extending an existing 5G 
testbed, in particular w.r.t. spectrum and traffic management features in ultra-dense 
environments. 

The main objective of SPEED-5G is to achieve a significantly better exploitation of 
heterogeneous wireless technologies, providing higher capacity together with the 
ultra-densification of cellular technology, and effectively supporting the new 5G 
Quality of Experience (QoE) requirements. 

In SPEED-5G we will develop new techniques for optimising spectrum utilisation, 
following three main dimensions: 

  Ultra-densification through small cells. 

  Additional spectrum. 

  Exploitation of resources across technologies. 

In SPEED-5G this three-dimensional model is referred to as extended-Dynamic 
Spectrum Allocation (DSA), where several spectrum bands, cells and technologies 
are jointly managed in order to offer improved QoE and a tremendous capacity 
increase in a cost-efficient manner. 

96 

 
 
 
 
 
 
 
 
SESAME - http://www.sesame-h2020-5g-ppp.eu/  

Planned start date:   July 2015 
Planned end date:  December 2017 

A fundamental component of SESAME will be the virtualisation 
of Small Cell and their utilisation and partitioning into logically 
isolated ‘slices’, offered to multiple operators/tenants. The main aspect of this 
innovation will be the capability to accommodate multiple operators under the same 
infrastructure, satisfying the profile and requirements of each operator separately. 

The Light DC execution platform will be used to support the required VNFs that 
implement the different features/capabilities of the Small Cells. Solutions for 
aggregation of data, transcoding of video content with optimised delivery in edge 
networks and caching at the very edge of the network, will enable a reduction in 
transport time and therefore, provide a crucial route to successfully reducing service-
level latency. 

SESAME targets innovations around three central elements in 5G: the placement of 
network intelligence and applications in the network edge through Network Functions 
Virtualisation (NFV) and Edge Cloud Computing; the substantial evolution of the 
Small Cell concept, already mainstream in 4G but expected to deliver its full potential 
in the challenging high dense 5G scenarios; and the consolidation of multi-tenancy in 
communications infrastructures, allowing several operators/service providers to 
engage in new sharing models of both access capacity and edge computing 
capabilities. 

SESAME proposes the Cloud-Enabled Small Cell (CESC) concept, a new multi-
operator enabled Small Cell that integrates a virtualised execution platform (i.e., the 
Light Data Center – Light DC) for deploying Virtual Network Functions (VNFs), 
supporting powerful Self-x management and executing novel applications and 
services inside the access network infrastructure. The Light DC will feature low-
power processors and hardware accelerators for time critical operations and will build 
a high manageable clustered edge-computing infrastructure. 

97 

 
 
 
 
 
 
SUPERFLUIDITY - http://superfluidity.eu/  

Planned start date:   November 2015 
Planned end date:  December 2017 

SUPERFLUIDITY offers a converged solution to counter the complexity emerging 
from three forms of heterogeneity: Heterogeneous data traffic and end-points; 
Heterogeneity in services and processing needs; Heterogeneity in access 
technologies and their scale. 

Many shortcomings affect today’s networks: overly long provisioning times; reliance 
on proprietary, hard-to-modify and cost-ineffective hardware devices; and the 
complexity arising from a wide range of heterogeneous access technologies. 

The main goal of SUPERFLUIDITY is to run network-processing virtualised, on-
demand, on third-party infrastructure located throughout the network, and to develop 
technologies allowing such services to be “superfluid”: 

  Fast instantiation times (in milliseconds). 

  Fast migration (in hundreds of milliseconds). 

  High consolidation (running thousands on a single server). 

  High throughput (10Gb/s and higher). 

SUPERFLUIDITY tackles these challenges with a multi-pronged comprehensive 
strategy: 

  Flexibility, via an architectural decomposition of network components and 

network services into elementary, reusable primitives. 

  Simplicity, via a cloud-based architecture. 

  Agility, via virtualisation of radio and network processing tasks. 

  Portability and viability, through platform-independent abstractions, permitting 

reuse of network functions across multiple heterogeneous hardware 
platforms. 

  High performance, via software acceleration, specialisation and adaptation to 

hardware accelerators. 

98 

 
 
 
 
 
 
 
 
5GEx - http://www.5gex.eu/  

Planned start date:   October 2015 
Planned end date:  March 2018 

The current market fragmentation results from having a multitude of 
telecommunications network and cloud operators each with a footprint focused on a 
speciﬁc region, while lacking inter-operator collaboration business models, services 
and supporting tools. This makes it infeasible to deploy and offer cost-effective 
infrastructure services spanning multiple countries. Existing services and inter-
operator collaboration tools are very limited and cumbersome. Our challenges are to 
invent technical and business solutions to autonomous orchestration of services 
across multi-domain and multi-technology environments. 

5G Exchange (5GEx) will enable efficient business and technical cross-domain 
orchestration of services over multiple administrations as well as multi-domain 
orchestration over single administrations. Such orchestration shall allow instantiating 
end-to-end networks and services into multi-vendor and heterogeneous technology 
resource environments. In order to overcome the traditional separation of network 
resources from compute and storage, 5GEx will realise composite services by 
seamlessly combining networking with computing and storage across domains. 
Service deployment, activation and further management can be viewed as the 
efficient mapping of service elements onto an abstracted model based on a 
virtualised substrate belonging to multiple operators.  

The goal of the 5GEx project is the automated assignment and mapping of 
virtualised service elements, which represent service and network functions and 
components, to the underlying (physical hardware) resources across domains. 5GEx 
cross-domain business models and orchestration shall optimise 5G business and 
operational policies and KPIs including substrate utilisation, OPEX reduction and 
revenue maximisation. Business-wise, 5GEx will create opportunities for operators to 
buy, sell, and integrate infrastructure services in an automated and cost-effective 
manner. 5GEx will build a working end-to-end system and deploy a proof-of-concept 
prototype, which includes the concept of a “Sandbox Exchange”. Sandbox Exchange 
will enable new ways of experimentation and use case validation close to an 
operating environment facilitating the transition from experimentation, to piloting and 
further to real-world operation. 5GEx will contribute to the relevant standard forums 
and Open Source communities. 

99 

 
 
 
 
 
5G-ENSURE - http://www.5gensure.eu/  

Planned start date:   November 2015 
Planned end date:  October 2017 

The need for a new security architecture is motivated by the fact that 5G is a platform that 
goes beyond telecoms and which will be far more decoupled from specific hardware and 
physical control of the network. 5G-ENSURE aims to deliver a reference security architecture 
for 5G which can be used by all 5G projects, and to support its use by providing a useful and 
useable initial set of security enablers addressing core concerns. The key security enablers 
considered in 5G-ENSURE are: 

  Authentication 

  Authorisation and Accounting 

  Privacy 

  Trust 

  Security Monitoring 

  Network Management and Virtualisation Isolation 

The overall objective of 5G-ENSURE is to become the reference project for 
everything that concerns security in 5G while contributing to 5G resilience. As such, 
this project will be focusing on 5G Security Architecture able to generate the 
necessary trust and confidence for 5G to be widely adopted and deliver its promises 
through applications enabled. To achieve this overall ambition, the following 
objectives will be targeted: 

  Collect, analyse and prioritise 5G security requirements. 

  Define security architecture for 5G. 

  Specify, develop and test an initial set of security enablers for 5G. 

  Demonstrate the potential of 5G Security enablers developed in the context of 

5G representatives’ showcases. 

  Advertise and exploit the Project’s results and offering to the 5G PPP 

community and beyond. 

  Foster 5G Security Vision by delivering a 5G Security Roadmap. 

  Act as pre-standardisation consensus builder. 

100 

 
 
 
 
 
 
8. Annex 3  Glossary  

Definition 

Actions  

The fact or process of doing something, typically to achieve an aim.  

Alignment  

Baseline 

Benchmarking 

Beneficiary 

Coherence 

Intermediaries undertake actions or interventions to support the 
participants of their initiatives. 

Business Information Technology alignment is the correspondence 
between the business objectives and the Information Technology 
requirements of an enterprise. These two factors often seem to 
contradict, but many economic and technical experts agree that 
alignment between them, maintained over time, is crucial to the success 
of an enterprise. 

A baseline assessment provides a critical reference point for assessing 
changes and impact.  It establishes a basis for comparing the situation 
before and after an intervention.  Baseline data enables inferences to be 
made about the effectiveness of an intervention. 

Benchmarking is the reference to a qualitative and quantitative standard 
for comparison of the performance of an intervention. Such a standard 
will often be the best in the same domain of intervention or in a related 
domain. Benchmarking is facilitated when, at the national or regional 
level, there is comparative information of good and not so good practice. 
The term benchmarking is also used to refer to the comparison of 
contextual conditions between territories. 

A beneficiary is a person or organisation directly benefitting from the 
intervention whether intended or unintended. Some people may be 
beneficiaries without necessarily belonging to the group targeted by the 
intervention. Similarly, the entire eligible group does not necessarily 
consist of beneficiaries. 

Coherence is the extent to which complementarity or synergy can be 
found within a programme and in relation to other programmes. The 
internal coherence refers to the correspondence between the resources 
allocated to a programme and its objectives. The external coherence 
refers to the adequacy between the evaluated programme and other 
related programmes and instruments. In European legal texts or 
working documents the term is often used equivalently to ‘consistency’. 

Cohort  

A group of people banded together or treated as a group.  The 

Common 
Indicator 

intermediaries regarded a cohort as a group of students recruited and or 
working together or a ‘year’ group.  

An indicator is a quantitative or qualitative factor or variable that provides 

a simple and reliable means to measure achievement, to reflect 
changes connected to an intervention, or to help assess the 
performance of a development actor. A common set of indicators, 
binding for all Member States, serves to measure achievements and 
changes at European level. 

Consistency 

Consistency is the harmony, compatibility, correspondence or uniformity 

among the parts of a complex thing. In European legal texts and 
working documents it is often used equivalently to coherence. 

Context 

Context is the socio-economic and environmental situation in which an 
intervention is implemented. The contextual situation and trends are 
taken into account in programming as well as in programme 

101 

 
 
 
evaluations. 

Criteria  

A principle or standard by which something may be judged or decided. 

Digital inclusion   Digital inclusion is a set of actions or policies intended to help people who 

are not online currently to be motivated, develop the skills and obtain 
access to affordable hardware, software, technical support/information 
and broadband connectivity, so they can begin to use the technology to 
improve their lives. The terms ‘Digital Inclusion’ and ‘eInclusion’ are 
used inter-changeably in many reports to avoid the awkward 
juxtapositioning of words. 

Digital 

Technology  

Digital Technologies are used to effectively find, analyse, create, 

communicate, and use information in a digital context. This 
encompasses the use of web 2.0 tools, digital media tools, 
programming tools and software applications. 

Effectiveness 

Effectiveness is the extent to which objectives pursued by an intervention 

are achieved. An effectiveness indicator is calculated by relating an 
output, result or impact to a quantified objective. 

Efficiency  

A level of performance that describes a process that uses the lowest 

amount of inputs to create the greatest amount of outputs. Efficiency 
relates to the use of all inputs in producing any given output, including 
personal time and energy. 

eInclusion   

eInclusion is a term used to describe the fact that the world can be 

divided into people who do and people who do not have access to - and 
the capability to use - modern information technology. According to 
advocates, einclusion has the power to: close the gap between 
developed and less developed countries; promote democracy and 
mutual understanding; and empower disadvantaged individuals, such as 
the poor, the disabled, and the unemployed.  The terms ‘eInclusion’ and 
‘Digital Inclusion’ are used inter-changeably in many reports to avoid the 
awkward juxtapositioning of words. 

Evaluation  

The systematic determination of a subject's merit, worth and significance, 

using criteria governed by a set of standards. It can assist an 
organisation, program, project or any other intervention or initiative to 
assess any aim, realisable concept/proposal, or any alternative, to help 
in decision-making; or to ascertain the degree of achievement or value 
in regard to the aim and objectives and results of any such action that 
has been completed.  The terms ‘evaluation’ and ‘monitoring’ are often 
sued inter-changeably in reports to avoid the awkward juxtapositioning 
of words. 

Evaluation plan 

An Evaluation Plan sets out the evaluation activities including the 

institutional arrangements (evaluation governance) and management 
provisions (evaluation management) for a whole programme 
implementation period.  

Ex ante 

conditionalities 

102 

Ex ante conditionality seeks to ensure that the necessary preconditions 
for investments to flourish are in place. Four types of preconditions can 
be identified: (i) regulatory, (ii) strategic, (iii) infrastructural-planning and 
(iv) institutional. Regulatory preconditions primarily relate to 
transposition of EU legislation. Strategic preconditions are linked to 

 
 
 
 
 
 
Financial 

instruments 

Governance 

strategic frameworks for investments; while infrastructural- planning 
preconditions relate to major infrastructure investments. Institutional 
preconditions aim to ensure institutional effectiveness and adequate 
administrative capacity. 

Financial instruments are measures of financial support provided on a 

complementary basis from the budget in order to address one or more 
policy objectives of the funding organisation. Such instruments may take 
the form of loans, guarantees, equity or quasi-equity investments, or 
other risk-sharing instruments, and may, where appropriate, be 
combined with grants. 

Governance is usually understood as the exercise of economic, political 
and administrative authority to manage a country’s affairs at all levels. It 
comprises the mechanisms, processes and institutions through which 
citizens and groups articulate their interests, exercise their legal rights, 
meet their obligations and mediate their differences. In contrast to older 
(narrower) definitions the term does not only indicate what a 
government does, but also includes structures set up and actions 
undertaken in partnership with the civil society and the private sector. 

Impact  

The force exerted by a new idea, concept, technology, or ideology.  In the 

context of this study four global impacts concerning the impact of 
eInlcusion have been identified – personal, social, cultural and 
economic. 

Information  

Information is stimuli that has meaning in some context for its receiver. 

When information is entered into and stored in a computer, it is 
generally referred to as data. After processing (such as formatting and 
printing), output data can again be perceived as information. When 
information is packaged or used for understanding or doing something, 
it is known as knowledge. 

Initiative  

 In the context of this study an initiative is particular action or set of 

Instruments  

Intermediary  

Intervention 

logic 

Macro, meso 
and micro  

Measuring  

activities undertaken in a complementary manner to assist participants.  
Intermediaries may provide one or many initiatives. 

A device for recording or measuring views or feedback.  In the context of 
this study the term instrument refers to the questionnaires and other 
methods to collect information. 

The term intermediary is used to describe an organization or group of 
individuals working at the operational level (below policymakers and 
strategists) to provide services that will achieve objectives. 

An intervention logic represents a methodological instrument which 
establishes the logical link between programme objectives and the 
envisaged operational actions. It shows the conceptual link from an 
intervention's input to its output and, subsequently, to its results and 
impacts. Thus an intervention logic allows an assessment of a 
measure's contribution to achieving its objectives. 

Terms used to describe scales of analysis or policymaking.  Macro is 
derived from the word macroscopic meaning large, meso refers to 
middle or intermediate, micro refers to small scales of analysis. 

Values made meaningful by quantifying into specific units. Measurements 
act as labels which make those values more useful in terms of details. 
For example, instead of saying that someone is tall, we can specify a 
measurement and specify that the individual is two meters tall. 

103 

 
 
 
Milestones 

Milestones express the intended progress towards each rural 

development priority. For rural development the intention is to define 
common performance indicators, based on output indicators, to 
establish the milestones for each priority. The Managing Authorities 
have to propose appropriate values for each of the relevant milestones, 
in relation to the measures, actions and resources programmed for each 
priority. 

Monitoring  

Observe and check the progress or quality of (something) over a period 

Objectives  

of time; keep under systematic review. The terms ‘monitoring’ and 
‘evaluation’ are used inter-changeably in this report to avoid the 
awkward juxtapositioning of words. 

A specific result that a person or system aims to achieve within a time 
frame and with available resources. In general, objectives are more 
specific and easier to measure than goals.   Objectives are basic tools 
that underlie all planning and strategic activities. They serve as the 
basis for creating policy and evaluating performance. 

Outcomes  

Determination and evaluation of the results of an activity, plan, process, 

or program and their comparison with the intended or projected results.  

Partnership  

Partnership 
agreement 

A partnership is an arrangement where parties agree to cooperate to 
advance their mutual interests. Partners in the understanding of the 
Common Strategic Framework are the parties involved in the 
Partnership Agreement concluded between the Member State and the 
Commission. According to the legal requirements, these partners shall 
comprise 
(a) competent regional, local, urban and other public authorities; 
(b)  economic and social partners; and  
(c)  bodies representing civil society, environmental partners, 

nongovernmental organisations, and bodies responsible for promoting 
equality and non- discrimination. 

Partnership Agreement means the document prepared by the Member 

State with the involvement of partners in line with the multi-level 
governance approach, which sets out the Member State's strategy, 
priorities and arrangements for using the CSF Funds in an effective and 
efficient way to pursue the Union strategy for smart, sustainable and 
inclusive growth, and which is approved by the Commission following 
assessment and dialogue with the Member State. 

Performance  

The accomplishment of a given task measured against preset known 

standards of accuracy, completeness, cost, and speed.   

Performance 
framework  

For each programme under the Common Strategic Framework, a 

performance framework shall be defined with a view to monitoring 
progress towards the objectives and targets set for each programme 
over the course of the programming period. The Commission should 
undertake a performance review in cooperation with the Member States. 
A performance reserve should be foreseen and allocated in 2019 where 
milestones set in the performance framework have been attained. 

Purpose  

The reason for which something is done or created or for which 

104 

something exists. 

 
 
 
 
 
Relevance 

Resources  

Risk 

assessment 

Relevance is the extent to which an intervention's objectives are pertinent 
to needs, problems and issues. Questions of relevance are particularly 
important in ex ante evaluation because the focus is on the strategy 
chosen or its justification. 

Economic or productive factor required to accomplish an activity, or as 
means to undertake an enterprise and achieve desired outcome. The 
two main resources investigated in this study are capital/funding and 
labour/personnel. 

Risk assessment is the determination of quantitative or qualitative value 

of risk related to a concrete situation and a recognized threat (also 
called hazard). In the case of the ex ante evaluation the risks to be 
assessed are related to (i) the possibility that a certain measure would 
not be applied and the corresponding funds not be used, e.g. due to the 
concomitant administrative burden (risk of inadequacy/inefficiency); or 
(ii) the possibility that a certain measure does not meet the needs of 
beneficiaries and/or does not help achieve the intended results (risk of 
irrelevance/ineffectiveness).  

Skills  

An ability and capacity acquired through deliberate, systematic, and 

sustained effort to smoothly and adaptively carryout complex activities 
or job functions involving ideas (cognitive skills), things (technical skills), 
and/or people (interpersonal skills).  

Social  

Of or relating to society or its organization 

Stakeholder 
mapping 

Stakeholder mapping is the comprehensive assessment and analysis of 

stakeholders relevant for a certain domain or issue (e.g. rural 
development, renewable energies, land use, etc.). The analysis 
concerns the stakeholders’ distinct interests, their relative weight (in 
respect to various criteria such as power, networking capacity, etc.) and 
the quality (intensity, harmony) characterizing their mutual relationships. 
From a pragmatic point of view, stakeholder mapping should not refer to 
a too large domain (as the complexity grows with the number of 
stakeholders and relationships). Stakeholder mapping is particularly 
revealing in facilitated dialogue settings (bringing together various 
stakeholders and their individual perspectives), and by using 
visualisation methods. 

Stakeholders  

A person, group or organization that has interest or concern in an 

organization.  In the context of this study examples of stakeholders are 
policymakers, organisations working with intermediaries and 
participants undertaking the initiatives provided by intermediaries. 

SWOT analysis   SWOT stands for Strengths, Weaknesses, Opportunities and Threats. 

The analysis of these four aspects has become the standard method for 
taking stock of the situation in an area, sector or theme and deciding on 
strategic priorities, objectives and measures. The SWOT should reflect 
evidence contained in the baseline and other indicators as well as more 
qualitative information. Ideally it should take into account stakeholder 
opinions. The strengths and weaknesses refer to the existing positive 
and negative attributes whereas the opportunities and threats to the 
future. 

Synergy 

Synergy denotes the fact that several public interventions (or several 
components of an intervention) together produce an impact which is 

105 

 
 
 
 
greater than the sum of the impacts they would produce alone (e.g. an 
intervention which finances the extension of an airport which, in turn, 
helps to fill tourist facilities, also financed by the intervention). Synergy 
generally refers to positive impacts. However, for phenomena which are 
contradictory, or reinforce negative effects, “negative synergy” or “anti- 
synergy” may also be used (e.g. an intervention subsidises the 
diversification of enterprises while a regional policy helps to strengthen 
the dominant activity). 

Target indicator  Target indicators should be linked, as directly as possible, to 

interventions, minimising the effect of external factors. They should be 
indicators which can be simply and regularly monitored, minimising the 
data requirements for beneficiaries and administrations, as the values of 
these indicators will be monitored regularly throughout the lifetime of an 
intervention. Wherever possible established indicators and methods 
should be used.  

Treatment  

Treatment is the term used in this study to describe the assistance, 

training or other activities provided by an intermediary to a participant at 
their initiative(s) 

Variables  

A characteristic, number, or quantity that increases or decreases over 

time, or takes different values in different situations. 

Workshop  

Training class or seminar in which the participants work individually 

and/or in groups to solve actual work related tasks to gain hands-on 
experience.   

106 

 
 
 
 
 
 
European Commission 

A Framework for monitoring the impact of 5G Public Private Partnership and 
the associated Key Performance Indicators (KPIs) – SMART 2015/0013 

Luxembourg, Publications Office of the European Union 

2017 - 107 pages  

ISBN 978-92-79-69011-2 

Doi:10.2759/79440

107 

 
 
 
 
 
 
 
 
K
K
-
0
1
-
1
7
-
5
2
0
-
E
N
-
N

108 

Doi:10.2759/79440                                                  ISBN 978-92-79-69011-2