A multi-dimensional framework to evaluate the innovation potential of digital public services : a step towards building an innovative public services observatory in the EU.
Date: 2020-09-14
Institutions: Joint Research Centre
Subjects: Internet of Things, artificial intelligence, communications policy, digital single market, electronic government, information network, information security, information society, information technology, innovation, new technology, research report

A multi-dimensional framework to 
evaluate the innovation potential of 
digital public services 

A step towards building an Innovative 
Public Services Observatory in the EU 

Authors: Ilenia Bruno, Anna Schiavone Panni, 
Valeria Marchetti, Francesco Molinari, Beatrice 
Valente Covino 

Editors: Gianluca Misuraca, Georges Lobo 

2020 

EUR 30355 EN 

 
 
 
 
 
 
 
This  publication  is  a  Technical  report by  the  Joint  Research  Centre  (JRC),  the  European  Commission’s  science 
and  knowledge  service.  It  aims  to  provide  evidence-based  scientific  support  to  the  European  policymaking 
process. The scientific output expressed does not imply a policy position of the European Commission. Neither 
the  European  Commission  nor  any  person  acting  on  behalf  of  the  Commission  is  responsible  for  the  use  that 
might be made of this publication. For information on the methodology and quality underlying the data used in 
this  publication  for  which  the  source  is  neither  Eurostat  nor  other  Commission  services,  users  should  contact 
the referenced source. The designations  employed and the presentation of material on the maps do not imply 
the expression of any opinion whatsoever on the part of the European Union concerning the legal status of any 
country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. 

Contact information  
Francesco Pignatelli 
Via Enrico Fermi, 2749 
21027 Ispra (VA), Italy 
francesco.pignatelli@ec.europa.eu 

EU Science Hub 
https://ec.europa.eu/jrc  

JRC121672 

EUR 30355 EN 

PDF 

ISBN 978-92-76-21455-7 

ISSN 1831-9424 

doi:10.2760/09628 

Luxembourg: Publications Office of the European Union, 2020  

© European Union, 2020  

The reuse policy of the European Commission is implemented by the Commission Decision 2011/833/EU of 12 
December 2011 on the reuse of Commission documents (OJ L 330, 14.12.2011, p. 39). Except otherwise noted, 
the reuse of this document is authorised under the Creative Commons Attribution 4.0 International (CC BY 4.0) 
licence (https://creativecommons.org/licenses/by/4.0/). This means that reuse is allowed provided appropriate 
credit is given and any changes are indicated. For any use or reproduction of photos or other material that is 
not owned by the EU, permission must be sought directly from the copyright holders. 

All content © European Union, 2020, except: Cover image #215481491, 2020. Source: stock.adobe.com 

How to cite this report: Misuraca, G., Lobo, G., (Eds.), Bruno I., Schiavone Panni A., Marchetti V., Molinari 
F., Valente Covino B., A multi-dimensional framework to evaluate the innovation potential of digital public 
services - A step towards building an Innovative Public Services Observatory in the EU, EUR 30355 EN, 
Publications Office of the European Union, Luxembourg, 2020, ISBN 978-92-76-21455-7, doi:10.2760/09628, 
JRC121672. 

The information and views set out in this report are those of the authors and do not necessarily reflect the 
official opinion of the Commission. The Commission does not guarantee the accuracy of the data included in this 
study. Neither the Commission nor any person acting on the Commission’s behalf may be held responsible for 
the use which may be made of the information contained therein. 

 
 
 
 
 
 
 
 
 
 
 
 
 
Contents 

Foreword .............................................................................................................. 2 

Acknowledgements ................................................................................................ 4 

Executive Summary ............................................................................................... 6 

1  Introduction ...................................................................................................... 8 

2  Identification of relevant technologies ................................................................ 12 

3  Interoperability issues and user centricity aspects ............................................... 16 

3.1  Interoperability assessment method ............................................................. 16 

3.2  User centricity assessment method .............................................................. 17 

3.3  Technology assessment results .................................................................... 19 

3.3.1  Artificial Intelligence ........................................................................... 19 

3.3.2  Communication Technologies .............................................................. 22 

3.3.3  Computing Infrastructures .................................................................. 25 

3.3.4  Digital Identity and Security ................................................................ 27 

3.3.5  Distributed Ledger Technologies .......................................................... 30 

3.3.6  Immersive Technologies ..................................................................... 33 

3.3.7  Internet of Things and Smart Devices ................................................... 35 

3.3.8  Software and Service Technologies ...................................................... 37 

3.4  Final considerations on the assessment of technologies .................................. 39 

4  Readiness concepts and levels ........................................................................... 42 

4.1  Technology Readiness Level ........................................................................ 42 

4.2  Societal Readiness Level ............................................................................. 43 

4.3  Organisational Readiness Level .................................................................... 44 

4.4  Legal Readiness Level ................................................................................ 45 

4.5  Application to pilot evaluation...................................................................... 46 

5  Knowledge repository ....................................................................................... 48 

5.1  Metadata .................................................................................................. 48 

5.2  Structure .................................................................................................. 51 

6  Pilot analysis and evaluation ............................................................................. 55 

7  Pilot shortlisting .............................................................................................. 60 

7.1  Dublinked ................................................................................................. 60 

7.2  IO App ..................................................................................................... 66 

8  Final considerations and outlook ........................................................................ 71 

List of Tables ...................................................................................................... 75 

List of Figures ..................................................................................................... 76 

i 

 
 
 
Foreword 

This  report  presents  the  results  of  a  study  conducted  under  the  action  2018.01 
“Innovative Public Services” (IPS) of the ISA² Programme1 - Interoperability solutions for 
Public  Administrations,  businesses  and  citizens.  The  programme,  coordinated  by  the 
Directorate  General  of  Informatics  (DIGIT)  has,  among  others,  the  purpose  to  support 
the digital transformation of the EU public sector, facilitating the deployment and use of 
digital  technologies,  by  proposing  concrete  interoperable  solutions  and  identifying  best 
practices to facilitate cross-border and cross-domain interoperability. 

More  specifically,  the  IPS  action  has  the  objective  to  assess  the  role  that  new  and  less 
new  digital  technologies  can  play  in  the  transformation  of  Public  Services  and  propose 
interoperable innovative solutions and support their piloting.  

The  aim  of  this  study  was  to  conduct  a  desk  and  field  research  on  available  evidence 
supporting  European  Public  Administrations  willing  to  embrace  new  digital  technologies 
and  deliver  innovative  public  services  according  to  the  4  layers  of  the  European 
Interoperability  Framework  (EIF)  and  in  alignment  with  the  user  centricity  principles 
defined  in  the  Tallinn  Declaration  in  2017,  to  ensure  that  adoption  of  new  technologies 
do not lead to creation of new silos. 

The main outcome of the study is an original multi-dimensional framework for evaluating 
digital  readiness,  interoperability,  and  user-centricity  of  innovative  public  services.  The 
framework  was  conceptualised  and  tested  in  the  context  of  the  research  funded  by 
DIGIT, supervised in collaboration with the JRC and conducted by KPMG Italy. 

The results of the research resonate well with the core principles of the ISA² Programme, 
which is in itself one of the instruments to support Digital Government Transformation in 
the  EU,  through  stimulating  cross-fertilisation  among  Member  States,  promoting 
exchange of practices and sharing of knowledge and digital solutions. 

In this regard, it is worth noticing that the idea of the study originated when we met in 
June  2018  in  Brussels  at  the  “Future  in  the  making”  conference  of  the  Future-Oriented 
Technology  Analysis  (FTA)  series  organised  by  JRC.  Two  years  later,  we  can  say  that 
after that discussion we have completed the first step of the future that we planned.  

The  results  of  the  work  initiated  with  this  study  were  instrumental  to  formalise  a 
strengthened collaboration between DIGIT and JRC, as part of the IPS Action of the ISA2 
Programme  now  co-led  also  with  CONNECT  colleagues,  and  somehow  ‘upgraded’  to 
become a tool to support the development of the future Digital Government policy.  

In  its  communication  “Shaping  Europe’s  Digital  Future2,  the  Commission  calls  for  a 
reinforced EU governments interoperability strategy  and a strengthened EIF for the end 
of  2021.  In  this  way,  the  orientation  outlined  in  the  Digital  Europe  Programme  (DEP) 
have  confirmed  interoperability  as  a  crucial  aspect  to  be  addressed,  not  only  from  a 
technical  perspective,  rather  giving  a  more  prominent  role  to  the  inherent  governance 
dimensions  of  public  sector  innovation  and  to  the  public  values  generated  by  digital 
public services. 

As the findings from this study show, not only organisational and legal aspects need to 
support  the  digital  transformation,  but  they  must  be  in  its  core  as  an  essential  part  of 
the next EU Governments Interoperability Strategy, to guarantee adoption of innovative 
public services and societal impact.  

1 https://ec.europa.eu/isa2/isa2_en 
2 https://ec.europa.eu/digital-single-market/en/content/european-digital-strategy 

2 

 
 
                                           
In fact, based on the results of this research, even the most promising technologies still 
present several challenges, in terms of technological readiness, interoperability and user 
centricity.  EU  governments  at  all  levels  need  to  invest  significantly  to  remove  existing 
barriers, in order to provide seamless access  to public services in line with citizens and 
businesses expectations.  

The framework proposed was outlined to contribute as a first step to develop a practical 
instrument to monitor and assess the multiple dimensions of technological readiness and 
interoperability  of  innovative  digital  public  services  and  respond  to  the  user-centricity 
challenges faced when implementing Digital Government across the EU.  

To  this  end,  it  paved  the  way  to  enhanced  cooperation  between  DIGIT  and  JRC,  which 
resulted  in  the  establishment  of  an  Administrative  Arrangement  to  conduct  a  feasibility 
study to set up an EU Innovative Public Services Observatory (IPSO).  

This exploratory research, implemented under the IPS Action of the ISA2 Programme for 
the  period  2020-2021,  aims  at  assessing  the  potential  for  formulating  an  initiative  at 
European level to support the innovation of public services with the help of emerging and 
disruptive technologies.  

The  framework,  proposed  in  this  report  and  tested  against  a  number  of  concrete  cases 
and promising pilots in EU Member States, should thus be considered a preparatory work 
to  further  develop  the  IPS  Observatory  and  assess  the  innovation  potential  of  digital 
services  in  terms  of  their  comprehensive  technological  readiness,  interoperability  and 
user-centricity dimensions.  

It is also complementary to other ways of measuring the progress of EU Member States 
towards  the  full  digitalisation  of  public  services,  such  as  the  yearly  eGovernment 
Benchmark, which monitors and rates the maturity of online public services in terms of 
user centricity, transparency, and other key enablers.  

Future research to be conducted within the context of the work stream of the IPS Action 
coordinated  by  JRC  and  DIGIT  should  thus  focus,  in  collaboration  with  CONNECT  and 
other relevant policy DGs, on further increasing the knowledge base that is being created 
and make sure that it could be easily used and constantly updated.  

In  this  perspective,  it  is  however  pivotal  to  preserve  the  methodological  rigour  of  the 
proposed framework, which should be the basis for developing technical and operational 
guidelines  for  Innovative  Public  Services  in  the  EU  and,  in  turn,  contribute  advance 
theory and practice of Digital Government Transformation and Public Sector Innovation. 

In  doing  so,  when  we  meet  at  the  next  FTA  conference,  we  hope  the  IPS  Observatory 
will be operational and digital by default, interoperable cross-border seamless innovative 
public services will be a closer reality and at all levels of governance across Europe. 

Gianluca Misuraca and Georges Lobo 

3 

 
 
 
 
 
 
 
Acknowledgements 

This  report  summarises  the  key  actionable  results  of  a  Study  conducted  by  KPMG  Italy 
for  the  European  Commission’s  DIGIT  in  collaboration  with  JRC  in  execution  of  a 
Framework  Contract  part  of  the  ISA²  Programme  -  Action  2018.01  “Innovative  Public 
Services” (IPS).  

The  authors  and  editors  of  this  report  are  grateful  to  the  experts  and  colleagues  who 
provided  input  to  the  Study  deliverables,  in  particular  Yannis  Charalabidis,  who 
supported  us  on  the  taxonomy  and  analysis  of  technologies  and  interoperability  issues, 
Elio De Tullio, who focused on legal aspects of pilots assessment, Cristina Cosma, policy 
officer at DIGIT, and Andrea Perego, former Scientific Officer at JRC, for their feedback.  

We also would like to thank other KPMG colleagues for their contribution on research and 
analytical activities and the vertical expertise provided on specific topics and in particular 
Pierluigi  Verbo,  Head  of  Government  and  Public  Sector  at  KPMG  Italy,  for  his  precious 
support and supervision. 

We  also  appreciated  the  cooperation  of  the  Smart  Dublin  Project  Coordinator  who 
provided  us  with  significant  information  on  their  initiative,  contributing  to  deepen  the 
analysis  of  the  Dublinked  project,  as  well  as  the  input  provided  by  the  Italian  Digital 
Transformation Team in the analysis of the IO App project. 

Moreover,  we  would  like  to  thank  all  the  participants  in  the  stimulating  IPS  Online 
Workshop  which  took  place  on  April  7th,  2020  who  showed  interest  in  the  Study  and 
supported  our  work  in  the  finalisation  of  the  report,  as  well  as  Peter  Ulrich,  who 
organised the event as part of the technical implementation of the IPS AA between JRC 
and DIGIT. 

Finally, we would like to express our special thanks to Natalia Aristimuño-Pérez, Head of 
the  Interoperability  Unit  and  Alessandro  Annoni,  former  Head  of  the  Digital  Economy 
Unit, for their guidance in the setting up of the follow-up activities of this study, that we 
expect will lead to the establishment of the EU Innovative Public Services Observatory.  

4 

 
 
 
 
Authors 

Ilenia Bruno (KPMG) 

Anna Schiavone Panni (KPMG) 

Valeria Marchetti (KPMG) 

Francesco Molinari (Senior External Expert) 

Beatrice Valente Covino (KPMG) 

Editors 

Gianluca Misuraca (JRC) 

Georges Lobo (DIGIT) 

5 

 
 
 
 
Executive Summary 

Nowhere  like  in  the  European  Union  is  digital  transformation  playing  a  decisive  role  in 
the  present  and  future  evolution  of  our  economy  and  society.  5  years  before  the 
outbreak  of  the  Covid-19  pandemic,  the  intuition  that  connecting  people,  communities, 
businesses and governments by the use of Information and Communication Technologies 
(ICTs)  would  help  create  and  reinforce  a  supranational  sociocultural  system  and 
therefore  strengthen  the  feeling  of  European  identity  was  already  evident  in  the 
European Commission priorities. 

A digital single market for online transactions would constitute an invaluable opportunity 
for  new  and  existing  enterprises  to  grow  and  boost  the  generation  of  more  numerous 
quality jobs. Citizens could and should be empowered in their capacity and skills to take 
full benefit of the Information  Society, including by seamless access to news and social 
media  on  their  mobile  and  smart  devices,  irrelevant  of  the  country  of  residence. 
Government  bodies  at  all  levels  (from  local  to  regional  and  national)  in  all  Member 
States were and are asked to digitalise existing public services and create new, natively 
online, facilities to increase their perceived quality, openness, diffusion and acceptance. 
In addition to interoperability and user centricity, a peculiar metric adopted since long in 
the  EU  eGovernment  benchmark  is  “cross-border  service  delivery”,  which  again  clearly 
points  at  a  future  where  location  will  no  longer  be  determinant  for  the  exercise  of 
citizens’ rights.  

Indeed, in a  well-functioning  digitally transformed economy and society, public services 
at all levels need to be able to ‘follow’ their users’ needs across administrative entities, 
policy  sectors  and  country  borders.  More  generally,  digital  transformation  of  the  public 
sector is as important for renewing its relationship with economic and societal players as 
it  is  for  the  improvement  of  the  efficiency  and  effectiveness  of  internal  policy, 
governance  and  service  delivery  processes.  However,  recent  research  conducted  at  the 
JRC  (Misuraca,  Ed.,  2019)  shows  that  a  big  gap  still  exists  between  expectations  and 
achievements. Not by chance, digitalisation remains a cornerstone of the new European 
Commission’s  5-year  strategy,  which  the  Covid-19  global  crisis  has  only  added  more 
reasons to. In this context, further support needs to be given to the widespread adoption 
and  implementation  of  digital  solutions  in  the  local,  regional  and  national  Public 
Administration. 

A  typical  approach  followed  by  government  bodies  and  agencies  innovating  their 
processes  is  built  on  the  execution  of  pilot  projects.  These  are  small-scale  experiments 
meant  to  reduce  the  costs  and  risks  inherent  to  a  diffused  introduction  of  new 
technologies  or  other  elements  of  transformation  of  the  “machinery”  of  Public 
Administration,  its  functioning  and  input  and  output.  By  definition,  pilot  projects  are 
more likely to be a failure, rather than a success. Whatever the outcomes, their “owners” 
–  responsible  persons  of  the  execution  and  sometimes  evaluation  of  pilot  results  –  are 
expected to learn useful lessons from them, which will possibly be shared with a broader 
audience on due time and are supposed to be useful (and used) internally to support the 
decision  of  whether  and  how  to  follow  up.  Most  of  these  pilot  projects  are  financially 
supported by third party grants, such as from the EU research, innovation, or territorial 
cooperation  programmes.  Such  circumstance  does  not  only  contribute  to  further 
reducing  the  costs  (and  to  some  extent  the  risks)  of  technology  introduction,  but  also 
provides the best guarantee that pilot results will be shared widely on the occasion of the 
periodic and final reports to the funding agencies. 

Unfortunately, and as the aforementioned JRC research also documented, the propensity 
of  pilot  owners  to  share  results  and  lessons  learned  in  a  structured  manner  is  quite 
limited and this adds noise to the evaluation of scalability potentials, not to mention the 
difficulty  of  defining  reliable  reuse  or  transfer  pathways  involving  other  public  sector 
organisations  than  those  in  the  original  pilots.  This  lack  of  information  is  particularly 

6 

 
undesired  in  case  of  new  and  emerging  technologies,  such  as  blockchain  or  Artificial 
Intelligence, which naturally lend themselves to being trialled in very similar – yet never 
too  much  –  pilot  contexts,  thus  increasing  the  risk  of  “reinventing  the  wheel”  –  i.e.  of 
duplication, if not proliferation, of limited size experiments. 

To  that  end,  the  present  study  started  with  identifying  8  thematic  clusters  of  relevant 
digital  technologies,  both  mature  and  emerging,  for  the  European  public  sector  and 
examining  their  potentialities  and  constraints  in  terms  of  service  delivery,  governance 
and policy making by focusing on  key interoperability and  user centricity aspects. More 
specifically,  scores  were  assigned  to  each  technology  cluster  on  the  most  significant 
interoperability  issues  and  risks  to  argue  about  the  presence  of  current  or  potential 
barriers  preventing  the  full  achievement  of  interoperability  targets.  Similarly,  for  each 
identified  cluster  of  technologies,  we  also  rated  and  commented  the  level  of  adherence 
to the 8 user centricity principles of the Tallinn Declaration.   

Then,  the  Study  team  moved  to  create  a  broad  collection  of  150+  meaningful  pilot 
projects  from  24  European  countries,  covering  all  the  8  technology  areas  identified 
previously.  As  expected,  many  pilots  pertained  to  EU  funded  initiatives,  such  as  H2020 
and territorial cooperation. The collection was intended to showcase the innovative public 
services that are  currently being  developed and/or tested across the  EU harnessing the 
technologies previously identified. Around 80 pilots were selected for deeper analysis.  

To carry out the assessment in a replicable fashion, we adopted a dynamically oriented 
and semi-quantitative approach, based on measuring the “transition pathways” that the 
pilots  experienced  across  time  (from  t0,  before  the  pilot  start,  to  t1,  at  the  pilot  end), 
under 4 distinct “readiness” dimensions: technological, societal, organisational and legal, 
as  well  as  to  assess  their  attention  to  user-centricity  principles  and  interoperability 
aspects. The original aspect of this exercise has been to highlight the crucial importance 
of all these aspects for the analysis of pilot results, contrary to the conventional wisdom 
giving differentiated weights – if not only a partial consideration – to those less directly 
related to the goal of technology trialling. In fact, it turned out that 31 innovative public 
service initiatives could be defined “successful”, as they  presented high and convergent 
scores  (around  7-8  on  a  scale  of  9)  for  all  4  readiness  levels  at  the  end  of  the  testing 
phase. Additionally, those initiatives showed good interoperability levels (2-3 on a scale 
of 4) and satisfied the majority (5 out of 8) of user centricity principles. 

Finally,  the  research  team  identified  two  promising  pilots  to  be  studied  more  in  depth, 
namely IO App and Dublinked. These  were  further analysed  in cooperation with project 
owners and preliminary advices for possible replication at EU level were outlined. 

Globally, the multi-dimensional framework presented in this report proved to be a useful 
tool to evaluate the success or failure of public service digitalisation projects. Looking at 
their evolution over time, it was possible to conclude that the pilots showing low scores 
in the 4 readiness levels at the beginning of their activities and capable of reaching high 
scores  at  the  end,  were  those  most  likely  to  produce  the  greatest  impacts  on 
government  modernisation  processes.  Conversely,  the  absence  of  a  strict  correlation 
among  the  trends  of  improvement  of  these  4  dimensions  quite  often  led  to  classify  the 
pilot  as  a  failure.  This  means  that  at  the  end  of  the  experimentation,  significant  gaps 
were  affecting  either  the  technological,  societal,  organisational  or  legal  aspects  of  the 
testing environment, in such a way that could not be neglected. 

We can conclude that this framework is a good complement to other ways of measuring 
the progress of EU Member States towards the full digitalisation of public services, such 
as the yearly eGovernment Benchmark, which monitors and rates the maturity of online 
public services in terms of user centricity, transparency, and other key enablers. Further 
research  activities  should  be  undertaken  to  expand  the  size  of  the  current  knowledge 
base and enhance the use of the framework to predict the scaling up potential of newly 
started innovative projects, based on the experience gained by looking into the existing 
pilots with the aim of building an Innovative Public Services Observatory in the EU. 

7 

 
1  Introduction 

Digitalisation is widely considered as a major driver of innovation, growth, modernisation 
and societal progress. It is a transversal phenomenon, affecting all areas of the economy 
and  communitarian  life,  including,  but  not  limited  to,  the  public  sector.  Technically 
speaking,  digitalisation  is  other  than  digitisation,  although  the  two  terms  are  often 
confused or used interchangeably. In fact, digitisation means  the conversion of existing 
information formats from analogue to machine readable3, a trend that has been with us 
since the late 20th century. Instead, digitalisation – quite often also referred to as digital 
transformation  –  is  a  more  modern  phenomenon,  alluding  to  a  change  in  the  “way  of 
thinking”4, even before actually “going digital”, i.e. starting to use digital technologies to 
innovate how products or services are  designed, manufactured, delivered or consumed. 
No surprise then that the younger generations are usually pointed at as “digital natives”5 
and that digitalisation of the communication spaces of European citizens has been often 
mentioned  as  the  embryo  of  a  (still  missing)  single  European  public  sphere,  cutting 
across  the  differences  in  cultures,  languages,  legislations  and  social  organisations 
between Member States. 

As  predicted  in  the  Europe  2020  strategy,  a  digital  single  market  for  online 
transactions would constitute an invaluable opportunity for new and existing enterprises 
to  grow  and  boost  the  generation  of  more  numerous  quality  jobs.  European  citizens 
could  and  should  be  empowered  in  their  capacity  and  skills  to  take  full  benefit  of  the 
information  society,  including  by  seamless  access  to  news  and  social  media  on  their 
mobile and smart devices, irrelevant of the country of residence. Government bodies at 
all levels (from local to regional and national) in all Member States were and are asked 
to digitalise existing public services and create new, natively online, facilities to increase 
to 
their  perceived  quality,  openness,  diffusion  and  acceptance. 
interoperability  and  user  centricity,  a  peculiar  metric  adopted  since  long  in  the  EU 
eGovernment  benchmark6  is  “cross-border  service  delivery”,  which  again  clearly 
points  at  a  future  where  location  will  no  longer  be  determinant  for  the  exercise  of 
European citizens’ rights. 

In  addition 

The  Tallinn  Declaration  on  eGovernment,  signed  in  2017,  has  been  an  important 
step in that direction: all Member States officially endorsed “the digital transformation of 
the  Public  Administration”  as  their  “collective  endeavour  at  national,  regional  and  local 
levels”  with  the  ultimate  aim  of  putting  “the  end-users  –  citizens,  businesses,  public 
sector  employees  –  truly  at  the  centre  of  services.”  But  there  is  more:  digitalisation  of 
public services  ensures  noteworthy savings of financial resources, giving new meanings 
to the well-known definitions of efficiency, effectiveness, and quality of government. 

Another key element to take into consideration in the development and delivery of digital 
the  new  European 
public  services 
Interoperability Framework (EIF) as “the ability of Public Administrations, any entity 
acting on their behalf or EU institutions or bodies, to interact towards mutually beneficial 
goals, involving the sharing of information and knowledge between these organisations, 

interoperability.  This 

is  defined  by 

is 

3 See https://en.wikipedia.org/wiki/Digitization  
4 Marek Kowalkiewicz, The transformational difference between ditisation and digitalisation, Medium.com post, 
27 October 2017. Retrieved online at: https://medium.com/qut-cde/digitise-or-digitalise-584c953e2d8  
5  Marc  Prensky,  Digital  Natives,  Digital  Immigrants,  article  originally  published  in  2001.  Retrieved  online  at: 
http://www.marcprensky.com/writing/Prensky_-_Digital_Natives,_Digital_Immigrants_-_Part1.pdf  
6 https://op.europa.eu/en/publication-detail/-/publication/c896937b-f554-11e9-8c1f-01aa75ed71a1  

8 

 
                                           
through  the  business  processes  they  support,  by  means  of  the  exchange  of  data 
between their ICT systems.”7   

Indeed, in a well-functioning digitally transformed economy and society, public services 
at all levels need to be able to ‘follow’ their users’ needs across administrative entities, 
policy  sectors  and  country  borders.  More  generally,  digital  transformation  of  the  public 
sector is as important for renewing its relationship with economic and societal players as 
it  is  for  the  improvement  of  the  efficiency  and  effectiveness  of  internal  policy, 
governance  and  service  delivery  processes.  However,  recent  research  conducted  at  the 
JRC8 shows that a big gap still exists between expectations and achievements.  

Not  by  chance,  digitalisation  remains  a  cornerstone  of  the  new  European  Commission’s 
5-year  strategy,  which  the  Covid-19  global  crisis  has  only  added  more  reasons  to. 
Already in her first 100 days of mandate, President Ursula von der Leyen has given new 
momentum  to  Shaping  Europe’s  Digital  Future9  through  a  pathbreaking  White 
Paper  on  Artificial  Intelligence10  and  a  new  European  strategy  for  data11,  both 
aimed to build “A Europe fit for the digital Age”, making digital transformation work for 
all  citizens  and  businesses,  while  helping  to  achieve  the  target  of  a  climate-neutral 
Europe by 2050. 

In  this  evolving  context,  further  support  needs  to  be  given  to  the  widespread  adoption 
and  implementation  of  digital  solutions  in  the  local,  regional  and  national  Public 
Administration. 

In  fact,  as  shown  in  the  2020  Digital  Economy  and  Society  Index  for  e-
Government  (DESI)12,  based  on  2019  data  and  assessing  the  status  of  the  digital 
economy and society prior to the outbreak of the pandemic,  both the quality and usage 
of  government  services  are  increasingly  progressing:  today  67%  of  EU  citizens  use 
public  services  online.  However,  differences  between  countries  still  persist:  top 
performers  such  as  Finland,  Sweden,  Netherlands  and  Malta  are  making  the  most  of 
digitalisation whereas countries like Bulgaria and Greece seem to be trailing behind. The 
situation  is  even  more  challenging  when  it  comes  to  availability  and  usability  of  cross-
border  services  for  businesses  and  the  general  public,  since  several  barriers  are  still 
present,  hampering  the  free  mobility  of  data  and  people  across  Europe  and  the 
ultimately the full realisation of the Digital Single Market.  

As stressed in the EIF, interoperability plays an important role in digital transformation, 
as it allows Public Administrations to digitally exchange data with unambiguous,  shared 
meaning  amongst  themselves  and  with  citizens  and  businesses.  To  further  improve 
public service delivery and link similar services across Europe, digital solutions should be 
carefully designed considering four interoperability layers: legal, organisational, semantic 
and technical. These layers are meant to significantly affect digital public service delivery 
in  the  EU,  from  the  preparation  of  the  legislation  to  the  organisation  of  Public 

Shaping 

7  New  European Interoperability Framework.  Promoting  seamless  services  and  data flows for  European public 
administrations. Retrieved online at https://ec.europa.eu/isa2/sites/isa/files/eif_brochure_final.pdf 
8 Barcevičius, E., Cibaitė, G., Codagnone, C., Gineikytė, V., Klimavičiūtė, L., Liva, G., Matulevič, L., Misuraca, 
G.,  Vanini,  I.,  Editor:  Misuraca,  G.,  Exploring  Digital  Government  transformation  in  the  EU  -  Analysis  of  the 
state  of  the  art  and  review  of  literature,  EUR  29987  EN,  Publications  Office  of  the  European  Union, 
Luxembourg, 2019, ISBN 978-92-76-13299-8, doi:10.2760/17207, JRC118857. 
9 
final. 
https://ec.europa.eu/info/sites/info/files/communication-shaping-europes-digital-future-feb2020_en_4.pdf 
10  White  Paper  on  Artificial  Intelligence  -  A  European  approach  to  excellence  and  trust,  COM(2020)  65  final. 
Retrieved  online  at  https://ec.europa.eu/info/sites/info/files/commission-white-paper-artificial-intelligence-
feb2020_en.pdf  
11 
https://ec.europa.eu/info/sites/info/files/communication-european-strategy-data-19feb2020_en.pdf  
12 https://ec.europa.eu/newsroom/dae/document.cfm?doc_id=67086  

COM(2020) 

COM(2020) 

Retrieved 

Retrieved 

European 

Europe’s 

strategy 

future, 

digital 

online 

online 

data, 

final. 

for 

66 

67 

at 

at 

A 

9 

 
                                           
Administration processes and to the development of  IT  systems for the implementation 
of public services.   

With the recommendations provided by the EIF, EU countries can follow a common and 
practical  approach  when  they  introduce  a  public  service  in  order  to  make  it  accessible, 
not  only  within  national  borders,  but  also  across  countries,  creating  a  favourable 
environment where Public Administrations can collaborate digitally.  

Since 1999, the European  Commission in accordance with the Member States  has been 
stimulating  in  different  ways  the  interchange  of  data  between  administrations.  Among 
the key initiatives, it is worth to mention the  eGovernment Action Plan 2016-2020, the 
eHealth Action Plan (2012-2020) and the Connecting Europe Facility (CEF). An important 
initiative supporting the EIF is the ISA2 Programme13 which is running since 2016. The 
Programme promotes the development of digital solutions to be made generally available 
for free to European Public Administrations, businesses and citizens in order for them to 
take benefit from interoperable, cross-border and cross-sector public services. The ISA2 
Programme  includes  several  coordinated  activities  at  EU  level  to  develop  the  necessary 
instruments  to  boost  interoperability  such  as:  a  revised  European  Interoperability 
Framework  (EIF);  a  revised  European  Interoperability  Strategy  (EIS);  the  European 
Interoperability  Reference  Architecture  (EIRA);  and  the  European  Interoperability 
Cartography  (EIC).  More  recently,  the  Digital  Europe  Programme  (2021-2027)  has 
been 
its  objectives  to  promote  the  digital 
transformation of public services and their EU-wide interoperability.  

launched,  which 

includes  among 

The Study reported about in this publication was part of the Action 2018.01 - Innovative 
Public  Services  (IPS)  of  the  ISA2  Programme,  which  includes  several  ongoing  activities 
both inside and outside ISA2, as depicted in Figure 1.   

Source: DIGIT 

Figure 1 - Overview of the IPS Action and related activities 

The  IPS  Study  rests  under  the  first  group  of  activities  (“General  landscaping  and 
studies”) and had four specific objectives: 

13 https://ec.europa.eu/isa2/isa2_en 

10 

 
 
                                           
1.  To  identify  key  technologies  -  both  mature  and  emerging  in  the  public  sector  - 
and assess their potential for the design and implementation of innovative public 
services; 

2.  To evaluate the interoperability issues and user centricity aspects related to those 

technologies; 

3.  To  collect  evidence  in  terms  of  IPS  pilots  and  the  conditions  for  their  success  or 

failure;  

4.  To  identify  the  most  promising  pilots,  as  successful  examples  of  digital 
transformation  of  public  services,  to  be  shared  and  reused  across  Europe,  also 
identifying the required activities for their further improvement. 

The structure of this report follows the track of the desk and field research work carried 
out by the IPS Study team, which can be outlined as follows:  

  Chapter 2: Identification of relevant technologies  

This section presents the eight thematic clusters gathering  relevant  technologies for 
the modernisation of European public services.  

  Chapter 3: Interoperability issues and user centricity aspects 

This  section  presents  the  methods  used  to  assess  the  identified  eight  technology 
areas  according  to  their  interoperability  issues/risks  and  main  user  centricity 
challenges.  It  also  overviews  the  results  obtained  from  the  application  of  the 
proposed methods to the technologies at hand. 

  Chapter 4: Readiness concepts and levels   

This section describes the four readiness concepts and levels used to assess the IPS 
pilot projects under the technological, societal, organisational and legal aspects. 

  Chapter 5: Knowledge repository  

This  section  overviews  the  MS  Excel  database  conceived  of  and  designed  to  make 
sure  that  the  results  of  data  collection  could  be  easily  used,  shared  and  constantly 
updated, also by future studies, and  could serve as a basis for the  realisation of an 
Innovative  Public  Services  Observatory 
the 
methodological lessons learned in the form of technical and operational guidelines.  

the  EU,  while  preserving 

in 

  Chapter 6: Pilot analysis and evaluation  

In this section the process followed for the analysis, the evaluation of each IPS pilot 
is  described,  and  the  main  results  are  outlined.  In  particular,  the  criteria  used  to 
define a pilot as successful are highlighted. 

  Chapter 7: Pilot shortlisting 

This section overviews the two pilots – Dublinked and IO App – shortlisted for further 
analysis in terms of readiness, interoperability and user centricity aspects. 

  Chapter 8: Final considerations and outlook  

This  section  discusses  the  main  findings  of  the  Study  and  draws  conclusions  and 
recommendations for future work. 

11 

 
2  Identification of relevant technologies 

The objective of the first Study activity was to identify the potentiality of emerging and 
mature  technologies  in the  public  sector  as  relevant  for  the  design  and  implementation 
of digital public service provision.  

To reach that objective, an extensive analysis was carried out of: 

  mature  technologies  i.e.  already  existing  in  the  state  of  the  art,  irrelevant  to 

whether they had been trialled in the public sector before or not; 

  emerging technologies under the condition that they had already been trialled 
in  a  public  sector  environment,  so  that  their  value  could  be  inferred  for  the 
provision of innovative public services. 

The  following figure is a high-level categorisation of the technologies  that  have been in 
the scope of this analysis. 

Source: authors’ elaboration 

Figure 2 - Analysed technologies 

Basically, the idea was to answer the following business questions:  

  BQ1:  In  the  current  landscape  of  technologies,  which  ones  have  been  already 

applied to public service delivery?  

  BQ2:  Narrowing  the  focus  on  mature  technologies,  which  are  those  potentially 

applicable to public service delivery?  

The  analysis  started  from  an  initial  list  of  12  technology  areas.  By  technology  area  it 
was meant “the extent of a determined technology enclosed within a specified boundary 
that can be subdivided into categories and uses/applications”.  

By  technology  category  it  was  meant  “an  exhaustive  division  of  the  technology  area 
according  to  the  proposed  system  of  classification”.  Finally,  uses  and  applications 
consisted in the “act of applying a technology to a particular purpose or use in real life”.  

The 12 areas have been thoroughly discussed with the European Commission officials in 
charge of supervising the Study, leading to a reduction in the initial number to 8 and a 
better articulation of the internal components of each area, not only including categories, 
but also some (exemplary, non-exhaustive) related uses and applications. 

The result of this endeavour, which is not intended to be a holistic taxonomy, is shown in 
Table 1.  

12 

Mature TechnologiesEmergingTechnologiesNevertrialledbeforein Public SectorAlreadytrialledin Public Sector 
 
Technology 
Areas 

1. Artificial 
Intelligence 

Technology Categories 

Technology Uses/Applications 
(exemplary, not exhaustive) 

1.1 Robotic Process Automation 

1.1 ERP System Reengineering; Smart 
Assistance 

1.2 Natural Language 
Processing, Text Mining, 
Computer Vision 

1.2 Advanced Simulation; Autonomous 
Vehicles; Evidence based Decision Making; 
Intelligence Agents, Bots; Sentiment 
Analysis; Virtual Assistance 

1.3 Machine Learning, Deep 
Learning 

1.3 Digital Twins; Evidence based Decision 
Making; Intelligent Agents, Bots 

1.4 Expert and Rule Based 
Systems 

1.4 Operational Decisions in PA 

1.5 Cognitive Computing, 
Predictive Analytics 

1.5 Big Data Analytics; E-Procurement; 
Government Efficiency 

2. 
Communication 
Technologies 

2.1 5G Networks and Handheld 
Devices 

2.1 Big Data Streaming; HQ Multimedia 
(training UX); Real-time Data Processing 

2.2 Software Defined Networks 

2.2 Programmable Networks; Network 
Visualisation 

3. Computing 
Infrastructures 

3.1 High Performance 
Computing 

3.1 Advanced Simulations and Visualisation 

3.2 Cloud Computing 

3.3 Edge Computing 

4. Distributed 
Ledger 
Technologies 

4.1 Blockchain 

3.2 Backup; Big Data Processing; Disaster 
Recovery; File Storage; Real-time Data 
Processing; Test and Development 

3.3 Autonomous Vehicles; Financial Sector; 
Healthcare; Industrial Manufacturing; 
Smart Cities 

4.1 Cryptocurrencies and Token; 
Distributed Governmental Registries; 
Distributed Multi-user Applications; e-
Voting (certification); Intellectual Property 
Rights; Personal Information Recording and 
Security (i.e. BSc, MSc, certifications, 
personal health information 

4.2 Other Distributed Ledger 
Technologies 

4.2 e-Procurement; ID Identification; Smart 
Contracts 

5. Digital 
Identity and 
Security 

5.1 Firewall and Protocols 

5.1 Business Security; Environmental 
Security; Human Resources Security; 
Protocol Security; Secure Exchanges and 
Transactions 

13 

 
Technology 
Areas 

Technology Categories 

Technology Uses/Applications 
(exemplary, not exhaustive) 

5.2 Antivirus and Vulnerability 
Scanners 

5.2 Backup Files Disclosure; File Disclosure; 
File Inclusion; Language based Security; 
Machine Learning and Artificial Intelligence 
Security; Vulnerability Discover 

5.3 Biometric Screening 

5.3 Biometric Analysis and Monitoring; 
Border Control; ID Identification; Public 
Safety; Video Surveillance 

5.4 Cloud-oriented 
Cybersecurity 

5.4 Privacy by Design 

5.5 Advanced User Analytics 

5.5 Fraud Prevention; Vulnerabilities 
Detection 

5.6 Mobile ID 

5.6  Advanced,  Proactive  Public  Services; 
User / Citizen Identification 

5.7 Digital Identity Frameworks 

6. Immersive 
Technologies 

6.1 Augmented Reality 

6.2 Virtual Reality 

5.7  Authentication  and  Trust  Frameworks 
(eIDAS);  Digital  Signature;  e-Voting; 
Financial  and  Commercial  Transactions 
Security;  Gaming  &  Gambling;  Security 
Framework  for  Internet  of  Things  and 
Blockchain 

6.1  Employee  and  User  Training;  Next 
Generation  User  Experience;  Gamification; 
Rich Experience Intelligent Agents 

6.2  Employee  and  User  Training;  Next 
Generation  User  Experience;  Gamification; 
Rich Experience Intelligent Agents 

7. Internet of 
Things and 
Smart Devices 

7.1  Mobile  Devices,  Wearables 
and Sensors 

7.1  Drones  and  Autonomous  Vehicles; 
Electronic  Appliances;  Robots;  Speaker 
Systems 

Internet 

7.2 
Platforms 

of 

Things 

7.2  Analytics;  Data  Management;  Devices 
Connection;  Smart  Cities;  Smart  Homes; 
Smart Personal Living 

8. Software 
and Service 
Technologies 

8.1  APIs,  Web  Services, 
Microservices 
including 
Registries and Marketplace 

Software 

8.1 
Integration 

Integration; 

System 

8.2  Enterprise  Service  Bus 
Technologies  and  Government 
Service Utilities 

8.2  Linked  Data  Collection,  Processing  and 
Diffusion;  Open Data  Collection,  Processing 
and  Diffusion;  Next  Generation  Public 
Service  Provision  Models; Proactive  Service 
Provision 

Table 1 - Technology Areas, Categories and Uses/Applications (authors’ elaboration) 

14 

 
In line with this representation, the following figure (Source: authors’ elaboration 

Figure 3) shows the technology areas selected as relevant, with respect to the purpose 
of the project. 

Source: authors’ elaboration 

Figure 3 - Technology areas supporting public service delivery 

15 

 
 
 
3  Interoperability issues and user centricity aspects  

Once the thematic clustering was  finalised of  relevant digital technologies, both mature 
and emerging, for the European public sector, the next activity was aimed at examining 
their  potentialities  and  constraints  in  terms  of  service  delivery,  governance  and 
policymaking by focusing on key interoperability and user centricity aspects.    

Basically, the idea was to answer the following business question:  

  BQ3:  What  is  the  current  or  prospective  impact  of  those  technologies  on  the 
design  and  implementation  of  digital  public  service  provision  (assessed  in 
particular with regards to interoperability and user centricity principles)?  

This section briefly overviews the methods used to assess the identified eight technology 
areas according to their interoperability issues/risks and major user centricity challenges 
and the results obtained from the exercise. 

3.1  Interoperability assessment method 

The adopted method focused on the 4 layers of the EIF Interoperability Model14 that are 
relevant to the process of establishing interoperable European public services. 

The EIF promotes electronic communication among European Public Administrations and 
stresses  the  importance  to  address  interoperability  issues  according  to  a  holistic 
approach, based on 12 principles, and keeping into account the four layers presented in 
the following Table. 

Interoperability 
Layers 

Description 

Legal 
Interoperability 

It represents how the provision of a European public service works within 
its own national legal framework. Legal interoperability is about ensuring 
that organisations operating under different legal frameworks, policies 
and strategies are able to work together and reuse innovative public 
solutions. 

Organisational 
Interoperability 

It refers to the way in which Public Administrations align their business 
processes, responsibilities and expectations to achieve commonly agreed 
and mutually beneficial goals. 

Semantic 
Interoperability 

It ensures that the precise format and meaning of exchanged data and 
information is preserved and understood throughout exchanges between 
Public Administrations, in other words “what is sent, is what is 
understood” and it is analysed by the semantic and syntactic aspects. 

14 https://ec.europa.eu/isa2/sites/isa/files/eif_brochure_final.pdf  

16 

 
 
 
 
 
                                           
Technical 
Interoperability 

It covers the applications and infrastructures linking systems and 
services. Aspects of technical interoperability include interface 
specifications, interconnection services, data integration services, data 
presentation and exchange, and secure communication protocols. 

Table 2 – 4 interoperability layers description 

The rating of interoperability was done using a 4-level scale, each of them denoting the 
level  of  caution  and  effort  that  should  be  exercised  in  ensuring  interoperability  along 
time, among European Union Member States:  

  1  –  Critical:  Important  considerations  have  to  be  taken  in  mind  for  each  of  the 

technologies / areas included in the domain.  

  2 – High: A significant part of the existing documentation and regulation on the 

topic should be updated, to ensure interoperability. 

  3  –  Medium:  Some  caution  should  be  exercised,  and  specific  references  should 
be  made  /  guidelines  should  be  given  in  the  relevant  documentation  (e.g.  EIF, 
EIRA, etc.). 

  4 – Low: No significant interoperability issues. A general mention of technologies 

in the relevant interoperability documentation is enough. 

3.2  User centricity assessment method 

The 12 principles of the EIF  Interoperability Model underline another relevant aspect to 
consider for the process of modernisation of European public services: the user centricity 
principle. User centricity is not a stand-alone EIF Principle (#6) but is closely related to 
two of them (#7 to #9) also concerning general user needs and expectations.  

Furthermore,  user  centricity  is  identified  as  a  key  aspect  in  the  Tallinn  Declaration  on 
eGovernment  signed  in  2017  by  32  Member  States.  Accordingly,  user  centricity  means 
putting the end-users (citizens, businesses, public sector employees) truly at the centre 
of services through the spread of digitalisation across all policy areas.   

The  eight  user  centricity  principles  as  acknowledged  by  the  Tallinn  Declaration  are 
presented below in Table 3. 

User Centricity 
Principles 

Description 

Digital Interaction 

Technology gives to citizens and businesses the option to digitally 
interact with administrations. 

17 

 
 
 
 
Accessibility, security, 
availability and 
usability 

Technology makes digital public services more accessible (including 
findable) and secure and can be used by all in a non-discriminatory 
manner, with appropriate assistance available upon need; 
Technology makes it so that the principles of universal design are 
applied to the setting up of the services and that the websites are 
simple to read and easy to understand; technology makes it so 
that the authenticity of digital public services is secured and can be 
recognised in a clear and consistent manner. 

Reduction of the 
administrative burden 

Technology supports Public Administrations’ efforts to reduce the 
administrative burden on citizens and businesses, namely by 
optimizing and/or creating digital processes and services where 
relevant and possible, and by offering personalised and pro-active 
services; technology facilitates citizens and businesses not to be 
asked to provide the same information to public services more than 
once, in due respect of data protection rules and regulations. 

Digital delivery of 
public services 

Technology procures that public services can as much as possible 
and appropriate, especially upon request of the user, be fully 
handled online, including the provision of any evidence required to 
obtain a right or fulfil obligations; technology ensures that the 
status of service delivery can be checked online where relevant. 

Citizen engagement 

Technology empowers citizens and businesses to voice their views, 
allowing policy makers to collect new ideas, involving citizens more 
in the creation of public services and ultimately providing better 
digital public services. 

Incentives for digital 
service use 

Technology helps remove barriers to use digital public services 
effectively, including by extending and promoting the benefits of, 
for example, higher confidence, speed, effectivity and reduced 
costs to individuals who are able to use them. 

Protection of personal 
data and privacy 

Technology allows the handling of personal data in compliance with 
the GDPR and privacy requirements in the EU and at national 
levels, when applicable through informing citizens about the use 
and storage of their personal data and allowing citizens to access 
and ask for the correction and deletion of personal data, where 
appropriate. 

18 

 
 
 
 
 
 
 
 
Redress and complaint 
Mechanisms 

This technology procures that redress mechanisms are available 
online and that citizens and businesses have access to complaint 
procedures online, while also in other available channel(s) of their 
choice. 

Table 3 – 8 user centricity principles description 

Similarly, to interoperability challenges, for each of the eight technology areas identified 
in  the  previous  activity  their  user  centricity  status  was  rated,  taking  on  a  EU  wide 
perspective.  To  that  end,  a  4-level  scale  was  introduced,  based  on  the  following 
convention15:  

1  –  Not  At  All:  No  evidence  that  this  specific  dimension  (aka  principle)  of  user 

centricity is active or meaningful in the technology at hand.  

2  –  To  Some  Extent:  Some  evidence  exists  that  such  dimension  (principle)  is 
present, but there are clear and evident limits – which can be of technological or 
even  non-technological  nature  –  preventing  a  more  intense  or  extensive 
occurrence of that character in the supported or facilitated public services. 

3 – To a Great Extent: Significant evidence shows that this dimension (principle) of 
user  centricity  is  active,  though  not  reaching  its  full  potential,  as  far  as  the 
associated public services are concerned. 

4 – Quite Likely  So: We can safely state that the corresponding dimension is fully 
operational and effective in the direction of promoting user centricity in the public 
services enabled by that technology area. 

3.3  Technology assessment results 

In  the  following  subsections,  we  provide  a  detailed  report  of  the  implementation  of  the 
above  methods  to  assess  the  capacity  of  the  8  chosen  thematic  clusters  to  be 
interoperable and user centric. 

3.3.1  Artificial Intelligence 

Artificial  Intelligence  (AI)  applies  advanced  analysis  and  logic-based  techniques  to 
interpret events, support and automate decisions, and take actions. AI is the simulation 
of  human  intelligence  processes  by  machines,  especially  computer  systems.  These 
processes may include learning (acquisition of information and rules to understand those 
input),  reasoning  (using  algorithms  to  reach  approximate  or  definite  conclusions)  and 
self-correction  (adjust  the  behaviour  according  to  conclusions).  AI  can  be  seen  as  an 
umbrella  term  encompassing  Machine  Learning  (ML)  and  Deep  Learning  (DL), 
Natural Language Processing (NLP), Text Mining (TM) and Computer Vision (CV), 
Cognitive Computing and Predictive Analytics, Expert and Rule Based Systems, 
to interpret events, provide support, automate decisions and take action. 

15  In  case  of  complex  dimensions,  i.e.  a  principle  composed  of  a  number  of  sub-principles,  the  assessment 
score is calculated as linear average and rounded where appropriate. 

19 

 
 
 
 
                                           
Source: authors’ elaboration 

Figure 4 - Artificial Intelligence Area, Categories and Uses/Applications 

Artificial Intelligence (AI) is spreading fast and the impact on the public service delivery 
is approaching. Consequently, huge implications will be propagated towards the existing 
and  foreseen  interoperability  documentation  and  guidance.  Moreover,  an  increasing 
number  of  government  agencies  are  considering  adopting  AI  in  the  decision-making 
process.  Thanks  to  AI,  all  governments  can  perform  more  efficiently,  improving 
outcomes and keeping costs down.   

Source: authors’ elaboration 

Figure  5  reports  about  the  interoperability  and  user  centricity  assessment  of  Artificial 
Intelligence. The explanation of the evaluation is provided below. 

20 

 
 
Source: authors’ elaboration 

Figure 5  - Artificial Intelligence Interoperability and User centricity assessment 

legal  and 
interoperability,  the  main  aspects  to  consider  are  both 
Regarding 
organisational,  pertaining  to  the  alignment  of  processes  but  also  ethical  considerations 
and governing frameworks across the Member States.   

The  alignment  of  Member  States’  legal  frameworks,  ensuring  that  European  citizens 
will feel “at home” in any Member States after the utilisation of Artificial Intelligence, will 
be a very hard task. This includes both innovation in operations (how AI will be used in 
the back and front office, in the long run or in real time) as well as ethical considerations 
(what  the  thresholds  or  limits  will  be),  at  each  Member  State.  EC  and  Member  States 
should continue to focus their efforts on the  realisation of a coordinated Action Plan for 
AI  to  provide  recommendations  on  policy  and  investment  and  to  set  guidelines  on  the 
ethical development of  AI. This aspect is also identified in the latest EC report  on AI in 
which it is clearly emphasised the need for a balanced regulatory framework, since many 
European Countries are developing national strategies to embrace AI.16 

Moreover, organisational interoperability poses major challenges in the application of 
Artificial Intelligence within the public sector. It is unclear what will be the pan-European 
processes that will govern the new possibilities that Artificial Intelligence offers, such as 
instant  decision  making  or  deep-learning  enabled,  big  data-based  forecasting  or 
budgeting. 

For  what  concerns  semantic  interoperability,  AI  is  not  expected  to  bring  extremely 
demanding  new  needs.  However,  fully  automated  service  delivery,  as  envisaged  by 
Artificial  Intelligence  applications  in  the  public  sphere,  will  generate  more  pressure  for 
full  coverage  of  existing  semantic  interoperability  issues.    In  other  words,  we  need  to 
solve  all  the  codification  and  semantic  standardisation  issues  to  work  in  an  acceptable 
way. 

Technical  Interoperability  is  foreseen  to  be  an  important,  but  less  critical,  issue  for 
Artificial  Intelligence.  Most  important  needs  will  come  from  the  aspect  of  integrating  AI 
infrastructures and dedicated systems within the overall reference architecture for Public 
Administrations,  as  well  as  from  the  ever-emerging  new  paradigms  of  utilizing  AI  in 

16 https://publications.jrc.ec.europa.eu/repository/bitstream/JRC113826/ai-flagship-report-online.pdf 

21 

 
 
                                           
service  provision.  The  emergence  of  technical  standards  for  AI  is  apparent  in  ISO  and 
IEEE,  in  recent  months.  However,  this  is  expected  to  be  a  dynamically  evolving 
phenomenon:  as  more  capabilities  and  applications  will  be  discovered,  the  need  for 
technical coordination and interoperability is bound to rise in the upcoming years. 

To date, technology and semantic standardisation needs will be below the  critical level, 
as  the  overall  AI  sphere  is  still  rapidly  expanding,  and  issues  of  AI  platform 
interoperability  or  semantic  codification  of  elements  are  not  yet  the  main  showstopper. 
Nevertheless,  as  AI  will  be  penetrating  every  form  of  service  provision,  new  needs  for 
interoperability guidance will emerge day-by-day. 

Regarding user centricity, Artificial Intelligence has the clear potential to provide citizens 
and  businesses  with  the  opportunity  to  digitally  interact  with  Public  Administrations.  It 
holds  a  largely  untapped  potential  to  make  digital  services  more  accessible  and  secure 
and  that  can  be  used  by  everyone  in  a  non-discriminatory  way,  at  the  same  time 
guaranteeing a high level of data protection.  

Moreover,  with  most  of  its  implementations  –  including  e.g.  Machine  Learning  and 
Predictive Analytics – AI can greatly contribute to support Public Administrations efforts 
to  restructure  existing  digital  processes  and  services  and/or  create  new  ones  where 
relevant  and  possible,  to  reduce  the  administrative  burden  on  citizens  and  businesses. 
Even though Artificial Intelligence engages  citizens less than other technologies used in 
the  public  sector,  there  are  important  incentives  for  its  usage,  and  this  is  seen  as  a 
starting point to breakdown the public service barriers. 

AI  is  probably  the  technology  area  that  will  mostly  affect  the  operations  of  Public 
Administrations,  businesses  and  citizens,  in  the  years  to  come.  There  are  several 
interoperability issues to be solved, specifically in legal and organisational fields, but the 
potential  of  this  technology  area  is  relevant.  Governments  can  exploit  the  power  of  AI 
with  the  purpose  to  innovate  or  redefine  policies  and  services  to  make  administrative 
tasks  as  a  process-driven  algorithm,  in  order  to  boost  efficiency  and  meet  citizen’s 
needs.   

3.3.2  Communication Technologies 

information.  The  main  Categories 

Communication  Technologies  refer  to  all  equipment  and  programs  that  are  used  to 
process  and  communicate 
identified  within 
Communication Technologies are 5G Networks and Handheld Devices and Software 
Defined Network. The first one represents a wireless technology that allows individuals 
to 
to  a broadband 
(smartphone, 
internet connection wirelessly through a mobile phone network. 5G Networks aim is to 
increase data communication speeds by up to 3 times compared to its predecessors. 

connect  handheld  devices 

tablet,  PDAs) 

Software  Defined  Networks  (SDN)  is  an  approach  to  network management  that 
enables  dynamic,  flexible  and  programmatically  efficient  network  configuration  in  order 
to  improve  current  network  performance.  This  is  a  new  architecture  that  is intended  to 
be the backbone of the 5G Infrastructure.  SDN is currently much more mature than 5G 
and  is  successfully  implemented  for  managing  large  datacentres  and  is  redefining  the 
architecture to bolster the development of the 5G ecosystem, which is still to come and 
will play a pivotal role in the creation of multiple network hierarchies.  

22 

 
 
Source: authors’ elaboration 

Figure 6 - Communication Technologies Area, Categories and Uses/Applications 

With  the  introduction  of  5G  networks  and  relevant  infrastructures  and  their  large-scale 
deployment in the years to  come, communication technologies will pave the ground for 
important challenges in the interoperability arena. Moreover, the novelty of 5G networks 
will support the promise to make digital public services more accessible and secure.  

In Source: authors’ elaboration 

Figure  7,  the  interoperability  and  user  centricity  assessment  of  Communication 
Technologies is summarised. The explanation of the evaluation is provided below. 

Source: authors’ elaboration 

Figure 7 - Communication Technologies Interoperability and User centricity assessment 

The analysis carried out suggests that significant efforts should be made to improve legal 
and  technical  interoperability.  This  relates  to  new  services  that  will  be  made  available, 
new  devices  that  should  be  embodied  in  the  relevant  technical  standardisation  by 
Member States and possible types of services that the new infrastructure may allow.  

Alignment of Member State legislation will be needed, in the areas of data roaming and 
cross-country  exchange  of  information  via  ultra-high-speed  networks,  following  on  the 
existing  robust  regulatory  framework  at  EU  level.  However,  since  5G  may  enable 
completely  new  capabilities  and  services  for  citizens  and  businesses,  most  of  the 

23 

 
 
 
countries will have to adapt or enhance their legislation in order to cover their alignment 
“by design”.    

Regarding  organisational  interoperability,  few  needs  are  foreseen:  the  governance 
structures are well in place in the area of Mobile Communication and regulation of Mobile 
Communication is well established through BEREC17.  

Semantic  interoperability  aspects  are  deemed  of  somewhat  lower  importance. 
Possible  issues  relate  to  the  development  of  controlled  vocabularies  and  code  lists  for 
various  types  and  subtypes  of  communication  infrastructures,  in  use  within  the  various 
service profiles and relevant directories in Member States. 

Technical interoperability issues might be important in the upcoming years, as a new 
generation  of  networks,  devices,  service  types  and  final  services  towards  citizens  and 
businesses will evolve.  Interoperability must be ensured among these new elements and 
layers,  keeping  in  mind  also  the  “mobile-first”  principle  for  digital  public  services. 
However, the effort will not be tremendous as the current technical standardisation level 
at  mobile  (3GPP)  and  IoT  (protocols)  is  deemed  robust  and  inclusive.  5G  will  certainly 
become an enabler that will provide interoperability in the IoT connectivity paradigm.    

Looking at user centricity, communication technologies will likely support the promise 
to  make  digital  public  services  more  accessible  and  secure.  They  can  be  used  by 
everyone, in a non-discriminatory manner. Moreover, this technology area can facilitate 
the  application  of  the  principles  of  universal  design  to  the  setting  up  of  the  services, 
ensuring  that  the  websites  are  simpler  to  read  and  easier  to  understand.  In  addition, 
they can contribute to making so that the authenticity of digital public services is secured 
and recognised in a clear and consistent manner. 

Communication Technologies can contribute to support Public Administrations’ efforts to 
restructure  existing  digital  processes  and  services  and  have  a  great  influence  on  the 
empowerment of citizens and businesses, reaching remote areas even where broadband 
is not economically viable. In general, a broader diffusion of such technologies can help 
to  create  the  conditions  in  the  public  sector  for  setting  up  redress  mechanisms  and 
complaint  procedures  that  citizens  and  businesses  can  have  access  to.  Beyond  the 
mobile use, 5G might revolutionise IoT networks that are largely used in the context of 
smart  cities,  maximizing  the  scores  of  several  principles  such  as  digital  interaction, 
accessibility or citizen engagement.   

In the European context, Communication Technologies must be considered the basis for 
important  challenges  in  the  interoperability  and  user  centricity  fields.  5G  networks  as 
well  as  Software  Defined  Network  architectures  will  enable,  within  the  near  future,  to 
increasingly reach remote areas and will revolutionise IoT networks and the consequent 
diffusion of Smart Cities.   

Regarding  the  interoperability  layers,  this  technology  area  does  not  present  particular 
issues.  A  specific  regulating  agency  (BEREC)  was  established  in  order  to  ensure  “high 
quality application of the  European  regulatory framework for electronic communications 
markets”.  Moreover,  technical  interoperability  is  solid  thanks  to  the  3GPP  group  and  to 
the  existent  protocols  enabling  communication  between  all  types  of  devices.  To  ensure 
the  full  deployment  of  communication  technologies  and  their  future  developments, 
consistent and coordinated efforts need to be made at a European level. 

17  Body  of  European  Regulators  for  Electronic  Communications:  regulating  agency  of  the  telecommunication 
market in the European Union, composed by EC and all National Regulators.   

24 

 
 
                                           
3.3.3  Computing Infrastructures 

Computing Infrastructures represent a collection of servers, networks, physical and cloud 
data centres and related equipment to provide a specified level of aggregate computing 
capacity.  It  provides  the  hardware  and  software  that  other  systems  and  services  are 
built  on.  The  main  categories  identified  are  High-Performance  Computing,  Cloud 
Computing and Edge Computing. 

Source: authors’ elaboration 

Figure 8 - Computing Infrastructures Area, Categories and Uses/Applications 

The  technology  area  of  Computing  Infrastructures  forms  the  fundamental  layer  of  the 
technology  stack  and  is  not  foreseen  to  pose  extreme  challenges  in  interoperability,  as 
most  of  the  issues  are  to  be  dealt  with  in  higher  layers  (e.g.  services,  applications, 
devices, etc.).  Moreover, it is a technology area that is built with a formal and explicit 
attention  to  the  user  centricity  principles,  therefore  it  could  considerably  help 
governments to put users at the heart of public services. 

Source: authors’ elaboration 

Figure  9  reports  the  interoperability  and  user  centricity  assessment  of  Computing 
Infrastructures. The explanation of the evaluation is provided below. 

25 

 
 
Source: authors’ elaboration 

Figure 9 - Computing Infrastructures Interoperability and User centricity assessment 

Even if Computing infrastructures does not present notable issues, important aspects of 
Cloud  Interoperability  need  to  be  dealt  with,  primary  at  a  legal  and  technical  level  and 
then at organisational and semantic levels.   

Regarding  legal  interoperability,  a  high  level  of  interoperability  guidelines  is  needed 
for  the  following  years.  These  guidelines  should  regulate  cloud  interoperability  at  an 
infrastructure  and  platform  level,  give  instruction  for  aligning  the  legislation  on  data 
storage and processing over the cloud and give directions on how to tackle ethical issues 
when  using  HPC  infrastructures  for  big  data  processing  (relating  also  to  Artificial 
Intelligence).  

interoperability,  medium 

interoperability 
Concerning  organisational 
guidelines  is  needed.  Main  issues  include  processes  and  process  alignment  for 
embedding  cloud  and/or  HPC  infrastructures  in  digital  public  services  and  internal 
procedures of the Member State public sector organisations. However, the procurement 
and selection processes for computing infrastructures by the public sector will be needing 
continuous adaptation, as more and more facilities will be moving towards the cloud.  

level  of 

Semantic interoperability aspects are deemed of medium importance. Possible issues 
relate  to  development  of  controlled  vocabularies  and  code  lists  for  various  types  and 
subtypes  of  computing  infrastructures,  in  a  way  that  they  will  facilitate  their 
categorisation and (automated) comparison for selection.  

Technical  interoperability  issues  are  still  important,  concerning  HPC  and  cloud 
infrastructures,  especially embracing the new edge and fog paradigms.  New guidelines 
and  items  of  standardisation  will  be  needed  for  Member  States  to  ensure  interoperable 
infrastructures  at  a  technical  level.  The  various  possible  types  of  Cloud  Infrastructures 
should  also  affect  the  Enterprise  Interoperability  Architecture  and  relevant  guidelines. 
Cloud  interoperability,  the  ability  for  a  public  sector  organisation  to  move  freely  within 
and among cloud infrastructure providers is critical.  

Regarding  user  centricity,  this  technology  area  could  be  considered  as  natively  built 
with a formal and explicit attention to the principles listed in the Tallinn Declaration. In 
fact, Computing Infrastructure Technologies are ready to support Public Administrations 
efforts  to  reorganise  and  enhance  existing  digital  processes  that  are  more  suitable  to 

26 

 
 
citizens  by  offering  accessible,  easier,  convenient  and  more  secure  digital  services  to 
users.  Moreover,  where  relevant,  this  technology  also  offers  new  services  and 
application, empowering citizen engagement. Looking at the protection of personal data, 
with  the  diffusion  of  Privacy  by  Design  principles  and  the  growing  awareness  of  the 
importance  of  data  protection,  it  would  be  impossible  to  conceive  a  Computing 
Infrastructure  not  allowing  the  handling  of  personal  data  in  compliance  with  the  GDPR 
and  privacy  requirements  in  the  EU.  However,  as  most  of  the  Cloud  providers  are  US 
based, it becomes difficult to ensure full protection of personal data and privacy, despite 
the GDPR. 

Overall,  Computing  Infrastructures  can  support  European  governments  to  create  more 
coherent, user-centric and digital public services for citizens and seamless efficient public 
sectors  across  Europe.  In  fact,  as  seen  from  the  assessment,  this  technology  does  not 
pose critical challenges for both interoperability and user centricity aspects. 

However,  in  the  long  run,  specific  efforts  should  be  made  to  tackle  legislative  and 
technical  challenges.  Even  if  Computing  Infrastructure  Technologies  encourage  citizens 
and  businesses  to  digitally  interact  with  Public  Administrations,  the  overall  take  up 
remain low.   

The  introduction  of  the  Cloud  in  Public  Administrations  is  enabling  the  attainment  of 
higher quality services increasing PAs transparency and security. 

3.3.4  Digital Identity and Security 

Digital Identity and Security technologies englobe both ICT security and physical security 
guaranteed through technological tools. This area comprises a set of products, services, 
organisation  rules  and  individual  behaviours  that  protect  the  ICT  system  of  an  entity. 
The  categories  embedded  in  Digital  Identity  and  Security  are  Firewall  and  Protocols, 
Antivirus  and  Vulnerability  scanners,  Advanced  user  analytics,  Biometric 
Screening,  Cloud-Oriented  Cybersecurity,  Mobile  ID  and  ultimately  Digital 
Identity Frameworks. 

Digital Identity and Cyber-Security infrastructures are horizontal elements in the digital 
public  service  infrastructures  of  any  country.  Consequently,  any  progress  in  such 
technologies  and  their  new  applications  usually  affects  the  whole  stack  of  digital  public 
service provision of administrations. To date, Member States are focusing their efforts on 
delivering easy-to-use and secure online public services that satisfy citizens’ needs.  

27 

 
 
Source: authors’ elaboration 

Figure 10 - Digital Identity and Security Area, Categories and Uses/Applications 

In Source: authors’ elaboration 

Figure  11  the  interoperability  and  user  centricity  assessment  of  Digital  Identity  and 
Security are reported. The explanation of the evaluation is reported below. 

28 

 
 
Source: authors’ elaboration 

Figure 11 - Digital Identity and Security Interoperability and User centricity assessment 

Legal  interoperability  issues  emerging  from  developments  in  Digital  Identity  and 
Security  technologies  are  foreseen  as  critical.  Developments  beyond  the  eIDAS 
specification  in  Member  States,  which  might  include  IoT-based  identity  protocols  or 
biometrics, are bound to create interoperability standardisation challenges, by means of 
the needed alignment of Member State legislation to allow for interoperable operations of 
such new developments in all Member States.  In parallel, envisaged infrastructure and 
data security developments in the post-GDPR era are also bound to create primarily legal 
interoperability challenges for the alignment of Member States’ legislation both from an 
operational  (how  the  new  services  will  operate)  and  ethical  points  of  view.  Therefore, 
specific efforts must be devoted for guidelines enabling the alignment of Member States 
legal frameworks in new uses of devices and biometrics for digital identity, as well as for 
the new security analytics and cloud cyber-security. 

New  challenges  in  organisational  interoperability  will  emerge  from  Digital  Identity 
and  Security  technologies,  pertaining  to  the  embodiment  of  new  technologies 
(biometrics,  mobile  ID,  security  analytics)  and  regulations  (eIDAS,  GDPR)  in  the  public 
service generic models and archetypes.  

Since  e-ID  is  the  common  denominator  in  almost  all  digital  public  services,  new 
documentation of semantic  components around identity and security (such as types of 
ID’s, types of data to be gathered, codifications of security-related data) will be needed. 
The  creation  of  core  components  for  digital  identities  for  persons,  devices  (IoT), 
personas (immersive technologies), bots and agents (AI) will also be required. 

With the development of new identity and security mechanisms, without considering the 
introduction  of  a  multiplicity  of  IoT  devices,  AI  agents  and  autonomous  systems,  the 
technical standardisation for e-identity and cyber-security needs significant updates. 

For  what  concern  user  centricity,  the  importance  to  ensure  data  protection  and  to  put 
end-users at the heart of public services has been stressed in the Tallinn Declaration. 

As  Computing  Infrastructures,  also  this  family  of  technologies  can  be  seen  as  a  key 
enabler  –  if  not  a  prerequisite  –  of  the  introduction  of  any  new  digital  public  service  or 
application.  In  that  regard,  the  support  for  citizens  and  businesses  willing  to  digitally 
interact with Public Administrations is fully evident. Indeed, there are some categories – 

29 

 
 
such  as  biometric  screening  –  that  have  only  recently  emerged  and  thus  cannot  be 
associated with a broad documentation of cases in the European public sector yet.  

Generally,  Digital  Identity  and  Security  technologies  are  likely  to  significantly  increase 
the uptake of eID by citizens, increase accessibility and participation and conduct secure, 
online  and  cross-border  transactions  with  public  and  private  sector  entities.  The  eIDAS 
Regulation  (EU  No  910/2014)  has  encouraged  the  widespread  and  seamless  use  of 
secure eID at a European level. 

This technology area does not pose extreme challenges in user centricity, except in the 
involvement  of citizens  in the creation of digital public services and in the procurement 
of  redress  mechanisms.  Digital  Identity  and  Security  Technologies  could  significantly 
contribute to support digital interaction among citizens and businesses willing to interact 
with  Public  Administrations.  However,  since  some  technology  categories  have  recently 
emerged,  more  research  is  needed  to  assess  how  these  technologies  foster  digital 
interaction and how they can be used in a non-discriminatory manner. 

This technology area presents interoperability challenges, above all with respect to legal 
aspects.  In  fact,  European  countries  have  varied  security  mechanisms  for  identification 
and  authentication,  which  are  based  on  different  philosophies  and  lack  cross-border 
recognition and validation.  

Specific effort has to be invested in order to create homogeneous guidelines that enable 
the alignment of Member States’ legal frameworks in new uses of devices and biometrics 
for digital identity, as well as for the new security analytics and cloud cyber-security.  

Digital Identity and Security technologies are fundamental to put end-users truly at the 
centre of public service delivery, by improving security of services offered to the citizens 
and businesses and assuring personal data protection. 

3.3.5  Distributed Ledger Technologies 

Distributed Ledger Technologies have been magnified by the  success  of Bitcoin and the 
explosion of potential use cases, becoming mainstream in research and policy agendas. 
This trend has created high expectations regarding the transformative role of blockchain 
for  both  the  industry  and  the  public  sector.  However,  ongoing  project  has  proven  to 
bring incremental changes rather than fundamental ones to the operational capacities of 
governments. 

A Distributed Ledger Technology (DLT) is a digital system for recording the transaction of 
assets in which the transactions and their details are recorded in multiple places at the 
same  time.  Distributed  ledger  technologies  have  therefore  the  potential  to  speed 
transactions  because  they  remove  the  need  for  a  central  authority  or  middleman. 
Moreover,  they  have  the  potential  to  reduce  costs  of  transactions  and  are  more  secure 
because each node of the network holds records, thereby creating a system that is more 
difficult  to  manipulate  or  successfully  attack.  Distributer  Ledger  Technologies  embrace 
two main categories: Blockchain and other DLTs.  

Distributed  Ledger  Technologies  concern  Distributed  Databases  or  Distributed  Registers 
that  can  be  updated,  managed,  controlled  and  coordinated  not  only  centrally,  but  in  a 
distributed way, by all players.  The prerequisites for Distributed Ledger Technology are 
in  the  creation  of  large  networks  consisting  of  a  series  of  participants  and  each 
participant is called upon to manage a node in this network. Each node is authorised to 
update  the  Distributed  Ledger  independently  of  the  others  but  under  the  consensual 
control  of  the  other  nodes.  Similarly,  updates  or  records  are  no  longer  managed,  as 

30 

 
 
traditionally  happened,  under  the  strict  control  of  a  central  authority,  but  are  instead 
created and uploaded by each node independently. In this way, each participant is able 
to process and control every transaction, but at the same time every single transaction, 
even  if  managed  independently,  must  be  verified  and  approved  by  the  majority  of  the 
participants of the network. Here lies the concept behind Distributed Ledger Technology 
or  the  concept  of  consensus.  The  autonomy  of  each  node  is  subject  to  reaching  a 
consensus  on  the  operations  that  are  carried  out  and  only  with  this  consent  they  are 
then  authorised  and  activated.  The  various  types  of  DLT  are  distinguished  primarily  in 
the way in which the consensus applies to the actions of writing on the Register.  

Blockchain is a category of DLTs and one of several consensus management applications 
used  to  apply  Distributed  Ledger  Technology.  Blockchain  organises  data  into  blocks, 
which  are  chained  together,  and  information  cannot  be  cancelled  out.  This  technology 
has been successfully implemented in the world of cryptocurrencies and has the potential 
of rewriting the notions of transaction. Moreover, Blockchain has and could been used to 
manage  identity  (birth  certificates,  marriage  licenses,  passport  and  death  records); 
personal records (health, insurance and financial); land title registry; voting; copyrights, 
risks and fraud, etc.  

Source: authors’ elaboration 

Figure 12 - Distributed Ledger Technologies Area, Categories and Uses/Applications 

Currently,  Distributed  Ledger  Technologies  and  primarily  Blockchain  are  “right  in  the 
middle”  of  a  techno-commercial  turmoil,  with  great  possibility  to  affect  all  European 
Union Member States and beyond, in the years to come. Given the very limited, if any, 
information  on  how  governments  and  public  sector  administrations  are  to  embrace  this 
new phenomenon, and include it within their National Interoperability Frameworks, Base 
Service  Registries  or  Service  Portals,  new  guidelines  are  needed  for  achieving 
interoperability in infrastructures, organisational issues, and final services.  

Even if Distributed Ledger Technologies are still considered “newcomers” in the European 
context,  and  thus  their  advantages  related  to  their  usage  are  not  demonstrated,  they 
can  be  expected  to  potentially  support  Public  Administration  by  reducing  administrative 
barriers and by fostering digital interaction with citizens and businesses. 

In Source: authors’ elaboration 

31 

 
 
Figure  13,  the  interoperability  and  user  centricity  assessment  of  Distributed  Ledger 
Technologies is summarised. The explanation of the evaluation is provided below. 

Source: authors’ elaboration 

Figure 13 - Distributed Ledger Technologies Interoperability and User centricity assessment 

The  estimates  suggest  that  the  most  crucial issues  revolve  around  legal,  organisational 
and  technical  interoperability  issues,  the  semantic  issues  being  slightly  easier  to  deal 
with.  As  a  new  legislation  will  evolve  in  the  upcoming  years  in  Member  States  and 
currently  documentation  on  the  use  of  DLT’s  in  the  public  sector  lacks,  new  guidelines 
should  be  developed  and  put  into  effort.    Alignment  of  Member  States  legal 
frameworks  concerning  DLTs  should  be  seen  both  from  an  administration  and  a 
business/citizen  point  of  view,  as  blockchain  will  be  radically  disrupting  the  status  quo 
(trusted  authorities,  data  protection,  etc.).  The  specific  issue  of  GDPR  that  contradicts 
the inalterability of the Blockchain to be forgotten is already being assessed by the latest 
European  Parliament  study  on  “Blockchain  and  the  General  Data  Protection 
Regulation”18.    

With  respect  to  organisational  interoperability,  the  application  of  DLTs  within  the 
organisational  processes  may  fundamentally  impact  the  way  services  are  provided, 
moving from the traditional trusted-server-based model to openly accessible, distributed 
infrastructures. As a result, the model of service provision will have to be revisited, from 
an  organisational  (but  also  technical)  point  of  view,  implying  that  organisational 
interoperability looks still complex to realise.  These changes will then propagate within 
service  types,  providing  alternate  means  and  processes  for  fulfilling  numerous  new 
services towards administrations, businesses and citizens.  

Although the DLTs phenomenon is still evolving,  semantic interoperability challenges 
might be the mildest of the 4 aspects. 

Moreover,  the  DLTs  and  blockchain  disruption  will  also  pose  critical  challenges  in  the 
technical interoperability layer: the distributed ledger will alter the  established ways 
that  services  are  provided,  through  base  registries,  trusted  API’s  and  Web  Services, 
centrally maintained Governmental Portals, etc.  A new, broader interoperability enabling 
reference  architecture  will  be  needed,  allowing  for  distributed  ledger  applications  and 

18  https://www.europarl.europa.eu/RegData/etudes/STUD/2019/634445/EPRS_STU(2019)634445_EN.pdf  

32 

 
 
                                           
services, or even hybrid approaches where the new technological elements co-exist with 
the  more  traditional  ones,  following  on  initial  pattern  as  depicted  in  the  recent  EC/JRC 
study on “Blockchain for digital government”.19 

Undoubtedly,  for  what  concerns  user  centricity,  Distributed  Ledger  Technologies  can 
enhance  citizens’  data  protection  and  privacy  assurance  and  the  effective  use  of  digital 
public  services.  On  the  other  hand,  this  technology  area  has  a  limited  influence  on  the 
empowerment  of  citizens  and  businesses  and  on  the  definition  of  redress  mechanisms 
and complaint procedures. 

As  a  whole,  Distributed  Ledger  Technologies  provide  above  all  transparency  and 
inalterability of transactions since any changes made in a blockchain are viewable by all 
parties of the system and once processed they cannot be removed or tampered with. In 
addition  to  this,  DLT  have  the  potential  to  reduce  costs  of  transactions  and  are  more 
secure because each node of the network holds records, thereby creating a system more 
difficult  to  manipulate  or  successfully  attack.  Moreover,  there  is  the  potential  to  speed 
transactions  because  the  need  for  a  central  authority  or  middleman  is  removed.  In  the 
near future, the main interoperability challenges will involve the legal, organisational and 
technical  fields.  Regarding  user  centricity,  the  main  challenges  will  be  related  to  the 
capacity  to  involve  citizens  in  the  creation  of  public  services  and  ultimately  provide 
better  digital  public  services.  On  the  other  hand,  the  main  issue  of  Distributed  Ledger 
Technologies  regards  its  limited  influence  on  the  empowerment  of  citizens  and 
businesses to express their opinion and on the creation of conditions in the public sector 
for  setting  up  redress  mechanisms  and  complaint  procedures,  which  businesses  and 
citizens have online access to, by means of various channels.   

3.3.6  Immersive Technologies 

Immersive  Technologies  are  defined  as  the  technology  that  digitally  simulates  the  real 
world by creating a sense  of immersion.  Immersive Technology could be divided in two 
Categories: Augmented Reality and Virtual Reality. 

Source: authors’ elaboration 

Figure 14 – Immersive Technologies Area, Categories and Uses/Applications 

19 https://joinup.ec.europa.eu/sites/default/files/document/2019-
04/JRC115049%20blockchain%20for%20digital%20government.pdf  

33 

 
 
 
                                           
Virtual  and  augmented  reality  might  have  significant  applications  in  new  service  design 
and deployment over the years to come, especially in the areas of training, gamification, 
and user experience. In fact, this technology area provides an environment where virtual 
objects are enriched by real vision, sound, and touch, allowing a sense of immersion for 
users in the physical world. Immersive Technologies hold a large untapped potential and 
could  improve  the  delivering  of  new  services  and  applications,  promoting  at  the  same 
time digital interaction between citizens/businesses and Public Administrations. 

Figure  15  reports  the  interoperability  and  user  centricity  assessment  of  Immersive 
Technologies. The explanation of the evaluation is explained below. 

Source: authors’ elaboration 

Figure 15 - Immersive Technologies Interoperability and User centricity assessment 

Based  on  the  current  status  and  interconnectivity  of  such  applications  with  the  rest  of 
the  public  sector  infrastructures,  their  resulting  need  for  interoperability  documentation 
and standardisation is currently foreseen as below average. 

Immersive  Technologies  present  very  limited  needs  for  legal  interoperability 
elements. There could be possibilities in aligning legal aspects with/of the area of ethics 
and proper usage of technologies. 

Regarding  organisational  interoperability,  specific  effort  should  be  made  for 
coordinated  guidance,  on  how  immersive  technologies  may  be  integrated  with  the 
existing service provision models, using new capabilities of 5G networks, IoT or Artificial 
Intelligence applications. 

Semantic  interoperability  does  not  pose  relevant  challenges,  only  in  the  areas  of 
categorisation and codification of different immersive technology types and applications. 

Technical  interoperability  does  not  present  significant  issues.  Specifically,  medium 
efforts  are  needed  for  interoperability-related  documentation,  mainly  in  the  areas  of 
types  and  formats  of  new,  rich  media  applications  and  the  ways  they  can  be 
interconnected with the backbone of digital service provision. 

Regarding  user  centricity,  Immersive  Technologies  constitute  a  new  and  innovative 
way  to  promote  an  effective,  purpose  oriented,  digital  interaction  between  citizens  and 
businesses,  on  the  one  hand,  and  Public  Administrations  on  the  other.  Indeed,  this 
potential  needs  to  be  further  explored  and  substantiated  with  evidence,  going  beyond 

34 

 
 
the  experimentation  level  and  documenting  a  transformative  impact  on  public  service 
delivery. 

Immersive  Technologies  can  significantly  improve  user  experience  since  technologies, 
such  as  Virtual  Reality  and  Augmented  Reality,  are  able  to  create  a  user-centric  world, 
immersing  users  in  the  physical  world  while  interacting  seamlessly  with  machines. 
However, the potential that lies behind this technology area is greater than explored so 
far/in the analysis; consequently, it is hard to affirm whether this technology area has a 
remarkable influence on the empowerment of citizens and businesses to voice their view 
and on setting up redress mechanisms. 

In  conclusion,  immersive  technologies  do  not  present  relevant  issues  concerning 
interoperability.  However,  there  is  a  need  for  coordinated  guidance  on  how  these 
technologies  could  be  introduced  in  the  public  service  provision  process  (specifically 
regarding  organisational  interoperability  and  technical  interoperability).  Research, 
documentations and prototypes   must  be  done  to  fully  understand  the  potential  that 
lies  behind  this  technology  area.  On  one  hand,  European  Member  States  lack  concrete 
examples  and  for  the  moment  it  is  only  possible  to  speculate  a  possible  role  of 
Immersive  Technologies.  On  the  other  hand,  the  great  impact  and  influence  that  these 
technologies can have in enriching user experience, accessibility, trust and convenience 
are recognised. 

3.3.7  Internet of Things and Smart Devices 

Internet of Things has been defined as a set of physical objects embedded with sensors 
or  actuators  and  connected  to  a  network.  An  IoT  platform  enables  the  development, 
deployment  and  management  of  solutions  that  connect  to  and  capture  data  from  IoT 
endpoints. 

IoT and Smart Devices could be classified in two Categories: the first one composed by 
Mobile Devices Wearables and Sensors; the second includes all IoT Platforms (IoT 
data platforms, IoT connectivity platforms, IoT cloud end to end platforms). 

Source: authors’ elaboration 

Figure 16 - Internet of Things and Smart Devices Area, Categories and Uses/Applications 

35 

 
 
 
The  technology  area  of  Internet  of  Things  is  foreseen  to  display  major  needs  for 
coverage by interoperability-related standardisation and guidelines in the years to come. 
Nowadays,  IoT  and  Smart  Devices  are  contributing  to  an  effective  removal  of  usage 
barriers  of  digital  public  services.  In  fact,  this  technology  area  can  be  considered  a 
building block for a new definition of digital public service applications. 

In Source: authors’ elaboration 

Figure 17, the interoperability and  user centricity assessment  of  Internet  of Things and 
Smart Devices are reported. The explanation of the evaluation is provided below. 

Source: authors’ elaboration 

Figure 17 - Internet of Things and Smart Devices Interoperability and User centricity assessment 

Regarding  interoperability,  the  envisaged  multiplicity  of  new  devices  that  need  to  find 
their  ways  in  the  digital  public  service  models  and  relevant  standardisation  will  create 
significant challenges at a technical and organisational interoperability aspect. 

Legal interoperability aspects of Internet of Things are considered very important, as 
new legislation will emerge for regulating the use of devices, the channelling and storage 
of  sensor-created  (big)  data,  and  the  overall  ethical  and  operational  issues  in  the  new 
smart cities and smart services of the future. 

For  what  concern  organisational  aspects,  significant  effort  should  be  devoted,  for 
updating  interoperability  guidelines,  process  descriptions  and  service  provision  models 
that will integrate the new Internet of Things devices, both at the citizen and enterprise, 
as well as the Public Administration sides. 

Moreover, new semantic elements and structures will be needed, for the identification, 
description (at data and metadata level) and overall operation of millions of new devices 
that will have a role in the digital service provision.  Also, the vast amount of data that 
will  be  generated  will  create  new  needs  for  metadata  specifications,  code  lists  and 
controlled vocabularies for their linking and publishing. 

Technical  interoperability  is  probably  the  most  important  aspect  of  interoperability 
concerning  IoT  and  Smart  Devices,  as  a  vast  variety  of  hardware  and  embedded 
software components will need to be made interoperable, at API and web services level 
with  the  existing  and  emerging  digital  public  service 
infrastructure.  Technical 
interoperability lies at different levels in IoT (connectivity, platform, data, identification, 

36 

 
 
etc).  There  are  plenty  of  solutions  and  standards  posing  a  threat  to  interoperability.  In 
addition,  new  approaches  to  reference  architectures  for  Smart  Cities  might  need  to  be 
broadly adopted as guidelines at EU level. 

There are no relevant issues concerning user centricity. The only field that needs to be 
improved  concerns  the  set-up  of  redress  mechanisms  and  complaint  procedures  that 
citizens and businesses can have access to  online. The capacity of this technology area 
to  allow  citizens  and  businesses  to  access  complaint  procedures  online  requires  careful 
coordination at the European Level. In addition, a full level of maturity has not yet been 
reached given the heterogeneity of technological components, many of which – such as 
wearables – are not well established in the European public sector. 

As a whole, the IoT and Smart Devices technology area presents some issues concerning 
interoperability. Specifically, the interoperability layer that requires more improvements 
is  the  technical  layer  because  there  are  plenty  of  solutions  and  standards  at  different 
level of  IoT, posing challenges to interoperability. Looking at  user  centricity, this family 
of technologies, developed only 15 years ago, can now be considered as a building block 
for  a  new  definition  of  digital  public  service  applications,  which  takes  considerable 
distance from the old-fashioned definition of government. In that regard, IoT and Smart 
Devices  can  be  intrinsically  supportive  of  citizens  and  businesses  willing  to  digitally 
interact with Public Administrations. 

3.3.8  Software and Service Technologies  

Software and Service Technologies englobes everything concerning software. An evident 
example of Software and Service Technologies is Software as  a Service (SaaS) which is 
owned,  delivered  and  managed  remotely  by  one  or  more  providers.  Within  this 
technology area, two macro-Categories are included: APIs, Web Services and Micro-
services  including  Registries  and  Marketplaces  on  the  one  hand;  Enterprise  Service 
Bus Technologies and Government Service Utilities on the other hand. 

Source: authors’ elaboration 

Figure 18 - Software and Service Technologies Area, Categories and Uses/Applications 

Software  and  Service  technologies  have  traditionally  been  the  most  “well  represented” 
technology area within the existing documentation for interoperability at a pan-European 
level.  However,  the  emerging  new  paradigms  in  the  areas  of  service  design,  new  APIs 
and  micro  services,  together  with  the  ever-maturing  infrastructures  for  service 

37 

 
 
 
orchestration  (aca  Enterprise  Service  Buses),  are  setting  the  tone  for  important 
challenges and needs from an interoperability point of view. 

In  the  European  context,  Software  and  Service  Technologies  can  be  considered  as  a 
common enabler for a wide variety of applications, supporting citizens and businesses to 
digitally interact with Public Administrations. 

Source: authors’ elaboration 

Figure 19 summarises the interoperability and user centricity assessment of Software and 
Service Technologies. The explanation of the evaluation is provided below in the relative 
subparagraphs. 

Source: authors’ elaboration 

Figure 19 - Software and Service Technologies Interoperability and User centricity assessment 

Regarding interoperability, organisational and semantic issues will be the most critical for 
the years to come. 

For  what  concern  legal  interoperability,  this  layer  is  not  considered  the  major  issue 
with  respect  to  the  development  of  standards  and  guidelines  for  the  next  years.  Work 
should  continue,  from  the  current  status  of  EIF  3.0,  for  making  legal  interoperability 
more  applied  than  more  designed  for  Service  Level  Agreements  (SLA’s)  for  the 
automated  provision  of  digital  public  services  through  open  API’s  in  an  area  that  will 
require  some  attention,  especially  considering  new  ways  of  providing  G2G  and  G2B 
services.  

A  critical  aspect  is  organisational  interoperability,  as  new  service  paradigms  should 
be developed, allowing also for new infrastructures and topologies to be integrated (e.g. 
with  respect  to  possible  Blockchain  applications  that  will  need  to  be  interoperable  with 
the existing software and service infrastructures). 

Very  important  semantic  issues  continue  to  be  a  challenge,  concerning  service 
categorisation,  EU-wide  service  catalogues  and  repositories,  converging  service 
typologies  for  all  member  states  and  the  needed  semantic  components  (controlled 
vocabularies and core components) for a vast number of entities.  

Technical  interoperability  is  still  a  major  issue,  but  also  a  well-tackled  one  over  the 
last  10  years.  Significant  effort  has  to  be  provided  both  towards  enhancing  the 
documentation  of  existing  service  models  and  reference  architectures,  as  well  as  for 

38 

 
 
allowing new paradigms and disruptive technological solutions to come into play, in the 
next years. 

Regarding  user  centricity,  Software  and  Service  Technologies  are  supportive  for  the 
application of the principles of universal design in several services and for the “never ask 
twice” principle. It helps to avoid that citizens and businesses are requested more than 
once for the same information, reducing loss in time.   

Software  Services  Technologies  can  be  considered  as  a  common  enabler  for  a  wide 
variety of applications, supporting citizens and businesses willing to digitally interact with 
Public  Administrations.  While  the  global  maturity  level  is  high,  the  top  may  not  have 
been  reached  in  the  European  public  sector  yet,  due  to  the  quite  high  variety  of 
components that are included. 

In general, Software and Service Technologies are largely aligned with the broad aim to 
make  digital  public  services  more  accessible,  personalised  and  secure.  Besides,  this 
technology  has  a  great  potential  influence  on  the  empowerment  of  citizens  and 
businesses to voice their views.  

From  the  analysis  carried  out  on  this  technology  area  and  considering  the  current 
European  context,  organisational  and  semantic  aspects,  such  as  service  categorisation 
and service  standardisation need to be improved, in order to guarantee interoperability 
of  Software  and  Service  Technologies  across  Member  States.  The  main  challenge  is  to 
develop  new  paradigms  and  allow  the  integration  of  new  infrastructures  and  service 
typologies. From a technological point of view, technical aspects are still a major issue, 
but also a well-tackled one over the last 10 years. 

3.4  Final considerations on the assessment of technologies 

The main results of the previous exercise are summarised in Table 4 below, in order to 
better  outline  which  technologies  score  lower,  in  terms  of  interoperability  and  user 
centricity and which layers seem to be the most critical.  

Source: authors’ elaboration 

Table 4 - Synopsis of Interoperability per technology area 

As shown in Source: authors’ elaboration 

Table 4, we can observe that Artificial Intelligence, Distributed Ledger Technologies, 
Digital Identity and Security, and Software and Service Technologies exhibit the 
lowest scores in terms of Interoperability.  

As  previously  outlined  in  subsection  3.3.5,  Distributed  Ledger  Technologies  show 
significant  limitations  in  being  interoperable  on  a  legal,  organisational  and  technical 
level;  challenges  are  respectively  linked  1)  to  GDPR  right  to  be  forgotten  and  the 

39 

TECHNOLOGY AREALEGAL  INTEROPERABILITY ORGANISATIONAL INTEROPERABILITY SEMANTIC INTEROPERABILITY TECHNICAL INTEROPERABILITY AVERAGEArtificial Intelligence 11221,5Communication Technologies34433,5Computing Infrastructures23222,3Distributed Ledgers Technologies11311,5Digital Identity and Security12221,8Immersive Technologies43433,5Internet of Things (IoT) and Smart Devices23222,3Software and Service Technologies31121,8AVERAGE2,12,32,52,1 
 
 
traditional 

inalterability of the Blockchain; 2) to the impact of the model of service provision - from 
to  openly  accessible,  distributed 
the 
infrastructures - that need to be revisited both from an organisational and technical point 
of view.   

trusted-server-based  model 

Digital  Identity  and  Security  shows  critical  results  in legal interoperability.  Currently 
this  technology  area  is  at  the  centre  of  an  ongoing  discussion  about  how  to  effectively 
achieve  strong  and  flexible  authentication  while  successfully  addressing  privacy  and 
security  concerns20.  Moreover,  even  if  protocols  were  developed  through  standard 
processes  and  released  openly,  technical  barriers  are  still  high;  standards  can  lead  to 
improved interoperability only if users, providers, and consumers start to adopt them.  

Similarly,  legal  challenges  raised  by  Artificial  Intelligence  refer  to  tensions  between 
protecting  rights  of  individuals,  personal  data  and  fostering  innovation  with  openness 
and  transparency.  In  addition,  it  has  relevant  limitations  with  regard  to  organisational 
interoperability  since  it  is  still  unclear  what  will  be  the  pan-European  services,  or  sub-
processes that will make use of the new possibilities of AI.   

Finally, Software and Service Technologies show critical scores in both organisational 
and semantic interoperability, since in the years to come, new service paradigms will be 
developed  requiring  new  infrastructures  to  be  integrated  and  EU  wide  vocabularies  and 
repositories will need to converge for all Member States.   

As  a  whole,  legal  and  technical  interoperability  pose  the  greatest  challenges  along 
the eight identified technology areas.  In fact, a solid legal and ethical framework at  EU 
level  is  fundamental  to  ensure  the  development  and  the  adoption  of  innovative 
technologies.  In  addition,  technical  interoperability  is  crucial  to  ensure  seamless 
information  exchange  and  free  movement  of  data;  however,  unless  semantic  is  fully 
standardised across Europe, this layer poses challenges. Technical interoperability could 
be enhanced through a greater reliance upon open standards making the environment of 
public/private  entities  and  citizens  more  interoperable.  In  addition,  organisational 
interoperability  also  presents  relevant  issues  when  dealing  with  commonly  agreed  and 
mutually  beneficial  standards  for  roles  and  processes  alignment  in  the  digital  public 
services delivery. Regarding semantic interoperability, it is possible to observe that this 
layer  is  likely  to  represent  a  lower  challenge  for  the  eight  technology  areas  even  if 
improvements  are  required.  For  instance,  in  the  case  of  electronic  invoicing,  many 
standards  and syntaxes are  still in use;  economic operators wishing to  carry out  cross-
border procurement  activities are often  required to comply with a new standard and/or 
codification system each time they access a new market. Therefore, a common European 
standard for the semantic data model of the core elements is needed to increase citizen 
uptake for any digital services delivered. 

Source: authors’ elaboration 

20 https://cyber.harvard.edu/interop/pdfs/interop-digital-id.pdf  

40 

TECHNOLOGY AREADigital Interaction Accessibility, security, availability and usabilityReduction of the administrative burdenDigital delivery of public servicesCitizen engagementIncentives for digital service useProtection of personal data and privacyRedress and complaint MechanismsAverageArtificial Intelligence33321341,52,6Communication Technologies3,53,53,53,53,53423,3Computing Infrastructures333332,53,533,0Distributed Ledgers Technologies22,52,5213412,3Digital Identity and Security3,53,53323423,0Immersive Technologies2,5322133,512,3Internet of Things (IoT) and Smart Devices3,5333,53,53,541,53,2Software and Services Technologies3,53,53,53,533433,4Average3,13,12,92,82,33,03,91,9 
 
                                           
Table 5 - Synopsis of User centricity challenges per technology area 

Another significant aspect to consider to fully exploit the technologies’ potentials and to 
allow the reuse of common IT solutions is user centricity. Moreover, the design of user-
centric  digital  Public  Administrations  is  growingly  seen  as  one  of  the  key  ingredients  of 
good governance. 

Source: authors’ elaboration 

Table  5  shows  how  the  analysis  of  the  different  technology  areas  highlights  positive 
trends that help to enrich user experience, accessibility and convenience and to  ensure 
data  protection  and  trust.  However,  citizen  engagement  and  redress  and  complaint 
mechanisms are still low for almost all the technologies.  

According  to  the  results  and  findings,  it  is  possible  to  observe  that  among  the  eight 
technology areas, Artificial Intelligence, Distributed Ledger Technologies  and  Immersive 
Technologies score the  lowest in terms of  user centricity whereas Software and Service 
Technologies,  Communication  Technologies  and  Internet  of  Things  and  Smart  devices 
register  the  highest  score.  The  reason  why  some  technologies  are  under  performing  is 
explained by the fact that there is still a lot of untapped potential to harness in order to 
make digital services more user oriented.  

In conclusion, as previously mentioned, EU governments at all levels still need to invest 
significantly in the exploitation of emerging and mature technologies, in order to provide 
seamless  digital  public  services.  They  must  adopt  a  proactive  attitude  to  better  tackle 
these  challenges  and,  while  introducing  a  digital  public  service,  all  the 
four 
interoperability layers have to be equally addressed and particular attention should also 
be dedicated to the eight user centricity principles. 

41 

 
 
 
 
4  Readiness concepts and levels 

This  section  provides  a  detailed  overview  of  the  approach  followed  in  the  Study,  to 
analyse the innovative pilot initiatives that harness digital technologies to improve public 
service  delivery.  In  a  nutshell,  the  approach  is  based  on  the  joint  consideration  of  4 
distinct  “readiness”  dimensions:  technological,  societal,  organisational  and  legal.  These 
are  presented  in  the  next  four  subsections,  forming  the  axes  of  a  composite  Maturity 
Model,  which  will  be  used  to  dynamically  assess  the  evolution  of  pilot  projects  across 
time, as explained in the last subsection. There, the approach is shown “in action”, with 
the help of an appropriate visual representation. 

4.1  Technology Readiness Level 

The  Technology  Readiness  Level  (TRL)  is  a  method  for  estimating  the  maturity  of  a 
certain  technology.  Being  based  on  readiness  (i.e.  closeness)  to  the  market,  it  has  the 
big  advantage  of  being  technology  neutral.  In  fact,  while  originally  invented  at  NASA 
headquarters in 1974 and consistently used within the aerospace industry until codified 
in the ISO 16290:2013 standard, it has become widespread in all technology sectors.  

The Technology Readiness Level (TRL) scale has been widely adopted at EU, national and 
regional levels in the current (2014-2020) programming period as a decision-making tool 
when  financing  Research,  Development  and  Innovation  investments  with  public  grants. 
Since  2014,  the  Technology  Readiness  Level  (TRL)  scale  has  become  part  of  the  EU 
Horizon 2020 Work Programmes and in many countries and regions of Europe has been 
widely adopted in the context of ERDF (European Regional Development Fund) supported 
Research,  Development  and  Innovation  investments.  TRLs  are  based  on  a  scale  from  1 
to 9, where 9 represents the most mature technology.  

The nine possible instances of TRL are reported in the Table below. 

MATURITY LEVEL 

DESCRIPTION 

TRL1 

TRL2 

TRL3 

TRL4 

TRL5 

TRL6 

TRL7 

TRL8 

TRL9 

Basic principles observed 

Technology concept formulated 

Experimental proof of concept 

Technology validated in lab 

Technology validated in relevant environment (industrially relevant 
environment in the case of key enabling technologies) 

Technology demonstrated in relevant environment (industrially relevant 
environment in the case of key enabling technologies) 

System prototype demonstration in operational environment 

System complete and qualified 

Actual system proven in operational environment (competitive 
manufacturing in the case of key enabling technologies, or in space) 

Source: authors’ elaboration 

Table 6 - TRL scale used in Horizon 2020 and ERDF 

42 

 
 
In  terms  of  readiness,  TRLs  1-3  pertain  to  the  initial  stages  of  any  Research  and 
Development project, while TRLs 4-6 belong to the validation and demonstration phases, 
which  can  be  done,  first,  in  a  closed  laboratory,  then  in  real  or  almost  real  world 
conditions.  TRL  7  is  the  final  stage  of  prototyping,  while  TRLs  8-9  constitute  the  usual 
pre-market and market launch conditions.  

However, this criterion has been subject to criticism, and one of the strongest objections 
has  been  related  to  the  non-linear,  cyclical  and  iterative  shape  of  technology 
development  processes,  the  TRL  scale  is  mostly  used  as  a  Research  and  Innovation 
Policy  tool.  The  aim  is  to  support  decision  making  when  financing  Research, 
Development and Innovation investments with public grants – rather than an ontological 
description  of  how  technologies  are  developed  and  rolled  out  in  the  respective  usage 
contexts. In this sense, it brings an invaluable help to clarify and specify the differential 
impact of the public grant on some, though maybe not all, of the sub processes leading 
to a certain progress along the TRL scale. 

4.2  Societal Readiness Level 

The SRL is an approach originally proposed by Innovation Fund Denmark21 and revised in 
this  Study  to  assess  the  level  of  societal  adaptation  of  a  certain  technology,  product, 
process, or intervention. The rationale is that any innovation – be it technical or social – 
must be integrated in the societal environment. Thus, the higher the SRL, the higher is 
such integration or the lower will be the need to set up ad hoc measures to promote “a 
realistic transition towards societal adaptation”.22 

The  SRL  is  analysed  through  the  readiness  of  the  society  to  adopt  the  solution.  The 
approach is technology neutral and, very importantly, there is no overlap with the TRL, 
which  makes  the  two  maturity  models  both  valid  and  rigorous  interpretative  lenses. 
According to its proposers, also the SRL has nine possible stages, which are reported in 
Table 7 (with slight changes from the original list)23. 

MATURITY LEVEL 

DESCRIPTION 

SRL1 

Identification of the generic societal need and associated readiness aspects 

SRL2 

SRL3 

SRL4 

SRL5 

Formulation of proposed solution concept and potential impacts; appraisal of 
societal readiness issues; identification of relevant stakeholders for the 
development of the solution 

Initial sharing of the proposed solution with relevant stakeholders (e.g. 
through visual mock-ups): a limited group of the society knows the solution 
or similar initiatives 

Solution validated through pilot testing in controlled environments to 
substantiate proposed impacts and societal readiness: a limited group of the 
society tests the solution or similar initiatives 

Solution validated through pilot testing in real or realistic environments and 
by relevant stakeholders: the society knows the solution or similar initiatives 
but is not aware of their benefits 

21 https://innovationsfonden.dk/sites/default/files/2019-03/societal_readiness_levels_-_srl.pdf  
22 ibidem 
23  Information about the society awareness and familiarity with the solution were formulated. 

43 

 
 
                                           
SRL6 

SRL7 

SRL8 

SRL9 

Solution demonstrated in real world environments and in co‐operation with 
relevant stakeholders to gain feedback on potential impacts: the society 
knows the solution or similar initiatives and awareness of their benefits 
increases 

Refinement of the solution and, if needed, retesting in real world 
environments with relevant stakeholders: the society is completely aware of 
the solution's benefits, a part of the society starts to adopt similar solutions 

Targeted solution, as well as a plan for societal adaptation, complete and 
qualified; society is ready to adopt the solution and have used similar 
solutions on the market 

Actual solution proven in relevant societal environments after launch on the 
market; the society is using the solution available on the market 

Table 7 - SRL scale (adapted from Innovation Fund Denmark) 

As shown in Table 7, the connection between TRL and SRL is very closed and direct if we 
think  of  the  target  “solution”  as  a  technological  one.  However,  the  approach  is  broader 
than that, as it encompasses new solutions that may not be technical – wholly or in part.  

In  the  above,  our  reading  of  SRLs  1‐ 2  is  that  they  reflect  the  growing  awareness  of  a 
research  Team  about  the  existence  of  a  societal  readiness  issue.  In  turn,  SRLs  3-6  are 
concerned with the more and more extended inclusion of societal stakeholders (such  as 
prospective users or other similar groups) in the testing, validation and demonstration of 
the Research and Development output. SRL 7 well matches TRL 7 in its being referred to 
the  final  stage  of  prototyping,  while  SRLs  8-9  belong  to  the  pre-market  and  market 
launch  phase  of  the  target  “solution”  –  a  phase  that  may  also  be  related  to  a  non-
commercial  situation,  as  can  evidently  be  the  case  for  a  newly  developed  digital  public 
service. 

4.3  Organisational Readiness Level 

While  the  TRL  and  SRL  scales  (particularly  the  former)  are  already  established  in  both 
literature  and  practice,  the  ORL  is  a  fully  original  approach  invented  here  for  the 
purposes  of  this  Study.  By  analogy  to  SRL,  ORL  must  be  seen  as  a  maturity  model 
related  to  the  organisational  impact  of  a  certain  technology,  product,  process,  or 
intervention. Following Eleanor D. Glor24, the term “impact” is defined to include both the 
results  of  an  innovation  (outcomes)  and  its  broader  effects  inside  the  perimeter  of  the 
organisation  implementing  it.  Key  impact  areas  span  from  professional  roles, 
competencies  and  skills 
functions,  processes  and  physical 
infrastructures.    Here,  we  are  more  interested  in  defining  the  preparedness  level  of  an 
organisation to receiving it, instead of analysing  the consequences of that innovation in 
terms  of  (for  e.g.)  improved  organisational  performance.  The  assumption  here  is  that 
any innovation – be it technical or social – requires being embedded in the organisational 
environment  to  become  permanently  adopted.  For  reasons  of  symmetry,  we  propose 
nine possible instances for the ORL, which are presented below. 

to  organisational 

24 https://pdfs.semanticscholar.org/8a6f/553006eda9aed6a4ea1b077374c37d5dcf98.pdf  

44 

 
 
 
 
                                           
MATURITY LEVEL 

DESCRIPTION 

ORL1 

ORL2 

ORL3 

ORL4 

ORL5 

ORL6 

ORL7 

ORL8 

ORL9 

Identification of the organisational need (infrastructures, capabilities, 
skills) and associated organisational readiness aspects 

Formulation of proposed solution concept and potential impacts; appraisal 
of organisational readiness issues; identification of relevant roles, 
processes, functions and structures for the solution 

Comprehensive description of proposed solution’s impacts within the 
organisation in terms of roles, competences and skills, physical 
infrastructures required 

Solution validated through simulation of major induced changes to 
substantiate proposed impacts and organisational readiness: the 
organisation which is developing the solution starts to acquire roles, 
competences and skills, physical infrastructures required 

Proposed solution validated through pilot testing in real or realistic 
organisational environments: the organisation which is developing the 
solution achieves roles, competences and skills, physical infrastructures 
required 

Solution demonstrated in real world environments and in co‐operation with 
relevant stakeholders to gain feedback in order to improve roles, 
processes, functions and infrastructures required 

Refinement of the roles, processes, functions and infrastructures required 
and retesting of the solution in relevant organisational environments 

Targeted solution, as well as a plan for organisational embedment, 
complete and qualified: roles, processes, functions and infrastructures are 
available 

Actual solution proven in relevant organisational environments: roles, 
processes, functions and infrastructures are correctly used for the solution 
on the market 

Table 8 - ORL scale (our elaboration) 

ORLs 1‐2 match SRLs 1-2 in reflecting the growing awareness of a research Team about 
the existence of an organisational readiness issue. In turn, ORLs 3-6 are concerned with 
the more and more extended consideration of roles, processes, functions and structures 
in the testing, validation and demonstration of the targeted output. ORL 7 matches both 
SRL  7  and  TRL  7  in  its  being  referred  to  the  final  stage  of  prototyping,  while  ORLs  8-9 
belong to the pre-market and market launch phase. 

4.4  Legal Readiness Level 

Like  ORL,  also  LRL  is  an  original  approach  invented  for  the  purposes  of  this  study. 
Conceptually, it is even less rooted than ORL in academic literature or practice. However, 
the  analogy  with  SRL  and  ORL  is  developed  enough  to  allow  the  formulation  of  a 
maturity model that looks at the legal and regulatory implications of innovations in terms 

45 

 
 
of compliance, but also transformative power. In fact, while no new technology, product, 
process, or intervention could survive if proven to go against the existing set of binding 
rules  that  govern  the  selected  domain,  the  opposite  is  also  true  (to  a  large  extent).  A 
legal system evolves over time, as a result of breakthrough innovations, bringing to the 
surface,  the  need  of  limiting  the  range  of  possibilities  or  configuring  new  spaces  of 
legitimate action. The nine possible instances for LRL are presented below in Table 9. 

MATURITY LEVEL 

DESCRIPTION 

LRL1 

LRL2 

Generic considerations of legal and ethical compliance aspects are 
observed but nothing has yet been done for the development of the 
solution 

Formulation of the need to enhance the legal normative, laws, rules and 
guidelines and solution concept; appraisal of legal and ethical compliance 
issues 

LRL3 

Abstract description of the proposed solution’s legal and ethical compliance 

LRL4 

LRL5 

LRL6 

LRL7 

LRL8 

LRL9 

Solution’s legal and ethical compliance prospects validated against any 
required or recommended changes in the legal and/or regulatory system 

Definition of the proposed solution’s legal and ethical compliance status 
after pilot testing in real or realistic organisational environments 

Detailed description of the required or recommended changes in relevant 
laws, regulations or organisational rules to ensure full compliance with the 
proposed solution 

Refinement of the solution within the existing legal and ethical system and, 
if needed, proposals for required or recommended changes to some 
aspects of it 

Targeted solution, as well as a legal and ethical compliance audit, 
complete, qualified and ready to be launched on the market 

Actual solution proven legally and ethically compliant after launch on the 
market 

Table 9 - LRL scale (our elaboration) 

4.5  Application to pilot evaluation 

The  application  of  our  4-axis  maturity  model  to  pilot  evaluation  is  exemplified  below  in 
Figure  20.  There,  “t”  stands  for  technology  readiness,  “l”  for  legal,  “s”  for  societal,  and 
“o”  for  organisational.  The  figure  shows  an  idealistic  example  of  a  trial  run  between  t0 
and  t1  in  a  certain  location,  where  both  technology  and  organisational  readiness 
progressed  considerably,  while  legal  and  societal  readiness  observed  a  slower  increase 
across time. Note that the TRL improvement is represented by the size of the small ball 
inside  the  cube,  which  becomes  bigger  between  t0  and  t1,  while  the  LRL,  SRL  and  ORL 
changes are all measured along the respective axis as shown. 

46 

 
 
The  global  change  occurred  in  the  four  dimensions  above  reflects  what  we  will  call  the 
“transition pathway” of the pilot. 

Source: authors’ elaboration 

Figure 20 - Framework in action and transition pathway 

The Readiness Levels, briefly exposed in this section, are very relevant in the context of 
the European public sector. Ideally, public services must be built on proven technologies 
(hence with high TRLs). However, it is also particularly important that they are used and 
accepted  by  the  targeted  audience  (hence  the  need  of  a  high  SRL)  and  that  business 
processes in the Public Administration are ready or flexible to implement the technology 
(hence the concern for a high ORL). Finally, although the private sector may quite often 
challenge existing laws when proposing new products and services (if not operate at the 
margin), any governmental service is tied to operate strictly within the boundaries of the 
law (hence the importance of a high LRL). 

47 

ORLLRLSRL(l1, s1, o1), t1(l0, s0, o0), t0 
 
 
 
 
 
 
 
 
 
 
5  Knowledge repository 

Taken  together,  the  methods  presented  in  section  3 
for  the  assessment  of 
interoperability and user centricity and the 4-axis readiness concept and model outlined 
in section 4  constitute  an original multi-dimensional framework to evaluate the success 
and  scalability  or  reuse  potentials  of  pilot  projects  aimed  to  experimentally  deploy 
innovative public services at the national, regional or local levels. 

As mentioned in the  introduction, the  framework was hosted in an interactive database 
as  knowledge  repository,  making  it  possible  to  filter  data  according  to  the  pilot 
catchment area, its readiness levels, its reference country, its starting or ending testing 
phase, and allow the desired data to be displayed. Generally speaking, the database was 
conceived  of  and  designed  to  make  sure  that  it  could  be  easily  used  and  constantly 
updated,  also  by  future  studies,  while  preserving  the  methodological  lessons  learned  in 
the  form  of  technical  and  operational  guidelines  and  be  the  foundation  of  the  possible 
development  of  an  Innovative  Public  Services  Observatory  to  be  developed  by  JRC  and 
DIGIT as a follow-up of this preparatory study. 

5.1  Metadata  

Within the database, gathered information for pilot analysis can be accommodated in 36 
columns. Some of these  are to be filled in with qualitative information whereas others 
with  quantitative  data,  such  as  the  scores  assigned  to  observed  dimensions  (namely 
TRL,  LRL,  SRL,  ORL  and  interoperability.  User  centricity  principles  are  flagged  when 
relevant to a pilot).   

The table below lists the metadata of the 16 qualitative columns.  

Column Name  

Description 

Name of the 
project  

Shows the short name of the project. 

Accepted/Rejected 
for the analysis 

Indicates if the case  study  has been considered for the project analysis. A 
pilot is "Accepted" for the analysis if it satisfies 5 specific requirements: (1) 
First and foremost it has to be applicable in the Public Administration; then, 
(2) it has to be a European project or an extra EU best practice; (3) it must 
use one of the 8 identified technologies; furthermore, (3) the project needs 
to  be  closed  or  ready  to  be  launched;  finally,  (5)  information  and 
documentation  need  to  be  available  to  proceed  with  the  pilot  evaluation.  
If at least one of the criteria of the high-level analysis is missed, the pilot is 
"rejected" for the analysis. 

Description of the 
project 

Reports in depth what the project was about, including identification of the 
technology vendor if available, the public service involved, the benefits and 
issues etc. 

Goals of the 
project 

Explains briefly the goal(s) of the pilot. 

Technology Area 

The extent of a determined technology enclosed within a specified boundary 
that can be subdivided into categories and uses/applications. 

Technology 

Individual  exhaustive  division  of  the  technology  area  according  to  the 

48 

 
 
Category 

proposed system of classification. 

Government 
broad objective 
(COFOG) 

The  Classification  of  the  Functions  of  Government  (COFOG) 
is  a 
classification  defined  by  the  United  Nations  Statistics  Division,  designed  to 
be a standard to indicate the government divisions of different countries. 

Sub-items 
(COFOG) 

Represents the single Activity Domain covered by the pilot. 

Target groups 

Group / entity the pilot / project is developed for. 

Reference country 

Country  where  the  pilot  is  implemented.  Most  of  times  the  country 
corresponds to the coordinator of the project. 

Catchment area 

Determines  the  wide  view  of  the  project,  considering  the  people  caught 
during the pilot or development of the programme. 

Turned into PS 

It  indicates  if  the  pilot  and  the  project  has  turned  or  not  into  a  public 
service. 

Starting period 

Year when the pilot/project started. 

Ending period 

Year when the pilot/project finished. 

Project source 

The link of the project website. 

List of sources 
used for the 
contract 

URL of sources used for the contract. 

Table 10 - Metadata of the qualitative columns in the database  

As  can  be  noted,  the  catalogue  of  cases  assigns  them  to  the  respective  public  sector 
domain following the COFOG, the Classification of the Functions of Government defined 
by  the  United  Nations  Statistics  Division,  to  facilitate  the  classification  of  digital  public 
services in thematic areas25. 

The table below lists the metadata of the 20 quantitative columns. More specifically:  

  8  columns  refer  to  the  4  readiness  criteria  (TRL,  SRL,  LRL  and  ORL)  with  the 
respective levels recorded both at the beginning of the pilot phase (t0) and at the 
end of it (t1) to monitor the evolution of the project across time;  

  8  columns  are  related  the  8  user  centricity  principles  mentioned  in  the  Tallinn 

declaration, to be flagged if/when relevant for the pilot at hand; 

  4  columns  point  at 

legal 
interoperability  layers,  each  to  be  attributed  a  score  depending  on  observed 
status, as explained in section 3 above.   

technical,  semantic,  organisational  and 

the 

25 https://unstats.un.org/unsd/classifications/Family/Detail/4  

49 

 
 
 
 
                                           
Column Name 

Description 

READINESS 
ANALYSIS 

Section where the results of the readiness analysis are collected. 

It uses a 1-9 scale and shows the initial and final status of the pilot 
in terms of maturity with respect to technical, organisational, societal 
and legal aspects in the country where the project has been carried 
out. 

TRL Level before 

The Technological Readiness Level when the project started. 

TRL Level after 

The Technological Readiness Level at the end of the pilot. 

SRL Level before 

The Social Readiness Level when the project started. 

SRL Level after 

The Social Readiness Level at the end of the pilot. 

ORL Level before 

The Organisational Readiness Level when the project started. 

ORL Level after 

The Organisational Readiness Level at the end of the pilot. 

LRL Level before 

The Legal Readiness Level when the project started. 

LRL Level after 

The Legal Readiness Level at the end of the pilot. 

INTEROPERABILITY 
ANALYSIS 

Section  where  the  results  of  the  interoperability  analysis  are 
collected. 

It  uses  a  1-4  scale  and  indicates  if  the  case  study  undercovers  the 
interoperability  solutions  for  Public  Administrations,  businesses  and 
citizens, according to the 4 layers of the EIF Interoperability Model. 

Legal 
Interoperability 

The  Legal  Interoperability  score  of  the  pilot  based  on  a  1-4  level 
scale (looking at legal framework, policies etc.)  

Semantic 
Interoperability 

The Semantic Interoperability score of the pilot based on a 1-4 level 
scale (looking at syntactic and semantics aspect such as format etc.)  

Organisational 
Interoperability  

The Organisational Interoperability score of the pilot based on a 1-4 
level scale (looking at business, processes, roles etc.) 

Technical 
Interoperability 

The Technical Interoperability score of the pilot based on a 1-4 level 
scale  (looking  at  interface,  specifications,  interconnection  services 
etc.) 

Section  where  the  results  of  the  user  centricity  analysis  are 
collected. 

USER CENTRICITY 
ANALYSIS 

It  measures  the  level  of  user  centricity  on  the  8  well-defined  user 
centricity  principles.  According  to  the  Tallinn  Declaration,  user 
centricity  means  putting  the  end-users  (citizens,  businesses,  public 
sector employees) truly at the centre of services through the spread 
of 
areas.  
User  centricity  principles  have  been  tagged  when  relevant  with  the 
objective and aim of the pilot.  

digitisation 

across 

policy 

all 

50 

 
Digital Interaction 

Digital  Interaction  principle  identified  and  flagged  when  relevant  to  a 
pilot.   

Accessibility, 
security, availability 
and usability 

Accessibility,  security,  availability  and  usability  principle  identified  and 
flagged when relevant to a pilot.   

Reduction of the 
administrative burden 

Reduction  of  the  administrative  burden  principle  identified  and 
flagged when relevant to a pilot.   

Digital delivery of 
public services 

Digital  delivery  of  public  services  principle  identified  and  flagged 
when relevant to a pilot.   

Citizen engagement 

Citizen engagement identified and flagged when relevant to a pilot.   

Incentives for digital 
service use 

Incentives for digital service use identified and flagged when relevant 
to a pilot.   

Protection of personal 
data and privacy 

Protection  of  personal  data  and  privacy  identified  and  flagged  when 
relevant to a pilot.   

Redress and 
complaint Mechanisms 

Redress  and  complaint  Mechanisms  identified  and  flagged  when 
relevant to a pilot.   

Table 11 - Metadata of the quantitative columns in the database  

5.2  Structure  

The database has been realised with the use of MS Excel. As mentioned, it was meant to 
be  as  interactive  and  user-friendly  as  possible.  5  distinct  sections  (guidelines,  pilot 
selection, cases analysis, pilot coverage, data entry) were created to ensure an intuitive 
and simple navigation for users, as  shown in Figure 21 – Database structure overview 
1. 

Source: authors’ elaboration 

Figure 21 – Database structure overview 

Within the “Guidelines” section, a general guidance on how to use the framework (“user 
manual”  on  the  use  of  the  worksheet),  and  an  overview  of  what  the  other  sections 

51 

 
 
 
contain is provided. Moreover, it is possible to consult the dictionary of the data columns 
inserted  for  the  pilot  analysis  as  well  as  the  scoring  classification  assigned  for  the 
parameters used for the assessment of pilots.  

In  the  “Data  entry”  section,  users  can  insert  the  information  available  on  the  pilot  at 
hand. 4 sections  can be filled with data input: 1)  pilot basic information  such as the 
name  of  the  project,  the  country,  and  the  technologies  used;  2)  the  readiness  levels 
i.e.  a  score  assigned  to  each  dimension  both  before  and  after  the  pilot’s  execution;  3) 
the  interoperability  layers  with  a  score  entered  for  each  of  them;  and  4)  user 
centricity principles inviting to select those that the pilot seems to satisfy.  

Source: authors’ elaboration 

Figure 22 - Data entry interface 

The  “Case  analysis”  section  shows  all  the  available  information  on  a  certain  pilot 
project.  The  section  maps  and  records  all  the  information  and  allows  the  user  to 
elaborate that information and edit the related analyses.  

In  the  “Pilot  Selection”  sheet,  users  can  filter  data  about  any  pilot  stored  in  the 
database,  according  to  the  value  or  qualitative  data  they  want  to  select.  The  slicers 
include,  e.g.:  the  reference  country  of  the  pilots,  its  starting  or  ending  period,  its 
readiness  criteria  before  and  after,  its  interoperability  level,  and  the  number  of  user 
in  Figure  23  and  Figure  24
shown 
centricity  principles 

satisfied,  as 

Source: authors’ elaboration 

Figure 23 - Data filter according to project ending in 2017 

52 

 
 
 
Source: authors’ elaboration 

Figure 24 - Data filter according to ORL “after” or in “t1” equal to 8 

In the “Pilot coverage” section, the system allows to synthetize and visualise inputted 
data  for  easier  understanding.  For  instance,  it  can  provide  a  representation  of  the 
technology categories and countries covered by the projects collected, as depicted below 
in Figure 25 and Figure 26.  

Source: authors’ elaboration 

Figure 25 - Technology coverage by collected pilots 

53 

 
 
 
Source: authors’ elaboration 

Figure 26 - EU and extra EU country coverage by collected pilots 

54 

 
 
 
6  Pilot analysis and evaluation 

Based on the multi-dimensional framework and database described in the previous sections, the process of analysis and evaluation of each pilot 
project considered relevant for the Study is composed by 8 sequential stages that are represented below in Source: authors’ elaboration 

Figure 27. 

Source: authors’ elaboration 

Figure 27 - Process of pilot analysis and evaluation 

55 

 
 
In the following paragraphs, those eight stages are explained in detail: 

  Desk Research: during this phase, relevant documentation was gathered and used 
as evidence base for the identification of cases. The main sources included European 
Commission’s  strategic  documents  and  official  communications,  studies  and  reports 
from International Organisations and private/global consulting firms including KPMG, 
academic  papers  from  leading  researchers  and  universities,  and  project  collections 
such as the Interreg and CORDIS databases.  

  High-level analysis of sources: during this phase, five specific conditions were to 
be verified upfront for the pilot to be retained. 1) First, it had to be applicable to the 
public  sector;  then  (2)  it  should  be  either  a  European  project  or  an  extra-EU  good 
practice example; afterwards (3) one of the  8  identified technology areas had to be 
represented; furthermore, (4) it was checked if the project  was closed or near to be 
finalised. This is because it was crucial to observe the evolution of  the pilot from an 
initial (t0) to a final point (t1). Finally, (5) enough information and documentation had 
to be available to proceed to a more in-depth analysis.  

  Pilot rejected from the analysis: based on the outcome of the previous stage, all 
the  pilots  that  did  not  meet  one  or  more  of  the  previous  five  conditions  were  not 
retained in the case study database. 

  Pilot accepted for the analysis and basic information  assessment: if all the 5 
criteria/conditions were satisfied, the pilot was considered "accepted" for the analysis 
and some basic information was collected and stored into the database; this included 
the description and goal of the project, the technology area and related category, the 
government’s  broad  objective,  the  target  group,  reference  country,  catchment  area 
and finally the pilot’s starting and ending time.  

  Readiness analysis: at the two points of time (t₀ the pilot’s start and t₁ the pilot’s 
end)  each  was  scored  under  the  4  criteria  of  technical,  societal,  organisational  and 
legal  readiness,  based  on  a  scale  of  values  from  1  to  9,  where  1  represents  the 
lowest and 9 the highest possible level. 

  Interoperability  analysis:  each  pilot  was  also  scored  under  the  4  interoperability 
layers (legal, organisational, semantic and technical), based on a scale of values from 
1  to  4,  where  1  denotes  the  highest  level  of  effort  that  should  be  exercised  in 
ensuring interoperability, whereas 4 implies that no particular issues should be taken 
in mind.  

  User centricity assessment: finally, the pilot was tagged according to which of the 

eight user centricity principles are represented in it.  

  Mapping  on  the 

readiness, 
interoperability  and  user  centricity  analyses  were  collected  and  inserted  in  the 
database to be mapped on the framework.  

framework:  All 

ratings  deriving 

from 

the 

Generally speaking, utmost attention was paid to the identification of pilot projects that 
could be easily shared and reused all over the EU,  including also some extra-EU stories 
that  could  be  considered  as  “good  practice  examples”  in  order  to  allow  a  clearer  and 
broader view of the current scenario. As mentioned, the Study team focused the analysis 
on both “closed and delivered” and “near to be delivered” projects, which may have been 
a  source  of  bias  towards  more  mature  and  consolidated  technologies.  The  assessment 
included  both  low  and  large-scale  pilots  and  covered  most  European  countries, 
technology  areas  and  categories.  More  specifically,  150  pilots  from  24  European 
countries  were  pre-  selected,  covering  all  the  eight  technology  areas  previously 
identified.  From  them,  80  pilots  were  “accepted”  for  deeper  analysis.  They  are 
visualised in the map below. 

56 

 
Source: authors’ elaboration 

Figure 28 - Representation of successful pilots across technology areas and European countries 

Within the 80 “accepted” pilots, the in-depth analysis allowed to identify the successful and unsuccessful cases. Generally speaking, a pilot 
was considered successful when it showed a homogenous growth of all four Readiness Levels, each of them nearing the top score in t1 (7-8 or 
higher),  demonstrated  good  interoperability  levels  (2-3  or  higher)  and  fulfilled  a  majority  (5  or  more)  of  user  centricity  principles.  Based  on 
such  evidence,  31  “successful”  pilots  were  identified  as  exemplary  and  supportive  of  the  digital  transformation  of  public  services,  with 
solutions that can be shared and reused across Europe (see Figure 30). 

The chart below (Figure 29) shows the detailed conditions to be satisfied by the successful pilots. 

57 

 
Source: authors’ elaboration 

Figure 29 - Criteria satisfied by successful pilots for future implementation 

The  sequence  of  evaluation  tasks  entails  4  consecutive  steps:  1)  Evaluation  of  generic  information,  2)  Readiness  analysis,  3)  interoperability 
analysis and 4) User centricity analysis. 

First,  it  must  be  ensured  that  the  launch  of  the  pilot  occurred  after  the  year  2011  and  end  after  2014,  to  prevent  it  from  being  considered 
“outmoded” or even “obsolete” (STEP 1). Second, successful pilots must show a TRL and LRL “after”26 higher than or equal to 7, and an ORL 
and SRL that increase simultaneously across time, bringing the score higher or equal to 8 (STEP 2). Otherwise, one might think that the pilot 
has had some malfunction in terms of legal, organisational or societal readiness. 

26 “after”: at the end of the pilot testing phase 

58 

 
 
                                           
STEP  3  focuses  on  the  four  interoperability  layers.  Successful  pilots  must  score  equal  to  or  above  2  for  the  technical,  semantic  and 
organisational layer, and equal to or above 3 for the legal one. Below that threshold, specific efforts should address interoperability challenges 
to allow an easy and effective sharing and reuse of the pilot solution. Finally, on the user centricity assessment (STEP 4), if the pilot satisfies at 
least 5 of the 8 principles, it can be considered an inspiring example for a possible replica or adoption (scaling out) in the European public sector 
context.   

Source: authors’ elaboration 

Figure 30 - Coverage and technologies used in the successful pilots 

59 

 
 
 
7  Pilot shortlisting 

Using the approach presented in the previous  section  and after consulting  with external 
experts and international researchers in the field of public sector’s digital transformation, 
the 31 successful projects were not only identified and extracted from the database, but 
also confirmed in their capacity to be transferred or reused at a larger scale. As Figure 30 
shows,  these  projects  are  representative  of  diverse  technology  applications  and  are 
located in 12 EU and 1 extra EU countries. Two of them, in agreement with the European 
Commission,  were  selected  for  a  more  in-depth  analysis  in  terms  of  readiness, 
interoperability and user centricity aspects: 

  Dublinked  –  the  Open  Data  store  for  the  Dublin  Region,  the  aim  of  which  is  to 
publish  quality  open  data,  foster  a  culture  of  knowledge  sharing  and  spur 
innovation  by  speeding  up  the  advancement  of  services  based  on  publicly 
available data.  

  IO App (literally “I App”)  – a project of the  Italian Digital Transformation Team, 
the goal of which is to increase the readiness of citizens and businesses to interact 
digitally  with  the  Public  Administration  and  make  it  easier  for  them  to  digitally 
manage their personal data, held by public authorities.  

For  both  of  them,  the  team  collected  additional  information  also  thanks  to  the 
collaboration  of  the  Project  Owners,  which  allowed  to  significantly  deepen  the  analysis, 
identify  possible  threats  and  future  opportunities  and  give  more  value  to  the  results 
obtained.  

The detailed results of the evaluation of the two shortlisted case studies are presented in 
the next subsections. 

7.1  Dublinked  

Dublinked: The Open Data Store for the Dublin Region 

Pilot overview 

Dublinked is an example of quality open data that is globally positioning the Dublin 
Region as a leader. Currently it is part of Smart Dublin an initiative aimed at using 
smart technologies, engage researchers, developers and citizens to solve challenges 
and improve city life.  

Within  Dublinked  Open  Data  Portal,  4  Dublin  Local  Authorities  (Dublin  City 
Council,  South  Dublin,  Fingal  and  Dún  Laoghaire  Rathdown)  can  share  data 
about  the  region  and  anyone  can  load,  use  and  reuse  data  to  spur  innovation  and 
economic development.  

Dublin  City  Council  (DCC),  one  of  the  4  Local  Authorities,  took  the  lead  of  the 
initiative  and  coordinated  the  whole  work  group  composed  by  the  3  Local 
Authorities,  Maynooth  University  (MU)  and  IBM.  The  partnership  between 

60 

 
 
 
 
entities from the private and public sector has been pivotal for Dublinked success, 
thanks to the combination of different skills and background. MU’s participation as a 
neutral independent party revealed to be crucial since it contributed to solve most of 
the issues and challenges faced in the start-up process. Besides, IBM was important 
to  ignite  the  initiative  thanks  to  its  data  understanding  such  as  the  identification  of 
most ‘valuable’ types of data to collect and release. More recent support is provided 
by Derilinx, a local Irish SME. 

Dublinked’s  aim  is  to  make  data  about  public  provisions  available  to  a  wider 
audience  (e.g.  water,  transport  and  infrastructures,  government  and  participation, 
population  and  communities,  art  and  culture,  environment  and  energy,  planning, 
public  health  and  safety),  to  enable  cross-sectoral  collaboration  and  promote  new 
data-driven businesses and innovation around the Dublin city region.  

Moreover, thanks to the potential of Open Data, Dublinked has enabled to contribute 
in  improving  transparency  and  efficiency  of  Government  and  created  potential  for 
business innovation and improvements in policy, to date.  

This  pilot  distinguishes  itself  from  other  open  data  initiatives  by  proactively 
supporting the creation of new communities of innovators and promotes the release 
of complex (yet potentially more valuable) types of datasets based on data streams 
and  live  feeds.  In  fact,  within  Dublinked,  anyone  can  download,  use,  reuse  and 
create something new from available data. The main objectives and benefits are to: 

  Make  the  city  of  Dublin  a  global  promoter  of  quality,  transparent  and 
standardised open data, as well as to foster a culture of knowledge sharing 
and civic engagement; 

  Improve  economic  development  by  speeding  up  the  advancement  of 

services based on data and enabling test bedding of new services; 

  Exploit  and  utilise  public  data  to  generate  new  revenue  streams  and 

address regional challenges; 

  Create a pool of high-value data for research purposes and to give Dublin-

based companies a significant advantage in this rapidly growing area. 

Like  EC,  Dublinked  uses  the  Creative  Commons  Attribution  (CC-BY)  licence. 
This  license  lets  others  distribute,  remix,  tweak,  and  build  upon  data,  even 
commercially, as long as users credit the original publisher for the original creation. 

Currently,  Dublinked  portal,  being  a  pioneer  in  the  field  of  open  data  portal  at  a 
regional  level,  is  harvested  to  the  Irish  National  Open  Data  Portal  data.gov.ie, 
that  promote  the  publication  of  datasets,  in  line  with  agreed  standards,  facilitating 
accessibility, re-use and interoperability. 

Before  the  launch  of  Dublinked,  many  cities  launched  open  data  initiatives  such  as 
online platforms where urban data was made available and was freely accessible. A 
key challenge in these cases was to turn raw data (such as planning application data, 
transport  movements,  water  flows)  into  useful  applications  that  improve  efficiency, 
quality and transparency of urban services.  

In  order  to  win  this  challenge,  Dublinked  logical  architecture  combines  an  Open 
Data  platform  with  a  network  of  innovative  solutions  based  on  data 

61 

 
visualisations,  app  for  smartphones,  data  downloads,  data  requests  received  by 
users and the release of open data. The collaboration with citizens, universities and 
lead  firms  in  this  field  helped  to  ignite  the  initiative,  showing  the  relevance  of  the 
data and the opportunity for the city. 

Technology Area 

Technology Category 

Software and Service Technologies 

Computing Infrastructure 

Internet of Things and Smart Devices 

APIs, Web Services, Microservices 
including Registries and Marketplaces 

Cloud Computing 

IoT Platforms 

Country 

Period 

Catchment area 

Ireland 

1st phase: 2011 - 
2016 

2nd phase: 2016 - 
ongoing 

Pilot Analysis 

Readiness of the pilot 

Regional 

User centricity principles satisfied 

62 

 
  
 
 
 
 
 
 
 
 
 
 
Digital 
interaction 

Accessibility, 
security, 
availability and 
usability 

Reduction of 
the 
administrative 
burden 

Digital Delivery 
of public 
services 

Citizen 
engagement 

Incentives for 
digital service 
use 

Protection of 
personal data 
and privacy 

Redress and 
complaint 
Mechanisms 

Focus on interoperability 

Legal  
interoperability 

Organisational 
interoperability 

Semantic  
interoperability 

Technical  
interoperability 

3 

3 

3 

3 

Benefits & Impacts 

In  the  following  paragraphs,  the  main  benefits  and  impacts  related  to  user 
centricity and interoperability analysis are reported. 

Regarding  user  centricity,  a  central  feature  of  Dublinked  is  user  involvement,  in 
fact  users  have  always  been  put  at  the  centre  of  Dublinked’s  implementation  and 
design, to speed up the processes, improve the overall performance of initiative and 
promote new co-operation networks around open data.  

Looking at the analysis on the eight user centricity principles, Dublinked satisfies 
nearly all of them apart from redress and complaint mechanisms.  

Regarding  interoperability,  Dublinked  is  a  regional  data  sharing initiative  that  has 
made unreleased public operational data available online for others to research and 
reuse,  thus  integrating  interoperability’s  aspects  right  from  the  beginning  of 
its  realisation.  The  project  has  been  strongly  influenced  by  the  European 
Interoperability  Framework  in  particular  with  regard  to  organisational,  semantic 
and technical interoperability aspects. 

  Legal Interoperability: Dublinked is fully compliant with European Privacy 
legislation  (GDPR),  PSI  and  Inspire  European  directive.  Dublinked  has 
introduced  some  relevant  advancements  regarding  Legal  Interoperability 
aspects  that  allowed  operation  in  a  cross-country  level,  for  instance  Smart 
Dublin  is  currently  responsible  for  coordinating  a  common  approach  to  data 
collection and publication. Moreover, the Dublinked team also participated on 
the Irish Open Data Advisory Group, which agrees on a National approach 
to the transposition of the EU Open Data Directive, including the identification 
of  high-value  datasets,  and  the  focus  on  the  publication  of  real-time  / 
dynamic datasets. 

  Organisational 

composed  by  4 
Municipalities and each of them have different approaches and architectures 
for  extracting,  managing  and  using  data.  In  fact,  South  Dublin  County 

Interoperability:  Dublinked 

is 

63 

 
 
 
 
 
 
 
 
 
 
 
Council  uses  ArcGIS  and  a  harvester  is  used  to  extract  data  to  the 
Dublinked  platform.  An  FME  is  used  to  extract  data  from  Dun  Laoghaire 
Rathdown  Council.  This  implies  that  the  organisational  interoperability  of 
Dublinked  has  still  not  reached  its  peak  of  performance  requiring  therefore 
some improvement.  In  particular, the standardisation of source systems and 
sources  processes  would  help  to  increase  organisational  interoperability. 
However,  some  efforts  have  been  made  to  introduce  advancements  in 
organisational  interoperability  and  promote  collaboration  between  the  4 
Municipalities:  the  Smart  Dublin  team  meet  regularly  to  set  out  common 
goals  and  review  existing  programmes  and  work  together  to  coordinate 
common  projects  (i.e.  the  SBIR  (Small  Business  Innovation  Research) 
initiative supports innovative projects that will impact all 4 Local Authorities).  

  Semantic Interoperability: any data associated to an Open Data License is 
considered Open Data by default. However, the data format is crucial for the 
exchange of data and can have significant implications for the usability of the 
data.  Within  Ireland  Open  Data  Initiative,  the  idea  is  to  publish  data  in  the 
most open way possible and at a minimum 3 Star such as CSV, JSON or XML. 
According  to  the  5-star  deployment  scheme  for  Open  Data  which 
measures the openness of the formats, most datasets published on Dublinked 
are  scored  3  Stars  out  of  5  on  the  openness  scale,  and  most  come  in 
multiple formats. The Dublinked portal is DCAT-compliant, meaning that all its 
metadata  is  available  in  a  standardised,  semantic  format.  The  team  is 
currently  reviewing  the  recently  launched  DCAT  v2,  for  its  adoption  into 
Dublinked. In particular, the inclusion of Web Service metadata is of interest. 
Regarding advancement about improving semantic interoperability, Dublinked 
had been part of the Open Data Advisory Group for the definition of the Open 
Data  Technical  Framework  national  project  to  identify  common  vocabularies 
and  help  standardise  datasets  (metadata,  description  tags,  schema,  field 
names etc.) to be used in the publication of Irish datasets. 

  Technical  Interoperability:  no  critical  intervention  is  needed.  Dublinked 
uses  open  specifications,  welcoming  the  publication  of  the  Open  Data 
Licenses (Creative Commons, also used by the National Open Data Platform 
data.gov.ie)  and  complies  to  the  INSPIRE  Directive  which  includes  key 
technical  interoperability  arrangements.  However,  some  efforts  need  to  be 
done,  since  the  4  municipalities  have  different  approaches  and  architectures 
for  managing  data.  Geospatial  data  is  harvested  from  an  ArcGIS  system  in 
South  Dublin  County  Council  to  the  Dublinked  platform,  while  FME  ETL 
processes  are  put  in  place  to  extract  data  from  Dun  Laoghaire-Rathdown 
County  Council.  Infrastructure  and  processes  have  been  put  in  place  to 
improve  organisational  interoperability,  which  are  continually  being  built  on. 
For  example,  a  Dublinked  AWS  infrastructure  is  now  available  to  all  4  Local 
Authorities  to  publish  real-time,  dynamic  or  large  datasets.  Moreover  to 
ensure  successful  data  integration  with  other  players’  sources  (linked  data) 
and  ensure  a  smooth  exchange  between  all  the  interesting  parties,  the  idea 
would  be  to  reach  the  maximum  rating  according  to  the  5  star-deployment 
scheme  and  ensure  a  uniform  approach  for  all  the  players  that  feed  the 
platform. 

64 

 
Main impacts and benefits: 

Dublinked has enabled the realisation of an open, machine readable platform fed by 
a  diverse  range  of  municipal  datasets  deriving  from  4  local  authorities,  which  could 
be  used  by  research  and  developer  communities  and  wherein  several  apps  and 
initiatives  developed  out  of  data.  Moreover,  Dublinked  has  allowed  the  creation  of 
new  communities  of  innovators  around  open  data  in  Dublin  and  Ireland  and 
increased  the  coordination  between  public  authorities,  and  increased  awareness 
about  the  value  of  open  data  in  society.  Moreover,  the  datasets  were  previously 
stored  on  enterprise-based  servers,  but  now  they  are  increasingly  stored  on  the 
cloud meaning they can reconfigure and react quickly.  

Risks & Challenges 

The  main  risks  and  possible  threats  Dublinked  could  face  in  future,  are  reported 
below: 

  Absence of revenue streams;  

  Cyber-security,  datasets  may  be  hacked  into,  or  shut  down.  If  services  or 
programs  are  dependent  on  the  data  (e.g.  through  apps)  they  would  be  let 
down; 

  Open data platform might provide a ‘back door’ into network or local authority 

systems; 

  Changing  priorities  and  personnel  in  Local  Authorities  which  can  increase  or 

decrease value placed and resources given to open data. 

The main challenges Dublinked could transform into opportunities:  

 

 

Invest in an effective software to prevent cyber-attacks; 

Invest resources to create a sustainable revenue model stream; 

  Accelerate  the  pace  in  order  to  contribute  in  recasting  the  EU  PSI  Directive 
which  promotes  ‘open  data  by  default’  and  data  sharing  by  private 
companies; 

  Harness  the  underlying  potential  of  IoT  devices,  APIs,  real  time  insights,  to 

scale services quickly; 

  Provide data available instantly to provide quicker response times; 

  Harness the potential of Linked data. 

Table 12 – Summary of Dublinked Analysis 

65 

 
 
 
 
7.2  IO App 

IO,  the  project  to  develop  the  public  services  app  -  Italian 
Digital Transformation Team 

Pilot overview 

The  IO  App  is  an  open  source  project  and  is  considered  an  important  pillar  of  the 
Italian  government's  vision  of  digital  citizenship.  Currently,  it  is  also  part  of  Co-
VAL best practices.  

Within the IO App, the Digital Transformation Team in collaboration with AgID 
which  objective  is  the  implementation  of  the  article  64  bis  of  the  Italian  Digital 
Administration Code27, is developing, since 2017, the “IO App” in order to provide 
the  main  functions  necessary  for  the  interaction  between  PA  and  citizens.  Thanks  to 
the  integration  with  other  enabling  platform  like  SPID,  PagoPA,  ANPR  and  the 
reliance  on  the  new  API  interoperability  model  and  on  the  various  tools  and 
guidelines of the Designers Italia and Developers Italia communities. 

The  new  interoperability  model  represents  a  cornerstone  of  the  3-year  IT  Plan 
designed to guarantee  the  functioning of the  entire  Information  System  of the 
Public Administrations. 

The  aim  of  IO  App  is  to  create  a  unique  interface  to  public  services  to  provide 
personalised and tailored services to each citizen wherein: 

 

 

they  can  receive  all  the  messages  of  the  Public  Administration  on  their 
smartphone,  manage  them  in  an  archive;  always  be  updated  on  deadlines  and 
manage alerts in an "integrated" mode; 
receive  and  store  documents,  receipts,  certificates  directly  on  their 
smartphone and share them with a public office in a few clicks;  

  make  digital  payments  to  the  Public  Administrations  through  PagoPA  channel, 
saving the preferred payment methods (credit cards and soon debit cards, PayPal, 
etc.); 

  pay any paper notice issued in the PagoPA circuit by simply framing a QR code; 

elect their digital home directly from the app. 

To  access  the  app,  citizens  need  to  create  a  SPID  account  to  ensure  that  whoever 
uses  the  app  is  an  individual  who  has  the  right  to  access  that  data.  Biometric 
screening to unlock the app is also foreseen.  

The revolutionary aspect of this app is that the user will not have to register actively 
for the services and will be directly reached by the services that know the tax code.  

Regarding the other side of the platform, namely Public Administrations, IO provides 

27 Art.64 CAD (Codice dell’Amministrazione Digitale) establishes a single access point for all digital services 

66 

 
 
 
                                           
them  many  benefits.  In  fact,  through  the  open  API,  PAs  can  easily  interact  with 
citizens by:  

  sending  electronic  communications  by  simply  knowing  the  tax  code  (without 

having to ask citizens for a contact address);  

  communicating deadlines and receiving electronic payments more easily; 

  sending and requesting documents in a simple and efficient way;  

  managing the preferences of each citizen in a centralised way; 

 

reducing  management  costs  (of  notifications,  payments,  etc.)  and  facilitate 
payments.  

The  citizen  could  at  any  time  deactivate  the  services  that  is  not  of  interest,  which 
from that moment.  

Where are we at? 

The  Closed  Beta  version  was  released  and  included  several  services,  such  as  the 
payment  of  fines,  TARI,  local  taxes;  notifications  reminders  for  appointments  with 
offices,  info  mobility  and  expiration  of  the  ZTL  pass.  Several  Italian  municipalities 
such  as  Milan;  Turin;  Ripalta  Cremasca;  Valsamoggia;  Palermo  were  tested  to 
date,  and  around  one  thousand  citizens  have  participated  to  the  closed  beta  phase; 
many signalled bugs have been resolved, others are under WIP.  

The  Open  beta  version  was  released  in  April  2020.  Anyone  will  be  able  to 
download from the Android and iOS store the app. The launch of the application will 
be  done  at  a  local  level,  in  the  territories  wherein  PAs  have  effectively  integrated 
enough services in the App. In January 2021, IO services will be made available as a 
webapp,  allowing  citizens  to  access  from  their  laptop,  using  the  browser.  Moreover, 
new  design  functionalities  will  be  validated  as  well  as  other  payments  methods  such 
as Bancomat Pay, PayPal and Satispay will be introduced. 

Technology Area 

Technology Category 

Software and Service Technologies 

Digital Identity and Security 

APIs, Web Services, Microservices 
including Registries and Marketplaces 

Enterprise Service Bus Technologies & 
Government Service Utilities 

Digital Identity Frameworks 

Country 

Period 

Catchment area 

67 

 
 
 
 
 
Italy 

2017 - ongoing 

National 

Pilot Analysis 

Readiness of the pilot 

User centricity principles satisfied 

Digital 
interaction 

Accessibility, 
security, 
availability and 
usability 

Reduction of the 
administrative 
burden 

Digital Delivery 
of public 
services 

Citizen 
engagement 

Incentives for 
digital service 
use 

Protection of 
personal data 
and privacy 

Redress and 
complaint 
Mechanisms 

Focus on Interoperability 

Legal  
interoperability 

Organisational 
interoperability 

Semantic  
interoperability 

Technical  
interoperability 

4 

3 

3 

3 

Benefits & Impacts 

In the following paragraphs, the main benefits and impacts related to  user centricity 
and interoperability analysis are reported. 

Regarding user centricity, the IO App was designed to put people at the centre of 
the  innovation  process  of  the  Public  Administration,  by  involving  citizens  in  the 

68 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
development  of  digital  processes.  Hence,  the  project  evolution  is  based  on  constant 
communication, co-design with users, along different development stages. In fact, in 
the IO roadmap what dictates priority are the needs, the experience and the requests 
of people, not the technology. 

If we look at the analysis made on the 8 User Centricity principles of the Tallinn 
Declaration,  IO  successfully  satisfies  all  of  them.  The  idea  in  fact  started  with  the 
vision to empower the relationship between the citizen and the Public Administration. 

Regarding  interoperability,  the  Digital  Transformation  Team,  together  with  AgID, 
developed a clearly defined Interoperability Model, in accordance to the EIF, based on 
Application Programming Interface (API) to allow the entire information systems of 
the Public Administration to communicate with each other. 

Observing  the  analysis  made  on  the  four  interoperability  layers  of  the  EIF,  IO 
App registers good interoperability scores.  

  Legal  Interoperability  is  fully  compliant  with  EU  GDPR  in  accordance  with  the 
principle  of  "privacy  by  design".  Documents  and  messages  are  stored  in  a 
database deployed on European datacentres that uses "encryption at rest". 

  Organisational  interoperability  scores  good.  A  guide  describing  the  steps 
necessary  for  PA  to  integrate  into  IO  and  be  part  of  the  project  have  been 
successfully  set  (PA  needs  to  identify  services  that  can  be  provided  through 
IO,  prepare  technological  integration  using  the  IO  API;  review  and  sign  legal 
documentation  to  join  IO;  communicate  to  citizens  that  they  will  be  able  to 
find the specific services on IO). 

  Semantic  Interoperability  also  does  not  pose  great  challenges.  The  aim  of 
the project is in line with the EIF: “what is sent is what is understood”. In the 
European  context  the  databases  of  national  interest  correspond  to  Base 
Register  (used  in  EIF  to  indicate  reliable,  authentic  and  official  sources  of 
particularly  relevant  data  produced  by  PA),  fundamental  components  to 
support  interactions  between  PA  and  between  these  and  private  individuals. 
These  data  constitute  the  foundation  for  the  construction  of  public  services 
(Codice dell'Amministrazione Digitale - CAD art. 60). 

  Technical interoperability scores high since PA can download the Open API 
specifications. This API enables Public Administration services to integrate with 
the  IO  platform.  IO  enables  services  to  communicate  with  Italian  citizens  via 
the IO app. To get access to this API, PA will need to register on a specific IO 
Developer Portal. Technical interoperability is therefore well defined. 

Main impacts and benefits:  

The  IO  App  will  allow  citizens  to  access  to  PA  local  and  national  services  in  a 
simple,  modern  and  secure  way.  In  this  way,  citizens  will  be  able  to  have  all  the 
services in one place and will benefit from an improved user experience. The app will 
allow  to  drastically  reduce  the  time  of  fulfilment  of  citizens’  duties  to  the  Public 
Administration. Thanks to a system of notifications, payments and deadlines, in a few 
minutes  it  is  possible  to  carry  out  operations  that  previously  took  much  longer.  The 
benefits will be both for citizens and PAs: citizens will have greater knowledge of the 
services  provided  and  will  no  longer  need  to  provide  and  update  their  personal  data 
(email  address,  mobile  telephone  number,  bank  details,  preferred  contact  method, 

69 

 
etc.)  on  each  site,  this  will  enable  PAs  to  dramatically  reduce  their  administrative 
burden, reducing costs for both parties and increasing the uptake of services. 

Risks & Challenges 

The  risks  and  future  threats  related  to  the  implementation  of  the  project  are  mainly 
focused on: 

•  Organisational  and  Semantic  issues:  today,  most  of  the  existing  public 
databases have been designed and built in a distinct way, without the support 
of  an  overall  vision,  useful  for  directing  regulatory  and  technical  actions 
capable  of  favouring  data  quality.  This  characteristic  has  produced  over  time, 
the  fragmentation  of  the  information  assets  of  the  Public  Administration  into 
real information silos: "containers" in which the data are  often replicated and 
stored in an uneven or even incoherent and misaligned with each other; 

•  Low users’ uptake of the solution; 

•  Low public authorities’ uptake of the solution; 

•  The project rests on experimental API that is (most probably) going to change 

as IO platform evolve.  

The main challenges that IO could face:  

• 

• 

Increase organisational readiness; 

Increase  the  uptake  and  the  readiness  of  European  Public  Administrations, 
citizens  and  businesses  to  interact  digitally  with  the  Public  Administrations 
through education and training programs.  

Table 13 - Summary of IO App Analysis 

70 

 
 
8  Final considerations and outlook 

A  typical  approach  followed  by  government  bodies  and  agencies  innovating  their 
processes  is  built  on  the  execution  of  pilot  projects.  These  are  small-scale  experiments 
meant  to  reduce  the  costs  and  risks  inherent  to  a  diffused  introduction  of  new 
technologies  or  other  elements  of  transformation  of  the  “machinery”  of  Public 
Administrations,  its  functioning  and  input  and  output.  By  definition,  pilot  projects  are 
more likely to be a failure, rather than a success. Whatever the outcomes, their “owners” 
–  responsible  persons  of  the  execution  and  sometimes  evaluation  of  pilot  results  –  are 
expected to learn useful lessons from them, which will possibly be shared with a broader 
audience on due time and are supposed to be useful (and used) internally to support the 
decision  of  whether  and  how  to  follow  up.  Most  of  these  pilot  projects  are  financially 
supported by third party grants, such as from  the EU research, innovation, or territorial 
cooperation  programmes.  As  already  mentioned,  such  circumstance  does  not  only 
contribute to further reducing the costs and the risks of technology introduction, but also 
provides the best guarantee that pilot results will be shared widely to the promoting and 
financing agencies. 

Unfortunately,  to  date,  as  also  reported  in  the  aforementioned  JRC  research28,  the 
propensity  of  pilot  owners  to  share  results  and  lessons  learnt  in  a  structured  manner 
seems  to  be  still  quite  low  and  hampers  the  possibility  to  fully  evaluate  their  scalability 
potentials,  and  the  definition  of  finding  reliable  reuse  or  transfer  pathways  involving 
other public sector organisations than those in the original pilots. This lack of information 
is  particularly  critical  in  case  of  new  and  emerging  technologies,  such  as  Blockchain  or 
Artificial  Intelligence,  which  naturally  lend  themselves  to  being  trialled  in  very  similar  – 
yet never too much – pilot contexts, thus increasing the risk of “reinventing the wheel” – 
i.e. of duplication, if not proliferation, of limited size experiments.  

In  response  to  this  and  other  needs,  the  Study  reported  about  in  this  publication  has 
developed  and  tested  an  innovative  and  original  multi-dimensional  framework  for 
evaluating public service digitalisation pilot projects and later for gathering pilot projects 
in a unique portal accessible to European Member States to foster a culture of knowledge 
sharing and build consensus on solutions and on their practical implementation. This is in 
order  to  satisfy  the  objective  of  the  IPS  action  and  to  give  continuity  to  the  follow  up 
activities of the ISA2 programme, that is aimed at developing insights for an Observatory 
at EU level. 

In  this  regard,  the  framework,  accompanied  by  a  MS  Excel  database  facilitating  data 
collection, searching and visualisation, has been purposefully designed to be, at the same 
time: 

  Respectful of the variety of innovation modes that characterise the digital public 
service  landscape  in  EU.  For  instance,  some  technological  innovations  may  be 
disruptive  if  properly  introduced,  despite  their  not  being  based  on  new  cutting-
edge technologies. Or the same consolidated and well-functioning technology may 
not  be  easily  transferred  to  another  similar  context,  unless  all  relevant  variables 
are controlled for; 

28 Barcevičius, E., Cibaitė, G., Codagnone, C., Gineikytė, V., Klimavičiūtė, L., Liva, G., Matulevič, L., Misuraca, 
G.,  Vanini,  I.,  Editor:  Misuraca,  G.,  Exploring  Digital  Government  transformation  in  the  EU  -  Analysis  of  the 
state of the art and review of literature, EUR 29987 EN, Publications Office of the European Union, Luxembourg, 
2019, ISBN 978-92-76-13299-8, doi:10.2760/17207, JRC118857. 

71 

 
                                           
  Attentive to the combination of contextual aspects (legal, organisational, societal 
and  sometimes  cultural  or  capacity  related)  that  enables  the  adoption  of  a 
technology, in order to contribute to the necessary changes; 

  Compatible  with  a  wide  variety  of  technological  and  even  non  technological 
domains,  some  with  definitions  that  may  look  sometimes  loose  or  actually  an 
'ensemble'  gathered  under  umbrella  terms,  and  thus  require  further  adaptation 
and specification to become meaningful at pilot project (and community) levels; 

  Verifiable as for TRL, also SRL, ORL and LRL include high level descriptions that 
are adaptable to different pilot contexts and nevertheless make the results of the 
analysis (in terms of lessons learned) comparable and replicable; 

  Parsimonious  collection  of  evidence  can  be  discontinued  at  any  time,  without 
procuring harm to the proposed model or hampering its heuristic and informative 
value for the cases already analysed; 

  Usable not only for the trivial reason that the framework works the same if/when 
the  number  of  collected  items  goes  up,  but  also  because  with  a  little  more 
attention paid to the details, each  surveyed pilot may genuinely and uniquely be 
associated  with  a  longer  string  of  cardinal  numbers.  The  first  4  of  them  are  the 
attributed/observed  values  for  the  (t,  s,  o,  l)  quadruplet,  while  others  reflecting 
the  contextual  conditions  domain  characteristics  and  impact  areas  of  the  pilot  at 
hand; 

  Interactive  not  only  for  the  use  of  buttons  and  “data  entry”  section  to  ensure 
users  a  simple  and  easy  navigation  and  fast  load  time,  but  also  because  of  the 
presence of data filter and data visualisation representations that provide users to 
interact with data, see and understand trends and patterns. 

The  framework  revealed  to  be  a  useful  tool  to  assess  innovative  pilots  in  the  public 
service  provision  domain.  On  one  hand,  it  has  the  potential  to  assess  the  societal, 
organisational,  technological  and  legal  readiness  level  of  the  developed  solutions  over 
time;  on  the  other  hand,  it  has  the  merit  of  highlighting  the  impacts  and  challenges  of 
interoperability aspects and user centricity principles.   

According  to  the  proposed  framework,  successful  projects  are  those  that  registered  a 
quite homogenous growth of all the 4 readiness levels, reaching, each of them, near the 
top score, at the end of the pilot testing phase, and those that during the deployment of 
their  innovative  solutions  have  managed  to  put  citizens  and  business  at  the  centre  of 
service  design  while  following  precise  guidelines  to  make  services  interoperable  at  EU 
level. 

Moreover, results suggest that projects that at the beginning registered low scores and at 
the  end  reached  high  grades  in  terms  of  readiness  are  those  that  are  likely  to  produce 
the greatest impact on the public sector modernisation. 

Conversely,  the  absence  of  a  strict  correlation  between  the  4  variables  leads  to  classify 
the  pilot  as  a  failure,  which  means  that  at  the  end  of  the  experimentation,  significant 
gaps  continue  to  exist  that  are  negatively  affecting  the  technological,  societal, 
organisational or legal dimensions of the pilot environment. 

72 

 
 
 
 
Key findings of the Study include: 

  The  preliminary  identification  of  8  thematic  clusters  of  existing  and/or  emerging 
digital  technologies  as  relevant  for  the  EU  public  sector  digital  transformation 
goals; 

  A  thorough  assessment  of  the  capacity  of  the  8  chosen  thematic  clusters  to  be 
interoperable  and  user  centric,  giving  special  attention  to  the  analysis  of 
challenges and barriers;  

  An initial case study database composed of 150+ IPS pilot initiatives coming from 

24 European countries; 

  80 pilot projects selected for deeper analysis; 

  31  of  them  defined  “successful”  based  on  the  application  of  the  proposed 

evaluative framework; 

  2  successful  projects  further  inspected  collecting  additional  information  also  in 

cooperation with the project owners. 

In light of the above, we consider this Study an important milestone in the assessment of 
the transformation potential of digital technologies in the public sector and the search for 
pilot applications that are truly transferable,  replicable and leading to the establishment 
of user centric and interoperable cross-border and cross-domain services. 

Future investigations as a follow up to the desk and field research done can additionally: 

  Refine and popularise the classification of digital technologies; 

 

Increase the number and territorial coverage of the initial case study database; 

  Reiterate the approach to determine the “success” of “failure” of digitalisation pilot 

projects; 

  Elaborate  on  the  “readiness”  concept  as  an  instrument  to  support  EU  policy 

making, as proposed in29; 

  Upgrade  the  framework  to  help  the  policy  makers  to  better  interface  with 
innovative  technologies  thanks  to  guidelines  and  lessons  learnt  and  build 
consensus  on  solutions  and  on  their  practical  implementation  to  enhance  public 
service delivery at an EU level; 

  Find additional elements to (dis)confirm the evaluation results and therefore make 
the proposed framework stronger and possibly reusable in other contexts (e.g. for 
non-digital innovation projects and/or outside the public sector). 

29 Bruno I., Lobo G., Molinari F., Valente Covino B., Marchetti V., Schiavone Panni A., Donarelli A., Technology 
Readiness  revisited:  A  proposal  for  extending  the  scope  of  impact  assessment  of  European  public  services. 
Paper approved for the forthcoming ICEGOV 2020 Conference, Athens, Greece. 

73 

 
 
 
 
 
                                           
In this regard, the findings of the  Study pave the way to the follow up activities part of 
the IPS Action of the ISA2 programme that is aimed at developing a proposal for a fully-
fledged  Observatory  at  EU  level.  This  should  be  considered  a  useful  tool  to  facilitate 
exchange  of  knowledge  and  provide  indications  to  policy  makers  for  further  developing 
comprehensive  strategies  for  digital  government  transformation  and  cross-border 
interoperability. This is in line with the key principles underpinning the ISA² programme 
that  has  been  pioneering  beyond  techno-centric  approaches  to  digital  transformation  of 
government,  as  it  is  acknowledged  in  the  vision  of  interoperability  supported  by  the 
programme. 

In particular, the IPS action of the ISA² programme aims to support the further revision 
of  the  European  Interoperability  Framework,  which  has  already  put  a  on  an  equal  foot, 
non-technical  aspects  such  as  governance,  organisational  and  legal  requirements  with 
technical ones to successfully implement digital government transformation.  

In this respect, as underlined in the new Digital Strategy for Europe adopted in February 
by  the  von  der  Leyen  Commission,  there  is  a  need  to  develop  a  reinforced  EU 
governments interoperability strategy, which should be completed by the end of 2021.  

Based  on  the  assessment  results,  EU  governments  at  all  levels  need  to  invest 
significantly  to  tackle  interoperability  and  user-centricity  challenges  of  most  promising 
technologies  in  the  Public  Sector,  in  order  to  provide  seamless  access  to  cross  border 
digital solutions in line with citizens and businesses expectations. 

The  results  of  this  Study  should  be  therefore  taken  into  consideration  in  supporting  the 
development of the seamless flow of data across sectors and borders,  while stimulating 
cross-fertilisation of innovative public services, through the establishment of a permanent 
Observatory to contribute sharing knowledge and practices across Public Administrations 
at a national, regional and city level in the EU. 

74 

 
 
 
List of Tables 

Table 1 - Technology Areas, Categories and Uses/Applications (authors’ elaboration) ...14 

Table 2 – 4 interoperability layers description .........................................................17 

Table 3 – 8 user centricity principles description ......................................................19 

Table 4 - Synopsis of Interoperability per technology area ........................................39 

Table 5 - Synopsis of User centricity challenges per technology area ..........................41 

Table 6 - TRL scale used in Horizon 2020 and ERDF .................................................42 

Table 7 - SRL scale (adapted from Innovation Fund Denmark) ..................................44 

Table 8 - ORL scale (our elaboration) .....................................................................45 

Table 9 - LRL scale (our elaboration) .....................................................................46 

Table 10 - Metadata of the qualitative columns in the database ................................49 

Table 11 - Metadata of the quantitative columns in the database ..............................51 

Table 12 – Summary of Dublinked Analysis ............................................................65 

Table 13 - Summary of IO App Analysis .................................................................70 

75 

 
 
 
 
List of Figures 

Figure 1 - Overview of the IPS Action and related activities ......................................10 

Figure 2 - Analysed technologies ...........................................................................12 

Figure 3 - Technology areas supporting public service delivery ..................................15 

Figure 4 - Artificial Intelligence Area, Categories and Uses/Applications .....................20 

Figure 5  - Artificial Intelligence Interoperability and User centricity assessment .........21 

Figure 6 - Communication Technologies Area, Categories and Uses/Applications .........23 

Figure 7 - Communication Technologies Interoperability and User centricity assessment
 ..........................................................................................................................23 

Figure 8 - Computing Infrastructures Area, Categories and Uses/Applications .............25 

Figure 9 - Computing Infrastructures Interoperability and User centricity assessment ..26 

Figure 10 - Digital Identity and Security Area, Categories and Uses/Applications .........28 

Figure 11 - Digital Identity and Security Interoperability and User centricity assessment
 ..........................................................................................................................29 

Figure 12 - Distributed Ledger Technologies Area, Categories and Uses/Applications ...31 

Figure 13 - Distributed Ledger Technologies Interoperability and User centricity 
assessment..........................................................................................................32 

Figure 14 – Immersive Technologies Area, Categories and Uses/Applications .............33 

Figure 15 - Immersive Technologies Interoperability and User centricity assessment ...34 

Figure 16 - Internet of Things and Smart Devices Area, Categories and 
Uses/Applications .................................................................................................35 

Figure 17 - Internet of Things and Smart Devices Interoperability and User centricity 
assessment..........................................................................................................36 

Figure 18 - Software and Service Technologies Area, Categories and Uses/Applications
 ..........................................................................................................................37 

Figure 19 - Software and Service Technologies Interoperability and User centricity 
assessment..........................................................................................................38 

Figure 20 - Framework in action and transition pathway ..........................................47 

Figure 21 – Database structure overview ...............................................................51 

Figure 22 - Data entry interface ............................................................................52 

Figure 23 - Data filter according to project ending in 2017 .......................................52 

Figure 24 - Data filter according to ORL “after” or in “t1” equal to 8 ..........................53 

Figure 25 - Technology coverage by collected pilots ................................................53 

Figure 26 - EU and extra EU country coverage by collected pilots ..............................54 

Figure 27 - Process of pilot analysis and evaluation .................................................55 

Figure 28 - Representation of successful pilots across technology areas and European 
countries .............................................................................................................57 

Figure 29 - Criteria satisfied by successful pilots for future implementation ................58 

Figure 30 - Coverage and technologies used in the successful pilots ..........................59 

76 

 
 
 
GETTING IN TOUCH WITH THE EU 

In person 

All over the European Union there are hundreds of Europe Direct information centres. You can find the 
address of the centre nearest you at: https://europa.eu/european-union/contact_en 

On the phone or by email 

Europe Direct is a service that answers your questions about the European Union. You can contact this 
service: 

- by freephone: 00 800 6 7 8 9 10 11 (certain operators may charge for these calls), 

- at the following standard number: +32 22999696, or 

- by electronic mail via: https://europa.eu/european-union/contact_en 

FINDING INFORMATION ABOUT THE EU 

Online 

Information about the European Union in all the official languages of the EU is available on the Europa 
website at: https://europa.eu/european-union/index_en 

EU publications 
You can download or order free and priced EU publications from EU Bookshop at: 
https://publications.europa.eu/en/publications. Multiple copies of free publications may be obtained by 
contacting Europe Direct or your local information centre (see https://europa.eu/european-
union/contact_en). 

 
 
 
 
 
K
J
-
N
A
-
3
0
3
5
5
-
E
N
-
N

doi:10.2760/09628 

ISBN 978-92-76-21455-7