REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL Results of the final evaluation of the ISA programme
Date: 2016-09-01
Institutions: Directorate-General for Digital Services, European Commission
Subjects: EU programme, administrative cooperation, cross-border cooperation, digitisation, exchange of information, information technology applications, public administration, public authorities

EN 

   EN 

   EUROPEAN COMMISSION  Brussels, 1.9.2016  COM(2016) 550 final   REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL Results of the final evaluation of the ISA programme {SWD(2016) 279 final}  
 
 
Contents 

1. INTRODUCTION ............................................................................................................................ 3 

2. INTEROPERABILITY AS AN ENABLER OF PUBLIC SECTOR MODERNISATION .................. 3 

3. BACKGROUND .............................................................................................................................. 4 

4. METHODOLOGY ........................................................................................................................... 5 

5. EVALUATION ISSUES AND QUESTIONS ................................................................................... 6 

6. FINDINGS ....................................................................................................................................... 7 

ACHIEVING PROGRAMME’S OBJECTIVES ......................................................................................... 7 

6.1 RELEVANCE ............................................................................................................................ 7 

6.2 EFFICIENCY ............................................................................................................................ 9 

6.3 EFFECTIVENESS ...................................................................................................................... 9 

6.4 UTILITY ................................................................................................................................. 10 

6.5 SUSTAINABILITY .................................................................................................................... 11 

6.6 COHERENCE ......................................................................................................................... 12 

6.7 COORDINATION ..................................................................................................................... 13 

7.   CONCLUSIONS OF THE FINAL EVALUATION ................................................................... 14 

8. RECOMMENDATIONS ................................................................................................................ 14 

8.1 

     COMMUNICATION  WITH  PUBLIC  ADMINISTRATIONS  AND  RAISING  THEIR  AWARENESS  ON 

INTEROPERABILITY ...................................................................................................................... 15 

8.2   ENHANCING THE HOLISTIC APPROACH AND CROSS-CUTTING VIEW OF INTEROPERABILITY ...... 16 

8.3   COOPERATION WITH OTHER EU POLICIES AND INITIATIVES .................................................. 16 

9. NEXT STEPS ................................................................................................................................ 17 

Page 2 of 17 

 
 
 
1. Introduction 

By decision of the European Parliament and the Council (the ISA Decision),1 the six year programme on 

interoperability  solutions  for  European  public  administrations  (the  ISA  programme)  was  launched  on  1 

January  2010  as  a  follow-on  to  the  programme  on  the  interoperable  delivery  of  pan-European 
e-government services to public administrations, businesses and citizens (IDABC).2 It was implemented 

from 1 January 2010 to 31 December 2015. 

As  mentioned  in  Article  1  of  the  ISA  Decision,  the  objective  of  the  ISA  programme  was  to  support 

cooperation  between  European  public  administrations  by  facilitating  efficient  and  effective  electronic 

cross-border and cross-sectoral interaction between them (including bodies performing public functions 

on  their  behalf),  with  a  view  to  enabling  the  delivery  of  electronic  public  services  supporting  the 
implementation of EU policies and activities.3  

This Report presents the main results of the final evaluation of the programme4, which was carried out in 

accordance with Article 13(3) of the ISA Decision. According to this provision, the final evaluation had to 

be  communicated  to  the  European  Parliament  and  the  Council  by  31  December  2015.  The  final 

evaluation was performed by the Commission from December 2014 until February 2016 when the draft 

Commission  report  was  accepted  by  the  steering  group.  However,  the  recent  changes  in  the 

Commission evaluation procedures resulted in a significant delay in meeting the deadline provided in the 

legal basis. 

During  the  final  evaluation  the  Commission  was  assisted  by  a  team  of  independent  experts  from  a 

consultancy  company  (the  final  evaluation  team).  Representatives  of  relevant  Commission  services 
oversaw the final evaluation through a Commission steering group.5 

2.  Interoperability  as  an  enabler  of  public  sector 
modernisation 

The modernisation of public administrations is one of the key priorities for the successful implementation 
of  the  Europe  2020  Strategy.6  The  annual  growth  surveys  published  by  the  Commission  from  2011  to 

1 Decision No 922/2009/EC of the European Parliament and of the Council of 16 September 2009 on interoperability 
solutions for European public administrations (ISA), OJ L 260, 3.10.2009, p. 20. 
2 Decision 2004/387/EC of the European Parliament and of the Council of 21 April 2004 on interoperable delivery of pan-
European eGovernment services to public administrations, businesses and citizens (IDABC), OJ L 144, 30.4.2004 (see OJ 
L 181, 18.5.2004, p. 25). 
3 Article 1(2) of the ISA Decision. 
4 Final Evaluation of the ISA programme, available at: http://ec.europa.eu/isa/library/official-documents/index_en.htm 
5 Composed of officials from DG for Informatics (DIGIT), DG for Internal Market, Industry Entrepreneurship and SMEs 
(GROW), DG for Taxation and Customs Union (TAXUD), the Joint Research Centre (JRC) and the Secretariat General. 
6 Europe 2020: a strategy for smart, sustainable and inclusive growth, Commission Communication (COM(2010) 2020 final, 
3.3.2010, p.12). 

Page 3 of 17 

 
 
                                                      
20157  show  that  the  quality  of  European  public  administrations  has  a  direct  impact  on  the  economic 

environment and is, therefore, crucial to stimulating productivity, competitiveness and growth.  

As  mentioned  in  its  2013  annual  growth  survey,8  the  Commission  considers  the  cross-border 

interoperability  of  online  services  and  the  digitalisation  of  European  public  administrations  to  be 

important  factors  for  growth  and  increased  efficiency.  Interoperability  between  administrations  is  a  key 

enabler in achieving a more efficient and effective delivery of digital services. 

The  Digital  Agenda  for  Europe  (DAE)9  outlines  how  interoperability  is  essential  in  helping  to  maximise 

the social and economic potential of ICT and, consequently, the DAE can only succeed if interoperability 

is  ensured.  Its  dedicated  pillar  on  ‘interoperability  and  standards’  urged  Member  States  to  undertake 
specific actions related to interoperability,10 namely the implementation of the commitments laid down in 

the Granada and Malmö ministerial declarations including the adoption of the European Interoperability 
Framework (EIF)11 in national legislation.  

On  24  and  25  October  2013,  the  European  Council  adopted  conclusions12  stressing  that  the 

modernisation  of  public  administrations  should  continue,  with  the  swift  implementation  of  services  that 

rely  on  interoperability  for  effective  functioning,  such  as  e-government,  e-health,  e-invoicing  and  e-

procurement.  This  will  lead  to  more  and  better  digital  services  for  citizens  and  enterprises  across 

Europe, cost savings and greater efficiency, transparency and quality of service in the public sector. 

However,  endeavours  from  Member  States  often  take  place  at  a  national  level  without  following  a 

common European approach, compromising European-level interoperability and the availability of cross-

border digital public services. Therefore, there is a high risk that new electronic barriers (‘e-barriers’) are 

created that prevent citizens and businesses from using public services in countries other than their own 

and impede the smooth functioning of the internal market. 

3. Background 

The  ISA  programme  was  established  building  on  progress  under  its  predecessor  (IDA  and  IDABC) 
programmes. As the ISA Decision3 made clear, the main objective of the ISA programme was to support 

cooperation between European public administrations with a view to enabling the delivery of electronic 

public  services  supporting  the  implementation  of  EU  policies  and  activities.  The  intermediate  objective 

7 http://ec.europa.eu/europe2020/index_en.htm  
8 Commission 2013 annual growth survey, COM(2012) 750 final, 28.11.2012, p. 13. 
9 Commission Communication to the European Parliament, the Council, the European Economic and Social Committee and 
the Committee of the Regions (COM(2010) 245 final/2), p. 5 
10 http://daeimplementation.eu/dae_actions.php?action_n=26&id_country=1  
11 ‘Towards interoperability for European public services’, Commission Communication to the European Parliament, the 
Council, the European Economic and Social Committee and the Committee of the Regions (COM(2010) 744 final, Annex 1). 
12 http://data.consilium.europa.eu/doc/document/ST-169-2013-INIT/en/pdf 

Page 4 of 17 

 
 
                                                      
was  to  facilitate  efficient  and  effective  electronic  cross-border  and  cross-sectoral  interaction  between 

those administrations. 

To  meet  these  objectives,  facilitate  the  activities  set  out  in  Articles  1  and  3  of  the  ISA  Decision,  and 
provide ‘common and shared solutions facilitating interoperability’,13 the programme involved launching 

studies, projects and accompanying measures, by means of specific actions, to support:  

 

the  operation  and  improvement  of  existing  common  services,  and    the  establishment  of 

new ones; 

 

the improvement of existing re-usable generic tools and the establishment of new ones; 

 

the establishment and improvement of common frameworks in support of cross-border and 

cross-sectoral interoperability; and 

 

the assessment of the ICT implications of EU legislation (ISA Decision, Article 3). 

As  defined  in  Article  2  of  the  ISA  Decision,  ‘common  services’  are  operational  applications  and 

infrastructures of a generic nature, which meet common user requirements across policy areas. ’Generic 

tools’  are  reference  platforms,  shared  and  collaborative  platforms,  common  components  and  similar 

building-blocks  that  meet  common  user  requirements  across  policy  areas.  ‘Common  frameworks’  are 

strategies, specifications, methodologies, guidelines and similar approaches and documents.  

The ISA programme is executed by means of implementing acts (ISA Decision, Article 9) on the basis of 

a rolling work programme, which is revised at least once a year and includes actions for funding relating 
to  one  or  more  layers  of  interoperability  (technical,  semantic,  organisational  and  legal).14  Annual  work 

programmes are determined following annual calls for proposals to Commission services and Member 

States, endorsed by the ISA Committee and adopted by the Commission before entering into force. The 

specific actions to be funded are listed in the rolling work programme. 

4. Methodology 

The final evaluation of the ISA programme was based on quantitative and qualitative data collected in a 

variety of ways (desk research, online surveys, interviews, written requests for additional information and 

case  studies),  involving  a  broad  range  of  representatives  from  Member  State  and  EU  services,  and  a 

limited  number  of  stakeholders  from  civil  society  and  private  organisations  with  which  the  programme 

interacted. 

13 Article 1(1) of the ISA Decision. 
14 See European Interoperability Framework (EIF) for European public services (COM(2010) 744 final, Annex 2), p. 21. 

Page 5 of 17 

 
 
 
                                                      
5. Evaluation issues and questions 

The  final  evaluation  used  the  same  set  of  criteria  as  the  interim  evaluation,15  which  serves  as  a 

benchmark  of  the  results.  The  results,  conclusions  and  recommendations  of  the  final  evaluation  are 

based on a broad range of data, which constitute the hard evidence required to verify the validity of the 

programme’s intervention logic and the achievement of its objectives. It focused on the following seven 

main criteria:  

–  relevance  —  to  what  extent  were  the  programme's  objective(s)  pertinent  to  meeting  evolving 

needs and priorities at national and EU levels? 

–  efficiency  —  how  economically  were  the  various  inputs  converted  into  outputs,  and  with  what 

impacts in terms of the best possible use of resources? Which aspects of the programme were 

the most (in)efficient , especially in terms of resources mobilised?  

– effectiveness — to what extent did the programme's results and impacts achieve its objectives? 

Are there aspects that were more or less effective than others and, if so, what lessons can be 

drawn from this? 

– utility — how did the programme's actual and anticipated results and impacts compare with the 

business needs they were intended to address? What measures could be taken to improve the 

utility of actions under the next programme? 

–  sustainability  —  to  what  extent  is  the  financial,  technical  and  operational  sustainability  of  the 

developed solutions, as maintained and operated through the programme, ensured? 

 –  coherence  —  to  what  extent  did  the  ISA  actions  form  part  of  a  ‘holistic’ approach  (in  terms  of 

coherence with each other and with other EU policies and initiatives)? How well were synergies 

achieved between programme actions and with other EU activities? 

–  coordination  —  to  what  extent  were  activities  involving  Member  States,  including  those  of  the 

ISA  Committee,  coordinated  to  ensure  stakeholders'  involvement  in  the  ISA  programme?  To 

what  extent  were  activities  coordinated  or  aligned  with  the  needs  of  other  stakeholders  with 

whom the Commission was supposed to interact in the ISA framework? 

15  Interim evaluation of the ISA programme, 2012,   
    http://ec.europa.eu/isa/documents/interim_evaluation_of_the_isa_programme.pdf  

Page 6 of 17 

 
 
                                                      
6. Findings 

Achieving Programme’s Objectives 

The  final  evaluation  found  that  the  ISA  programme  was  effective  in  achieving  its  objectives  and 

delivering  operational  solutions  to  “support  cooperation  between  European  public  administrations  by 

facilitating the efficient and effective electronic cross-border and cross sectoral interaction between such 

administrations, [….] enabling the delivery of electronic public services supporting the implementation of 
Community  policies  and  activities”.3  This  is  particularly  the  case  for  common  frameworks,  reusable 

generic  tools  and  common  services  and  for  the  assessment  of  ICT  implications  of  new  EU legislation. 

Overall,  the  achieved  and  anticipated  results  and  impacts  of  the  ISA  programme  largely  address  the 

business needs that they intended to although some actions have more utility than others.  

The evaluation identified a number of key areas where changes in the successor ISA² programme have 

the potential, collectively, to be more relevant to the current needs on interoperability and to improve its 

utility.  It  also  confirms  that  the  ISA  programme  contributed  to  sharing  and  re-use,  through  the 

establishment  of  common  frameworks.  Finally,  the  ISA  programme  had  the  opportunity  to  revise  the 
‘Better  Regulation’  guidelines16  to  ensure  that  the  assessment  of  ICT  implications  is  clearly  part  of 

European Commission’s Impact Assessment process.  

The overall findings are summarised below: 

6.1 Relevance 

The  ISA  programme’s  objective  (‘to  support  cooperation  between  European  public  administrations  by 

facilitating the efficient and effective electronic cross-border and cross-sectoral interaction between such 

administrations’),  is  still  pertinent  to  meeting  evolving  needs  at  both  national  and  EU  levels.  This  was 

demonstrated  by  the  programme’s  final  evaluation  on  the  impact  of  the  ISA  results  but  also  from  the 

political  agendas  of  Member  States  clearly  indicating  the  need  for  efficient  and  effective  digital  public 
service  delivery.12  In  addition,  the  EIF  (European  Interoperability  Framework)  was  developed, 

implemented  and  monitored  by  the  ISA  programme.  The  EIF’s  recommendations  and  guidelines 

demonstrate how interoperability should be implemented by Member States in the public administrations 

and they aim to: 

a)  promote  and  support  the  delivery  of  European  public  services  by  fostering  cross-border  and 

cross-sector interoperability;  

b)  guide  European  public  administrations  in  their  work  of  providing  services  to  businesses  and 

citizens; and  

16 Better Regulation guidelines, Commission staff working document (SWD(2015) 111 final);  
Guidelines on impact assessment (COM(2015) 215 final) (SWD(2015) 110 final), p. 18. 

Page 7 of 17 

 
                                                      
c)  complement  and  tie  together  the  various  national  interoperability  frameworks  (NIFs)  at 

European level.  

The EIF provides a common understanding on interoperability in the Union, which is acknowledged by 

the  DSM  strategy  that  includes  a  particular  action  on  interoperability  between  public  administrations 

(action  15:  adoption  of  a  priority  ICT  standards’  plan  and  extending  the  European  Interoperability 

Framework for public services).  

The  ongoing  revision  of  the  EIF  builds  on  work  undertaken  over  the  past  few  years  by  the  National 
Interoperability Framework Observatory (NIFO),17 which monitors and supports its implementation in the 

Member States.  

The  ISA  programme  contributes  to  other  DSM  actions
where  interoperability  acts  as  an  underlying 
enabler.18  The  programme’s  objectives  are  well  aligned  with  the  DAE  and  the  2010  –  2015  European 

eGovernment Action Plan, which both identify interoperability as a condition for success. In addition, the 

programme  aligns  well  with  the  CEF,  which  aims  to  develop  connectivity  and  interoperability  of 
European digital services, as some of its building blocks19 (e.g. e-delivery, translation, e-invoicing) rely 
on  ISA  results.20  The  CEF  can  provide  operational  support  to  solutions  developed  and  matured  under 

the ISA programme. 

The ISA programme actively contributed to a number of EU policies such as the Internal Market,21 the 
Infrastructure for Spatial Information in the European Community (INSPIRE),22 the Common Information 
Sharing  Environment  (CISE),23  the  Pan-European  Open  Data  Portal,24  and  competition  law,25  which 
have benefited from its support in the form of dedicated actions and shared solutions.26 These policies 

have identified goals and objectives that rely on interoperability for their implementation.  

The needs of European public administrations, such as the modernisation of public service delivery, the 

improvement  of  coordination  between  Member  States’  administrations  and  Commission  services,  the 

17 Developed under the IDABC programme, the NIFO supports the implementation of the EIF at Member State level, 
through the transposition of the EIF into national interoperability frameworks. http://ec.europa.eu/isa/actions/04-
accompanying-measures/4-2-3action_en.htm  
18 Examples are its contributions to the interconnection of business registers, to the implementation of the ’once only’ 
principle (DSM action16) through its activities in the area of base registries and to the ‘digital single gateway’ objective, with 
the work on semantics and catalogue of services. 
19 https://joinup.ec.europa.eu/community/cef/og_page/catalogue-building-blocks 
20 Final evaluation of the ISA programme, table 10: ‘Alignment of the ISA programme with CEF’, p. 54 
21 Working together to create new growth’ – the Single Market Act: 12 levers to boost growth and strengthen confidence, 
Commission Communication to the European Parliament, the Council, the European Economic and Social Committee and 
the Committee of the Regions (COM(2011) 206 final),p. 10. 
22 Article 7 of Directive 2007/2/EC of the European Parliament and of the Council of 14 March 2007 establishing an 
Infrastructure for Spatial Information in the European Community (INSPIRE). 
23 Draft roadmap towards establishing the Common Information Sharing Environment for the surveillance of the EU maritime 
domain, Commission Communication to the European Parliament (COM(2010) 584 final), , p. 3 
24 Open data: An engine for innovation, growth and transparent governance, Commission Communication to the European 
Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions (COM(2011) 882 
final), p. 11. 
25 Report on Competition Policy 2012, Commission report (COM(2013) 257 final), p.12. 
26 Final evaluation of the ISA programme, table 21: ‘Level of adoption of ISA solutions by Commission services’, p. 131. 

Page 8 of 17 

 
 
                                                      
organisational  alignment  in  public  administrations  and  the  greater  reusability  of  solutions,  identified  at 
the  time  of  the  ISA2  ex-ante27  and  ISA  interim  evaluations,28  still  remain.  Facilitating  cross-border  and 

cross-sector  service  delivery  in  a  uniform  and  resource-efficient  way  requires  greater  reuse  of  existing 

interoperability  solutions  and  increased  collaboration  between  Member  States'  administrations  and 
between them and the Commission. Common approaches ensure no duplication of efforts.29 

6.2 Efficiency 

Overall,  the  ISA  programme  can  be  considered  to  have  been  efficient.  First,  the  evidence  from  the 
monitoring  and  evaluation  action30  found  that  it  has  been  considered  ‘on  track’  throughout  the  2010  - 
2015 period, with an average delay of less than 5 %.31 Second, the total executed budget for the 2010 – 

2014  work  programmes  amounted  to  EUR128.2  million,  approximately  2  %  less  on  average  than  the 
cumulative  budget  allocated  for  the  period.32  Given  these  findings,  the  programme  can  be  deemed 
efficient in budgetary terms.30 

In  addition,  90  %  of  surveyed  Member  State  representatives  and  Commission  officials  from  relevant 
services affirmed that the programme delivered on time and within its original scope.33 As mentioned in 

the  ISA  final  evaluation,  a  majority  of  these  stakeholders  felt  fully  or  somewhat  involved  in  the  annual 
revisions of the programme.34 

The procedure for admitting proposals for new actions and the funds’ release process can be regarded 
as efficient.35 The efficiency in terms of the allocation of human resources is confirmed by the fact that 

the actual allocation was always lower than that envisaged in the original ISA legislative proposal. 

6.3 Effectiveness 

The  ISA  programme  has  delivered  operational  solutions,  i.e.  common  frameworks,  re-usable  generic 

tools  and  common  services  that  help  to  achieve  the  objective  of  facilitating  effective  collaboration 

between  European  public  administrations.  Overall,  the  majority  of  interviewees  and  online  survey 
respondents felt that all ISA solutions facilitate interoperability between those administrations.36 

27 Document accompanying proposed ISA2 Decision (COM(2014) 367 final). 
28 Interim evaluation of the ISA programme, 2012, p. 172  
    http://ec.europa.eu/isa/documents/interim_evaluation_of_the_isa_programme.pdf 
29 Final evaluation of the ISA programme, ‘Pertinence of the ISA objectives to respond to needs of EU public 
administrations’, p. 45 
30 http://ec.europa.eu/isa/dashboard/isadashboard  
31 Final evaluation of the ISA programme, ‘ISA programme efficiency’: % of delay, p. 82. 
32 Final Evaluation of the ISA programme, ‘Budget allocation and execution’, p. 67. 
33 Final evaluation of the ISA programme, ‘Conclusion 4’, p. 86. 
34 Final evaluation of the ISA programme, ‘Revision process of the ISA Work Programme’, p. 59. 
35 Final evaluation of the ISA programme, ‘Allocation of financial resources’, p. 66. 
36 Final evaluation of the ISA programme, ‘Conclusion 6’, p. 111. 

Page 9 of 17 

 
                                                      
The  establishment  of  common  frameworks  such  as  the  EIF,  also  supported  the  sharing  and  re-use  of 
solutions with its dedicated recommendations.37 A sharing and reuse framework started to be developed 
under ISA and is currently being finalised under the ISA2 programme.  

The  programme  started  work  on  the  top  two  layers  of  interoperability,  the  organisational and  the  legal 

interoperability  layers,  which  are  areas  with  great  potential  impact  on  Member  States’  public 

administration modernisation and need to be further developed and consolidated. 

The  assessment  of  the  ICT  implications  of  new  EU  legislation,  such  as  impact  related  to  the 

implementation  of  new  or  the  adaptation  of  existing  ICT  solutions  was  included  in  the  Commission’s 

impact  assessment  process,  under  the  Better  Regulation  guidelines.

16

  The  ISA  programme  has  gone 

further by promoting such assessments and supporting DGs and services undertaking them. 

The  effort  that  the  programme  invested  in  the  area  of  semantic  interoperability  and  also  in  developing 

stronger links with standardisation organisations represented important contributions to increased cross-

sector interoperability. 

Although  the  programme  and  users  of  ISA  results  re-use  existing  common  frameworks,  common 

services and reusable generic tools, the programme should further promote their re-use as pointed out 
by surveyed Member State representatives and Commission officials.38  

6.4 Utility 

The achieved and anticipated results and impacts of the ISA programme largely address the business 
needs  identified.39  It  led  to  the  production  of  23  solutions,  made  freely  available  on  its  website40  or 

provided  as  shared  services.  These  solutions  vary  from  systems  supporting  implementation  of  EU 

policies  such  as  the  Internal  Market  Information  System  (IMI),  to  common  specifications  such  as  the 

Data Catalogue Application Profile (DCAT-AP) used in the Pan-European Open Data Portal. Of the 23 

solutions, 15 are used by some or all Member States, 14 are used by several Commission services, and 

five are used by a handful of other EU institutions. This success is even more striking given that Member 

States are not obliged to reuse the solutions. 

The  method  developed  for  the  assessment  of  ICT  implications  of  new  EU  legislation  included  in  the 

Commission’s  Better  Regulation  Toolbox,  is  being  widely  reused  in  the  context  of  the  impact 

assessment process.

16

37 Final evaluation of the ISA programme, ‘Establishment and improvement of common frameworks’, p. 94. 
38 Final evaluation of the ISA programme, ‘Conclusions 7 and 8’, p. 112 and 140. 
39 Final evaluation of the ISA programme, table 18 ‘Utility scores at action level', p. 116. 
40 http://ec.europa.eu/isa/ready-to-use-solutions/index_en.htm  

Page 10 of 17 

 
  
                                                      
Other  actions  have  proven  their  utility  through  their  widespread  usage  as  shown  by  the  statistics 
collected by the monitoring and evaluation action.41 

Member State representatives and Commission officials identified the following as the top four benefits 
of the ISA programme for Member States:42  

1) 

the use of ISA solutions;43  

2) 

the existence of the programme itself in raising awareness of the importance of interoperability 
and keeping it on the political agenda;44  

3) 

the provision of references supported by NIFO, such as the EIF; and  

4) 

the services provided, such as the sTESTA network and the CIRCABC platform.  

The final evaluation highlights the low level of awareness of the programme among the key stakeholders 
who were interviewed and surveyed, Member State representatives and Commission officials.45 This is 

confirmed by the fact that the additional measures suggested by interviewees and survey respondents 
are all either in place or being developed or expanded.46  

The programme’s contribution to other EU policies and initiatives and to the modernisation of the public 
sector  in  Europe,  improved  its  utility.30  Lastly,  actual  use  (including  the  number  of  users)  of  each  ISA 

solution should be systematically measured and the benefits of the solutions promoted. 

6.5 Sustainability 

The  final  evaluation  found  that  the  developed  solutions,  as  maintained  and  operated  under  the 

programme, were sustainable from a financial, technical and operational point of view. 

The technical and operational sustainability of an ISA solution is assessed indirectly by its operation and 

usage, as measured under the monitoring and evaluation action. 

In  financial  terms,  sustainability  is  guaranteed  primarily  by  the  follow-on  ISA²  programme  or  other  EU 

initiatives  and  programmes,  e.g.  the  CEF.  In  certain  cases,  funding  options  such  as  cost  sharing  and 

paying for use could be envisaged as a way to ensure long-term financial sustainability. 

Organisational  sustainability  is  also  fundamental  to  overall  sustainability.  Long-term  governance  of 
solutions  and  services  should  be  systematically  planned  to  ensure  their  continuity.47  It  is  important  to 

41 Final Evaluation of the ISA programme,  ‘Facilitation of effective interaction between European public administrations’, p. 
88 and 89. 
42 Final evaluation of the ISA programme, Main ISA results perceived, p. 88. 
43 Final evaluation of the ISA programme,  ‘General perceptions on the benefits delivered by the programme’, p. 113 and 
114. 
44 Final evaluation of the ISA programme, figure 37 ‘Respondent’s agreement levels on the awareness raising of the ISA 
programme', p. 171 
45 Final evaluation of the ISA programme, ‘Conclusion 9’, p. 141. 
46 Final evaluation of the ISA programme, ‘Coherence’, p. 17. 
47 Final evaluation of the ISA programme, ‘Conclusion 10’, p. 148. 

Page 11 of 17 

 
                                                      
note that information on the governance of solutions funded under the programme is always requested 

as part of the annual application process.  

6.6 Coherence 

The  final  evaluation  found  that  the  actions  of  the  ISA  programme  feed  into  a  holistic  approach.  This 
holistic  approach,  presented  to  the  ISA  Committee  in  January  2013,48  recognises  that  genuine 

interoperability  among  public  administrations  requires  working  on  various  lines:  strategic  objectives, 

frameworks  and  architectural  guidelines,  development  of  specifications  and  the  implementation, 

operation  and  dissemination  of  solutions.  In  addition,  monitoring  and  mapping  the  interoperability 
landscape  in  Europe  and  awareness-raising  are  considered  important  enablers  of  interoperability.  A 

holistic  approach  allows  for  gaps  to  be  identified  and  relevant  activities  or  projects  to  be  launched  to 

address them.  

This  holistic  approach  is  not  apparent  to  ISA  stakeholders,49  who  are  more  familiar  with  the  individual 

actions  than  with  the  programme  itself.  Low  awareness  of  programme  synergies  was  also  confirmed 

among  the  interviewees  and  survey  respondents,  although  the  final  evaluation  pointed  to  an 

improvement since the interim evaluation. 

Stakeholders  mentioned  only  one  ISA  action  (EUSurvey)50  significantly  more  than  others  in  the 
interviews and online surveys as not being coherent with the objectives of the programme.36  

Coherence was enhanced in the course of the programme by a considerable number of synergies that 

emerged between its actions and other EU policies and initiatives where interoperability is a key enabler 

for  their  implementation.  What  was  clearly  demonstrated  and  tackled  through  the  programme  was  the 

need  to  act  horizontally  to  ensure  interoperability  across  policy  domains.  This  horizontal  dimension  is 

missing  in  the  domain  specific  implementations.  ISA  created  synergies  with  areas  like  internal  market, 

competition  law,  INSPIRE,  CISE,  Pan  European  Open  Data  Portal  and,  more  recently,  the  CEF,  which 

builds, to some extent, on the programme’s results. 

A cross-cutting approach to interoperability brings tangible benefits that could not be achieved otherwise 

in sector specific approaches and solutions. The areas of public procurement, public sector information, 

ICT  standardisation,  justice  and  home  affairs,  and  taxation  would  benefit  from  closer  alignment  with 

cross-sector interoperability solutions at EU level and the possible ensuing cooperation. 

In  the  latter  stages  of  the  ISA  programme,  synergies  were  developed  with  Commission  services 

responsible for the European Structural and Investment Funds (ESIF). Under the thematic objective 11, 

which  aims  to  enhance  the  institutional  capacity  and  efficiency  of  public administrations,  the  ESIF  can 

support  national,  regional  and  local  administrations’  efforts  to  improve  interoperability  as  a  means  of 

48 ISA Committee meeting, 23 January 2013 – presentation of the results of the interim ISA programme evaluation. 
49 Final evaluation of the ISA programme, ‘Conclusion 11’, p. 165. 
50 EUSurvey is an open source, free-of-charge, web-based application which makes it easier both for respondents to answer 
the consultations and for survey managers to analyse the results. 

Page 12 of 17 

 
                                                      
modernising  their  public  administrations.51  Closer  cooperation  under  the  ISA2  programme  could  bring 

mutually beneficial results for the programme and ESIF implementation in Member States. In addition, 

active  involvement  of  DG  DIGIT  in  the  recently  launched  Inter-service  Group  on  Public  Administration 
Quality and Innovation52 ensures cooperation between the programme and the ESIF. 

Efforts under the programme and by DG DIGIT led to closer links with Commission’s IT governance. 

6.7 Coordination 

During  the  implementation  of  the  programme,  a  good  coordination  of  formal  activities  was  established 

with the Member States and with Commission services in line with the ISA Decision (Articles 12 and 14). 

Two thirds of the Member State representatives and over half of the Commission officials agreed fully or 

somewhat  that  the  interaction  between  ISA  and  its  stakeholders  was  effective  to  ensure  consistent 
exchange of information, views and best practices.53 

In  addition,  a  good  coordination  of  activities  with  stakeholders  outside  public  administrations  was 

established under the programme to ensure their involvement where appropriate. These include, where 

relevant  and  appropriate,  standardisation  organisations,  academics,  private  sector  organisations  and 

non-EU countries, e.g. the United States, Japan and Australia. 

The final evaluation recognised the concerted efforts that have been invested in coordination at EU level 
with  undeniably  positive  results.54  The  Inter-service  Group  on  Public  Administration  Quality  and 

Innovation will be instrumental in improving the coordination of  all EU-level initiatives that contribute to 

the modernisation of public administrations in Europe. 

The evaluation concluded that there is scope for Member State representatives on the ISA Committee, 

in the Coordination Group and the Working Group to coordinate better with their national colleagues on 
interoperability issues to contribute to the follow-on programme’s success.55 

The  programme  provides  a  good  overview  of  interoperability  developments  at  national  level  by 

consolidating  the  sets  of  NIFO  and  e-government  Member  States’  factsheets  in  one  location  on  the 
Joinup platform.56  

A majority of Member State representatives and Commission officials agreed fully or somewhat that the 

electronic  platforms  used  by  ISA  to  deliver  consistent  interaction  between  stakeholders  and  with  the 

Commission (e.g. Joinup, the ISA website and dashboard) were adequate. However, there was a lack of 

51 Final evaluation of the ISA programme, ‘Internal coordination within the European Commission’, p. 167. 
52 Inter-service Group on Public Administration Quality and Innovation (Ares (2015) 4431139, 20.10.2015). 
53 Final evaluation of the ISA programme, ‘Extent to which interaction between ISA and its stakeholders was effective’, p. 
172. 
54 Final evaluation of the ISA programme, ‘Internal coordination within the European Commission’, p. 167 
55 Final evaluation of the ISA programme, ‘Conclusion 13’, p. 187. 
56 https://joinup.ec.europa.eu/community/nifo/home 

Page 13 of 17 

 
                                                      
awareness  among  stakeholders  of  the  results  achieved  at  national  (e.g.  the  reuse  of  the  programme) 
and EU level, and of the use of ISA results by Commission services.57 

7. Conclusions of the Final Evaluation 

The final evaluation was largely positive, concluding that the ISA programme was aligned with the Union 
policy priorities and Member States’ needs.58 It found that it was implemented efficiently and coherently, 

delivering  results  that  are  re-used  by  both  Commission  services  and  Member  States.  The  ISA 

programme has been effective in achieving its objectives and delivering operational solutions facilitating 

effective  collaboration  between  European  public  administrations,  including  the  assessment  of  ICT 

implications of new EU legislation. Overall, the achieved and anticipated results and impacts of the ISA 

programme largely address the business needs that they intended to and it coordinated its activities well 

with its stakeholders to ensure their engagement. 

The  conclusions  found  that  there  is  a  need  to  focus  on  improving  the  reuse  of  ISA  solutions  and 

addressing new evolving needs. In addition, there is a need to build on the coordination of initiatives at 

EU level, in particular among DGs, in the framework of the ‘Inter-service Group on Public Administration 

Quality and Innovation’, and between Member States and the Commission to ensure greater coherence 

with other EU and national initiatives and coordination with stakeholders. 

The  overall  strategic  approach  taken  by  the  programme  since  2013  demonstrates  that  a  coherent 

holistic  approach  within  the  framework  the  ISA  programme  exists.  Synergies  were  achieved  between 

programme actions and with other EU activities but scope for further synergies remains.  

8. Recommendations 

It  is  important  to  note  that  the  ISA  Decision  focused  the  programme’s  activities  on  achieving 

interoperability  between  public  administrations.  This  is  likely  to  have  had  an  impact  on  levels  of  utility, 

awareness and re-use because the scope of the programme did not include public administrations’ links 

with businesses and citizens. Nevertheless, in some instances, the programme extended its stakeholder 

base  in  a  pragmatic  way  to  involve  businesses  and  citizens  in  the  development  of  interoperable 

solutions and put together a more holistic view than would otherwise have been possible. 

The need, when developing interoperable solutions, to consider the link between public administrations, 

on  the  one  hand,  and,  on  the  other  hand,  citizens  and  business  and  their  particular  needs,  becomes 

57 Final evaluation of the ISA programme, ‘Reuse by ISA of results delivered by other EU initiatives’, p. 163. 
58 Final evaluation of the ISA programme, ‘Conclusions 1, 2 and 3’, p. 56, 57 and 58. 

Page 14 of 17 

 
 
 
                                                      
increasingly apparent, given that the latter are the end-users of all public administration services.59 The 

ISA  Decision  did  not  provide  for  action  to  address  the  needs  of  end-users,  but  experience  of 

implementing  the  programme  and  consultation  with  action  ‘owners’  and  Member  States  made  it  clear 

that this was necessary. This limitation is addressed in the recommendation to extend the scope of the 

new ISA² programme to businesses and citizens. 

The  evaluation  found  that  the  ISA²  programme  should  develop  a  more  systematic  ‘business-case’ 

approach, which would also serve to highlight further the role of interoperability in producing economic 
and  societal  benefits.60  In  addition,  in  order  to  ensure  the  full  efficacy  of  the  ISA²  programme,  the 
Commission  should  endeavour  to  respect  its  staffing  targets,61  particularly  as  the  actual  allocation  of 

human resources was always lower than that originally envisaged in the programme. 

As regards the recommendations of the interim evaluation, the programme sufficiently addressed that on 

improving interaction with stakeholders and presence at major events, and made consistent progress in 

addressing the others. However, the evaluation concluded that the following areas require ongoing effort 

under the new ISA² programme: 

 

 

 

communication with public administrations and raising their awareness on interoperability; 

enhancing the holistic approach and horizontal cross-cutting view of interoperability; and 

cooperation with other EU policies and initiatives. 

In  addition,  the  final  evaluation  team  considers  that  two  of  the  11  recommendations  from  the  final 
evaluation of the IDABC programme are still applicable to the ISA programme; these are:62 

 

a  common  ‘promotion’  document  should  be  produced,  focusing  on  policy  alignments  and 

synergies between Union programmes and initiatives. (IDABC recommendation 5); and 

 

information on projects’ financial and operational sustainability should be rendered more visible 

and explained better to external stakeholders. (IDABC recommendation 8) 

The main recommendations are below: 

8.1 Communication with public administrations and raising their 
awareness on interoperability  

Between 2010 and 2015, approximately two thirds of the events that the ISA programme organised and 

held with Member State representatives, and 63 % (25) of the 40 events organised by Member States in 

which  the  ISA  programme  participated,  were  held  after  2013,  i.e.  after  the  interim  evaluation.  The 

59 Final evaluation of the ISA programme, ‘Consideration of results achieved by Member States in the establishment of ISA 
solutions’, p. 184. 
60 Final evaluation of the ISA programme, ‘Recommendation 5’, p. 200. 
61 Final evaluation of the ISA programme, ‘Recommendation 6’ p. 201. 
62 Final evaluation of the ISA programme, ‘Recommendations from previous evaluations’, p. 190. 

Page 15 of 17 

 
                                                      
programme increased its presence at conferences and other events; of over 60 events in which it was 
involved, three quarters took place after the interim evaluation.63  

The ISA² programme should update and implement a communication strategy, with a focus on targeted 

engagement, including with sector-specific stakeholders. This would address the lingering gap between 
the actual performance of the programme and stakeholders’ perception of it.64 

8.2 Enhancing the holistic approach and cross-cutting view of 
interoperability 

The  holistic  approach  taken  since  2013,  should  be  continued  in  the  ISA²  programme,  as  legal  and 

organisational  interoperability  are  still  in  their  infancy.  The  European  Interoperability  Reference 
Architecture  (EIRA)65  and  the  European  Interoperability  Cartography  (EICart),66  documenting  the 

existing services and tools, will be instrumental in supporting this holistic approach by helping to identify 
and  re-use  existing  solutions  and  highlighting  where  interoperability  enablers  are  lacking.67  EIRA  and 

EICart have a vital role in the dissemination of results and better identification of needs. They allow for 

more awareness, improved coordination and the emergence of greater synergies. 

Now that the assessment of the ICT implications of new EU legislation has been included in the impact 

assessment  process,  the  ISA²  programme  should  continue  to  play  its  intended  role  and  deploy  its 
expertise to promote this exercise and support the DGs and services undertaking such assessments.68  

Lastly,  the  ISA²  programme  should  develop  a  more  systematic  approach  to  supporting  and  monitoring 

the use of common services and generic tools, but also the application and implementation of common 
frameworks.69 

8.3 Cooperation with other EU policies and initiatives 

Efforts to align the ISA/ISA² programme with other relevant EU policies continue, taking as an example 

the  CEF,  which  follows  up  on  its  results.  Also,  from  2016  onwards,  account  should  be  taken  of  the 

revised EIF, when solutions are implemented in support of other EU policies.  

The ISA² programme should build on the clear improvements in the coordination of activities relating to 

interoperability  and  e-government  across  the  Commission  that  the  ISA  programme  has  brought  about 

63 Final evaluation of the ISA programme, ‘Conclusion 14 and extent to which the ISA programme raised awareness of the 
ISA solutions’, p. 188. 
64 Final evaluation of the ISA programme, ‘Recommendation 4, p. 199, and ‘Perceived performance of the ISA programme’, 
p. 83. 
65 The EIRA is a reference architecture for delivering interoperable digital public services across borders and sectors.  
66 The EICart tool is a tool that can be used to describe and catalogue the Solution Building Blocks (SBBs) of enterprise 
architectures in line with the EIRA. 
67 Final evaluation of the ISA programme, Recommendation 8 of 9, p. 202. 
68 Final evaluation of the ISA programme, Recommendation 3 of 9, p. 198. 
69 Final evaluation of the ISA programme, Recommendation 9 of 9, p. 202. 

Page 16 of 17 

 
                                                      
since the interim evaluation.70 This includes the ongoing cooperation with DG CNECT on the CEF and 

the new 2016-2020 e-Government Action Plan. In addition, there are plans to cooperate with DG EMPL 
and DG REGIO to identify existing or new ISA/ISA2 actions that have the potential  to contribute to the 

ESIF  thematic  objectives  2,  (‘enhancing  access  to,  and  use  and  quality  of  ICT’),  and  11,  (‘enhancing 

institutional capacity of public authorities and stakeholders and efficient public administration’). This is in 

addition  to  work  on  evaluating  national  operational  programmes  and  identifying  conditions.  Lastly,  the 

ISA  programme  has  shown  strong  potential  for  greater  cooperation  with,  inter  alia,  DG HOME,  DG 

JUST, DG MARE, DG FISMA, DG GROW and the JRC. 

The final evaluation sees the new Inter-service Group on Public Administration Quality and Innovation as 

the  main  driver  to  achieve  consistent  cooperation  and  coherence  among  EU  policies  and  initiatives 
contributing to public sector modernisation.71 

9. Next steps 

In  implementing  the  follow-on  ISA²  programme,  the  Commission  will  pay  the  utmost  attention  to  the 

above  findings  and  recommendations,  analysing  them  to  validate  and  address  issues  raised  in  close 

cooperation,  wherever  appropriate,  with  the  Member  States  and  other  stakeholders,  including 

businesses and citizens. 

70 Final evaluation of the ISA programme, Recommendation 7 of 9, p. 201. 
71 Opinion of the European Economic and Social Committee on the Communication from the Commission - Annual Growth 
Survey 2014 (COM(2013) 800 final); see OJ C 214, 8.7.2014, p. 46–54. 

Page 17 of 17