REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL Interim evaluation of the ISA programme
Date: 2013-01-18
Institutions: European Commission
Subjects: electronic government, exchange of information, intergovernmental cooperation (EU), telematics, transmission network

EUROPEAN 
COMMISSION 

Brussels, 18.1.2013  
COM(2013) 5 final 

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND 
THE COUNCIL 

Interim evaluation of the ISA programme 

EN 

   EN 

 
 
 
 
  
 
REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND 
THE COUNCIL 

Interim evaluation of the ISA programme 

 
 
1. 

INTRODUCTION 

By decision of the European Parliament and the Council (the ISA Decision)1 the six-
year  programme  on  interoperability  solutions  for  European  public  administrations 
(the ISA programme) was launched on 1 January 2010 as a follow-up programme to 
the IDABC programme2. 

The  purpose  of  this  report  is  to  present  the  findings  and  recommendations  of  the 
interim evaluation of the ISA programme. The need to carry out the evaluation stems 
from  Article  13(3)  of  the  ISA  Decision,  which  also  requires  the  European 
Commission to communicate the results of the evaluation to the European Parliament 
and the Council by 31 December 2012. 

The  evaluation3  was  performed  by  the  Commission  using  a  team  of  independent 
experts  from  a  consultancy  company  (the  evaluation  team).  Representatives  of 
Commission  services  have  overseen  the  evaluation  through  a  Commission  steering 
group4. 

2. 

BACKGROUND 

The  objective  of  the  ISA  programme  is  to  support  cooperation  between  European 
public  administrations  by  facilitating  the  efficient  and  effective  electronic  cross-
border and cross-sectoral interaction between such administrations, including bodies 
performing public functions on their behalf, with a view to enabling the delivery of 
electronic  public  services  supporting  the  implementation  of  EU  policies  and 
activities5. 

As  defined,  the  objective  covers  both  the  ultimate  goal  of  supporting  cooperation 
between  European  public  administrations  with  a  view  to  enabling  the  delivery  of 
electronic  public  services  supporting  the  implementation  of  EU  policies  and 
activities  and  the  intermediate  objective  of  facilitating  efficient  and  effective 
electronic  cross-border  and  cross-sectoral  interaction  between  European  public 
administrations to this end. 

To meet both the ultimate and intermediate objectives and to provide ‘common and 
shared solutions facilitating interoperability’6, the ISA programme launches studies, 
projects and accompanying measures, in specific actions, to support the 

1 

2 

3 

4  
5 
6 

Decision  No  922/2009/EC  of  the  European  Parliament  and  of  the  Council  of  16  September  2009  on 
interoperability solutions for European public administrations (ISA), OJ L 260, 3.10.2009, p. 20. 
Decision 2004/387/EC of the European Parliament and of the Council of 21 April 2004 on interoperable 
delivery  of  pan-European  eGovernment  services  to  public  administrations,  businesses  and  citizens 
(IDABC), OJ L 144, 30.4.2004 (see OJ L 181, 18.5.2004, p. 25). 
The full evaluation report is available at:     
http://ec.europa.eu/isa/documents/interim_evaluation_of_the_isa_programme.pdf 
DG CONNECT, DIGIT, MARKT, SG, TAXUD. 
Article 1(2). 
Article 1(1). 

EN  

3  

EN 

 
 
                                                 
– 

– 

– 

– 

Establishment  and  improvement  of  common  frameworks  in  support  of  cross-
border and cross-sectoral interoperability, 

Operation  and  improvement  of  existing  common  services  as  well  as  the 
establishment of new ones, 

Improvement of existing reusable generic tools as well as the establishment of 
new ones, 

Assessment of the ICT implications of EU legislation. 

Common  frameworks  are  strategies,  specifications,  methodologies,  guidelines  and 
similar approaches and documents. 

Common services are operational applications and infrastructures of a generic nature 
which meet common user requirements across policy areas. 

Generic  tools  are  reference  platforms,  shared  and  collaborative  platforms,  common 
components and similar building blocks that meet common user requirements across 
policy areas. 

Specific  actions  funded  by  the  programme  are  listed  in  the  rolling  ISA  Work 
Programme, which the Commission amends at least once a year after consulting the 
ISA Committee, established by the ISA Decision. 

3. 

METHODOLOGY 

The  interim  evaluation  of  the  ISA  programme  relied  on  multiple  methods  of 
investigation  for  collecting  quantitative  and  qualitative  data,  namely  desk  research, 
online  surveys,  interviews,  written  requests  for  additional  information  and  case 
studies,  involving  a  broad  range  of  representatives  from  Member  States  and  EU 
services as well as a limited number of other stakeholders. 

4. 

EVALUATION ISSUES AND QUESTIONS 

The evaluation focused on the following six main criteria: 

– 

– 

– 

Relevance  —  to  what  extent  do  the  actions  funded  by  the  ISA  programme 
contributes to the achievement of the programme objective(s)? To what extent 
are  the  ISA  programme’s  objective(s)  still  pertinent  in  relation  to  evolving 
needs and priorities at both national and EU level? 

Efficiency  —  how  efficiently  are  the  various  inputs  and  actions  being 
converted  into  outputs  and  results?  What  aspects  of  the  programme  are  the 
most efficient or inefficient, especially in terms of resources mobilised? 

Effectiveness — how effective have the ISA programme’s results and impacts 
been in achieving its specific and general objectives? Are there aspects that are 

EN  

4  

EN 

 
 
more or less effective than others, and — if so — what lessons can be drawn 
from this? 

– 

– 

– 

Utility — how do the ISA programme’s actions, results and impacts, achieved 
and anticipated, compare with the needs they are supposed to address? To what 
extent could measures be taken to improve the utility of the ISA programme’s 
actions, and what measures would these be? 

Sustainability  —  to  what  extent  is  the  financial,  technical  and  operational 
sustainability of solutions ensured? 

Coherence — to what extent do the actions form part of a ‘holistic’ approach 
within the programme framework? How well are synergies achieved between 
programme actions and with other EU activities? 

Apart  from  the  specific  questions  linked  to  the  evaluation  criteria,  a  number  of 
related  questions  were  raised.  Furthermore,  during  the  evaluation  the  Commission 
steering group added an additional criterion: 

 —      Coordination — to what extent are activities coordinated or aligned with the 
needs  of  other  stakeholders  and  Member  States?  To  what  extent  are  the  activities 
under other EU initiatives coordinated with ISA actions? 

5. 

FINDINGS 

Based on the data collected, the evaluation team has presented its findings in relation 
to the evaluation issues and questions defined. They may be summarised as follows: 

5.1. 

Relevance 

With  a  view  to  the  ultimate  objective  of  the  ISA  programme,  the  Commission  has 
demonstrated that the programme is fully aligned with the intermediate objective, as 
88 %  of  the  ISA  actions  facilitate  efficient  and  effective  cross-sectoral  interaction 
between European public administrations and 94 % facilitate electronic cross-border 
interaction  between  public  administrations  across  Europe.  However,  the  cross-
sectoral aspect of the intermediate ISA objective was not mentioned in the majority 
of the interviews and survey responses. 

The ISA programme is closely aligned with both the Digital Agenda for Europe and 
the  eGovernment  Action  Plan  2011-2015  and  is  perceived  as  contributing  to  the 
implementation  of  the  Single  Market  Act,  the  Service  Directive,  the  environmental 
protection directives and the Reuse of Public Sector Information Directive. 

In  addition,  the  programme  is  considered  to  be  relevant  in  responding  to  Member 
States’  needs,  as  the  needs  addressed  are  still  recognised  as  current  needs  of 
European public administrations. 

EN  

5  

EN 

 
 
The improvement of coordination between the ISA programme and both the Member 
States and Commission services and the increased reusability of ISA solutions were 
additional needs identified by Member States. 

5.2. 

Efficiency 

Overall,  the  ISA  programme  is  considered  to  be  efficient.  First  of  all,  the  smooth 
process leading to the ISA Committee’s favourable opinions on the ISA rolling work 
programme and its first and second revisions, as well as the swift release of funds on 
average  six  working  days  after  adoption  of  the  work  programme  and  its  revisions, 
have  ensured  efficient  implementation.  Secondly,  as  the  budget  allocated  is  almost 
equal  to  the  budget  committed,  and  the  total  budget  for  the  period  2010-12  is  very 
close  to  the  estimated  financial  cost  for  that  period,  the  budgetary  resources  are 
considered to be allocated and consumed efficiently. As to programme performance, 
it is generally on track, despite some delays in meeting some milestones 

Some aspects of the ISA programme are not perceived as efficient, two in particular 
being singled out. Firstly, the engagement of stakeholders is not considered efficient, 
in  particular  the  Member  States’  involvement  in  the  programme,  more  specifically 
that of national interoperability experts, which can partly be explained by a lack of 
resources at national level. Second, the allocation of human resources is considered 
by  the  evaluators  as  only  partially  adequate,  given  the  high  turnover  and  the 
distribution  of  resources  compared  with  the  figures  specified  in  the  financial 
statement accompanying the proposal for the ISA Decision when it was adopted by 
the Commission and transmitted to the European Parliament and the Council. Some 
interviews pointed to the lack of expertise in specific domains, a lack of budgetary 
resources and inefficiency in implementing the programme, but the evaluators found 
no evidence confirming these perceptions. 

5.3. 

Effectiveness 

A specific assessment of the results achieved by the programme is not yet feasible as 
the ISA programme is still in its early stages and very few new actions have so far 
delivered  results.  Common  frameworks  were  perceived  as  the  most  effective 
solutions provided by the programme as compared to common services or reusable 
generic  tools.  The  action  ‘Assessment  of  ICT  implications’  was  highlighted  in 
particular as not yet having delivered the expected benefits.  

5.4. 

Utility 

Assessing  the  utility  of  a  programme  at  the  time  of  an  interim  evaluation  is 
questionable  and,  in  fact,  this  is  normally  only  done  in  final  evaluations.  The 
evaluators have, nevertheless, addressed the reusability aspect of ISA solutions. 

The  results  of  the  ISA  programme  are  partially  reused  at  both  Member  State  and 
Commission  level.  The  majority  of  Member  State  stakeholders  stated  that  ISA 
solutions  were  being  reused  at  Member  State  level.  Although  concrete  solutions 
could  not  always  be  named,  some  24 %  of  the  ISA  solutions  are  said  to  have  been 
reused, representing roughly 65 % of the total budget of the programme, with sTesta 

EN  

6  

EN 

 
 
accounting for the major share. A representative number of interviewees considered 
that  ISA  solutions  are  being  reused  in  the  Commission,  although  this  perception  is 
not  shared  by  a  small  number  of  respondents  to  the  online  surveys,  possibly 
indicating a lack of awareness of the ISA solutions. 

5.5. 

Sustainability 

As with utility, sustainability is typically not assessed during an interim evaluation. 
Even though the evaluators addressed this aspect, no definitive conclusion is drawn 
regarding the sustainability of the ISA solutions for two main reasons. First, the ISA 
programme  is  still  in  its  early  stages  and  has  not  yet  yielded  enough  results  and 
impacts  that  might  be  sustained.  Second,  as  very  little  evidence  could  be  gathered, 
the validity of any conclusion would be highly questionable. 

The  evaluation  report  noted  the  need  to  identify  possible  means  to  ensure  the  long 
term  sustainability  of  an  increasing  number  of  solutions  developed  by  the  ISA 
programme (and its predecessors). 

Regarding  financial  sustainability  the  proposed  Connecting  Europe  Facility  (CEF) 
was  identified  by  the  evaluators  as  one  of  the  possible  means  of  sustainability  of 
some of the operational activities of ISA.  

5.6. 

Coherence 

Synergies are fairly well established within the ISA programme, as 78 % of the ISA 
actions have yielded internal synergies. More generally, the majority of stakeholders 
perceive  synergies  as  not  well  or  not  well  enough  established  or  leading  to  some 
overlaps, which some stakeholders attribute to the lack of information communicated 
to stakeholders, the lack of synergies identified in the programme and the lack of a 
control mechanism to ensure the reuse of ISA solutions. 

On  the  other  hand,  the  ISA  programme  is  achieving  synergies  with  other  EU 
initiatives.  In  fact,  some  ISA  actions  have  synergies  with  the  ICT  Policy  Support 
Programme  under  the  Competitiveness  and  Innovation  Framework  Programme,  the 
Open Data Portal and eParticipation, and to a lesser extent, with the ICT programme 
under  the  seventh  Framework  Programme  (FP7-ICT),  ICT  standardisation  and  the 
European eJustice action plan. 

5.7. 

Coordination 

Member  States  are  not  sufficiently  engaged  in  the  ISA  programme.  Despite  their 
involvement in defining the priorities of the programme, a work programme revision 
process that includes Member States, and the high participation of Member States in 
the ISA Committee and Coordination Group7 meetings, the interaction between the 
ISA  programme  and  Member  States  to  ensure  that  the  ISA  programme  is 
implemented  in  line  with  the  needs  and  priorities  of  the  Member  States  is  not 

7 

The ISA Coordination Group is a working group established by the ISA Committee to address issues 
referred to it, assist the Commission in translating the European Interoperability Strategy into concrete 
actions and coordinate the alignment of national and ISA actions. 

EN  

7  

EN 

 
 
                                                 
perceived to be sufficiently effective. In the opinion of the evaluation team, the lack 
of involvement at national level can explain this gap and also explain the low level of 
awareness and reuse of the ISA solutions across Member States. 

Coordination  between  the  ISA  programme  and  other  EU  initiatives  is  ensured 
through  various  means:  formal  and  informal  meetings,  the  Commission’s  IT 
governance  and  its  inter-service  consultations,  presentations  to  the  CTI  (‘Comité 
Technique Informatique’), communication activities and external events. As regards 
the perception among Commission services of the effectiveness of this coordination, 
a  small  majority  of  interviewees  considered  it  not  sufficiently  effective  to  prevent 
duplication  of  work.  They  perceive  this  ineffective  coordination  as  a  barrier  to 
raising  awareness  of  the  ISA  solutions  across  Member  States  and  Commission 
services, and, consequently, to the reuse of ISA solutions by these stakeholders. 

6. 

RECOMMENDATIONS 

The evaluation was largely positive, describing the ISA programme as aligned with 
the  policy  priorities  of  the  European  Commission  and  the  needs  of  Member  States 
and implemented efficiently and coherently, delivering results that are reused by both 
Commission  services  and  Member  States.  Nevertheless  the  evaluation  report  also 
highlights some shortcomings and makes recommendations with regard to: 

•  Communication and raising awareness 

•  Engagement of stakeholders and project management continuity 

•  Avoiding  overlaps  and  duplications,    of  work,  increasing  reusability  and 

ensuring sustainability 

Furthermore, the evaluators consider that four of the eleven recommendations from 
the final evaluation of the predecessor programme IDABC are still applicable to the 
ISA programme. 

6.1. 

Communication and raising awareness 

The  ISA  programme  must  ensure  that  all  stakeholders  involved  in  the  ISA 
programme are well aware of the objectives of each action, the contribution of these 
actions  to  the  programme’s  objectives,  and  the  intended  and  actual  results.  Even 
though these aspects are covered in the work programme documentation and on the 
ISA portal, they should be made available in a simplified form to communicate more 
effectively  to  a  wider  audience.  In  addition,  the  cross-sector  aspects  of  the  ISA 
actions should be described more precisely8. 

8  

Recommendation 1. 

EN  

8  

EN 

 
 
 
                                                 
The  ISA  programme  should  conduct  regular  meetings  with  all  action  owners  to 
exchange  information  on  the  current  progress  of  actions  and  explore  potential 
synergies9. 

The  ISA  programme  should  reinforce  promotion  and  communication  activities 
regarding  the  ISA  solutions  that  have  produced  concrete  results  by  continuously 
participating  in  events  at  national  level,  organising  ISA  events,  and  issuing 
publications  on  these  solutions,  as  well  as  by  playing  an  active  role  in  other  EU 
activities or programmes supporting opportunities for the reuse of ISA solutions. 

In  line  with  the  evaluators'  recommendations,  the  ISA  programme  will  increase 
collaboration  with  other  stakeholders,  i.e.  other  European  institutions  and  the  ICT 
industry.  Furthermore,  taking  into  consideration  the  evaluators  recommendation  it 
will identify relevant contacts within academia and private organisations that could 
bring added value through their involvement in specific ISA actions 10. 

The Commission will analyse the cost-effectiveness of these recommendations. 

The  ISA  programme  made  important  efforts  last  year  to  increase  participation  in 
national events and improve communication and information dissemination. Further 
to  these  efforts,  the  Commission  is  revising  the  communication  strategy  and  will 
complement  the  overall  strategy  by  pursuing  dedicated  communication  activities  in 
specific areas. 

6.2. 

Engagement of stakeholders and project management continuity 

Members  of  the  specialised  working  groups  established  by  the  ISA  Committee 
should report nationally to the ISA Coordination Group members to ensure that ISA 
solutions are aligned with needs and initiatives at national level. In addition, relevant 
stakeholders  from  the  respective  public  administrations  should  be  identified  by  the 
ISA  Coordination  Group  members  and  involved  on  an  ad-hoc  basis  in  formal 
meetings and workshops 11. 

The  Commission  proposes  to  support  the  Member  States  in  implementing  this 
recommendation by, for example: 

•  making use of the established networks with EU regions 

•  exploring means of remote participation in meetings (videoconferences) 

•  other means to be identified together with the Member States. 

9  
10  
11  

Recommendation 5. 
Recommendation 8. 
Recommendation 7. 

EN  

9  

EN 

 
 
                                                 
In  addition  the  ISA  programme  will  give  priority  to  activities  to  assess  the  ICT 
implications  of  EU  legislation,  which  Member  States  consider  to  be  an  important 
issue that has not yet been addressed12. 

The ISA programme should ensure continuity in the project management of actions 
by analysing the issues leading to the high overall turnover of human resources in the 
programme and by identifying mitigation actions13. 

The  evaluators  recalled  the  recommendation  from  the  IDABC  final  evaluation  that 
the  Commission  should  ensure  that  the  contractual  frameworks  are  in  place  in  due 
time for the launch of the next programme, allowing actions to be launched as soon 
as a work programme is adopted14. 

The Commission strongly promotes mobility among its staff. In order to assure that 
mobility  does  not  compromise  project  continuity,  and  to  exploit  competences 
available  in  different  services  (which  cannot  all  be  found  in  the  same  unit),  the 
Commission  is  exploring  synergies  among  its  services  by  promoting  inter-service 
contacts and stronger cooperation between services. 

The  Commission  is  closely  following  up  the  procedure  for  framework  contracts, 
which will assure the timely launch of the required call for tenders. 

6.3. 

Avoiding  overlaps  and  duplications,  increasing  reusability  and  ensuring 
sustainability 

The ISA programme should apply a business case approach in the selection of new 
actions  proposed  by  Member  States  and  Commission  services  and  involve  the 
Commission’s IT governance bodies in the evaluation of business cases for proposals 
from the Commission services15. 

The  Commission’s  IT  governance  bodies  should  ensure  that  external  synergies 
between ISA actions and other EU initiatives are identified and documented upfront. 

DIGIT's  participation  in  the  IT  governance  bodies  will  ensure  that  the  ISA 
programme is associated with the proceedings of these bodies so as to better identify 
cases and where ISA solutions can be reused16. 

The ISA programme should establish a control mechanism to ensure the reuse of ISA 
solutions. Reusability should be a feature of the ISA solutions selected for funding. 
These solutions must include documentation for their reuse.  

In  this  respect,  new  proposals  should  include  an  adequate  governance  structure  to 
support  reuse.  Reusable  solutions  and  opportunities  for  reuse  should  be  identified 
and communicated to relevant stakeholders 17. 

12  
13  
14  
15  
16  

Recommendation 3. 
Recommendation 4. 
IDABC Recommendation 9. 
Recommendation 2. 
Recommendation 6. 

EN  

10  

EN 

 
 
                                                 
The ISA programme should consider in due time the sustainability of ISA solutions 
by  identifying  different  sustainability  options,  such  as  charge-back  methods, 
financial  support  from  Commission  services  that  have  developed  specific  ISA 
solutions,  and  financial  support  from  an  ISA  follow-on  programme  or  other  EU 
programme1819. 

The  Commission  takes  a  holistic  approach  to  addressing  the  reusability  and 
sustainability of interoperability solutions by acting at different levels: 

1.  At governance level, by improving the links between the Commission’s IT 
governance20, the ISA programme and through the ISA committee with the 
Member States. The ISA is currently presented annually to the CTI and  
individual ISA actions led by Commission services must be approved by the 
ISPMB; 

2.  At the strategic level, by putting more emphasis on reusability and 

sustainability in future ISA work plans and if necessary by a possible review 
of the European Interoperability Strategy21 (EIS) 22; 

3.  At operational level, by taking action to ensure better reusability of 

interoperability solutions and to develop sustainability enablers (e.g. 
European Interoperability Architecture (EIA), European Federated 
Interoperability Repository (EFIR), Assessment of trans-European networks 
supporting EU policies, Sharing and reuse strategy) 23 and by considering 
possible means of financial sustainability including the proposed CEF. 

Finally,  in  line  with  the  evaluator's  recommendation,  the  ISA  unit  will  identify 
actions  producing  concrete  results  by  reviewing  ISA  solutions  every  two  years  as 
specified in Article 13(2) of the ISA Decision24. 

17  
18  
19  

20  

21  

22  

23  

24  

Recommendation 9. 
Recommendation 11. 
Related  IDABC  final  evaluation  recommendations:  (1)  The  Commission  should  have  further  open 
discussions  with  its  stakeholders  to  agree  on  which  of  the  efforts  and  budget  under  the  new  ISA 
programme  should  be  allocated  to  operating  IT  infrastructures,  developing  new  actions  and 
promoting/reusing  existing  solutions.  (2)  Information  on  project  sustainability  and  on  financial  and 
operational  sustainability  should  be  rendered  more  visible  and  be  better  explained  to  external 
stakeholders. 
Commission IT Governance bodies are: Comité Technique Informatique (CTI), High Level Commitee 
on  Information  Technologies  (HLCIT),  Activity  Based  Management  Steering  Committee  covering  IT 
(ABM+IT) and the Information Systems Project Management Board (ISPMB). 
Also addressing one recommendation in the IDABC final evaluation: a common ‘promotion’ document, 
focusing on the policy alignments and the synergies between the different eGovernment programmes, 
should be produced. 
The  "Annex  I  to  the  Commission  communication  on  interoperability  -  European  Interoperability 
Strategy (EIS)" is available at http://ec.europa.eu/isa/documents/isa_annex_i_eis_en.pdf 
These 
http://ec.europa.eu/isa/library/index_en.htm under the "ISA Work Programme" heading. 
Recommendation 10. 

ISA  Work 

Programme 

actions 

recent 

found 

can 

the 

at: 

be 

in 

EN  

11  

EN 

 
 
                                                                                                                                                         
7. 

CONCLUSIONS 

In the on-going implementation of the ISA programme, the Commission will pay the 
utmost 
associated 
recommendations, analysing them to validate and address the issues raised, wherever 
appropriate in close cooperation with the Member States. 

shortcomings  highlighted 

attention 

and 

the 

the 

to 

EN  

12  

EN