EN    EN  
 EUROPEAN  
COMMISSION 
Brussels, 18.1.2013  
COM(2013) 5 final 
  
REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND 
THE COUNCIL 
Interim evaluation of  the ISA programme 
 
 
 REPORT FROM THE COMMISSION TO  THE EUROPEAN PARLIAMENT AND 
THE COUNCIL 
Interim evaluation of the ISA programme 
 
EN   3   EN  
 1. I NTRODUCTION  
By decision of the European Parliame nt and the Council (the ISA Decision)1 the six-
year programme on interoperability solutio ns for European public administrations 
(the ISA programme) was launched on 1 January 2010 as a follow-up programme to 
the IDABC programme2. 
The purpose of this report is  to present the findings and recommendations of the 
interim evaluation of the ISA programme. Th e need to carry out the evaluation stems 
from Article 13(3) of the ISA Decisi on, which also requires the European 
Commission to communicate the results of the evaluation to the European Parliament and the Council by 31 December 2012. 
The evaluation
3 was performed by the Commission using a team of independent 
experts from a consultancy company (the  evaluation team). Representatives of 
Commission services have overseen the ev aluation through a Co mmission steering 
group4. 
2. B ACKGROUND  
The objective of the ISA programme is to support cooperation between European 
public administrations by facilitating the efficient and effective electronic cross-
border and cross-sectoral interaction betw een such administrations, including bodies 
performing public functions on their behalf, with a view to enabling the delivery of 
electronic public services supporting th e implementation of EU policies and 
activities5. 
As defined, the objective covers both the ultimate goal of supporting cooperation 
between European public administrations w ith a view to enabling the delivery of 
electronic public services supporting th e implementation of EU policies and 
activities and the intermediate objective of facilitating efficient and effective 
electronic cross-border and cross-sectoral interacti on between European public 
administrations to this end. 
To meet both the ultimate and intermedia te objectives and to provide ‘common and 
shared solutions facilitating interoperability’6, the ISA programme launches studies, 
projects and accompanying measures, in  specific actions, to support the 
                                                 
1 Decision No 922/2009/EC of the European Parliament and of the Council of 16 September 2009 on 
interoperability solutions for European public admi nistrations (ISA), OJ L 260, 3.10.2009, p. 20. 
2 Decision 2004/387/EC of the European Parliament and of the Council of 21 April 2004 on interoperable 
delivery of pan-European eGovernment services to public administrations, businesses and citizens 
(IDABC), OJ L 144, 30.4.2004 (see OJ L 181, 18.5.2004, p. 25). 
3 The full evaluation report is available at:     
http://ec.europa.eu/isa/documents/interi m_evaluation_of_the_isa_programme.pdf 
4  DG CONNECT, DIGIT, MARKT, SG, TAXUD. 
5 Article 1(2). 
6 Article 1(1). 
 
EN   4   EN  
 – Establishment and improvement of co mmon frameworks in support of cross-
border and cross-sectoral interoperability, 
– Operation and improvement of existing common services as well as the 
establishment of new ones, 
– Improvement of existing reusable generi c tools as well as the establishment of 
new ones, 
– Assessment of the ICT impli cations of EU legislation. 
Common frameworks are strategies, speci fications, methodologies, guidelines and 
similar approaches and documents. 
Common services are operational applications and infrastructures of a generic nature 
which meet common user requirements across policy areas. 
Generic tools are reference platforms, sh ared and collaborativ e platforms, common 
components and similar building blocks th at meet common user requirements across 
policy areas. 
Specific actions funded by the programme  are listed in the rolling ISA Work 
Programme, which the Commission amends at  least once a year after consulting the 
ISA Committee, established by the ISA Decision. 
3. M ETHODOLOGY  
The interim evaluation of the ISA pr ogramme relied on multiple methods of 
investigation for collecting quantitative and qualitative da ta, namely desk research, 
online surveys, interviews, written requests for additional information and case studies, involving a broad range of repres entatives from Member States and EU 
services as well as a limited number of other stakeholders. 
4. E
VALUATION ISSUES AND QUESTIONS  
The evaluation focused on the following six main criteria: 
– Relevance  — to what extent do the actions funded by the ISA programme 
contributes to the achievement of the programme objective(s)? To what extent are the ISA programme’s objective(s) st ill pertinent in relation to evolving 
needs and priorities at both  national and EU level? 
– Efficiency  — how efficiently are the va rious inputs and actions being 
converted into outputs and results? Wh at aspects of the programme are the 
most efficient or inefficient, especially in terms of resources mobilised? 
– Effectiveness  — how effective have the ISA programme’s results and impacts 
been in achieving its specific and genera l objectives? Are there aspects that are 
 
EN   5   EN  
 more or less effective than others, a nd — if so — what lessons can be drawn 
from this? 
– Utility  — how do the ISA programme’s actions , results and impacts, achieved 
and anticipated, compare with  the needs they are supposed to address? To what 
extent could measures be taken to impr ove the utility of the ISA programme’s 
actions, and what measures would these be? 
– Sustainability  — to what extent is the fina ncial, technica l and operational 
sustainability of solutions ensured? 
– Coherence  — to what extent do the actions fo rm part of a ‘ho listic’ approach 
within the programme framework? How well are synergies achieved between 
programme actions and with  other EU activities? 
Apart from the specific questions linked to  the evaluation criteria, a number of 
related questions were raised. Furthermor e, during the evalua tion the Commission 
steering group added an  additional criterion: 
 —      Coordination  — to what extent are activities coordinated or aligned with the 
needs of other stakeholders and Member States? To what extent are the activities under other EU initiatives coor dinated with ISA actions? 
5. F
INDINGS  
Based on the data collected, the evaluation t eam has presented its findings in relation 
to the evaluation issues and questions defined. They may be summarised as follows: 
5.1. Relevance 
With a view to the ultimate objective of the ISA programme, the Commission has 
demonstrated that the programme is fully al igned with the intermediate objective, as 
88 % of the ISA actions facilitate efficient and effective cross-sectoral interaction 
between European public administrations and 94  % facilitate elec tronic cross-border 
interaction between public administrati ons across Europe. However, the cross-
sectoral aspect of the intermediate ISA objective was not mentioned in the majority 
of the interviews and survey responses. 
The ISA programme is closely aligned with both the Digital Agenda for Europe and 
the eGovernment Action Plan 2011-2015 and is perceived as contributing to the 
implementation of the Single Market Act, the Service Directive, the environmental protection directives and the Reuse of Public Sector Information Directive. 
In addition, the programme is  considered to be relevant in responding to Member 
States’ needs, as the needs addressed ar e still recognised as  current needs of 
European public administrations. 
 
EN   6   EN  
 The improvement of coordination between the ISA programme and both the Member 
States and Commission services and the incr eased reusability of ISA solutions were 
additional needs identified by Member States. 
5.2. Efficiency 
Overall, the ISA programme is considered to be efficient. First of all, the smooth 
process leading to the ISA Committee’s fa vourable opinions on the ISA rolling work 
programme and its first and second revisions, as well as the swift release of funds on 
average six working days after adoption of  the work programme and its revisions, 
have ensured efficient implementation. Sec ondly, as the budget allocated is almost 
equal to the budget committed, and the total budget for the period 2010-12 is very close to the estimated financial cost fo r that period, the budgetary resources are 
considered to be allocated and consumed efficiently. As to programme performance, 
it is generally on track, despite some delays in meeting some milestones 
Some aspects of the ISA programme are not pe rceived as efficient, two in particular 
being singled out. Firstly, the engagement of stakeholders is not c onsidered efficient, 
in particular the Member States’ involvement in the programme, more specifically that of national interoperability experts, wh ich can partly be explained by a lack of 
resources at national level. Second, the allo cation of human resour ces is considered 
by the evaluators as only partially adequate, given the high turnover and the distribution of resources compared with the figures specified in the financial 
statement accompanying the proposal for the ISA Decision when it was adopted by the Commission and transmitted to the Eur opean Parliament and the Council. Some 
interviews pointed to the lack of expert ise in specific domains, a lack of budgetary 
resources and inefficiency in implementi ng the programme, but the evaluators found 
no evidence confirming these perceptions. 
5.3. Effectiveness 
A specific assessment of the results achieved  by the programme is not yet feasible as 
the ISA programme is still in its early stag es and very few new actions have so far 
delivered results. Common frameworks we re perceived as the most effective 
solutions provided by the programme as compared to common services or reusable generic tools. The action ‘Assessment of  ICT implications’ was highlighted in 
particular as not yet having deliv ered the expected benefits.  
5.4. Utility 
Assessing the utility of a programme at the time of an interim evaluation is 
questionable and, in fact, this is norma lly only done in final evaluations. The 
evaluators have, nevertheless, addressed the reusability aspect of ISA solutions. 
The results of the ISA programme are part ially reused at both Member State and 
Commission level. The majority of Member  State stakeholders stated that ISA 
solutions were being reused at Member State level. Although concrete solutions 
could not always be named, some 24
 % of the ISA solutions ar e said to have been 
reused, representing roughly 65  % of the total budget of th e programme, with sTesta 
 
EN   7   EN  
 accounting for the major share. A representa tive number of interv iewees considered 
that ISA solutions are being reused in the Commission, although this perception is 
not shared by a small number of responde nts to the online surveys, possibly 
indicating a lack of awaren ess of the ISA solutions. 
5.5. Sustainability 
As with utility, sustainability is typica lly not assessed during an interim evaluation. 
Even though the evaluators addressed this  aspect, no definitive conclusion is drawn 
regarding the sustainability of the ISA solu tions for two main reasons. First, the ISA 
programme is still in its early stages and has not yet yielded enough results and 
impacts that might be sustained. Second, as  very little evidence could be gathered, 
the validity of any conclusi on would be highly questionable.  
The evaluation report noted the need to id entify possible means to ensure the long 
term sustainability of an increasing number of solutions developed by the ISA 
programme (and its predecessors). 
Regarding financial sustainability the pr oposed Connecting Europe Facility (CEF) 
was identified by the evaluators as one of  the possible means of sustainability of 
some of the operational activities of ISA.  
5.6. Coherence 
Synergies are fairly well establis hed within the ISA programme, as 78  % of the ISA 
actions have yielded internal synergies. Mo re generally, the majority of stakeholders 
perceive synergies as not well or not we ll enough established or leading to some 
overlaps, which some stakeholders attribut e to the lack of information communicated 
to stakeholders, the lack of synergies identified in the programme and the lack of a 
control mechanism to ensure the reuse of ISA solutions. 
On the other hand, the ISA programme is  achieving synergies with other EU 
initiatives. In fact, some ISA actions ha ve synergies with the ICT Policy Support 
Programme under the Competitiveness and Innovation Framework Programme, the Open Data Portal and eParticipation, and to a lesser extent, with the ICT programme 
under the seventh Framework Programme (F P7-ICT), ICT standardisation and the 
European eJustice action plan. 
5.7. Coordination 
Member States are not sufficiently engage d in the ISA programme. Despite their 
involvement in defining the priorities of the programme, a work programme revision 
process that includes Member States, and the high particip ation of Member States in 
the ISA Committee and Coordination Group
7 meetings, the interaction between the 
ISA programme and Member States to ensure that the ISA programme is 
implemented in line with the needs and priorities of the Member States is not 
                                                 
7 The ISA Coordination Group is a working group established by the ISA Committee to address issues 
referred to it, assist the Commission in translating th e European Interoperability Strategy into concrete 
actions and coordinate the alignment of national and ISA actions. 
 
EN   8   EN  
 perceived to be sufficiently effective. In the opinion of the evaluation team, the lack 
of involvement at national le vel can explain this gap and al so explain the low level of 
awareness and reuse of the ISA solutions across Member States. 
Coordination between the ISA programme and other EU initiatives is ensured 
through various means: formal and info rmal meetings, the Commission’s IT 
governance and its inter-servi ce consultations, presenta tions to the CTI (‘Comité 
Technique Informatique’), communication activ ities and external events. As regards 
the perception among Commission services of  the effectiveness of  this coordination, 
a small majority of interviewees considered it not sufficiently effective to prevent duplication of work. They perceive this ineffective coordination as a barrier to 
raising awareness of the ISA solutions across Member Stat es and Commission 
services, and, consequently, to the reuse of ISA solutions by these stakeholders. 
 
6. R
ECOMMENDATIONS  
The evaluation was largely pos itive, describing the ISA pr ogramme as aligned with 
the policy priorities of the European Co mmission and the needs of Member States 
and implemented efficiently and coherently, delivering results that  are reused by both 
Commission services and Member States. Nevertheless the evaluation report also 
highlights some shortcomings and ma kes recommendations with regard to: 
• Communication and raising awareness 
• Engagement of stakeholders and project management continuity 
• Avoiding overlaps and duplications,  of  work, increasing reusability and 
ensuring sustainability 
Furthermore, the evaluators consider that four of the eleven recommendations from 
the final evaluation of the predecessor progr amme IDABC are still applicable to the 
ISA programme. 
6.1. Communication and raising awareness 
The ISA programme must ensure that all stakeholders involved in the ISA 
programme are well aware of the objectives of each action, the contribution of these 
actions to the programme’s objectives, a nd the intended and actual results. Even 
though these aspects are cove red in the work programme  documentation and on the 
ISA portal, they should be made available in a simplified form to communicate more 
effectively to a wider audience. In add ition, the cross-sector aspects of the ISA 
actions should be described more precisely8. 
                                                 
8  Recommendation 1. 
 
EN   9   EN  
 The ISA programme should conduct regular meetings with all action owners to 
exchange information on the current pr ogress of actions a nd explore potential 
synergies9. 
The ISA programme should reinforce pr omotion and communication activities 
regarding the ISA solutions that have produced concrete results by continuously 
participating in events at national level, organising ISA events, and issuing publications on these solutions, as well as by playing an active role in other EU 
activities or programmes supporting opportuni ties for the reuse of ISA solutions. 
In line with the evaluators' recommendations, the ISA programme will increase 
collaboration with other stakeholders, i.e. other European institutions and the ICT industry. Furthermore, taki ng into consideration the ev aluators recommendation it 
will identify relevant contacts within academ ia and private organisations that could 
bring added value through their i nvolvement in specific ISA actions 
10. 
The Commission will analyse the cost-e ffectiveness of these recommendations. 
The ISA programme made important efforts last year to increase participation in 
national events and improve communication and information dissemination. Further to these efforts, the Commission is revi sing the communication strategy and will 
complement the overall strategy by pursui ng dedicated communica tion activities in 
specific areas. 
6.2. Engagement of stakeholders and project management continuity 
Members of the specialised working groups established by the ISA Committee 
should report nationally to the ISA Coordination Group me mbers to ensure that ISA 
solutions are aligned with needs and initiati ves at national level. In addition, relevant 
stakeholders from the respective public administrations should be identified by the ISA Coordination Group members and invo lved on an ad-hoc basis in formal 
meetings and workshops 
11. 
The Commission proposes to support the Member States in implementing this 
recommendation by, for example: 
• making use of the established networks with EU regions 
• exploring means of remote participa tion in meetings (videoconferences) 
• other means to be identified together with the Member States. 
                                                 
9  Recommendation 5. 
10  Recommendation 8. 
11  Recommendation 7. 
 
EN   10   EN  
 In addition the ISA programme will give pr iority to activities to assess the ICT 
implications of EU legislation, which Memb er States consider to be an important 
issue that has not yet been addressed12. 
The ISA programme should ensure continuity  in the project management of actions 
by analysing the issues leading to the high overall turnover of human resources in the 
programme and by identifying mitigation actions13. 
The evaluators recalled the recommendati on from the IDABC final evaluation that 
the Commission should ensure that the cont ractual frameworks are in place in due 
time for the launch of the next programme, a llowing actions to be launched as soon 
as a work programme is adopted14. 
The Commission strongly promotes mobility am ong its staff. In order to assure that 
mobility does not compromise project c ontinuity, and to exploit competences 
available in different services (which cannot all be found in the same unit), the 
Commission is exploring synergies among its  services by promoting inter-service 
contacts and stronger coope ration between services. 
The Commission is closely following up th e procedure for framework contracts, 
which will assure the timely launch of the required call for tenders. 
6.3. Avoiding overlaps and duplications, increasing reusability and ensuring 
sustainability 
The ISA programme should apply a business ca se approach in the selection of new 
actions proposed by Member States an d Commission services  and involve the 
Commission’s IT governance bodies in the ev aluation of business cases for proposals 
from the Commission services15. 
The Commission’s IT governan ce bodies should ensure that external synergies 
between ISA actions and other EU initiativ es are identified and documented upfront. 
DIGIT's participation in the IT govern ance bodies will ensu re that the ISA 
programme is associated with the proceedings  of these bodies so as to better identify 
cases and where ISA solutions can be reused16. 
The ISA programme should establish a contro l mechanism to ensure the reuse of ISA 
solutions.  Reusability should be a feature of the ISA solutions selected for funding. 
These solutions must include doc umentation for their reuse.  
In this respect, new proposals should incl ude an adequate govern ance structure to 
support reuse. Reusable solu tions and opportunities for reuse should be identified 
and communicated to re levant stakeholders 17. 
                                                 
12  Recommendation 3. 
13  Recommendation 4. 
14  IDABC Recommendation 9. 
15  Recommendation 2. 
16  Recommendation 6. 
 
EN   11   EN  
 The ISA programme should consider in due time the sustainability of ISA solutions 
by identifying different sustainability options, such as charge-back methods, financial support from Commission servic es that have developed specific ISA 
solutions, and financial support from an  ISA follow-on programme or other EU 
programme
1819. 
The Commission takes a holis tic approach to addressing the reusability and 
sustainability of interoperability solutions by acting at different levels: 
1. At governance level, by improving the links between the Commission’s IT 
governance20, the ISA programme and through the ISA committee with the 
Member States. The ISA is currently presented annually to the CTI and  
individual ISA actions le d by Commission services must be approved by the 
ISPMB; 
2. At the strategic level, by putting more emphasis on reusability and 
sustainability in future ISA work plan s and if necessary by a possible review 
of the European Interoperability Strategy21 (EIS) 22; 
3. At operational level, by taking actio n to ensure better reusability of 
interoperability solutions and to de velop sustainabili ty enablers (e.g. 
European Interoperability Archite cture (EIA), European Federated 
Interoperability Repository (EFIR), Assessment of trans-European networks supporting EU policies, Sharing and reuse strategy)
 23 and by considering 
possible means of financial sustainab ility including the proposed CEF. 
Finally, in line with the evaluator's recommendation, the ISA unit will identify 
actions producing concrete results by re viewing ISA solutions every two years as 
specified in Article 13( 2) of the ISA Decision24. 
                                                                                                                                                         
17  Recommendation 9. 
18  Recommendation 11. 
19  Related IDABC final evaluation recommendations: (1) The Commission should have further open 
discussions with its stakeholders to agree on which of the efforts and budget under the new ISA 
programme should be allocated to operating IT  infrastructures, developing new actions and 
promoting/reusing existing solutions. (2) Information on project sustainability and on financial and 
operational sustainability should be  rendered more visible and be better explained to external 
stakeholders. 
20  Commission IT Governance bodies are: Comité Technique Informatique (CTI), High Level Commitee 
on Information Technologies (HLCIT), Activity Based Management Steering Committee covering IT 
(ABM+IT) and the Information Systems Project Management Board (ISPMB). 
21  Also addressing one recommendation in the IDABC final evaluation: a common ‘promotion’ document, 
focusing on the policy alignments and the synerg ies between the different eGovernment programmes, 
should be produced. 
22  The " Annex I to the Commission communication on interoperability - European Interoperability 
Strategy (EIS) " is available at http://ec.europa.eu/isa/docume nts/isa_annex_i_eis_en.pdf  
23  These actions can be found in the recent ISA Work Programme at: 
http://ec.europa.eu/isa/library/index_en.htm  under the "ISA Work Programme" heading. 
24  Recommendation 10. 
 
EN   12   EN  
 7. C ONCLUSIONS  
In the on-going implementation of the ISA programme, the Commission will pay the 
utmost attention to the shortcomi ngs highlighted and the associated 
recommendations, analysing them to validate and address the issues raised, wherever appropriate in close cooperat ion with the Member States.