COMMISSION STAFF WORKING DOCUMENT eu-LISA EVALUATION Accompanying the document Report from the Commission to the European Parliament and the Council on the functioning of the European Agency for the large-scale IT systems in the area of Freedom, Security and Justice (eu-LISA)
Date: 2017-06-29
Institutions: Directorate-General for Migration and Home Affairs, European Commission
Subjects: EU office or agency, European security, Schengen Information System, area of freedom, security and justice, exchange of information, external border of the EU, illegal migration, information system, personal data, rights of aliens

EN 

   EN 

   EUROPEAN COMMISSION  Brussels, 29.6.2017  SWD(2017) 249 final   COMMISSION STAFF WORKING DOCUMENT eu-LISA EVALUATION Accompanying the document Report from the Commission to the European Parliament and the Council  on the functioning of the European Agency for the large-scale IT systems in the area of Freedom, Security and Justice (eu-LISA) {COM(2017) 346 final} {SWD(2017) 250 final}  
 
 
CONTENTS 

1. Introduction .......................................................................................................................... 3 

1.1. The purpose and the scope of the evaluation ................................................................... 3 

2. Background to the initiative ................................................................................................ 4 

2.1. Objective of the initiative - eu-LISA ............................................................................... 4 

2.2. Intervention logic ............................................................................................................. 5 

2.3. Baseline ........................................................................................................................... 6 

3. Main Evaluation Questions ................................................................................................. 7 

4. Method ................................................................................................................................... 7 

4.1. Timeline and steering ...................................................................................................... 7 

4.2. Consultation (Information sources) ................................................................................. 8 

4.3. Evaluation risk assessment .............................................................................................. 9 

4.4. Limitations of the evaluation ........................................................................................... 9 

5. Implementation state of play (Results) ............................................................................. 10 

5.1. State of Play ................................................................................................................... 10 

5.2. Cooperation with the main stakeholders ....................................................................... 11 

6. Findings - Answers to the evaluation questions ............................................................... 12 

6.1. Effectiveness .................................................................................................................. 12 

6.2. Efficiency ....................................................................................................................... 14 

6.3. Coherence ...................................................................................................................... 17 

6.4. Relevance and added-value ........................................................................................... 18 

7. Ways to improve ................................................................................................................. 19 

7.1. Which are the concrete needs and opportunities to further increased practical 
cooperation with Member States and EU bodies? ................................................................ 19 

7.2. To what extent could eu-LISA's current mandate, tasks and/or capabilities address 
these needs? .......................................................................................................................... 20 

7.3. How could eu-LISA's mission, tasks, working practices or activities be further 
developed in order to meet these needs? .............................................................................. 20 

7.4. How could eu-LISA increase its added value and its contribution towards the EU and 
the Member States in the future using the capabilities and competencies already in place? 20 

8. Other impacts ..................................................................................................................... 21 

9. Conclusions ......................................................................................................................... 21 

9.1. Main findings ................................................................................................................. 21 

9.2. Recommendations ......................................................................................................... 21 

2 

 
 
1. INTRODUCTION 

The European Agency for the operational management of large-scale IT systems in the area of 
freedom,  security  and  justice  (eu-LISA)  was  set  up  to  provide  a  long-term  solution  for  the 
operational  management  of  large-scale  IT  systems  in  this  area.  The  Agency  is  currently 
responsible for  the  operational  management  of  the  following  IT  systems:  second  generation 
Schengen Information System (SIS II), the Visa Information System (VIS) and Eurodac.1 

This  Staff  Working  Document  is  based  on  the  final  evaluation  report  issued  by  an  external 
contractor (Ernst and Young — EY)2 and summarises the findings of the external evaluation 
of the action of eu-LISA since it took up its core tasks in December 2012. It also outlines the 
possible impact of the evaluation on the applicable legal framework, in particular on the eu-
LISA establishing Regulation. 

1.1. Purpose and scope of the evaluation 

The purpose of the evaluation is laid down in Article 31 of Regulation (EU) 1077/2011 of the 
European  Parliament  and  of  the  Council  of  25 October  2011  establishing  the  European 
Agency  for  the  operational  management  of  large-scale  IT  systems  in  the  area  of  freedom, 
security  and  justice3  (hereinafter  referred  to  as  the  establishing  Regulation).  The  Regulation 
stipulates  that,  ‘within  3  years  from  1 December  2012,  and  every  4  years  thereafter,  the 
Commission, in close consultation with the Management Board, shall perform an evaluation 
of the action of the Agency’.4  

In accordance with Article 31(1), the evaluation must: 

i)  examine  the  way  and  the  extent  to  which  the  Agency  effectively  contributes  to  the 
operational management of large-scale IT systems in the area of freedom, security and 
justice and fulfils its statutory tasks; and 

ii)  assess  the  role  of  the  Agency  in  the  context  of  a  Union  strategy  aimed  at  a 

coordinated, cost-effective and coherent IT environment at Union level. 

1   The  system  for  comparing fingerprints  of  asylum  seekers and  some  categories of  illegal  immigrants,  which 

facilitates the application of the Dublin Regulation. 

2   Independent  external  evaluation  of  the  European  Agency  for  the  operational  management  of  large-scale  IT 
systems  in  the  area  of  freedom,  security  and  justice  -  eu-LISA,  Final  evaluation  report,  March  2016; 
http://bookshop.europa.eu/is-bin/INTERSHOP.enfinity/WFS/EU-Bookshop-Site/en_GB/-
EUR/ViewPublication-Start?PublicationKey=DR0 116464. 

3   OJ L 286, 1.11.2011 p. 1. 
4   Article 31 of Regulation (EU) No 1077/2011 of the European Parliament and the Council of 25 October 2011 
provides  in  paragraph  1  that  ‘Within  3  years  from  1 December  2012,  and  every  4  years  thereafter,  the 
Commission, in close consultation with the Management Board, shall perform an evaluation of the action of 
the Agency. The evaluation shall examine the way and extent to which the Agency effectively contributes to 
the operational management of large-scale IT systems in the area of freedom, security and justice and fulfils 
its tasks laid down in this Regulation. The evaluation shall also assess the role of the Agency in the context of 
a Union strategy aimed at a coordinated, cost- effective and coherent IT environment at Union level that is to 
be established in the coming years’. Paragraph 2 provides that ‘On the basis of the evaluation referred to in 
paragraph  1,  the  Commission,  after  consulting  the  Management  Board,  shall  issue  recommendations 
regarding changes to this Regulation, also in order to bring it further in line with the Union strategy referred 
to in paragraph 1. The Commission shall forward those recommendations, together with the opinion of the 
Management  Board,  as  well  as  appropriate  proposals  to  the  European  Parliament,  the  Council  and  the 
European Data Protection Supervisor’. 

3 

 
 
 
                                                            
Paragraph  (2)  of  Article  31  further  states  that,  on  the  basis  of  the  present  evaluation,  the 
Commission, after consulting the Management Board, will issue recommendations regarding 
possible changes to the Regulation.  

The  evaluation  examined  the  implementation  of  eu-LISA’s  tasks,  taking  into  account  the 
relevant  legal  and  policy  framework,  i.e.  the  eu-LISA  establishing  Regulation  and  the 
legislative instruments that govern the IT systems the Agency operates at central level. Given 
that no formal Union strategy has been put in place, the evaluation focused on the Agency’s 
contribution to the establishment of a coordinated, effective and coherent IT environment for 
managing large-scale IT systems that support the implementation of Justice and Home Affairs 
(JHA) policies. 

In  addition  to  assessing  whether  the  Agency  has  fulfilled  its  current  statutory  tasks,  the 
evaluation  also  assessed  the  extent  to  which  eu-LISA  has  efficiently  implemented  the  new 
tasks that it assumed in 2013 and 2014. This includes VISION5 and DubliNet6, which are the 
communication  tools  that  allow  central  visa  authorities  to  consult  on  visa  applications 
(VISION) and ensure the exchange of information for the effective application of the Dublin 
Regulation by the Member States (DubliNet). 

The evaluation also looked into the way in which eu-LISA has organised and implemented ad 
hoc projects such as the 2015 pilot on the Smart Borders proposals, which was entrusted to it 
by way of a delegation agreement.7 It also looked into major evolutions of existing systems 
under its management, e.g. the Biometric Matching System (BMS) or the Eurodac Recast.8  

The evaluation also examined how eu-LISA’s role, work and/or organisation could be adapted 
and improved, with the possible need for revision of or extension of the tasks entrusted to it in 
the Regulation. 

In line with the Commission's Guidelines on Better Regulation9 the evaluation was conducted 
by  assessing  eu-LISA's  action  against  the  following  criteria:  effectiveness,  efficiency, 
coherence, relevance and EU-added value.  

2. BACKGROUND TO THE INITIATIVE 

2.1. Objective of the initiative — eu-LISA 

As stated in the impact assessment that accompanies the Regulation, eu-LISA was set up to 
achieve  synergies  by  ensuring  the  operational  management  of  large-scale  IT  systems  in  a 

5   Service level agreement on the temporary modalities of cooperation between the Member States and Iceland, 
Liechtenstein  and  Norway  using  VISION,  represented  by  the  Presidency  of  the  Council  of  the  European 
Union  and  the  European  Agency  for  the  operational  management  of  large-scale  IT  systems  in  the  area  of 
freedom, security and justice, signed on 27 May 2013. 

6   Service  level  agreement  between  the  European  Agency  for  the  operational  management  of  large-scale  IT 
systems  in  the  area  of  freedom,  security  and  justice  and  the  services  of  the  European  Commission  (DG 
HOME)  for  the  transfer  of  the  technical  support  for  the  operational  management  of  DubliNet  to  eu-LISA, 
signed on 31 July 2014. 

7   Delegation agreement between the European Union represented by the European Commission, represented for 
the  purposes  of  signing  this  delegation  agreement  by  Matthias  Ruete,  Director  General,  DG  Migration  and 
Home Affairs and the European Agency for the operational management of large-scale IT systems in the area 
of freedom, security and justice represented by Krum Garkov, signed on 14 January 2015.  

8   Regulation  (EU)  No 603/2013  of  the  European  Parliament  and  of  the  Council  of  26 June  2013,  L18 0, 

29.6.2013, p. 1. 

9   COM(2015) 111 final. 

4 

 
 
                                                            
single  entity,  benefitting  from  economies  of  scale,  creating  critical  mass  and  ensuring  the 
highest utilisation rate of capital and human resources. 

The  Agency  began  operations  on  1 December  2012.  The  Agency’s  seat  is  Tallinn,  Estonia, 
whilst its operational data centre is in Strasbourg, France. There is also a backup site in Sankt 
Johann im Pongau, Austria. 

Apart from the three systems it currently manages, its establishing Regulation specifies that 
eu-LISA  may  also  be  made  responsible    for  the  preparation,  development  and  operational 
management of other large-scale IT systems in the area of freedom, security and justice, only 
if so provided by relevant legislative instruments based on Articles 67 to 89 TFEU.10 

Since  it  was  established,  eu-LISA  has  taken  on  tasks  other  than  those  envisaged  in  the 
legislative instruments on the systems or in the establishing Regulation, such as DubliNet and 
VISION.  It  also  carried  out  a  Smart  Borders  pilot  project  in  2015  in  order  to  help  the 
Commission prepare for the revised Smart Borders initiative. This resulted in particular in the 
new proposal for a Regulation for an Entry/Exit System (EES).11 In accordance with Article 
1(3)  of  the  establishing  Regulation,  the  EES  proposal  envisages  eu-LISA  taking  on  the 
development and operational management of the EES. 

2.2. Intervention logic 

For the purpose of the evaluation, a detailed analysis was carried out identifying the Agency's 
objectives (general/specific) and tasks and the links between them. This intervention logic is 
based  on  the  Regulation  establishing  the  Agency  as  well  as  the  relevant  Regulations  and 
Decisions that govern the systems under its operational management. A detailed analysis was 
carried  out  that  identifies  the  Agency’s  objectives  (general/specific),  tasks  and  the  links 
between them. 

The diagram below depicts the intervention logic. The tasks can be considered ‘macro-tasks’, 
which can be broken down into several more detailed specific tasks entrusted to eu-LISA by 
the applicable legal framework. These specific tasks which are not illustrated in the diagram 
have been examined by the external contractor in Annex 10 of the final evaluation report. The 
fulfilment of these tasks is assessed later on in the chapters on effectiveness and efficiency. 

10 ‘The Agency may also be made responsible for the preparation, development and operational management of 
large-scale IT systems in the area of freedom, security and justice other than those referred to in paragraph 2, 
only if so provided by relevant legislative instruments, based on Articles 67 to 89 TFEU, taking into account, 
where appropriate, the developments in research referred to in Article 8 of this Regulation and the results of 
pilot schemes referred to in Article 9 of this Regulation.’ (Article 1(3) of the Regulation). 

11 Proposal for a Regulation of the European Parliament and of the Council establishing an Entry/Exit System 
(EES) to register entry and exit data and refusal of entry data of third country nationals crossing the external 
borders of the Member States of the European Union and determining the conditions for access to the EES for 
law enforcement purposes and amending Regulation (EC) No 767/2008 and Regulation (EU) No 1077/2011. 
COM(2016) 194 final of 6.4.2016. 

5 

 
 
 
                                                            
On  the  basis  of  the  intervention  logic,  the  Commission  defined  the  evaluation  questions  in 
close  cooperation  with  eu-LISA  (and  the  Agency’s  Management  Board).  EY  then 
supplemented  them  with  additional  questions  to  ensure  that  all  evaluation  criteria  and  all 
components of the intervention logic were covered. 

A  detailed  analytical  framework  was  then  developed.  Precise  evaluation  criteria,  indicators 
and descriptors were identified for each evaluation question (for details, see Annex 3 of the 
annexes  of  the  final  evaluation  report)  as  well  as  the  underlying  sources  of  data.  The 
exceptional cases where the indicators could not be analysed (e.g. due to the lack of Service 
Level Agreements, indicators and data quality controls used for incident management) were 
duly reported, and corresponding recommendations were made in the final report. 

2.3. Baseline – situation before December 2012 

Before a dedicated Agency was created, the Commission was in charge of the development 
and management of Eurodac and VIS. The Commission also developed SIS II. 

However, the legal instruments that establish SIS II and VIS entrusted the Commission with 
the operational management of the systems for a transitional period only, with the possibility 
to delegate operational management and tasks related to the implementation of the budget to 

6 

 
 
 
national  public  sector  bodies  in  two  Member  States.  The  same  legal  instruments  also 
established the location of the central systems in Strasbourg (France), with a backup in Sankt 
Johann  im  Pongau  (Austria).  However,  they  did  not  establish  which  specific  entity  (a 
management  authority  funded  from  the  general  budget  of  the  European  Union)  should  be 
responsible for long-term management. 

A  long-term  management  solution  for  SIS  II,  VIS  and  Eurodac  was  needed.  Five  policy 
options12 were analysed, and the creation of a dedicated regulatory agency was chosen as the 
most efficient solution.13  

The  first  evaluation  of  eu-LISA  is  the  first  opportunity  to  gauge  the  extent  to  which  the 
objectives of the long-term solution have been met and to what extent eu-LISA has achieved 
its mandate. 

3. MAIN EVALUATION QUESTIONS 

In  accordance  with  the  terms  of  reference,  the  evaluation  aimed  to  formulate  findings, 
conclusions and recommendations, in particular to answer the following questions: 

  Whether  changes  to  eu-LISA’s  legal  framework  should  be  considered  and,  if  so, 

which? and 

  Whether  other  structural,  organisational  or  staffing  changes  and/or  changes  to  eu-
LISA’s  working  practices  and  administrative  documents,  which  do  not  require 
amendments to its legal framework, should be considered and, if so, which? 

Specific  evaluation  questions  were  formulated  in  the  terms  of  reference  and  adapted  to  the 
intervention logic. For details on these questions and replies to them, see Chapter 6. 

4. METHOD 

4.1. Timeline and steering 

The  evaluation  covered  the  period  from  1 December  2012  to  30 September  2015,  and  took 
into account two issues: first, that the Agency took over its core tasks in December 201214, 
and second, the completion of the Smart Borders pilot project in September 2015. 

The fieldwork and data collection phase ran from March to September 2015, with additional 
ad hoc data collection on specific questions, such as financial management or matters related 
to data protection, in late 2015 and early 2016. The deliverables of the contractor consisted of: 

 
 
 
 

the inception report; 
two progress reports; 
the draft final report and the final report and its annexes; and 
two  presentations  to  the  eu-LISA  Management  Board  (March  and  November  2015 
meetings). 

12 1) The Baseline option (i.e. the Commission fully responsible for Eurodac and shared responsibilities for SIS 
II  and  VIS  with  two  Member  States);  2)  The  Baseline  +  option  (shared  responsibilities  between  the 
Commission and two Member States for SIS II, VIS and Eurodac); 3) New Regulatory Agency, which would 
assume  responsibility  for  the  long-term  management  of  SIS  II,  VIS,  and  EURODAC;  4)  Handing  over  the 
management of the three systems to FRONTEX; 5) The Commission for VIS and Eurodac, and Europol for 
SIS II. 

13  For  details,  refer  to  the  Impact  Assessment  accompanying  the  proposal  for  a  Regulation  establishing  a 
European Agency for the operational management of large-scale IT systems in the area of freedom, security 
and justice, SEC(2009) 837. 

14  eu-LISA took over VIS on 1 December 2012, SIS II in May 2013 and Eurodac in June 2013. 

7 

 
 
                                                            
Each  deliverable  was  the  subject  of  online  consultations  via  SharePoint  and  meetings 
involving a dedicated Inter-Service Steering Group (ISSG) that supported the evaluation and 
was  made  up  of  the  representatives  of  relevant  Commission  departments  (DG  HOME,  DG 
JUST  (+SRD),  DG  HR,  DG  BUDG,  DG  DIGIT,  DG  CNECT,  Secretariat-General,  Legal 
Service). A Steering Committee chaired by the Commission then reviewed and approved all 
deliverables  submitted  by  the  contractor.  The  Steering  Committee  was  composed  of  three 
representatives  of  the  Commission/DG  HOME,  two  members  of  the  eu-LISA  Management 
Board (delegates of Estonia and Finland) and one eu-LISA representative. 

The contract between the Commission and EY was scheduled to run from March to December 
2015  but  the  evaluation  exercise  proved  to  be  too  complex  for  the  contractor  to  deliver  the 
required quality within this timeframe. To perform all the tasks, a thorough analysis of both 
the  complex  legal  framework  of  this  Agency,  as  laid  down  in  the  systems’  legislative 
instruments and in the establishing Regulation and its functioning was needed. In addition, the 
two  monitoring  bodies  (the  Steering  Committee  and  ISSG)  that  closely  followed  the 
evaluation  provided  valuable  feedback  for  the  contractor,  who  needed  time  to  ensure  the 
appropriate follow-up. 

All these objective factors led to more review cycles and to the overall project schedule being 
updated, resulting in the contract being extended to 30 March 2016. However, there was no 
financial impact on the EU budget. 

4.2. Consultation (information sources) 

The  methodology  took  into  account  the  views  of  relevant  stakeholders  at  European  and 
national level. Different data collection tools, an e-survey and interviews were used to consult 
the following stakeholders in particular: 

  EU  institutions  (European  Commission,  European  Parliament,  Council  of  the 
European  Union,  European  Court  of  Auditors)  and  the  European  Data  Protection 
Supervisor; 

  eu-LISA  stakeholders 

(Management  Board  and  Advisory  Groups,  external 

contractors); 

  National competent authorities (authorities responsible for/with access to SIS II, VIS 
and  Eurodac  at  national  level)  in  Member  States,  Schengen  and  Dublin/Eurodac 
Associated Countries (Iceland, Liechtenstein, Norway, Switzerland); 

  EU  agencies  active  in  the  JHA  field  (Eurojust,  Europol,  CEPOL,  EASO,  FRA, 

EBCGA) and ENISA. 

The eu-LISA staff and management were also interviewed to ensure that the experience and 
views of the Agency were duly taken into account. 

In all, 50 interviews were conducted with eu-LISA staff in Tallinn and Strasbourg, as well as 
47 interviews with other stakeholders (European Commission, European Parliament, Council, 
European  Court  of  Auditors,  European  Data  Protection  Supervisor,  eu-LISA  Management 
Board and Advisory Groups, EU agencies). 

The  e-survey  sent  to  the  eu-LISA  Management  Board  and  Advisory  Group  members  had  a 
50 % response rate, with 92 replies in total. 

In  addition,  the  contractors  observed  one  Management  Board  meeting  and  three  Advisory 
Group meetings. 

An extensive document  review and five thematic project case studies were  also carried out. 
For details, see Annexes 4, 5, 9 and 12 to the final report. 

8 

 
 
4.3. Evaluation risk assessment 

A number of risks were identified for the evaluation, as detailed in the inception and progress 
reports.  Appropriate  mitigation  measures  were  identified  for  each  risk, including  the parties 
responsible for implementing the corrective measures. 

The most significant risks were identified and ranked in order of impact and probability: 

 
 

 

requested documents may not exist or do not include the expected information; 
interviewees  may  not  be  able  to  answer  questions  on  specific  topics  due  to 
confidentiality concerns or lack of information; 
the  benchmark  exercise  cannot  be  completed  due  to  lack  of  data  comparability  and 
unavailability of agencies; 

  stakeholders’ written comments may contradict each other. 

In the end, only the risks related to the unavailability of data materialised in part. This was the 
case for the benchmarking exercise, which failed to yield the expected results despite repeated 
attempts;  this  was  mainly  due  to  the  unique  nature  of  the  evaluated  entity.  It  proved 
impossible to benchmark the activities of the Agency carried out within its mandate against an 
equivalent  organisation  dealing  with  IT,  not  only  at  the  corporate  level  but  also  at  the  core 
business  level.  The  efforts  made  to  obtain  at  least  comparable  data,  even  from  other  JHA 
agencies,  were  equally  futile.  However,  it  did  manage  to  use  some  data  collected  from  the 
Agency for benchmarking in the analysis, in particular in the assessment of cost-effectiveness. 

4.4. Limitations of the evaluation 

  Approach limitations for assessment criteria 

The terms of reference for the external evaluation provided that ‘eu-LISA’s activities and in 
particular its core tasks of the operational management of SIS II, VIS and Eurodac, shall be 
evaluated  taking  into  account  the  evaluation  criteria’.  When  responding  to  the  questions 
relating  to  effectiveness  and  efficiency,  it  was  possible  to  evaluate,  in  many  instances,  eu-
LISA’s  specific  tasks  due  to  their  direct  link  to  the  Agency’s  objectives.  However,  on 
coherence, relevance and added-value, it was not possible to evaluate each individual task in 
turn as questions relating to these evaluation criteria related primarily to the Agency’s overall 
activities. The evaluation therefore assessed eu-LISA’s overall activities against these criteria. 

  Approach limitations for assessment of security 

The security requirements described by the legal instruments governing the Agency and the 
systems under its responsibility can be categorised in three levels as follows:  

  Security  controls  and  measures  that  must  be  implemented  in  technical  terms  in  the 

systems; 

  Security requirements related to the management of the systems; and 
  Security requirements related to the organisation of security functions and roles in the 

Agency. 

EY was able to  evaluate the effective implementation of the security controls and measures 
that  must  be  technically  implemented  in  the  system  (level  1),  with  the  exception  of  the 
requirements that would have involved the evaluation team accessing the systems for testing, 
which  was  not  legally  possible.  In  addition,  EY  failed  to  share  all  the  details  of  the 
methodology that applied to the IT and physical security assessment (as specified in Annex 7 
of the final evaluation report) with the Steering Committee. 

9 

 
 
 
  Data availability (confidentiality of documentation) 

The high level of criticality and confidentiality of some documents prevented access to all the 
requisite documents. However, the triangulation of data that was gathered allowed coverage, 
with  a  few  exceptions,  of  all  the  indicators  and  descriptors.  Where  the  confidentiality  of 
documentation  impacted  the  provision  of  an  answer  to  the  evaluation  questions,  this  was 
indicated in the final report. In addition, the evaluators received security clearance in order to 
ensure that documentation that was inaccessible due to security concerns could be accessed as 
appropriate. 

It is also worth noting that the restrictions due to confidentiality were not significant. Of the 
157 documents required for the evaluation, only three15 were not provided due to reasons of 
confidentiality. 

A distinction between documentation not available due to confidentiality and non-existence is 
provided in the annexes to the final report (Annex 5). 

  Detail and quality of reports 

The  current  eu-LISA  budgeting  system  does  not  allow  a  systematic  comparison  of  costs 
versus  individual  tasks.  This  limited  the  extent  to  which  the  evaluation  could  draw 
conclusions on cost effectiveness. 

5. IMPLEMENTATION STATE OF PLAY (RESULTS) 

5.1. State of play 

The Agency took up its core tasks on 1 December 2012 and gained financial independence in 
May 2013. The seat agreement with the Estonian government was signed in November 2014 
and became applicable on the same date.  

The site agreements with Austria and France were signed in May 2013 and December 2013 
respectively and entered into force on 13 October 2013 and 28 August 2016 respectively. By 
March  2016,  12  Management  Board  meetings  had  been  held.  In  principle,  four  sessions  of 
each Advisory Group (AG) (for SIS II, VIS and Eurodac) are organised per year, resulting in 
15 regular meetings of SIS II AG and 16 meetings of VIS and Eurodac AG by March 2016. 
An extraordinary ad hoc meeting of all three AGs was held in late February 2016 to discuss 
the 2017 draft annual work programme. 

On the basis of the annual work programmes and the annual activity reports of eu-LISA or the 
European Court of Auditors’ discharge reports, the evaluation confirmed that although still a 
work  in  progress,  eu-LISA  has  already  reached  ‘cruising  speed'.  Against  the  background  of 
ever-changing developments in the JHA area and budgetary restrictions, the Agency gradually 
took  up  its  responsibilities  with  increasing  success.  It  also  took  a  proactive  approach  to 
developing  a  constructive  relationship  with  the  stakeholders,  handled  new  tasks  efficiently 
and further explored the potential to fulfil its 2014-2020 strategy.16 

While acknowledging that the Agency made a successful start, the evaluation also identified 
room for improvement under all evaluation criteria. 

15  List  of  ITIL  certified  staff;  List  of  PRINCE  2  certified  staff;  Report  of  the  last  Business  Investments 

Committee (BIC). 

16  eu-LISA Strategy 2014-2020 as set out in Document 2014-029 adopted by the eu-LISA Management Board 

of 11-12 March 2014, http://europa.eu, ISBN 978-92-95203-04-4. 

10 

 
 
 
                                                            
5.2. Cooperation with the main stakeholders 

eu-LISA was clearly instrumental in meeting the needs of the Member States by guaranteeing 
operational management at central level. Judging by the outcome of the evaluation, the level 
of satisfaction with eu-LISA’s performance among Member States can be considered positive, 
with implementation of the Eurodac recast deemed exemplary. However, the evaluation also 
established  that  the  Agency  could  make  better  use  of  the  expert  potential  of  the  Advisory 
Groups, in particular through adequate consultation both in terms of content and timing on the 
programming  documents.  With  cooperation  progressively  developing,  certain  legal  and 
resource  constraints  to  strengthening  the  practical  cooperation  between  the  Agency  and  the 
Member States were identified. 

The Commission plays a dual role in the governance of the Agency, both as a Member of the 
Management  Board  and  the  Advisory  Groups  and  as  an  EU  institution.  As  a  result,  the 
relationship between the Commission and the Agency is relatively complex. The rules on the 
cooperation  mechanisms  between  the  Commission  and  eu-LISA  were  developed  in  the 
Memorandum  of  Understanding  required  by  the  establishing  Regulation  and  signed  on 
18 June 2014.17  

To  ensure  coordinated  action,  a  video  conference  takes  place  every  2  weeks  between  the 
Agency  and  the  partner  Directorate-General  (DG  HOME).  Whilst  cooperation  is  generally 
satisfactory and mutually beneficial, some issues for concern were identified, in particular on 
reporting and research monitoring. While reporting done by the Agency on implementation of 
its annual work programme at the start of its operations was deemed insufficient in terms of 
transparency, coherence and frequency, there was a gradual improvement in 2014 and 2015. 
However, there is still room for improvement. Both the preparation and content of eu-LISA 
programming documents could be improved and strictly aligned with its mandate in line with 
the Commission’s opinions. They should also be clearer. The Agency must also ensure that its 
monitoring of research activities is closely coordinated with its main stakeholders. There has 
also  been  the  occasional  inconsistency  between  the  Agency’s  interventions  and  the 
Commission’s  policy  line  (e.g.  intervention  in  the  European  Parliament  on  sensitive  policy 
matters without prior coordination with the Commission). 

Both Europol and Eurojust, which are observers in the Management Board and the Advisory 
Groups as well as end-users of SIS II, were satisfied with the support provided by the Agency. 
However, although Europol legally has access to VIS and Eurodac, it had still not connected 
to these systems by the end of the evaluation, despite eu-LISA’s readiness to assist with the 
necessary technical implementation. 

On overall  inter-agency cooperation, eu-LISA has  so far concluded working  arrangements 
that  cover  common  fields  of  interest  and  seek  possible  synergies  with  CEPOL  (2013), 
FRONTEX  (now  EBCGA)  (2014)  and  EASO  (2014).  During  the  evaluation,  eu-LISA  also 
made  progress  on  negotiations  for  a  similar  working  arrangement  with  Europol  (signed  on 
22 March 2016). In 2015, eu-LISA successfully chaired the JHA Agencies network. 

17  Commission  Decision  of  11 June  2014  on  the  adoption  of  a  Memorandum  of  Understanding  between  the 
European Commission and the European Agency for the operational management of large-scale IT systems in 
the area of freedom, security and justice C(2014)3486 final. 

11 

 
 
 
                                                            
6. FINDINGS — ANSWERS TO THE EVALUATION QUESTIONS 

6.1. Effectiveness 

The aim of this criterion was to assess the extent to which eu-LISA achieved its objectives as 
described in the intervention logic. 

6.1.1 To what extent has eu-LISA been effective in ensuring the operational management, technical 
development  and  security  of  large-scale  IT  systems  in  the  JHA  area  entrusted  to  it,  and  their 
improvement?  

The evaluation found that  the Agency had achieved its objective of ensuring  effectively  the 
operational management of the three IT systems entrusted to it in general. According to the 
findings of the evaluation, the Agency: 

  successfully fulfilled the tasks stemming from the Agency establishing Regulation; 
  met the Service Level Agreements put in place; 
  correctly implemented the evolution of the systems; and 
  met the requirements from security, data protection and industry best practices.  

The Agency also established the appropriate capacities and organisational framework to cope 
with  tasks  related  to  the  evolution  of  systems  under  its  responsibility  (ITILv3,  Prince2, 
ISO 2700x18). 

The evaluation also identified ways to improve, in particular on the implementation of ITILv3 
best  practices.  While  the  Agency  made  progress  on  this  (e.g.  by  establishing  appropriate 
project management tools for projects such as VIS Evolutions, Testa-NG), there is still work 
to  be  done.  For  example,  the  performance  of  implemented  IT  processes  could  be  better 
monitored  and  the  scope  of  performance  indicators,  currently  limited  to  corporate  business 
performance, should be extended to cover the performance and functioning of the systems 

The evaluation highlighted a risk to business continuity linked to the absence of a unique and 
transversal Disaster Recovery Plan (DRP) and Business Continuity Plan (BCP) that covers all 
three systems (e.g. personnel, facilities, resources etc.). This risk was identified by the Agency 
and was being addressed by an ongoing project during the evaluation period. Although there 
is a DRP and BCP for each system, the Agency should ensure that all possible scenarios (e.g. 
necessity  to  transfer  all  three  systems  to  the  backup  site  at  the  same  time)  are  taken  into 
account when defining them at organisational level. 

On  capacity  management,  the  evaluation  confirmed,  by  means  of  the  VIS  evolution  case 
study, the need for the Agency to establish  and formalise a review process that allows for a 
regular  review  with  Member  States  of  the  systems’  capacity  needs  based  on  the  statistics 
generated by the systems and a forward-looking exercise. 

On IT security, the evaluation identified the need to implement a cross-functional architecture 
management function tasked with: 

  validation of infrastructure choices; 
  selection of application technology; and 

18  As part of the IT best practices framework, ITILv3 emphasises the concept that IT is a service that supports 
business  goals.  Prince  2  (PRojects  IN  Controlled  Environments)  is  a  process-based  method  for  effective 
project  management  recognised  and  used  internationally.  ISO  2700x  is  a  series  of  standards/objective 
benchmarks  that  provide  a  comprehensive  set  of  security-related  topics  and  objective  means  for  measuring 
compliance. 

12 

 
 
                                                            
 

implementation at transversal level rather than at system level.  

There is a need for a common strategy and roadmap for improving the integration of secure 
architecture principles into all three systems to reduce security-related risks. 

On  data  protection,  there  is  a  need  to  ensure  that  appropriate  data  protection  clauses  are 
included in the agreements concluded with external contractors. The evaluation also identified 
concerns  about  the  production  of  additional  statistics  and  data  quality/data  analysis  reports. 
The  evaluation  therefore  highlighted  the  need  to  provide  the  possibility  for  eu-LISA  to 
produce additional statistics, data quality/data analysis reports in a  future amendment of the 
current legal framework. It also identified the need to reinforce the data protection capacity in 
Strasbourg  by  either  transferring  the  Data  Protection  Officer  (DPO)  from  Tallinn  to  the 
technical site or appointing a deputy DPO there. 

On  eu-LISA’s  obligations  for  the  production  of  documents  and  reports,  the  Agency  should 
increase the quality control carried out on the documents/reports delivered. 

6.1.2a. To what extent has eu-LISA been effective in monitoring research? 

The task given to eu-LISA of monitoring of research is deemed appropriate and necessary for 
the  effective  and  efficient  functioning  of  the  Agency  and  development  of  the  systems. 
However, no clear examples of its impact could be identified due to the low level of maturity 
of  this  activity  during  the  Agency’s  first  years  of  operation.  The  Agency  identified  the 
importance  of  monitoring  research  and  acknowledged,  in  its  2015  work  programme,  that 
greater  effort  should  be  made  to  feed  the  results  of  the  activity  into  the  operational 
management of the systems. 

The  evaluation  also  found  that  efforts  need  to  be  made  to  ensure  that  synergies  are  created 
with other entities in this area, including the Commission. This would apply in particular to 
coordination  with  the  Commission’s  research  programmes.  An  example  of  wasteful 
duplication  of  effort  is  in  the  research  monitoring  activities  on  biometric  identifiers.  The 
Agency issued a report on this in 2015 due to its links with the systems it manages, although 
specific  research  activities  have  already  been  developed  in  the  area  of  biometrics  under 
Horizon 2020 and other research programmes. 

6.1.2b. To what extent has eu-LISA been effective in providing training? 

Since it was established, the Agency has made laudable efforts to develop appropriate training 
activities, both technical and practical, in line with the needs of national authorities. National 
contact  points  (NCP),  which  consist  of  Member  State  representatives,  prepare  the  training 
strategy each year. 

The  evaluation  found  that  training  needs  to  be  aligned  further  with  the  technical  needs. 
However,  this  could  in  part  be  due  to  the  passive  participation  of  the  relevant  stakeholders 
(e.g.  the  Advisory  Group  and  the  NCPs)  in  developing  the  training  strategy,  despite  the 
Agency’s efforts to keep them involved. 

6.1.2c.  To  what  extent  has  eu-LISA  been  effective  in  its  tasks  particular  to  Eurodac  relating  to 
transmission, collection and comparison of data, access to and correction or erasure of data? 

Since  eu-LISA  took  over  Eurodac,  it  has  been  effective  in  its  tasks,  specific  to  the  system, 
with processes and procedures put in place to control the effective transmission of data as well 
as the comparability of fingerprints sent by the Member States. 

13 

 
 
6.1.3 To what extent has eu-LISA been capable to respond to the new tasks entrusted to it? 

The Agency has demonstrated its capabilities in responding to new tasks entrusted to it. On 
DubliNet  and  VISION,  the  Agency  successfully  ensured  the  effective  integration  and 
management of these operational tasks. While the Agency was successful, no formal reporting 
currently exists in terms of the impact on the Agency’s resources for assuming these tasks.  

On the Smart Borders pilot project, the Agency was effective in implementing the technical 
aspects and assisting the Member States in performing tests. The implementation of the pilot 
also  showed  that  the  Agency  staff  in  charge  should  have  been  trained  in  financial 
management in relation to EU grants management. Another lesson learned from the pilot is to 
ensure that the Agency develops sufficient project management and development capacity so 
that the different systems always receive the attention they need and new projects do not rely 
on ad hoc reshuffling of staff and priorities. 

6.1.4  To  what  extent  has  eu-LISA  been  successful  in  establishing  partnerships  and  looking  for 
synergies with other EU agencies and in building strong and trusting relations with its stakeholders 
when operating and maintaining the systems under its mandate? 

eu-LISA has developed adequate strategies and working practices to ensure a sufficient level 
of  communication  with  key  stakeholder  groups.  This  enables  the  Agency  in  particular  to 
ensure that the  allocation of roles  and responsibilities at EU and national level is respected. 
The  Member  States’  level  of  satisfaction  appears  high,  and  the  relationship  with  the 
Commission  is  also  deemed  adequate.  The  organisational  structure  of  eu-LISA  facilitates 
communication  and  discussion  through  the  Management  Board  and  Advisory  Groups, 
although the potential of these bodies, in particular the latter, could be better exploited in the 
future. 

Since  it  was  established,  the  Agency  has  developed  cooperative  and  effective  relationships 
with  other  JHA  agencies.  By  the  end  of  the  evaluation  period,  it  had  signed  three  working 
arrangements that formalise cooperation with other EU agencies (CEPOL, FRONTEX (now 
EBCGA),  EASO).  This  cooperation  was  considered  a  success  in  the  evaluation,  with  the 
synergies helping to fulfil eu-LISA’s tasks (i.e. cooperation with FRONTEX (now EBCGA) 
on the Smart Borders pilot project). However, the Agency establishing Regulation provides a 
limited  mandate  for  such  cooperation.  As  a  result,  when  developing  cooperation  with  other 
JHA  agencies  within  the  scope  of  its  mandate,  eu-LISA  should  take  all  necessary  steps  to 
ensure that it gives priority to its core business which is the operational management of the 
systems. 

6.2. Efficiency 

The  aim  of  this  criterion  was  to  assess  the  costs  (financial,  staff,  time,  expertise)  directly 
incurred  by  the  Agency  to  achieve  results,  and  also  the  way  resources  are  allocated  and 
managed. 

6.2.1 To what extent has eu-LISA been efficient in implementing the tasks set out in its mandate as 
laid down in the Agency establishing Regulation? 

Since  it  was  established,  the  Agency  has  undertaken  significant  work  to  align  its  functions, 
operations  and  internal  processes  with  the  management  of  an  IT  framework.  eu-LISA 
introduced  changes  to  the  operational  department  in  order  to  identify  and  introduce 
operational synergies. 

The evaluation found that there is a need for a review of how staff resources are allocated in 
certain areas which are not considered as the core task of the agency (e.g. cooperation with 
agencies and other EU bodies, communication with industry) in order to ascertain whether it 

14 

 
 
is  in  proportion  to  the  objectives  and  whether  some  resources  should  be  reallocated.  The 
evaluation  also  found  (e.g.  case  study  on  Eurodac  Recast)  that  there  was  a  need  to  put 
appropriate controls in place to ensure that sufficient staff resources are available for project 
management based on project needs. 

6.2.1a To what extent did external factors influence the efficiency of the Agency? 

While the evaluation found that the Agency faced issues due to external factors (recruitment 
in Tallinn, lack of synergies in system procurement due to the Agency being tied to existing 
contracts),  it  is  expected  that  the  second  problem  will  be  resolved  over  time,  with 
procurement  and  contracting  arrangements  eventually  falling  into  line  with  the  Agency’s 
approach (i.e. concluding contracts that cover all three systems). 

6.2.1b  To  what  extent  are  the  internal  and  external  mechanisms  for  monitoring,  reporting  and 
evaluating  eu-LISA  adequate  for  ensuring  accountability  and  an  appropriate  assessment  of  the 
overall performance of eu-LISA? 

The  Agency  has  gradually  improved  reporting  on  the  implementation  of  its  annual  work 
programmes. However, the evaluation identified a need for the Agency to further improve the 
quality of its reporting in order to ensure transparency of the Agency’s overall performance, 
with issues identified on the accountability and transparency of the Agency’s activities in its 
reports (e.g. difficulties in identifying links between the work planned in the work programme 
and that actually undertaken in the annual activity report). 

6.2.2 Were the annual budgets of the Agency implemented in  an efficient way and  with a  view to 
achieving results? 

With regard to budget execution in 2013 and 2014, the evaluation found a low execution level 
of  annual  commitment  appropriations  in  2014.  Considerable  progress  was  made  in  2015 
compared  to  2014  in  achieving  the  target  level  of  execution  in  general,  i.e.  on  both  the 
commitments  and  the  payments.  On  payment  execution,  a  gradual  improvement  was  noted 
over  the  whole  evaluation  period.  Nevertheless,  the  Agency  should  continue  to  develop  its 
planning capacities, allowing more detailed work programmes to be developed in particular in 
terms of their multi-annual dimension (activities and related financial estimates) and a close 
follow-up of the implementation process. 

6.2.3a  To  what  extent  are  eu-LISA’s  organisational  solutions,  HR  and  procedures  adequate  for 
carrying out the work entrusted to it and the actual workload? 

EU-LISA should  make use  of the  flexibility that exists  in  the Staff Regulations of Officials 
and other Servants of the Union on recruitment grades, judging on a case-by-case basis when 
the recruitment of highly skilled and sought-after staff justifies the use of this flexibility. The 
evaluation  underlined  that  anticipating  changes  in  the  Agency’s  workload  and  new  tasks  is 
crucial to allow enough time to adapt existing resources and deploy additional means where 
necessary. 

However, it has been underlined by the external evaluation and on the basis of the execution 
of the case study on the Smart Borders pilot in particular that if eu-LISA were to be entrusted 
with the development of any new large-scale IT system or additional tasks, the current level 
of resources and the required staff profiles would need to be adapted accordingly as eu-LISA 
was initially staffed to operate three systems, not to develop new ones. 

6.2.3b Is the planning cycle of the Agency (annual work programme and budget) in line with the 
objective of achieving efficient results? 

The evaluation found that the process of preparing the annual work programme and budget is 
adequate  as  the  Agency  introduced  adequate  and  appropriate  accounting  practices  and 

15 

 
 
systems.  However,  there  is  a  lack  of  foresight  due  to  the  current  lack  of  a  detailed  multi-
annual  work  programme  which,  in  accordance  with  Article  32(1)  of  the  Commission 
delegated Regulation (EU) No 1271/201319 and of the Agency’s Financial Regulation, should 
be incorporated in the single programming document. Some improvements should be made, 
such as: 

involving the stakeholders earlier in the process; and 

 
  strengthening alignment with the budget and the multi-annual work programme.  

To  this  end,  eu-LISA  should  swiftly  adopt  a  detailed  multi-annual  work  programme,  as 
envisaged  in  the  establishing  Regulation,  to  ensure  effective  planning  by  the  Agency.  In 
addition,  the  Agency  should  carry  out  systematic  ex-ante  and  ex-post  evaluations  of 
programmes and activities that entail significant spending as required by Article 29.5 of the 
Commission  delegated  Regulation  (EU)  No 1271/2013  and  of  the  Agency’s  Financial 
Regulation. 

6.2.3c To what extent has eu-LISA succeeded in building up the in-house capacities for handling 
various tasks entrusted to it? 

While the organisational set-up and internal processes are still evolving, the evaluation found 
that  these  are  properly  structured  and  described.  The  Agency  has  been  able  to  identify 
solutions to carry out the assigned tasks. This was observed in particular in the project case 
studies  (e.g.  Smart  Borders  pilot  project,  Common  Shared  Services  Studies).  On  new  tasks, 
the Agency has dealt with these efficiently reallocating staff internally, prioritising tasks and 
limiting or reducing the effort on implementation and subcontracting.   

However, it may be difficult to deal with such challenges in the future if new responsibilities 
that cannot be handled by existing internal resources are allocated. 

On  subcontracting,  the  evaluation  found  that  there  is  a  need  for  a  formal  and  transverse 
sourcing  strategy  as  well  as  for  internal  staff  to  closely  manage  the  work  carried  out  by 
external contractors.   

6.2.4  Do  eu-LISA’s  processes  and  procedures  successfully  contribute  to  the  efficiency  of  its 
operations? 

The  Agency  has  made  laudable  efforts  to  set  up  the  appropriate  policies,  processes  and 
procedures  that  allow  it  to  govern  structure  and  organise  its  operations  and  deliver  the 
required level of service. This was seen, for example, in the implementation of the project on 
the Common Shared Services Study. 

The evaluation found a need for the implementation of a quality management system aimed at 
updating and reviewing all processes, policies and procedures on a regular basis. 

In  the  case  studies  undertaken  for  the  evaluation,  a  need  was  also  identified  to  ensure 
availability of complete and sufficiently transparent documentation that deals with all aspects 
of project management (VIS Evolutions). 

19  Commission  delegated  Regulation  (EU)  No 1271/2013  of  30 September  2013  on  the  framework  financial 
regulation for the bodies referred to in Article 208 of Regulation (EU, Euratom) No 966/2012 of the European 
Parliament and of the Council. 

16 

 
 
                                                            
6.2.5 To what extent has eu-LISA’s governance, organisational structure and locations, as created 
by the Agency establishing Regulation, been conducive to its efficiency and to achieving economies 
of scale? 

No  major  changes  are  needed  to  the  current  governance  system.  There  is  room  for 
improvement in the active participation of Advisory Group members in providing expertise to 
the Agency on the operational management of the three IT systems. The Management Board 
functions efficiently, with no major changes required to current practice. Nevertheless, current 
use of the written procedure may in some cases fail to ensure that issues are given sufficient 
attention. In addition, the evaluation found (i.e. case study on Common Shared Service Study) 
that the Agency could potentially improve its level of communication on progress related to 
projects. 

On multi-site arrangements, the evaluation found that the geographical dispersal of eu-LISA 
has  an  impact  on  the  efficient  implementation  of  eu-LISA’s  tasks  and  responsibilities.  It 
generates  costs  that  would  not  have  been  incurred  had  the  Agency  been  established  on  just 
two sites (technical and backup), necessary due to security reasons. While direct and indirect 
costs  exist,  these  are  considered  justified  and  reasonable.  Since  the  political  reasoning 
associated  with  the  establishment  of  the  seat  in  Tallinn  also  continues  to  be  relevant,  the 
evaluation concluded that the justification for the current multi-site arrangement is still valid. 
For more details, please refer to point 4.9.3 of the final report. 

6.3. Coherence 

The aim of this criterion was to examine the extent to which eu-LISA cooperation activities 
are  aligned  in  an  effective  way  with  those  of  stakeholders,  including  Member  States  and 
Associated Countries, the European Commission, the other EU institutions and EU bodies, to 
ensure  complementarity  and  avoid  duplication  of  effort.  The  evaluation  also  assessed  the 
alignment  of  strategies,  the  coherence  of  cooperation  activities,  the  risk  of  overlaps,  the 
procedures  in  place  to  ensure  that  eu-LISA’s  cooperation  activities  are  coherent  with  the 
policies  and  activities  of  its  stakeholders  and  stakeholders’  perception  of  the  quality  of  the 
latter. 

6.3.1  To  what  extent  is  eu-LISA  acting  in  cooperation  with  the  European  Commission  and  other 
EU bodies to ensure complementarity and avoid duplication of efforts? 

The  Agency  establishing  Regulation  and  the  Memorandum  of  Understanding  between  the 
Agency  and  the  Commission  provide  a  solid  framework  for  effective  cooperation,  which  is 
considered  to  be  largely  satisfactory.  The  Commission  provided  adequate  support  to  the 
Agency  during  its  establishment  until  it  took  over  its  tasks  on  1 December  2012  and  in  the 
transition period, until the Agency reached budgetary independence on 22 May 2013. 

The cooperation established with other EU institutions is considered to be beneficial to ensure 
complementarity in the implementation of EU policy and strategy, provided the cooperation 
remains  within  the  mandate.  The  relationships  developed  with  the  private  sector  by  way  of 
industry-related  events  have  created  a  useful  platform  for  discussion  on  future  needs  and 
opportunities in relation to IT development. 

The following improvements were identified: 

  Coherence  of  the  management  of  communication  infrastructure  tasks,  which  are 
divided between the Agency and the Commission, could be improved by transferring 
the  Commission’s  tasks  (in  particular  implementation  of  the  budget,  acquisition  and 
renewal, and contractual matters) to the Agency. The Agency has the competence and 
capacities to cope with these tasks, which could also lead to greater efficiency; 

17 

 
 
 

In  addition,  a  risk  of  duplication  was  identified  in  the  Agency’s  activities  on 
monitoring of research with regard to the operational management of the systems. An 
example was the Agency's report on biometric identifiers in 2015 while other studies 
on biometrics had already been carried out by the Commission under the framework of 
Horizon  2020  and  other  research  programmes.  These  should  be  aligned  with  the 
Commission’s  research  work  in  this  area  and  coordinated  with  the  Commission  in 
accordance with Article 15 of the Memorandum of Understanding; and 

  The  evaluation  considers  that  the  Agency  could  continue  to  develop  cooperation 
activities  (with  other  EU  agencies)  provided  that  they  comply  with  the  Agency’s 
mandate  and  the  core  activities  are  not  impacted,  for  example  in  terms  of  quality  of 
service and promptness in responding or reporting. 

6.3.2  To  what  extent  is  eu-LISA  acting  in  cooperation  with  the  Member  States  and  Associated 
Countries to ensure complementarity and avoid duplication of efforts? 

The  evaluation  found  that  the  Agency  has  been  largely  successful  in  respecting  the  clear 
allocation of roles and responsibilities at EU and national level concerning its core business of 
managing large-scale IT systems. This stems from the clear division of responsibilities in the 
legal framework of the Agency, as provided in the Agency establishing Regulation and more 
specifically  in  the  systems’  legislative  instruments.  While  no  duplication  of  effort  was 
identified,  further  complementarity  could  be  possible  in  relation  to  increased  interaction 
among  Advisory  Group  members  as  well  as  increased  involvement  of  the  Agency  in  the 
provision and analysis of statistics and reports, including those on the data quality statistics of 
Member States. 

6.3.3  To  what  extent  are  eu-LISA  activities  coherent  with  the  strategy  documents  adopted  in  the 
policy field? 

The  evaluation  found  that  the  activities  are  well  aligned  in  general  with  the  relevant  JHA 
policy  frameworks  and  strategies.  The  coherence  of  the  Agency’s  activities  was  confirmed 
through  eu-LISA  strategy  2014-2020.  Due  to  the  low  level  of  maturity  of  this  strategy,  the 
evaluation cannot gauge its impact. However, the alignment with JHA policies is expected to 
continue as the strategy develops over the coming years. This should also be demonstrated by 
the alignment of the eu-LISA strategy with the first multi-annual work programme, which has 
not yet been adopted. 

6.3.4. Are the procedures put in place effective to ensure that eu-LISA’s cooperation activities are 
coherent with the policies and activities of its stakeholders? 

The  evaluation  found  that  the  Agency  has  developed  a  coherent  cooperation  strategy  for 
engaging  with  different  stakeholders  in  a  structured  way  by  adopting  its  stakeholder 
management strategy. However, no specific mechanisms are in place at Agency level for eu-
LISA to ensure coherence between its cooperation strategy and its mandate. 

The Agency has  undertaken cooperation activities  within the JHA area with  EU institutions 
and other JHA agencies. However, greater efforts need to be made by the Agency to ensure 
coherence between these activities and its mandate and better coordination with the Member 
States and with the Commission on this topic. 

6.4. Relevance and added value 

The  aim  of  this  criterion  was  to  examine  the  extent  to  which  the  creation  of  the  Agency 
brought benefits in terms of efficiency and effectiveness compared with previous operational 
arrangements. 

18 

 
 
6.4.1.  What  have  been  the  benefits  of  acting  at  Agency  level  from  an  operational  and  strategic 
perspective? 

The  establishment  of  a  single  management  authority  to  assume  operational  management  of 
the  three  IT  systems  inherently  created  a  high  level  of  added  value,  to  the  extent  that  the 
Agency carries out its tasks in an effective and efficient manner. The ultimate added value of 
eu-LISA is therefore highly dependent on its ability to fulfil its core tasks in an efficient and 
effective manner. 

The  key  elements  to  eu-LISA’s  added  value  are  the  pooling  of  expertise,  harnessing  of 
synergies by bringing systems ‘under one roof’ and the creation of a more flexible operational 
framework. 

6.4.2  To  what  extent  has  eu-LISA  been  more  effective  in  achieving  its  results  compared  to  other 
past, existing or alternative national or EU level arrangements? 

By making a comparison of the costs linked to Eurodac (i.e. the only system for which such a 
comparison could be carried out) before and after the Agency was created, the evaluation did 
not identify economies deriving from its creation, with costs appearing to be higher. While the 
difference in costs can be explained, in some cases, by the tasks associated with the systems 
(i.e.  training,  Advisory  Group  meetings),  an  overall  comparative  assessment  of  the  costs 
cannot be made due to a difference in how costs are recorded (e.g. for central unit and backup 
central unit, quality assurance, hardware and software) before and after Eurodac was handed 
over by the Commission to eu-LISA. The evaluation identified a lack of an internal recording 
process to measure all costs associated with each system. 

The comparison of operational costs identified the need for clear recording of costs for each 
system  (activity-based  management)  in  the  future  in  order  to  be  able  to  ascertain  whether 
efficiency gains  have been  achieved. The Agency should therefore ensure in the accounting 
system  that  all  costs  associated  with  each  system  are  recorded  (including  staffing  costs 
maintenance costs, evolution costs) and can be identified for each system. 

6.4.3  To  what  extent  has  eu-LISA  strengthened  the  involvement  of  Member  States  in  the 
operational management and evolution of the systems entrusted under its mandate? 

The  evaluation  noted  positive  feedback  in  general  on  the  level  of  involvement  and 
communication with national level stakeholders. The Agency’s governance contributes to the 
quality of communication and involvement of national authorities, ensuring that a platform is 
provided  to  Member  States  to  be  involved  in  the  Agency  and  to  ensure  that  their  needs  are 
taken into account. 

While the Advisory Groups are a relevant and valuable forum for Member State involvement, 
the evaluation found that the wealth of expertise within these groups could be better exploited 
in order to improve the quality of dialogue with national authorities. This could take place by 
establishing guidelines for the Advisory Groups and setting out annual key objectives for the 
Advisory Groups. 

7. WAYS TO IMPROVE 

7.1. What are the specific needs and opportunities to ensure increased practical 
cooperation with Member States and EU bodies? 

To  be  able  to  identify  the  needs  in  the  Member  States  on  the  development,  operational 
management  and  evolution  of  the  respective  systems,  there  is  a  need  to  ensure  closer 
cooperation and dialogue in the Management Board and the Advisory Groups, for instance on 
the monitoring of research or training. Another particular area of support that eu-LISA could 
offer  the  Member  States  could  be  an  enhanced  role  for  the  Agency  in  the  provision  of 

19 

 
 
additional  statistics  and  data  analysis/data  quality  reports.  The  Smart  Borders  pilot  project 
also demonstrated the added value of closer practical cooperation between the Agency and the 
Member States. On data protection, the evaluation also identified the need  to develop better 
communication  with  the  European  Data  Protection  Supervisor.  On  the  JHA  agencies,  the 
practical cooperation has proved beneficial so far, and further practical cooperation could be 
sought, whilst fully respecting the mandate, in order to leverage potential synergies and avoid 
the risk of overlap and duplication. 

7.2. To what extent could eu-LISA’s current mandate, tasks and/or capabilities 
address these needs? 

Some of the tasks mentioned above, such as monitoring of research and training, are already 
covered  by  the  establishing  Regulation,  and  increased  cooperation  could  therefore  be 
undertaken in accordance with  the Agency’s mandate.  However,  for other  activities  such  as 
those  related  to  statistics  or  cooperation  with  other  Agencies,  the  mandate  is  not  explicit  or 
does not exist at all. Similarly, on increased practical cooperation between the Agency and the 
Member States in relation to ad hoc support for the implementation of systems, there are clear 
legal and financial limitations to what eu-LISA can deliver. 

In addition, the Smart Borders pilot experience pointed out the limitations that currently exist 
in the establishing Regulation with regard to pilot schemes.20 

7.3. How could eu-LISA’s mission, tasks, working practices or activities be 
further developed in order to meet these needs? 

Some of the needs identified, in particular those related to monitoring of research, training or 
the Advisory Groups’ added value, can be addressed to some extent by developing the current 
working  practices  within  the  Agency  and  enhanced  dialogue  with  the  Commission  and  the 
Member States. With regard to taking on new projects, the working practices of the Agency 
could  be  improved  in  order  to  increase  internal  expertise  and  experience  on  grant 
management. Other needs, such as cooperation with other Agencies, the provision of further 
statistics,  data  analysis  and  data  quality  reports  and  the  provision  of  ad-hoc  support  to 
Member States would require a clarification and/or extension of the mandate. 

7.4. How could eu-LISA increase its added value and its contribution to the EU 
and the Member States in the future using the capabilities and competencies 
already in place? 

By  assuming  a  central  role  in  the  operational  management  of  large-scale  IT  systems  in  the 
area of freedom, security and justice, the Agency can provide added value due to synergies it 
can offer, ensuring that all systems are working as effectively and efficiently as possible. As 
the  needs  of  the  EU  and  Member  States  develop  due  to  ever-changing migratory  flows  and 
internal  security  challenges,  the  Agency  should  ensure  that  the  existing  and  future  systems 
take into account developments in technology, in particular with regard to biometrics.  

To ensure that the systems are working as effectively and efficiently as possible, the Agency 
could  increase  the  interoperability  of  the  systems  while  respecting  the  legal  framework 
applicable  to  each  system.  eu-LISA  can  also  provide  additional  added  value  to  the  EU 
institutions  by  identifying  technical  solutions  for  legislative  proposals  on  existing  or  new 
systems. It can also provide added value to the Member States by advising on matters linked 
to fine-tuning their national systems.  

20 Article 9 of the Regulation. 

20 

 
 
                                                            
This  could  be  done  by  means  of  increased  communication  with  Member  States  through  the 
Management Board and the Advisory Groups or through ad hoc support. 

8. OTHER IMPACTS 

As was the case with the impact assessment accompanying the proposal for the establishing 
Regulation, the evaluation questions did not look into the social or environmental impact as it 
was not considered relevant to the purpose or the mandate of the Agency. The assumption that 
the  establishment  of  the  Agency  has  no  significant  social  or  environmental  impact  remains 
valid for the period covered by the eu-LISA evaluation. 

Although the creation of the Agency generated some new costs and the cost effectiveness of 
some  of  the  Agency’s  actions  have  not  yet  been  fully  proven,  this  is  considered  to  be 
outweighed  by  the  Agency’s  added  value.  This  EU  intervention  therefore  also  has  no 
significant economic or financial impact. 

9. CONCLUSIONS 

9.1. Main findings 

In general, the evaluation provided reassurance that the Agency contributes to the operational 
management  of  large-scale  IT  systems  in  the  area  of  freedom,  security  and  justice  and 
effectively fulfils the tasks laid down in the Regulation as well as new tasks entrusted to it. 

It  also  found  that  by  bringing  the  management  of  the  three  systems  together,  the  envisaged 
pooling  of  resources  and  more  consistent  approach  were  effectively  ensured.  This  therefore 
allowed  the  Agency  to  contribute  to  the  establishment  of  a  more  coordinated,  effective  and 
coherent  IT  environment  for  the  management  of  large-scale  IT  systems  that  support  the 
implementation of JHA policies. 

However,  the  evaluation  also  identified  room  for  improvement  at  both  the  corporate  and 
business level. 

9.2. Recommendations 

The recommendations corresponding to the findings are numerous and mixed in nature. Most 
have  an  organisational  and  financial  impact,  although  some  also  have  a  legal  impact. 
Altogether, the evaluation produced 66 recommendations. 

In  terms  of  significance,  seven  recommendations  are  considered  critical  in  the  area  of 
business  continuity  and  security  (effectiveness),  financial  management  and  procedures 
(efficiency) and cost management (common to coherence, relevance, added value): 

1)  To prepare for the possible need to transfer all systems to the backup site at the same 
time, the Management Board should formally define and agree the sequence of switch-
over/switch-back between the three systems. The organisation that is currently being 
implemented  for  DRP  and  BCP  should  be  aligned  with  the  operational  needs  of  the 
Member States and leading industry practices, especially in terms of scope, roles and 
responsibilities,  testing  (periodicity,  procedures  and  scenarios)  and  emergency 
procedures. 

2)  Since  the  continuity  and  disaster  recovery  of  the  three  systems  are  tested  separately 
and  not  at  the  same  time,  it  is  also  recommended  that  the  Agency  defines,  together 
with  the  Advisory  Groups,  a  realistic  and  feasible  Recovery  Point  Objective  (RPO) 
and Recovery Time Objective (RTO) for each large-scale IT system. 

3)  To  strengthen  its  security  management  competencies,  the  Agency  should  review  the 
segregation  of  security  duties  between  security  staff  members.  Implementation, 

21 

 
 
enforcement and testing should be conducted by specific staff members to guarantee 
an independent approach and efficient management of all potential security risks. 
4)  The Agency should define and implement an activity-based management approach to 
improve eu-LISA’s planning capabilities and reinforce transparency and accountability, 
as well as facilitate greater prioritisation and trade-offs by providing accurate data on 
costs. Technical staff should be involved in developing this approach. 

5)  The  written  procedure  should  not  be  used  to  enable  decisions  on  points  with  a 

significant financial impact without prior discussion in the Management Board. 

6)  To  improve  efficiency  in  the  Agency’s  activities,  the  communication  infrastructure 
tasks under the Commission’s responsibility should be transferred to the Agency. 
7)  To be able to measure the efficiency gains of the Agency, eu-LISA should ensure that 
all  costs  associated  with  each  system  are  recorded  and  can  be  identified  per  system. 
This recommendation is also associated with another non-critical yet ‘very important’ 
recommendation (the Agency should consider putting in place appropriate controls to 
ensure  that  sufficient  staff  resources  are  available  for  project  management  based  on 
project needs and that the use of in-house capacities are prioritised over outsourcing in 
the Agency’s sourcing strategy). 

The  vast  majority  of  the  recommendations  can  be  followed  up  at  management  and 
organisational level by way of modifications to the rules of procedure etc.  
Pursuant to Article 12(o) of the Regulation,21 the March 2016 Management Board asked the 
Executive Director of the Agency ‘to prepare and provide for discussion an assessment for an 
action  plan  at  the  next  Management  Board  meeting  [in  November  2016]  and  further  to 
provide  for  discussion  and  adoption  not  later  than  end  of  March  2017  of  the  action  plan  to 
address the recommendations not requiring a legislative amendment.’ 

At the same time, the evaluation identified the possible need for revision or extension of the 
tasks  entrusted  to  eu-LISA  in  the  establishing  Regulation  and/or  the  systems’  legislative 
instruments. 

The  recommendations  for  legislative  amendments  to  the  Agency  Regulation  made  in  the 
external evaluation can be summarised as follows: 

  The Commission’s responsibilities relating to the communication infrastructure should 
be  transferred  to  eu-LISA.  This  amendment  will  also  require  amendments  to  the 
systems’  instruments,  the  SIS  II  Security  Decision  and  the  Memorandum  of 
Understanding between the Commission and the Agency. It will entail a transfer of the 
related budget. 

  A  new  provision  on  the  cooperation  framework  between  eu-LISA  and  other  JHA 

agencies should clarify the scope of cooperation within the eu-LISA mandate. 

  A risk assessment and ex-ante assessment should be prepared for projects of more than 
EUR  500 000  that  are  carried  out  by  eu-LISA  within  its  current  mandate  (i.e.  not 
derived from a legislative instrument that entrusts it with a new system, for which an 
impact assessment will be provided by the Commission).22 

21  ‘In  order  to  ensure  that  the  Agency  carry  out  its  tasks,  the  Management  Board  shall:  ….ensure  adequate 
follow-up to the findings and recommendations stemming from the various internal or external audit reports 
and evaluations;'. 

22  As  explained  in  the  report  from  the  Commission  to  the  European  Parliament  and  the  Council  on  the 
evaluation of the European Agency for the operational management of large-scale IT systems in the area of 
freedom,  security  and  justice  (eu-LISA),  this  recommendation  does  not  require  a  change  of  the  Agency 
Regulation as Article 29.5 of the Commission delegated Regulation (EU) No 1271/2013 and of the Agency’s 

22 

 
 
                                                            
 
An  interim  report  should  be  adopted  by  the  Management  Board  by  the  end  of  August  each 
year  on  progress  made  on  the  implementation  of  planned  activities  covering  the  first  six 
months of that same year. The scope of pilot projects that can be entrusted to eu-LISA by the 
Commission  (Article  9)  should  be  extended.  The  scope  is  currently  limited to  pilot  projects 
referred to in Article 54(2)a) of the Financial Regulation, i.e. those that may be implemented 
without a basic act. This should be extended at least to pilot projects with an existing basic 
act.  The  external  evaluation  also  made  other  recommendations  for  amendments  to  the 
Agency’s  mandate.  These  should  be  inserted  into  the  systems’  legislative  instruments  and 
would, in principle, not require an amendment to the Agency Regulation: 

Increased responsibility for eu-LISA to generate/publish the statistics for each system. 

 
  A new task for eu-LISA to produce data quality reports and data analysis reports. 

Financial Regulation already require prior and subsequent evaluations of programmes and activities that entail 
significant spending. 

23