COMMISSION STAFF WORKING DOCUMENT EVALUATION of the European Union Agency for the Operational Management of Large-Scale IT Systems in the Area of Freedom, Security and Justice (eu-LISA) in accordance with Article 39 of Regulation (EU) 2018/1726 Accompanying the document Report from the Commission to the European Parliament and the Council on the evaluation of the European Union Agency for the Operational Management of Large-Scale IT Systems in the Area of Freedom, Security and Justice (eu-LISA) in accordance with Article 39 of Regulation (EU) 2018/1726
Date: 2026-01-26
Institutions: Directorate-General for Migration and Home Affairs, European Commission
Subjects: European Agency for the Operational Management of Large-Scale IT Systems in the Area of Freedom, Security and Justice, European security, Schengen Information System, area of freedom, security and justice, exchange of information, external border of the EU, illegal migration, operation of the Institutions, personal data, rights of aliens

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 26.1.2026  SWD(2026) 980 final  COMMISSION STAFF WORKING DOCUMENT  EVALUATION  of the European Union Agency for the Operational Management of Large-Scale IT Systems in the Area of Freedom, Security and Justice (eu-LISA) in accordance with Article 39 of Regulation (EU) 2018/1726 Accompanying the document Report from the Commission to the European Parliament and the Council on the evaluation of the European Union Agency for the Operational Management of Large-Scale IT Systems in the Area of Freedom, Security and Justice (eu-LISA) in accordance with Article 39 of Regulation (EU) 2018/1726 {COM(2026) 32 final}  
 
 
Table of Contents 

1. 

INTRODUCTION .............................................................................................................................. 1 

1.1. 

Purpose and scope of the evaluation ........................................................................... 1 

2.  WHAT WAS THE EXPECTED OUTCOME OF THE INTERVENTION? ..................................... 3 

2.1 

2.2 

Description of the intervention and its objectives ....................................................... 3 

Points of comparison ................................................................................................... 7 

3.  HOW HAS THE SITUATION EVOLVED OVER THE EVALUATION PERIOD? ....................... 9 
EVALUATION FINDINGS ............................................................................................................. 12 
4. 

4.1. 

To what extent was the intervention successful and why? ........................................ 12 

4.1.1. 

4.1.2. 

4.1.3. 

4.2. 

4.3. 

Effectiveness ...................................................................................................... 12 

Efficiency ........................................................................................................... 29 

Coherence .......................................................................................................... 49 

How did the EU intervention make a difference and to whom? ............................... 53 

Is the intervention still relevant? ............................................................................... 54 

5.  WHAT ARE THE CONCLUSIONS AND LESSONS LEARNED? ............................................... 60 

5.1. 

5.2. 

Conclusions ............................................................................................................... 60 

Lessons learned ......................................................................................................... 62 

ANNEX I:   PROCEDURAL INFORMATION ......................................................................................... 65 
ANNEX II. METHODOLOGY AND ANALYTICAL MODELS USED ................................................. 67 
ANNEX III. EVALUATION MATRIX ..................................................................................................... 75 
ANNEX IV. OVERVIEW OF BENEFITS AND COSTS ....................................................................... 106 
ANNEX V. STAKEHOLDERS CONSULTATION - SYNOPSIS REPORT ......................................... 108 

 
 
 
 
 
 
 
 
Term or acronym 

Meaning or definition 

Glossary 

AFSJ  

AG  

AI   

API 

CD   

CEAS  

CI  

CIR  

Area of Freedom, Security, and Justice 

Advisory Groups 

Artificial Intelligence 

Advanced Passenger Information 

Continuous Delivery 

Common European Asylum System 

Continuous Integration 

Common Identity Repository 

COVID-19  

Coronavirus Disease 2019 

CRRS  

CSI  

CSIRT  

DPA  

EBCGA 

ECRIS RI 

Central Repository for Reporting and Statistics 

Common Shared Infrastructure 

Computer Security Incident Response Team 

Data Protection Authorities 

European Border and Coast Guard Agency 

European  Criminal  Records  Information  System  Reference 
Implementation 

ECRIS-TCN  

European  Criminal  Records  Information  System  –  Third 
Country Nationals 

EDPB  

EES  

EPMO  

EPPO  

EPSO  

ESP  

ETIAS   

EU   

European Data Protection Board 

Entry/Exit System 

Enterprise Project Management Office 

European Public Prosecutor’s Office 

European Personnel Selection Office 

European Search Portal 

European Travel Information and Authorisation System 

European Union 

 
 
 
EUAN   

Eurodac  

Eurojust  

eu-LISA  

Europol  

e-CODEX    

HR   

ICD  

ISMS  

IT  

JHA  

JIT  

EU Agencies Network 

European Asylum Dactyloscopy Database 

European Union Agency for Criminal Justice Cooperation 

European Union Agency for the Operational Management of 
Large-Scale  IT  Systems  in  the  Area  of  Freedom,  Security 
and Justice 

European Union Agency for Law Enforcement Cooperation 

e-Justice Communication via Online Data Exchange 

Human Resources 

Interface Control Document 

Information Security Management System 

Information Technology 

Justice and Home Affairs 

Joint Investigation Team 

JITs CP  

Joint Investigation Teams Collaboration Platform 

KoM  

LFS  

LLM  

MB  

MID  

MS  

MVP  

NCA  

PMB  

PNR 

SAC  

sBMS  

SDLC  

Kick-off Meeting 

Legislative Financial Statement 

Legal Language Model 

Management Board 

Multiple-Identity Detector 

Member States 

Minimum Viable Product 

National Competent Authorities 

Programme Management Board 

Passenger Name Record 

Schengen Associated Countries 

shared Biometric Matching System 

Software Development Life Cycle 

 
 
SIEM  

SIS  

SLA  

TDD  

TEF  

TOF  

VIS  

Security Information and Event Management System 

Schengen Information System 

Service Level Agreements 

Test-Driven Development 

Transversal Engineering Framework 

Transversal Operation Framework 

Visa Information System 

 
 
 
 
1. 

INTRODUCTION 

1.1.  Purpose and scope of the evaluation 

The European Union Agency for the Operational Management of Large-Scale IT Systems 
in  the  Area  of  Freedom,  Security  and  Justice  (eu-LISA)  was  established  by  means  of 
Regulation (EU) No 1077/20111 with a view to ensuring the operational management of 
existing large-scale IT systems in the area of freedom, security and justice (at that stage: 
the second generation Schengen Information System – SIS II, the Visa Information System 
– VIS, and Eurodac) and potentially to support the development and operation of additional 
IT systems to be established. The establishing Regulation was amended on 20 July 2015 
by  Regulation  (EU)  No  603/2013  (Eurodac  recast  Regulation)2  to  reflect  the  changes 
brought by this Regulation with regard to the Agency's tasks related to Eurodac. 

Following an evaluation of the Regulation in 20173, the mandate of eu-LISA was revised 
based  on  Regulation  2018/1726  (hereinafter  referred  to  as  “eu-LISA  Regulation”),  the 
subject of this evaluation. While Regulation 2011/1726 was fully replaced and repealed by 
eu-LISA Regulation, the new Regulation builds largely upon the previously established 
legal  framework. The general  objective of the Agency thus  remains  to  “contribute to  a 
high-level  security  within  the  area  of  freedom,  security  and  justice,  including  by 
facilitating  border  management  and law enforcement”.  In  addition,  the Agency has the 
following specific objectives based on eu-LISA Regulation:  

•  To  contribute  to  a  coordinated,  cost-effective  and  coherent  IT  environment  at 
Union  level  in  the  area  of  freedom,  security  and  justice  that  is  in  line  with 
fundamental rights;  

•  To  contribute  to  the  development  of  IT-based  solutions  in  the  area  of  freedom, 

security and justice; and  

•  To contribute within the scope of its mandate to evidence-based Union migration 
and security policy-making and to the monitoring of the proper functioning of IT 
systems in the area of freedom, security and justice. 

1  Regulation  (EU)  No  1077/2011  of  the  European  Parliament  and  the  Council  of  25  October  2011 
establishing a European Union Agency for the operational management of large-scale IT systems in the 
area of freedom, security and justice, OJ L 286/1, 01.11.2011.  

2  Regulation  (EU)  No 603/2013  of  the  European  Parliament  and  of  the  Council  of  26 June  2013  on  the 
establishment of 'Eurodac' for the comparison of fingerprints for the effective application of Regulation 
(EU)  No 604/2013  establishing  the  criteria  and  mechanisms  for  determining  the  Member  State 
responsible for examining an application for international protection lodged in one of the Member States 
by a third-country national or a stateless person and on requests for the comparison with Eurodac data 
by  Member  States'  law  enforcement  authorities  and  Europol  for  law  enforcement  purposes,  and 
amending  Regulation  (EU)  No 1077/2011  establishing  a  European  Agency  for  the  operational 
management of large-scale IT systems in the area of freedom, security and justice  (recast), OJ L 180, 
29.6.2013, p.1-30 

3 European Commission 2017, Report from the Commission to the European Parliament and the Council on 
the functioning of the European Agency for the operational management of largescale IT systems in the 
area of freedom, security and justice (eu-LISA), COM(2017) 346 final, Brussels. 

1 

 
 
 
Article 394 of the eu-LISA Regulation calls for an evaluation of the performance of the 
Agency in relation to its objectives, mandate, locations and tasks with a view of identifying 
potential  areas  for  improvements.  The  Commission  is  also  required  to  report  to  the 
European Parliament (EP), the Council and the Management Board (MB) of eu-LISA on 
the findings of the evaluation.  

The Commission carried out the supporting study between October 2023 and November 
2024, covering the period from the entry into force of the current eu-LISA Regulation in 
2018  until  March  2024.5  The  evaluation  also  compared  the  situation  with  the  period 
between 2012 and 2018 before the current regulation entered into force. While the study 
formally  concluded  in  2024,  the  evaluation  integrates  developments  up  to  June  2025. 
Extending  the  evaluation  period  allowed  the  Commission  to  reflect  the  significant 
organisational  and  management  changes  that  the  Agency  underwent  in  parallel  to  the 
study, thereby ensuring that the assessment captures the most recent progress and provides 
a more comprehensive and accurate picture of the Agency’s performance. 

As regards methodology, a mixed methods approach has been used, including information 
gathered via desk research, surveys, interviews and field visits, and was informed by the 
triangulation of a variety of sources. A range of methodological tools and techniques were 
used. For more details on the methodology please see Annex II. 

As part of this evaluation, a wide range of stakeholders were consulted through various 
methods  (interviews,  surveys,  workshops,  etc.).  The  stakeholders  consulted  include  the 
European  Commission,  staff  and  contractors  of  eu-LISA,  Member  States  authorities 
participating in the governance bodies of eu-LISA (including Advisory Groups, Project 
Management Boards and Management Board members) from both the Home Affairs and 
Justice  domains,  EU  Agencies  and  Offices  (including  the  European  Border  and  Coast 
Guard  Agency,  Eurojust  and  European  Public  Prosecutor’s  Office),  the  European  Data 
Protection  Supervisor  (EDPS)  and  national  data  protection  authorities.  A  detailed 
description of the stakeholder consultation is presented in Annex V.  

This evaluation analyses whether the eu-LISA Regulation contributed to the achievement 
of the general objectives of the Agency in an effective, efficient and coherent manner. This 
evaluation includes an analysis of the legal and policy frameworks governing eu-LISA and 
focus  largely  on  the  performance  of  the  tasks,  the  organisation  set-up,  including  the 
governance and locations of the Agency, and the management of economic, human and 
technical resources. 

Furthermore, the evaluation analyses the relevance and EU added value of the Agency, by 
assessing  to  what  extent  the  new  mandate  of  the  Agency,  introduced  in  2018,  has 
contributed to achieving the objectives of the Agency, and to which extent it has supported 
Member States in the development and implementation of the large-scale IT systems in the 

4 Regulation (EU) 2018/1726 of the European Parliament and of the Council of 14 November 2018 on the 
European  Union  Agency  for  the  Operational  Management  of  Large-Scale  IT  Systems  in  the  Area  of 
Freedom, Security and Justice (eu-LISA), and amending Regulation (EC) No 1987/2006 and Council 
Decision 2007/533/JHA and repealing Regulation (EU) No 1077/2011, OJ L 295, 21.11.2018, p. 99–137 
5  Study  to  support  the  evaluation  of  Regulation  (EU)  2018/1726  of  the  European  Parliament  and  of  the 

Council of 14 November 2018 on eu-LISA. ISBN: 978-92-68-34401-9 

2 

 
 
Area of Freedom, Security and Justice. Additionally, the evaluation analyses whether the 
objectives  of  eu-LISA  could  have  been  achieved  sufficiently  by  Member  States  acting 
alone. 

Finally,  the  evaluation  looks  at  the  relevance  of  eu-LISA  in  the  context  of  current  and 
emerging  needs  and  challenges.  This  includes  assessing  the  relevance  of  the  eu-LISA 
regulation in terms of its overall scope and objectives. 

The  evaluation  takes  stock  of  the  state  of  play  in  the  implementation  of  the  eu-LISA 
Regulation and identifies a series of areas of attention for improvement that will feed into 
the future work of the Commission, eu-LISA and Member States. 

2.  WHAT WAS THE EXPECTED OUTCOME OF THE INTERVENTION? 

2.1 Description of the intervention and its objectives 

Objective of eu-LISA 

The establishment of a European Agency for the operational management of large-scale 
IT  systems  in  the  area  of  freedom,  security  and  justice  (“eu-LISA”)  dates  back  to  the 
adoption of its Regulation in 2011 as a solution for the long-term operational management 
of SIS II, VIS and EURODAC. In 2017, the Commission published its first evaluation of 
the  initial  eu-LISA  Regulation.6  Albeit  the  evaluation  concludes  that  eu-LISA  overall 
effectively  ensured  the  management  of  large-scale  IT  systems  and  created  clear  added 
value, it found significant room for improvement. 

Following  the  2017  evaluation,  the  Commission  adopted  a  new  eu-LISA  Regulation, 
Regulation (EU) 2018/1726,7 which is the subject of this evaluation. The objective was to 
address  the recommendations esteeming from  the evaluation,  as  well as to improve the 
functioning of the Agency and to enhance and strengthen its role to ensure that the mandate 
met  the  challenges  at  EU  level  in  the  area  of  freedom,  security  and  justice.  This  new 
instrument, aimed also at inserting in the Regulation changes deriving from policy, legal 
or  factual  developments  and  in  particular  to  reflect  the  fact  that  new  systems  were  be 
entrusted to the Agency and the responsibility of the Agency to develop the interoperability 
between the large-scale IT systems. In particular, these changes included: 

-  Giving eu-LISA enhanced responsibilities with regard to data quality subject to the 

adoption of specific legislative amendments/proposals. 

-  Giving  eu-LISA  responsibility  for  the  development  of  interoperability  actions 

subject to the adoption of the relevant legislative proposals. 

-  Addressing  the  possible  need  for  eu-LISA  to  develop,  manage  and/or  host  joint 
technical  solutions  for  the  national  implementation  of  technical  aspects  of 
obligations deriving from EU legislation on decentralised systems in the area of 
freedom, security and justice for interested Member States; and 

6  European  Commission  2017,  Report  on  the  functioning  of  the  European  Agency  for  the  operational 
management  of  large-scale  IT  systems  in  the  area  of  freedom,  security  and  justice  (eu-LISA),  COM 
(2017) 346 final, Brussels. 

7 Regulation (EU) 2018/1726 of 14 November 2018 on the European Union Agency for the Operational 
Management of Large-Scale IT Systems in the Area of Freedom, Security and Justice (eu-LISA), and 
amending Regulation (EC) No 1987/2006 and Council Decision 2007/533/JHA and repealing Regulation 
(EU) No 1077/2011, Official Journal of the European Union. L295/99, 21.11.2018. 

3 

 
 
-  Addressing changes required by the adoption of the ECRIS-TCN proposal. 

This new instrument extended the mandate of the Agency to allow it to take over new tasks 
in order to render border management more effective and secure and to reinforcing security 
and combatting and preventing crime. The general objective of the Agency thus remains 
to  “contribute  to  a  high-level  security  within  the  area  of  freedom,  security  and  justice, 
including  by  facilitating  border  management  and  law  enforcement”.  In  addition,  the 
Agency  has  the  following  specific  objectives  based  on  the  establishing  eu-LISA 
Regulation, and the amendments introduced by the IT systems’ establishing regulations 
(cf. the intervention logic, see Figure 1):  

-  To  contribute  to  a  coordinated,  cost-effective  and  coherent  IT  environment  at 
Union  level  in  the  area  of  freedom,  security  and  justice  that  is  in  line  with 
fundamental rights. 

-  To  contribute  to  the  development  of  IT-based  solutions  in  the  area  of  freedom, 

security and justice, including the takeover of e-CODEX; and  

-  To contribute within the scope of its mandate to the implementation of EU policies 
and  to  the  monitoring  of  the  proper  functioning  of  IT  systems  in  the  area  of 
freedom, security and justice. 

The evaluation  

Article 39 of the eu-LISA Regulation  calls  for an evaluation  of the performance of the 
Agency in relation to its objectives, mandate, locations and tasks with a view to identifying 
potential areas for improvement.  

The strategic objective of this evaluation was to assess whether, based on its Regulation8, 
eu-LISA functions smoothly and with high added value to ensure the effective and efficient 
implementation  of  EU  policies  in  the  area  of  cross-border  law  enforcement,  border 
management, and justice. It aimed to examine how eu-LISA Regulation contributes to a 
coordinated,  cost-effective  and  coherent  IT  environment  with  a  view  to  identifying 
potential areas for improvement (cf. Article 39). 

As regard the subject matter scope, the evaluation examined the implementation of eu-
LISA Regulation, covering all objectives and tasks of eu-LISA based on the Regulation as 
well as aspects relating to its organisational set-up, management and oversight. As a result, 
the evaluation developed a series of recommendations based on the findings relating to the 
previous  points  and  overarching  conclusions  and  recommendations  on  the  performance 
and functioning of the Regulation.  

In terms of the scope from a time perspective, the evaluation concerns the period after 
the entry into force of eu-LISA Regulation, i.e. November 2018 until March 2024, but also 
compares the situation against the time before it entered into force based on the previous 
evaluation9. While the evaluation is retrospective, recommendations for improvement have 

8 Regulation (EU) 2018/1726 of the European Parliament and of the Council of 14 November 2018 on the 
European  Union  Agency  for  the  Operational  Management  of  Large-Scale  IT  Systems  in  the  Area  of 
Freedom, Security and Justice (eu-LISA), and amending Regulation (EC) No 1987/2006 and Council 
Decision 2007/533/JHA and repealing Regulation (EU) No 1077/2011. 

9 The  eu-LISA Regulation extend the mandate of eu-LISA.  However, some of the tasks entrusted by the 
establishing  regulation  remain.  Therefore,  while  the  primary  focus  of  the  evaluation  is  to  assess  the 

4 

 
 
 
also been developed. The geographical scope covers the EU-level, EU Member States and 
Schengen Associated Countries (“Member States”). 

Intervention logic 

For  the  purpose  of  the  evaluation,  a  detailed  analysis  was  carried  out  identifying  the 
Agency’s  objectives  (general  and  specific)  and  tasks  and  the  links  between  them.  The 
intervention logic below (Figure 1) serves to depict the chain of expected effects associated 
with the eu-LISA Regulation, by identifying the needs and objectives of eu-LISA, as well 
as inputs and actions to achieve the outputs, outcomes, and impacts in line with the five 
evaluation criteria (relevance, coherence, effectiveness, efficiency and EU added value).  

The  intervention  logic  shows  the  needs  leading  to  the  intervention  logic,  namely  the 
smooth  operation  of  large-scale  IT  systems  to  respond  to  challenges  relating  to  border 
management and law enforcement within the Schengen area. 

In particular, the intervention logic depicts the objectives of eu-LISA at different levels: 

•  One general objective: to contribute to a high level of security within the area of 
freedom, security and justice, including by facilitating border management and law 
enforcement within the area of freedom, security and justice. 

•  Three specific objectives:  

o  To contribute to a coordinated, cost-effective and coherent IT environment 
at Union level in the area of freedom, security and justice that is in line with 
fundamental rights;  

o  To  contribute  to  the  development  of  IT  based  solutions  in  the  area  of 

freedom, security and justice; and 

o  To  contribute  to  evidence-based  Union  migration  and  security  policy-
making and to the monitoring of the proper functioning of IT systems in the 
area of freedom, security and justice; and 

•  Fourteen operational objectives. 

The  intervention  logic  identified  the  inputs,  including  human,  financial  and  technical 
resources,  structures  and  processes  dedicated  to  achieving  the  operational,  specific  and 
ultimately general objectives of the Agency.  

Finally,  the  intervention  logic  translates  the  objective  and  inputs  into  a  hierarchy  of 
expected effects, including at the level of outputs, outcomes and impacts. 

The effectiveness is measured by how the objectives (general, specific and operational) are 
translated into expected outputs, expected outcomes, and ultimately expected effects. The 
efficiency compares the inputs with expected outputs, expected outcomes, and expected 
effects. The coherence measures to what extent the Regulation avoid gaps and overlaps 
and assesses the degree of internal coherence as well as the extent to which it is align with 
other relevant EU legal instruments and policies. The relevance measures to which extent 
the objectives (general, specific and operational) identified to achieve the needs translate 

changes since the introduction of the eu-LISA Regulation, it is also important to consider the aspects of 
eu-LISA that have not significantly changes since its establishment.  

5 

 
 
into the expected outputs, outcomes and impacts. Finally, the EU added value measures 
how to which extent the EU intervention helped to achieve the defined objectives.  

6 

 
Figure 1: Intervention logic of eu-LISA’s Regulation 

7 

 
 
 
 
2.2  Points of comparison  

In  2017,  pursuant  to  Article  31  of  the  eu-LISA  establishing  Regulation  (EU)  No 
1077/201110,  the  European  Commission  carried  out  an  evaluation  of  the  Agency 
(hereinafter the 2017 evaluation) to: 

•  Examine the way and the extent to which the Agency effectively contributes to 
the operational management of large-scale IT systems in the area of freedom, 
security and justice and fulfils its statutory tasks; and 

•  Assess  the role of the  Agency in  the context of  a Union strategy aimed  at  a 
coordinated, cost-effective and coherent IT environment at Union level.  

Given  that  no  formal  Union  strategy  for  a  coordinated,  cost-effective  and  coherent  IT 
environment had been put  in  place  at  the time, the evaluation  focused on the Agency’s 
contribution to the establishment of a coordinated, effective and coherent IT environment 
for managing large-scale IT systems that support the implementation of Justice and Home 
Affairs (JHA) policies. 

The 2017 evaluation examined the way and the extent to  which the Agency effectively 
contributes to the operational management of large-scale IT systems in the area of freedom, 
security  and  justice  and  the  implementation  of  eu-LISA's  tasks  taking  into  account  the 
relevant  legal  and  policy  frameworks,  namely  the  eu-LISA  Agency  establishing 
Regulation and the legislative instruments governing the IT systems the Agency manages. 
The 2017 evaluation covered the period from 1 December 2012 to 30 September 2015. 

The  2017  evaluation  identified  two  general  objectives:  (i)  to  ensure  the  operational 
management of SIS II, VIS and Eurodac; and (ii) to ensure the preparation development 
and  operational  management  of  other  large-scale  IT  systems  in  the  ASFJ.  In  addition, 
seven specific objectives, considered the expected results of the Agency, are identified: (i) 
to  ensure  effective,  security  and  continuous  operation  of  large-scale  IT  systems;  (ii)  to 
ensure efficient and financially accountable management of large-scale IT systems; (iii) to 
ensure an adequately high quality of service for users of large-scale IT systems; (iv) to 
ensure continuity and uninterrupted services; (v) to provide a high level of data protection; 
(vi)  to  provide  an  appropriate  level  of  data  and  physical  security;  and  (vii)  to  use  an 
adequate project management structure for efficiently developing large-scale IT systems. 
Finally,  the  2017  evaluation  identified  sixteen  tasks  of  eu-LISA  and  analyses  how  the 
Regulation  has  contributed,  through  the  different  tasks,  to  the  achievement  of  the 
objectives.  

The  2017  evaluation  confirmed  that  the  Agency  effectively  ensures  the  operational 
management of large-scale IT systems in the area of justice and home affairs and fulfils 
the tasks laid down in the establishing eu-LISA Regulation as well as new tasks entrusted 
to  it.  It  also  found  that  eu-LISA  effectively  contributed  to  the  establishment  of  a  more 
coordinated, effective and coherent IT environment for the management of large-scale IT 

10  Regulation  (EU)  No  1077/2011  of  the  European  Parliament  and  the  Council  of  25  October  2011 
establishing a European Union Agency for the operational management of large-scale IT systems in the 
area of freedom, security and justice, OJ L 286/1, 01.11.2011.  

8 

 
 
 
systems that support the implementation of Justice and Home Affairs (JHA) policies. The 
2017 evaluation therefore provided the necessary reassurance about the continuing validity 
of the rationale behind creating eu-LISA and its further existence. In particular: 

• 

• 

• 

In  regard  to  effectiveness:  the  2017  evaluation  found  that  the  Agency  had 
overall  achieved  its  objective  of  effectively  ensuring  the  operational 
management of the three IT systems entrusted to it. 
In regard to efficiency: the 2017 evaluation found that there is a need for a full 
review of the allocation of resources for core and non-core tasks of eu-LISA, 
to be performed by the Agency, in order to ascertain whether the allocation is 
proportionate to the aims to be achieved and whether some reallocation could 
occur  (for  instance  the  development  of  new  systems).  The  Evaluation  also 
found that there was a need to put appropriate controls in place to ensure that 
sufficient staff resources are available for project management based on project 
needs. 
In terms of coherence: the 2017 evaluation found that the activities of eu-LISA 
are  generally  well-aligned  with  the  relevant  JHA  policy  frameworks  and 
strategies and the coherence of the Agency’s activities was confirmed. 

•  Finally, in terms of relevance and added value: the 2017 evaluation found that 
the  establishment  of  a  single  management  authority  to  assume  operational 
management of the three IT systems inherently creates a high level of added 
value, to the extent that the Agency effectively and efficiently carries out its 
tasks.  However,  ultimate  added  value  of  eu-LISA  is  thus  logically  highly 
dependent  on  its  ability  to  fulfil  its  core  tasks  in  an  efficient  and  effective 
manner. 

Nevertheless, the 2017 evaluation also identified certain shortcomings for all evaluation 
criteria  at  both  operational  and  organisational  levels  and  recommended  a  number  of 
changes to address them. Although the Agency proved that it could do more with the same 
level of resources and can adapt with a high degree of flexibility to new needs, the 2017 
evaluation also concluded that, should eu-LISA be made responsible for new IT systems, 
it would not be able to manage them with the resources available. In its actions, the Agency 
could be more efficient, effective and coherent and could also strengthen its relevance and 
added value. 

The 2017 evaluation came up with a set of recommendations for legislative amendments 
to the eu-LISA establishing Regulation and to other related instruments, which included: 

•  The  transfer  of  the  European  Commission’s  responsibilities  relating  to  the 

communication infrastructure to eu-LISA. 

•  A  new  provision  on  the cooperation  framework  of  eu-LISA  with  other  JHA 

agencies to clarify the scope of cooperation within eu-LISA’s mandate. 

•  An extension of the scope of the pilot schemes that can be entrusted to eu-LISA. 
•  Additional tasks in relation to eu-LISA management of the systems, including: 
(i) an extended responsibility for eu-LISA in generating statistical reports for 
each  system;  and  (ii)  a  new  task  to  produce  data  quality  and  data  analysis 
reports.  

Finally, the evaluation identified the possible need to increase the tasks entrusted to eu-
LISA and introduced a limited set of recommendations for legislative amendments to the 
eu-LISA  establishing  Regulation  and  to  other  related  instruments,  including  changes 

9 

 
derives from amended EU legislation (such as the Financial Regulation and the Framework 
Financial  Regulation),  changes  derived  from  the  adoption  by  the  co-legislators  of  the 
Commission  proposals  that  entrust  new  systems  to  the  Agency  such  as  the  Entry/Exit 
System,  and  changes  to  allow  eu-LISA  to  provide  assistance  or  support  to  the  relevant 
Commission  services  on  technical  issues  related  to  existing  or  new  systems,  where 
requested.  

The same day of the adoption of the 2017 evaluation report, the Commission tabled, on the 
basis of the evaluation recommendations, a proposal to amend the eu-LISA establishing 
Regulation  and  the  system’s  instruments.  eu-LISA’s  mandate  was  revised  based  on 
Regulation (EU) 2018/1726 (the “eu-LISA Regulation”), and the role in the development 
of new large-scale IT systems considerably expanded, as further detailed in Section 3.  

3.  HOW HAS THE SITUATION EVOLVED OVER THE EVALUATION PERIOD? 

Considering the responsibilities of the Agency, the 2018 eu-LISA Regulation established 
a series of tasks that were latter extended through amendments of the eu-LISA Regulation 
by the IT systems establishing Regulations11. The tasks of the Agency detailed in the eu-
LISA Regulation (article 3 to 16) are the following:  

•  Adding responsibilities relating to new IT systems and Interoperability (Article 6, 

7, 8, 8a, 8b, 8c, 13):  

11 The 2018 eu-LISA Regulation was amended in 2019, 2022, 2023, and 2024 by:  

-  Regulation  (EU)  2018/1816  of  the  European  Parliament  and  of  the  Council  of  17  April  2019 
establishing  a  centralised  system  for  the  identification  of  Member  States  holding  conviction 
information  on  third-country  nationals  and  stateless  persons  (ECRIS-TCN)  to  supplement  the 
European Criminal Records Information System and amending Regulation (EU) 2018/1726 ;  
-  Regulation  (EU)  2019/817  of  the  European  Parliament  and  of  the  Council  of  20  May  2019  on 
establishing  a  framework  for  interoperability  between  EU  information  systems  in  the  field  of 
borders and visa and amending Regulations (EC) No 767/2008, (EU) 2016/399, (EU) 2017/2226, 
(EU)  2018/1240,  (EU)  2018/1726  and  (EU)  2018/1861  of  the  European  Parliament  and  of  the 
Council and Council Decisions 2004/512/EC and 2008/633/JHA;  

-  Regulation  (EU)  2019/818  of  the  European  Parliament  and  of  the  Council  of  20  May  2019  on 
establishing a framework for interoperability between EU information systems in the field of police 
and judicial cooperation, asylum and migration and amending Regulations (EU) 2018/1726, (EU) 
2018/1862 and (EU) 2019/816; 

-  Regulation (EU) 2022/850 of the European Parliament and of the Council of 30 May 2022 on a 
computerised  system  for  the  cross-border  electronic  exchange  of  data  in  the  area  of  judicial 
cooperation  in  civil  and  criminal  matters  (e-CODEX  system),  and  amending  Regulation  (EU) 
2018/1726 (Text with EEA relevance); 

-  Regulation  (EU)  2023/969  of  the  European  Parliament  and  of  the  Council  of  10 May  2023 
establishing  a  collaboration  platform  to  support  the  functioning  of  joint  investigation  teams  and 
amending Regulation (EU) 2018/1726; 

-  Regulation (EU) 2024/982 of the European Parliament and of the Council of 13 March 2024 on the 
automated  search  and  exchange  of  data  for  police  cooperation,  and  amending  Council 
Decisions 2008/615/JHA and 2008/616/JHA and Regulations (EU) 2018/1726, (EU) No 2019/817 
and (EU) 2019/818 of the Europe an Parliament and of the Council (the Prüm II Regulation); and 

-  Regulation (EU) 2025/12 of the European Parliament and of the Council of 19 December 2024 on 
the collection and transfer of advance passenger information for enhancing and facilitating external 
border checks, amending Regulations (EU) 2018/1726 and (EU) 2019/817, and repealing Council 
Directive 2004/82/EC; and Regulation (EU) 2025/13 of the European Parliament and of the Council 
of  19  December  2024  on  the  collection  and  transfer  of  advance  passenger  information  for  the 
prevention,  detection,  investigation  and  prosecution  of  terrorist  offences  and  serious  crime,  and 
amending Regulation (EU) 2019/818. 

10 

 
 
 
o  The  Entry/Exit  System  (EES),  the  European  Travel  Information  and 
Authorisation  System  (ETIAS),  DubliNet,  and  the  interoperability  of  IT 
systems were included in the 2018 eu-LISA Regulation;  

o  The computerised system for the cross-border electronic exchange of data 
in the area of judicial cooperation in civil and criminal matters (e-CODEX), 
the  European  Criminal  Records  Information  System  for  third-country 
nationals  (ECRIS-TCN),  the  European  Criminal  Records  Information 
System  Reference  Implementation,  and  the  joint  investigation  teams 
collaboration platform (JITs CP) were added by later amendments in 2019, 
2022 and 2023 triggered through the systems’ establishing Regulations);  

•  Detailing of the tasks relating to the operation of IT systems, including requiring 
eu-LISA  to  develop  solutions  to  develop  technical  solutions  and  meet  the 
requirements for systems which are running 24 hours a day, 7 days a week, ensuring 
data quality and interoperability (Article 10, 12, 13); 

•  Specification of tasks relating to the communication infrastructure (Article 11);  
•  Providing  a  possibility  for  eu-LISA  to  support  the  Member  States  and  the 

Commission on request (Article 16);  

•  Enhancing the Agency’s responsibilities in research and pilot testing (Article 15); 

• 

and  
Inclusion  of  a  possibility  for  eu-LISA  to  collaborate  with  international 
organisations, only with the authorisation of the Management Board after having 
received the Commission’s prior approval (Article 43). 

A summary of the tasks of eu-LISA can found in Figure 2 below. 

Figure 2: Overview of eu-LISA's taks 

It should be noted that additional tasks have been added after the start of this evaluation, 
in particular in regards to API-PNR (Advanced Passenger Information – Passenger Name 

11 

 
 
 
Record), as well as in relation to the tasks related to the Prüm II router (Article 8d)12. These 
tasks are, however, not in scope of this evaluation.  

The  governance  structure  of  eu-LISA  has  also  evolved  since  2018,  consisting  of  the 
following bodies:  

•  The  Management  Board  (MB)  is  made  up  of  representatives  from  the  EU 
Member States and the European Commission, represented by DG HOME (Article 
20).  In  addition,  Schengen  Associated  Countries  (SAC)  are  Members  of  the 
Management Board13, while Eurojust, Europol, Frontex and the European Public 
Prosecutor’s Office (EPPO) have observers status. The role of the MB is to provide 
the overall orientation for eu-LISA’s activities and oversee the implementation of 
eu-LISA’s tasks (Article 19). The Management Board also appoints the Executive 
Director, who is accountable to the MB (Article 25). The MB meets ca. five times 
annually. To focus on specific business-related aspects, the MB created an Audit 
Compliance and Financial Committee (ACFC) p, an informal sub-group of the MB 
The Management Board adopted the ACFC’s first mandate on 20-21March 2018. 
The ACFC held its first meeting on 19 September 2018. The mandate of the ACFC 
was revised  by Management Board Decisions on  20 September 2022 and 15-16 
November 2023).  

•  The Advisory Groups (AGs): the objective of the AGs is to provide the MB with 
the necessary technical expertise relating to the IT systems managed by eu-LISA. 
The AGs are made up of representatives from them EU Member States, SAC and 
the Commission, as well as in some cases representatives from Europol, Eurojust, 
Frontex and EPPO as observers.  

The eu-LISA Regulation established 4 AGs (Article 27): SIS II, VIS, Eurodac and EES-ETIAS 
AGs. The number of AGs extended since 2018 to a total of 9, including: the ECRIS-TCN AG 
(set up in 2019), the Interoperability AG (2019).  

•  Five  Programme  Management  Boards  (PMBs)  on  ETIAS14,  EES15,  ECRIS-
TCN16,  e-CODEX17  and  JITs  CP18  have  been  established  by  the  establishing 
Regulations  of  the  relevant  IT  systems.  They  comprise  seven  Member  State 
representatives, the Commission and the chair of the AG of the same system. In 
some cases, additional eu-LISA representatives may join the PMB (e.g. in the case 
of the EES  and ETIAS  PMBs). PMBs advise the MB on aspects  relating  to  the 
development and/or handover of the related systems.  

12 The "Prüm II router" is a central, secure point for EU member states to automatically exchange police and 
criminal  data  under  the new  Prüm  II  Regulation. It  acts  as  a  central  "message  broker"  that  connects 
national databases, eliminating the need for each member state to establish direct connections with every 
other country. This new  router facilitates faster and more  efficient searches of data  like fingerprints, 
facial images, and police records for law enforcement purposes.  

13  The  Schengen  Associated  countries  have  approved  the  Arrangement  of  their  participation  in  eu-LISA. 
They have voting rights limited to the systems in which they participate, and to the items listed in Article 
2(2) of the Arrangements.  

14 Regulation (EU) 2018/1240, Article 73. 
15 Regulation (EU) 2017/2226, Article 37. 
16 Regulation (EU) 2019/816, Article 11. 
17 Regulation (EU) 2022/850, Article 13. 
18 Regulation (EU) 2023/696, Article 11. 

12 

 
 
While the e-CODEX PMB is established as a permanent body (Article 13.1 of the 
e-CODEX Regulation19), the other PMBs are established on a temporary basis20 
during  the  designing  and  development  phase  of  the  systems.  In  addition,  the  e-
CODEX  PMB  has  an  additional  responsibility  to  monitor  the  respect  for  the 
principle of the independence of the judiciary and prompt preventive or corrective 
action where necessary.  

Within eu-LISA, the MB Secretariat plays a central role in supporting the governance 
bodies and ensuring their smooth functioning, including by organising and documenting 
meetings,  ensuring  the  implementation  of  decisions  taken  and  communicating  with 
members of different bodies.21 

Figure 3: Governance structure of eu-LISA 

During this evaluation, the implementation of eu-LISA’s tasks was organised through three 
separate  departments  including  10  units  and  29  sectors.  In  addition,  there  were  in  total 
seven executive units directly connected to the Executive Director. This also comprised 
the security unit which is, therefore, set up independently from any internal or operational 
departments.  During  the  Evaluation  period,  the  Agency  underwent  a  revision  of  the 
organisational set up.  

The Agency’s seat is located in Tallinn, Estonia, while two operational sites are situated 
in  Strasbourg,  France.  Whereas  the  Executive  Director,  the  corporate  governance  and 
corporate  services  departments,  as  well  as  the  security  unit  are  located  in  Tallinn,  the 
operations department is based in Strasbourg22. A third site, which operates as a backup for 
the systems, is located in St. Johann im Pongau, Austria. The Agency also has a liaison 
office in  Brussels.  Most  of eu-LISA’s staff is  located in  Strasbourg (73%). 26% of the 
Agency’s staff is located in Tallinn and 1% in Brussels. 

19  Regulation  (EU)  2022/850  of  the  European  Parliament  and  of  the  Council  of  30  May  2022  on  a 
computerised system for the cross-border electronic exchange of data in the area of judicial cooperation 
in civil and criminal matters (e-CODEX system) and amending Regulation (EU) 2018/1726 (Text with 
EEA relevance), OJ L 150, 1.6.2022, p. 1–19.  

20 PMBs are established as sub-groups of the MB. Their members are also attending the MB. 
21 eu-LISA, 2022, Annual Activity Report 2021, p. 63.  
22  Article  17(3)  of  the  e-CODEX  Regulation  provides  that  the  tasks  relating  to  the  development  and 

operational management of the e-CODEX shall be carried out in Tallinn, Estonia. 

13 

 
 
 
 
 
4.  EVALUATION FINDINGS  

4.1.  To what extent was the intervention successful and why?  

The different evaluation criteria are analysed in the section below. 

4.1.1.  Effectiveness 

Key evaluation findings  

At the level of the general and specific policy objectives of the eu-LISA Regulation, the 
findings of the evaluation are mixed. With its achievements relating in particular to the 
management and operation of IT systems, notably the Schengen Information System, the 
Visa Information System and Eurodac, eu-LISA was able to contribute to a coordinated, 
cost-effective and coherent IT environment at Union level in the area of freedom, security 
and  justice.  Thereby,  the  Agency  made  contributions  to  ensuring  a  high-level  security 
within the area of freedom, security and justice. However, there is room for improvement 
based on the shortcomings relating to the development of the systems and the limited use 
of Articles 15 and 16 relating to pilot testing and ad hoc support to Member States. 

At the level of the operational objectives, the extent to which eu-LISA has been effective in 
achieving its objectives differs between its tasks:  

- The operational management of existing systems is generally considered as very effective 
and has indeed been highlighted as a strength. This also includes the handover / takeover 
of already existing systems (i.e. e-CODEX);  

-  The  development  of  systems  is  considered  less  effective,  a  conclusion  based  on  the 
significant  delay  in  the  time  to  markets,  which  originated  from  quality  issued  of  the 
intermediary deliverable of the systems;  

The effectiveness in conducting tasks  that are not relating to the systems, such as research, 
pilot  testing,  or  ad  hoc  support  to  the  Member  States,  is  considered  mixed.  While  the 
evaluation  has  found    the  positive  outputs  on  research,  there  has  been  limited  use  of 
Articles 15 and 16 relating to pilot testing and ad hoc support, which is e.g. related to the 
limited scope of Article 16(4). 

This  evaluation  criterion  examined  the  extent  to  which  eu-LISA  has  fulfilled  its 
expectations and met its objectives. The evaluation assessed to which extent eu-LISA has 
contributed to a high level of security within the area of freedom, security and justice, and 
in  particular,  eu-LISA’s  contribution  a  coordinated,  cost-effective  and  coherent  IT 
environment  at  Union  level  in  the  area  of  freedom,  security  and  justice,  eu-LISA’s 
contribution to the development of IT-based solutions in the area of freedom, security and 
justice, and eu-LISA contribution within the scope of the mandate to evidence-based Union 
migration and security policy-making and to the monitoring of the proper functioning of 
IT systems in the area of freedom, security and justice. 

To  what  extent  are  procedures  in  place  to  ensure  the  coherence  of  eu-LISA’s 
cooperation activities with the policies and activities of its stakeholders? 

14 

 
As outlined in the 2018 eu-LISA Regulation, the core objectives of the Agency embody 
the following:23 

-  The development of large-scale IT systems using an adequate project management 

structure for efficiently developing such systems; 

-  The effective, secure and continuous operation of large-scale IT systems; 
-  The efficient and financially accountable management of large-scale IT systems; 
-  An adequately high quality of service for users of large-scale IT systems; 
-  Continuity and uninterrupted service; 
-  A  high  level  of  data  protection,  in  accordance  with  Union  data  protection  law, 

including specific provisions for each large-scale IT system; and 

-  An  appropriate  level  of  data  and  physical  security,  in  accordance  with  the 
applicable rules, including specific provisions for each large-scale IT system. 

Based  on  the  available  evidence,  the  above-mentioned  core  objectives  are  met  to  a 
moderate- to large extent. In regard to safeguarding effective, secure and continuous 
operation; the continuity of uninterrupted services; the provision of data protection; 
the  provision  of  high-quality  services  to  users;  and  ensuring  data  and  physical 
security,  the  Agency  has  met  its  objective  to  a  large  extent.  This  is  backed  by  the 
available evidence, both from interviews and from the online surveys. In fact, 24 out of 35 
(69%) AG and PMB members who answered this specific question indicated in the online 
survey that eu-LISA’s management of the systems contributes to a ‘large’ or ‘very large’ 
extent to a coordinated, cost-effective and coherent IT environment at EU level in the area 
of freedom, security and justice.  

As  concerns  the  development  of  large-scale  IT  systems  as  well  as  the  efficient  and 
financially accountable management of IT systems, eu-LISA has met its objectives to a 
moderate extent due to delays in development processes and lack of accountability. 

As concerns the tasks of eu-LISA, we found that system-specific tasks, as well as providing 
support  to  Member  States  are  working  well.  However,  according  to  interviewed 
stakeholders,  there  is  still  room  for  improvement  on  communication  even  within  the 
governance bodies to facilitate more efficient decision-making processes. Nearly half of 
the  respondents  of  the  AG  and  PMB  survey  acknowledged  that  Member  States  are 
supported to a “very large” or “large extent”. However, 18% of the respondents did not 
have a  clear opinion on  this matter which could  indicate a need for  eu-LISA to  further 
improve  communication  and  engagement  with  certain  stakeholders  to  ensure  that  the 
benefits of its mandate are fully understood and utilised. 

Stakeholder interviews reinforce the perception that eu-LISA is capable of meeting its core 
objectives. At the same time, stakeholders have been raised concerns that the broadened 
scope  of  the  latest  Regulation  could  risk  generating  a  higher  workload  without  a 
corresponding increase in staff. Thee LFS are prepared by the Commission in cooperation 
with eu-LISA.  

23 See Regulation (EU) 2018/1726 of the European Parliament and the Council of 14 November 2018 on eu-

LISA, http://data.europa.eu/eli/reg/2018/1726/2025-01-28. 

15 

 
 
Stakeholders interviewed also pointed out concerns about internal administrative processes 
within the Agency that delay the effective allocation of staff. This suggests communication 
issues, as well as an internal organisation that does not evolve in line with the growing 
responsibilities of the Agency. 

How has been the performance of eu-LISA in operationally managing large-scale IT 
systems? 

For the purpose of addressing this question on operational management performance, it is 
important  to  assess  several  indicators,  including  systems  availability,  number  of 
operational incidents, response to incidents, as well as user satisfaction with data quality 
or the overall IT infrastructure provided. 

Based on available evidence, the Agency is performing well in the context of operational 
management of its IT systems with some existing areas for improvement. 

The  perception  of  eu-LISA  stakeholders  on  the  Agency’s  performance  in  operationally 
managing large-scale IT systems has been largely positive. The Agency has demonstrated 
a strong capability in handling technical issues, with AG and PMB members expressing 
satisfaction  with  response  times,  incident  management,  resolution  times,  and  the 
prevention of operational disruptions. 

Most respondents also recognised eu-LISA's contribution to creating a coordinated, cost-
effective, and coherent IT environment within the EU's Area of Security and Justice (62 
out  of  74,  84%).  From  the  perspectives  of  users  of  the  IT  systems  this  satisfaction  is 
reflected, too. The management of DubliNet, Eurodac, SIS, and VIS is generally seen as 
satisfactory, with high marks for system availability and operational support. The overall 
level of satisfaction among AG and PMB members is high: 86% of respondents indicated 
that they are satisfied or very satisfied. 

However, the feedback also shows that the levels of satisfaction vary between different 
areas: while the level of satisfaction is highest concerning security-related aspects (93% 
were  ‘satisfied’  or  ‘very  satisfied’),  the  general  level  of  satisfaction  is  slightly  lower 
concerning  both  the  Agency’s  services  (86%  were  ‘satisfied’  or  ‘very  satisfied’)  and 
operational requirements for IT systems (77% were ‘satisfied’ or ‘very satisfied’). Only a 
few users have experienced issues such as temporary system unavailability and data entry 
errors. . 

For  users  of  the  IT  systems,  the  efficiency  of  maintenance  work  (37  out  of  43,  86%), 
technical development (28 out of 40, 70%), and system availability (39 out of 43, 91%) 
has been well-received, indicating that eu-LISA is able to ensure its systems to be reliable 
and  up  to  date.130  However,  there  are  mixed  perceptions  on  the  cost-efficiency  of 
communication tools, suggesting a need for eu-LISA to evaluate and possibly optimise the 
tools used for interacting with Member States. 

Despite these successes, stakeholders have identified areas for improvement, such as the 
use of management systems, tracking of change requests, and communication regarding 
production issues and test environments.  In particular: 

16 

 
-  VIS: Limited interoperability, limited scalability of system (due to its age), limited 

data quality, and limited search functionalities; 

-  SIS:  Limited  capacity  of  eu-LISA  to  provide  further  development  and 
improvements, burdensome administrative processes, limited interoperability with 
other databases and systems, limited data quantity (i.e. not all information that is 
available  in  a  particular  Member  State),  limited  data  quality  (e.g.  the  system 
contains a very high number of alerts on issued documents without any significant 
practical effect), limited file sizes, limited extent of automation, downtimes, and 
limited awareness among users concerning specific (new) types of alerts. 

-  Eurodac:  Limited  functionalities,  limited  information  exchange  after  biometric 
correspondence,  and  limited  ability  to  triangulate  with  data  in  case  of  false 
positives. There is room for improvement concerning reaction times. 

At the same time, however, available data from the survey with AG and PMB members 
shows that the Agency has been responsive to feedback and suggestions for improvement 
of the systems, at least to a moderate or large extent.24  

To  what  extent  has  eu-LISA  been  effective  in  ensuring  the  secure  and  continuous 
operational management of large-scale IT systems entrusted to it? 

Overall, eu-LISA has been very effective in ensuring the secure and continuous operational 
management of large-scale IT systems during the evaluation period.  

The IT systems currently under operation and management of eu-LISA (in particular SIS 
and VIS) have so far worked without any major incidents. They have proven to be highly 
reliable  and  capable  of  delivering  uninterrupted  operations,  effectively  handling  a 
significant volume of requests 25262728. The takeover process of existing IT systems (e.g. e-
CODEX) was considered to be efficient to a very large or large extent. 

Case study: Handover/takeover process for e-CODEX 

e-CODEX  was  developed  between  2010  and  2016  by  21  EU  Member  States  with  the 
participation of Turkey, Jersey, the Council of Bar Societies of Europe, and the Council of 
the Notaries  of the European Union, under the lead of ministry for justice North-Rhine 
Westphalia.29 Since 2016, the development was maintained through the Me-CODEX and 
Me-CODEX II projects. 

Examining how well the Agency has performed in taking over already existing systems is 
also relevant for the assessment on how well eu-LISA is contributing to the development 

24  Study  to  support  the  evaluation  of  Regulation  (EU)  2018/1726  of  the  European  Parliament  and of  the 

Council of 14 November 2018 on eu-LISA. ISBN: 978-92-68-34401-9, page. 60 

25 eu-LISA, 2023, Annual Activity Report 2022. 
26 eu-LISA, 2023, Report on the technical functioning of central SIS II 2021-2022.  
27 eu-LISA, 2022, Report on the technical functioning of the Visa Information System (VIS). 
28 eu-LISA, 2023 Eurodac Annual Report 2022. 
29  The  development  was  financed  half  financed  by  EU  grants  from  the  competitiveness  and  innovation 
framework  programme  of  DG  CNECT  and  through  an  action  grant  administered  by  DG  JUST  the 
Commission's Directorate-General for Justice and Consumers(DG JUST) under the Justice programme, 
and half by the Member States that participated in the project. 

17 

 
 
of  IT-based  solutions  overall.  This  is  particularly  relevant  in  case  of  the  e-CODEX 
handover/takeover  process,  which  was  transitioned  from  a  managing  consortium  to  eu-
LISA in Spring 2024.  

The  evaluation  found  that  the  transition  process  was  efficient  and  overall  positive:  the 
process  appears  was  completed  largely  within  the  agreed  timeframe  with  only  minor 
delays, which have been outside of the responsibility/control of eu-LISA. The expertise of 
staff involved in the process on the side of eu-LISA was a point underlined by stakeholders, 
pointing  to  the  need  to  include  experienced  personnel  into  the  handover/takeover  of 
systems; a finding that relevant to consider in the transition of other systems in the future. 

The evaluation identified some challenges in the takeover process such as the complexity 
of the system, the granularity and shifting focus  of the takeover terms, the bureaucratic 
process, the unclear responsibility allocation and the communication with the Commission.  

Moreover, the evaluation highlighted the high level of user satisfaction regarding several 
dimensions,  such  as  security,  data  protection,  and  service  continuity.  In  particular,  the 
evaluation  underlined  the  service  reliability  and  efficient  security  measures  of  the 
operating systems. However, the evaluation pointed to differences in the (high) levels of 
satisfaction between VIS, SIS, and Eurodac and less satisfied with eu-LISA’s performance 
in relation to Eurodac. 

Nevertheless, the evaluation also unveils areas for improvement. For example, response 
timeliness  to  incidents  and  service  disruptions  could  be  improved,  with  the  evaluation 
indicating  a  need  for  enhanced  agility  and  responsiveness.  For  example,  for  Eurodac, 
certain reported downtimes were, in fact, rather attributed to the national interfaces. the 
evaluation also pointed to the lack of information about the origins of outages, likely again 
attributed to Member States. For the case of the SIS recast, the evaluation highlighted the 
slow responsiveness of the Agency to questions, as well as the unavailability of the test or 
production environment at certain uncalculatable times. The proposed 'software factory' 
model,  employing  DevSecOps 
  (Development,  Security,  Operations)  principles, 
underscored  the  Agency's  efforts  to  bolstering  control  and  reducing  dependencies  on 
external contractors. 

To what extent has eu-LISA fulfilled its obligation in relation with data protection, data 
and  physical  security  in  accordance  with  the  applicable  rules,  including  specific 
provisions for each large-scale IT system? 

Overall, eu-LISA has fulfilled its obligations reliably concerning data protection, data, and 
physical security in alignment with applicable rules and specific provisions for each large-
scale IT system. 

The  Agency's  proactive  approach  to  data  protection  is  evidenced  by  several  strengths. 
Notably,  eu-LISA  has  invested  in  comprehensive  training  and  awareness  initiatives  to 
sensitise  personnel,  ensuring  a  culture  of  compliance  with  data  protection  principles30. 
Additionally, the establishment of clear storage limitation rules, categorised into Category 
1 and Category 2 data, aligns with GDPR principles, ensuring purpose limitation and data 

30 eu-LISA, 2022, Consolidated Annual Activity Reports. 2018-2022.  

18 

 
 
minimisation31. The involvement of the Data Protection Officer (DPO) in initiatives such 
as Change Management reflects a robust commitment to data protection by design and by 
default32. Monthly updates of the Record of Processing Activities (RoPA)33 and frequent 
monitoring  of  the  data  breach  register  further  underscore  eu-LISA's  commitment  to 
compliance and proactive risk management. 

However, the Agency faces significant challenges in the ever-evolving landscape of digital 
threats  and  global  instabilities.  The  surge  in  cybercrime  poses  a  substantial  risk, 
necessitating  continuous  adaptation  to  ensure  the  security  of  large-scale  IT  systems34. 
Additionally, the lack of a clear categorisation of incidents, for example, whether or not, 
they  affect  system  availability  poses  challenges  in  implementing  adequate  mitigating 
measures promptly35. 

Furthermore,  the  workload  on  the  Data  Protection  Office  raises  concerns  about  the 
effectiveness of their involvement in various initiatives, highlighting a need for internal 
resource allocation improvements and a clearer definition of priorities. Ensuring that the 
DPO is adequately resourced and able to focus on the most pressing issues is essential for 
allowing  them  to  support  the  Agency  in  identifying  and  addressing  challenges,  thereby 
reinforcing the Agency’s role in developing the systems. This workload is only expected 
to  increase  in  the  future  and  technological  solutions  should  be  considered  according  to 
interviewees. The rapid advancement of disruptive technologies such as AI presents both 
opportunities  and  risks,  requiring  a  delicate  balance  between  innovation  and  data 
protection36. For example, AI can support the DP by automating routine tasks, identifying 
potential data privacy issues through advanced analytics, and enhancing compliance efforts 
through real-time monitoring and predictive analysis. 

Despite these challenges, stakeholders generally perceive eu-LISA's work on cybersecurity 
and data protection positively. The Agency's management of communication infrastructure 
using  TESTA-ng  ensures  secure  data  transfer,  supporting  effective  decision-making  by 
national authorities37. 

In conclusion, while eu-LISA has made significant strides in upholding data protection and 
security  obligations,  there  is  always  a  need  for  continued  vigilance  and  adaptation  to 
address emerging challenges effectively. 

How has eu-LISA performed in the development of new large-scale IT systems? 

eu-LISA's performance in developing new large-scale IT systems shows a mixed picture, 
characterised  by  successes,  but  more  increasing  and  pressing  challenges  with  the 
development of selected new systems such as the EES.  

31 eu-LISA, 2023, Eurodac 2022 Annual Report 
32 eu-LISA, 2023 DPO Annual Work Report. 2022 
33 eu-LISA, 2022, Consolidated Annual Activity Reports. 2018-2022. 
34 EU Parliament, 2023, High common level of cybersecurity at the institutions, bodies, offices, and 

agencies of the Union. 
35 eu-LISA, 2023 EDPS Report.  
36 eu-LISA, 2022, DPO Annual Work Reports. 2018- 2021. 
37 eu-LISA, 2020, Single Programming Document 2022–2024. 

19 

 
 
Notably,  while  users  expressed  satisfaction  with  the  efficiency  of  maintenance  work, 
uninterrupted availability, and responsiveness of eu-LISA in managing existing systems, 
sentiments  by  survey  respondents  and  interviewees  regarding  the  development  of  new 
systems were less favourable.38 

Delays  in  the  development  of  the  EES,  but  also  ETIAS  and  ECRIS-TCN,  have  eroded 
stakeholders' confidence in eu-LISA's capacity to deliver efficiently in the future as can be 
seen from consultations with the PMB, the user survey as well as concluded from several 
interviews  from  both  the  operational  and  management  perspective.  In  relation  to  these 
systems,  significant  quality  and  delivery  issues  have  been  encountered  which,  in  turn, 
resulted  in  cascading  delays  that  impacted  dependent  work  streams  and  systems  For 
example, the revised roadmap at the time of the supporting study indicated that EES was 
planned  to  commence  operations  in  late  2024,  while  ETIAS  was  scheduled  to  become 
operational in the first half of 2025. According to the current planning EES is scheduled 
in  late  2025  and  ETIAS  in  late  2026.  It  should  be  noted,  however,  that  developing  a 
completely new system and revising or transferring an existing one, such as the recast of 
the VIS, present distinct challenges. While new system development offers possibilities 
for  flexibility  and  innovation,  it  requires  extensive  planning  and  testing  to  ensure 
compatibility and functionality. Recasting an existing system involves modifying existing 
infrastructure,  which  can  expedite  deployment,  but  may  result  in  complexities  in 
integrating with legacy systems and accommodating evolving requirements. Whereas eu-
LISA was less effective with regard to the development of completely new systems, there 
are good practices and positive lessons learned from the recast of existing systems, such 
as the SIS. 

Moreover,  there  have  been  challenges  in  taking  up  new  justice-related  systems,  where 
functional  knowledge  had  to  be  built  up  at  eu-LISA  such  as  ECRIS-TCN  and  the  JITs 
collaboration platform, underscore the complexity of the Agency's mandate. In particular, 
the  development  and  takeover  of  justice  systems  such  as  e-CODEX  has  also  been 
highlighted as challenging by stakeholders from the justice area, including members of the 
e-CODEX AG and PMB. This stems from the fact these IT systems are very different in 
their functionality and (business) processes from the other systems (such as SIS) that eu-
LISA  has  been  managing  before.  Hence,  the  necessary  awareness  and  expertise  (e.g. 
regarding  the  respect  of  the  principle  of  the  independence  of  the  judiciary)  had  to  be 
gradually built up in order to ensure that no infringements (e.g. at the technical level) take 
place. On the positive side, the structured and regular participation of members from the 
e-CODEX  community,  including  those  represented  in  the  PMB,  has  proven  beneficial. 
According  to  several  interviewees  from  the  justice  community  and  in  particular  the  e-
CODEX AG and PMB, the extensive experience with and maturity of e-CODEX when it 
was  handed  over  have,  overall,  facilitated  a  smooth  transition.  According  to  their 
perception, eu-LISA has a strong track record of bringing together various communities 
and has gained substantial knowledge about e-CODEX over the year during the handover. 
Most  stakeholders  also  see  various  opportunities  for  further  involvement  in  the 
management of justice systems within the Agency, such as the possibility for eu-LISA to 

38  Study  to  support  the  evaluation  of  Regulation  (EU)  2018/1726  of  the  European  Parliament  and of  the 

Council of 14 November 2018 on eu-LISA. ISBN: 978-92-68-34401-9, section 5.1.2.5. 

20 

 
 
contribute with high-level technical expertise in the comitology, as it is now the case in the 
Home Affairs systems39, to become a recognised institution to align technical approaches 
between Member States and to become a technical pillar of the justice community. At the 
current stage, however, it was highlighted that the expertise within the Agency is rather 
technical/law enforcement-related, still requiring additional expertise in the justice sector 
before it can participate in a role as described above.  

While  some  specific  interoperability  systems  such  as  the  European  Search  Portal  and 
Shared Biometric Matching Service are perceived as moderately to largely developed by 
survey respondents, others such as the Multiple Identity Detector exhibit limited progress. 
Consequently, the Agency's ability to contribute to the timely development of new systems 
is, on average, rated as only moderately successful by respondents.  

A  significant  concern  highlighted  by  stakeholders  is  the  over-reliance  on  external 
contractors for system development. This dependency raises issues of knowledge retention 
and  control  over  development  processes,  potentially  compromising  flexibility  and 
adaptability  in  the  long  term.  Additionally,  resource  constraints  and  shifting  priorities 
contribute to discrepancies between planned and actual activities, posing further challenges 
to  effective  system  development.  Despite  these  challenges,  eu-LISA's  introduction  of 
interoperability components such as the CSI programme shows that the Agency is actively 
addressing historical inefficiencies and enhancing interoperability. The CSI platform aims 
to  streamline  operations  and  improve  system  scalability,  aligning  with  the  Agency's 
transformative objectives.  

In conclusion, while eu-LISA has been effective in managing existing systems and some 
new interoperability components, the development of new large-scale IT systems presents 
distinct challenges  that require proactive strategies and collaboration with stakeholders. 
Overall, this reflects a need for strategic adjustments and a concerted effort to overcome 
existing challenges. 

To  what  extent  have  eu-LISA  tripartite  set-up  and  governance  structure  affected  the 
performance of the Agency in delivering its core tasks? 

This  question  concerns  the  split-up  of  eu-LISA  over  three  main  locations  in  Tallinn 
(Estonia), Strasbourg (France) and St. Johann im Pongau (Austria), and how this set-up 
affects the performance of the Agency in delivering its core tasks, which are defined as 
development and operation of the large-scale IT-systems. For this purpose, it is important 
to (i) analyse the separation of tasks between the three locations and (ii) analyse survey 
and interview perceptions on the tripartite set-up.  

Based  on  literature  analysed40  and  stakeholders  consulted,  it  became  apparent  the 
headquarters in Tallinn are hosting more horizontal and managerial tasks, whereas the two 
operational sites in Strasbourg are concerned with the actual development and operation 

39  eu-LISA  participated  in  the  SIS  SIRENE  Committee,  Smart  Borders  Committee,  Interoperability 

Committee and Visa Committee.  

40 E.g. PwC (2020). Future analysis “Governance model for the European Union IT Agency (eu-LISA)”  

21 

 
 
of the IT-systems. St. Johann im Pongau serves also as an operational set-up, providing a 
back-up in case the systems in France should not be usable.  

In general, most staff show understanding for the separation of locations in Tallinn and 
Strasbourg, particularly as concerns the political dimension of the Agency. In terms of the 
operational dimension, given that the different main areas of work are clearly separated 
between different locations, daily tasks can be usually organised and completed within a 
single location, reducing the need for administrative coordination between the different 
locations.  In  case  of  a  need  for  cross-location  coordination,  existing  communication 
channels and methods seem to be sufficient to meet the requirements of eu-LISA staff. The 
back-up  site  in  Austria  is  at  the  current  stage  only  marginally  included  in  coordination 
activities with the other two sites, allowing for it to perform its tasks effectively by itself. 
These  findings  are  also  underlined  by  the  AG/PMB  survey,  with  roughly  75%  of 
stakeholders pointing out that the tripartite set-up does only to a “moderate” “low” or “no” 
extent have an effect on the Agency’s efficiency.  

However, some stakeholders from within eu-LISA consulted via interviews have pointed 
out  that  the  geographical  separation  of  locations  can  occasionally  hinder  mutual 
understanding between staff regarding the concrete and operational needs of other sites.. 
The  physical  distance  between  the  sites  may  also  create  technical  difficulties,  such  as 
ensuring availability, connectivity, latency, maintenance, and regular back-ups of systems 
between the two technical centres in Strasbourg and St. Johann in Pngau.  

In addition, stakeholders have highlighted that both main locations come with location-
specific  advantages  and  disadvantages,  leading  to  particular  challenges  at  the  different 
locations. In particular Tallinn was portrayed as a major obstacle for recruiting, given the 
comparably low salaries (for IT talent) as well as the attractiveness of Tallinn as a city 
(remote location, harsh weather conditions, connectivity). The office premises, however, 
were portrayed as central, inviting and modern. Strasbourg’s premises, on the other hand, 
have been described as rather outdated and uninviting, whereas the geographical location 
of the city is considered an asset. To ensure that staff feel equally valued across eu-LISA’s 
main locations, further adaptations and investments should be considered to enhance parity 
in working conditions, for instance in terms of facilities, infrastructure and overall working 
environment.  In  the  context  of  a  highly  competitive  IT  labour  market,  where  working 
conditions  are  key  to  attracting  and  retaining  skilled  professionals,  ensuring  balanced 
conditions across sites would help strengthen cohesion, and support staff retention within 
the Agency.  

Nonetheless and only with few exceptions, stakeholders largely agreed that the current set-
up as foreseen within eu-LISA’s mandate is overall effective and that no major adaptions 
are needed within the mandate. Adaptions are rather recommended on the operational and 
organisational levels, through e.g. increased coordination and teaming efforts. 

To  what  extent  have  eu-LISA’s  management  structures  and  monitoring  of  external 
contractors been effective to assure compliance and timely implementation of contracts 
related to the development and operational management of large-scale IT systems? 

22 

 
This question concerns the effectiveness of eu-LISA’s outsourcing approach in particular 
with regard to management structures and monitoring capabilities of the Agency. For this 
reason, it is important to analyse not only how the current management structures look like 
but also how stakeholders assess the performance of the current structures. In the second 
step, an assessment of the appropriateness of the structures follows.  

As  concerns  eu-LISA’s  management  structures,  eu-LISA  has  long  prolonged  a  rather 
system silo-centric outsourcing approach, in which tasks have been outsourced based on a 
system-centric model..41 To overcome this approach, the Agency has established the more 
horizontal  TEF/TOF  (Transversal  Engineering  Framework  /  Transversal  Operation 
Framework’) outsourcing framework, in which different types of services (related to e.g. 
operations,  infrastructure,  application,  testing,  development,  maintenance,  etc.)  can  be 
provided by different consortia of contractors for each system. Monitoring is conducted by 
different  actors  as  e.g.  the  Finance  and  Budget  Sector,  the  Vendor  and  Contract 
Management Sector or the Business Relations Management Sector. Given the split over 
different departments,  it  should be noted that responsibilities are sometimes not  clearly 
distributed, which can lead to inefficiencies in the monitoring process.  

As  concerns  the  assessment  of  stakeholders,  most  stakeholders  confirmed  that  the 
outsourcing of certain tasks is necessary and unavoidable, and that eu-LISA is following a 
collaborative  approach,  working  closely  with 
the 
implementation  of  existing  and  planned  systems.  Nonetheless,  the  performance  of  the 
management  structures  is  considered  rather  poor.  Many  stakeholders  consider  the 
TEF/TOF model to be too complex, given the different layers of services and still silo-
centric focus on single IT-systems. In fact, the framework is so complex that a specific 
framework  contract  has  been  signed  with  a  contractor  to  manage  the  delivery  in  other 
framework contracts.  

the  external  contractors  on 

In  addition,  the  general  outsourcing  strategy  was  described  as  insufficient,  which  is 
mirrored by the survey results: more than half of respondents (53%, n=39) consider the 
current  strategy  “not  at  all”,  to  “a  low  extent”  or  to  a  “moderate  extent”  successful  in 
delivering new systems. In addition, the following weaknesses could be identified based 
on the data available:  

-  The overall procurement process is described as rather lengthy and administrative, 
often not pointing out the requirements of the procurement adequately in its full 
complexity. Moreover, the fixed price and scope of the contracts leave too little 
flexibility to undertake adaptions needed due to the dynamic developments in fields 
relevant  for  the  Agency-managed  IT-systems  (e.g.  through  technological 
advancements such as in the field of AI or new EU and national policies in relevant 
areas);  

- 

It  was  underlined  frequently  that  the  administrative  burden  of  managing  the 
external contracts is very high, as concerns documentation obligations;  

41 Government Office of Estonia (2020). Future analysis “Governance model for the European Union IT 

Agency (eu-LISA)”, p.38-39.  

23 

 
 
-  Most  stakeholders  pointed  out  that  through  the  complexity  of  the  contracts 
managed,  a  lack  of  clearly  allocated  responsibilities  leads  to  a  delusion  of 
responsibilities,  strongly  affecting  the  effectiveness  of  administrative  processes 
and slowing down operations;  

-  Contractors  of  the  Agency  noted  that  eu-LISA’s  current  mandate  is  rather 
restrictive  in  terms  of  using  new  technologies,  such  as  e.g.  cloud  technology. 
Therefore, contractors must fall back on more complex and sometimes outdated 
technologies. Problems with the current contract system, in particular as concerns 
the development of new systems, is highlighted by the survey.  

Overall, the management and monitoring of external contractors must be concluded to be 
rather insufficient, which is also underlined through the number of delays and poor quality 
of work of external contractors, as is highlighted by most stakeholders consulted. Even 
though  there are positive developments  since the arrival  of the new Executive Director 
(investment into co-sourcing strategies, working under ‘one roof’ at the same premises), 
there seems to be significant room for improvement.  

To  what  extent  has  eu-LISA  been  effective  in  ensuring  the  efficient  and  financially 
accountable management of large-scale IT systems? To what extent has eu-LISA been 
effective 
to 
objectives/expenditures, are accounts compliant with EU budget rules, are discrepancies 
between actuals and budgets explained? 

its  budget:  could  budgets  expenditures  be  related 

in  managing 

eu-LISA has demonstrated an overall effective management of its large-scale IT systems, 
maintaining a high level of financial accountability.  

The Agency has faced an evolving landscape of challenges, such as complex multi-annual 
planning  and  rigorous  compliance  requirements  that  have  necessitated  a  micro-
management  approach  to  budgeting.  These  challenges  have  led  to  some  difficulties, 
particularly  in  the  flexibility  and  strategic  clarity  of  outsourcing  strategies  and  agile 
development  practices.  Issues  with  outsourcing  strategies,  mainly  the  complexity  and 
strategic approach, have particularly affected the delivery of key IT systems, like the EES. 

In examining the relationship between budget expenditures and objectives, eu-LISA has 
put in place mechanisms to align spending with strategic goals. However, the specificity 
and strictness of legal bases have restricted the organisation’s agility in budget utilisation, 
making it challenging to swiftly adapt to changing technological and political demands. 
Despite these constraints, actual expenditures generally correspond to outlined objectives, 
as supported by eu-LISA's high budget implementation rates. 

Regarding  compliance  with  EU  budget  rules,  eu-LISA  follows  a  structured  budgetary 
process. The Agency submits its budget requests, negotiates the amount with the European 
Commission and eu-LISA's management, and then adheres to EU Financial Regulations. 
Interviewees have expressed concerns about the restrictive nature of this process and the 
limited involvement of Member States and the Management Board during negotiations. 
While this might suggest a need for greater transparency, there is no evidence that eu-LISA 
has  failed  to  comply  with  EU  budget  rules.  For  2022,  the  budget  accounts  provide  a 
comprehensive  picture  of  the  budget  implementation,  and  the  outstanding  budgetary 

24 

 
commitments  not  yet  expensed  represent  the  correction  of  the  budgetary  RAL  (reste  à 
liquider)  with  estimated costs  determined  by  applying  the  accrual-based  principle.  This 
indicates a structured approach to relating budgets and expenditures to objectives42. 

Discrepancies between actuals and budgets are an inevitable part of financial management, 
particularly in a rapidly changing IT environment. eu-LISA's financial reports indicate that 
discrepancies are closely monitored and generally explained by the Agency's adjustments 
to  unforeseen  changes  and  challenges.  For  instance,  interviews  highlight  the  need  for 
improved  multi-annual  planning  capabilities  and  more  flexible  budgeting  practices  to 
handle such discrepancies proactively. 

In  2022,  eu-LISA  effectively  managed  significant  budget  appropriations,  with 
commitment  appropriations  having  a  99.0% 
implementation  rate  and  payment 
appropriations  a  91.6%  rate  by  year-end.  These  figures  attest  to  the  Agency's  financial 
diligence  and  ability  to  achieve  defined  objectives  within  established  financial 
frameworks.  Financial  KPIs  are  consistently  met,  further  showcasing  eu-LISA's 
commitment to financial accountability43. 

In conclusion, while eu-LISA has faced challenges in refining its financial planning and 
outsourcing practices, it remains effective in disbursing its budget, ensuring expenditures 
reflect  objectives,  and  providing  accountability  for  any  variances.  The  Agency's  high 
budget  implementation  rates,  alongside  ongoing  efforts  to  enhance  budget-related 
processes and planning, indicates efficiency and financial accountability. 

How effective has eu-LISA been in ensuring high quality, continuous and uninterrupted 
services for users of large-scale IT systems? How effective has eu-LISA been in meeting 
the Service Level Agreement(s), in particular for availability of service, for each large-
scale IT system? 

eu-LISA  has  demonstrated  a  high  level  of  effectiveness  in  ensuring  the  quality  and 
continuity  of  services  provided  by  its  large-scale  IT  systems,  while  also  fulfilling  its 
obligations under SLAs in terms of service availability. Evidence from desk research and 
several  interviews  suggests  that  eu-LISA  has  implemented  measures  to  maintain  and 
enhance the data quality across its systems. The Agency has utilised advanced technologies 
and procedures to ensure accuracy, consistency, and timeliness of the data processed and 
stored.  Regular  audits,  quality  control  mechanisms,  and  feedback  loops  with  Member 
States contribute to the high data standards eu-LISA upholds. Concrete metrics, such as 
error rates and correction times, have reportedly been within acceptable thresholds, though 
specific figures would provide a clearer picture of the performance in this area.44 

Further evidence (e.g. from the user survey) shows that the Agency has managed to provide 
continuous  and  uninterrupted  services  through  robust  infrastructure  and  redundancy 
mechanisms designed to minimise downtime. This has resulted in high system availability 
and reliability for end-users.  As the reports from operational metrics indicate the system 

42 eu-LISA, 2023, Annual Budget Report 2022, p.27. 
43 eu-LISA, 2023, Annual Budget Report 2022, p.3. 
44 eu-LISA, 2023, Annual Activity Report 2022, pp. 24 ff.  

25 

 
 
uptimes have consistently met or exceeded the targets  set  forth  in  the respective SLAs, 
although exact uptime percentages would offer more substantive evidence. 

For each IT system managed by eu-LISA, the achievement of SLA objectives, particularly 
regarding availability of service, is as follows: 

•  DubliNet: The system has maintained operational continuity with few reported 
incidents,  meeting  its  SLA  target  for  availability.  Downtime  has  been 
minimised through effective incident response strategies45; 

•  Eurodac:  Eurodac  has  shown  high  availability  rates,  aligning  with  SLA 
requirements. eu-LISA has addressed technical challenges promptly to ensure 
the system remains accessible to users4647; 

•  SIS  (Schengen  Information  System):  SIS  has  upheld  excellent  service 
availability, as eu-LISA has implemented comprehensive monitoring and rapid 
issue resolution to comply with SLA standards4849; and 

•  VIS (Visa Information System): VIS has similarly maintained high availability, 
with eu-LISA conducting regular maintenance and upgrades to prevent service 
disruptions in line with SLA commitments5051. 

Despite  these  achievements,  eu-LISA  continues  to  face  challenges,  such  as  ensuring 
consistent data quality and addressing interoperability requirements. The development and 
integration of new systems such as EES,ETIAS and ECRIS-TCN have experienced delays, 
but eu-LISA's is actively addressing these issues with task forces and managerial escalation 
(e.g.  through  the  ED).  Continuous  efforts  to  update  and  refine  operational  processes, 
systems  enhancements,  and  stakeholder  engagement  will  be  key  to  further  improving 
service quality and reliability. Despite these challenges in the development of new systems, 
the Agency's emphasis on interoperability and the introduction of components such as the 
European Search Portal (ESP) and Common Identity Repository (CIR) demonstrate efforts 
to enhance operational capabilities and secure data exchange among systems. 

Overall, eu-LISA has demonstrated effectiveness in managing large-scale IT systems, with 
a  focus  on  ensuring  continuous  service  delivery,  meeting  SLAs,  and  addressing  user 
satisfaction.  However,  ongoing  efforts  are  needed  to  address  identified  challenges  and 
enhance  operational  efficiency,  particularly  in  the  development  of  new  systems  and 
improving interoperability across platforms. 

To  what  extent  have  the  research  and  monitoring  of  research  activities  of  eu-LISA 
contributed to the performance in delivering its core tasks? 

Since 2018, eu-LISA has ensured regular publication of various research reports and other 
documentation  that  are  accessible  on  eu-LISA  website.52  Between  2018  and  2023,  the 

45 eu-LISA, 2023, Annual Activity Report 2022, p.35.  
46 Eu-LISA, 2023, Eurodac Annual Report 2022. 
47 eu-LISA, 2023, Annual Activity Report 2022, p.35.  
48 eu-LISA 2023, SIS II technical functioning 2021-2022 Factsheet, 

https://www.eulisa.europa.eu/Publications/Reports/Factsheet%20SIS%20II%20technical%20functioni
ng%202021-2022.pdf.   

49eu-LISA 2023, Report on the technical functioning of central SIS II 2021-2022. 
50 eu-LISA, 2023, Annual Activity Report 2022., p.31. 
51 eu-LISA, 2022, Report on the technical functioning of the Visa Information System (VIS). 
52 https://www.eulisa.europa.eu/our-publications/reports.  

26 

 
 
Agency  has  published  86  reports.  These  reports  include  technical  reports,  statistical 
documentation,  and  conference  notes.  Most  thematic  report  series  are  published  on  an 
annual basis, others are produced irregularly. 

The  expansion  of  eu-LISA’s  mandate  widened  the  scope  of  the  Agency  from  not  only 
managing  to  also  developing  large-scale  IT systems.  The development  aspect  yields  an 
increase  in  its  role  of  monitoring  and  conducting  research  activities.  In  general,  it  is 
essential for eu-LISA to monitor and assess emerging innovative technologies to guarantee 
that  the  Agency  can  successfully  achieve  its  strategic  goals  of  maintaining  secure  and 
efficient external borders, and to implement state-of-the-art solutions within its systems. 

However,  research  activities  have  arguably  contributed  to  a  very  limited  extent  to  the 
performance  of  delivering  eu-LISA’s  core  tasks.  This  can  be  based  on  the  following 
arguments.  As  mentioned  above,  most  reports  published  disclose  e.g.  technical  and 
statistical information and only 12% of the total reports published between 2018 and 2023 
address  thematic  research  topics,  including  feasibility  studies.  Hence,  the  number  of 
reports contributing to the development of new systems is rather limited. Acknowledging 
the  small  number  of  staff  dedicated  to  conducting  research  within  eu-LISA,    and 
considering  the  rapid  pace  of  technological  advancements  and  the  emergence  of  new 
technologies, the overall resources -both in terms of staff and budget- are relatively limited. 
Given these constraints, eu-LISA’s resources dedicated to monitoring of research should 
be  prioritised  on  areas  directly  related  to  the  systems  developed  and  managed  by  the 
Agency, thereby generating greater added value. By refocusing its research activities, the 
Agency  could  increase  its  focus  on  future  needs  and  challenges  such  as  Artificial 
Intelligence,  biometrics,  or  quantum  computation  that  had  been  stressed  by  survey 
respondents as potential future opportunities. 

Has the Agency effectively performed its tasks related to developing new systems in line 
with  its  current  mandate?  How  effective  was  the  Agency  to  provide  the  technical 
capacity, to develop the systems? How effective was the governance structure provided 
by the Agency to support the development of the systems? 

Besides  the  operational  management  of  existing  large-scale  IT  systems,  eu-LISA  is 
responsible for the development of systems if requested by the Union. According to the 
EU  Regulation  2018/1726,  the  Agency  is  obliged  to  govern  tasks  relating  to  the 
development of new systems as laid out in specific Union legal acts as well as providing 
training  on  the  technical  system  use  (Chapter  II,  Article  9).  This  question  concerns  the 
effectivity of eu-LISA developing new large-scale IT systems in light of its governance 
bodies (i.e. Management Board, Programme Management Boards, and Advisory Groups). 

Based on evidence, the development of systems is considered less effective. Even though 
sufficient  technical  capacity  is  available,  recent  developments  suffered  from  significant 
delays.  Moreover,  the  provision  of  support  from  the  governance  bodies,  which  became 
more complex over time, was effective only to a moderate extent. 

The technical capacity to develop these systems has been acknowledged by stakeholders, 
with a majority indicating that the Agency provided the necessary expertise to a moderate 
to  very  large  extent.  However,  the  Agency's  success  in  contributing  to  the  timely 

27 

 
development of new large-scale IT systems appears to be moderate to low (39 out of 80 
respondents, 49%), as per the feedback from the Advisory Group members. 

Even though the governance bodies of eu-LISA have been evaluated as internally coherent 
to a large extent, there are identified areas for improvement, particularly in speeding up 
decision-making  processes,  ensuring  representation  from  the  judicial  domain,  and 
enhancing communication and collaboration with the Advisory Group and Member States, 
including with eu-LISA’s external contractors. This highly resonates with several inputs 
received from interviewed stakeholders. While the governance structure was described as 
functioning  well  overall,  there  are  some  issues  including,  for  instance,  its  complexity. 
Some interviewees stressed the rising number of different governance bodies in the past 
years  and  a  resulting  increase  in  workload  for  different  stakeholders.  In  addition,  the 
inclusion of justice systems into eu-LISA’s portfolio was stated to have contributed to this 
complexity, including because the bodies now represent different communities. 

Has eu-LISA effectively handled incidents on each large-scale IT systems operated by 
the Agency? 

eu-LISA's  management  of  incidents  across  its  large-scale  IT  systems  reflects  a  robust 
framework  with  generally  positive  outcomes,  although  some  challenges  have  been 
identified. 

Survey  responses  indicate  a  high  level  of  satisfaction  with  the  handling  of  incidents, 
particularly highlighting fast response times and efficient incident management. However, 
concerns  have  been  raised  regarding  system  interruptions  and  data  entry  problems, 
highlighting areas for improvement. 

Examining specific systems under eu-LISA's operation provides further insights. The SIS 
and VIS have demonstrated high availability rates, with minimal disruptions to operations 
For  the  SIS,  in  the  year  2022,  the  system  had  a  high  availability  rate  of  99.94% 
accompanied by a response time of 98.98%.  

For the VIS, it demonstrated a good level of availability, with the central system achieving 
an availability rate of 99.6%. Moreover, in terms of response time performance, the VIS 
central system showed a performance indicator of 99.7%. Overall, the VIS central system 
demonstrated a strong level of availability and responded efficiently to user requests. 

No critical security incidents occurred during the reporting period. The Agency's security 
unit  continues  to  maintain  and  develop  security  measures,  operating  its  Information 
Security Management System (ISMS) in compliance with ISMS standards and ISO27001. 

The VIS experienced a major upgrade on 20 May 2025 to ensure interoperability with the 
EES,  migration  to  a Common Shared  Infrastructure, and  integration with sBMS. These 
efforts were complemented by a reassessment of security risks and the implementation of 
new security and business continuity plans. 

Similarly,  the  Eurodac  system  has  exhibited  exceptional  reliability  and  responsiveness, 
with an availability rate of 99.9% and efficient response times. 

28 

 
Interoperability readiness is a key focus for eu-LISA, with integration efforts underway to 
enhance  data  exchange  and  collaboration  between  systems53.  Security  measures  remain 
robust, with proactive initiatives like the establishment of a Computer Security Incident 
Response  Team  (CSIRT)  and  cybersecurity  exercises  to  strengthen  incident  response 
capabilities. 

The  Agency's  support  function,  including  24/7  monitoring  and  effective  incident 
resolution, has been lauded by the Member States (e.g. in the SWOT workshops and in 
several interviews). Additionally, both, end-users as well as Management Board and AGs 
express  satisfaction  with  the  support  provided  through  the  survey,  underscoring  the 
importance of eu-LISA's role in  ensuring  the smooth  functioning  of IT systems  at  both 
national and EU levels. 

Overall, eu-LISA has showcased a very good track record in incident management, with a 
focus on timely response, high availability, and proactive security measures.  

Does  eu-LISA  Regulation  allow  the  Agency  to  cooperate  with  all  international 
organisations, which are relevant for carrying out its tasks efficiently? 

According to Article 43 of eu-LISA Regulation (2018/1726), the Agency is legally entitled 
to cooperate with international organisations that are relevant for eu-LISA to carry out its 
tasks efficiently if certain conditions are met. More specifically, according to Article 43.1, 
such cooperation must be provided for by a Union legal act. Moreover, the cooperation 
must be necessary for the performance of eu-LISA’s tasks. For all working arrangements 
established with international organisations or bodies, Article 43.2. underlines the need to 
specify  terms  of  scope,  nature,  purpose,  and  extent.  Furthermore,  the  working 
arrangements  can  only  be  concluded  with  the  authorisation  of  the  Management  Board. 
Prior  to  such  authorisation,  the  Management  Board  is  required  to  receive  an  official 
approval by the European Commission. 

For  cooperations  in  the  context  of  e-CODEX,  technical  support  shall  be  allowed  to  be 
requested and received by the cooperation parties. This shall be handled by one natural 
person per organisation.54 

Has  eu-LISA  effectively  implemented  the  recommendations  of  the  previous  external 
evaluation completed in 2016? 

The  2017  evaluation  of  eu-LISA  proposed  84  specific  recommendations  to  eu-LISA, 
particularly  on  the  Agency’s  legal  framework  as  well  as  on  structural,  organisational 
changes.55  The  recommendations  focused  on  improving  current  tasks  and  addressing 
possible future needs of the Agency. 

Overall, eu-LISA has incorporated the recommendations of the 2016 evaluation to a mixed 
extent.  In  addition,  the  recently  introduced  operational  restructuring  within  the  Agency 

53 eu-LISA, 2021, eu-LISA Strategy 2021-2027, https://www.eulisa.europa.eu/Publications/Corporate/eu-

LISA%20Strategy%202021-2027.pdf.  

54 See Article 17 of Regulation 2022/850 of the European Parliament and of the Council of 30 May 2022. 
55  See  European  Commission,  2016,  Independent  external  evaluation  of  the  European  Agency  for  the 
operational management  of  large-scale  IT  systems  in  the  area  of  freedom,  security  and  justice  –  eu-
LISA, Final evaluation report. 

29 

 
 
aims  at  further  implementing  recommendations  concerning  staff  matters  and  agility. 
Moreover,  the  management  of  eu-LISA’s  Information  Security  Management  System 
(ISMS)  was  strengthened  and  considerable  resources  have  been  invested  to  bolster 
information  security  across  its  spectrum  of  activities.  The  Agency  carries  out  thorough 
security evaluation on the systems it manages. In addition, with the publication of technical 
and statistical system reports, eu-LISA addresses the 2016 recommendation of regularly 
reviewing systems’ capacity needs. For instance, since 2018 the Agency published in total 
29 reports on technical aspects related to the different IT systems. However, there is still 
room for improvement in terms of eu-LISA’s research activities and access to Member 
States system data and audits to ensure adherence to prescribed standards and directives, 
thereby fortifying the ISMS. 

Recommendations, which were considered as “critical” in the previous evaluation, which 
have not yet been fully addressed, include, for instance, the improvement of a backup site 
for  switch  over/switch  back  sequences  between  systems,  a  more  detailed  accounting 
system to better manage budgeting of resources or to strengthen the security management 
competencies. 

Regarding the use of the backup site based in Austria, technical experts stressed during the 
interviews that due to the physical distance between Strasbourg and St. Johann im Pongau, 
an uninterrupted service in case of system failure cannot be guaranteed as switching to the 
backup  site  is  taking  too  long.  Moreover,  the  recommendation  relating  to  accounting 
practices have not been addressed to a satisfying extent. The current processes have been 
criticised by interviewees for their lack of transparency. Another significant concern that 
remains to be addressed is the absence of a robust accounting and planning framework. 
This has been pointed out as a critical issue, making it challenging to project and manage 
finances effectively in the medium and long term. 

Other recommendations classified as “very important” that have not yet been addressed 
include, among others, the definition of a sourcing strategy to improve procurement, to 
leverage  the  technical  expertise  in  the  Advisory Groups  or  to  establish  and  formalise  a 
review process allowing for a regular review of the systems’ capacity needs. Importantly, 
the Agency still lacks a common sourcing strategy. A noticeable gap has been identified 
between  the  human  resources  and  procurement  divisions  located  in  Tallinn  and  the 
operations  department  in  Strasbourg.  This  division  has  reportedly  led  to  various 
inefficiencies  over  recent  years  up  to  the  present,  particularly  in  the  processes  of 
recruitment (e.g. length of procedures) and staff development (e.g. contract prolongation), 
as  well  as  in  overseeing  contractors  (such  as  selecting,  managing,  and  monitoring 
contractor  performance).  Furthermore,  as  concerns  the  technical  expertise  within  the 
Advisory Groups, stakeholders pointed to slow decision-making processes within the AGs, 
given  that  consensus  must  be  reached.  Moreover,  once  a  decision  is  made  and 
communicated to the Management Board, technical expertise is often missing in the latter 
governance body and an adequate problem assessment is challenging. 

Hence,  to  conclude,  while  many  of  the  recommendations  have  been  addressed,  key 
outstanding recommendations include: 

30 

 
-  Address technical challenges stemming from the split between the Central Unit in 
Strasbourg and the backup site in St. Johann im Pongau to ensure uninterrupted 
services, hence, consider another back-up site in closer reach. 

-  Define a sourcing strategy. Based on such as strategy, the procurement strategy and 

principles for involving outside competences should be defined; and 

-  Establish  and  formalise  a  review  process  allowing  for  a  regular  review  of  the 
systems’ capacity needs with Member States based on the statistics generated from 
the systems and a prospective exercise. 

4.1.2.  Efficiency 

Key findings 

While the Agency seems to be operating efficiently relating to the operational management 
of existing IT systems, there is room for improvement with regard to the development of 
new IT systems. Challenges in this regard stem, for example, from difficulties in managing 
the various contractors involved.  

Considering  the  organisational  and  governance  set-up,  the  evaluation  shows  that  the 
Agency is working rather efficiently and puts regular efforts in increasing the efficiency of 
internal  processes.  Nevertheless,  challenges  relating  to  the  internal  set-up  exist,  for 
example,  concerning  the  rather  rigid  unit-based  set-up,  the  large  size  of  some  units, 
overlaps  between  the  units  dealing  with  stakeholder  relations  as  well  as  a  perceived 
disconnect across different locations that led to inefficiencies, e.g. relating to hiring and 
the management of contractors.  

The governance bodies are generally functioning efficiently, although the high workload 
relating to the organisation of various meetings has been criticised. In addition, there is 
room  for  increased  efficiency  by  streamlining  and  ensuring  better  coordination  of  the 
different  governance  bodies,  which  could  potentially  lead  to  synergies  and  reduce  the 
organisational efforts. 

The assessment of the Agency’s efficiency covers a) the financial resources and planning, 
b) the operational management of IT systems, c) the development of IT systems, and d) 
the governance and organisational set-up.  

To what extent has the governance structure of eu-LISA supported its ability to perform 
its tasks, having regard to its size, composition, organisation and working processes? 

As regards the efficiency of eu-LISA’s governance bodies (i.e. its MB, AGs, PMBs) and 
their  structure  and  how  this  structure  has  supported  the  ability  of  eu-LISA  as  an 
organisation  to  perform  its  tasks,  the  evaluation  found  that  the  Agency  has  overall 
successfully managed to perform its tasks, considering the size, composition, organisation 
and working processes of eu-LISA’s governance structure. 

Concerning  the  size  and  composition  of  the  bodies,  the  evaluation  found  that  Member 
States  were  able  to  contribute  to  the  Advisory  Groups  and  the  Management  Board. 
Participation by Member States varies which can lead to an underrepresentation of certain 
points  of  view.  The  representation  of  only  some  Member  States  in  the  PMBs  was 
considered critical, and some Member States pointed to a lack of participation possibilities 
in  terms  of  upcoming  systems.  The  size  of  the  Advisory  Groups  and  the  Management 

31 

 
 
Board was  considered by some stakeholders to  negatively  impact  an  efficient  decision-
making, given the need to align multiple different positions among sometimes more than 
30 actors.  

As concerns the organisation and working processes of the different governance bodies, 
the evaluation found the coordination between the bodies sufficient: while the Advisory 
Groups discuss the more technical questions, the Management Board is responsible for the 
overall decision-making and the Programme Management Boards are responsible for the 
preparation on new systems and acting as an intermediary between the other two bodies. 
In this context, the e-CODEX PMB is an exception as, unlike other PMBs, it is established 
as a permanent body, ensuring ongoing support, oversight and guidance for the e-CODEX 
system's development and long-term sustainability. This permanence is crucial in ensuring 
the integrity and effectiveness of the e-CODEX system, particularly in safeguarding the 
respect of the principle of the independence of the judiciary  within its framework. This 
difference is accentuated by the absence of e-CODEX PMB members in the MB, which 
stems  from  the  decision  taken  within  the  e-CODEX  PMB  aiming  at  safeguarding  the 
principle of independence of the judiciary. Nevertheless, as a result the e-CODEX PMB 
Member are less familiar with the Agency's  broader operations. The evaluation pointed 
possible improvements concerning unclear distribution of responsibilities, as well as the 
significantly high administrative burdens (e.g. for chairs) resulting from the coordination 
needs between the bodies. 

How  efficient  is  the  Agency  governance  structure  to  reach  its  goals  and  complete  its 
tasks? 

The  evaluation  assessed  the  high-level  implications  of  the  governance  structure  of  the 
Agency to reach its targets and complete its tasks. The evaluation found the governance 
structure to be overall efficient. The three primary governance entities – the Management 
Board,  Programme  Management  Boards,  and  Advisory  Groups  –  were  found  to  be 
successful and functioning efficiently, with the implementation of decisions made by these 
bodies  operating  efficiently.  Stakeholders  highlighted  that  the  representation  of  all 
Member  States  and  EU  Agencies  using  the  systems  in  the  Advisory  Groups  and 
Management Board to be a major asset, helping the end-beneficiaries of eu-LISA to raise 
a voice on how the Agency’s overall goals could be reached better.  

However, the evaluation pointed to the challenges that the increase of governance bodies 
– in line with the increasing tasks of the Agency under the new mandate – raised, leading 
to an increasing complexity of the governance structure in  eu-LISA’s overall set-up.  In 
particular,  the  governance  structure  was  seen  as  not  sufficiently  reflecting  the 
interoperability of the systems, as the bodies as well as eu-LISA’s internal organisation 
continue to operate largely on a system-centric basis. These challenges concern: 

-  The complexity of the governance structure: the number of governance bodies 
has increased exponentially in the past years. The inclusion of justice systems into 
eu-LISA’s  portfolio  was  stated  to  have  contributed  to  this  complexity,  as  the 
governance  bodies  now  represent  different  communities.  This  led  to  a  high 
administrative  burden  as  the  coordination  between  the  different  bodies  can  be 
extremely challenging and time-consuming. In general, the AGs and PMBs meet 
in different frequencies, varying between monthly meetings (e.g. ECRIS-TCN and 

32 

 
EES-ETIAS AG) and four times per year (e.g. SIS AG).56 For AGs meeting on a 
monthly basis, there is a constant need to prepare the meetings (agenda, sharing of 
materials),  follow-up  on  meetings  (executive  summaries  and  detailed  minutes) 
communicate,  collaborate,  and  coordinate.  Since  the  expansion  of  the  Agency’s 
mandate  to  the  development  of  new  systems,  the  time  and  resources  needed  to 
deliver on this managerial task has significantly grown. Overall, there can be up to 
around 200 different meetings per year.  

-  The  potential  risks  of  duplications  or  contradictions:  while  eu-LISA’s 
Regulation allows for the cooperation between the AGs (Article 27(5)), in practice, 
the  different  bodies  work  separately  and  can  lead  to  instances  of  inconsistent 
decisions among bodies, such as contradictory opinions in different AGs about the 
same matter, and the same topic may be discussed several times. While it is the role 
of  the  Chairs  and  of  the  Agency  to  note  and  react  on  inconsistencies,  such 
inconsistencies  cause  additional  work.  Coherence  in  the  opinions  of  the  AGs  is 
necessary to ensure interoperability between the different IT systems. Stakeholders 
also underlined that Programme Management Fora (PMFs) are being requested by 
Member States even in cases where there is no explicit legal base, highlighting the 
practical  demand  for  mechanisms  to  coordinate  governance  and  cross-system 
developments.  

-  The practical work of the different governance bodies: challenges relate to the 
expertise of the members of the MB and AGs across Member States and thematic 
fields, misunderstandings of the MB members of eu-LISA’s financial rules or the 
operational  work  of  eu-LISA.  This  silo  approach  is  reinforced  by  limited 
coordination between the different silos at Agency level, while at the same time 
national  coordination  mechanisms  could  also  be  strengthened  to  ensure  more 
coherent participation and input across governance structures. 

The increasing complexity also impacted the administrative burden and potential risks for 
duplications  and/or  contradictions  in  the  Agency’s  work.  While  the  current  governance 
structure may still be sufficient for eu-LISA to reach its goals at present, a potential change 
of the mandate on this point could reduce the complexity of the governance structure.  

Consultations carried out with stakeholders suggested considering reducing the number of 
separate bodies by moving away from a systems-based approach to a clustered or sectorial 
approach. Suggestions referred to clustering the governance bodies by area of operation, 
namely (1) border management, including EES, ETIAS, VIS; (2) migration and asylum, 
including Eurodac, DubliNet; (3) law enforcement and customs, including SIS, API and 
PNR, and Prüm; and (4) justice, including ECRIS-TCN, e-CODEX and the JITs CP. In 
this  context,  several  stakeholders  representing  their  Member  State  in  the  governance 
structure of the Agency highlighted that they currently considered the justice area to be 
underrepresented  within  the  governance  bodies  of  the  Agency,  which  might  reduce  the 
effectiveness and efficiency in the fulfilling eu-LISA’s tasks in the justice sector.  

Other  stakeholders  highlighted  that  the  governance  structure  should  considered  the 
individual systems’ specificities relating to their set-up and that the profiles responsible for 
the system at the national level may vary. At the same time, governance currently reflects 
a rather decentralised approach to the implementation of systems, with experts focusing 
on the priorities of their respective systems, which may not always align with a common 

56 Study to support the evaluation of Regulation (EU) 2018/1726 of the European Parliament and of the 

Council of 14 November 2018 on eu-LISA. ISBN: 978-92-68-34401-9, section 2.2.1. 

33 

 
 
overarching goal. The evaluation did not find evidence suggesting that the proposed set-
up  of  clustering  the  governance  bodies  by  area  of  operation  would  be  more  efficient. 
Therefore,  looking  forward,  it  may  be  worth  exploring  an  alternative  approach  to  the 
development of systems, potentially moving towards a more centralised governance model 
and  considering  a  different  involvement  of  national  bodies  to  support  more  coherent 
decision-making and alignment across systems. 

Case study: Governance structure and its efficiency 

Overall findings 
The evaluation of eu-LISA's governance structure is mixed in terms of efficiency. While the 
Agency  is  efficient  regarding  the  operational  management  of  systems,  challenges  arise  in 
developing new IT systems, notably in outsourcing management. The governance bodies seem 
to  function  efficiently,  but  the  analysis  revealed  concerns  regarding  the  proliferation  of 
governance  bodies  with  the  expanding  mandate,  leading  to  increased  complexity  and 
administrative burden. Proposals for improvement include streamlining governance bodies by 
operational domain and addressing the underrepresentation of the justice sector perceived by a 
limited number of stakeholders. Furthermore, evidence suggests that challenges may arise due 
to the silo-based governance model. 

Challenges of silo-based governance 
eu-LISA  operates  within  a  governance  framework  tailored  to  efficiently  manage  critical  IT 
systems essential for the area of freedom, security and justice . However, this framework, while 
effective within the area of freedom, security and justice, faces scrutiny for its adaptability and 
scalability  beyond  this  domain.  One  primary  concern  is  the  silo-based  configuration  of  eu-
LISA's  governance  model.  While  suitable  for  intra-area  of  freedom,  security  and  justice  
operations, this structure poses limitations when extending the Agency's responsibilities to other 
policy areas. 

Considering  the  intricate  relationship  between  border  management  and  migration  policies,  a 
comprehensive  approach  is  essential,  encompassing  aspects  of  security,  human  rights,  and 
international cooperation. The current governance model, with its compartmentalised approach, 
may  struggle  to  facilitate  the  necessary  coordination  and  collaboration  across  these  diverse 
policy domains. 

Similarly, in the overall context of law enforcement, seamless integration and interoperability 
of IT systems are crucial for effective operations. However, the siloed nature of the governance 
model may hinder the development and implementation of integrated solutions that cater to the 
specific needs of these interconnected policy areas. 

Moreover, as new challenges emerge and EU policy priorities evolve, the rigid structure of the 
current governance model may inhibit the Agency's ability to adapt and respond effectively. The 
fragmented decision-making processes inherent in silo-based governance could impede timely 
and coordinated action, potentially jeopardising the Agency's ability to fulfil its mandate in an 
ever-changing landscape. 

Potential for improvement 
Transitioning towards a sector-specific governance model could facilitate greater coherence and 
integration across policy areas. Clustering governance bodies by operational domain, such as 
border  management  or  law  enforcement,  could  promote  more  targeted  decision-making  and 
resource  allocation.  Strengthening  coordination  mechanisms  between  governance  bodies  is 
essential  for  overcoming  silo-based  inefficiencies.  Establishing  formalised  channels  for 
communication  and  collaboration,  such  as  joint  working  groups  or  inter-body  forums,  can 
facilitate  information  sharing  and  promote  a  more  cohesive  approach  to  decision-making. 
Fostering a culture of cross-domain expertise within governance bodies can help bridge silos 
and facilitate a more comprehensive understanding of interconnected policy issues. Encouraging 

34 

 
 
 
interdisciplinary collaboration and providing training opportunities on cross-cutting topics can 
empower stakeholders to navigate complex challenges more effectively. 

Pathways for improvement 
Looking  ahead,  while  the  overall  state  is  positive,  the  findings  of  the  evaluation  suggest 
considering  strategic  reforms  to  fortify  the  governance  structure's  coherence  and  efficiency. 
Ideas for improvement by stakeholders include consolidating bodies by operational domain and 
fostering closer cooperation mechanisms to streamline decision-making processes. For example, 
stakeholders have proposed enhancing the efficiency by consolidating governance bodies for 
both the existing systems and those currently in development, thereby aligning decision-making 
processes with operational domains, facilitating more agile responses to emerging challenges 
and opportunities within the organisation. Additionally, implementing cross-functional teams or 
task  forces  that  include  representatives  from  different  governance  bodies  could  facilitate 
collaboration  and  ensure  alignment  of  strategies  and  priorities  across  the  organisation. 
Additionally, there is a consensus on the need for a more sector-specific orientation within the 
governance framework, with some stakeholders noting that the justice sector could benefit from 
stronger representation. To support this, stakeholders suggested establishing ‘liaison officers’ 
between  the  predominant  governance  bodies  overseeing  the  other  systems  and  the  relevant 
justice  sector.  In  essence,  while  challenges  remain,  there  is  a  prevailing  sense  of  optimism 
regarding the potential for improvement. Strategic reforms guided by stakeholder feedback offer 
a pathway to enhance eu-LISA's governance efficiency. By embracing these recommendations 
and adapting to evolving needs, eu-LISA can strengthen its position as a trusted custodian of EU 
IT  systems,  poised to  navigate  the dynamic landscape  of the  digital  era  with  confidence  and 
efficacy. 

To  what  extent  have  eu-LISA’s  organisational  solutions,  including  outsourcing 
strategies, affect the performance of the Agency in developing the new systems? 

The evaluation found that the organisational strategies adopted by eu-LISA, ranging from 
various  solutions  including  outsourcing,  have  yielded  a  mixed  impact  on  the  Agency's 
ability to develop new systems.  

Over the past years, the evaluation found that eu-LISA largely outsourced the work related 
to the development of new IT systems. This resulted in a series of challenges concerning 
the delivery of these IT systems in appropriate quality and efficiency, ranging from the 
complexity  of  the  previous  and  current  outsourcing  approach,  the  division  in  roles  and 
responsibilities between eu-LISA and external contractors, the contractual management, 
to the volume of outsourcing. Stakeholders note that eu-LISA does not involve contractors 
in governance meetings with Member States, limiting transparency and exposure to the 
wider governance bodies. 

The Commission’s Internal Audit Service report of 202057 found that, compared to other 
Agencies,  the  proportion  of  structural  service  providers  was  high  and,  in  addition, 
structural service providers were not only used for temporary or ad-hoc duties but also for 
core operational and/or administrative tasks. While there was a steep rise of eu-LISA staff 
since the adoption of the legal mandate in 2018, the number of tasks and the complexity 
related  to  the  development  of  new  large-scale  IT  systems  has  also  increased,  and  the 
evaluation showed that the Agency still did not possess sufficient personnel to handle all 
these tasks within the Agency itself. In addition, throughout the evaluation period, eu-LISA 
was in strong competition with the private sector and other EU institutions as concerns the 
filling of vacancies for highly skilled IT staff, fuelling the problems to recruit enough staff 

57 Audit report: IAS.A2-2019-EU-LISA-001, 5 June 2020. 

35 

 
 
to fill all open vacancies. Hence, though a controversial decision within eu-LISA itself, the 
Agency had to continuously pursue the outsourcing of core tasks to external contractors. 

Furthermore,  the  work  conducted  by  eu-LISA  is  regulated  through  the  relevant  legal 
instruments.  The  results-focussed  perspective  of  some  legal  requirements  in  practice 
forced the Agency (i.e. not through the legal text itself but through the operational practices 
chosen by the Agency in response to the legal text) to an outsourcing model in which IT 
developments  were  contracted  according  to  the  waterfall  principle  instead  in  an  agile 
fashion. This resulted in lengthy development cycles and difficulties adapting to changing 
needs or technological advancements, ultimately impacting project timelines and quality.  

The  evaluation  showed  the  lack  of  a  strategic  approach  for  the  outsourcing  of 
resources. Due to the historic development of the Agency and its overall set-up, this has 
led to a situation in which outsourcing was, and still is, the default modus operandi of the 
Agency. This has led to challenges with regard to the development and delivery of new 
systems, such as EES, which was the Agency’s very first own development. However, in 
recent months, there has been a noticeable evolution, with eu-LISA taking steps to adapt 
its  approach  towards  outsourcing,  including  greater  integration  and  co-location  with 
contractors to improve supervision and control. 

SIS, VIS, and Eurodac were systems developed before the creation of the Agency, and the 
Agency  has  been  entrusted  with  the  responsibility  to  manage  these  three  systems 
operationally.  Therefore,  originally,  the  Agency  used  a  system-centric  silo  approach  to 
outsource work related to SIS, VIS, and Eurodac. The same approach was taken in relation 
to EES, with the main purpose to externalise any substantial risks. However, stakeholders 
have pointed out that the Agency was not sufficiently equipped in recent years to manage, 
guide, and control the EES contractors and that the contractual vehicles themselves were 
not well-equipped for the development of an IT system.58 

The Agency changed its outsourcing approach from a system-centric silo approach with a 
limited  number  of  consortium  partners  per  system  to  a  more  horizontal  approach  of 
‘layered’  outsourcing.  Under  this  model,  different  types  of  services  (related  to  e.g. 
operations,  infrastructure,  application,  testing,  development,  maintenance  etc.)  can  be 
provided by different consortia of contractors for each system. The restructuring indicates 
a shift from the limitations posed by isolated silos, such as fragmented service provision, 
lack of standardised operations, and reduced interoperability. The aim is to allow for more 
agile, standardised, timely, and cost-effective service provision. 

However,  the  implementation  of  this  approach  has  encountered  some  limitations.  For 
example,  under  this  approach,  technical  experts  who  had  been  hired  based  on  their 
technical  expertise,  skills,  and  experience,  have  been  assigned  to  contract  management 
roles in which they were not able to apply their specialised skills. Challenges appeared in 
managing  the  transitions  between  different  phases  of  outsourcing  contracts  (“sunset 
period”).  The  absence  of  dedicated  internal  transition  teams  accompanying  contractors 
have contributed to complexities in the handover process (e.g. concerning decisions that 
still  need  to  be  taken,  operational  developments  still  outstanding,  potential  areas  of 

58  Study  to  support  the  evaluation  of  Regulation  (EU)  2018/1726  of  the  European  Parliament  and of  the 

Council of 14 November 2018 on eu-LISA. ISBN: 978-92-68-34401-9, section 2.3.3. 

36 

 
 
improvement,  testing  results,  bugs  in  the  software,  etc.),  leading  to  disruptions  and 
inefficiencies, risking delays in the development processes of large scale-IT systems. The 
establishment of internal transition teams emerges as a crucial aspect for enhancing eu-
LISA's operational efficiency and project management. 

Therefore, the evaluation found that the Agency did not strike an appropriate balance in 
recent  years  between  delivering  technical  developments  inhouse  and  simultaneously 
ensuring  that  contractors  deliver  on  time  in  the  necessary  quality.  The  evaluation 
emphasises  that  the  Agency  should  maintain  strong  internal  control  over  system 
development,  keeping  key  technical  expertise  and  developments  in-house,  even  while 
leveraging external contractors. 

To  improve  the  situation,  the  evaluation  highlighted  the  importance  for  the  Agency  to 
increase the share of insourced work to reduce dependency on external contractors, and 
investing  in  internal  capabilities  –  namely  hiring  (more)  technical  staff  that  is  actually 
tasked  with  technical  work  for  which  their  expertise  is  needed,  rather  than  managing 
contracts – to enhance efficiency and oversight. A clearer strategic approach to outsourcing 
and improved contractual management, to mitigate risks and ensure the timely delivery of 
quality  systems,  are  considered  necessary  by  the  evaluation  given  the  escalating 
complexity of the IT systems entrusted to eu-LISA. 

The evidence gathered for this evaluation showed certain positive developments on this 
topic.  For  instance,  eu-LISA  has  transitioning  towards  a  more  integrated  outsourcing 
approach  and  strengthening  supervision  over  contractors  and  started  to  invest  into  co-
sourcing  with  contractors  (i.e.  a  closer,  more  integrated  working  relationship  with 
contractors, for instance, on-site alongside eu-LISA staff) and implement the policy that 
everyone  has  to  work  ‘under  one  roof’.  This  means  that  contractors  carry  out 
developmental work related to the new system on premise in Strasbourg and provides eu-
LISA staff with increased supervision and control possibilities.  However, achieving the 
optimal balance between in-house development and outsourcing remains a challenge. 

How efficient have been the management of resources by eu-LISA? To what extent has 
eu-LISA efficiently used its human and financial resources to ensure the development 
and the effective, secure and continuous operation of large-scale IT systems? To what 
extent has eu-LISA succeeded in building up in-house capabilities for handling various 
tasks entrusted to it? 

The  evaluation’s  assessment  of  eu-LISA's  management  of  resources,  particularly  its 
human and financial resources, revealed a mixed picture of efficiency and challenges.  

The  evaluation  found  that  the  Agency  demonstrated  success  in  certain  areas,  such  as 
utilising  advanced  technology  for  central  IT  systems  such  as  SIS  and  VIS,  ensuring 
effective,  secure  and  continuous  operation,  and  fostering  coordination  among  Member 
States and providers.  

However, while the management of existing IT systems contributed to a coordinated and 
cost-effective environment, the evaluation pointed to moderate successes and challenges 
associated with the development of new IT systems and the handover/takeover processes. 
Challenges  arising  from  slow  bureaucratic  processes  and  unclear  tasks  have  been 
identified, indicating inefficiencies in project management. 

Outsourcing  practices,  although  moderately  successful,  were  criticised  for  their 
complexity, contractual management, limited accountability and volume. The evaluation 
found that the Agency's default reliance on outsourcing, without a balanced approach to 
in-house  development,  resulted  in  difficulties  in  delivering  tasks  related  to  the 
development of new systems (see previous question). 

37 

 
Nonetheless, the evaluation also found that the Agency made progress in building its in-
house capabilities to manage various tasks. Its recruitment efforts and HR processes have 
been digitalised, with initiatives like Agile Recruitment and video interviews. However, 
challenges remain, such as legal constraints on adopting new technologies and an inflexible 
organisational  structure.  At  the  same  time,  ongoing  projects  such  as  the  cultural 
transformation  and  employer  branding  show  eu-LISA's  dedication  to  improving  its 
capabilities and staying competitive in the IT field. 

The  evaluation  found  that  the  financial  planning  and  reporting  processes  showed 
improvement over the evaluation period.  

The  distribution  of  resources  and,  accordingly,  competences  and  responsibilities,  is 
allowed  between  IT  systems  in  relation  to  staff,  as  well  as  infrastructure  and  operating 
expenditures  (administrative  costs).  However,  the  operational  expenditures  –  by  far  the 
Agency’s largest budget item – are allocated according to systems.59 This is considered to 
be a deterrent factor for the distribution of resources across systems and, thus, a challenge 
to practically respond to ad hoc issues and urgent needs. Particularly noteworthy within 
this  context  is  the  role  of  e-CODEX  and  the  obligation  to  respect  the  principle  of  the 
independence of the judiciairy.60 Such a full separation of resources can lead to silo-based 
operational  approaches,  and  eventually  cause  inefficiencies  as  concerns  knowledge 
transfers within the Agency.  

The evaluation found that the increasing number of tasks of the Agency posed particular 
challenges  concerning  the  planning.  While  initially  eu-LISA  was  an  Agency  set  up  to 
manage three systems, it has become an Agency to manage and update three systems and, 
in  parallel,  develop  additional  new  ones  from  scratch.  While  the  appropriate  planning 
capabilities were eventually established, this led to gaps between the actual and planned 
progress over time. 

The  evaluation  pointed  to  the  need  for  eu-LISA  to  professionalise  its  accounting  and 
planning function through the implementation of appropriate tools.61 In this regard, multi-
annual  planning  is  an  area  for  improvement,  both  from  a  strategic  perspective  and  the 
concrete  tooling.  Over  the  previous  years,  emphasis  was  given  to  the  annual  planning 
which led in the Agency to ask for too many financial resources, which could, ultimately, 
not  be  used  entirely.  This  is  reflected  in  the  budgetary  absorption  rates  (see  Figure  4 
below).62 While particularly low absorption rates in the past stemmed from delays in the 

59 See eu-LISA, Annual Accounts, 2018-2024. 
60  Article  14(2)  of  the  e-CODEX  Regulation  provides  that  in  order  to  respect  the  principle  of  the 
independence of the judiciary “eu-LISA shall commit the resources provided to it for the e-CODEX system 
in  their  entirety  to  its  operation  under  the  Regulation  and  shall  ensure  the  involvement  of  justice 
representatives in the management of the e-CODEX system”. 
61  Study  to  support  the  evaluation  of  Regulation  (EU)  2018/1726  of  the  European  Parliament  and of  the 

Council of 14 November 2018 on eu-LISA. ISBN: 978-92-68-34401-9, section 2.3.3. 

62 Figure 4 shows the annual absorption rate as well as the amount of financial means received and spent 
from  2018  until  2024.  The  revenue  consists  of  subsidies  from  the  European  Commission,  financial 
contributions from associated countries and the transfer of assets. The expenses incorporate various costs 
such as fixed asset related expenses, staff expenses, external service provide expenses, or administrative 
and IT expenses. 

38 

 
 
development of the EES (and sub-sequent delays in the payment of contractors), over the 
last years, the Agency was able to improve its budgetary absorption.  

Figure 4: Evolution of budgetary absorption of eu-LISA, 2018-2024, in million EUR 

Source: eu-LISA, Annual Accounts, 2018-2024. 

Are  the  resources  (human,  financial  and  technical)  used  for  the  operational 
management  and  maintenance  of  existing  systems  adequate  in  comparison  with  the 
results achieved? Could the efficiency gains be obtained with a different approach to 
outsourcing or with different technology choices? 

The  evaluation  found  that  the  operational  management  and  maintenance  of  IT  systems 
remains  one  of  the  Agency’s  strong  areas.  eu-LISA  implemented  these  tasks  through  a 
combination of internal capacity and outsourced support (human and financial). Whereas 
the 24/7 management of incidents was done by internal staff, the operational support of 
users  is  outsourced  to  external  service  providers.  This  was,  in  principle,  considered 
adequate by the evaluation: while challenging and security-critical tasks were implemented 
by staff, the more routine tasks have been outsourced. 

As  there  only  very  few  incidents  occurred  during  the  evaluation  period,  with  existing 
systems  running  smoothly,  the  evaluation  assessed  whether  or  not  staffing  the  24/7 
operational  management  of  the  systems  with  internal  resources  was  the  most  efficient. 
While  such  activities  are  undoubtedly  security-critical,  they  can  also  become  routine, 
raising the issue of whether internal resources are always the best fit for these functions, 
particularly in light of challenges in recruiting for round-the-clock services and shortages 
of staff in system development. In this respect, it is key that the Agency carries out a clearer 
assessment of which tasks should be maintained intramuros and which ones can be more 
effectively externalised, so as to maximise the use of available resources. 

The  Final  audit  report  on  staff  recruitment,  management,  and  engagement  of  structural 
service  providers  and  organisational  ethics  in  eu-LISA,  prepared  by  the  European 
Commission Internal Audit Service in 2020 expressed similar concerns, noting that ‘the 

39 

 
 
 
Agency needs to improve the methodology on which management and stakeholders reach 
decisions  on  the  use  of  its  staff  versus  the  use  of  structural  service  providers  in  the 
production of its deliverables. The documentation of the justification of the current mix 
(staff versus structural service providers) is still weak. Compared to other agencies, EU-
LISA has a relatively high proportion of structural service providers, and they are not only 
used for temporary or ad-hoc duties but also for core operational and/or administrative 
tasks’.63 This remarks were also echoed in 2024 in the Final audit report on procurement 
and contract management in the European Union Agency for the Operational Management 
of Large-Scale IT Systems in the Area of Freedom, Security and Justice (eu-LISA), noting 
that,  in  relation  to  eu-LISA  2023  Consolidated  Annual  Activity  Report,  ‘weakness  still 
exist  in  the  sourcing  of  human  resources,  controls  over  procurement  and  contract 
management activities in the context of operational programmes and projects’.64  

As concerns the choice of technical resources, the evaluation found that eu-LISA did yet 
establish a clear strategy and roadmap for the implementation and practical use of artificial 
intelligence  within  the  operational  management  of  systems  (e.g.  concerning  incident 
detection and management, as well as the supportive communication with users through 
chatbots). Tasks were performed manually, blocking financial and human resources that 
could,  eventually,  be  better  used  elsewhere.  The  evaluation  therefore  found  that  the 
technology choices of eu-LISA were considered to be an important lever to increase the 
efficiency of the Agency in this regard. 

Are the resources (human, financial and technical) used for the development of new 
systems adequate in comparison with the results achieved? Could the efficiency gains 
be  obtained  with  a  different  approach  to  outsourcing  or  with  different  technology 
choices? 

In  view  of  the  delays  and  quality-related  issues  in  relation  of  the  delivery  of  the  new 
systems,  the  evaluation  considered  that  the  resources  used  for  the  development  of  new 
systems were only partially adequate. Efficiency gains could have been acquired through 
a larger share of insourcing development work and more proactive investment into new 
technologies. 

The development of new systems was largely outsourced over the past years. This has led 
to severe delays and quality concerns, in particular with regard to the delivery of EES. A 
significant portion of the Agency’s financial resources were spent on external contractors 
that were not fully able to deliver, while at the same time too few resources were spent of 
staff guiding contractors or co-developing systems.  

In addition,  technical  eu-LISA staff was mainly  in  charge of project  management tasks 
contractors rather than performing technical work: A job for which specific expertise and 
experience, as well as the right internal processes and contractual levers are needed in order 

63 Final audit report on staff recruitment, management and engagement of structural service providers and 
organisational  ethics  in  the  European  Agency  for  the  Operational  Management  of  Large-Scale  IT 
systems in the area of Freedom, Security and Justice (eu-LISA). 

64  Final  audit  report  on  procurement  and  contract  management  in  the  European  Union  Agency  for  the 
Operational Management of Large-Scale IT Systems in the Area of Freedom, Security and Justice (eu-
LISA). 

40 

 
 
to be effective. As a consequence, eu-LISA was not fully able to deliver on the task to 
develop new systems in alignment with the expectations of stakeholders. 

Therefore,  eu-LISA  could  have  been  more  efficient  with  regard  to  the  development  of 
systems  if  it  had  chosen  a  different  approach  to  outsourcing:  Rather  than  silo-based 
outsourcing or even using the layered approach under TEF/TOF (Transversal Engineering 
Framework/Transversal Operation Framework), better results could eventually have been 
achieved  through  increased  insourcing  of  resources  and  better  leveraging  the  technical 
expertise already available within eu-LISA. 

For this purpose, the evaluation identified two crucial factors for future improvement: The 
introduction of a portfolio management approach within eu-LISA, as well as the definition 
and execution of a long-term sourcing strategy (‘make and/or buy’). 

To what extent are procurement, finances, personnel and any other corporate services 
organised efficiently? Are resources commensurate with results achieved? 

The evaluation of the efficiency and organisation of procurement, finances, personnel, and 
other  corporate  services  within  eu-LISA  showed  a  blend  of  strengths  and  areas  for 
enhancement.  The  Agency's  heavy  reliance  on  external  resources  facilitated  the 
management  of  workload  demands  associated  with  both  legacy  and  planned  systems. 
However,  whether  resources  were  commensurate  with  the  results  achieved  presents  a 
mixed picture. 

The  evaluation  highlighted  the  importance  of  enhancing  compliance  and  strengthening 
procurement  processes.  Main  issues  identified  included  the  need  for  additional 
procurement  managers  to  handle  workload  demands  effectively,  for  more  detailed 
attention  to  the  technical  requirements  outlined  in  the  terms  of  references  to  ensure 
successful procurement outcomes, and for clearer processes to navigate the complexity of 
contractual management. Recommendations from various stakeholders included exploring 
creative procurement approaches, such as outcome-based contracts, to align with the fast-
paced nature of the technology market.  

However, despite ongoing efforts, the evaluation pointed to continuous challenges faced 
by eu-LISA in its procurement processes. Inconsistent stakeholder responsiveness and a 
lack  of  clearly  defined  processes  were  identified  as  challenging  factors.  For  instance, 
variations in  stakeholder engagement and input can lead to  delays and discrepancies in 
procurement  decisions.  Without  well-defined  processes  outlining  roles,  responsibilities, 
and timelines, a risk of inefficiencies and misunderstandings in the procurement workflow 
appears.  The  shift  towards  alternative  procurement  models,  such  as  agile  delivery,  was 
suggested to address these challenges and increase control over service delivery.65 

The evaluation pointed to the need for additional human resources in certain departments, 
particularly in activities related to the Operations and Corporate Services Departments.66 
Reporting  discrepancies,  extreme  dependency  on  contractors,  and  challenges  in 
recruitment and retention were found by the evaluation to contribute to resource shortages. 

65 Deloitte (2024): Study on the Operational Management of eu-LISA: Preparation for the Evaluation of the 

Agency. 

66 Deloitte (2024): Study on the Operational Management of eu-LISA: Preparation for the Evaluation of the 

Agency. 

41 

 
 
These challenges can be effectively addressed through a more strategic and integrated use 
of existing resources. 

The evaluation also found that delays and challenges in project delivery are an indication 
of  potential  inefficiencies  in  resource  allocation  and  management.  Furthermore, 
complexities within the organisational  structure  and governance processes highlight the 
need  for  streamlining  operations  to  enhance  resource  efficiency.  To  address  these 
disparities  effectively,  resource  distribution  strategies  could  be  reassessed  to  ensure 
optimal  outcomes.  Overall,  the  evaluation  finds  out  that  that  eu-LISA  should  optimise 
internal  resource  allocation,  avoid  over-reliance  on  external  partners,  and  maintain 
integrated oversight of project development rather than leaving different aspects to external 
service providers. 

In  this  context,  challenges  related  to  the  organisational  split  between  Tallinn  and 
Strasbourg  were  also  noted  in  the  evaluation  as  a  factor  impacting  collaboration, 
communication,  and  operational  efficiency.  The  perceived  disconnection  between 
corporate  services  and  operations  highlights  the  need  for  improved  clarity  of  roles  and 
responsibilities and closer alignment between units.  

Additionally, several interviewees expressed concerns about staffing levels and contractual 
arrangements, which underscores the importance of addressing recruitment and retention 
challenges,  while adapting to  evolving work preferences, including remote and flexible 
working arrangements. Stakeholders noted that the Agency tends to keep the individual 
allocation of resources and personnel tied to systems rather than managing projects in a 
more integrated and coordinated manner.  

Overall, while eu-LISA demonstrated transparency through its published information and 
efforts  to  strengthen  procurement  and  compliance,  whether  these  resources  were 
commensurate  with  results  achieved  present  a  mixed  picture.  Strategic  resource 
management  should  emphasise  optimal  internal  use,  coordination  across  systems,  and 
retaining  control  over  critical  project  aspects  to  ensure  efficiency  and  effectiveness. 
improving 
Additional,  addressing  concerns  about  claimed 
organisational  effectiveness,  and  enhancing  collaboration  between  units  are  critical 
priorities for optimising efficiency and achieving desired outcomes. This  is particularly 
important for units such as the Corporate Services Unit and Program and Project Delivery 
Unit, ensuring that allocated resources align effectively with the Agency’s objectives.. 

resource  shortages, 

To what extent has the tripartite set-up of eu-LISA affected the efficiency of the use of 
resources by the Agency? How does eu-LISA's split location affect its activities? How 
has eu-LISA ensured efficient working on three separate locations? 

The evaluation did not find the split-up of the Agency to be a major hindrance in terms of 
resource  efficiency.  The  evaluation  noted  that  the  split-up  can  lead  to  additional 
organisational  costs,  given  the  need  for  frequent  travelling  and  the  establishment  of 
additional coordination activities, and increased costs for maintenance and office rents.  

The evaluation found that, overall, eu-LISA has found a way to efficiently handle the split-
up  of  its  locations.  The  separation  of  tasks  reduced  the  need  for  administrative 

42 

 
coordination, allowing for the processing of tasks largely within a certain location. Teams 
are rarely split-up, ensuring connectivity of the team members.  

In  summary,  while  the  evaluation  found  the  Agency’s  efficiency  is  not  significantly 
impacted by the split over different locations, improvements in terms of additional teaming 
activities and better online coordination tools could be implemented to mitigate for certain 
effects of the tripartite location of the Agency.  

Having regards to the organigram of the Agency, the distribution of competences, and 
the  sharing  of  responsibilities  between  the  different  entities,  to  what  extent  does  the 
internal  organisation  of  the  Agency  impact  the  efficiency  of  eu-LISA  to  reach  its 
objectives? 

The  evaluation  found  that  eu-LISA’s  internal  organisation  as  satisfactory,  with  a  high 
approval  rate  by  stakeholders  of  the  efficiency  of  the  internal  structures,  escalation 
procedures and decision-making procedures. The evaluation pointed at two aspects of the 
Agency’s  internal  organisation  which  could  be  improved:  (i)  the  distribution  of 
responsibilities and competencies and (ii) the teaming and coordination between the two 
locations in Tallinn and Strasbourg.  

The evaluation pointed to unclear roles and overlapping responsibilities and competencies 
among  different  units  and  sectors  within  the  Agency,  leading  to  inefficiencies  and 
coordination issues. Despite having bottom-up descriptions, there was a lack of clear top-
down  strategic  definitions  of  roles  and  responsibilities  in  several  departments.  For 
example, three separate units have governance bodies within their responsibilities, with 
some overlap between the executive support and stakeholder relations unit and the liaison 
office in Brussels. In addition, multiple entities are responsible for programme and project 
management, creating potential for overlap and the need to clarify these roles.  Some of 
these overlaps and role ambiguities have been addressed and clarified over the course of 
the evaluation period, contributing to improved coordination and operational clarity. 

The evaluation also pointed to the loss of information that naturally occurs in coordination 
activities between the two main sites in Strasbourg and Tallinn. This concerned a perceived 
disconnect between operations (France) and both the HR and procurement units (Estonia). 
This  has  led  to  some  inefficiencies  in  hiring  procedures  (e.g.  length  of  procedures), 
development  of  staff  (e.g.  contract  prolongation),  and  management  of  contractors  (e.g. 
responsibility to select, manage, and control contractors). Moreover, the evaluation also 
noted that Strasbourg currently hosts more junior staff whereas the more  senior staff is 
located  in  Tallinn,  leading  to  an  unequal  distribution  of  knowledge  within  the  Agency. 
However, despite certain hindrances, the overall cooperation between the two sites in terms 
of internal organisation is found overall to be satisfactory. 

To what extent were the annual budgets of the Agency implemented in an efficient way 
and in view on achieving results? 

The  Agency’s  budget  is  structured  in  three  larger  clusters:  (1)  staff  expenditure;  (2) 
infrastructure and operating expenditures; and (3) operational expenditure, as illustrated in 
Figure 5. Between 2018 and 2024, eu-LISA’s budget increased twofold, from EUR 152 

43 

 
million to EUR 307 million (compound annual growth rate of +13.8% annually)67. For the 
financial  year  2024,  staff  expenditure  accounted  for  15%  of  the  Agency’s  budget, 
infrastructure and operating expenditure accounting for 8%, and operational expenditure 
accounting for 77%. This distribution indicates a strong emphasis on operational activities, 
reflecting  the  Agency’s  priority  to  enhance  IT  systems  and  support  functions  while 
maintaining relatively lower administrative and infrastructure costs. 

Figure 5: Evolution of eu-LISA’s budget from 2018 to 2024, in million EUR 

Source: eu-LISA, Annual Accounts, 2018-2024. 

The analysis of specific budget items from 2018 to 2024 (illustrated in Figure 6) indicates 
that expenditures on: 

- 

IT  system development increased  by  661%, reaching  EUR  76.1 million in  2024. 
This substantial increase aligns with the Agency’s expanded mandate to develop 
IT  systems.  However,  this  increase  should  be  read  together  with  the  delays  and 
quality  issues  in  output  (explained  in  the  effectiveness  section),  indicating 
inefficiencies in this area;   

-  Operational  management  increased  by  362%,  reflecting  an  EUR  167  million 
increase  by  2024  compared  to  2018.  Despite  this  relative  increase,  eu-LISA 
demonstrated  successful  contributions  in  operational  management,  suggesting 
more efficient resource-use in this field. 

-  System  security  and  business  continuity,  part  of  the  operational  management 
ledger,  increased  by  17%,  while  shared  system  infrastructure  expenditure 
decreased by 297%, totalling EUR 3.39 million in 2024. 

As regards staff (Figure 7), staff expenditure increased by 160%, an increase consistent 
with  the  increase  in  staff  numbers.  In  2024,  eu-LISA  had  404  staff,  translating  to  an 
average cost of EUR 104 207 per staff member.  

67 eu-LISA (2024), Annual accounts (https://www.eulisa.europa.eu/about-us/finance).  

44 

 
 
 
Figure 6: Evolution of eu-LISA’s expenditure on specific items from 2018 to 2024, in 
million EUR 

Source: eu-LISA, Annual Accounts, 2018-2024 

Figure 7: Evolution of eu-LISA’s staff and staff expenditure, 2018 to 2024, in million 
EUR 

Source: eu-LISA, Annual Accounts, 2018-2024 

45 

 
 
 
 
 
 
 
 
 
 
Despite  high  budget  implementation  rates  (100%  for  commitment  appropriations  and 
89.7% for payment appropriations in  2024), the efficiency of budget allocation remains 
questionable  due  to  the  disparity  between  expenditures  and  results,  particularly  in  new 
system  development.  The  increased  operational  budget  was  better  utilised,  reflecting 
successful  contributions,  and  administrative  and  infrastructure  expenditures  remained 
relatively low compared to other EU agencies, suggesting efficient resource management 
in these areas. However, bureaucratic barriers delayed budget allocation, leading to rapid 
and inefficient spending towards the end of budgetary years, indicating a need for more 
flexible and timely budgeting processes. 

The  evaluation  also  pointed  out  to  the  time-consuming  nature  of  the  annual  budget 
allocation process, which often led to delayed and rushed spending. More importantly, the 
evaluation found that the Agency lacks comprehensive multi-annual planning capabilities, 
hampering efficient budget utilisation, with stakeholders noting the need for more strategic 
and less compliance-centric budget management. The iterative nature of IT development 
requires flexible resource availability, often constrained by rigid budget structures. Legal 
and structural constraints, such as the e-CODEX Regulation requiring full separation of 
resources, limit the Agency’s agility and contribute to inefficiencies. 

In conclusion, while eu-LISA demonstrated adherence to EU budget rules and displayed 
financial diligence, there a is need to enhance its financial planning processes to ensure 
efficient  implementation  of  annual  budgets  aligned  with  achieving  results.  The  Agency 
would need to develop a comprehensive multi-annual overview and planning capability 
across its entire portfolio of tasks and services to establish a balanced scorecard, portfolio 
and pipeline management. A shift towards more strategic and flexible planning will better 
utilise resources, support overcoming bureaucratic barriers, enhance budget efficiency and 
maximising the Agency's impact. 

Having  regards  to  the  Human  Resource  strategy,  are  the  allocation  of  positions 
especially  managerial  as  well  as  secretarial  positions  and  the  distribution  of  projects 
within the Agency’s staff in line with the Agency’s objectives? 

The evaluation found there was no HR strategy in place by the Agency. eu-LISA used the 
Single Programming Documents as well as the overall eu-LISA strategy as guidelines on 
this matter. The evaluation pointed to the lack of capacities and resources allocation for 
the establishment of such strategy. 

46 

 
Figure 8: Evolution of posts and vacancy rate (2017-2024) 

Source: eu-LISA, Annual Accounts, 2018-2024 

The substantial expansion of the Agency’s mandate was accompanied by a sharp increase 
in  the posts  available (+46% from  2018 to  2019)68. In a similar fashion, the number of 
posts has increased steadily since then and more than doubled compared to 2018. At the 
same  time,  however,  from  2019  to  2023,  eu-LISA  faced  challenges  concerning  the 
recruitment, retention, and development of human resources, which has led a continuously 
high number and proportion of vacancies (reaching a high of 18,2% in 2022). 

The distribution of tasks within the Agency referred to the overall tasks that needed to be 
performed in order to meet the overarching objectives. This included, among others, the 
management of the large-scale IT systems, the development of new systems, or conducting 
relevant research according to the scope of the mandate. Hence, a clear HR strategy would 
better align needs and capacities of and within the Agency, decrease individual workload, 
and improve the hiring of suitable staff. 

In the past, the Agency adopted a delivery model where staff was primarily responsible for 
overseeing  and  ensuring  the  delivery  of  contractors'  work  (see  also  section  above 
concerning  outsourcing).  Concerns  have  been  voiced  about  the  potential  dilution  of 
responsibilities, the creation of silos, the lack of accountability and a diminishing focus on 
in-house  development  and  subsequently  the  knowledge-base.  Hence,  according  to 
stakeholders from within the Agency, knowledge management became a challenge, with 
eu-LISA staff being overloaded with contract management responsibilities and in addition 
to risking the loss of institutional knowledge. 

The  evaluation  found  that  Agency  performed  very  well  with  regard  to  the  operational 
management of existing IT systems, yet it struggled concerning the development of new 
systems, including due to an inefficient use and management of contractors.  

The  evaluation  found  that  the  Agency  did  not  approach  the  management  of  human 
resources from a sufficiently strategic perspective. During the previous growth phase of 
the Agency, staff was hired to fill an ‘ever-expanding’ number of vacant positions. The 

68 eu-LISA, Staffing situation of eu-LISA, 2021-369. 

47 

 
 
 
 
challenges related to the development of new IT systems showed that eu-LISA did not yet 
possess clear strategic visibility about its human resources’ pipeline and needs. 

Hence a holistic HR strategy, as well as the appropriate tooling with which the Agency 
could effectively plan, prioritise, and steer the allocation of its staff holistically according 
to functions, roles and profiles in view of existing and future business needs, rather than 
positions, is missing. 

The evaluation also found potential for improvement with regard to the Agency’s internal 
knowledge  management.  To  enhance  knowledge  management  within  the  Agency, 
implementing  a  job  rotation  programme  could  prove  beneficial.  Such  initiative  would 
entail periodically rotating staff across diverse roles, systems, or domains, addressing the 
issue of knowledge silos and fostering knowledge transfer. By exposing staff to various 
facets of the Agency's operations, this approach would cultivate a broader perspective and 
would equip staff to address technical challenges from multiple angles. Furthermore, job 
rotation would present new challenges and avenues for skill development. 

Lastly,  a  critical  step  in  developing  the  Agency’s  knowledge  management  would  be  a 
review and assessment of existing knowledge management processes, thus implementing 
a unified Knowledge Management strategy. Such an approach would streamline efforts, 
mitigate  duplication  of  work,  and  ensure  effective  capture,  sharing,  and  utilisation  of 
valuable knowledge throughout the organisation.  

Is the Agency’s Human Resources strategy adequate to attract and retain high-skilled 
profiles in the context of the IT labour market? 

The evaluation pointed to  a lack of a dedicated  HR  strategy document or roadmap  and 
assessed the existing challenges with regard to IT talent attraction and retention stemming 
from this gap. The evaluation identified various challenges in relation to the recruitment 
and staff retention: 

•  Strategic factors: Availability of positions; Competition with private industry; 
Purpose and mission of the Agency (i.e. its branding); Location and facilities 
of the Agency; 

•  Contractual  factors:  Necessity  of  security  clearance;  the  need  of  short-term 
contracts; Salary and compensation package. These issues, however, could be 
mitigated through strengthened human resource management, including more 
strategic planning and resource allocation practices; 

•  Operational  considerations:  Ability  to  work  with  cutting-edge  technology; 

Rigid ways of working. 

In addition, many stakeholders from within the Agency pointed towards process-related, 
contractual,  and  operational  challenges,  for  example  the  bureaucratic  nature  and  undue 
length of the hiring processes, types of contracts available to applicants, and the Agency’s 
limited ability to offer working with cutting-edge technology stack, in particular AI. 

While the evaluation found that the Agency performed well with regard to the operational 
management  of  IT  systems,  it  faced  challenges  concerning  the  development  of  new 
systems  (see  previous  question),  which  could  eventually  have  been  avoided  with  more 
strategic workforce planning. The Agency’s adoption of a delivery model where staff was 

48 

 
primarily responsible for overseeing and ensuring the delivery of contractors' work led to 
knowledge management challenges and had negatively impacted staff retention. 

The evaluation found that eu-LISA did not approach the management of human resources, 
including of highly-skilled profiles, from a strategic perspective. A clear strategic visibility 
about  the  human  resource  pipeline  was  missing,  as  the  recruitment  was  still  being 
performed rather ‘position by position’ in order to deliver as effectively and efficiently as 
possible on its tasks while simultaneously avoiding shortages. 

To  what  extent  does  the  internal  organisation  of  eu-LISA  allow  and  provide  the 
distribution  of  resources,  competences  and  sharing  of  responsibilities  between  IT 
systems? 

The  evaluation  found  that  the  internal  organisation  of  eu-LISA  should  allow  for  the 
distribution of resources, competences and sharing of responsibilities between IT systems. 
In practice, however, the existing structures within the Agency have led to team allocations 
specifically tailored for large implementation initiatives. While this approach enabled deep 
specialisation and holds the promise of optimal IT delivery and improved time to market, 
it  also  created  knowledge  silos  and  posed  risks  in  terms  of  sharing  expertise.  This  is 
particularly  relevant  given  that  one  of  the  Agency’s  core  tasks  involved  ensuring 
interoperability across large-scale IT systems, yet its internal structure continue to reflect 
a silo-based approach. Additionally, heavy reliance on outsourcing further aggravated the 
issue,  as  it  increased  the  risk  of  lack  of  knowledge-sharing  and  created  dependence  on 
external vendors. At this moment, the Agency depends on essential Subject Matter Experts 
(SME) and various documents to facilitate the sharing of expertise among the teams. 

Enhancing eu-LISA's flexibility implies a conscious shift towards a more interconnected, 
agile  and  fluid  organisational  structure.  It  is  important  to  mention  that  the  Agency  is 
already  adopting  agile  practices,  diminishing  its  dependence  on  traditional  waterfall 
project management, which will further enhance teamwork and the exchange of expertise 
across large-scale programmes. As a consequence, eu-LISA has already introduced and is 
currently scaling the use of more agile squads and waves in order to break up existing silos. 

Although operational work in relation to existing systems and systems under development 
is highly specific to each of the systems, there are also horizontal aspects that need to be 
aligned across systems. On the one hand, eu-LISA ensures the proper implementation of 
such aspects through allowing functional / horizontal procurement as part of TEF / TOF 
(Transversal  Engineering  Framework/Transversal  Operation  Framework).  On  the  other 
hand, there seems to be only a practical separation between existing systems and systems 
under  management.  Eu-LISA  staff  only  being  able  to  work  on  one  or  the  other  makes 
sense, given that different skills, expertise and experience are required in relation to both 
of them. 

However, with e-CODEX, there is also an exception: The e-CODEX Regulation specifies 
that resources devoted to e-CODEX may only conduct work in relation to this system and 
not  to  others.  This  is  augmented  through  the  e-CODEX  Regulation  specifying  that  the 
operational management of e-CODEX shall take place in Tallinn (which is not consistent 
with eu-LISA’s organisational set-up, as the IT facilities and relevant staff are based in 
Strasbourg). Hence, does not allow eu-LISA to apply its tested and proven business model 

49 

 
but  necessitates  changes,  both  in  the  organisational  structure,  processes  (including 
recruitment and budgeting). 

Given that the human and financial resources allocated to e-CODEX remain comparatively 
small,  these  organisational  restrictions  could  create  limitations  for  maximising  the 
system’s  potential  and,  more  broadly,  for  enabling  the  Agency  to  fully  deliver  on  its 
objectives of increased effectiveness and efficiency.  

Is the technology used by the Agency the most suitable to deliver its tasks? Is the Agency 
using the most advanced technology? 

The evaluation of whether eu-LISA's technology is the most suitable for its tasks and if the 
Agency  is  using  the  most  advanced  technology  paints  a  nuanced  picture.  While  the 
evaluation noted eu-LISA's proactive approach in adopting cutting-edge technologies such 
as  cloud,  AI,  and  DevSecOps,  operational  constraints  and  regulatory  barriers  posed 
challenges to fully leveraging these innovations.  

In particular, the Agency’s mandate and the establishing Regulations of IT systems include 
legal  barriers  to  working  with  cutting-edge  technologies  such  as  cloud  computing, 
quantum computing, and AI. For example, Article 5 of the eu-LISA Regulation provides 
requirements to process all data on-premise for all operations, which limits the flexibility 
in  technology  choices  and  introduces  challenges  common  to  large-scale  IT  systems 
leveraging on-premise solutions.  

Despite  these  challenges,  there  is  recognition  of  eu-LISA's  pursuit  of  innovation  and 
operational  efficiency  by  key  actors  of  the  Agency,  who  see  untapped  potential  of 
innovation  that they are  currently  exploring.  For  example, the Agency is  exploring AI-
driven analytics for real-time monitoring and the integration of AI-based applications such 
as chatbots and LLMs (Large Language Models) for first-level support. 

Survey responses further provide insights into various aspects of eu-LISA's technological 
landscape.  According  to  the  majority  of  responses  (82%)  the  Agency  has  succeeded  or 
largely succeeded inusing the best available technology for core systems SIS and VIS.69 
Additionally, there is a recognition of the importance of keeping up with new technologies, 
increasing  research  and  pilot  testing  capabilities,  implementing  stronger  feedback  and 
cooperation  mechanisms,  and  addressing  new  needs  such  as  AI  for  identification  and 
biometric  data,  cloud  services,  cybersecurity,  agile  workflow,  communication,  and  test 
environment improvements (e.g. test automation).  

To  bridge  the  gap  between  aspirations  and  operational  realities,  the  stakeholder 
consultations for the evaluation period reveals that the Agency should invest in strategic 
areas such as infrastructure maintenance and discuss a change of the mandate with DG 
HOME  regarding  the  usage  of  sovereign  cloud  solutions.  By  enhancing  infrastructure 
maintenance planning, investing in cloud technologies, and cultivating cloud-native teams, 
the Agency may unlock the full potential of advanced technologies to enhance operational 
efficiency and effectiveness, which remained an untapped potential in the past. 

69  Study  to  support  the  evaluation  of  Regulation  (EU)  2018/1726  of  the  European  Parliament  and of  the 

Council of 14 November 2018 on eu-LISA. ISBN: 978-92-68-34401-9, section 5.1.3.13. 

50 

 
 
How does the tripartite set up of the Agency affect the allocation of Human Resources?   

The evaluation assessed in particular how human resources of the Agency were allocated 
within eu-LISA and how the tasks differ between the different locations.  

The evaluation pointed to a general understanding for and acceptance of the decision to 
locate the Agency’s sites in three Member States (with the liaison office being established 
in Belgium, a fourth Member State). Staff distribution across sites as of 31 December 2022 
differs significantly, with the largest share of staff working in Strasbourg (73%), followed 
by Tallinn at 26%, while Brussels had a much lower share (1%)70. The tripartite set-up of 
the Agency mirrors the distribution of competencies within the Agency, with the site in 
Tallinn largely working on horizontal and administrative tasks, whereas the locations in 
Strasbourg hosting the technical staff and resources (e.g. central systems). The back-up 
site in St. Johann im Pongau is usually staffed on a rotating basis, with staff officially being 
located at different locations. The rest of staff is located in Brussels (of which most in the 
liaison office), working on publicity and lobbying-related tasks and ensuring the visibility 
of the Agency at the relevant institutions.71  

While the reasoning of the split and the staff allocation was not challenged, the evaluation 
noted  certain  practical  challenges,  as  the  split  of  human  resources  across  the  locations 
could be problematic for example in terms of team spirit and understanding of the tasks of 
the  staff  in  the  other  location.  The  most  frequently  mentioned  challenges  concerning 
Tallinn-Strasbourg  were:  (1)  Detachment  of  corporate  services  from  operations;  (2) 
Remoteness and practical reachability of both Tallinn and Strasbourg; and (3) Secluded 
location of the Central Unit and the rented office space in Strasbourg. The evaluation found 
that the location of the  Agency across separate  sites  mainly  have had an impact  on the 
hiring (e.g. length of procedures) and development of staff (e.g. contract prolongation), as 
well as the management of contractors.  

The  location  of  corporate  services  in  Tallinn  were  perceived  to  lead  to  a  lack  of 
understanding among staff based in Tallinn for the actual tasks delivered in Strasbourg, as 
well as the needs and practical challenges of the colleagues there. In addition, the location 
of control functions  in  Tallinn  (e.g. audit) posed certain  practical  challenges as there is 
only limited direct contact to those persons, processes, and data that should be controlled. 
While  the  collaboration  between  corporate  services  and  operations  was  generally 
considered  to  work  well,  including  through  frequent  (virtual)  exchanges  and  the  co-
location of some units and sectors at both sites, the physical distance between Strasbourg 
and  Tallinn  was  regarded  as  a  cause  of  inefficiencies  concerning  the  operational  and 
financial  planning,  demand  management  (e.g.  of  positions,  technical  infrastructure  and 
equipment),  recruitment  of  staff,  technical  implementation  of  the  Agency’s  tasks  (i.e. 
management and development of systems), and management of the contractors.  

As the Agency is located in Strasbourg and Tallinn, with its back-up site in Sankt Johann 
im  Pongau,  similar  concerns  were  voiced  with  regard  to  the  collaboration  and 
communication with the Commission based in Brussels. Specifically, a limited number of 
stakeholders from outside the Agency have highlighted the challenges stemming from the 

70 eu-LISA (2023): Internal document on the retrospective staff overview. 
71 Information based on 2023 note to the MB concerning a retrospective staff overview.  

51 

 
 
liaison office's role in Brussels, which, while originally intended to bridge policy-making 
in Brussels with the Agency's development in Strasbourg/Tallinn, has evolved into a more 
active  and  strategic  entity  (although  the  Agency  does  not  have  a  mandate  for  policy-
making). This expansion of eu-LISA’s activities, alongside overlaps with other units such 
as the executive support and stakeholder relations unit, has led to instances of inconsistent 
communication and increased coordination efforts, impacting operational efficiency. 

As concerns the split of the two technical sites – the Central Unit in Strasbourg and the 
back-up  site  in  St.  Johann  in  Pongau,  the  distance  between  both  sites  brought  about 
technical  challenges  with  regard  to  the  provision  of  uninterrupted  services,  particular 
concerning the  availability, connectivity, latency, maintenance, and regular back-ups of 
the systems.  

The evaluation found that the geographical location of the Agency’s sites posed challenges 
in terms of attracting and retaining staff. Tallin, in particular, was frequently highlighted 
by stakeholders as presenting difficulties for recruitment, while in Strasbourg, the Central 
Unit’s placement on the outskirts of the city was not seen as an asset for attracting talent. 

As a mitigation action, the evaluation pointed to the possibility for eu-LISA to consider 
drafting a fully-fledged Human Resources and Procurement Strategy for the future, where 
the above-mentioned aspects would find consideration.  

4.1.3.  Coherence  

Key findings 

The eu-LISA Regulation is coherent internally and with other relevant EU and national 
instruments. No challenges relating to the internal coherence of eu-LISA Regulation 
itself have been identified and only few challenges relating to the coherence of eu-LISA 
Regulation with other legal instruments have been found. 

This evaluation criterion examined the extent to which eu-LISA’s activities were aligned 
in an effective and efficient way with those of other stakeholders, including the Member 
States and the Schengen Associated Countries, the European Commission and other EU 
institutions  and  agencies,  to  ensure  coherence  and  avoid  duplication  of  efforts.  The 
evaluation also assessed the coherence of eu-LISA’s cooperation activities, management 
structure  and  activities,  whether  the  Agency’s  services  were  coherent  with  user’s 
expectations and needs, and the coherence of the Agency’s mandate with the other legal 
acts of the information systems it is responsible for. 

To what extent is eu-LISA acting in cooperation with the European Commission and 
other  EU  institutions  and  agencies,  as  well  as  with  Member  States  and  Schengen 
Associated Countries to ensure coherence and avoid duplication of efforts? 

Ensuring coherence and avoiding duplication is crucial to maintain the effectiveness and 
efficiency of eu-LISA, both in relation to the management of operational systems, as well 
as the development of new systems.  

52 

 
 
As  regards  the  systems  already  implemented,  the  evaluation  found  that  eu-LISA’s 
management of these systems contributes to a coordinated and coherent IT environment, 
with  limited  operational  cooperation  between  the  Agency  and  the  Commission  being 
necessary. However, as assessed in the efficiency section, the evaluation found challenges 
with regard to the strategic and budgetary planning and supervision of the Agency by the 
Commission. Users of the existing systems (Member States and SACs, other EU agencies) 
have expressed a high level of satisfaction with the Agency’s support and performance. 
An  area  for  improvement  identified  by  stakeholders  is  the  governance  of  the  agency, 
notably  relating  to  the  administrative  burden  the  governance  structure  entails,  the 
operation, cooperation and necessary expertise needed within AGs and PMBs (see section 
on efficiency on governance). The overall coherence of the governance structures of the 
Agency could be improved to further enhance the efficient management and operation of 
the Agency. 

As  regards  the  development  of  new  systems,  the  evaluation  found  that  the  Agency  has 
showed  a  constructive  collaboration  with  Member  States  by  being  responsive  to  the 
feedback and suggestions for improvement of the new systems. The evaluation pointed to 
the importance of strategic and operational cooperation with  the Member States, SACs, 
the Commission and other EU agencies to avoid gaps, duplications or grey areas in the 
future. 

However, the evaluation found that certain ambiguities persist regarding the division of 
responsibilities between eu-LISA and other EU institutions, agencies, Member States, and 
private sector actors (e.g. contractors). In particular, the respective roles of the Agency vis-
à-vis  the  Commission,  Frontex,  and  Member  States  are  not  always  clearly  delineated. 
These uncertainties are also reflected in the budgetary dimension, notably in relation to the 
Legislative Financial Statements (LFS). While the LFS are negotiated on a year-by-year 
basis,  stakeholders  underlined  the  importance  of  strengthening  the  Agency’s  internal 
planning for the use of the LFS, ensuring that resources are aligned with long-term strategic 
needs. The  evaluation  noted that eu-LISA  currently lacks a dedicated tool  or system  to 
effectively  support  multi-annual  planning  and  resource  allocation,  which  limits  the 
Agency’s  ability  to  anticipate  and  coordinate  needs  over  several  years.  Reliance  on 
traditional annual budgetary cycles further constrains the effectiveness of coordination and 
communication, particularly during the preparatory stages of the LFS. Introducing more 
structured multi-annual planning mechanisms would help enhance clarity, predictability, 
and efficiency of the Agency’s resource planning.  

To what extent are eu-LISA’s management structure and activities internally coherent 
for an efficient management and operation of the Agency? 

The management structure covers both the Agency’s governance structure, together with 
its  internal  organisational  set  up  (i.e.  departments,  units  and  sectors).  eu-LISA’s 
management structure consists of the Management Board (and various sub-bodies), nine 
Advisory  Groups72,  as  well  as  five  Project  Management  Boards.  In  addition,  the 

72 The API-PNR Advisory Group was established during the evaluation period. 

53 

 
 
Management  Board  secretariat  plays a central  role in  supporting the governance bodies 
and ensuring their smooth functioning. 

The evaluation found that the management structure is internally coherent and contributes 
to the Agency’s overall effectiveness and efficiency. In particular, the involvement of all 
stakeholders in multiple governance bodies was underlined as an important success factor 
by  Member  States,  as  the  responsibility  on  the  on  the  continuous  oversight  and 
improvement  of  the  respective  IT  system  should  lie  with  the  “community  of  Member 
States”. However, the numerous parallel governance bodies in which all Member States 
and EU agencies who are users of the systems are represented, brings its own complexities 
(e.g. high administrative burden), risks of duplications and contradictions, and may lead to 
shortcomings  relating  to  the  efficiency  of  the  governance  structures,  as  outlined  in  the 
efficiency section. 

The organisational set-up of the Agency evolved during the evaluation period. The current 
organigramme with its three departments, as well as sub-sequent 11 units and 35 sectors  
and three executive units directly reporting to the Executive Director was developed based 
on the Commission’s model in mind. While the organigram does not seem to have any 
obvious structural gaps, the evaluation found there is room for improvement with regard 
to the Agency’s organisational effectiveness and efficiency. For example, past situations 
were  noted  in  which  the  practical  roles  and  responsibilities  of  units  and  sectors  lacked 
clarity,  and  instances  occurred  where  multiple  units  worked  on  similar  tasks  without 
sufficient  coordination.  During  the  evaluation  period,  the  organigram  was  revised,  and 
several  of  the  shortcomings  previously  identified  have  been  addressed  to  some  extent; 
however,  some  inconsistencies  and  overlaps  remain,  indicating  the  need  for  continued 
attention to organisational alignment.  

In addition, stakeholders from within the Agency pointed out that the organisational split 
of the Agency between Tallinn and Strasbourg into corporate services and operations is 
bringing  about  challenges  (e.g.  a  perceived  disconnect  between  both  the  HR  and 
procurement units in Tallinn and the operations department in Strasbourg) which have led 
to inefficiencies concerning the hiring and development of staff, as well as the management 
of contractors. 

The evaluation found that, overall, the Agency’s management structure and activities are 
internally  coherent.  At  the  same  time,  opportunities  exist  to  further  enhance  the 
management  and  operational  effectiveness  of  the  Agency,  which  is  already  actively 
pursuing a transformation aimed at increasing efficiency.  

To what extent have eu-LISA’s services been coherent with the expectations of the users 
of large-scale IT-systems? How does the Agency’s service correspond to the needs and 
expectations of the Member States as users of the systems? 

Services provided by eu-LISA include,  among  others, the development  and operational 
management of large-scare IT systems in the area of freedom, security, and justice, creating 
an effective and coherent IT environment, or ensuring system continuity and security. The 
evaluation found that eu-LISA has met, to a large extent, users’ needs, including strong 

54 

 
system  availability,  incident  responsiveness,  or  operational  support,  with  the  provided 
large-scale IT systems within the scope of eu-LISA’s Regulation.  

The evaluation identified areas requiring further attention by the Agency, such as improved 
search  capabilities  and  interactivity  in  DubliNet,  and  addressing  issues  related  to  false 
positives, system scalability, interoperability, and search functionalities in Eurodac, VIS, 
and SIS. The need for enhanced pilot testing, stronger feedback mechanisms, and ensuring 
adequate  resources  and  system  security  to  meet  future  demands  were  other  areas 
highlighted by stakeholders. Some stakeholders have expressed concerns regarding system 
usability improvements and data quality.  

To  what  extent  are  procedures  in  place  to  ensure  the  coherence  of  eu-LISA’s 
cooperation activities with the policies and activities of its stakeholders? 

The strategic direction of eu-LISA is encapsulated in its 2021-2027 agenda73, emphasising 
the importance of coherent interactions with stakeholders to meet the operational needs of 
essential  systems  such  as  SIS  II,  VIS,  and  Eurodac,  while  also  undertaking  the 
development  of  new  systems  such  as  EES,  ETIAS,  and  ECRIS-TCN.  These  tasks  are 
projected to shape eu-LISA's role as a holistic IT service provider across the entire JHA 
landscape.  

The evaluation found that the procedures ensuring the coherence of eu-LISA’s cooperation 
activities with stakeholder policies and activities demonstrate a structured approach aimed 
at aligning with stakeholder expectations and operational needs.  

The  evaluation  showed  that  the  Agency  employed  a  multifaceted  approach  to  ensure 
alignment with both stakeholder policies and activities. This entailed engaging in day-to-
day  operational  activities  and  actively  participating  in  policy  dialogues  and  strategic 
planning exercises with  key stakeholders. By embedding itself within the policymaking 
process, eu-LISA can better anticipate and adapt to evolving policy frameworks, ensuring 
that  its  cooperation  activities  remain  coherent  with  stakeholder  policies.  An  enhanced 
coordination especially with the European Commission, is essential to ensure eu-LISA's 
activities  are  firmly  aligned  with  its  mandate74.  Additionally,  the  Agency  implements 
robust feedback mechanisms to gather insights from stakeholders, enabling it to fine-tune 
its  activities  in  line  with  their  expectations  and  requirements  through  the  governance 
bodies.  This  proactive  approach  not  only  enhanced  coherence  but  also  fosters  a 
collaborative environment conducive to achieving shared objectives.  

In addition, the evaluation found that the Agency’s ongoing efforts to enhance stakeholder 
management  strategies  and  improve  the  efficiency  of  service  delivery,  align  with  the 
Agency's strategic objectives and solidifying the coherence of eu-LISA's cooperations75. 

73 eu-LISA, 2021, eu-LISA Strategy 2021-2027, https://www.eulisa.europa.eu/Publications/Corporate/eu-

LISA%20Strategy%202021-2027.pdf.  

74 Government Office of Estonia and PWC, 2020, Future analysis Governance model for the European 

Union IT Agency (eu-LISA) 

75 Ibid. 

55 

 
 
To  what  extent  are  the  activities  of  the  Agency  as  well  as  the  scope  of  its  mandate 
coherent with the legal act governing the Information Systems the Agency is responsible 
for? 

Albeit not impacting negatively on the coherence, the evaluation found that the complexity 
of the legal framework in which eu-LISA operates presents a significant challenge. This 
framework comprises the various Regulations governing each of the large-scale IT systems 
managed by the Agency and the eu-LISA Regulation itself. Interpreting these Regulations, 
and  successfully  developing,  implementing,  and  operating  the  systems  within  the 
accompanying regulatory, budgetary, and organisational  constraints, has  proven to  be a 
demanding task. 

The evaluation  has  not  found major incoherences between eu-LISA Regulation  and the 
Regulations governing the systems the Agency is responsible for. One exception relates to 
the e-CODEZ Regulation, which presents certain specific features that merit attention:  

(1)  The  e-CODEX  Regulation  specifies  that  the  operational  management  of  e-
CODEX shall take place in Tallinn76. While this provision is intended to safeguard 
the  respect  of  the  principle  of  independence  of  the  judiciary,  it  is  not  entirely 
coherent  with  eu-LISA’s  organisational  set-up,  as  the  relevant  IT  facilities  and 
staff are based in Strasbourg. 

(2)  The e-CODEX PMB is institutionalised as a permanent body with members that 
are not part of the MB. This arrangement differs from the other system’s PMB and 
may require additional efforts to  align priorities and decisions between the MB 
and the e-CODEX PMB. Moreover, as e-CODEX PMB members are not part of 
the  MB,  they  do  not  always  have  an  overarching  view  of  the  Agency’s 
management and priorities.   

(3)  The e-CODEX Regulation specifies that resources dedicated to e-CODEX should 
focus exclusively on this system. This creates constraints for the Agency’s overall 
resource management. In practice, this approach may limit eu-LISA’s capacity to 
mobilise additional specialised expertise for the management of e-CODEX when 
needed, thereby reducing flexibility and potentially affecting the efficiency of its 
operational management. 

4.2.  How did the EU intervention make a difference and to whom? 

Key findings 

The added value of the eu-LISA Regulation compared to what Member States could have 
achieved alone has clearly been confirmed in this evaluation. The main conclusion is that 
without a single body such as eu-LISA responsible for several IT systems in the area of 
freedom,  security  and  justice,  the  current  IT  landscape  would  be  drastically  more 
fragmented and it would likely be more challenging to ensure security.  

This section assesses the extent to which the Agency’s mandate has brought added value 
and benefits compared with those that Member States would have achieved alone. 

76 Article 14(2) of Regulation (EU) 2022/850 of the European Parliament and of the Council of 30 May 2022 
on a  computerised  system  for  the  cross-border  electronic  exchange  of  data  in  the  area  of  judicial 
cooperation  in  civil  and  criminal  matters  (e-CODEX system),  and amending  Regulation  (EU) 
2018/1726 (OJ L 150/1, 01.06.2012). 

56 

 
 
Could  the  objectives  set  up  in  eu-LISA  Regulation  have  been  achieved  without  EU 
intervention? 

The evaluation has found that the Agency is playing a crucial role in contributing to high 
levels of security within the area of freedom, security and justice and that such degree of 
security could have not been achieved without a central EU Agency. The systems that are 
centrally managed by eu-LISA are the precondition for the existence of the Schengen area 
and are needed to ensure that the external Schengen borders, as well as the free movement 
of citizens and objects within Schengen, all take place within a uniformly secure, lawful 
and well-coordinated manner. 

The evaluation confirmed that the development and management of systems only at the 
national  level  by  Member  States  and/or  by  Schengen  Associated  Countries  would  be  a 
considerably  less  cost-effective  approach,  and  would  highly  increase  the  need  for 
coordination activities between Member States/Schengen Associated Countries, which is 
prone  to  misunderstandings  and  likely  creating  additional  administrative  and  financial 
burden. In addition, the level of information exchanges between competent authorities of 
the  Member  States  would  have  likely  been  lower,  encroaching  on  Member  States’ 
cooperation.  This  would  lead  to  a  more  fragmented  IT  landscape  and  more  prone  to 
security  challenges,  increasing  the  risks  of  unlawful  Schengen  border  crossings  or 
complicating the search for criminals or missed persons within the Schengen area. 

However,  the  evaluation  underlined  the  need  for  the  Agency  to  remain  flexible  and 
adaptive,  in  order  to  continue  to  fulfil  its  mandate  in  accordance  with  the  objectives 
underpinning its existence. This would include, amongst others, the strengthening of the 
tasks in research monitoring and development function to contribute to the development 
of IT-based solutions in the area of freedom, security and justice. In this respect, eu-LISA 
should not carry out pure research activities, but rather strengthen its capacity to monitor 
technological  and  research  developments,  establishing  a  robust  technological  watch 
function, which will allow them to identify emerging solution that may be relevant for the 
systems it manages. This would enable the Agency to ensure that innovative technologies 
are not only identified at an early stage, but also assessed in terms of their applicability and 
integrated swiftly and efficiently into the systems where relevant. In parallel, enhancing 
cooperation  with  third-country  actors  and  international  organisations,  as  well  as  further 
refining its internal organisation, will allow eu-LISA to remain efficient and effective.  

How has the new mandate of the Agency contributed to supporting Member States in 
developing the information systems at national level? 

Concerning  the  effects  of  Article  16  of  the  2018  eu-LISA  Regulation,  concerning  the 
advice, requests of ad hoc support, or the development of IT components to assist Member 
States  in  implementing  technical  aspects  of  obligations  deriving  from  Union  law,  the 
findings of the evaluation can be read in the relevance section. In particular, the evaluation 
found  that  the  the  requesting  procedure  for  IT-components  under  Article  16(4)  of  the 
current Regulation is rather lengthy and complicated, which has led to the Article not being 
invoked.  In  addition,  as  to  the  hand-in  of  the  reports,  no  cases  of  support  for  testing 
procedures  on  the  MS-level  were  known,  also  pointing  to  significant  room  for 
improvement. 

57 

 
4.3. 

Is the intervention still relevant? 

Key findings 

The continued overall relevance of eu-LISA’s mandate and objectives has been confirmed 
in this evaluation. Indeed, the relevance of the Agency has become even more important, 
considering external developments such as the increased mobility of EU citizens and third-
country nationals, technological developments and increased risks from cyber-attacks as 
well as threats from ongoing geopolitical developments and conflicts (e.g. in Ukraine and 
the Middle East). 

This evaluation criterion assessed the relationship between the needs and problems at the 
time of the adoption of the new mandate of the Agency and during its implementation, as 
well as the relationship between the current and future needs and problems in the EU.  

To what extent did the scope of the mandate and the objectives of eu-LISA Regulation 
remain relevant over the implementation period? 

The evaluation found that, overall, the scope of the mandate and the objectives of eu-LISA 
Regulation  (see  the  effectiveness  section  on  the  definitions  of  general  and  specific 
objectives) remained relevant. Throughout the evaluation period, the systems managed by 
eu-LISA functioned appropriately in view of political, social and other external challenges. 
However, the evaluation showed that the operational management of the systems could be 
improved to maintain a high relevance from an operational perspective. This concerned, 
for example, the extent to which the Agency was able to adapt its IT systems and operations 
to the technological progress. With particular regard to the development of new systems, 
the  evaluation  showed  that  the  Agency’s  mandate  also  stayed  relevant  across  the 
implementation period.  

While  certain  stakeholders  expressed  less  satisfaction  in  this  regard  compared  to  the 
operational management of existing systems, the Agency’s mandate remains nonetheless 
relevant.  In  particular  with  regard  to  existing  security  and  migratory  challenges,  the 
operational performance of the Agency is a key success factor for the Agency to remain 
relevant in the future. 

How did the objectives of eu-LISA Regulation correspond to wider EU policy goals 
and priorities? 

The wider EU policy goals and priorities considered by the evaluation concern those of the 
Union in the area of freedom, security and justice. The evaluation found that the general, 
specific  and  operational  policy  objectives  of  eu-LISA  Regulation  are  aligned  with  the 
wider EU policy goals and priorities in the area of freedom, security and justice.  

The eu-LISA Regulation and the specific establishing Regulations enable the Agency to 
develop and manage operational IT systems aimed at delivering on the specific objectives 
in these three areas of the area of freedom, security and justice, namely, border checks, 
asylum and immigration, police cooperation, and judicial cooperation in civil and criminal 
matters. While the IT systems concerning border checks, asylum and immigration were 
already mentioned in the 2018 eu-LISA Regulation, the systems concerning judicial and 

58 

 
police  cooperation  were  added  by  later  amendments  in  2019,  2022  and  2023  triggered 
through the systems’ establishing Regulations. 

The evaluation confirmed that the objectives of the eu-LISA Regulation positively enhance 
overarching goals of the EU in the area of freedom, security and justice. 

To what extent has eu-LISA adapted to the new needs and challenges, in particular to 
the technological or scientific progress since the adoption of eu-LISA Regulation? 

Since  the  adoption  of  the  new  eu-LISA  Regulation  in  2018,  new  needs  and  challenges 
emerged for the Agency.  This  included new technological  advancements  such as  cloud 
computing,  artificial  intelligence,  improved  cybersecurity  measures,  or  blockchain 
technology which the Agency has not integrated into its operations. On a broader scale, 
the  Agency  was  also  impacted  by  the  increase  in  global  migration  flows  that  affected 
Member States. 

The evaluation found that, as concerns the technological advancements, the Agency has 
adapted to the new technological  environment to  a moderate extent. On the other hand 
however, eu-LISA was able to deal to a large extent with the challenges to its activities as 
a  follow  up  to  the  increase  in  global  migration  flows.  For  this  purpose,  the  evaluation 
assessed the number of incidents for the respective large-scale IT systems. In regard to the 
number of incidents, SIS maintained an availability rate of 99.9% in 2022 with a response 
rate of 99.0%. The same stable picture holds for other large-scale IT systems that are linked 
to migration flows. 

Overall, the evaluation underlined the importance of intensifying research activities within 
the Agency and pilot testing to be able to address technological or scientific progress. New 
needs  and  challenges  identified  during  consultations  include  AI  for  identification, 
biometric data, more services for judiciary, cloud services, cybersecurity, agile workflow, 
and improvement regarding communication and test environments. This holds for potential 
challenges for DubliNet, Eurodac, SIS, and VIS, as these systems must be kept up to date 
as well as deal with increasing flows of migration.  

Has the nature of the work of eu-LISA experienced such a change that a change of 
mandate/ objectives is needed? 

Through the implementation of the revised Regulation since 2018, the Agency experienced 
a pressure to not only deliver on the operational management of existing systems but to 
develop new systems. The evaluation underlined specific challenges with regard to parts 
and the flexibility of the mandate (see effectiveness section).  

The evaluation looked into reasons both in favour and against a change in the mandate of 
the Agency. An update of the mandate could enhance the Agency's adaptability to evolving 
needs  and  technological  advancements,  potentially  fostering  innovation  and  efficiency. 
However,  a  change  in  the  mandate  could  bring  disruptions  to  ongoing  operations  at  a 
moment where stability is needed to fulfill current objectives. In addition, changes to the 
current  mandate  would  introduce  additional  complexities  and  administrative  burdens, 
potentially  diverting  resources  and  attention  away  from  critical  operational  priorities. 
Thus, a careful consideration and evaluation of the ‘pro and cons’ arguments are essential 

59 

 
in  determining  the  best  course  of  action  that  balances  adaptability  with  operational 
continuity.  

The  evaluation  also  noted  that  the  Agency  has  recognised  and  acknowledged  that  the 
change  of  the  nature  of  its  work  has  brought  about  challenges  and,  thus,  proactively 
initiated a transformation process. 

Has eu-LISA’s mandate provided the necessary flexibility to address unexpected 
challenges? 

The evaluation found that the overall mandate did not leave significant flexibility for the 
Agency to be entrusted with new tasks without amending the legal framework.  

While Article 1(5) and Article 9 of eu-LISA Regulation explicitly refer to the possibility 
for  eu-LISA  to  be  entrusted  with  the  preparation,  development  and  operational 
management of additional large-scale IT systems not yet referred to in the Regulation, this 
is only possible if ‘provided by relevant Union legal acts governing those systems’ (Article 
1.5  of  eu-LISA  Regulation).  In  practice,  this  triggers  an  amendment  of  eu-LISA 
Regulation, a procedure which has taken time based on the based on the timeline of an 
ordinary legislative procedure (around one to two years).  

Article  16(4)  could  present  another  potential  source  of  flexibility,  as  it  provides  the 
possibility for a group of at least five Member State to entrust the Agency with developing, 
managing or hosting a common IT component. This provision has not been used, based on 
its limited scope and rigid procedures. Therefore, the evaluation found that this provision 
has not contributed to the flexibility of the Agency in taking on additional tasks. 

The evaluation also found limitations relating to the flexibility in implementing existing 
tasks. In particular, the evaluation pointed to the to the legal framework of the  establishing 
Regulations of the individual IT systems as highly prescriptive with regard to the technical 
implementation of the IT systems, impacting the development and testing of the IT systems 
as well as the choice of potential IT solutions. While these separate Regulations are not the 
focus of this evaluation, these are relevant to consider to the extent to which they impact 
on eu-LISA’s activities and may interact with eu-LISA Regulation.  

For example, the establishing EES Regulation specified that the system must go live in all 
Member States  at the same time.77 As a consequence, eu-LISA was not able to test the 
system in a beta version, for example, at a small number of selected border points, which 
is common practice in comparable projects. However, it must be noted that in 2025, the 
European  Parliament  and  the  Council  adopted  Regulation  (EU)  2025/153478  enabling  a 
progressive start of operations of the EES, which allows Member States to start operating 
the system gradually. 

77 Article 66 of Regulation (EU) 2017/2226 of the European Parliament and of the Council of 30 November 
2017 establishing an Entry/Exit System (EES) to register entry and exit data and refusal of entry data of 
third-country  nationals  crossing  the  external  borders  of  the  Member  States  and  determining  the 
conditions  for  access  to  the  EES  for  law  enforcement  purposes,  and  amending  the  Convention 
implementing the Schengen Agreement and Regulations (EC) No 767/2008 and (EU) No 1077/2011, 
Official Journal of the European Union, L327/20, 09.12.2017 (EES-Regulation).  

78 Regulation (EU) 2025/1534 of the European Parliament and of the Council of 18 July 2025 on temporary 
derogations from certain provisions of Regulations (EU) 2017/2226 and (EU) 2016/399 as regards the 
progressive start of operations of the Entry/Exit System 

60 

 
 
In  addition,  there  are  requirements  in  establishing  Regulations  of  IT  systems  that  the 
systems  should  be hosted by eu-LISA in  its  technical  sites (e.g.  Article  7.2 of the EES 
Regulation79; Article 8 of the amending Regulation80). As a consequence, eu-LISA could 
not consider the use of cloud services, which could be more cost efficient and would save 
server capacity in eu-LISA’s technical site, decreasing pressure on eu-LISA’s data centres.  

With regard to specific unexpected challenges, the evaluation found that the Agency’s (in-
) ability to directly and effectively respond to the Ukraine refugee crisis within the range 
of its core expertise – IT development – is in large part due to the limited flexibility of its 
mandate: 1)  In a time of crisis,  eu-LISA’s mandate was not  flexible enough to  support 
within the range of its core expertise; and 2) the processes governing potentially amending 
eu-LISA’s  mandate  are  not  swift  enough  in  order  to  enable  the  Agency  to  provide 
operational support. As a result, despite eu-LISA being the IT agency of the EU, it was not 
able to effectively respond to one of the most challenging crises and security threats in 
recent history. 

To what extent has the Agency been able to undertake additional tasks established in 
legal acts adopted after the entry into force of eu-LISA Regulation? To what extent 
does the Agency’s mandate enable eu-LISA to undertake new responsibilities? 

As regards additional tasks eu-LISA established outside of the 2018 eu-LISA Regulation, 
the major block of additional tasks was delegated to eu-LISA by the establishment of new 
large-scale  IT  system  Regulations,  in  particular  for  ECRIS-TCN  (in  2019),  the  new  e-
CODEX  Regulation  (in  2022)  and  the  adoption  of  the  JITs  CP  Regulation  (in  2023). 
Within these Regulations, eu-LISA was transferred the responsibility for the development 
and operational management of the systems, as well as related tasks (such as e.g. ensuring 
data security, training on use, providing support services, etc.). The situation with the e-
CODEX Regulation was different, as it was an already established and operable system, 
managed  by  a  consortium  of  Member  States  and  organisations.  During  the  evaluation 
period, the takeover by eu-LISA was completed. eu-LISA will continue to further develop 
the system, including through technical updates and the integration of new components. 

In addition to the new systems’ Regulations, Regulation (EU) 2021/1134 on VIS and the 
Regulations  on  the  renewed  SIS  from  November  201881,  as  well  as  the  2019 
Interoperability  Regulation  (Regulation  (EU)  2019/817),  adding  new  functionalities  to 
existing systems, brought about significant changes in the form of additional tasks of the 
Agency. These tasks include in particular the preparation and development of the SIS’s 
and VIS’s new components and adding of additional access rights, as well as monitoring 
of research and development efforts with regard to interoperability components such as the 
Common  Identity  Repository  (CIR),  Common  Repository  for  Reporting  and  Statistics 

79 Article 66 of the EES-Regulation.  
80 Article 8 of Regulation (EU) 2021/1134 of the European Parliament and of the Council of 7 July 2021 
amending Regulations (EC) No 767/2008, (EC) No 810/2009, (EU) 2016/399, (EU) 2017/2226, (EU) 
2018/1860, (EU) 2018/1861, (EU) 2019/817 and (EU) 2019/1896 of the European Parliament and of the 
Council and repealing Council Decisions 2004/512/EC and 2008/633/JHA, for the purpose of reforming 
the  Visa  Information  System,  Official  Journal  of  the  European  Union,  L248/11,  13.07.2021  (VIS-
Regulation).  

81 Regulation (EU) 2018/1860, Official Journal of the European Union, L312, 07.12.2018; Regulation (EU) 
2018/1861, Official Journal of the European Union, L312, 07.12.2018; Regulation (EU) 2018/1862, 
Official Journal of the European Union, L312, 07.12.2018 

61 

 
 
(CRRS),  European  Search  Portal  (ESP),  Multiple  Identity  Detector  (MID)  and  shared 
Biometric Matching Service (sBMS). 

The evaluation found that, overall, the Agency has been able to successfully take on and 
manage the growing number and variety of tasks assigned to it. The Agency has established 
additional  governance  bodies  (such  as  AGs  and  PMBs)  concerned  with  preparing  and 
working  out  specifications  of  the  new  systems  and  has  slowly  and  steadily  progressed 
towards  the  implementation  of  the  systems  and  of  the  interoperability  components,  for 
which a separate AG has been established. As for other IT systems, external contractors 
have  been  included,  ensuring  that  all  necessary  technical  skills  are  represented  to 
implement the tasks. In particular, the evaluation found that the work in the governance 
bodies, eu-LISA’s professional and well-developed technical environment, as well as the 
Agency’s secure environment were major assets of the Agency. 

However, compared to the work on already operational systems, the evaluation found that 
the Agency need stronger management of external contractors, better planning and project 
management, as well as more comprehensive project and technical documentation in order 
to carry out these tasks successfully.  

However, challenges in relation to new tasks did not only stem from eu-LISA Regulation 
itself: although it may be possible undertake new responsibilities from a legal perspective, 
the evaluation found that the extent to which this is possible also depends on the practical 
constraints  of  the  Agency  concerning  its  operation  (budget,  positions,  prioritisation  of 
tasks,  available  technology,  timeframe  etc.).  Hence,  undertaking  new  responsibilities 
successfully is a matter of aligning legislation and its practical implementation. 

To what extent would the Agency be capable to play a role in the governance of the 
systems from a business perspective? 

The  expansion  of  the  Agency’s  mandate,  together  with  the  increasing  scope  and 
complexity of its tasks, has required eu-LISA to gradually evolve from a service provider 
into  an  organisation  that  addresses  the  broader  business  dimensions  of  large-scale  IT 
systems,  including  their  development,  implementation,  and  management  in  the  area  of 
freedom, security and justice. 

The evaluation found that under the current governance structure, the conduct of business 
analysis could benefit from further strengthening. Business analysis encompasses both the 
software development perspective of IT systems and the practical implementation of new 
systems at national level, taking into account the diverse conditions and capabilities across 
Member  States.  At  present,  the  Management  Board’s  responsibilities  do  not  extend  to 
certain business-related issues – such as staffing and financial resources linked to system 
development,  monitoring,  or  workflow  management  –  which  suggests  that  additional 
consideration in this area could be beneficial. 

In addition, the evaluation noted that the limited scope of business analysis also affects the 
monitoring  of  tasks  performed  by  external  contractors.  Given  the  Agency’s  significant 
reliance  on  contractors,  it  is  important  for  eu-LISA  to  maintain  strong  oversight  of  the 
delivery of their work. While contractors carry out essential functions, their participation 

62 

 
in  the  meetings  of  governance  bodies  has  so  far  been  limited,  which  may  reduce 
opportunities for alignment and information exchange. 

Overall, the evaluation suggests that, in light of the Agency’s expanded mandate, further 
reflection  on  business-related  aspects  of  development  projects  would  be  valuable.  A 
stronger business-oriented approach could help ensure that governance structures are fully 
aligned with the Agency’s evolving objectives and scope, thereby reinforcing control over 
operations and development processes. 

Is the current mandate and organisation of the Agency correctly reflecting the need to 
respect the principle of independence of the judiciary? 

eu-LISA Regulation states in its Article 19 that the Management Board shall “ensure that 
all  decisions  and  actions  of  the  Agency  affecting  large-scale  IT  systems  in  the  area  of 
freedom, security and justice respect to the principle of independence of the judiciary”.82  

The evaluation analysed instances related to eu-LISA’s work where this principle could be 
at  risk  or  was  compromised  and  found  that  eu-LISA  has  complied  this  principle.  The 
organisational setup of the Agency favours this principle of independence: the Agency's 
current structure, encompassing both management and governance as well as operational 
functions, ensures the judiciary's independence is upheld from a broad perspective as the 
Agency deals with the technical architecture and infrastructure of large-scale IT systems 
and does not interfere with transmitted data within the systems. In line with this principle, 
eu-LISA’s staff does not have access to the operational data contained in the systems. In 
addition, eu-LISA implemented security measures such as security clearance of staff and 
separating the individual systems on a technical level. In the context of e-CODEX, Article 
14 (2) of the establishing Regulation restricts the Agency to only commit resources for the 
IT system in its entirety. The Agency dedicated five staff to work exclusively on e-CODEX 
topics to further respect the principle of independence of the judiciary.  

82  Article  14  of  EU  Regulation  2022/850  emphasises  the  respect  of  the principle of  independence of  the 

judiciary in the context of e-CODEX. 

63 

 
 
 
5.  WHAT ARE THE CONCLUSIONS AND LESSONS LEARNED? 

5.1.  Conclusions 

Overall, despite a complex operational and policy environment—marked by accelerating 
digitalisation,  rising  security  threats,  and  geopolitical  instability—this  evaluation 
concludes with a broadly positive assessment of the performance, relevance, coherence, 
efficiency, and EU added value of eu-LISA. The Agency has proven indispensable for the 
implementation  and  operational  management  of  large-scale  IT  systems  in  the  area  of 
freedom, security and justice, supporting Member States and EU institutions in delivering 
coordinated, secure, and effective digital infrastructure. 

The  eu-LISA  Regulation  remains  highly  relevant  to  addressing  current  and  future 
challenges.  The  Agency’s  importance  has  increased  in  light  of  developments  such  as 
increased  mobility  of  EU  citizens  and  third-country  nationals,  rapid  technological 
advancements  heightened  cyber-attacks  risks,  and  ongoing  geopolitical  tensions, 
particularly those linked to the Russian war of aggression against Ukraine and the conflicts 
in the Middle East. The digitalisation of the area of freedom, security and justice constitutes 
a major objective of the European Union, and it is at the top of EU’s policy and political 
agendas, requiring a common and coordinated response by all Member States.  

The objectives of the eu-LISA Regulation could have not been achieved sufficiently 
by  Member  States  acting  alone.  The  current  mandate  of  eu-LISA,  as  introduced  by 
Regulation  (EU)  2018/1726,  plays  a  pivotal  role  in  advancing  the  Agency’s  mission  to 
“contribute  to  a  high-level  security  within  the  area  of  freedom,  security  and  justice, 
including by facilitating border management and law enforcement”. By centralising the 
operational  management of  multiple large-scale  IT systems,  eu-LISA not  only  supports 
Member States in their implementation efforts but also leverages significant economies of 
scale,  fostering  greater  efficiency  and  consistency  across  the  Union.  The  evaluation 
concludes that, while there is room for improvement in the development of new systems, 
the absence of an EU level Agency like eu-LISA would lead to a drastically more fragment 
IT landscape, compromising the security and interoperability essential to the justice and 
home affairs domains. Accordingly, the evaluation confirms the EU added value of the 
Regulation. 

The current eu-LISA Regulation is to a large extent legally coherent both internally 
and externally. Only a limited number of challenges have been identified in relation to its 
coherence with other legal acts, most notably concerning its interaction with the e-CODEX 
Regulation.  As  detailed  in  Section  4.1.3,  the  e-CODEX  Regulation  imposes  specific 
operational  constraints  -such  as  mandating  that  is  management  must  be  conducted 
exclusively  in  Tallinn,  and  restricting  the  resources  provided  for  the  e-CODEC  to  be 
committed in their entirety to that system, in order to provide for the respect of the principle 
of the independence of the judiciary.  These  provisions  hamper the efficient  operational 
management of the system by eu-LISA, showing areas where regulatory adjustments could 
enhance administrative flexibility and operational synergies.  

No challenges relating to the internal coherence of the eu-LISA Regulation were identified. 
The  evaluation  analysed  the  extent  to  which  the  current  structure  of  the  eu-LISA 

64 

 
Management  Board  is  coherent  with  safeguarding  the  respect  of  the  principle  of 
independence of the judiciary. eu-LISA is a technical Agency that is responsible for the 
management and development of IT systems, which means that it deals with the technical 
architecture  and  infrastructure,  but  eu-LISA’s  staff  does  not  have  access  to  the  data 
transmitted  via  the  systems  they  manage.  While  there  is  still  a  risk  that  there  might  be 
(criminal) incidents of infringements, the Agency has taken preventative measures. The 
evaluation concluded that, at this stage, it is not expected that the current organisational 
and governance set-up of eu-LISA would interfere with the respect of the principle of the 
independence of the judiciary.  

The assessment of eu-LISA’s effectiveness is mixed.  While eu-LISA performs well in 
the management of existing large-scale IT systems, with positive stakeholder feedback on 
service  continuity  and  incident  response,  there  is  an  important  scope  for  improvement 
concerning the development of new systems. 

The effectiveness of the eu-LISA Regulation in achieving its objectives differs between 
tasks: 

-  The operational management of existing systems is generally very effective, and it 

has been identified as a strength. 

-  The development of new systems is less effective, with the Agency encountering 
challenges related to, for example, contractor management, internal coordination, 
and  resource  allocation.  These  challenges  have  resulted  in  significant  delays, 
notably in relation to the EES, ETIAS, ECRIS-TCN, and revised VIS. 

-  The  effectiveness  of  the  Agency  in  conducting  tasks  that  are  not  related  to  the 
systems is mixed, with positive outputs in research monitoring, but the limited use 
of  pilot  testing  and  ad  hoc  support  to  the  European  Commission  and  Member 
States, as provided in Articles 15 and 16 of the current eu-LISA Regulation. 

With  its  achievements  relating  to  the  development,  management  and  operation  of  IT 
systems as well as to monitoring of research, piloting and testing, eu-LISA was able to 
contribute to a coordinated, cost-effective and coherent IT environment at Union level in 
the area of freedom, security and justice, the development of IT-based solutions in the area 
of freedom, security and justice as well as the implementation of evidence-based policy-
making in this field. Thereby, the Agency contributed to a high-level security within the 
area of freedom, security and justice. However, there is room for improvement based on 
the shortcomings relating to the development of the systems. 

In regard to the objective of contributing to a coordinated, cost-effective and coherent IT 
environment at Union level in the area of freedom, security and justice, eu-LISA is, to a 
large-extent, effective. The management of existing IT systems has been eu-LISA’s core 
strength over the past years. Stakeholders’ satisfaction with the performance of the Agency 
is high in particular with regard to the services the Agency offers concerning the system 
operated by the Agency, as well as their security. Evidence suggests that there is  small 
room for improvement with regard to operational requirements. 

eu-LISA’s effectiveness with regard to the development of IT-based solutions in the ASFJ 
is  positive  but  there  is  ample  room  for  improvement  for  the  Agency  as  a  whole,  in 

65 

 
particular with regard to the Agency’s overall operating model and strategy and concerning 
the organisation and resource management of the Agency. 

Finally,  the  effectiveness  with  regard  to  the  implementation  of  evidence-based  Union 
migration and security policy is limited. While the Agency is compiling and publishing 
relevant  statistics  and  produces  statistical  reports,  and  the  website  of  eu-LISA  contains 
links to relevant documentation about the Agency’s and the governance bodies’ activities, 
the  accessibility  and  user-friendliness  of  these  publications  is  limited.  This  negatively 
impacts on the extent to which the information can be used in practice and, therefore, the 
transparency of the Agency’s activities overall. 

The assessment of eu-LISA’s efficiency presents a nuanced picture While the Agency 
has demonstrated strong efficiency in the operational management of existing large-scale 
IT  systems,  there  remains  considerable  scope  for  improvement  in  the  development  and 
delivery  of  new  systems.  Challenges  in  this  regard  largely  stem  from  difficulties  in 
contractor management and difficulties adhering to timelines for major projects, such as 
the  EES  and  ETIAS.  From  an  organisational  and  governance  perspective,  eu-LISA 
operates  efficiently  and  continuously  strives  to  optimise  its  internal  processes. 
Nevertheless,  certain  structural  issues  have  been  identified  that  hinger  the  Agency’s 
optimal  performance,  for  instance,  the  rigid  unit-based  set-up,  oversized  teams  and 
overlapping responsibilities, particularly in stakeholder engagement.  

Additionally, eu-LISA’s operations are fragmented across different locations in Tallinn, 
Strasbourg  and  Sankt  Johann  im  Pongau.  These  factors  have,  at  times,  contributed  to 
inefficiencies, notably in contract management and oversight, as well as recruitment of the 
Agency’s  staff.  The  Agency’s  governance  framework  is  inherently  complex  and  can 
generate  substantial  administrative  overhead.  While  this  complexity  is  to  some  extent 
justified  by  the  need  to  ensure  robust  coordination  across  Member  States  and  EU 
institutions  concerns  have  been  raised  about  the  administrative  burden  resulting  from 
meeting-heavy  schedules.  The  evaluation  concludes  that  streamlining  governance 
mechanisms  and  enhancing  coordination  among  governance  bodies  could  generate 
valuable  synergies,  reducing  organisational  overhead.  At  the  same  time,  eu-LISA’s 
excessive  reliance  on  external  contractors  for  technical  delivery  poses  persistent 
challenges, limiting its ability to effectively oversee complex IT projects. Addressing this 
dependency by boosting internal knowledge, improving talent retention, and strengthening 
project ownership is essential for ensuring long-term organisational resilience. 

 eu-LISA  is  recognised  for  its  competent  and  efficient  operational  management  of  IT 
systems. Stakeholders’ perception of eu-LISA’s performance in this regard is positive: in 
the limited number of incidences or service disruptions that have taken place, the Agency 
was  able  to  swiftly  implement  mitigating  actions  to  ensure  the  continuity  of  service. 
However, there is room for improvement to enhance efficiency through finding a strategic 
balance  between  handling  support  functions  in-house  and  outsourcing  them  to  external 
providers.  In  contacts,  the  efficiency  of  new  system  development  is  more  limited.  The 
development of the EES, for instance, was subject to multiple delays and, therefore, was 
ultimately not developed within the foreseen schedule. Although stakeholder feedback is 
not  overly  critical,  satisfaction  levels  of  the  Agency’s  efficiency  in  terms  of  system 
development lag behind those for operational management.  

66 

 
5.2.  Lessons learned 

The evaluation concluded that the current mandate and objectives of the Agency remain 
largely adequate. While the Agency has demonstrated effective performance in fulfilling 
several aspects of its mandate, there remain opportunities for improvements in the areas of 
effectiveness, efficiency and coherence. 

Challenges persist, particularly concerning the development of IT systems. There is also 
scope for improvement in as regards the alignment of the current eu-LISA Regulation with 
other relevant legal instruments, cooperation with international organisations, as well as 
piloting, testing, and ad hoc support to Member States. These challenges are linked to a 
range  of  internal  factors,  including  strategic  planning,  budget  and  human  resources 
management, technology infrastructure, contractor oversight, and aspects of the Agency’s 
organisational and governance structure. 

Taking these findings and the complex operating environment into account, the evaluation 
confirms that, at this stage, eu-LISA’s mandate remains appropriate. Since the adoption of 
the  current  eu-LISA  Regulation,  two  of  the  critical  systems,  the  renewed  Schengen 
Information  System  and  the  Entry/Exit  System,  as  well  as  the  first  component  of  the 
Interoperability  framework,  the  shared  Biometric  Matching  Service,  while  four  others 
(ECRIS-TCN, Eurodac, ETIAS and the Communion Identity Repository) are scheduled 
for  operational  launch  in  2025-2026.  Although  modifications  to  eu-LISA’s  current 
mandate could be beneficial in some areas, any significant alterations to the mandate at 
this stage could divert the Agency from its core objectives and jeopardise its capacity to 
deliver the planned priority IT systems on time, resulting in further delays in the rollout of 
the Interoperability Framework.   

In particular, modifying eu-LISA's mandate by amending the current eu-LISA Regulation, 
in  the  near  future,  would  pose  significant  challenges  at  both  strategic  and  operational 
levels: 

• 

• 

Increased risk exposure: modifying the mandate, in particular changes to Article 
16(4)  of  the  current  eu-LISA  Regulation,  regarding  ad  hoc  support  to  Member 
States, could heighten the Agency’s risk exposure. While greater flexibility in this 
area could be desirable, the evaluation suggests that eu-LISA should completing 
the current on-going projects before undertaking additional tasks. Any additional 
tasks entrusted to eu-LISA through a modification of Article 16(4) would increase 
the risks of further delays in the Interoperability Project.    

Instability  and  delays:  introducing  a  revised  mandate  at  this  time  could  create 
instability given the Agency’s commitment and task to deliver four systems and 
one interoperability component within the next two years. A change of the mandate 
at this moment would likely increase the risk of delays. .  

•  Budgetary  constraints:  an  amendment  to  the  mandate  does  not  automatically 
mean  additional  resources.  Therefore,  any  increase  in  tasks  would  necessitate 
commensurate  financial  support,  including  a  new  Legislative  and  Financial 
Statement, which would also increase financial instability.   

67 

 
Certain  weaknesses  identified  in  the  evaluation,  such  as  the  cooperation  with  third-
countries  or  the  use  of  innovative  technological  solutions  for  the  operation  and 
management of the system, including the use of cloud technology, would require a change 
of  eu-LISA’s  mandate.  In  contrast,  a  large  number  of  other  issues  identified  can  be 
addressed  operationally  without  an  immediate  mandate  change.  Therefore,  while  the 
Agency would benefit from a change in the mandate, most of the immediate shortcoming 
identified can be addressed through operational changes, providing the Agency with the 
necessary stability to deliver on the more time pressing projects.  

The evaluation finds that not amending the mandate at this stage could for now better 
serve eu-LISA to deliver effectively and efficiently on its ongoing tasks. Therefore, eu-
LISA should prioritise organisational stability and strategic operational improvement over 
immediate changes to its mandate. The challenges identified by the evaluation that may 
require a mandate revision can be addressed once eu-LISA has completed its most pressing 
tasks  such  as  delivering  the  planned  IT  systems  and  implementing  the  interoperability 
architecture.   

Based on this conclusion, the evaluation includes a list of 54 recommendations (under the 
heading  ‘global  assessment  of  the  areas  of  attention’)  have  been  prepared.  The  priority 
order of the recommendations in the list is based on (1) their urgency; (2) their feasibility; 
and (3) the evaluation criteria. Considering the evaluation’s finding that mandate changes 
could create further risks and impediments for eu-LISA in the current context, mandate-
related recommendations have been de-prioritised in the short term.  

On  this  basis,  the  evaluation  has  identified  the  following  11  recommendations,  to  be 
implemented  by  eu-LISA,  as  the  most  important,  as  they  can  be  implemented  without 
changes  to  the  current  eu-LISA  mandate  and  will  contribute  directly  to  organisational 
stability and the timely delivery of the Agency’s tasks: 

-  Develop  a  Target  Operating  Model  (TOM)  to  help  eu-LISA  operate  effectively 
while  managing  risks,  making  decisions  quickly,  and  ensuring  clarity  in 
responsibilities, in particular, in case of incidents or emergencies. 
-  Develop and implement a comprehensive medium-term HR strategy. 
-  Establish and implement a medium to long-term technology strategy. 
- 
- 

Implement a multiannual portfolio management approach. 
Improve resource planning functionalities, e.g. through establishing processes and 
using appropriate tools to address incomplete view of staff allocation and steering. 
-  Deploy an integrated tool for the development cycle (such as JIRA) across eu-LISA 
and contractors to streamline project management and collaboration, minimising 
disruptions, optimising workflows, and ensuring relevance. 

-  Establish and implement a medium-term sourcing and outsourcing strategy. 
-  Strengthen existing capacities concerning in-house legal services. 
-  Update the service catalogue with full descriptions of services and define clear roles 

- 

and responsibilities for service owners and managers. 
Improve  visibility  about  current  resource  allocation,  project  status,  processes, 
bottlenecks,  shortages,  in  order  to  enable  more  informed  discussion  about  the 
prioritisation of projects, also at political level. 

-  Decrease  dependency  from  external  contractors,  while  increasing  project 

ownership and knowledge retention.  

68 

 
ANNEX I:   PROCEDURAL INFORMATION 

1.  Lead DG, Decide reference and Work Programme reference 

DG Migration and Home Affairs (DG HOME) is the lead DG. There is no reference to the evaluation and the review in the Commission Work Programme 2024. 

2.  Derogations granted and justification 

In  conducting  the  evaluation,  no  exceptions  from  the  usual  procedural  requirements  described  in  the  Better  Regulation  Guidelines  were  required.  Public 
consultation did not take place as eu-LISA’s core tasks are highly technical, sensitive and of confidential nature and it was concluded that a public consultation 
would not bring specific added value. However, as part of the evaluation, the public was given the possibility to respond to the call for evidence. Furthermore, 
extensive targeted consultation of relevant stakeholders was planned and took place during the evaluation process (as described in Annex V). 

3.  Organisation and timing 

The  Terms  of  Reference  (ToR)  for  engaging  an  external  contractor  to  carry  out  the  study  to  support  the  evaluation  of  the  European  Union  Agency  for  the 
Operational Management of Large-Scale IT Systems in the Area of Freedom, Security and Justice (eu-LISA) were drawn up in June 2023. A request for service 
was issued on 3 August 2023, and a contractor selected by an evaluation committee consisting of staff from DG HOME in October 2023. The kick-off meeting 
for the study took place on 6 November 2023 and the final report was submitted on 14 June 2024.  
In line with the Better Regulation Guidelines, an Inter-Service Group (ISG) was set up within the Commission to accompany the evaluation process. Several 
Directorates-General (DGs) within the Commission were invited to nominate representatives to the ISG. DG HOME chaired the ISG  meetings. The ISG was 
regularly consulted over the course of the evaluation process, typically in conjunction with the submission of specific draft reports by the contractor responsible 
for carrying out the external study. These consultations took place both in regular meetings, and via written consultations. The following list provides an overview 
of the ISG’s work over the course of the evaluation and review process: 

-  The ISG was convened for the first time on 19 July 2023 in order to receive a presentation of the evaluation of eu-LISA’s Regulation and to discuss and 
provide feedback on the draft ToR prepared for the external contractor. The meeting also served as an opportunity to discuss and comment on the call for 
evidence  and  the  draft  consultation  strategy,  which  e.g.  described  how  the  external  contractor  should  carry  out  targeted  consultations  with  relevant 
stakeholders in the context of the study, in cooperation with and under the supervision of DG HOME and the ISG. The participants were invited to provide 
additional written feedback after the meeting. 

69 

 
 
 
 
 
 
 
-  On 1 December 2023, the ISG received the draft Inception Report for written comments and subsequent approval. On the basis of the ISG’s written 

feedback, the draft Inception Report was further revised by the external contractor.  

-  On 1 February 2024, the ISG received the draft Interim Report for written comments and subsequent approval. The Interim meeting with the ISG and the 
external contractor took place on 13 February. On the basis of the ISG’s written feedback, the draft Interim Report was further revised by the external 
contractor. 

-  The Commission accepted the Final Report on 10 July 2024. 
-  The ISG was consulted during the drafting of this Staff Working Document. 

In line with the Better Regulation Guidelines, an Inter-Service Group (ISG) was set up within the Commission to accompany the evaluation process. In addition, 
an extended steering group was set-up, composed by the representatives of Belgium and Estonia in the eu-LISA Management Board, and tasked with providing 
advice to the Commission during the evaluation.   

4.  Consultation of the Regulatory Scrutiny Board  

Not applicable, as this evaluation was not subject to consultation and scrutiny of the Regulatory Scrutiny Board. 

5.  Evidence used together with sources and any issues regarding its quality  

The evaluation and the review primarily drew on the supporting study entitled “Study to support the evaluation of Regulation (EU) 2018/1726 of the European 
Parliament and of the Council of 14 November 2018 on eu-LISA”, commissioned by DG HOME in August 2023 and carried out by the external contractor (EY). 
The objectives of the study were to assess the performance of eu-LISA in relation to objectives, mandate, locations and tasks with a view of identifying potential 
areas for improvements. The study assessed the relevance, effectiveness, efficiency, coherence, and EU added value of the eu-LISA Regulation. Beside an in-
depth review of relevant documentation, including legal documents, policy and strategic documents, programming and internal business documents, Agency’s 
outputs and publications, financial and other documents. the study also relied on extensive targeted consultations in the form of surveys and interviews with 
relevant stakeholders. More specifically, the external contractor conducted 3 surveys targeting Member States, and EU Agencies and bodies, with 353 replies. 
EY also conducted 111 interviews with stakeholders. 
During the process, the Commission provided updates and consulted Member States in the eu-LISA Management Board meetings.  

6.  Use of external expertise 

Apart from the study carried out by the external contractor, no other groups were consulted.  

70 

 
 
 
 
 
 
 
Additional information on the stakeholder consultation is provided in Annex V. 

71 

 
 
ANNEX II. METHODOLOGY AND ANALYTICAL MODELS USED 

In line with the ToR, the work plan was structured into four Tasks: (i) Preparation, (ii) Fieldwork, (iii) Analysis and (iv) Finalisation of the evaluation. The 
following figure presents an overview of these tasks, their main objectives, activities and outputs. 

Figure 1 – Overview of Tasks, activities and outputs 

72 

 
 
 
 
 
During the kick-off meeting for this project, DG HOME indicated that the open public consultation initially proposed by the study team should not be carried out 
for this project in line with the Better Regulation Guidelines. In line with the suggestions from DG HOME, the study team devoted the resources initially foreseen 
for this activity to conduct an additional survey with the Advisory Group members (Activity 2.3). 

Data collection strategy 
As part of the assignment, we collected various types of primary and secondary data using the appropriate analysis techniques to assess each type of data and 
each piece of information: 

•  Qualitative information was coded and thoroughly cross-analysed. This involved searching for common themes, or areas where opinions differ across 

the data; 

•  Quantitative data collected was checked for quality and coherence, and structured for quantitative analysis, including: (1) Producing descriptive statistics 
(counts, cross-tabulations); (2) Producing graphs and charts to visualise trends over time and comparisons; and (3) Exploring the data to detect trends, for 
example, in relation to different stakeholders, different activities; and 

•  Cost data in line with the Better Regulation Toolbox (BRG), Tools #56 and #57. 

We conducted the evaluation using a multi-criteria analysis, including a cost-benefit assessment and assessment of the administrative impacts under the "one-in-
one-out" approach. Data undergone a critical assessment for quality, limitations, and strengths. We systematised information aligned with the evaluation matrix, 
which is structured according to the evaluation criteria / questions and covers all three dimensions as defined by the Commission in the ToR. We emphasised 
comprehensive  coverage  and  employed  triangulation  of  data  between  different  stakeholders  and  across  desk  and  field  research.  In  summary,  our  analysis 
involved two steps: data collection based on the evaluation criteria and questions, and dimension-specific analysis. Triangulation iteratively verifies data, fostering 
nuanced answers through pattern identification and differences examination. We collected the data both via (i) desk research  and (ii) field research, which is 
described in the following. 
Desk research 
Desk research was integral to our approach in evaluating the Agency’s performance. It served as a foundational data source, informing the assessment of the 
Regulation's implementation, eu-LISA's functioning and performance, and (future) needs related to its activities. We mapped existing information, in order to 
identify gaps for potential field research, and ensured that all team members had a thorough understanding of eu-LISA's mandate and activities. Our research 
covered various document types, detailed in the table below and extended by the data the Commission has e.g. provided through CircaBC.  

of 

Table 1 – Selected categories of sources covered by desk research 
Types 
documents 
Academic 
and 
grey  literature  on 
activities  of  eu-
LISA 

Main objectives 

Assessment of the context and functioning of the Agency as well as existing needs with regard to eu-LISA’s role  

73 

 
 
 
of 

and 

Types 
documents 
Policy 
legislative 
documents 
Documents 
the 
to 
relating 
monitoring 
and 
evaluation  of  eu-
LISA’s activities 
Documents 
the 
relating 
to 
governance 
and 
structures  of  eu-
LISA 
Documents 
relating 
the 
to 
performance of eu-
LISA’s tasks  
Documents 
relating 
to 
technical 
performance of the 
IT systems 
Cost data  
Documentation  of 
supervision  of  eu-
LISA  (e.g.  reports 
by the EDPS) 

the 

Main objectives 

Analysis of the policy and legislative framework the Agency operates in, for example, to understand overall policy objectives.  

Assessment of the functioning of the Agency and the implementation of eu-LISA Regulation 

Examination of eu-LISA’s governance and internal functioning  

Examination of the performance of eu-LISA in relation to its different tasks 

Examination of the adequacy of the technical solutions applied eu-LISA  

Analysis of the management of financial resources and the cost-effectiveness  

Examination of the extent to which eu-LISA is working within legal boundaries and identification of any issues in this respect  

At the end of the project, the study team has gathered and analysed around 140 documents. Given the sheer number of documents, the study team prepared a 
well-structured approach on registering and assessing the documents in an overview Excel, thus enabling all members to easily gain access to sources relevant to 
them.  

Field research  
The consultation strategy followed a mixed-method approach (see Table  below) that ensured an extensive coverage (written surveys) with targeted tools that 
allowed collecting more detailed information and having in-depth discussions with relevant stakeholders (interviews, focus groups and workshops). We aimed 
for a profound integration of relevant stakeholders, which was crucial to ensure access to relevant information (validity of findings) and a high level of credibility 

74 

 
 
 
of the results (acceptance). The approach was based on a careful analysis of the objectives and context of this evaluation and ensured an active engagement of 
both eu-LISA and relevant external stakeholders at the EU and national levels. 
While the burden on stakeholders was kept to a minimum and engaging each stakeholder only to the extent necessary and with the suitable tools, our strategy 
involved a wide range of stakeholders within the Agency’s units (including e.g. the advisory groups), systems’ users, other relevant EU institutions, bodies, 
offices and agencies and different political stakeholders. 

Table 2 – Consultation tools used  
Tools  
Task 1 – Preparation  

Main objectives and justification  

Exploratory visit to eu-LISA 

Task 2 – Fieldwork  

Targeted written surveys 

Interviews & focus groups with 
eu-LISA staff, the Advisory 
Groups, the PMBs and the 
Management Board 

Interviews with stakeholders 
at the EU level 

Interviews and focus group 
with private sector actors 
(contractors of eu-LISA, 
carriers) 

SWOT workshops on the 
different systems  

Two exploratory visits to eu-LISA's main seats in Tallinn and Strasbourg were conducted in the initial phase of the study – postponed due 
to scheduling issues to mid-January. The visits served a dual purpose: (i) establishing a working relationship with eu-LISA for successful 
cooperation  in  the  evaluation,  and  (ii)  gathering  insights  to  refine  the  methodological  approach,  including  practical  aspects  for  later 
consultations, and confirming access to written sources, incl. internal documents. The visits were, in addition, used to get in contact with 
relevant stakeholders (in particular eu-LISA staff) in order to gather first qualitative insight into the operation mechanisms of the Agency.  

We  conducted  three  written  surveys  targeting  diverse  groups  efficiently:  (i)  users  of  various  IT  systems,  (ii)  advisory  boards,  and  (iii) 
supervisory authorities. The surveys included tailored questions relevant to each group, ensuring standardised and comparable evidence 
collection and engaging users and authorities with role-specific perspectives. The participation in the survey can be assessed as successful, 
with 175 responses registered for the AG survey, 132 responses registered for the users and 46 responses registered for the supervisory 
authorities. Even though the surveys included some unit and item non-responses, overall participation was very well, and most questions 
posed could be analysed based on a sufficient amount of respondents.  
To  comprehend  eu-LISA's  internal  operations,  we  contacted  69  stakeholders  via  interviews  and  focus  groups  within  key  departments, 
gathering detailed insights. Additionally, engaging with Advisory Groups for each eu-LISA system was crucial for understanding business-
critical  processes,  including  governance  structures,  roadmap  evolution,  and  system  training  plans.  The  Management  Board,  a  key 
governance body with representatives from Member States, Eurojust, Europol, Frontex, and the EPPO, were consulted to capture diverse 
perspectives on the functioning of eu-LISA and its governance set-up. 
We conducted interviews with external stakeholders, such as additional Commission DGs, EU agencies, and cooperation partners to gather 
diverse  perspectives  on  eu-LISA.  This  approach  allowed  us  to  obtain  detailed  insights  from  each  stakeholder,  considering  the  unique 
perspectives they offer on the functioning of eu-LISA. 
In order to assess how the private sector perceives the work of eu-LISA, 4 interviews with external contractors of eu-LISA and a focus group 
was conducted. As part of the focus group, air, land and sea carriers were invited to share their view on eu-LISA and the obligations stemming 
from the work of eu-LISA. Overall, twenty stakeholders participated in the virtual focus group, and were able to share their experiences in 
two breakout sessions. Both formats helped understanding how eu-LISA is perceived by the private sector and which recommendations 
they would have for the Agency.  
To comprehensively assess eu-LISA's overall functioning and performance across its diverse tasks and IT systems, we conducted 6 SWOT 
workshops with system users and other national stakeholders working with the systems in different ways. These workshops helped identify 
strengths, weaknesses, current challenges, and future opportunities. Most workshops focused on a single system (e.g. for SIS, VIS, Eurodac) 
or on the overall topic of interoperability. Some workshops were clustered for comparable systems (e.g. EES & ETIAS or ECRIS-TCN & e-

75 

 
 
Tools  

Main objectives and justification  
CODEX)  and  conducted  for  both  systems  simultaneously,  also  allowing  stakeholders  to  point  out  to  synergies  or  shared  challenges.  In 
general,  the  workshops  followed  the  clustering  of  eu-LISA’s  AGs.  On  average,  roughly  eight  participants  participated  in  the  workshops. 
Regardless  of  the  number  of  participants,  all  workshops  helped  gathering  important  evidence  thanks  to  the  active  participation  of  most 
participants. 
We conducted six case studies to explore crucial aspects of eu-LISA's functioning. These included the following topics:  

Case studies on key activities 
/ projects 

Task 3 – Analysis  
Design thinking workshop 

Presentation of the final report 

•  The flexibility of eu-LISA’s mandate; 
•  Respect of the principle of the independence of the judiciary; 
•  Lessons learned from the go-live of the renewed SIS; 
•  The efficiency of eu-LISA’s governance structure; 
•  The introduction of squads; and  
•  The impact of age structure and organisation membership on the functioning of the Agency.  

The case studies served to strengthen and underline insights from prior consultations and involved additional interviews (depending on the 
case) and an in-depth analysis of existing data and desk research. The case studies were scheduled for the conclusion of Task 2, integrating 
insights from earlier consultation activities. 

The design thinking workshop will be conducted in unison with the interservice steering group meeting on 6 May 2024.  
On 11 June, we will hold a presentation of the final report to the Commission as well as the Management Board of eu-LISA to inform about 
the main conclusions and recommendations of the evaluation and provide them the possibility to ask questions concerning the evaluation. 

Quality of information and data collected 
The data collected for the completion of this evaluation included: 

-  Documents analysed as part of desk research; 
-  Exploratory interviews with the European Commission; 
- 

(Group) interviews with eu-LISA management, staff and governance bodies, some conducted during the visits to Tallinn and Strasbourg that also included 
tours of the premises; 
Interviews and focus groups with private sector actors; 

- 
-  Three targeted written surveys for Member States representatives, users of the IT systems and data protection authorities; and 
-  Six SWOT Workshops with national experts of individual systems and interoperability. 

Concerning the document review, around 140 documents were analysed (public and not public). These documents include legal provisions applicable to eu-
LISA  and  those  regulating  other  EU  large-scale  IT  systems,  technical  documentation,  system  reports,  documents  related  to  projects  and  studies,  strategic 
documents,  governance-,  or  human  resources  related  information,  among  others.  The  quality  of  these  documents  was  satisfying  and  met  our  expectations, 
contributing to a sound basis for the evaluation relating to all three dimensions covered. Nevertheless, the information was not fully complete. In particular, the 
Study Team did not receive documentation regarding information requested. For around 10 (types of) documents, including, among others, statistics on data 

76 

 
 
 
protection, exit-survey outcomes with former staff, training evaluation forms, statistics on job advertisements (e.g. numbers of application, number of rejections, 
etc.), documentation on the volume of outsourcing, the Study Team had not had access. A list of documents received from eu-LISA is presented in the Annex 
(section Error! Reference source not found.). 

The interviews conducted have provided insights from various perspectives. Overall, the vast majority of interviewees were very responsive in the scheduling of 
the interviews and were forthcoming with providing helpful and constructive information, openly sharing their experiences and opinions. Several persons indeed 
highlighted the importance of the evaluation from their perspective. As concerns the interviewees with eu-LISA, some of them tended to focus on their personal 
challenges and making suggestions for future improvements. While this is highly relevant, efforts have been put in ensuring that the interviews were balanced in 
(1) covering not only challenges, but also strengths relating to the current performance of eu-LISA; and (2) focusing on the past five years (retrospective view). 
With the completion of this tasks, we conducted interviews with 106 different stakeholders. This includes eu-LISA management staff, eu-LISA governance 
bodies, national authorities, contractors, sea-, land-, and air carrier, as well as other EU Agencies and bodies. The interviews and focus group conducted were 
scheduled for up to one hour each. The content derived from the interviews as well as specific information of interviewees was encrypted to ensure complete 
anonymity of participants. 
Apart from the insights received from the virtual and on-site interviews, the visits of eu-LISA sites also provided highly valuable insights for the evaluation 
based on well-organised tours of the premises. In particular, they contributed significantly to the understanding of the work of the Agency, including the Agency’s 
tasks as well as work culture and atmosphere at the different sites. 
The targeted written surveys were open from the end of January 2024 until 29 February 2024. In total, the Study Team has received 353 completed responses, 
including 175 for the survey with Advisory Group members, 132 for the survey with users and 46 for the survey with data protection authorities as presented in 
section 0. The quality of the responses received is adequate considering that 41% of all respondents completed the full survey and 57% of all respondents across 
the three surveys completed the survey to at least 50%. Table  below shows the statistical outcomes of the three individual surveys.  

Table 3 – Overview of reponses to the targeted written surveys 

Survey 

Responses 
received 

AG & PMB 
members 
Users of IT 
systems 
Data protection 
authorities 
Total 

175 

132 

46 

335 

Number of 
responses 
100% 
completed 

100% 
completion rate 

Number of 
responses 
>=50% 
completed 

>=50% 
completion 
rate 

67 

55 

17 

139 

43% 

42% 

37% 

41% 

101 

68 

22 

191 

64% 

52% 

48% 

57% 

Source: External Study. 

77 

 
 
 
Participants  also  used  the  open  questions  within  the  surveys  to  provide  additional  qualitative  input,  including  e.g.  valuable  insights  relating  to  areas  for 
improvement (e.g. tasks outsourced, governance structure or the operational management improvements). In addition, numerous participants voluntarily flagged 
their availability for a follow-up interview. Throughout the process, the Study Team has been constantly monitoring response rates and quality of the responses 
and implemented the planned mitigation actions (e.g. sending out reminders) as needed. 

As  an  additional  part  of  the  data  collection  exercise,  we  conducted  six  different  SWOT  Workshops  on  the  different  large-scale  IT  systems  as  well  as  on 
interoperability between 21 February 2024 and 7 March 2024.83 These workshops were prepared for national users and members of the respective Advisory 
Groups of the different systems who provided us information about the strengths, weaknesses, opportunities and threats of the specific systems from their national 
perspectives. Across all SWOT workshops, participants represented a wide range of Member States and Schengen Associated Countries. An average session was 
joined by eight diverse participants. This number allowed to obtain in-depth details about system-related aspects. For collection information in an interactive 
manner,  the  Study  Team  prepared  session-specific  Mural  boards  on  which  inputs,  thoughts  and  ideas  were  collected.  Once  the  strengths,  opportunities, 
weaknesses, and threats were collected and ranked by prioritisation, participants derived potential recommendations and assessed their individual impact and 
feasibility. 

Description of problems encountered and mitigation actions  
The identification of potential risks and/or problems encountered, and the development of mitigation actions is an integral part of the quality control measures. 
The following table presents the problems encountered.  

Table 1 – Problems encountered and mitigation actions 

Problems encountered 
Facing a wealth of information, 
potentially hindering the review process 

Challenges and delays in scheduling 
and coordinating visits at eu-LISA  

High number of interviews planned, 
several data collection efforts going on 
in parallel  

Mitigation actions 
The Study Team has prepared an overview of around 140 documents that were made available. This overview structures our 
analysis, as it includes the main contents of each document and the review status. In addition, we have clearly allocated each 
document to a specific team member to spread the work across the team.  
While it was challenging to identify suitable dates for the visits of the two eu-LISA sites in Tallinn and Strasbourg, this was 
solved via a close coordination with the dedicated contact point within eu-LISA. Together with the contact point, two weeks, 
in which the majority of relevant staff was available were identified, so that we were able to conduct a high number of interviews 
in a short time frame. Staff who was not available during those days has been contacted after the visits to schedule virtual 
interviews.  
The Study Team has taken deliberate decisions on which activities and stakeholders to focus on in the beginning of Task 2. 
In  particular,  the  activities  that  are  more  challenging  to  organise  (eu-LISA  visits,  online  surveys,  SWOT-Workshops)  and 
stakeholders that provide essential inputs for the content of the evaluation and/or act a important multipliers (e.g. eu-LISA, 
MB, AGs) were prioritised. 

83 The workshops were clustered according to eu-LISA’s Advisory Group setups. 

78 

 
 
 
 
 
Problems encountered 
Reliance on multipliers for distributing 
online surveys  

Coordination of SWOT workshops 

Low survey response rates 

Mitigation actions 
For  the  distribution  of  online  surveys,  the  Study  Team  relied  on  the  Advisory  Group  chairs  and  members  as  well  as  on 
members  of  the  EDPB.  Based  on  the  efforts  put  into  clearly  communicating  the  added-value  of  the  surveys  and  actively 
following up with the contact points, the distribution of the surveys functioned well and led to satisfying response rates. 
Setting up the SWOT workshops has been challenging, considering that we had to find suitable time slots for a large group 
of persons. Our approach to ensuring a high participating rate includes: (1) planning with ample time ahead, in particular for 
sessions involving (super-) users who still needed to be identified; (2) being flexible relating to the participants to be involved 
and time slots; and (3) proactively following up with the contact points.  
Low  response  rates  to  online  surveys  can  impact  the  quality  of  data  collected.  Based  on  the  efforts  put  into  clearly 
communicating  the  added-value  of  the  surveys  and  actively  following  up  with  the  contact  points,  the  response  rate  was 
satisfying. 

79 

 
 
 
ANNEX III. EVALUATION MATRIX 

The following tables present the evaluation matrix, consisting of separate tables for each evaluation criterion.  

Relevance 

Understanding  

Analytical and 
methodological 
approach 

Potential limitations  

Alignment of eu-LISA Regulation's scope and mandate with the needs of the Member States, the EU, and external stakeholders; 
Degree to which the Regulation's objectives and purpose are relevant in view of existing and future problems observed;  
Evaluation of the extent to which the Regulation remains relevant in addressing the evolving demands of its stakeholders and the changing 
landscape of technology and security. 
Mapping of eu-LISA Regulation's objectives and mandate vis-à-vis existing problems and evolving demands, as well as any relevant EU policy 
goals and priorities, ensuring a thorough understanding of their scope and evolution; 
Data collection through desk research, written surveys, and interviews with EU institutions, stakeholders, and experts to assess the alignment 
between eu-LISA's objectives and broader EU policy needs; 
Comparative Analysis to determine the extent to which eu-LISA's objectives correspond to evolving trends, challenges, and emerging needs, 
especially in fields such as technology, law enforcement, and other relevant areas. 
Assessing “relevance” may involve subjective judgments, as perceptions of what is relevant can vary among stakeholders and over time; 
Subjective judgments will be made transparent and discussed with DG HOME; 
Stakeholders’ inputs potentially biased due conflicts of interest. Project team will scrutinise evidence for potential bias; 
Potentially limited availability and accuracy of data, especially when assessing the alignment with broader EU policy goals and priorities. 
Besides, some information may be confidential or difficult to access (project team has an expert with security clearance involved). 

Table 1- Relevance – Evaluation matrix 

Evaluation 
question 

Sub-questions (if 
applicable) 

Judgement criteria  

Indicators / information needs (examples)  

EQ 1: To what extent did the scope of the 
mandate and the objectives of eu-LISA 
Regulation remain relevant over the 
implementation period? 

The scope and objectives of 
eu-LISA Regulation have 
remained relevant throughout 
the implementation period. 

Mapping of eu-LISA's scope of mandate and objectives 
Mapping of key developments, new needs and challenges 
during the implementation period, as well as threats to be 
addressed 
Mapping and comparison of eu-LISA's scope and objectives 
with emerging trends, needs and challenges 

Information sources 
(indicative) 

Desk research: 
eu-LISA Regulation 
Online literature and 
sources on key trends, 
needs and challenges etc. 
Targeted written surveys 
Interviews with EU 
institutions, eu-LISA 
stakeholders, national 
authorities 

80 

 
 
 
 
 
EQ 2: How did the objectives of eu-LISA 
Regulation correspond to wider EU policy 
goals and priorities? 

The objectives of eu-LISA 
Regulation corresponded to 
wider EU policy goals and 
priorities 

EQ 3: To what extent has eu-LISA 
adapted to the new needs and challenges, 
in particular to the technological or 
scientific progress since the adoption of 
eu-LISA Regulation? 

eu-LISA has adapted to new 
needs and challenges, 
including technological and 
scientific progress. 

EQ 4: To what 
extent have the 
Agency’s tasks 
evolved since the 
adoption of eu-
LISA Regulation, 
and, has eu-LISA 
been able to cope 
with the increasing 
number of tasks? 

EQ 4.1. To what 
extent have the 
Agency’s tasks 
evolved since the 
adoption of eu-LISA 
Regulation 

EQ 4.2: Has eu-
LISA been able to 
cope with the 
increasing number 
of tasks? 

n.a.  

The Agency coped well with 
new or adapted tasks, both 
from a qualitative and a 
quantitative perspective 

Mapping of the objectives of eu-LISA Regulation 
Mapping of other relevant EU policy goals and priorities (e.g. 
GDPR Digitalisation and IT policies, or initiatives like the 
European Counter Terrorism Centre (ECTC)) 
Mapping and comparison of eu-LISA's objectives with the 
EU's broader policy priorities, e.g. in the field of technology, 
law enforcement cooperation etc. 
Extent to which eu-LISA's objectives have supported or been 
contradictory to wider EU goals 
Mapping of key new trends, needs and challenges since the 
adoption of eu-LISA Regulation 
Mapping of actions taken by eu-LISA to adapt to the 
identified new trends, needs and challenges, such as:  
Mapping of any actions taken by eu-LISA to adapt to 
emerging technologies (such as cloud computing, artificial 
intelligence, or blockchain) and challenges (e.g. refugee 
crises) in its IT systems and operations 
Collecting any actions taken by eu-LISA to adapt its IT 
infrastructure to accommodate new technological trends (e.g. 
the integration of modern cybersecurity measures, data 
encryption protocols, and alignment with data protection 
regulations) 
Identification of any key new trends, needs or developments 
in relation to which eu-LISA has not yet taken any action 
Assessment of the extent to which the IT infrastructures 
employed by eu-LISA correspond to recent standards, e.g. 
based on the identification of better suited technologies (e.g. 
more secured and advanced technologies) 
Mapping of changes in eu-LISA's tasks since the adoption of 
eu-LISA Regulation (by year and type of task), including the 
number of new or adapted tasks per area 
Comparison of the previous tasks with current responsibilities 
to identify any shifts, including complexity, volume, and 
variety of new or adapted tasks compared to the original 
mandate 
Historical workload data, including the number of projects, 
tasks, and associated timelines 
Allocation of additional human and technical resources in 
response to the growing workload, including the possibility 

Desk research: 
Legal documents of eu-
LISA regulation 
Other EU policy 
documentations 
Interviews with EU 
institutions, and eu-LISA 
stakeholders 

Desk research: 
Online sources concerning 
key new trends, needs and 
challenges 
Data analysis of technical 
documents, system 
updates, and security 
measures 
Targeted written surveys 
with eu-LISA users from the 
MS 
Interviews and focus 
groups with eu-LISA staff, 
including technical experts 
and management 
SWOT workshops 

Desk research: 
Interviews and focus 
groups with eu-LISA staff 
Interviews with EU-level 
stakeholders 

Desk research: Key 
Performance Indicators 
(e.g. from eu-LISA’s KPI 
reports) which measure the 
efficiency of task execution 

81 

 
 
EQ 5: Has the nature of the work of eu-
LISA experienced such a change that a 
change of mandate/objectives is needed? 

n.a. 

for/flexibility of existing resources to efficiently take on/deal 
with an increasing number of tasks 
Measurement of the efficiency of task execution through key 
performance indicators (KPIs) such as project completion 
time, error rates, or cost-effectiveness 
Identification of resource gaps, such as staffing, 
infrastructure, or technology, that have hindered eu-LISA's 
ability to manage the increased workload effectively 
Mapping of changes in the nature of tasks performed by eu-
LISA over time (e.g. complexity index based on factors such 
as skill requirements, technical intricacy, and resource 
intensity) 
Share of emerging technologies effectively incorporated into 
task execution 
Evidence that eu-LISA has not been able to executive all 
relevant types of tasks required due to limitations in the 
scope of the mandate or objectives of eu-LISA (e.g. new 
tasks associated with emerging challenges and needs) 

EQ 6: Has eu-LISA’s mandate provided 
the necessary flexibility to address 
unexpected challenges? 

Eu-LISA’s mandate has 
provided the necessary 
flexibility to address 
unexpected challenges 

Timeliness and effectiveness of responses to unexpected 
challenges, measured by response time and outcomes (both 
immediate/short-term and long-term challenges) 
Evidence of situations where unexpected challenges could 
not be addressed based on restrictions in the mandate and 
reasons for these 
The need and adequacy of Article 16(4) of eu-LISA 
Regulation 

82 

such as project completion 
times, cost effectiveness, or 
resource utilisation 
Interviews and focus 
groups with eu-LISA staff 
Interviews with EU-level 
stakeholders 

Interviews and focus 
groups with eu-LISA staff 
and other EU-level 
stakeholders on the 
evolving nature of tasks 
and alignment with the 
current mandate 
Desk research: 
Examination of legal 
documents and regulations 
to assess the capability of 
the existing mandate with 
emerging needs. Legal 
documents of eu-LISA 
regulation 
Review of eu-LISA's 
internal reports, strategic 
plans, and task records 
related to evolving tasks 
Interviews with EU-level 
stakeholders 
Desk research and 
Document review: 
Examination of legal 
documents to assess the 
compatibility of the existing 
mandate with emerging 
needs 
Analysis of documents 
related to the adaptation of 
eu-LISA's mandate in 

 
 
EQ 7: To what extent has the Agency 
been able to undertake additional tasks 
established in legal acts adopted after the 
entry into force of eu-LISA Regulation? 

The Agency has effectively 
taken on additional tasks and 
responsibilities in line with the 
legal acts adopted post-
enactment of eu-LISA 
Regulation 

EQ 8: To what extent would the Agency 
be capable to play a role in the 
governance of the systems from a 
business perspective? 

The Agency would be capable 
of playing a role in the 
governance of new systems 
from a business perspective. 

EQ 9: To what 
extent does the 
Agency’s mandate 

EQ 9.1.: Does the 
Agency’s mandate 
enable it to 

Eu-LISA Regulation is set up 
in such a way that it is 
sufficiently flexible to 

response to crises or 
emerging needs 
Interviews and focus 
groups with eu-LISA staff 
Interviews with EU-level 
and national stakeholders 
Desk research: 
Review of Agency reports, 
task implementation plans, 
and records of resource 
allocation for tasks 
established by new legal 
acts 
Analysis of records 
demonstrating the Agency's 
adherence to legal acts and 
their requirements in 
relation to new tasks 
Interviews and focus 
groups with eu-LISA staff 
Interviews with EU-level 
and national stakeholders 
SWOT workshops 
Desk research, e.g. eu-
LISA’s strategic plan; 
Interviews and focus 
groups with eu-LISA staff; 
Interviews with EU-level 
and national stakeholders 
SWOT workshops. 

Desk research, in particular 
eu-LISA Regulation 

Mapping of relevant legal acts adopted post-enactment of eu-
LISA Regulation and identification of new and additional 
tasks for eu-LISA stemming from these acts 
Processes within eu-LISA to identify any additional tasks to 
be conducted based on the legal acts 
Time taken to initiate and fully implement additional tasks 
mandated by the relevant legal acts 
Identification of any additional tasks stemming from the 
relevant legal acts that have not (yet) been taken on by eu-
LISA including the respective reasons for delays or 
barriers/roadblocks, e.g. based on eu-LISA's capacities to 
take on additional tasks relating to system operation, 
management and development  
Resource allocation efficiency, measured by the alignment of 
resources with the demands of new tasks  
Stakeholder satisfaction and feedback regarding the 
Agency's ability to handle additional responsibilities and tasks 
Comparison of the governance models of the different IT 
systems and identification of patterns 
Needs of each system in terms of business processes 
Analysis of the difference in the governance models as 
regards to the business models 
Identification of the role of the Agency in designing the 
business processes 
Effectiveness of the Agency in designing the business 
processes, including based on an identification of processes 
that have been designed, their implementation and 
perception by stakeholders 
Presence of a clearly defined business governance structure 
for new systems 
Qualifications and relevant experience of personnel 
Processes within eu-LISA for identifying new responsibilities 
eu-LISA should take on 

83 

 
enable eu-LISA to 
undertake new 
responsibilities? 

undertake new 
responsibilities? 

accommodate new 
responsibilities, ensuring 
adaptability to evolving needs 

EQ 9.2.: Is the 
current mandate 
legally sufficient for 
new tasks? 

Eu-LISA Regulation provides a 
well-defined legal framework 
that is sufficiently flexible for 
eu-LISA to take on key new 
tasks when necessary 

EQ 10: Are all activities performed by eu-
LISA relevant and necessary considering 
the mandate of the Agency? 

All activities performed by eu-
LISA are relevant and 
necessary considering the 
mandate of the Agency. 

EQ 11: Is the current mandate and 
organisation of the Agency correctly 
reflecting the need to respect the principle 
of independence of the judiciary? 

eu-LISA's current mandate 
and organisational structure 
respect the principle of 
independence of the judiciary.  

Mapping and identification of any new key responsibilities 
that eu-LISA has taken on within the current scope of its legal 
mandate 
Mapping and identification of any new key factors that eu-
LISA has not been able to take on due to restrictions and 
limitations based on its legal mandate 
Comparison of eu-LISA Regulation to similar regulations in 
other EU agencies (e.g. Frontex Regulation (EU) 2016/1624, 
or Eurojust Regulation (EU) 2018/1727) to identify common 
features related to legal flexibility 
Analysis of the language and provisions in eu-LISA 
Regulation 
Evidence where the mandate of the Agency limits its role in 
supporting IT-based border management and law 
enforcement 
Stakeholders' perceptions on whether eu-LISA’s role should 
remain to be purely IT development and management, or 
whether this should be broadened with new tasks and 
responsibilities beyond IT 
Analysis of the extent to which the current mandate is 
sufficient to take on new responsibilities beyond IT 
Mapping of eu-LISA's key activities to the objectives and 
scope defined in eu-LISA Regulation 
Identification of any activities that are currently executed by 
eu-LISA that cannot clearly be connected to its mandate 
Stakeholder perceptions of the relevance and necessity of 
eu-LISA's current activities in fulfilling its mandate 
Identification of any decisions and actions of the Agency 
affecting large-scale IT systems in AFSJ that may 
compromise the independence of the judiciary, including 
based on views from relevant stakeholders, such as judicial 
authorities, legal experts, and supervisory authorities 
Description of the approach of the Management Board to 
ensure that eu-LISA’s decisions and actions do not 
compromise the independence of the judiciary, e.g. any 
processes established, discussions on these aspects in the 
meetings 

Interviews with EU-level 
stakeholders 
Interviews and focus 
groups with eu-LISA staff 

Desk research: 
Legal documents on eu-
LISA Regulation 
Legal documents of similar 
EU regulations (e.g. 
Frontex Regulation 
2016/1624) 
EU legal frameworks 
Interviews with EU-level 
stakeholders 
Interviews and focus 
groups with eu-LISA staff 
SWOT workshops 

Desk research 
Interviews and focus 
groups with eu-LISA 
management 
Interviews with EU-level 
and national stakeholders 
Desk research, including 
e.g. internal documents on 
relevant governance 
procedures, minutes of the 
MB meetings 
Targeted written surveys 
Interviews and focus 
groups with eu-LISA 
Interviews with national 
stakeholders and others 

Effectiveness 

84 

 
 
 
Understanding  

Analytical and 
methodological 
approach 

Potential 
limitations  

Assessment of the degree to which eu-LISA Regulation's objectives have been achieved or progress been made towards their achievement, including 
general, specific, and operational objectives; 
Investigation of the factors contributing to the success or failure of achieving these objectives, providing insights into what has worked and what has 
not; 
Examination of the impact of eu-LISA's activities concerning different users and purposes, offering a nuanced understanding of their effectiveness. 
Variety of methods, including desk research, interviews, focus groups, (SWOT) workshops, written surveys, and case studies; 
Evaluation of multiple dimensions of eu-LISA's performance and objectives, addressing a wide range of criteria; 
Gathering and analysis data from internal sources (documents, records) and internal as well as external stakeholders (interviews, written surveys, 
focus groups, workshops) to ensure a comprehensive assessment; 
Utilisation of combination of quantitative (metrics, KPIs) and qualitative (perceptions, opinions) indicators. 
Reliance on self-reported data and perceptions from stakeholders may introduce bias. Project team will carefully scrutinise for bias and be transparent 
about it in the deliverables; 
Potentially limited accuracy and completeness of historical data, especially in terms of incident reports and budget records. Will be discussed with 
Commission and mitigated through use of multiple data collection methods and analytical approaches; 
Interpretation of qualitative data and the assessment of subjective indicators might be influenced by individual perspectives. Project team is very 
experienced in this regard and will make any subjective judgments transparent to the Commission; 
Influence of external factors on eu-LISA's performance may not be fully isolatable from internal factors. 

Table 2 - Effectiveness – Evaluation matrix 

Evaluation question  Sub-questions 

Judgement criteria  

Indicators / information needs (examples) 

EQ 12: How and to 
what extent did eu-
LISA meet its core 
objectives and tasks 
set out in eu-LISA 
Regulation? 

[see sub-questions 
below as well as more 
detailed evaluation 
questions 13 ff.] 

Eu-LISA has achieved its 
objectives and tasks as defined 
in eu-LISA Regulation  

EQ 12.1 Operational 
Objectives: 
To carry out the 
operational 
management of relevant 
large-scale IT systems 

eu-LISA effectively operates 
large-scale IT systems and the 
related communication 
infrastructure  
Upon request, new IT systems 
are developed efficiently  

Extent to which each of the objectives have been 
achieved or progress been made towards the 
achievement of the objectives  
Identification of internal or external factors hindering 
progress  
Stakeholder’s perception of the Agency's progress 
towards achieving the operational objectives. 
Performance of eu-LISA in relation to the operational 
management of relevant large-scale IT systems and 
the related communication infrastructure  
Performance of eu-LISA in relation to the 
development of IT systems  

Information sources 
(indicative) 

Desk research  
Targeted written surveys  
Interviews & focus groups 
with eu-LISA staff and MB 
Interviews with EU-level and 
national stakeholders 
SWOT workshops 
Case studies 

85 

 
 
 
 
 
 
 
and the related 
communication 
infrastructure  
To develop the IT 
systems in an efficient 
manner  
To ensure the effective, 
secure and continuous 
functioning of the IT 
systems  
To ensure 
interoperability between 
different IT systems  
To ensure a high service 
quality for users  
To ensure a high level of 
data protection  
To implement an 
appropriate level of data 
and physical security  
To manage the systems 
in an efficient and 
financially accountable 
manner  
To provide advice and 
support to MS and the 
EC  
To contribute to training 
and awareness for users 
of the IT systems  
To compile and publish 
statistics and produce 
statistical reports  
To make suggestions for 
necessary technical 
changes to the IT 
systems  
To engage in research 
and pilot projects  

The IT systems are constantly 
and securely available  
Where applicable, data can be 
exchanged seamlessly between 
different IT systems  
Users can easily navigate the IT 
systems and identify and use all 
functions relevant  
Information is processed within 
legal boundaries and respect of 
fundamental rights  
Data and eu-LISA premises are 
sufficiently protected  
The systems are managed in an 
efficient manner and eu-LISA is 
held accountable in case of any 
concerns in this respect  
Member States and the EC 
receive useful advice and 
support in relation to specific 
requests in a timely manner  
Users are sufficiently competent 
in using the IT systems  
Statistics on eu-LISA fulfil quality 
standards and are published 
timely in a lucid manner  
The IT systems are constantly 
re-assessed and where 
applicable improved  
Relevant trends in research are 
recorded sufficiently and, where 
applicable, research and pilot 
projects are actively supported  
Interested stakeholders and the 
public are able to access 
information on eu-LISA’s 
activities. 

Performance of eu-LISA in relation to ensuring the 
effective, secure and continuous functioning of the IT 
systems  
Performance of eu-LISA in relation to ensuring 
interoperability  
Performance of eu-LISA in relation to ensuring a high 
service quality for users  
Performance of eu-LISA in relation to ensuring a high 
level of data protection  
Performance of eu-LISA in relation to implementing 
an appropriate level of data and physical security  
Performance of eu-LISA in relation to managing the 
systems in an efficient and financially accountable 
manner  
Performance of eu-LISA in relation to providing 
advice and support to MS and the EC  
Performance of eu-LISA in relation to contributing to 
training for users of the IT systems, including based 
on: the methodology pursued by the Agency to 
provide training on the technical use of the systems 
and the experiences of participants with such training 
the establishment of different forums by the Agency 
to communicate with end-users and the efficiency of 
these forums in the context of preparation, exchange 
of good-practice and trainings assessment if the 
chosen forums are adequate to the targeted 
audience  
Performance of eu-LISA in relation to compiling and 
publishing statistics and producing statistical reports  
Performance of eu-LISA in relation to compiling and 
publishing statistics and producing statistical reports  
Performance of eu-LISA in relation to research and 
pilot projects  
Performance of eu-LISA in relation to ensuring 
transparency on its activities. 

86 

 
To ensure transparency 
on eu-LISA’s activities. 
EQ 12.2 Specific 
Objectives:  
To contribute to a 
coordinated, cost-
effective and coherent 
IT environment at Union 
level in the AFSJ that is 
in line with fundamental 
rights  
To contribute within the 
scope of its mandate to 
evidence-based Union 
migration and security 
policy-making and to the 
monitoring of the proper 
functioning of IT 
systems in the AFSJ  
To contribute to the 
development of IT-
based solutions in the 
AFSJ. 
EQ 12.3 General 
Objectives 
To contribute to a high 
level of security within 
the AFSJ, including by 
facilitating border 
management and law 
enforcement within the 
AFSJ 

The Union level IT environment 
is operated in a coordinated, 
cost-effective and coherent way 
and in line with fundamental 
rights  
eu-LISA’s provides input to 
evidence-based Union migration 
and security policy-making and 
to the monitoring of the IT 
systems  
eu-LISA contributes significantly 
to the development of IT-based 
solutions in the AFSJ. 

Contributions of eu-LISA to a coordinated, cost-
effective and coherent IT environment at Union level 
in the AFSJ that is in line with fundamental rights  
Contributions of eu-LISA to evidence-based Union 
migration and security policy-making and to the 
monitoring of the proper functioning of IT systems in 
the AFSJ  
Contributions of eu-LISA to the development of IT-
based solutions in the AFSJ. 

Contributions of eu-LISA to a high level of security in 
the AFSJ  

eu-LISA contributes to a high 
level of security in the AFSJ by 
facilitating border management 
and law enforcement within the 
AFSJ through the development, 
management and operation of 
large-scale IT systems in these 
fields 

EQ 13: How has the performance of eu-LISA 
been in operationally managing the large-scale 
IT systems? 

eu-LISA has ensured system 
availability, effectively minimised 
operational incidents, adhered to 
Service Level Agreements 
(SLAs), and efficiently resolved 
operational incidents of the 
managed large-scale IT 
systems. 

Average system availability over a specified period 
Pro-active actions and strategy taken to minimise 
and address operational incidents  
Number of operational incidents and their impact on 
system performance  
Actions and time taken to address operational 
incidents (taking into account the complexity of the 
incidents)  

Desk research  
Targeted written surveys for 
users  
Interviews and focus groups 
with eu-LISA  
Interviews with EU-level and 
national stakeholders. 

87 

 
EQ 14: To what extent has eu-LISA been 
effective in ensuring the secure and continuous 
operational management of large-scale IT 
systems entrusted to it? 

eu-LISA has been effective in 
ensuring the secure and 
continuous operational 
management of the entrusted 
large-scale IT systems. 

EQ 15: To what extent has eu-LISA fulfilled its 
obligation in relation to data protection, data and 
physical security in accordance with the 
applicable rules, including specific provisions for 
each large-scale IT system? 

eu-LISA has fulfilled its 
obligation in relation to data 
protection, data and physical 
security in accordance with the 
applicable rules, including 
specific provisions for each 
large-scale IT system. 

Level of correspondence with Service Level 
Agreements (SLAs) for each large-scale IT system 
(including any deviations and explanations of the 
reasons for this and how / how timely they were 
addressed)  
Compliance with data quality criteria set up in the 
legal act governing each system 
User satisfaction with: 
IT infrastructure 
The system availability  
Data quality 
The number of operational incidents  
The time and actions taken to address incidents  
The level of adherence with SLAs. 
Number of security breaches or data incidents over 
time 
Comprehensiveness and adequacy of security 
measures and protocols (i) in place and (ii) their 
implementation in practice 
User satisfaction regarding system security and 
continuous operational management 
Frequency and impact of system interruptions due to 
security issues (including time taken to address the 
interruptions, taking into account their complexity) 

For each large-scale IT system: 
Adequacy and alignment of security strategies and 
measures with operational needs and applicable 
rules, including specific provisions for each large-
scale IT system 
Level of preparedness for physical and IT security 
risks 
Type and relevance of implemented physical and IT 
security controls 
Number of recorded security issues and incidents 
Type and relevance of security guidelines and 
trainings for staff 
Notification of security incidents to the security officer 

88 

Desk research 
Incident reports related to 
security breaches  
Security audit reports  
Records of system 
interruptions and their 
causes  
Targeted written surveys  
Interviews and focus groups 
with eu-LISA  
Interviews with EU-level and 
national stakeholders  
SWOT workshops. 
Desk research: 
Security procedures and 
guidelines  
Data protection procedures 
and guidelines  
Previous security and data 
protection risk assessments’ 
results  
Identified security measures’ 
implementation roadmap 
and advancement 
documents  

 
Notification of security incidents to users 
Assessment of risks related to data protection 
Strategy adopted by the Agency to comply with data 
protection law 
Type and relevance of implemented data protection 
measures 
Number of recorded data protection issues and 
incidents, including measures taken to address these 
and prevent that they take place again 
Type and relevance of data protection guidelines and 
training for staff 
Notification of data protection incidents to the DPO 
Existence of business continuity and recovery 
procedures 
Efficiency of business continuity and recovery plan 
User satisfaction with data protection, data and 
physical security 
Degree to which new large-scale IT systems have 
been launched and successfully developed and 
deployed on schedule, including relevant, necessary 
and timely inputs from eu-LISA  
Assessment of the cost-effectiveness of system 
development, including e.g. based on adherence to 
projected budgets  
Assessment of the quality of work in relation to the 
development of new IT systems, including e.g. based 
on timely delivery, number of critical system defects 
or issues identified post-launch, user feedback 
Technical capacity of eu-LISA to comply with 
readiness criteria required for the start of operation 
by the legal acts governing the systems in 
development or being taken over 
Frequency and effectiveness of communication and 
collaboration among the three parties (European 
Commission, Member States, eu-LISA) 
Description of the role of the Management Board, 
Programme Management Board, Advisory Groups, 
Project Management fora and the working groups 
established under the auspices of eu-LISA, including 

Interviews and focus groups 
with eu-LISA staff  
Targeted written survey with 
eu-LISA users  
Interviews with EU-level and 
national stakeholders  
SWOT workshops. 

Desk research: 
Project management 
records and documentation  
Records of system testing 
and quality assurance 
processes  
Targeted written surveys  
Interviews & focus groups 
with eu-LISA staff and MB  
Interviews with EU-level and 
national stakeholders  
SWOT workshops. 

Desk research / document 
review: 
Meeting minutes, 
communication records, and 
collaboration reports  
Timelines and records of 
decision-making processes  

89 

EQ 16: How has eu-LISA performed in the 
development of new large-scale IT systems? 

eu-LISA provided important 
inputs for the development of 
new large-scale IT systems, 
meeting project goals, timelines, 
and quality standards 

EQ 17: To what extent have eu-LISA tripartite 
set-up and governance structure affected the 
performance of the Agency in delivering its core 
tasks? 

The tripartite set-up and 
governance structure have 
positively contributed to eu-
LISA's performance in delivering 
its core tasks, ensuring efficient 
decision-making and 
collaboration among 
stakeholders. 

 
 
EQ 18: To what extent have eu-LISA’s 
management structures and monitoring of 
external contractors been effective to assure 
compliance and timely implementation of 
contracts related to the development and 
operational management of large-scale IT 
systems? 

EQ 19: To what extent has eu-LISA been 
effective in ensuring the efficient and financially 
accountable management of large-scale IT 
systems? To what extent has eu-LISA been 
effective in managing its budget: could budgets 
expenditures be related to 
objectives/expenditures, are accounts compliant 
with EU budget rules, are discrepancies 
between actuals and budgets explained? 

Eu-LISA's management 
structures and monitoring of 
external contractors have 
effectively ensured compliance 
and timely implementation of 
contracts, contributing to the 
successful development and 
operational management of 
large-scale IT systems. 
eu-LISA has effectively 
managed large-scale IT systems 
and its budget, ensuring 
efficiency, financial 
accountability, correspondence 
with EU budget rules, and clear 
explanations for discrepancies 
between actual expenditures 
and budgets 

an identification of the division of tasks and 
responsibilities and/or shared responsibilities 
Existence of coordination between the different fora, 
including e.g. based on identifying formal and 
informal processes of communication 
Existence of synergies, contradictions and overlaps 
between the work of the different fora 
Timeliness of decision-making processes related to 
core tasks and system development 
Level of alignment between governance structure 
activities and Agency performance 
Stakeholder satisfaction with the tripartite set-up and 
governance structure 
Types and frequency of reported challenges or 
bottlenecks related to the governance structure 
Percentage of contracts completed on time and 
within budget 
Frequency and effectiveness of contract monitoring 
activities 
Number of reported contractual issues and their 
resolution 
Stakeholder satisfaction with contract management 
and implementation 

Budget allocated to the operational management of 
large-scale IT systems, broken down by cost items 
and objectives  
Budget utilisation/absorption rate for large-scale IT 
systems compared to allocated funds, broken down 
by cost items and objectives  
Evidence of any budget expenditure that could not 
clearly be related to objectives/expenditures  
Correspondence with EU budget rules and 
regulations  
Existence and adequacy of explanations for 
discrepancies between actual and budgeted 
expenditures  
Transparency of eu-LISA's financial planning and 
documentation 

Reports on the alignment of 
governance activities with 
performance  
Documentation of 
challenges or issues related 
to the governance structure  
Interviews & focus groups 
with eu-LISA staff, MB, AGs 
and PMB  
Interviews with EU-level and 
national stakeholders. 

Interviews and focus groups 
with eu-LISA, including 
management and internal 
services  
Interviews with EU-level and 
national stakeholders, 
including contractors. 

Desk research: 
Financial records and 
reports, including budget 
allocations and expenditures  
Audits and financial 
performance assessments  
Documentation on EU 
budget rules  
Reports explaining 
discrepancies in 
expenditures  
Interviews and focus groups 
with eu-LISA  
Interviews with EU-level 
stakeholders and 
supervisors. 

90 

 
 
EQ 20: How effective has eu-LISA been in 
ensuring high quality, continuous and 
uninterrupted services for users of large-scale IT 
systems? How effective has eu-LISA been in 
meeting the Service Level Agreement(s), in 
particular for availability of service, for each 
large-scale IT system? 

eu-LISA has effectively ensured 
high-quality, continuous, and 
uninterrupted services for users 
of large-scale IT systems and 
consistently met Service Level 
Agreements (SLAs), especially 
for system availability. 

EQ 21: To what extent have the research 
activities of eu-LISA contributed to the 
performance in delivering its core tasks? 

eu-LISA's research activities 
have significantly contributed to 
enhancing the performance of 
delivering its core tasks and 
objectives 

EQ 22.1: Has the 
Agency effectively 
performed its tasks 
related to developing 
new systems in line with 
its current mandate? 

The Agency has effectively 
performed its tasks in 
developing new systems, in line 
with its current mandate 

EQ 22: Has the 
Agency effectively 
performed its tasks 
related to developing 
new systems in line 
with its current 
mandate? How 
effective was the 
Agency to provide 
the technical 

Stakeholder satisfaction with financial management 
practices 
Percentage of system availability compared to SLA 
targets  
Frequency and duration of system downtime or 
interruptions  
Timeliness and relevance of actions taken to address 
any service interruption  
Correspondence with SLA requirements, including 
response times and problem resolution  
Records of system incidents and their impact on 
operations 

Number of research initiatives and projects 
undertaken by eu-LISA (linked to the objectives of 
the Agency)  
Comparison of the research activities conducted by 
the Agency against the technological needs of the 
Agency to manage and the develop the Union’s IT 
systems as well to anticipate future needs 
Resources spent on research initiatives in total, as a 
range and on average  
Demonstrable impact of research findings on 
improving the efficiency or effectiveness of core 
tasks, including reduction of costs or more efficient 
working arrangements  
Integration of research outcomes into operational 
practices and decision-making processes. 
Types of tasks performed in connection with the 
development of new systems  
Timeliness of new system development projects in 
accordance with its mandate  
Budget adherence and cost-effectiveness in 
developing new systems  
Capacity to address technical challenges and 
changes during system development. 

Desk research: 
SLA reports and 
performance metrics  
Incident logs and records of 
system downtime  
Internal performance 
monitoring and reporting  
Targeted written surveys 
with users, 
Interviews and focus groups 
with eu-LISA  
Interviews with EU-level and 
national stakeholders 
Desk research: 
Documentation of research 
projects, reports, and 
publications  
Records of actions taken 
based on research 
recommendations 
Internal assessments of the 
influence of research on 
operational tasks  
Interviews and focus groups 
with eu-LISA. 

Desk research, in particular 
on project reports, timelines 
and milestones  
Targeted written surveys 
with advisory groups  
Interviews and focus groups 
with stakeholders with eu-
LISA  
Interviews with EU-level and 
national stakeholders  

91 

 
 
 
capacity, to develop 
the systems? How 
effective was the 
governance structure 
provided by the 
Agency to support 
the development of 
the systems? 

EQ 22.2: How effective 
was the Agency in 
providing the technical 
capacity to develop the 
systems? How effective 
was the governance 
structure provided by 
the Agency to support 
the development of the 
systems? 

EQ 22.3: How effective 
was the governance 
structure provided by 
the Agency in 
supporting the 
development of the 
systems? 

The Agency effectively provided 
the necessary technical capacity 
for system development 

The governance structure 
provided by the Agency 
effectively supported the 
development of systems 

EQ 23: Has eu-LISA effectively handled 
incidents on each large-scale IT systems 
operated by the Agency? 

eu-LISA has effectively 
managed and resolved incidents 
on each large-scale IT system, 
ensuring minimal disruption and 
adherence to incident 
management procedures. 

EQ: 24 Has eu-LISA effectively implemented the 
recommendations of the previous external 
evaluation completed in 2016? 

eu-LISA has effectively 
implemented the 
recommendations from the 2016 
external evaluation, resulting in 
improved performance. 

Appropriateness of the type and volume of technical 
expertise and resources provided  
Alignment between technical resources and system 
development needs  
Timely completion of system development tasks  
Adequate quality of different aspects of system 
development  
User feedback on system development  

Evidence that the different eu-LISA governance 
bodies have executed their role in relation to the 
development of systems in an adequate and timely 
fashion  
Evidence of delays, roadblocks and barriers with 
regard to the involvement of the different governance 
bodies of eu-LISA in the system development  
Stakeholder engagement and collaboration within the 
governance structure. 
Per large-scale IT system: 
Number and severity of incidents over time  
Incident response time  
Incident resolution time  
Impact on operations of incidents  
Number of cases where minimal disruptions was not 
ensured, including the reasons for this and any 
actions taken to prevent reoccurrence in the future  
Incident management procedures and their 
implementation  
Improvements of incident management procedures  
User satisfaction with incident handling. 
Timely implementation of recommendations from the 
2016 external evaluation (by recommendation)  
Adherence to budget and cost-effectiveness in 
implementing recommendations  
Identification of any recommendations that were no 
longer relevant following the adoption of the 2018 
Regulation  

SWOT workshops. 
Desk research, in particular 
on project reports, timelines, 
resource allocation, and 
milestones  
Targeted written survey with 
users  
Interviews and focus groups 
with eu-LISA  
Interviews with EU-level and 
national stakeholders  
SWOT workshops. 
Desk research, in particular, 
governance structure 
assessments and reports  
Interviews and focus groups 
with eu-LISA management  
Interviews with EU-level and 
national stakeholders 
including monitoring. 

Desk research, e.g. review 
of Incident logs and records 
maintained by eu-LISA  
Targeted written surveys 
with users  
Interviews and focus groups 
with eu-LISA  
Interviews with EU-level and 
national stakeholders  
SWOT workshop 

Desk research, e.g. review 
of the 2016 evaluation report  
Interviews and focus groups 
with eu-LISA  
Interviews with EU-level and 
national stakeholders, 
especially supervisors. 

92 

 
EQ 25: How successful has the Agency been in 
progressing towards the objective set up in eu-
LISA Regulation? 
EQ 26: What external factors have affected 
progress towards the objectives and how are 
they linked to the performance of the Agency 
and the implementation of eu-LISA Regulation? 

[see detailed judgement criteria and indicators for EQ12] 

Alignment of eu-LISA's action plan with the goals of 
the 2016 external evaluation 

External factors affecting 
progress towards the objectives 
of eu-LISA Regulation are 
identified and their impact on the 
Agency's performance and 
Regulation implementation is 
understood. 

Identification of key external factors having a positive 
or negative impact on eu-LISA’s performance  
Assessment of how external factors have affected 
the achievement of specific objectives outlined in eu-
LISA Regulation  
Analysis of the Agency's adaptability and 
responsiveness to external factors. 

EQ 27: Does eu-LISA Regulation allow the 
Agency to cooperate with all international 
organisations, which are relevant for carrying out 
its tasks efficiently? 

Eu-LISA Regulation enables the 
Agency to efficiently cooperate 
with all relevant international 
organisations for the effective 
execution of its tasks. 

Analysis of the legal framework and provisions within 
eu-LISA Regulation related to international 
cooperation, including limitations with regard to 
cooperation with specific stakeholders  
Identification of international organisations that are 
relevant for the Agency's tasks and objectives  
Assessment of the Agency's ability to engage in 
cooperation and collaboration with international 
organisations based on eu-LISA regulation  
Evidence that cooperation with certain key 
organisations is currently not possible based on eu-
LISA Regulation. 

Interviews / focus groups 
with eu-LISA staff and EU 
and national stakeholders 
Targeted written surveys  
Interviews and focus groups 
with eu-LISA  
Interviews with EU-level and 
national stakeholder groups  
Desk research / document 
review  
SWOT workshop  
Case studies. 
Desk research, e.g. review 
of eu-LISA Regulation and 
relevant legal documents  
Interviews and focus groups 
with eu-LISA  
Interviews with EU-level 
stakeholders. 

93 

 
 
 
Efficiency  

Understanding  

Analytical and 
methodological approach 

Limitations  

Evaluation of the efficient translation of resources into results (such as, funds, human resources, expertise, and time); 
Analysis of the cost-effectiveness of operating eu-LISA's systems, considering the benefits and costs incurred by end-users and other 
beneficiaries; 
Assessment of the overall relationship between resource allocation and the outcomes achieved, identifying areas where improvements in 
efficiency are possible. 
Data is gathered from diverse sources such as document reviews, stakeholder interviews, focus groups, SWOT workshops, and targeted 
written surveys; 
Assessment of a multitude of aspects related to eu-LISA's efficiency, encompassing resource management (including outsourcing strategy), 
organisational structure, governance, technology, and HR strategies; 
Application of a blend of quantitative metrics (e.g. budget allocation, cost-effectiveness) and qualitative indicators (e.g. stakeholder opinions, 
organisational policies) for a holistic evaluation; 
Deployment of comparative analysis techniques, including benchmarking and cost-benefit assessments, to provide insights into eu-LISA's 
performance in relation to industry standards and best practices. 
Potentially limited data accuracy that heavily relies on the availability and completeness of data from various sources. Will be discussed with 
Commission and mitigated through use of multiple data collection methods and analytical approaches; 
Stakeholder interviews, surveys, and focus groups involve subjective opinions, potentially introducing bias; Project team will carefully 
scrutinise for bias and be transparent about it in the deliverables; 
Efficiency may be influenced by external factors beyond eu-LISA's control, such as changes in EU policies or international events. 

Table 3 - Efficiency – Evaluation matrix 

Evaluation question  Sub-questions 

Judgement criteria  

Indicators / information needs (examples) 

EQ 28: To what extent have eu-LISA’s 
organisational solutions, including 
outsourcing strategies affected the 
performance of the Agency in developing 
the new systems? 

The organisational solutions and outsourcing 
strategies implemented by eu-LISA have 
positively affected the performance of the 
Agency in developing new systems, resulting 
in improved efficiency and effectiveness. 

Role of eu-LISA’s own organisational 
solutions in developing new systems  
Types of solutions development outsourced 
over time  
Indications that outsourcing is happening 
strategically or tactically  
Criteria and processes used for selecting 
outsourcing partners  
Flexibility and scalability of outsourcing 
strategy  
If available: eu-LISA outsourcing performance 
data (e.g. success, quality)  

Information sources 
(indicative) 

Desk research: 
Review of organisational 
and outsourcing policies 
and strategies within eu-
LISA  
Examination of project 
timelines and budget 
allocation for new system 
development  
Interviews and focus 
groups with eu-LISA staff  
Interviews with EU-level 
and national stakeholders. 

94 

 
 
 
 
Direct and indirect costs and benefits of 
outsourcing (overall and per solution/item 
outsourced), including compared to: 
Inhouse alternatives  
Industry standards  
Benchmarks from other private and public 
organisations 
Cost-effectiveness of outsourcing strategies 
and implementation, and organisational 
solutions in new system development (‘value 
for money’, ‘return on investment’)  
Identification of tasks the Agency is 
outsourcing in the context of developing new 
systems, takeover or renewal of existing 
systems 
Effects of the outsourcing strategy and 
practice on the timeliness and effectiveness of 
developing new systems, e.g. based on the 
quality and success of new systems 
developed within the scope of outsourcing 
arrangements vs. own organisational 
solutions (“value for money”) and the impact 
on the Agency's internal knowledge of the 
systems in development 
Mapping of the key roles and tasks of the 
different governance bodies, including their 
competencies and mandate  
Mapping of the number, types and functions 
of stakeholders involved in the different 
governance bodies  
Mapping of the key processes of the different 
governance bodies  
Clarity of internal processes to employees 
and key stakeholders  
Efficiency of key decision-making and other 
key processes within the governance 
structure, including replacement procedures 
of key staff, e.g.: 
Time taken for decisions  

Desk research: 
eu-LISA's governance 
documentation  
organisational charts and 
policies  
Interviews and focus 
groups with eu-LISA  
Interviews with EU-level 
and national stakeholders. 

EQ 29: To what extent has the governance 
structure of eu-LISA supported its ability to 
perform its tasks, having regard to its size, 
composition, organisation and working 
processes? 

The governance structure of eu-LISA has 
effectively supported the Agency's ability to 
perform its tasks in alignment with its size, 
composition, organisation, and working 
processes. 

95 

 
Frequency and length of high-level decision 
meetings  
Existence and application of processes and 
procedures to involve the management board 
in the decisions regarding the operational 
management of the large-scale IT systems  
Efficiency of escalation procedures  
Level of specialisation versus generalisation 
of the bodies and stakeholders involved in the 
governance structure. 
Correspondence of the governance structure 
with eu-LISA's strategic objectives and tasks  
Responsiveness of the governance structure 
to emerging challenges in (home) security 
and needs. 
See the indicators regarding EQ29. In 
addition: 
Potential redundancies of structures, 
processes, including across locations  
Resource allocation and utilisation within the 
governance structure, including the efficiency 
and flexibility to address “business as usual” 
tasks and objectives, as well as changing 
needs, challenging, tasks and responsibilities 
Budget allocation and absorption overall and 
per cost item (direct and indirect) relating to 
the operation and management of large-scale 
IT systems, broken down by year, system 
etc., e.g.: 
Capital investments in IT (hardware, software, 
cloud / data storage infrastructure)  
Ongoing maintenance of systems (hard- and 
software), including through outsourced 
services, licenses, and subscriptions to 
services  
Communication costs (network and telecom 
costs)  
Energy costs (power, utilities, water) 
Training of staff  
Awareness raising, info, and training of users  

Desk research: 
Governance documentation  
Records of decision-
making timeline  
Interviews with EU-level 
and supervising 
stakeholders  
Interviews and focus 
groups with eu-LISA. 
Desk research: 
Financial reports and 
budget analysis  
Records on procurement 
and resource allocation  
Targeted written surveys 
with users  
Interviews / focus groups 
with eu-LISA staff  
Interviews with EU-level 
stakeholders, including 
supervisors. 

EQ 30: How efficient is the Agency 
governance structure to reach its goals and 
complete its tasks? 

The governance structure of the Agency 
demonstrates efficiency in facilitating the 
achievement of its goals and successful 
completion of tasks. 

eu-LISA has efficiently managed its financial 
resources, demonstrating adherence to 
budgets and cost-effectiveness. 

EQ 31.1: How 
efficiently has eu-
LISA managed its 
financial and 
budgetary 
resources to 
support the 
development and 
operation of large-
scale IT systems? 

EQ 31: How efficient 
the management of 
resources by eu-LISA 
has been? To what 
extent has eu-LISA 
efficiently used its 
human and financial 
resources to ensure 
the development and 
the effective, secure 
and continuous 
operation of large-
scale IT systems? To 
what extent has eu-
LISA succeeded in 
building up in-house 

96 

 
 
capabilities for 
handling various 
tasks entrusted to it? 

eu-LISA has successfully built in-house 
capabilities for handling various tasks. 

EQ 31.2: To what 
extent has eu-LISA 
efficiently 
developed in-
house capabilities 
for handling 
various tasks 
entrusted to it? 

EQ 32: Are the 
resources (human, 
financial and 
technical) used for 
the operational 

EQ 32.1: Are the 
resources (human, 
financial, and 
technical) used for 
the operational 

The resources allocated for operational 
management and maintenance are adequate 
in achieving the desired result 

97 

Recruiting costs  
Staff salaries  
Budget adherence and cost-effectiveness in 
managing financial resources, including 
reasons for any changes compared to plan 
Timeliness and effectiveness of resource 
allocation for system development and 
operation 
Efficiency in procurement processes for 
external services 
Flexibility in shifting resources based on 
emerging needs or obtaining additional 
resources and constraints imposed by the 
procedures on the management of resources 
User satisfaction with the (perceived) cost-
effectiveness of the support provided by eu-
LISA in relation to the development and 
operation of the large-scale IT systems  
Types of in-house capabilities and skills, 
including development over the years, by type 
of key task to be performed (including the mix 
of senior and junior profiles) 
Development and retention of human 
resources, including unfilled positions per 
year, by type of key task 
Number and % of staff with permanent 
contracts vs. temporary staff, by type of key 
task 
Evidence of specific skills gaps over time, and 
how these have hampered the efficient 
execution of tasks 
Comparison of in-house capacity with external 
outsourcing 
Success in recruiting and retention of different 
staff categories 
For the operational management and 
maintenance of existing systems: 
Budget allocation and absorption overall and 
per cost item, broken down by year, system 
etc. 

Desk research: 
Internal assessments of in-
house capabilities  
Recruitment and retention 
records  
Human resources statistics 
provided by eu-LISA  
Interviews and focus 
groups with eu-LISA 
management and internal 
services  
Interviews with EU-level 
stakeholders, including 
supervisors. 

Desk research  
Interviews and focus 
groups with eu-LISA  
Interviews with EU-level 
stakeholders  

 
 
management and 
maintenance of 
existing systems 
adequate in 
comparison with 
the results 
achieved? 

management and 
maintenance of 
existing systems 
adequate in 
comparison with the 
results achieved? 
Could the efficiency 
gains be obtained 
with a different 
approach to 
outsourcing or with 
different technology 
choices? 

EQ 32.2: Could 
these efficiency 
gains be obtained 
with a different 
approach to 
outsourcing or with 
different 
technology 
choices? 
EQ 33.1: Are the 
resources (human, 

EQ 33: Are the 
resources (human, 

Evidence that the desired results were 
achieved within the allocated budget and 
allocated human and technical resources  
Evidence that the volume and skills (including 
the level of seniority) of the human resources 
involved in the operational management and 
maintenance of existing systems were 
adequate to achieve the intended targets  
Evidence that the technical resources for the 
operational management and maintenance of 
existing systems functioned adequately 
Evidence that certain tasks were not possible 
to execute in a timely fashion to the required 
quality, volume etc. due to a lack of 
appropriately qualified staff 
Reasons for any major divergences with 
regard to the budget allocated and absorbed 
for the operational management and 
maintenance of existing systems 
Resource adequacy for operational 
management 
Identification of efficiency gains through 
alternative outsourcing approaches or 
technology choices 
Identification of tools and forums used / 
established by the Agency to communicate 
with the Member States authorities and others 
having access to or participating to the 
systems as well as related benefits and costs 
Identification of potential efficiency 
improvements, particularly related to 
increased use of outsourcing and alternative 
technology choices 
Comparative analysis / Benchmarking of 
costs and benefits. 

SWOT workshop. 

Desk research, e.g. review 
of cost analysis reports  
Interviews and focus 
groups with eu-LISA staff  
Interviews with EU-level 
and national stakeholders, 
particularly supervision. 

Efficiency gains are possible through 
alternative outsourcing approaches or 
technology choices. 

Resources allocated for system development 
are adequate in achieving the desired results. 

For the development of new systems: 

Desk research, e.g. review 
of cost analysis reports  

98 

 
 
financial, and 
technical) used for 
the development of 
new systems 
adequate in 
comparison with 
the results 
achieved? 

financial and 
technical) used for 
the development of 
new systems 
adequate in 
comparison with the 
results achieved? 
Could the efficiency 
gains be obtained 
with a different 
approach to 
outsourcing or with 
different technology 
choices? 

EQ 33.2: Could 
these efficiency 
gains be obtained 
with a different 
approach to 
outsourcing or with 
different 
technology 
choices? 
EQ 34.1: To what 
extent are 
procurement, 
finances, and 
corporate services 

EQ 34: To what 
extent are 
procurement, 
finances, personnel 
and any other 

Evaluation of potential efficiency gains 
through alternative outsourcing approaches or 
technology choices 

The organisation of procurement, financial 
management, and corporate services has 
been efficient. 

99 

Budget allocation and absorption overall and 
per cost item, broken down by year, system 
etc.  
Evidence that the desired results were 
achieved within the allocated budget and 
allocated human and technical resources  
Evidence that the volume and skills (including 
the level of seniority) of the human resources 
involved in the development of new systems 
were adequate to achieve the intended 
targets  
Evidence that the technical resources for the 
development of new systems functioned 
adequately  
Evidence that certain tasks were not possible 
to execute in a timely fashion to the required 
quality, volume etc. due to a lack of 
appropriately qualified staff  
Reasons for any major divergences with 
regard to the budget allocated and absorbed 
for the development of new systems  
Resource adequacy for the development of 
new systems  
Identification of efficiency gains through 
alternative outsourcing approaches or 
technology choices. 
Identification of potential efficiency 
improvements, particularly related to different 
approaches to outsourcing, collaboration with 
external partners, and alternative technology 
choices 
Identification of potential draw-backs of 
different outsourcing approaches and 
collaboration with external partners 
Comparative analysis of costs and benefits 
Evidence of procurement and financial 
management efficiency, e.g.: 
Variance between budgeted and actual 
expenses  

Interviews and focus 
groups with eu-LISA  
Interviews with EU-level 
stakeholders, including 
supervision. 

Desk research, e.g. review 
of cost analysis reports  
Interviews and focus 
groups with eu-LISA 
management and internal 
services  
Stakeholder interviews with 
supervision stakeholders. 

Desk research, e.g. review 
of efficiency benchmarking 
reports  
Interviews and focus 
groups with eu-LISA staff  

 
 
corporate services 
organised efficiently? 
Are resources 
commensurate with 
results achieved? 

organised 
efficiently? 

Human resources are efficiently managed and 
compliant with established efficiency 
standards and benchmarks for human 
resource management. 

EQ 34.2: Are 
human resources 
efficiently 
managed to 
achieve 
commensurate 
results? 

EQ 35: To what extent has the tripartite set-
up of eu-LISA affected the efficiency of the 
use of resources by the Agency? How does 
eu-LISA's split location affect its activities? 
How has eu-LISA ensured efficient working 
on three separate locations? 

The tripartite set-up of eu-LISA and its split 
location do not negatively impact the 
efficiency of resource utilisation, enabling the 
Agency to effectively carry out its activities 
across three separate locations. 

100 

Time it takes to complete procurement 
processes from initiation to contract award  
Cost of procurement (e.g. administrative 
costs) as a percentage of the total 
procurement value  
Percentage of contracts managed within 
agreed-upon terms and timelines  
Alignment with efficiency standards and 
benchmarks  
Resource allocation compared to service 
efficiency. 
Evidence of human resource management 
efficiency: 
Number of HR staff per employee 
Percentage of employees who leave the 
organisation within a specified period  
Average time taken to fill job vacancies and 
the time it takes for new hires to become fully 
productive  
Investment in employee training and 
development programs as a percentage of the 
HR budget  
Readiness of employees to fill key roles, as 
assessed through succession planning  
The approach adopted by the Agency to 
negotiate contracts with external contractors 
Comparison of results achieved with 
workforce size. 
Measures taken by eu-LISA to ensure 
efficient working within and across its split 
locations  
Consequences of the tripartite location on the 
performance of the Agency to deliver its core 
tasks, including based on evidence of 
additional costs and duplication of work, 
resources etc. due to the tri-partite set-up 
Consequences of the tripartite location on the 
internal organisation and management of eu-
LISA, e.g. based on the communication 

Interviews with supervision 
stakeholders. 

Desk research, e.g. review 
of efficiency benchmarking 
reports and HR statistics 
provided by eu-LISA  
Interviews and focus 
groups with eu-LISA  
Interviews with supervising 
stakeholders. 

Desk research  
Interviews and focus 
groups with eu-LISA  
Interviews with EU-level 
and supervising 
stakeholders. 

 
 
 
EQ 36: Having regards to the 
organigramme of the Agency, the 
distribution of competences, and the 
sharing of responsibilities between the 
different entities, to what extent does the 
internal organisation of the Agency impact 
the efficiency of eu-LISA to reach its 
objectives? To what extent does it impact 
the Agency's accountability? 

The internal organisation of eu-LISA is 
structured and managed in a way that 
enhances efficiency, facilitates decision-
making, and contributes to the achievement of 
the Agency's objectives. 

101 

Desk research: 
Current organisational 
structure adopted by the 
Management Board on 21 
March 2018  
Records of organisational 
changes and improvements 
implemented  
Internal reports and 
assessments of decision-
making processes and 
workflow. 
Interviews and focus 
groups with eu-LISA staff  
Interviews with EU-level 
and supervising 
stakeholders. 

dynamics between the managerial and 
operation levels 
Consequences of the tripartite location on the 
management of resources, e.g. based on the 
efficiency of resource allocation and utilisation 
in relation to the tripartite set-up 
Efficiency of resource allocation and utilisation 
across eu-LISA's three separate locations 
Mapping of the key roles and tasks along key 
positions of the organigram 
Mapping of the relations between different 
parts of the organigramme and the distribution 
of competences and responsibilities 
Mapping of the number, types, functions and 
required competencies for key positions 
according to the organigram of the Agency 
Mapping of key processes and functions, 
including the sharing of responsibilities 
between different entities 
Efficiency of key decision-making and other 
key processes within the internal organisation  
Adequacy of the internal organisation with a 
view to facilitating the performance of eu-
LISA, e.g. based on evidence of roadblocks 
and delays in relation to key decision-making 
and other processes, hampering the Agency 
to reach its objectives in an efficient fashion 
Existence and application of processes and 
procedures to involve the management board 
in the decisions regarding the operational 
management of the large-scale IT systems 
Efficiency of escalation procedures 
Level of specialisation vs. generalisation of 
the bodies and stakeholders involved  
Effects of the internal structures and division 
of tasks on the accountability of the Agency, 
e.g. based on the existence of mechanisms to 
ensure accountability, instances where 
responsibility for an activity cannot clearly be 
attributed 

 
EQ 37: To what extent were the annual 
budgets of the Agency implemented in an 
efficient way and in view on achieving 
results? 

The annual budgets of eu-LISA are efficiently 
implemented in a manner that aligns with the 
achievement of results and objectives outlined 
in the budget. 

EQ 38: Is the Agency’s Human Resources 
strategy adequate to attract and retain 
high-skilled profiles in the context of the IT 
labour market? 

The Agency's Human Resources strategy is 
effective in attracting and retaining high-
skilled profiles in the competitive IT labour 
market. 

Responsiveness of the internal organisation 
to emerging challenges in (home) security 
and needs 
Comparison of actual budget expenditure 
(budget absorption) with planned budget 
allocations, and explanations of the reasons 
for any discrepancies / changes 
Relationship between budget implementation 
and the attainment of specified results 
Evidence of objectives and targets being 
achieved within the planned and absorbed 
budget 
Efficiency gains achieved through budget 
management practices 
Evidence that benefits have outweighed costs 

Recruitment success rate for high-skilled 
positions 
Retention rates for high-skilled employees 
Number and % of positions not filled, where 
high-skilled employees are required over time 
Comparison of eu-LISA's compensation and 
benefits packages with industry standards 
Evidence from existing employee satisfaction 
and engagement surveys (if available) 

EQ 39: To what extent does the internal 
organisation of eu-LISA allow and provide 
distribution of resources, competences and 
sharing of responsibilities between IT 
systems? 

The internal organisation of eu-LISA 
effectively facilitates the distribution of 
resources, competences, and sharing of 
responsibilities between IT systems, leading 

Identification of synergies and overlaps of the 
resources and skills required for the different 
IT systems 

102 

Desk research: 
eu-LISA's annual budget 
reports and financial 
statements  
Internal financial 
management documents 
and records  
Reports on budget 
execution and financial 
performance  
Interviews and focus 
groups with eu-LISA staff  
Interviews with EU-level 
and national stakeholders. 
Desk research:  
Documents relating to the 
overall monitoring and 
evaluation of eu-LISA’s 
activities, including e.g. 
evaluation reports, 
statistics” and “Cost data 
eu-LISA's recruitment and 
retention records and 
reports  
Any available employee 
surveys and feedback 
mechanisms  
Analysis of Compensation 
and benefits data and 
benchmarking studies  
Interviews and focus 
groups with eu-LISA staff.  
Interviews and focus 
groups with eu-LISA staff. 

 
 
to optimised operational performance and 
efficiency. 

EQ 40: Is the technology used by the 
Agency the most suitable to deliver its 
tasks? Is the Agency using the most 
advanced technology? 

The technology used by the Agency is well-
suited for delivering its tasks efficiently and 
effectively, and it aligns with state-of-the-art 
solutions to maximise performance 

EQ 41: Having regards to the HR strategy, 
are the allocation of positions especially 
managerial as well as secretarial positions 
and the distribution of projects within the 
Agency’s staff in line with the Agency’s 
objectives? 

The allocation of positions, including 
managerial and secretarial positions, as well 
as the distribution of projects within the 
Agency's staff, align with the objectives and 
priorities set forth in the HR strategy. 

EQ 42: Are the job descriptions coherent 
with the actual tasks executed by the staff? 

Job descriptions accurately reflect the tasks 
and responsibilities carried out by staff 

103 

Evidence of flexible and coordinated 
management and use of resources (budget, 
staff, infrastructure) among IT systems 
Allocation of responsibilities and roles among 
different units within the organisation 
Coordination mechanisms for managing 
cross-system activities 
Compatibility of existing technology with the 
Agency's tasks 
Evidence of use and effective application of 
emerging technologies in eu-LISA’s systems 
(AI, blockchain. Cloud Computing, IoT, etc.), 
taking into account the possibilities to use 
such technology within the budget allocation 
limits 
Evaluation of the technology's capability to 
adapt to evolving IT challenges 
Benchmarking of the Agency's deployed 
technologies against industry standards and 
best practices, taking into account the 
possibilities to use such technology within the 
budget allocation limits 
Analysis of the Agency's investment in 
research and development for technological 
advancements, including outsourcing 
practices 
Alignment of managerial positions with 
strategic objectives and needs 
Distribution of projects and tasks to staff 
based on their expertise and skills 
Assessment of the HR strategy's 
effectiveness in achieving organisational 
goals, including e.g. the conditions of 
employment, the distribution of responsibility 
within the teams, career evolution 
perspective, practices of grade allocation and 
the approach to advertise vacancies 
For selected, key profiles/ roles/ functions: 
Correspondence and discrepancies between 
job descriptions and actual job 

Desk research: 
Industry benchmarks and 
comparative studies  
Documentation of R&D 
initiatives and outcomes  
Interviews and focus 
groups with eu-LISA staff  
Interviews with EU-level 
and national stakeholders  
SWOT workshops. 

Interviews and focus 
groups with eu-LISA. 

Desk research: 
Job descriptions and staff 
assignments  

 
members, ensuring coherence and alignment 
with their actual roles. 

EQ 43: How does the tripartite set up of the 
Agency affect the allocation of Human 
Resources? 

The tripartite set-up of the Agency has a clear 
impact on the allocation of human resources, 
and this impact is well-understood and 
effectively managed. 

responsibilities, including based on a review 
of job descriptions against documented tasks 
and assignments 
HR policies and procedures related to job 
descriptions 
Allocation of human resources across the 
three locations (Tallinn, Strasbourg, and 
Brussels), including synergies, coordination 
and duplication of roles and functions 
Staff management strategies and 
implementation 
Distribution of staff roles and responsibilities 
in relation to the tripartite set-up 
Recruitment and retention strategies specific 
to each location 

Documented task and 
project assignments  
HR policies and guidelines  
Interviews and focus 
groups with eu-LISA staff. 
Desk research, e.g. HR 
allocation records and 
reports  
Interviews / focus groups 
with eu-LISA staff. 

Coherence 

Understanding  

Analytical and 
methodological approach 

Potential limitations  

Assessment of internal coherence (i.e. within the Regulation itself), ensuring that it avoids internal overlaps, gaps, grey areas, or 
inconsistencies; 
Assessment of external coherence (alignment of the Regulation's activities and objectives with those of other legal instruments and 
perceptions of external stakeholders, including Member States, the European Commission, EU Agencies, and non-EU entities); 
Analysis to which degree the Regulation's goals are harmonised with broader European and international initiatives and interventions in the 
field. 
Interviews and focus groups with EU-level stakeholders, stakeholders from Member States, and other key actors; 
Documentary Review of relevant documents, including eu-LISA's organisational charts, strategic plans, policy documents, and management 
decisions; 
Review of eu-LISA's procedures for stakeholder engagement and cooperation and assessment the alignment between eu-LISA's 
cooperation activities and the policies and objectives of its stakeholders.  
Analysis of the legal act governing the Information Systems for alignment with eu-LISA's mandate and objectives (including reviews of legal 
texts, regulations, and reports related to the implementation of the legal act) 
Stakeholder interviews and focus groups involve subjective opinions and perspectives which can translate into biases. Project team will 
carefully scrutinise for bias and be transparent about it in the deliverables. 

Table 4 - Coherence – evaluation matrix 

104 

 
 
 
 
 
 
 
 
 
Evaluation 
question 

Sub-questions 

Judgement criteria  

Indicators / information needs (examples) 

EQ 44: To what extent is eu-LISA acting 
in cooperation with the European 
Commission and other EU bodies, as well 
as with Member States and Associated 
Countries to ensure coherence and avoid 
duplication of efforts? 

eu-LISA's actions and 
collaborations demonstrate a 
commitment to coherence with the 
European Commission, EU bodies, 
Member States, and Associated 
Countries, resulting in effective 
coordination and avoidance of 
duplicative efforts 

EQ 45: To what extent are eu-LISA’s 
management structure and activities 
internally coherent for an efficient 
management and operation of the 
Agency? 

eu-LISA's management structure 
and activities demonstrate a high 
degree of internal coherence, 
facilitating efficient management 
and operation of the Agency. 

Types and quality of relationships  
Allocation of responsibilities between eu-LISA and 
Member States, SAC, as well as other EU bodies  
Evidence of collaboration agreements and memoranda 
of understanding with relevant stakeholders  
Assessment of the extent of information exchange and 
coordination with the European Commission, EU bodies, 
Member States, and SAC  
Monitoring of overlapping or duplicate initiatives and 
efforts in the field  
Feedback and perceptions of stakeholders on the 
effectiveness of eu-LISA's efforts in ensuring coherence 
and coordination. 
Level of correspondence of eu-LISA's management and 
organisational structure with its strategic goals and 
objectives  
Interdepartmental coordination and collaboration within 
the Agency  
Effectiveness of communication channels and 
information sharing within the Agency. 

EQ 46: To what 
extent have eu-
LISA’s services 
been coherent with 
the expectations of 
the users of large-
scale IT-systems? 
How does the 
Agency’s service 
correspond to the 
needs and 
expectations of the 
Member States as 

EQ 46.1: To what 
extent have eu-
LISA’s services 
been coherent with 
the expectations of 
the users of large-
scale IT-systems 

eu-LISA's services align with the 
expectations and requirements of 
the users of large-scale IT systems, 
ensuring efficiency and satisfaction. 

Level of user satisfaction regarding the functionality and 
usability of eu-LISA’s large-scale IT systems  
Timeliness and responsiveness of eu-LISA's support in 
addressing user needs and issues  
Correspondence of eu-LISA's services with the 
operational requirements and workflows of users. 

EQ 46.2: How does 
the Agency’s 
service correspond 

eu-LISA's services meet the needs 
and expectations of Member States 
as users of large-scale IT systems, 

Relevance and effectiveness of eu-LISA's services in 
supporting MS’ operational requirements  

Information sources 
(indicative) 

Desk research: Regulations 
and implementing rules, 
internal processes and 
procedures  
Interviews with EU-level 
and national stakeholders  
Interviews and focus 
groups with eu-LISA staff. 

Desk research: 
eu-LISA's organisational 
charts and strategic plans  
Policy documents and 
management decisions  
Interviews and focus 
groups with eu-LISA staff  
Interviews with EU-level 
and national stakeholders. 
Targeted survey with eu-
LISA advisory groups and 
users  
Interviews and focus 
groups with eu-LISA 
advisory groups  
Interviews with EU-level 
and national stakeholders  
SWOT workshops. 

Targeted written surveys 
with advisory groups and 
users  

105 

 
 
 
 
Evaluation 
question 

users of the 
systems? 

Sub-questions 

Judgement criteria  

Indicators / information needs (examples) 

to the needs and 
expectations of the 
Member States as 
users of the 
system? 

facilitating efficient cross-border 
cooperation. 

Contribution of eu-LISA to enhanced cross-border 
information exchange and cooperation among MS  
Improvements in the interoperability and effectiveness of 
MS' systems. 

EQ 47: To what extent the procedures in 
place ensure the coherence of eu-LISA’s 
cooperation activities with the policies and 
activities of its stakeholders? 

Procedures in place effectively 
ensure the coherence of eu-LISA’s 
cooperation activities with the 
policies and activities of its 
stakeholders. 

Review of eu-LISA’s procedures for stakeholder 
engagement and cooperation  
Correspondence between eu-LISA’s cooperation 
activities and the policies and objectives of its 
stakeholders. 

EQ 48: To what extent are the activities of 
the Agency as well as the scope of its 
mandate coherent with the legal act 
governing the Information Systems the 
Agency is responsible for? 

The activities of the Agency and the 
scope of its mandate align 
effectively with the legal act 
governing the Information Systems 
it is responsible for. 

Analysis of the Agency's mandate and objectives as 
defined in the legal act  
Review of the legal framework governing the information 
systems  
Consistency between the Agency's activities and the 
provisions of the legal act. 

Information sources 
(indicative) 

Interviews and focus 
groups with eu-LISA’s 
advisory groups  
Interviews with EU-level 
and national stakeholders. 
Interviews and focus 
groups with eu-LISA 
management, 
implementation, and 
internal services  
Stakeholder interviews with 
EU-level and national 
stakeholders. 
Desk research: 
eu-LISA's mandate 
documents and strategic 
plans  
Documentations of eu-
LISA’s IT systems 
Legal texts and regulations  
Reports and documentation 
related to the 
implementation of the legal 
act  
Records of any legal 
reviews or assessments 
conducted by relevant 
authorities. 

EU Added Value 

Understanding  

Assessment of the added value brought by eu-LISA Regulation in comparison to what could have been achieved without EU intervention  

106 

 
 
 
 
 
 
 
Analytical and 
methodological 
approach 

Potential limitations  

Examination of the extent to which the Regulation has enhanced cross-border cooperation among national competent authorities, particularly 
in terms of information exchange and security  
Evaluation of the tangible benefits, e.g. in terms of operational efficiency and security, that have been realised at the European level through 
the activities and mandates of eu-LISA. 
Extensive review of relevant documents, including eu-LISA Regulation, legislative documents, national-level assessments, reports on Member 
States' capabilities, and records of cross-border cooperation activities  
Stakeholder Interviews to gain deeper insights and perspectives with key actors involved in eu-LISA's activities, including representatives from 
Member States, eu-LISA itself, and other relevant stakeholders  
Focus Groups as forums for structured discussions among stakeholders. 
Potentially limited data availability, time constraints, and data confidentiality. Will be discussed with Commission and mitigated through use of 
multiple data collection methods and analytical approaches. Team includes expert with security clearance  
Potential for biases and limitation of specific data sources, such as varying levels of detail and reliability in national-level reports and 
feedback. Project team is very experienced in this regard and will make any subjective judgments transparent to the Co 
mission  
Inherent challenges in quantifying EU added value, particularly in assessing intangible benefits, may limit the precision of the evaluation's 
findings. 

Table 5 – EU Added Value – Evaluation matrix 

Evaluation 
question 

Sub-questions 

Judgement criteria  

Indicators / information needs (examples) 

EQ 49: Could the objectives set up in 
eu-LISA Regulation have been achieved 
without EU intervention? 

The objectives outlined in eu-
LISA Regulation could not have 
been effectively achieved through 
national-level initiatives alone, 
demonstrating the added value of 
EU intervention. 

Extent of cross-border cooperation and information 
exchange facilitated by the Regulation  
Analysis of the tangible benefits, in terms of operational 
efficiency and security, that have been realised at the EU 
level (see also above). 

Information sources 
(indicative) 

Desk research: 
eu-LISA Regulation and 
related legislative 
documents  
National-level assessments 
and reports on capabilities 
and initiatives of Member 
States  
Records of cross-border 
cooperation and information 
exchange activities  
Interviews and focus groups 
with eu-LISA advisory 
groups  
Interviews with EU-level and 
national stakeholders  
SWOT workshops. 

107 

 
 
 
EQ 50: How has the new mandate of the 
Agency contributed to supporting 
Member States in developing the 
information systems at national level? 

The new mandate of the Agency 
has demonstrably contributed to 
supporting Member States in the 
development of information 
systems at the national level, thus 
fulfilling its role in enhancing 
cooperation and capacity-
building. 

Identification of specific new tasks and activities mandated 
to eu-LISA in support of MS’ information system 
development  
Extent to which MS have actively engaged with eu-LISA in 
seeking assistance and collaboration  
Outcomes and impacts of eu-LISA's involvement in 
supporting national-level information systems development  
Reported improvements in the capacity and efficiency of 
MS' information systems. 

Interviews and focus groups 
with eu-LISA implementation  
Interviews with EU-level and 
national stakeholders  
Case studies. 

108 

 
 
 
 
ANNEX IV. OVERVIEW OF BENEFITS AND COSTS  

Table 1. Overview of costs and benefits identified in the evaluation 
MS administrations  

EU institutions  

Businesses and citizens 

Quantitative  

Comment 

Quantitative  

Comment 

Quantitative 

Comment  

Costs 

Direct 
compliance costs 

recurrent  

Year-to-year increase 
in opening 
contribution in 2018 - 
2024 of approximately 
107% 

Difference between 
opening and closing 
contribution in general 
between 0.41% and 
9.73%.  

Contribution from 
associated countries 

In 2018: EUR 2.2 mil.  
In 2019: EUR 3.7 mil. 
In 2020: EUR 4 mil. 
In 2021: EUR 32 mil. 
In 2022: EUR 16.4 
mil. 
In 2023: EUR 21 mil. 
In 2024: EUR 23 mil. 

Overall contribution to 
eu-LISA budget 

Opening EU 
contribution  
In 2018: EUR 92.7 mil. 
In 2019: EUR 136.4 mil. 
In 2020: EUR 229.8 mil. 
In 2021: EUR 260.8 mil. 
In 2022: EUR 270.2 mil. 
In 2023: EUR 293 mil. 
In 2024: EUR 258.4 mil. 

Closing EU 
contribution: 
In 2018: EUR 93.6 mil. 
In 2019: EUR 138 mil. 
In 2020: EUR 233.4 mil. 
In 2021: EUR 263.9 mil. 
In 2022: EUR 296.5 mil. 
In 2023: EUR 294.2 mil. 
In 2024: EUR 260.1 mil. 

N/A 

N/A 

Member States costs related 
to the implementation of the 
EU IT systems at national 
level (e.g. costs related to 
the connection of national 
systems to EU IT systems). 
These costs could not be 
quantified due to limited 
available information. 
However, the use of 
common IT tools lead to a 
reduction of administrative 
costs and burden on 
individual Member States. 

The Agency also receives 
contributions from 
associated countries to the 
Schengen area, namely 
Norway, Switzerland, 
Iceland and Liechtenstein 
(due to their use of some IT 
systems managed by the 
Agency). 
The contributions of the 
associated countries nearly 
increased 10 times during 
the evaluation period, in line 
with the increase of the 
mandate of the Agency. 

109 

 
 
 
                        
 
 
 
 
 
Enforcement 
costs 

Indiect costs 

N/A 

N/A 

N/A  

Direct benefits 

recurrent 

Benefit
s 

N/A 

Indirect benefits  recurrent 

No enforcement costs 
(e.g. monitoring 
generally carried out 
the EU institutions). 

N/A 

Differences in EU 
opening and closing 
contributions (see 
above) represent an 
increase in EU budget 
available for other 
uses. 
On the other hand, 
cancellations can be 
signal of issues with 
planning of resources 
and activities and/or 
inefficiencies in 
implementation. 

The development and 
management of EU 
level IT large scale 
systems contribute to a 
high level of security 
within the EU and the 
Schengen area. 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

The development and 
management of large scale 
IT systems at EU level 
contributed to a more 
efficient pooling of 
resources, exchange of 
information, increased 
cooperation and 
coordination of activities 
among Member States and 
associated countries.  

N/A 

A centralised management 
of large scale IT systems in 
the area of freedom, security 
and justice benefits Member 
States and associated 
countries indirectly by 
creating a more harmonised 
IT landscape, with 
competent authorities 
having access to timely 
information. 

Businesses and 
citizens operating in 
areas relevant to eu-
LISA activities (IT 
equipment) indirectly 
benefitted from 
centralised 
management of IT 
systems by the 
Agency, i.e. from the 
increased security 
within the Schengen 
area. 
These are indirect 
effects hard to assess. 

N/A 

N/A 

N/A 

110 

 
 
 
 
 
 
 
 
ANNEX V. STAKEHOLDERS CONSULTATION - SYNOPSIS REPORT  

This Annex provides a concise summary of all consultation activities conducted in the context of the “Study to support the evaluation of Regulation (EU) 
2018/1726 on eu-LISA”. 
Specifically, the present  synopsis report offers an overview of the consultation process, including  the stakeholder clusters  targeted and key topics addressed 
through each consultation activity. Moreover, it provides a concise summary of the main results of each consultation activity, aggregating relevant information 
as appropriate. An overview on the methodology and analytical models and tools used.  

Call for evidence  
The feedbacks received on the call for evidence differed widely in their thematic orientation, length and attention to detail.  
Stakeholders from within or representing the interests of the airlines particularly focused their feedback on the implementation of the EES and ETIAS systems. 
The professional organisations expressed support for the objectives of the EES and ETIAS systems but raised concerns about the lack of effective preparedness 
by eu-LISA, which could risk EU connectivity, hamper the passenger experience, and disrupt industry operations. They emphasized the need for operational 
readiness before implementation and offered to work closely with eu-LISA for smooth system implementation. The airlines themselves highlighted the need for 
eu-LISA to allocate well-qualified staff, establish mechanisms to address inefficiencies, and streamline implementation processes. They recommended improving 
communication methods and project management to ensure successful IT system operations.  
Feedback from the public sector, represented only by the Ministry of the Interior of Estonia, focused on  potential amendments to the Agency's current legal basis. 
The recommendations were not included in the feedback itself but in an additional attached document, which included an analysis prepared for the Ministry.84 
Feedback received from the EU and non-EU citizens covered a wide range of different topics, including (i) the good performance of eu-LISA in managing systems 
like VIS, SIS, and Eurodac, (ii) suggestions that the Agency's activities might become insufficient given the current migration and security challenges, (iii) the 
need for improved project management and better stakeholder integration, and (vi) recommendations for enhanced operational readiness to address the challenges 
faced by eu-LISA in managing the large-scale IT systems.  
The feedback was noted within the context of the study but did not provide significant additional inputs for the study team. The main findings were rather derived 
from the other consultation methods outlined above, also including many of the stakeholder groups who have answered to the call for evidence.  

Consultation methods and stakeholders consulted  

84 PwC (2020). Future analysis “Governance model for the European Union IT Agency (eu-LISA)”. Government Office of Estonia,  

111 

 
 
 
 
 
 
The consultation activities performed in the context of the present Evaluation Study include exploratory and expert interviews (individual and in focus groups), 
workshops and a survey. Overall, consultation activities reached around 500 stakeholders, including more than 114 contributors via interviews/focus groups 
(including  exploratory  interviews),  roughly  40  participants  in  SWOT-workshops,  and  353  respondents  via  the  online  survey.  The  overall  objective  of  the 
consultations conducted was to better understand the current functioning of eu-LISA, and where challenges arise or where the performance of the Agency could 
be improved with regard to its current mandate.  
The table below provides an overview of the involvement of each stakeholder category into the consultation activities conducted. The stakeholder groups consulted 
include: officials from the European Commission; stakeholders from within eu-LISA’s staff and the Agency’s governance bodies (Management Board, Advisory 
Groups, Programme Management Boards); stakeholders within other EU institutions and bodies; private sector carrier organisations (air, land, sea) affected by 
the systems managed by eu-LISA; contractors of eu-LISA; national data protection authorities; and finally users of the large-scale IT-systems.  

Table 1 - Overview on consultations conducted by stakeholder group 

Activity 

Officials from COM  

eu-LISA  staff  and 
governance bodies 

Users  of 
systems 

the 

IT-

Other 
EU 
stakeholders 
agencies) 

level 
(e.g. 

sector 

Private 
actors 
(contractors, 
carriers) 

National 
protection 
authorities 

data 

Exploratory 
interviews 
Interviews  with  eu-
and 
staff 
LISA 
governance bodies 
Interviews 
focus 
groups  with  EU  and 
level 
national 
stakeholders 
Online survey 
SWOT-Workshops 

/ 

✓ 

✓ 

✓ 

✓ 
✓ 

✓ 

✓ 

✓ 
✓ 

✓ 

Interviews  
Throughout  the  project,  the  Study  Team  performed  two  rounds  of  interviews,  namely  (i)  six  exploratory  interviews  with  officials  from  the  European 
Commission (DG HOME AND DG JUST) (ii) 108 expert interviews with stakeholders from within (i) eu-LISA staff, and the governance bodies of eu-LISA, 
and (ii) EU-level and national level stakeholders, comprising the European Commission (in particular DG HOME), other EU-level institutions and agencies, 
including Frontex, Europol, and the EPPO, as well as private sector actors, in particular contractors of eu-LISA. All interviews were semi-structured and conducted 

112 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
by  means  of  interview  guides  that  were  approved  by  the  Commission  as  part  of  the  preparation  of  the  inception  report  and  coined  towards  the  respective 
stakeholder group. 

Exploratory interviews 
In the inception phase of the study, the study team conducted six one-hour exploratory interviews with stakeholders from within the European Commission. This 
included stakeholders from, amongst others, the following units of DG HOME and DG JUST: DG HOME Directorate B (i.e. DG HOME B3, DG HOME E1) 
and DG JUST H4 and DG JUST A4. Most interviews were conducted virtually via MS-Teams, with one exception where the interview was held in-person.  
The interviews aimed at deepening the study team’s understanding of the functioning of eu-LISA, and to finalise the evaluation framework, intervention logic 
and data collection instruments. Moreover, the interviews allowed the study team to collect further documentation and identify additional data sources.  
The  interviews  followed  a  semi-structured  approach  to  provide  sufficient  guidance,  ensuring  that  the  relevant  aspects  were  covered  while  leaving  sufficient 
flexibility  to  tailor each  conversation to  the focus area and expertise of the interviewee. To allow for the semi-structuring of the interviews, the  study team 
prepared interview guidelines in advance of the consultations, which followed the five evaluation criteria provided in the EU Commission’s Better Regulation 
Guidelines85:  Effectiveness,  Efficiency,  Relevance,  Coherence,  and  EU  Added  Value.  In  addition,  the  study  team  prepared  questions  following-up  on  the 
recommendations provided in the previous evaluation of the eu-LISA Regulation conducted in 2016/2017, as well as on lessons learned and the way forward.  

Expert interviews 
During the fieldwork phase of the study, overall, 108 interviews were conducted with (i) stakeholders within eu-LISA and the governance bodies; and (ii) EU-
level  and  national  level  stakeholders.  Most  of  the  interviews  were  conducted  virtually  via  MS-Teams,  however,  the  study  team  used  the  opportunity  of  the 
exploratory visits to Tallinn and Strasbourg to also conduct interviews with eu-LISA staff in-person (overall 28 interviews were conducted in-person). The main 
goal of these interviews was to gather detailed insights into the views of eu-LISA staff and members of the governance bodies, to better understand the current 
functioning of the Agency as well as existing challenges and areas where room for improvement exist.  
In total 70 interviews (of the overall 108 conducted interviews, i.e. roughly 65% of all expert interviews) were conducted with stakeholders within eu-LISA and 
its  governance  bodies.  Of  the  70  interviews,  46  interviews  (66%)  were  conducted  with  eu-LISA  staff  whereas  24  interviews  (34%)  were  conducted  with 
representatives from the governance bodies. An overview on the number of interviews per department/unit is found below in the table. 

Table 2 - Stakeholders reached within eu-LISA staff 

85  European  Commission  (2021)  Better  Regulation  Guidelines:  Available  at:  https://commission.europa.eu/law/law-making-process/planning-and-proposing-law/better-regulation/better-

regulation-guidelines-and-toolbox_en, lastly accessed 13 May 2024.  

113 

 
 
 
 
 
Department/unit/type of stakeholder 

Management (Executive Director, Deputy Executive Director) 
Data Protection Officer 
Internal Audit Capability 
Security Unit  
Corporate Governance Department  
Corporate Services Department  
Operations Department  
Liaison Office  
eu-LISA staff representation  
SUM:  

Total number of stakeholders with whom interviews were 
conducted 
2 
1 
1 
1 
5 
9 
18 
3 
6 
46 

On the side of the governance bodies, the remaining 24 stakeholders contacted were either from the Management Board or from the Advisory Groups and the 
e-CODEX PMB. Overall, 12 stakeholders were contacted respectively for each of the two bodies, covering different Member State perspectives. As concerns 
the Advisory Groups and e-CODEX PMB, at least one stakeholder could be reached for each AG and for the e-CODEX PMB. Given that a maximum of two to 
three interviews per AG and e-CODEX PMB were initially planned and that the same stakeholder groups were also contacted via the online survey, this is a 
satisfactory result.  
As for the exploratory interviews, the one-hour interviews were conducted following a semi-structured approach, following pre-agreed upon interview guidelines. 
For all interviews, questions were clustered by the three main dimensions covered by the evaluation, which were provided in the ToR. These included (i) the 
mandate, tasks and objectives of the Agency, (ii) the governance and location of the Agency, and (iii) the management of resources. In addition, the study team 
drafted specific question blocks for each stakeholder group to allow for deep-dives concerning specific topics according to the expertise of the interviewees. 
Before the interviews, team members were briefed to ensure familiarity with the interview guides and approach to conducting the interviews. 
As concerns the remaining 38 EU and national stakeholders reached via interviews/one focus group, an overview of the stakeholders contacted can be found in 
the table below.  

Table 3 – EU and national level stakeholders reached 

Type of stakeholder 

Total number of stakeholders with whom interviews were conducted 

EU agencies and bodies 
EU Commission (DG HOME units A4, B3, B4, C3, H4) 
Europol  

6 
3 

114 

 
 
 
Type of stakeholder 

Total number of stakeholders with whom interviews were conducted 

Frontex  
EPPO  
National level stakeholders  
Contractors of eu-LISA  
Carriers (air, land, sea) – conducted in one focus group 
SUM  

4 
1 

4 
20 
38 

For the interviews with EU Agencies and bodies as well as the contractors, interview guidelines were drafted. Similar to the  above interviews, the interview 
guides were approved by the Commission as part of the submission of the inception report and coined towards the respective stakeholder group, covering aspects 
such as administrative procedures, cooperation with eu-LISA, questions about eu-LISA’s current operations as well as future needs and developments to consider. 
As for the other interviews, the study team’s interviewers were briefed prior to the start of the fieldwork phase to ensure compliance with the guidelines.  
The one-hour focus group was conducted on 22 February 2024 via MS-Teams with 20 participants representing air (five representatives), land (one representative) 
and sea carriers (11 representatives), as well as three other representatives from other carrier groups. The focus group was conducted with the help of the online 
whiteboard tool MURAL, which allowed the participants to contribute their ideas and perceptions live during the focus group. In addition, the study team allowed 
for more detailed exchanges in two break-out sessions. In order to foster the discussions, the study team prepared a number of guiding questions prior to the focus 
group.  

Online survey 
A written online survey was distributed among three different stakeholder groups in order to gather information on the functioning of eu-LISA, experiences made 
with the large-scale IT-systems operated by the Agency, and on the Agency’s adherence to data protection standards and guidelines. Stakeholders contacted 
comprised: (i) Advisory Group members and the e-CODEX PMB; (ii) The current and future users of the different IT systems (i.e. national competent authorities 
(NCAs), such as national law enforcement authorities, border control authorities and immigration authorities as well as the judiciary); and (iii) national supervisory 
authorities. The surveys covered aspects relevant to these groups, containing filter questions to ensure that only relevant questions were posed to the stakeholders. 
The different surveys were approved by DG HOME as part of the submission of the inception report. Overall, 353 answers were recorded through the survey. An 
overview of the quality of responses to the targeted written surveys can be found in the main body of the report in Table . 

As concerns the survey distributed among AG members and the e-CODEX PMB, the study team reached 175 stakeholders via the survey. The figure below 
provides an overview on the distribution of respondents within the AGs and the e-CODEX PMB. Most answers were received from members of the EES-ETIAS 
Advisory Group whereas the e-CODEX PMB, as to be expected due to its size, provides the smallest number of answers. It should be noted that the survey 

115 

 
 
 
 
participants were able to provide multiple answers to this question, which is why the number of respondents in this specific question is higher than the overall 
number of stakeholders reached (175, see above). 

Figure 1 - Share of AG and e-CODEX PMB survey participants by AG/PMB (n=179) 

For SIS, VIS and Eurodac, the questions focused on the operation and functioning of the systems, whereas the questions for ECRIS-TCN, EES-ETIAS and e-
CODEX had a more forward-looking focus on development-related aspects of eu-LISA’s work. For participants of the surveys related to the judicial systems, 
there were additional questions on the principle of the respect of the independence of the judiciary and the handover / takeover process of e-CODEX from a 
consortium to eu-LISA. For all respondents, the survey concluded with a block of questions on organisational matters and governance, as well as future challenges 
and needs.  
Regarding the survey distributed among current and future users of the large-scale IT-systems, the study team reached overall 132 stakeholders via the 
survey.  Even  though  not  all  EU  Member  States  and  Schengen-Associated  Countries  (SACs)  were  reached  via  the  survey,  the  study  team  roughly  reached 
respondents in three quarters of the countries contacted (77%, 24 of 31). An overview of the participating countries and the number of responses recorded can be 
found below.  

116 

 
 
 
 
 
 
 
Figure 2 - Share of user survey participants by country (n=90) 

The overwhelming majority of participants were members of national SIRENE Bureaux (33%, 30 of 90 respondents who answered the question), followed by 
police officials (27%, 24 of 90), immigration, naturalisation and visa issuing authorities (9%, 8 of 90), border control (7%, 6 of 90) and asylum authorities (4%, 
4 of 90). In addition, stakeholders from judicial authorities (2), other national law enforcement authorities (2), Europol (2) and the EPPO (1) were reached.  
Given  that  the  survey  focused  on  currently  operational  systems,  participants  were  asked  which  system  they  wish  to  provide  answers  for.  The  majority  of 
respondents indicated that they would like to answer questions for SIS (71%, 64 of 90 respondents who answered the question). Second most answers were 
provided for VIS (24%, 22 of 90), followed by Eurodac (18%, 16 of 90) and lastly DubliNet (6%, 5 out of 90). It should be noted that respondents had the 
opportunity to provide answers for multiple systems, which is why the percentages do not add up to 100%. The questions within the survey focused on the use 
and satisfaction of the respondents with the IT-systems and also included additional questions on system development and the respect of the principle of the 
independence of the judiciary where applicable. As for the AG and e-CODEX PMB survey, the user survey concluded with a block on future challenges and 
needs.  

117 

 
 
Lastly, the study team distributed a survey among national supervisory authorities, which reached 46 stakeholders. Stakeholders were able to provide answers 
to  questions  concerning  the  General  Data  Protection  Regulation  (GDPR)86  other  legislation,  as  well  as  specific  questions  as  concerns  the  data  protection 
requirements and challenges of the single operational IT-systems (SIS, VIS, Eurodac, DubliNet). Overall, 65% (20 of 31) of contacted Member States and SACs 
provided at least one answer to the survey. Answers differed between one and three per country, with respondents from Germany providing most answers as a 
single country. In terms of expertise, most participants indicated to have core expertise for the SIS (83%, 24 of 29), followed by the VIS (65%, 19 of 29), Eurodac 
(45%, 13 of 29) and DubliNet (only one participant indicated to have core expertise for this system). As above, the percentage does not up add to 100% due to 
participants’ possibility to choose multiple answers.  

SWOT-workshops  
The  last  consultation  method  used  in  the  study  constituted  the  conducting  of  six  SWOT  workshops,  aiming  at  identifying  current  Strengths,  Weaknesses, 
Opportunities and Threats within eu-LISA’s operations. The thematic focus of the single workshops was clustered by eu-LISA’s Advisory Group structure, the 
operational status and the overall domain of the topics addressed in the respective workshop, leading to the following overall clustering of workshops conducted:  

-  ECRIS-RI, ECRIS-TCN and e-CODEX (21.02.2024) 
-  SIS (26.02.2024) 
-  VIS (28.02.2024) 
-  Eurodac and DubliNet (01.03.2024) 
- 
Interoperability (04.03.2024) 
-  EES and ETIAS (07.03.2024) 

Overall, roughly 40 stakeholders were reached within the context of the workshops, with differing participation rates depending on the workshop. Participation 
rates were particularly strong within Western and Middle European countries, including multiple participants from Austria, Czechia, Hungary, Germany, and the 
Netherlands. In addition, participants from France, Iceland, Italy, Lithuania, Luxembourg, Malta, Norway, Poland, Portugal, Slovakia, Spain, and Switzerland 
also participated in some of the workshops respectively.  
The workshops were all conducted virtually via MS-Teams with the help of the online whiteboard tool MURAL, which served as a platform to collect inputs 
from the participants and structure the outputs derived. All workshops followed the same structure: First, the study team provided a short overview of the contents 
of the study, sharing a PowerPoint presentation with participants via screen share. Following the introduction, participants received instructions on how to use 
MURAL and a short tour de table were held. Afterwards, participants were able to contribute to the SWOT analysis by adding notes (“post-its”) where they saw 
specific strengths, weaknesses, opportunities and threats in a number of different overall categories such as technology, user perspective, governance, fundamental 

86 Regulation (EU) 2016/679 of the European Parliament and the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free 

movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation), 4.5.2016, L119/1.  

118 

 
 
 
rights, processes, or other aspects. Lastly, the notes rated most important by participants were used to derive recommendations on how eu-LISA could further 
foster its strengths and upcoming opportunities as well as how to mitigate the effects of weaknesses and threats. Each workshop was concluded with a short 
summary by the study team.  

Call for evidence  
In the context of the study, the Commission has launched a call for evidence. Calls for evidence are used in the context of planning and preparing law and describe 
the problem to be tackled and objectives to be met, explains why EU action is needed, outlines policy options and, where applicable, describes the main features 
of the consultation strategy. In response to the call for evidence, private actors are able to provide feedback and recommendations on the current status of the 
proposed law. 
The call for evidence concerning the evaluation of eu-LISA’s mandate started on 18 August 2023 and was open for roughly a month until 15 September 2023. 
Overall, the Commission received feedback from 15 people, of which 14 feedbacks were published and one remained unpublished. Feedback givers were able to 
file their feedback online via the Better Regulation Portal.  
The sources of the feedback included in particular actors from the carrier sector (exclusively airlines or professional organisations representing airlines, 6 of the 
15 feedbacks). In addition, a number of private persons (EU and non-EU citizens) have provided feedback, without closer defining which organisation they belong 
to (7 of 15 feedbacks). Lastly, one actor from the Ministry of the Interior in Estonia and a Spanish non-governmental organisation have participated in the call. 
In general, the feedback received covered a wide range of EU and non-EU countries, including actors from Austria, Belgium, Czechia, Estonia, Germany (2 
actors), Israel, Romania, Slovakia, Spain (3 actors), Sweden, and the United States of America (2 actors). 

Results of the consultation activities  
The following part of the synopsis report provides a high-level overview of the main results of the consultation activities.  

Interviews 

Exploratory interviews 
The  exploratory  interviews  aimed  at  deepening  the  study  team’s  understanding  of  the  functioning  of  eu-LISA,  and  to  finalise  the  evaluation  framework, 
intervention logic and data collection instruments. The main take-aways comprised:  

-  The rise in migration to the EU and the international character of organised crime continuously underline the importance of the Agency.  
-  The  Agency  provides  considerable  EU  Added  Value,  given  that  single  Member  States’  approaches  would  create  significant  administrative  and 

cooperation burden.  

119 

 
 
 
 
 
-  The Agency is overall perceived as professional and effectively aligns its objectives with its operations. Its core business, the operation of the large-

scale IT system, works smoothly and without major technical incidents.  

-  The transparency and insufficient long-term planning of budgetary and organisational procedures were considered areas for improvement.  
-  The development of new IT systems through the Agency as well as the current outsourcing approach were considered to be key challenges by most 

interviewees.  

-  The difference between the traditional eu-LISA systems in the area of home affairs compared to the justice systems was emphasised. Questions were 

raised on how effective eu-LISA will be in transferring the current e-CODEX system from a consortium to the Agency.  

Expert interviews 
The main findings of the expert interviews are as follows:  

Dimension 1: Mandate, tasks and objectives of the Agency  
Mandate and objectives: 

-  Most stakeholders confirmed the overall adequacy of the mandate of eu-LISA. In particular for eu-LISA staff, the mandate and objectives tended to be 

considered as “given” and the focus lies on fulfilling the mandate.  

-  The flexibility of the mandate was considered to be rather restrictive by most interviewees within eu-LISA who addressed this topic, which was stated 

to potentially lead to problems in terms of timely and appropriate reactions to unforeseen circumstances.  

-  Concerning  the  cooperation  with  international  organisations  (Article  43  of  the  eu-LISA  Regulation)  and  third  countries,  some  stakeholders  from  the 
Agency voiced that this provision has not been used successfully. The provision specifies that such a collaboration can only take place if provided by a 
Union legal  act  and following the approval  process that involves the Management Board (MB) and the Commission.  However, based on the current 
provision, past attempts have not been successful. 

-  A majority of the justice stakeholders confirmed that the mandate is adequate for the justice systems, however, they voiced concerns as regards the respect 
of the principle of the independence of the judiciary, given that eu-LISA might serve as a connector between the executive branch and the judiciary, 
thus not fully respecting the principle. In addition, justice stakeholders emphasised their wish to be more strongly represented in the Agency’s governance 
bodies. 

Tasks: 

-  Article 15 on piloting and testing and Article 16(4) on ad hoc support to Member States were considered very relevant by most interviewees who answered 
questions on topic, based on that several requests were made by different Member States according to these Articles in the past years. However, some 
interviewees from eu-LISA noted that the implementation of these provisions was hindered due to the complex and time-consuming approval processes 
specified in the provisions. On this basis, the requesting Member States in the past changed their mind and found other solutions, as it took too long to 
secure the needed support by eu-LISA. 

120 

 
 
 
-  As concerns the implementation of its tasks, a majority of stakeholders from eu-LISA and governance bodies noted that the management of existing 
system and communication infrastructure as well as the provision of training works relatively well. Significant room for improvement was identified with 
regard to the development of new systems (delays, extensive outsourcing), the monitoring of research and support function to the Member States.  

Dimension 2: Governance, location and organisation of the Agency  

Governance:  

-  Overall, the work of the governance bodies is considered important and running smoothly by a majority of stakeholders within and outside of the Agency, 

including the cooperation between different bodies and with eu-LISA.  

-  As concerns areas where room for improvement was identified, many interviewees highlighted a high administrative burden of each governance body 

- 

as well as potential risks of duplications and/or contradictions as key challenges for the Agency.  
In addition, many stakeholders from the governance bodies raised that the practical work of the governance bodies is sometimes challenging due to the 
varying levels of expertise of its members.  

-  As concerns coordination with the Member States, most stakeholders confirmed well-working coordination mechanisms, with only very few exceptions 

highlighted.  

-  Finally,  several  interviewees  from  the justice area indicated that increasing challenges are expected to  arise in  the  MB due to the increased  future 
importance of the justice area. In particular, this has brought up questions relating to the potential inclusion of justice profiles into the so far mainly 
home affairs focused MB. 

-  The efficiency of governance could be improved through shifting from the current silo-based governance approach towards a sector-specific governance 

model, by clustering governance bodies, and establishing formalised channels for communication and collaboration.  

Location:  

-  As concerns the political dimension, most stakeholders from the Agency and its governance bodies showed a general understanding for and acceptance 

of the decision to locate the Agency’s sites in three Member States (with the liaison office being established in Belgium, a fourth Member State). 

-  With  regard  to  the  operational  dimension,  a  large  majority  of  interviewees  stated  that  in  particular  the  split  of  the  locations  between  Tallinn  and 
Strasbourg, as well as between Strasbourg and St. Johann im Pongau, has de facto caused operational challenges and roadblocks over the past five years. 
The  most  frequently  mentioned  challenges  concerning  Tallinn-Strasbourg  were:  (1)  Detachment  of  the  corporate  services  from  the  operations;  (2) 
Remoteness and practical reachability of both Tallinn and Strasbourg; and (3) The secluded location of the Central Unit and the rented office space in 
Strasbourg. 

-  Addressing the attractiveness dimension, the location of the Agency’s sites was criticised by some interviewees from eu-LISA as a major reason for a 
comparatively limited attractiveness of eu-LISA to potential applicants. Tallinn was considered unattractive due to its location far north and the cold 
weather conditions. In Strasbourg, the Central Unit is located at the southern outskirts of the city in a residential neighbourhood at the end of a street next 
to a farmland with very limited lunching/dining options or opportunities to run errands. 

121 

 
 
 
Organisation:  

-  The clarity of the organisational set-up and structure of the Agency within departments, units and sectors was considered by stakeholders within and 

outside of the Agency as acceptable.  

-  At the same time, the perceived disconnect between corporate services (in Tallinn) and operations (in Strasbourg) were viewed as problematic, creating 

communication and cooperation difficulties between both departments.  

-  Some stakeholders from the Agency criticised the lack of clear role descriptions and responsibilities, which might lead to overlaps in the work of eu-

LISA.  

-  As concerns the size of the organisational entities, several stakeholders from eu-LISA, but also from the governance bodies voiced concerns about the 
current organisational structure, which may not be adequately capable of managing the number of staff allocated to each of the units and sectors. From a 
practical perspective, interviewees argued that the size of the units and sectors is too large to be managed efficiently. 

Dimension 3: Management of human, technical, and financial resources 
Human resources 

-  The available human resources are the Agency’s most important asset. This was also acknowledged by several interviewees within the Agency, who 
specifically pointed out that the collaboration with their colleagues and peers is one of their most important sources of motivation to work at eu-LISA. 
-  Based on the interviews conducted, there are challenges with regard to the career development within eu-LISA: With only a finite number of positions 

available at different levels, it seems that challenges exist for staff to ‘climb up’ the career ladder. 

-  With regard to the challenges in recruitment, retention and development of HR, the following factors were mentioned most frequently by interviewees 

within eu-LISA and in the governance bodies:  

o  Strategic factors: Availability of positions; Competition with private industry; Purpose and mission of the Agency (i.e. its branding); Location 

and facilities of the Agency; 

o  Contractual factors: Necessity of security clearance; Necessity of short-term contracts; Salary and compensation package; 
o  Operational considerations: Ability to work with cutting-edge technology; Rigid ways of working. 

-  Most stakeholders consulted also noted that the Agency has undertaken efforts to mitigate the HR-related challenges, comprising e.g. the digitalisation of 
the HR process, a restructuring of the organisation of HR management, changes in the organisational development (launch of cultural transformation 
projects, updating of existing role descriptions), as well as improved employer branding.  

-  Lastly, it was noted by some stakeholders within eu-LISA that the Agency must provide a fully-fledged HR strategy in the future.  

Technical resources  

-  According  to  various  stakeholders  from  within  and  outside  the  Agency,  eu-LISA  is  aims  at  adopting  state-of-the-art  technologies,  signifying  a 

noteworthy pursuit of innovation and operational efficiency.  

122 

 
 
-  This being said, many stakeholders noted that there is untapped potential in the fields of AI-based applications and cloud technologies, notably due to 

restrictions in the mandate.  

-  Stakeholders from within the Agency and its governance bodies also identified challenges in the software development life cycle. The layered approach 
where  testing,  delivery,  and  application  maintenance  emerge  as  distinct  layers  managed  by  different  contractors  under  various  framework  contracts, 
introduces complexities in coordination and communication across the different phases of the SDLC. This exacerbates the challenge of achieving seamless 
end-to-end delivery and needs to be addressed with specific approaches and tooling.  
In addressing these problems, eu-LISA has shifted from a more traditional waterfall model towards more agile, iterative and collaborative methods. At 
the heart of agile implementation at eu-LISA is the strategic formation of 'squads.' These squads are not only cross-functional teams; they are a dynamic 
fusion of diverse expertise and experience that operate with focused objectives, autonomy, and collective ownership. 

- 

-  Given its extensive outsourcing, the Agency sometimes struggles with the retention of technical expertise and experience within eu-LISA.  

Financial resources 

-  The financial planning and reporting are based on traditional year-by-year public budgetary processes. Within this context, many stakeholders with 
expertise in the area highlighted that the collaboration between the Commission (in particular DG HOME) and the Agency has positively improved over 
the last years, but it has still sometimes been challenging for eu-LISA to provide the Commission with the requested information in time.  

-  The growth of tasks of the Agency has also led to the situation of managing and updating three systems and, in parallel, develop additional new ones 
from scratch. It seems that appropriate planning capabilities were only established over time, which led to gaps between the actual and planned progress 
over time. 

-  With specific regard to multi-annual planning, stakeholders with specific expertise in the field pointed out that this is an area for improvement, both 
from a strategic perspective and the concrete tooling. Over the previous years, too much emphasis was given to the annual planning, which, in turn, led to 
a  lack  of  strategic  planning  of  resources  over  time.  This  led  in  the  past  to  the  Agency  having  asked  for  too  many  financial  resources,  which  could, 
ultimately, not be used entirely. 

-  As concerns the Agency’s outsourcing practices, many stakeholders within the Agency and even on the side of external contractors themselves criticised 
aspects  such  as  legislative  predicament,  the  outsourcing  strategy  (or  lack  thereof),  the  complexity  of  the  outsourcing  approach,  the  contractual 
management, and the volume of outsourcing.  
In  particular  justice  stakeholders  noted,  that  in  the  context  of  e-CODEX,  the  respective  Regulation  states  that  “eu-LISA  shall  commit  the  resources 
provided (…) for the e-CODEX system in their entirety to its operation” (Article 14(2) of the e-CODEX Regulation). Hence, eu-LISA resources allocated 
to  e-CODEX  are solely  focused on this specific  IT system  itself in  order to  ensure the respect  of the principle of the independence of the judiciary. 
However, (i) currently there are five FTE working on e-CODEX which is insufficient and (ii) the legal mandate impedes efficiency as resources are rigid. 

- 

Online survey 
Below, an overview of the main findings clustered by the Better Regulation Guideleines’ evaluation criteria is found. 

123 

 
 
 
Effectiveness 

-  As concerns eu-LISA’s general objective, the Agency is considered effective in contributing to a high-level of security within the AFSJ. The primary 
reason for this assessment is the smooth functioning of the existing IT systems. However, in view of the current room for improvement concerning the 
development of new IT systems, the effectiveness of the Agency could be even higher. 

-  As concerns eu-LISA’s effective management of the large-scale IT-systems, a majority of respondents consider eu-LISA to contribute to a coordinated, 
cost-effective and coherent IT environment at Union level. 24 out of 35 (69%) AG and PMB members who answered this specific question indicated in 
the online survey that eu-LISA’s management of the systems contributes to a ‘large’ or ‘very large’ extent to a coordinated, cost-effective and coherent 
IT environment.  

o  As concerns the overall levels of performance, 86% (64 out of 74) of respondents from eu-LISA have said to be ‘very satisfied’ or ‘satisfied’ 
with the performance. Users’ assessment resembles this largely, with 85% (65 out of 75) of respondents indicating they are ‘very satisfied’ or 
‘satisfied’ with the performance  

o  Within the single systems, performance of SIS and VIS are rated best among AG and e-CODEX PMB members (for both roughly 50% very 

satisfied), whereas Eurodac only receives 40% (29 of 74) ‘very satisfied’-ratings.  

o  Openness to feedback is also considered sufficient, however, the overall ratings are not as good as for system performance with only 53% of 

respondents (19 of 36) considering eu-LISA to be open to feedback to a ‘large’ or ‘very large’ extent. 

o  The takeover of e-CODEX was handled successfully to a ‘very large’ or ‘large’ extent according to 65% (17 of 26) of stakeholders from the 

AGs and e-CODEX PMB consulted. 

o  Ratings  for  data  protection  by  consulted  national  supervisory  authorities  were  also  largely  positive,  with  80%  (19  of  24)  of  respondents 
considering eu-LISA to be largely in accordance with the GDPR87, and 75% (18 of 24) of respondents considering eu-LISA to be largely in 
accordance with the Law Enforcement Directive.88 

Efficiency 

-  With regard to its efficiency, the operational management of existing IT-systems on the side of eu-LISA was rated positively by most respondents from 
the AGs and e-CODEX PMB consulted, with ‘very satisfied’ and ‘satisfied’ ratings accounting for at least 70% (n=74) in all categories asked (e.g. system 
availability, continuity of services, timeliness of service delivery, support responsiveness,  quality and frequency of communication, reaction times to 
security issues) 

87 Regulation (EU) 2016/679 
88 Directive  (EU) 2016/860 of the European Parliament and the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data by competent 
authorities for the purposes of the prevention, investigation, detection or prosecution of criminal offences or the execution of criminal penalties, and on the free movement of such data, and 
repealing Council Framework Decision 2008/977/JHA, 4.5.2016, L119/89.  

124 

 
 
 
-  As concerns the performance of eu-LISA’s contractors, most respondents from the AGs and the e-CODEX PMB are ‘very satisfied’ or ‘satisfied’ (for 
each category asked at least 70% of respondents, n=38) with the performance, even though differences were seen between the categories. Respondents 
were comparably strongly or moderately dissatisfied with the response time (23%, 9 of 38) and limitations of impact on the operations (26%, 10 of 38) 
on the side of the contractors. Best  scores were  reached for incident  management, with  an overall 84% (32 of 38) of respondents  being  either ‘very 
satisfied’ or ‘satisfied’ with the work of the contractors. Even though work of the contractors was rated comparably high, it is notable that the Agency’s 
overall outsourcing strategy is rather rated unfit to deliver its tasks sufficiently, with 74% (44 of 60) of AG and e-CODEX PMB respondents rating the 
strategy successful to ‘a moderate extent’, ‘low extent’, or ‘not at all’.  
In general, the stakeholders in the AGs and e-CODEX PMB wished for eu-LISA take over more responsibilities from external contractors, with 45% of 
respondents (21 of 47) thinking to a ‘very large’ or ‘large’ extent that the Agency should do so.  

- 

-  As concerns the development of new systems, the AG and e-CODEX PMB members participating in the survey painted a mixed picture of eu-LISA’s 
efficiency. Roughly 30% of respondents (14 of 68) were either ‘very satisfied’ or ‘satisfied’ with the work of eu-LISA, whereas nearly half of respondents 
(45%,  30  of  68)  were  only  satisfied  to  a  moderate  extent.  25%  of  respondents  (17  of  68)  were  either  ‘dissatisfied’  or  ‘not  at  all’  satisfied  with  the 
performance of the Agency.  
In terms of staff covering the required competencies to make the Agency work efficiently, most respondents (76%, 55 of 72) from the AGs and e-
CODEX PMB agree that the staff qualifications available satisfy the needs of eu-LISA to ‘a very large’ or ‘large’ extent’, thus painting a positive picture 
on the recruitment practices of the Agency.  

- 

-  Asking about the efficiency of the governance set-up of the Agency, stakeholders from the AGs and the e-CODEX PMB paint a mixed picture. Even 
though very low and low ratings remain an exception, roughly a third of respondents (20 of 59) consider the set-up only to a ‘moderate’ extent efficiently. 
Nonetheless, a majority of 34 of 59 stakeholders (58%) answered the question with a positive result (‘very large’ or ‘large’  extent). A more nuanced 
picture is painted by stakeholders  as regards the functioning of the internal organisation, with decision-making procedures and escalation procedures 
highlighted by at least 30% of respondents as only ‘partially’ or ‘not at all’ appropriate.  

Coherence 
The internal and external coherence of the eu-LISA Regulation was assessed entirely through qualitative methods, which is why no survey item is included here. 

EU Added Value  

-  The added value of the eu-LISA Regulation compared to what Member States could have achieved alone has clearly been confirmed in this evaluation. 
Without a single body such as eu-LISA responsible for several IT systems in the AFSJ, the current IT landscape would be drastically more fragmented, 
and it would likely be more challenging to ensure security. 

-  As already presented above, 69% of the surveyed AG and e-CODEX PMB indicated that eu-LISA’s management of the systems contributes to a large or 
very large extent to a coordinated, cost-effective and coherent IT environment at EU level in the AFSJ, which the Member States could not have achieved 
alone. 

125 

 
 
 
- 

In terms of support for the MS in developing the information systems on the national level, the AG and e-CODEX PMB members paint a more mixed 
picture, with only 45% (26 of 58) of respondents indicating that eu-LISA supports the MS to a ‘very large’ or ‘large’ extent.  

Relevance 

-  The continued overall relevance of eu-LISA’s mandate and objectives has been confirmed in this evaluation. Indeed, the relevance of the Agency 
has  become  even  more  important,  considering  external  developments  such  as  the  increased  mobility  of  EU  citizens  and  third-country  nationals, 
technological developments and increased risks from cyber-attacks as well as threats from ongoing geopolitical developments and conflicts (e.g. in Ukraine 
and the Middle East). 

-  As concerns how well eu-LISA performed in view of existing challenges, 71% of respondents (27 of 38) replied that eu-LISA performs well to a ‘very 

large’ or ‘large’ extent, underlining the relevance of the Agency with a view to upcoming challenges.  

-  As concerns which challenges are considered particularly important, stakeholders from the AGs and the e-CODEX PMB pointed out the expansion of eu-
LISA’s responsibilities (54%, 43 of 79), the ‘war for talent’  (45%, 36 of 79) and ensuring sufficient  budget  and financing (43%, 34 of 79) as ‘very 
important’ challenges.  
In order to keep up its current relevance, stakeholders highlighted the need to implement stronger feedback and cooperation mechanisms, increasing eu-
LISA’s attractiveness as an employer, increasing the Agency’s pilot testing capabilities and adapting the current organisational structure (all considered 
at least to 70% ‘very important’ or ‘important’).  

- 

-  To improve its current relevance, 53% of respondents from the AGs and e-CODEX PMB currently consider eu-LISA to take feedback provided into 

consideration to a ‘very large’ or ‘large’ extent, pointing out the need for eu-LISA strengthen its current feedback mechanisms.  

SWOT-workshops  
The study team conducted six SWOT workshops, focussing on the experiences eu-LISA governance body members and national users made with eu-LISA’s 
large-scale  IT  systems.  The  results  are  presented  in  in  the  following  (clustered  by  workshop).  After  finalising  the  SWOT  analysis,  the  participants  derived 
recommendations based on challenges identified. An overview on the results is given in Figure  at the end of this chapter.  

ECRIS-RI, ECRIS-TCN & e-CODEX 
Strengths 

-  Strong regulative framework, structure and organisation capable to deal with the complex framework  
-  Secure and reliable environment provided by eu-LISA  
-  Centrally positioned, professional and skilled support on the side of eu-LISA for the entire product  
-  Well-equipped premises for meetings  
-  Use of state-of-the-art technology and access to standardised tools and applications 

126 

 
 
 
 
Weaknesses 

-  Strictly defined role of eu-LISA within its mandate 
-  Lack of exchange of documentation documents (and general lack thereof) between different projects  
-  Long decision-making procedures within the Agency  
-  Differing knowledge between different Member States, eu-LISA, and Commission  

Opportunities 

-  eu-LISA should contribute more strongly within the Commission comitology  
-  eu-LISA can become a main pillar of the e-CODEX community 
- 
It is recommended to strengthen the role of the AGs and PMBs 
-  The Agency can become a stronger actor in providing standardisation and harmonise the approaches between the different Member States 
-  eu-LISA should also try to include new target groups of the systems  
-  The Agency can ensure the respect of the principle of the independence of the judiciary 

Threats  

-  The heterogenous governance structure can lead to diffusions  
-  The complexity of e-CODEX might not be well understood by many members of the community  
-  The speed of developments is too fast for eu-LISA to cope with  
-  There might be a shortage of funding and resources  
-  The Member States might not be able to implement all necessary technical infrastructures and need more time than expected to implement all measures 

necessary  

Schengen Information System (SIS)  
Strengths 

-  Centralised technology and centralised solutions for the whole system  
-  Good service by the Service Desk  
-  A very supportive eu-LISA team and very pragmatic and realistically thinking staff on the ground  
-  Strong systems for training Member States  
-  Good data protection measures, conformant to legal base  

Weaknesses  

-  Limited information for end-users / limited resources in general and extensive outsourcing on the side of eu-LISA  
-  Sometimes issues in availability  
- 

It takes a long time to implement updates or changes. Overall change management is considered inefficient 

127 

 
 
-  Testing performed is often too limited and strict, leading to the identification of errors when it is already too late  
-  Member States’ positions are not considered sufficiently by eu-LISA  
-  Timelines set as concerns changes in the system are too ambitious and optimistic  
-  Participant management for meetings need to be improved (contacting the right people, sending invitations timelier, etc.)  

Opportunities  

Insourcing of more developers, to keep knowledge within the Agency  
Improving the quality and stability of technical documents  
Increase the sharing of best practices  

- 
- 
- 
-  Take Member States’ concerns stronger into consideration and better communicate with MS upcoming changes or problems  
-  Build more partnerships (e.g. with Europol)  

Threats  

-  Lack of resources can hinder the proper development and keeping of knowledge within the Agency  
-  Lack of proper communication might in the long-term hinder the effective and efficient cooperation with Member States  
-  The Agency might be too political, not considering Member States’ needs sufficiently in the future  

Visa Information System (VIS)  
Strengths 

-  The VIS is robust and reliable  
-  Member States can use their own applications  
-  The VIS provides a helpful communication tool via VIS mail  
-  Strong data protection framework  
-  Good knowledge about relevant legislation in the VIS team  

Weakness  
- 
Insufficient knowledge management (e.g. via a SharePoint)  
-  eu-LISA takes too much time to process requests and questions  
- 
-  eu-LISA should be stronger involved in drafting of legal basis  
-  Access rights are considered quite strict  
-  Too many Interface Control Documents (ICDs) are used to achieve a usable result  

Internal and external communication with eu-LISA is not good; there are too many working groups increasing the administrative burden of communication  

Opportunities  

-  Recommended to make documentation available via an own wiki page or something comparable  

128 

 
 
Include more national experts and, in general, include the Member States in further decisions  

- 
-  eu-LISA should get more flexibility in design (e.g. technical and functional specifications of the system)  

Threats 

-  The dynamic environment and frequent changes of the IT environment are not considered and addressed timely 
- 
Interoperability with other systems is not implemented successfully  
-  Too ambitious implementation plans might lead to delays in the long-term 
-  The skill shortage in IT might lead to recruitment problems for the Agency 
-  Pressure from the side of the Commission might lead to unfinished problems being published  

Eurodac & DubliNet  
Strengths  

-  The system is user-friendly  
-  Good service by the Service Desk  
-  The system is considered safe and efficient  
-  The data on Eurodac is very limited, which reduces the risk of data pollution  
-  The roles of the AG and PMB are highlighted very positively as supportive to Member States’ cooperation  

Weaknesses  

-  Renewal of certificates (every two years) is labour intensive and disturbs the work flow  
-  New certificates are often linked to old accounts, sometimes preventing Member States to open attachments sent with new certificates  
-  There is no properly established change management procedure  
-  There is a lack of formal communication with the Member States; preparation of AG meetings’ communication (e.g. sending around relevant files) could 

be improved  

-  Changing incorrect information in Eurodac is a very complicated and lengthy process  
-  End-users sometimes lack knowledge on how to use the system properly  

Opportunities  

Include Eurodac in the European interoperability framework  

- 
-  DubliNet is quite old, therefore a more modern, user-friendly communication system can be developed  
- 
- 
-  Setting up and finalising the new Eurodac Regulation 
- 

Increase the piloting, testing and innovation activities around DubliNet and Eurodac (e.g. use of other kinds of data capturing technology)  
Improve the preparation of the AG meetings (e.g. timely sharing of presentations, papers, etc.) 

Increase eu-LISA’s role in standardisation  

129 

 
 
Threats 

-  The deadline for the revised Eurodac Regulation might not be kept, which could lead to delays  
-  There are no clear guidelines on how to deal with the use of facial images  
-  The connecting of the systems through interoperability components with other systems might not work sufficiently  

Interoperability  
Strengths  

-  eu-LISA’s high level architecture environment seems flexible and scalable for future enhancements 
-  Good support for training 
-  eu-LISA has the necessary competencies to support the Member States  
-  The work of the Interoperability AG is quite efficient  
-  The interoperability roadmap was very well communicated from eu-LISA to the Commission and, thereafter, approved 

Weaknesses 

-  The planning of interface control documents (ICDs) was poor; they were not developed according to best practices for technical specifications.  
-  Technology  decisions  are  not  always  documented  and  described  well  on  the  side  of  eu-LISA;  making  it  hard  for  Member  States  to  understand  how 

decisions were made (not user-friendly) 

-  Lack of communication overall (with eu-LISA, sub-contractors) 
-  The necessary framework for standardisation across the IT systems is not well developed 
-  The timeline for the component development is too tight  
-  The simulators provided by eu-LISA have not proven to be as useful as expected; the Member States have started using their own simulators  
-  eu-LISA does not properly listen to the Member States but is rather focused on the feedback they receive from the Commission and the contractors  
-  The demand process is too time-consuming  
-  The AG is not chaired by a Member State but by eu-LISA, which Member States assess as a weakening of the interests of the Member States  
-  eu-LISA’s project management could be improved and be more agile  
-  Responsibilities are not clearly allocated (e.g. for testing activities) between the Member States and eu-LISA 

Opportunities  

-  Development of an open-source strategy, in which between Member States could be included  
- 
-  Establishment of pilot development of new systems by a small group of volunteering Member States, who would prepare stable environments for all 

Improvement of the ICDs quality  

Member States  

-  Strengthening of eu-LISA’s role from a passive system integrator to a more self-confident, proactive actor 

130 

 
 
-  Establishment of more interactive, transparent, modern and available communication 

Threats 

-  Further delays in the implementation of components can affect eu-LISA’s reputation and Member States’ good will  
-  Limited amount of testing might lead to implementation of faulty components  
-  Other developments within eu-LISA might move focus away from the interoperability components, given that eu-LISA has limited resources  
-  Lack of consulting eu-LISA on side of the Commission, EP and Council of the EU when new legislative proposals are drafted  
-  “Big bang go live” strategies are considered to be quite risky (e.g. entry into operations is more likely to be delayed due to errors)  

EES & ETIAS 
Strengths:  

-  The work of the AGs is structured and well-organised  
-  Technical documents (such as e.g. the ICDs) are well-structured and provide a clear vision of how systems shall work in the future  
-  Member States’ and users’ needs are considered sufficiently in the process 

Weaknesses  

-  Communication on the side of eu-LISA could be better, information is sometimes received too late and in an unclear manner  
-  There are significant shortcomings in the management of external contractors, leading to delays in the systems’ implementation  
-  Lack of transparency on how important products (e.g. ICDs, mock-ups) are produced  
-  Even though well structured, ICDs and other documentation are in different formats, making it difficult to process them for the single MS 
-  The planning of eu-LISA lacks a user perspective and is too much focused on the provisions of the regulations  

Opportunities  

Improve stakeholder communication  

- 
-  eu-LISA could facilitate more discussions in the AG, which can help the Member States to more strongly support each other  
-  More work could be moved from external contractors to eu-LISA  
-  National experts should be more strongly involved into the drafting of solutions  

Threats 

-  Lack of standardisation might come with serious problems in implementing the systems  
- 
-  Unrealistic time planning on the side of the Commission might reduce time for testing  

Improper implementation and lack of enough testing might lead to long waiting times at airports, ferry terminals, etc. 

131 

 
 
 
Figure 3 - Overview on recommendations derived in the workshops 

132