COMMISSION STAFF WORKING DOCUMENT EVALUATION Accompanying the document REPORT OF THE EUROEAN COMMISSION TO THE EUROEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS assessing the implementation and achievements of the 2014-2020 justice programme
Date: 2022-03-22
Institutions: Directorate-General for Justice and Consumers, European Commission
Subjects: EU financial instrument, EU programme, access to the courts, commitment of expenditure, drug addiction, fundamental rights, independence of the judiciary, judicial cooperation in civil matters in the EU, judicial cooperation in criminal matters in the EU, rule of law, vocational training

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 22.3.2022  SWD(2022) 64 final  COMMISSION STAFF WORKING DOCUMENT  EVALUATION  Accompanying the document   REPORT OF THE EUROEAN COMMISSION TO THE EUROEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS assessing the implementation and achievements of the 2014-2020 justice programme {COM(2022) 121 final} - {SWD(2022) 69 final}  
 
 
Table of contents 

1. 

INTRODUCTION ................................................................................................................................ 3 

Purpose and scope ....................................................................................................................... 3 

2. 

BACKGROUND TO THE INTERVENTION ..................................................................................... 4 

Description of the intervention and its objectives ....................................................................... 4 

Budget ......................................................................................................................................... 7 

Baseline and points of comparison .............................................................................................. 9 

3. 

IMPLEMENTATION ......................................................................................................................... 11 

4.  METHOD............................................................................................................................................ 25 

Short description of methodology ............................................................................................. 25 

Limitations and robustness of findings ..................................................................................... 27 

5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS .......................................... 28 

Effectiveness ............................................................................................................................. 28 

Efficiency .................................................................................................................................. 31 

Relevance .................................................................................................................................. 32 

Coherence, Complementarity and Synergies ............................................................................ 33 

EU added value ......................................................................................................................... 34 

Equity ........................................................................................................................................ 35 

Scope for simplification ............................................................................................................ 37 

CONCLUSIONS .......................................................................................................................................... 38 

ANNEX 3: METHODS AND ANALYTICAL MODELS .......................................................................... 50 

1 

 
 
Glossary 

Term or acronym 

Meaning or definition 

DG JUST 

Directorate-General for Justice and Consumers 

EU 

AG 

OG 

HES 

OTH 

PRC 

REC 

PUB 

AWP 

JCOO 

JTRA 

JACC 

JDRU 

MMF 

European Union 

Action grant 

Operating grant 

Higher secondary education 

Other non-profit organisation 

Private company 

Research organisation 

Public Institution 

Annual work programme 

Judicial Cooperation 

Judicial Training 

Access to Justice 

Drug prevention policy 

Multi Annual Financial Framework 

2 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1. 

INTRODUCTION 

Purpose and scope 

This staff working document accompanies the report from the European Commission to 
the European Parliament, the Council, the Economic and Social Committee and the 
Committee of the Regions in accordance with the obligation specified in Article 14.2(c) 
of the legal base of the justice programme.1 

The  Regulation  states  that  the  European  Commission  shall  provide  the  European 
Parliament  and  the  Council  with  an  ex-post  evaluation  report  of  the  programme  by  31 
December 2021. The report shall assess the long-term impact and the sustainability of the 
effects  of  the  programme,  with  a  view  to  informing  a  decision  on  a  subsequent 
programme. In 2021, however, a considerable number of projects, around 30 %, are still 
ongoing and many completed ones have just produced the first outputs. This situation is 
also  exacerbated  by  the  COVID-19  pandemic,  which  has  resulted  in  the  timeframe 
several projects being extended. 

For these reasons, and in order to have a meaningful evaluation of the long-term results 
or impacts of the programme, the ex post evaluation will be carried out in two parts. This 
staff  working  document  accompanying  the  report  to  the  European  Parliament,  the 
Council,  the  Economic  and  Social  Committee  and  the  Committee  of  the  Regions 
represents  the  first  part  of  the  evaluation.  It  is  based  on  the  currently  available  data, 
provides  an  overview  of  the  funding  distribution  and  assesses  the  achievements  of  the 
justice  programme  so  far.  While  the  new  justice  programme  2021-2027  was  already 
adopted,  based  amongst  others  on  the  results  of  the  interim  evaluation  of  the  justice 
programme  2014-2020,  this  first  part  of  the  ex-post  evaluation  of  the  previous 
programme  will  be  of  added  value  in  order  to  shed  light  on  potential  areas  for 
improvement in the implementation phase of the new justice programme. 

The second part of the evaluation will be carried out at a later stage, once all final data 
will  be  available  and  in  concomitance  with  the  interim  evaluation  of  the  succeeding 
justice programme (2021-2027). This second part will assess the long-term impacts and 
sustainability of the effects of the justice programme and will provide recommendations 
for the new MFF after 2027, as appropriate.  

This two-step approach will enable the Commission to conduct an in-depth evaluation of 
the entire programming period, while being fully transparent on the rationale for splitting 
the reports.  

Against  this  backdrop,  the  first  part,  covering  available  data  from  2014  to  2020  will 
provide  a  good  overview  of  the  funding  distribution  and  achievements  of  the 
programmes so far.  By  contrast, the second report will provide further insights into the 
effectiveness and long-term impacts of the programme. 

1 REGULATION (EU) No 1382/2013 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL 
of 17 December 2013 establishing a Justice Programme for the period 2014 to 2020 

3 

 
 
                                                           
In  terms  of  geographical  coverage,  the  justice  programme  is  open  to  all  EU  Member 
States with  the exception of Denmark and the United Kingdom.2  In  addition  to  the EU 
Member  States,  it  is  open  to  the  European  Free  Trade  Association  States  party  to  the 
European  Economic  Area,  candidate  countries,  potential  candidates  and  countries 
acceding  to  the  EU,  provided  that  they  conclude  an  agreement  with  the  Union  laying 
down  the  details  of  their  respective  participation  in  the  programme.  Albania  joined  the 
programme in 2017 and Montenegro in 2018. 

2.  BACKGROUND TO THE INTERVENTION 

Description of the intervention and its objectives 

In accordance with the Treaty on the Functioning of the European Union, which provides 
for  the  creation  of  a  European  area  of  freedom,  security  and  justice  based  on  mutual 
recognition  of  judicial  decisions  and  mutual  trust  among  Member  States,  the  justice 
programme was set up to overcome obstacles to the functioning of an effective European 
area of justice. The justice programme was also designed to encourage national judicial 
systems  to  build  trust  in  each  other’s  standards  of  fairness  and  justice,  as  this  is  a 
prerequisite to achieve a European area of justice. 

More specifically, the justice programme as set out in in Article 4 of its legal base3 is to 
contribute to the development of a European area of justice: 

(a)  facilitating and supporting judicial cooperation in civil and criminal matters; 

(b)  supporting and promoting  judicial training, including language training on legal 

terminology, with a view to fostering a common legal and judicial culture; 

(c)  facilitating effective access to justice for all, including promoting and supporting 
the rights of victims of crime, while respecting the rights of the defence; and 

(d)  supporting  initiatives  in  the  field  of  drug  policy  as  regards  judicial  cooperation 
and  crime  prevention  aspects  closely  linked  to  the  general  objective  of  the 
programme, insofar as they are not  covered by the Internal Security Fund (ISF) 
for financial support for police cooperation, preventing and combating crime, and 
crisis management or by the Health for Growth Programme. 

This vision to create a European area of justice is to enable everyone in the EU to fully 
enjoy their full right to the freedom of movement, the respect for their fundamental rights 
and respect  for  common principles, such as non-discrimination,  gender equality, access 
to justice for all, the rule of law and well-functioning independent judicial systems. 

The justice programme also has links, potential synergies, and is complementary to other 
relevant  initiatives  of  the  European  Union  (EU)  and  other  international  organisations, 
such as the United Nations (UN). These include:  

2 The justice programme has legal bases that belong to Part V of title III of the TFEU. Therefore, protocols 
21  and  22  to  the  TEU  and  the  TFEU  apply  and  Denmark  and  the  UK  never  participated  to  the  Justice 
programme. 
3  Regulation  (EU)  No  1382/2013 of  the  European  Parliament  and  of  The  Council  of  17  December  2013 
establishing a Justice Programme for the period 2014 to 2020, available online at: EUR-Lex - 32013R1382 
- EN - EUR-Lex (europa.eu) 

4 

 
 
 
                                                           
(a)  The European Agenda for Justice for 2020, adopted in 2014;  

(b) The European Agenda for Security, adopted in 2015; 

(c)  The 2030 Sustainable Development Agenda;  

(d) The Charter of Fundamental Rights, which came into effect in 2000, and is linked 
to  the  Convention  for  the  Protection  of  Human  Rights  and  Fundamental 
Freedoms;  

(e)  The “Juncker Priorities” for 2015-2019; 

(f)  The Von der Leyen's Commission priorities for 2019-2024. 

These documents and initiatives have similar objectives, particularly regarding issues of 
effectiveness of judicial remedies, judicial cooperation, and drug prevention policy. 

Description of the programme 

With the aim of meeting the above specific objectives of the justice programme, the 
following paragraphs outline the programme’s main building blocks. 

Judicial cooperation in civil and criminal matters 

To achieve its specific objective to facilitate and support judicial cooperation in civil and 
criminal  matters,  the  justice  programme  offers  financial  support  for  activities  that 
promote  the  effective  and  coherent  application  of  the  EU  law  by  building-up  and 
improving  data  collection  and  statistics  on  the  application  of  EU  law.  The  programme 
fosters  judicial  cooperation  in  civil  and  criminal  matters  by  funding  activities  that 
contribute  to  the  better  enforcement  of  EU  law  and  judicial  decisions,  in  particular 
resulting  from  cross-border disputes.  This  includes, Directive 2011/99  on the European 
Protection Order and Framework Decision 2002/584/JHA on the European arrest warrant 
in criminal matters. Regulation (EC) No 861/2007 establishing a European small claims 
procedure  and  Regulation  (EU)  No  1215/2012  on  jurisdiction  and  the  recognition  and 
enforcement of judgments in civil and commercial matters are also relevant pieces of EU 
law  in  civil  matters.  EU  legislation  on  judicial  cooperation  in  civil  matters  also  covers 
multiple  aspects  of  family  law  (parental  responsibility,  parental  child  abduction, 
maintenance, matrimonial property rights and succession, to give a few examples). The 
justice programme contributes by funding improvements to the exchange of information 
among professionals in order to boost operational cooperation and mutual trust in the EU. 
Under this specific objective, the justice programme facilitates the EU’s participation in 
The Hague Conference  of Private  International  Law4 and  action  at  the national  level  to 
set  up  and  strengthen  networks  in  the  framework  of  the  European  Judicial  Network  in 
civil and commercial matters. Last but not least, this specific objective encompasses calls 
for  proposals,  launched  in  the  wake  of  terrorist  attacks  in  the  EU,  to  prevent 
radicalisation in prisons  leading to terrorism and violent extremism. In this context, the 
European  Commission  cooperates  closely  with  the  Council  of  Europe  on  two  specific 

4 The EU is party to The Hague Conference on Private International Law since 2007. The purpose of this 
international  intergovernmental  organisation  is  to  work  for  the  progressive  unification  of  the  rules  of 
private international law in the participating countries. Most of the conventions developed by The Hague 
Conference fall within exclusive or partial external competence of the EU and are part of the EU law. 

5 

 
 
 
 
 
 
                                                           
initiatives:  the  SPACE  report  on  prison  statistics  and  on  setting  up  the  EU  network  of 
prison monitoring bodies in the Member States. 

Judicial training 

Under  the  specific  objective  to  facilitate  and  support  judicial  training,  activities  funded 
by the justice programme cover the training of professionals such as judges, prosecutors, 
court  staff,  bailiffs,  lawyers,  notaries,  prison  staff,  probation  officers,    mediators,  and 
legal interpreters. The topics of trainings courses are wide and varied, including EU civil 
and  criminal  law,  fundamental  rights,  judicial  ethics  and  the  rule  of  law.  Activities 
funded  under  this  specific  objective  are  also  designed  to  provide  language  training  on 
legal terminology to foster a common legal and judicial culture. In addition to measures 
that  aim  to  create  a  shared  understanding  of  legal  terminology,  the  justice  programme 
provides  funding  for  seminars  on  specific  aspects  of  civil  and  criminal  law,  e-learning 
and exchanges of staff and experience.5 For example, it covers specialised training in the 
field of competition law, which typically requires a significant amount of specialisation. 
Support for judicial training provided by the justice programme is strongly linked to the 
European judicial training strategy, which was adopted in 2011.6 In conjunction with this 
training  strategy,  the  European  Judicial  Training  Network  is  a  key  structure  funded  by 
the justice programme to reach more practitioners from across the Member States and to 
promote trust in the judicial systems of different Member States. The European Judicial 
Training  Network  is  an  essential  structure  for  judicial  training.  It  is  also  referenced  in 
Article 6.2 of the legal base of the justice programme as a recipient of an operating grant 
to co-finance expenditure on its permanent work programme. In summary, by providing 
the  training  opportunities  outlined  above,  the  justice  programme  aims  to  help  create  a 
common  legal  and  judicial  culture  and  foster  mutual  trust  of  legal  practitioners  in  the 
judicial systems of EU Member States.  

Access to justice for all 

The  remit  of  the  specific  objective  to  facilitate  effective  access  to  justice  for  all,  is 
divided in two sub- objectives. One key topic is promoting and supporting the rights of 
victims  of  crime.  The  other  is  procedural  rights,  which  ensure  respect  for  the  right  of 
defence.  To  this  end,  the  aim  of  the  projects  financed  is  to  provide  EU  citizens  with 
effective remedies if EU law is breached. In particular, where national procedures prove 
too complex. In other words, the measures are designed to promote the use of remedies 
and non-remedies developed in  the EU that can provide quick, efficient and less costly 
solutions to disputes, for example through the e-Justice portal.7 Access to justice is also 
an enabling right for victims of crime to be able to enforce their rights and seek redress. 
When the Treaty of Lisbon came into effect, a specific reference to access to justice was 

5 European Commission, Directorate-General for Justice and Consumers, European Judicial Training 2016. 
Available online at: http://ec.europa.eu/justice/criminal/files/final_report_2015_en.pdf 
6 Communication from the Commission to the European Parliament, the Council, the European Economic 
and  Social  Committee  and  the  Committee  of  the  Regions:  Building  Trust  in  EU-Wide  Justice  -  A  New 
Dimension to European Judicial Training, COM (2011) 551 final. 

7 Communication from the Commission to the European Parliament, the Council, the European Economic 
And  Social  Committee  and  the  Committee  Of  The  Regions  The  EU  Justice  Agenda  for  2020  - 
Strengthening  Trust,  Mobility  and  Growth  within  the  Union,  Strasbourg,  11.3.2014,  COM  (2014)  144 
final. 

6 

 
 
 
                                                           
also made in the Treaty on the Functioning of the EU, which states that ’the Union shall 
facilitate  access  to  justice,  in  particular  through  the  principle  of  mutual  recognition  of 
judicial and extrajudicial decisions in civil matters’.8 

Initiatives in the field of drug policy 

This specific objective covers the promotion of initiatives in the field of drug prevention 
policy, which mainly focus on judicial cooperation and crime prevention. Thus, the main 
aim of activities funded by the justice programme is to foster the practical application of 
drug-related  research.  Activities  under  this  objective  are  also  designed  to  help  civil 
society  organisations  and  key  stakeholders  expand  the  knowledge  base  and  develop 
innovative methods  to  address  the phenomenon  of new psychoactive substances. These 
aims  are  also  pursued  by  the  European  agenda  on  security.  Under  this  agenda,  the  EU 
should  continue  to  support  action  taken  by  the  Member  States  to  counter  both  the 
demand and the supply of illicit drugs, including prevention work. In the context of drug 
prevention  policy,  through  its  focus  on  judicial  cooperation  and  crime  prevention,  the 
justice programme complements other EU initiatives such as the Internal Security Fund, 
which  tackles  the  sale,  transport,  import,  and  export  of  illegal  drugs.  The  justice 
programme complements the health for growth programme, which focuses on actions to 
reduce the health-related damage caused by drugs. 

Budget 

The initial total budget for the justice programme for 2014-2020 was EUR 377 604 000. 

The  justice  programme  has  three  main  funding  mechanisms:  action  grants,  operating 
grants and procurement.  

  action  grants  are  mainly  provided  to  civil  society  organisations,  Member 

State authorities and universities;  

  operating  grants  (support  to  networks)  fund  mainly  European  networks 
active in the following areas:  judicial training as set out in the programme’s 
legal base, judicial cooperation and access to justice; 

  procurement  actions  (Commission  initiatives)  fund  mostly  conferences, 
seminars,  studies,  surveys,  awareness-raising  activities,  but  also  specific  IT 
projects  and  the  EU’s  membership  in  The  Hague  Conference  of  Private 
International Law.  

In  order  to  achieve  its  objectives,  the  justice  programme  can  support  a  wide  range  of 
activities.  In  particular,  under  Article  6  of  the  Regulation  establishing  the  justice 
programme for the period 2014 to 2020, the programme can finance: 

  analytical  activities,  in  particular:  the  collection  of  data  and  statistics;  the 
development of common methods and, where appropriate, indicators or benchmarks; 
studies,  research,  analyses  and  surveys;  evaluations;  drafting  and  publishing  guides, 
reports  and  educational  material;  workshops,  seminars,  expert  meetings  and 
conferences.  

8  European  Union,  consolidated  version  of 
lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:12012E/TXT&from=EN.  

the  Treaty  of 

the  European  Union,  http://eur-

7 

 
 
 
                                                           
  mutual  learning,  cooperation,  awareness  raising  and  dissemination  activities,  in 
innovative 
identifying,  and  exchanging  concerning  good  practice, 
particular: 
approaches  and  experiences;  organising  of  peer  reviews  and  mutual  learning; 
organising  of  conferences,  seminars,  information  campaigns;  compilation  and 
publishing  of  materials  to  publicise  about  the  justice  programme  and  its  results; 
developing,  operating  and  maintaining  systems  and  tools,  using  information  and 
communication technologies, including developing of the European e-justice portal as 
a tool to improve public access to justice.  

  training,  for  instance  staff  exchanges,  workshops,  seminars,  train-the-trainer  events, 
language training on legal terminology,  and the development of online training tools 
or other training modules for members of the judiciary and judicial staff. 

Measures to support the main actors that implement the programme’s objectives, such 
as:  support  for  Member  States  in  implementing  EU  law  and  policies;  support  for  key 
European actors and European-level networks, including in the field of judicial training; 
support  for  networking  at  EU  level  among  specialised  bodies  and  entities  as  well  as 
national, regional and local authorities and non-governmental organisations. 

The  stakeholders  eligible  for  support  can  be  either  public  or  private  organisations 
(usually  non-profit-oriented),  established  in  one  of  the  countries  participating  in  the 
programme, or international organisations (Council of Europe). 

In  terms  of  target  groups,  defined  as  the  groups  that  can  receive  support  directly  or 
indirectly from the justice programme, potentially all people in the EU are included. This 
broad  target  group  is  based  on  the  overarching  objective  of  the  justice  programme  to 
create  a  European  area  of  justice,  where  everybody  is  aware  of  and  can  exercise  their 
rights. More specifically, the programme covers support for judicial practitioners, public 
authorities, universities, international organisations, non-governmental organisations and 
insolvency  or  pre-insolvency 
other  research  groups;  companies  going 
proceedings, citizens accused or victims of a crime, families, consumers, minors, groups 
in need of specialised legal protection and people at risk of radicalisation. 

through 

The  figure  overleaf  summarises  the  intervention  logic  of  the  justice  programme  by 
illustrating causal mechanisms to an effective area of justice. 

8 

 
 
Figure 1 Intervention logic of the justice programme 2014-2020 

Baseline and points of comparison  

The  2011  impact  assessment  used  as  part  of  the  interim  evaluation  of  the  justice 
programme  serves  as  a  baseline  and  a  point  of  comparison.9  The  document  outlines  a 
‘baseline  scenario’  against  which  the  policy  options  for  the  justice  programme  were 
benchmarked. The document also describes a series of expected outcomes, connected to 
the  development  of  the  justice  programme.  The  baseline  analysis  showed  that,  if  no 
changes had been made between the 2007-13 and the 2014-20 programming periods, the 
programme would have continued to be successfully implemented, but it would not have 
reached its full potential. In this context, it is worth underlining that before its launch in 
2014, the justice programme was preceded by three different funding instruments, which 
each covered the thematic areas now covered by one programme. 

Before launching the justice programme, the Commission identified the following issues 
as potentially critical: 

 

 

the  lack  of  flexibility  in  the  funding  instruments  would  have  been  an  obstacle, 
given the pace of change and reform in the area of justice; 

the  fragmentation  of  funding,  especially  where  funding  is  needed  to  support 
horizontal  and  cross-cutting  issues,  could  have  reduced  the  capacity  of  the 
predecessor programmes to deliver results; 

  having multiple funding programmes with different rules and procedures, in the 
context  of  rising  numbers  of  applicants,  could  have  increased  the  administrative 
burden, potentially resulting in delays in procedures for the beneficiaries. 

Given these bottlenecks and difficulties, the criminal justice programme, the civil justice 
programme  and  the  drug  prevention  and  information  programme  were  merged  into  the 
justice  programme.  The  2011  impact  assessment  used  in  the  interim  evaluation  also 

9 SEC (2011) 1364 final 

9 

 
 
                                                           
assessed  the  effect  of  the  consolidation  in  terms  of  relevance  and  scope,  effectiveness, 
complementarity, EU added value, efficiency and potential for simplification. 

 

 

 

 

In  terms  of  the  relevance  and  scope  of  the  consolidated  programme,  the  impact 
assessment  found  that  funding  would  be  available  for  all  of  the  areas  and  activities  
covered  by  predecessor  programmes.  In  addition,  the  expectation  was  that  it  would 
reduce fragmentation and overlaps, as policies with close links to each other, such as 
civil and criminal justice, were streamlined under the same programme. The  aim of 
streamlining predecessor programmes into the justice programme was also to reduce 
the  potential  for  overlap  with  other  funding  instruments.  In  terms  of  flexibility,  the 
broader scope of the justice programme was designed to progressively integrate new 
actions  that  better  support  justice  policies  as  the  policy  environment  continuously 
evolves.   

In terms of effectiveness, the new comprehensive justice programme was expected to 
promote  the  concentration  of  funds  and  the  development  of  stronger  partnerships. 
The  applicants  would  coordinate  their  action  in  broader  partnerships  with  more 
funding  available.  It  was  also  expected  to  lead  to  a  more  balanced  geographical 
spread among beneficiaries as a result of larger partnerships.  

In  terms  of  efficiency,  the  expectation  was  that the  justice  programme  would  bring 
about  improvements  by  simplifying  the  funding  procedures  for  applicants  and 
beneficiaries.  In  particular,  merging  the  predecessor  programmes  under  a  broader 
justice  programme  aimed  to  put  an  end  to  the  practice  of  beneficiaries  oscillating 
between the previous three funding programmes based on their individual fit to them. 
Last but not least, the aim was to improve efficiency by reducing the number of calls 
for proposals, which also became more thematically focused.   

In  terms  of  complementarity,  it  was  expected  that  merging  the  three  predecessor 
programmes  would  ensure  a  coordinated  approach  to  identifying  annual  priorities. 
This approach had to be applied also vis-à-vis programmes in other EU policy areas 
with links to justice and rights. Coordination during the annual programming process 
would  make  sure  that  funding  is  not  duplicated  and  that  funds  are  used  in  a 
complementary way, in the best public interest. 

  With a view to reducing the administrative burden, reducing the three annual work 
programmes to one and reducing the number of calls for proposals were expected to 
yield  considerable  improvements.  Harmonising  the  management  procedures  would 
lighten  the  administrative  burden  for  applicants  and  beneficiaries.  It  would  also 
enable the Commission to halve the number of staff members assigned to managing 
calls for proposals.  

In addition to the findings of the impact assessment of the interim evaluation, all ex-post 
evaluation reports on the previous funding programmes confirmed that overall they were 
effective  and highlighted that their specific objectives and priorities  were  on the whole 
specific, attainable and realistic, but not always measurable. 

In particular, it identified the following problems common to the three programmes: 

1.  the  funds  were  diluted  and  spread  across  many  small-scale  projects  with  limited 
impact and EU dimension; 

2.  no  balanced  geographical  spread  among  the  organisations,  receiving  funding;  

3. complex and bureaucratic procedures for applicants; 

10 

 
 
4.  high  administrative  burden  on  the  European  Commission  and  longer  procedures 
due to the multiplication of procedures for the different programmes.  

In  summary,  the  baseline  and  point  of  comparison  of  the  justice  programme,  is 
characterised by a fragmented funding landscape, which put heavy administrative burden 
and  complex  procedures  in  place.  As  a  result,  the  Commission  took  measures  to  make 
EU  funding  more  accessible  and  to  promote  more  targeted  action  to  reach  its  policy 
goals.  

3. 

IMPLEMENTATION  

Programme management 

The  European  Commission  implements  the  justice  programme  via  direct  management. 
This approach was taken in order to ensure a close relationship between the programme 
management and EU policy-making. This management mode also enables the European 
Commission to tailor funded activities to policy priorities and policy needs and thereby 
to directly target specific groups of stakeholders. It also facilitates close contacts with the 
programme's beneficiaries and results in better knowledge of the needs on the ground.  

State of play  

This section presents the state of play of the programme’s implementation on the basis of 
the  information  available  at  the  time  of  writing.  As  explained  in  the  introduction,  a 
significant number of projects, around 30%, are still ongoing, and therefore the ex post 
evaluation will be carried out in two parts. The first part, based on the currently available 
data, is prepared in 2021 and assesses the achievements of the justice programme so far. 
The second part of the evaluation will be carried out at a later stage, once all final data 
will  be  available  and  in  concomitance  with  the  interim  evaluation  of  the  succeeding 
justice programme (2021-2027). This second part will assess the long-term impacts and 
sustainability of the effects of the justice programme and will provide recommendations 
for the new MFF after 2027, as appropriate.  

Annual work programmes 

The  annual  work  programmes  of  the  justice  programme  set  out  goals  to  pursue,  by 
funding  the  measures.  In  other  words,  the  annual  work  programmes  outline  how  the 
specific  objectives  of  the  justice  programmes  are  to  be  met  through  targeted  activities. 
The  annual  work  programmes  will  also  seek  to  adapt  to  emerging  needs  so  that 
beneficiaries  can  adapt  to  any  changes  in  the  policy  environment  of  the  justice 
programme. 

In 2014, the first annual work programme was overall balanced. The specific objectives 
of judicial cooperation, access to justice and judicial training received approximately the 
same  level  of  funding.  The  beneficiary  identified  by  the  legal  base  of  the  justice 
programme,  the  European  Judicial  Training  Network,  received  an  operating  grant  of 
EUR  7  880  000  the  highest  amount  of  funding  in  that  year.  Action  grants  for  work  to 
meet  the  specific  objective  on  judicial  training  had  a  rather  broad  focus  on  family  and 
succession matters, as well as on criminal law. Calls for proposals on access to justice for 
all  covered  the  expansion  of  the  e-justice  portal  and  work  on  procedural  and  victims’ 
rights.  Action  grants  under  the  specific  objective  on  judicial  cooperation  in  civil  and 

11 

 
 
 
 
 
 
criminal  matters  focused  on  family  matters  such  as  divorce,  legal  separation,  parental 
custody,  and  succession.  Last  but  not  least,  the  drafting  of  new  legislation  on  new 
psychoactive  substances  and  support  to  implement  the  EU  drug  strategy  were  financed 
through procurement and action grants in the field of drug prevention policy.  

The 2015 annual work programme underscored the importance of the specific objective 
of  access  to  justice  for  all.  In  total  there  were  six  calls  for  proposals  focusing  on  this 
specific objective. In parallel, action grants promoting the specific objective on judicial 
training  were  aligned  with  the  topics  pursued  in  2014  under  the  specific  objectives  of 
judicial cooperation and access to justice.  Action grants for judicial training  focused on 
training  judges  on  legal  instruments  in  family  matters  and  successions.  Other  training 
courses  were  focused  on  international  debt  recovery  in  civil  matters  and  commercial 
matters.  On criminal  matters, training  sessions  focused on procedural  rights  in  criminal 
proceedings.  In  addition,  judicial  training  was  promoted  through  the  operating  grant  to 
the European Judicial Training Network with a budget of EUR 8 800 000. The European 
Judicial  Training  Network  supported  implementation  of  EU  legislation  in  commercial 
matters,  for  example  by  working  on  the  Commission  recommendation  for  a  new 
approach  to  business  failure  and  insolvency.  The  Network  also  supported  action  on 
criminal  matters  to  implement  Framework  Decision  2002/584/JHA  on  the  European 
Arrest Warrant. 

In  2016,  the  budget  reserved  for  funding  the  work  of  the  European  Judicial  Training 
Network grew to EUR 9 500 000, underlining the importance of the network. The 2016 
annual  work  programme  also  attributed  major  importance  to  the  specific  objective  of 
access  to  justice  for  all  through  procurement,  which  funded  the  evaluation  of  existing 
criminal  justice  instruments  and  the  preparation  of  possible  new  tools.  For  the  specific 
objective  on  judicial  cooperation,  funding  supported  the  management,  analytical  work 
and  information  activities  of  the  European  Judicial  Network  in  civil  and  commercial 
matters. Procurement funding supported the maintenance and upgrading of the European 
Criminal Records Information System.  

The  specific  objective  on  judicial  training  was  a  salient  aspect  of  the  annual  work 
programme in 2017. The funding for the European Judicial Training Network grew and 
amounted  to  EUR  10  200  000.  With  a  view  to  judicial  cooperation,  family  matters, 
succession  and  commercial  law  enforcement  remained  key  topics  in  civil  matters.  On 
criminal  matters,  the  focus  was  on  implementation  and  practical  application  of  mutual 
recognition  instruments,  such  as  the  European  Arrest  Warrant,  the  transfer  of  prisoners 
and  the  European  Supervision  Order.  For  the  specific  objective  on  drug  prevention 
policy, the focus remained on identifying new psychoactive substances. 

In  2018,  the  European  Judicial  Network  in  Civil  and  Commercial  Matters  and  the 
European  Judicial  Training  Network  were  high  on  the  agenda  of  the  annual  work 
programme.10  Support  for  the  European  Judicial  Network  in  Civil  and  Commercial 
Matters  was  designed  to  improve  implementation  and  case-handling  under  civil  justice 
legislation. Training courses tackled gaps in cross-border training for court staff, bailiffs 
and prison staff. Another priority on the training front was to facilitate access to training 
and instructing training providers. By  contrast, there was a reduction in  funding for the 

10 The European Judicial Training Network received EUR 11 000 000 in 2018. 

12 

 
 
 
 
 
                                                           
specific  objective  on  access  to  justice  for  all  and  for  judicial  cooperation,  except  for 
support for the European Judicial Network in Civil and Commercial Matters. 

In 2019, there were continued reductions in the budget allocated to the specific objectives 
of  judicial  cooperation  and  access  to  justice.  However,  activities  continued  to  promote 
cooperation between national networks in the field of civil justice under the umbrella of 
the European Judicial Network in Civil and Commercial Matters. The 2019 annual work 
programme  provided  funding  to  ensure  that  Member  States  create  national  networks  to 
boost judicial cooperation. In addition, the funding was designed to consolidate existing 
structures.  On  judicial  training,  the  level  of  funding  remained  relatively  stable.11  The 
continued high importance of judicial training can be partially attributed to the fact that 
judicial training became a crucial part of the European Commission’s work programme, 
under the priorities ’An Area of Justice and Fundamental Rights based on Mutual Trust” 
and “a Union of democratic change’. Judicial training also claimed a central role with the 
European e-justice portal expanded by rolling out the European Training Platform.  

In  2020,  the  last  annual  work  programme  under  this  programming  period,  there  was  a 
particular  focus  on  digital  technologies  and  their  impact  on  the  goals  of  the  justice 
programme.  In  the  area  of  judicial  cooperation  in  criminal  matters,  the  focus  was  on 
access to electronic evidence as a means to  counter criminal  activities.  Under access to 
justice,  funding  was  targeted  to  promote  new  initiatives  to  implement  the  European  e-
justice strategy for 2019-2023. As in the previous year, maintaining and upgrading of the 
e-justice  portal  remained  an  important  topic.  This  included  creating  synergies  between 
the  work  on  e-justice  and  the  Connecting  Europe  Facility  programme  with  the  aim  of 
contributing  to  the  creation  of  the  digital  single  market.  Digital  technologies  also 
influenced work under the specific objective of judicial training. The potential impact of 
frontier  technologies  such  as  artificial  intelligence  and  machine  learning  on  law 
enforcement, amplified the importance of judicial training.12 Therefore, the focus was on 
training  judicial  staff  on  the  use  of  such  technologies  and  on  tools  to  counter  digital 
threats.  Technological  developments  also  affected implementation of the programme in 
the field of drug prevention policy, in particular in combating the online drugs trade. 

The  developments  described  above  are  reflected  in  the  tables  below,  which  show  the 
budget for each annual work programme and provide an overview of funding across the 
specific objectives for the entire programming period.  

Table 1 Budget planned per year 

Budget Year 

2014 
2015 
2016 
2017 

Amount planned in the 
annual work programme  € 
€ 45 812 000 
€ 48 051 000 
€ 52 250 000 
€ 52 631 000 

Annual change % 

- 
4,9% 
8,7% 
0,7% 

11 The European Judicial Training Network received EUR 11 000.000 in 2019. 
12 The European Judicial Training Network received EUR 11.000 000 in 2020. 

13 

 
 
 
 
 
                                                           
 
2018 
2019 
2020 
Total 

€ 45 949 000 
€ 43 675 000 
€ 45 603 000 
€ 333 971 000 

Source: annual work programmes 2014-2020 

-12,7% 
-4,9% 
4,4% 

The  overall  budget  increased  steadily  up  to  2017,  when  it  reached  a  peak  of  EUR 
52.631.000.  It  fell  afterwards  to  reach  approximately  the  same  level  as  in  2014.  The 
budget  decreased  as  funds  of  the  justice  programme  were  reallocated  to  the  Civil 
Protection  Mechanism  (CPM  -  COM(2017)772)  and  to  provide  eu-LISA  with  the 
necessary financial resources to develop the European Criminal Records System for third 
country nationals. 

Table 2 Budget planned per specific objective 2014 - 2020 (in EUR thousands) 

11 661 

17 570 

13 373 

Specific 
Objective 

Judicial 
Cooperation 

2014 

Planned 

2015 

2016 

2017 

2018 

2019 

2020 

Planned 

Planned 

Planned 

Planned 

Planned 

Planned 

14 228 

14 415 

14 570 

15 789 

12 000 

11 600 

Judicial Training 

14 825 

16 515 

16 900 

18 048 

17 870 

17 570 

Access to Justice 

13 775 

13 296 

16 268 

15 662 

13 330 

11 630 

Initiatives in the 
field of drugs 
policy 

3 004 

3 000 

2 512 

3 123 

2 749 

2 875 

2 999 

Total 

45 812 

48 051 

52 250 

52 631 

45 949 

43 675 

45 603 

Source: annual work programmes 2014-2020 

The  distribution  of  funds  across  specific  objectives  shows  that  every  year,  the  specific 
objective on judicial training received a large share of the financial resources available. 
In total, 36,12% of the budget or (EUR 119 908 000) was allocated to judicial training. 
Access to justice was the second most funded specific objective, with EUR 99 364 000 
and  29,94%  of  the  budget.  This  was  closely  followed  by  judicial  cooperation,  with 
28,45% of the budget,  (EUR  94 418  000).  Initiatives  in the field  of drugs policy  had  a 
significantly lower budget. Only 6,11 % of the budget, (EUR 20 281 000) was earmarked 
for this specific objective. The distribution of funds is set out in the allocation of funds to 
specific  objectives  in  the  Annex  to  the  Regulation  1382  establishing  the  justice 
programme 2014-2020.  

Table 3 Budget planned per specific objective 2014 -2020 (in EUR) 

Action  grants 

Operating grants   Procurement 

Other 

Amount planned 
€ 

Share of total 

€ 49 089 600 

€9 258 400 

€ 35 833 000 

€ 237000,00 

€ 94 418 000 

Specific 
objective 
Judicial 
Cooperation 
Access to 
Justice 
Judicial 
Training 

€ 48 454 400 

€ 14 473 170 

€ 36 436 430 

€ 44 493 000 

€ 69 380 000 

€ 6 035 000 

Drugs  

€16 781 000 

- 

€ 3 500 000 

14 

- 

- 

- 

€ 99 364 000 

€ 119 908 000 

€ 20 281 000 

28,45% 

29,94% 

36,12% 

6,11% 

 
 
 
 
 
 
 
 
 
 
 
Specific 
objective 

Total 

Action  grants 

Operating grants   Procurement 

Other 

€ 158 818 000 

€ 93 111 570 

€ 818 044 30 

237 000 

Source: annual work programmes 2014-2020 

Amount planned 
€ 
€ 333 971 000 

Share of total 

100% 

Calls for proposals 

In the period covered by the 2014-2020 annual work programmes, 118 calls for proposals 
were  launched  for  all  types  of  funding  under  the  justice  programme:  action  grants, 
operating  grants  and  procurement.  Concerning  the  specific  objectives  of  the  justice 
programme, a large share of the calls for proposals (40 in total) were to support judicial 
cooperation  in  civil  and  criminal  matters.  The  second  highest  number  of  calls  (36  in 
total),  were  to  support  the  specific  objective  on  effective  access  to  justice  for  all, 
including  promoting  and  supporting  the  rights  of  victims  of  crime,  and  respecting  the 
rights  of  defence.  The  third  highest  number  of  calls  were  for  the  specific  objective  to 
facilitate  and  support  judicial  training  (26  calls  for  proposals).  Finally,  16  calls  for 
proposals  were  run  to  support  initiatives  in  the  field  of  drug  prevention  policy.  The 
distribution of calls over the entire programming period shows that the yearly number of 
calls  peaked  in  the  initial  years  of  the  programme,  before  dropping  as  a  result  of  the 
merge in 2016 to manage the justice programme more efficiently. 

Table 4 Number of calls for proposals launched per specific objective. 

Judicial cooperation  Access to justice 

Judicial training 

Drugs policy 

Specific objective 

Year 

2014 

2015 

2016 

2017 

2018 

2019 

2020 

Total 

7 

5 

7 

6 

6 

4 

5 

40 

8 

6 

5 

5 

4 

4 

4 

36 

3 

4 

4 

4 

4 

3 

4 

26 

Source: annual work programmes 2014-2020 

2 

5 

1 

2 

2 

2 

2 

Total 

20 

20 

17 

17 

16 

13 

15 

16 

118 

Action grants and operating grants 

The  table  overleaf  presents  the  number  of  projects  per  specific  objective  financed 
between 2014 and 2020. A large share of action and operating grants is allocated to the 
specific  objective  on  access  to  justice  for  all.  The  second  largest  share  of  projects 
provided  funding  to  advance  judicial  training.  Third  in  terms  of  number  of  projects  is 
judicial cooperation in civil and criminal matters. The first three specific objectives have 
an  equal  allocation  in  terms  of  the  projects  related  to  them.  By  contrast,  with  only  32 

15 

 
 
 
 
 
 
 
 
 
projects over the entire programming period, the fewest projects were financed under the 
specific objective of drug prevention policy.13 

Table 5: Number of funded projects per year (action grants and operating grants)14 

SOs 

2014 

2015 

2016 

2017 

2018 

2019 

2020 

TOT 

Judicial 
Cooperation 
Access to 
Justice 
Judicial 
Training 
Drugs  

28 

40 

37 

7 

16 

33 

32 

6 

29 

28 

32 

5 

29 

32 

30 

5 

25 

22 

25 

4 

16 

12 

22 

5 

19 

14 

16 

- 

162 

181 

194 

32 

112 

Total. 
96 
Source: Data for 2014-2015 are retrieved from the interim evaluation of the justice programme. The projects funded 
for 2016-2020 were reconstructed using the project database  

569 

76 

87 

49 

55 

94 

In terms of the type of activities, implemented through action and operating  grants, the 
figure below depicts the distribution over the entire programming period.15 16 

Figure 2 Types of activities implemented by projects funded by grants under the programme (2014-
2020) 

Source: Data for 2014-2015 are retrieved from the interim evaluation of the justice programme. The projects funded 
for 2016-2020 were reconstructed using the project database. 

A  large  share  of  projects  implemented  were  analytical  and  monitoring  activities, 
followed by awareness-raising, information and dissemination and training activities. 

Procurement 

13 The specific objective of drug prevention policy was the only one without operation grants. 
14 Data included in this report mirror those provided by DG Justice and Consumers, for the years available. 
15 Multiple activities per project were considered when creating the overview. 
16 To see the types of activities funded by the justice programme in 2014, 2015 and 2016, see the mid-term evaluation 
of the justice programme 2014-2020, Ernst & Young Financial-Business Advisors, Final Report, April 2018, p. 71-72. 

16 

 
 
 
 
 
                                                           
Over  the  whole  programming  period  2014-2020,  a  total  of  EUR  81,7  million  was 
earmarked for procurement in the 2014-2020 annual work programmes. The largest share 
of the budget for procurement was allocated to the specific objective on judicial training 
(36%), followed by access to justice (30%), judicial cooperation (28%) and initiatives in 
the field of drug prevention policy (6%).  

Table 6 Number of procurement contracts signed per year 

SOs 

Judicial 
Cooperation 
Access to 
Justice 
Judicial 
Training 
Drugs  

Total. 

2014 

2015 

2016 

201717 

201818 

201919 

2020 

32 

1 

28 

2 

63 

36 

2 

23 

1 

62 

22 

1 

19 

- 

42 

26 

1 

31 

- 

58 

40 

2 

32 

- 

74 

24 

- 

45 

- 

69 

21 

- 

28 

- 

49 

Information  communication  technology  tools  and  other  IT-related  activities  were  the 
most  common  types  of  procurement  activities  contracted  over  2014-2020.  These  IT-
related  activities  were  followed  up  by  mutual  learning  initiatives.  By  contrast,  training 
activities were the least most common activity contracted, as shown in Figure 3. 20 

Figure 3 Types of procurement projects (2014-2020) 

Source: Detail on procurement data provided by DG JUST. 

Although training accounted for the fewest in absolute number of projects, their average 
size was the largest in terms of the budget spent per procurement contract. By contrast, 
mutual learning activities, which rank second highest in absolute numbers, had the lowest 
average budget.21 

17 For 2017, one procurement contract was not associated to a specific objective. 
18 For 2018, two procurement contracts were not associated to a specific objective. 
19 For 2019, two procurement contracts were not associated to a specific objective. 
20 The  average  is  computed  over  the  procurement  contracts  provided by  DG  JUST,  which,  for  2014  and 
2016, is lower than the numbers included in the Interim Evaluation.  
21 The  average  is  computed  over  the  procurement  contracts  provided by  DG  JUST,  which,  for  2014  and 
2016, is lower than the numbers included in the Interim Evaluation.  

17 

020406080100120140160180TrainingAwareness raisingAnalytical and monitoringMutual learningICT tools and related activities 
 
 
 
                                                           
Figure 4 Average budget of procurement contracts in thousands EUR (2014-2020) 

Source: Detail on procurement data provided by DG JUST. 

Target groups  

In  terms  of  the  target  groups  reached  per  specific  objective  through  funding  under  the 
justice  programme,  text  mining  of  project  documents  (such  as  final  reports  of  funded 
projects) yielded the result shown in the figures below. 

Figure 5: Target groups per specific objective (2014-2016 left; 2016-2020 right) 

Several  similarities  and  differences  can  be  spotted  between  the  two  periods  2014-2016 
and  2016-2020.  In  particular,  researchers  were  the  most  frequently  mentioned  target 
group  between  2014  and  2016  but  they  were  replaced  by  legal  professionals  between 
2016  and  2020.  Young  people  and  children  also  remain  important  target  groups, 
interestingly  for  all  specific  objectives.  The  target  groups  also  largely  overlap  between 
the two periods, although there were more groups in 2016-2020. 

Distribution of resources by type of beneficiary 

The figure below summarises the distribution of funding to key groups of beneficiaries.  

18 

92138139214512€-€100 €200 €300 €400 €500 €600 Mutual learningICTAwareness raisingAnalytical and monitoringTrainingThousands 
 
 
 
 
 Figure 6: Yearly distribution of funding (2014-2020) 

Source: Interim evaluation of the  justice programme (for 2014-2015), detail on project monitoring data provided by 
DG JUST (for 2016-2020). 

According  to  the  data,  there  is  rather  strong  prevalence  of  higher  secondary  education 
institutions  (HES)  and  research  organisations  (REC)  between  2014  and  2018.  Only  in 
2015,  public  institutions  (PUB),  non-profit  organisations  (OTH)  and  private  companies 
(PRC)  participated  more  frequently  in  funding  under  the  justice  programme.  Although 
higher  secondary  education  institutions  and  research  organisations  still  had  a  relatively 
high share of the budget until 2018, the amount distributed to them decreased steadily as 
of  2017.  Beginning  in  2018,  public  institutions,  non-profit  organisations  and  private 
companies together received over 50% of the available budget. Also as of 2018, the share 
of the budget attributed to public institutions and non-profit organisations rose to, reach 
over 60 % of funds in 2020.  

Distribution of funds and applications 

In  terms  of  geographical  distribution  of  resources,  as  for  predecessor  programmes, 
participation in the justice programme was concentrated in a number of Member States 
(e.g.  Italy,  Belgium,  Spain,  Germany,  the  Netherlands  and  France).  Organisations  from 
these Member States have consistently remained the lead beneficiaries, despite changes 
in  the  numbers  of  applications  over  the  years.  In  contrast,  there  is  a  relatively  limited 
demand for support from countries like Latvia, Poland, Croatia, Slovakia and Cyprus. In 
fact,  when looking at  the selected project  beneficiaries, it  emerges that  not  all Member 
States  participated  in  an  equal  way  in  the  programme:  about  22%  of  all  beneficiaries 
come from either Italy or Belgium. In absolute terms, Spain (157), the Netherlands (120), 
France  (113)  and  Germany  (110)  are  the  other  countries  with  over  100  organisations 
selected for grants under the justice programme.  

The figure below summarises the described geographical allocation of funds. 

Figure  7 : Number of applications,22 and  grants (action and  operating) awarded by country (2014-
2020)23 

22 Including  all applications (successful and unsuccessful).  
23 The figure does not include information on non-eligible countries (i.e. CH, DK, MK, RS, UK), including 
international organisations based in these countries (e.g. IOM). 

19 

 
 
 
                                                           
Source: Data for 2014-2015 are taken from the interim evaluation of the justice programme. The projects funded for 
2016-2020 are taken from the project database. Population data Eurostat 

Similar  to  the  actual  distribution  of  funds,  the  same  EU  countries  that  received  most 
funds,  are  also  the  most  frequent  applicants.  Looking  at  the  whole  period,  Italy  (889 
applicants,  17.8%)  and  Spain  (503,  10%)  had  the  most  applicant  organisations, 
comprising combined over 25% of all applicants. Belgium (295), France (274), Germany 
(284)  and  Romania  (272)  also  had  over  250  organisations  applying  to  the  justice 
programme between 2014 and 2020.  

The  overleaf  figure  provides  an  overview  of  the  number  of  the  applications  and  funds 
awarded over the programming period.  

Figure 8 Number of applications and grants awarded (2014-2020) 

Source: Data for 2014-2015 are taken from the interim evaluation of the justice programme. Data on the 
projects funded for 2016-2020 are taken the project database. 

Overall,  the  number  of  applications  fell  over  the  course  of  the  programming  period  to 
reach the lowest point in 2019 with 143 applications. The number of applications started  
increasing again in 2020. The fall in applications resulted in a similar reduction in terms 
of grants awarded to projects. The number of selected projects fell from 112 in 2014 to 
48 in 2020, the lowest number over the whole programming period.  

Progress in indicators 

20 

 
 
 
In  order  to  measure  progress  against  the  baseline  situation,  the  Commission  created  a 
series  of  programme  indicators  for  the  justice  programme  2014-2020.  This  section 
presents, the results available for each indicator, against the 2020 target, the baseline, the 
milestone  and  the  actual  results  achieved.  The  results  are  either  based  on  the  analysis 
carried  out  or  on  official  sources,  including  information  reported  in  the  draft  general 
budget of the EU for the 2020 and 2021 financial years.24 

Indicator tracking progress on the general objective 

The  Regulation  establishing  the  justice  programme  does  not  stipulate  an  indicator  to 
track  its  general  objective.25  However,  the  European  Commission’s  programme 
statements, which accompany the draft general budget of the EU,26 refer to the percentage 
of legal practitioners trained as the main indicator for measuring the justice programme’s 
impact.  The  indicator  is  defined  as:  the  cumulative  number  of  legal  professionals 
receiving training, not only through the justice programme, on EU law or law of another 
Member State in the area of civil justice, criminal justice and fundamental rights.  

Table 7 - Indicator tracking progress on the general objective: legal practitioners trained 

Target 

Baseline 

Results 

Overall 
assessment 

700 000 legal practitioners will have received training by 2020. 

239 000 legal practitioners were trained in 2013. 

By  2014,  371  000  practitioners  had  received  training.  In  2016,  the 
cumulative  number  of  trainees  had  increased  to  approximately  640 
000.  According  to  the  2020  programme  statement,  820  199  legal 
practitioners  were  trained  by  2017,  of  which  180  000  practitioners 
had been  trained in  2017 alone.  In 2018, the cumulative number of 
trained legal practitioners surpassed one million. 

The target has been reached ahead of schedule. 

Result indicators measuring progress towards the specific objectives 

For  the  specific  objective  to  support  and  facilitate  judicial  cooperation  in  civil  and 
criminal  matters,  the  justice  programme's  performance  is  measured  by  tracking  two 
main indicators: the average time of the surrender procedure under the European Arrest 
Warrant  in  cases  where  the  person  consents  to  the  surrender,  and  the  number  of 
exchanges of information in the European Criminal Records Information System. 

Table 8 - Result indicator one: average time of the surrender procedure under the European Arrest 
Warrant in cases where the person consents to the surrender 

Target 

The average time is equal to 10 days by 2020. 

24 COM (2019) 400 - Draft Budget of the European Union for the financial year 2020. Working document 
Part I – Programme Statements of operational expenditures; and COM (2020) 300 - - Draft Budget of the 
European  Union  for  the  financial  year  2021.  Working  document  Part  I  –  Programme  Statements  of 
operational expenditures.  
25 To contribute to the further development of a European area of justice based on mutual recognition and 
mutual trust, in particular by promoting judicial cooperation in civil and criminal matters 
26  COM  (2017)  400  -  May  2017,  Draft  Budget  of  the  European  Union  2017.  Information  on  indicators 
related to the general objectives of the Programmes and indicators related to the specific objectives of the 
programmes are provided by this document. 

21 

 
 
                                                           
Baseline 

Results 

Overall 
assessment 

According to DG Justice’s analysis of Member States reports to the 
Council, the average time was 14-20 days in 2013. 

In 2014, the average time was 19,4 days, falling to 16 days by 2015. 
In 2017, the average time fell further to 15 days but increased again 
to 16,4 days in 2018. For subsequent years, no data are available. 

Two main factors impede the accuracy of this result indicator. First, 
data  are  difficult  to  collect.  Second,  data  that  is  available  is 
susceptible  to  influence  by  external  factors,  such  as  the  different 
degree of incorporation of EU directives at national level, as well as 
the  different  degree  of  judicial  system  reforms  within  EU  Member 
States that have a significant impact on the level of enforcement of 
EU law27.  In turn, these factors make it very difficult to estimate to 
which  extent  the  justice  programme  contributes  to  a  change  in  the 
average number of days. On the basis of the current data, the target 
for this result indicator was not reached. 

Table  9  Result  indicator  two:  number  of  exchanges  of  information  in  the  European  Criminal 
Records Information System. 

Target 

Baseline 

Results 

The  number  of  exchanges  of  information  in  the  European  Criminal 
Records Information System is equal to 3 500 000. 

In 2012, 300 000 exchanges were recorded. 

In 2014, there were 1 250 000 exchanges, increasing to 1 978 000 in 
2016. This increasing trend continued with 2 963 789 exchanges in 
2018 and 3 700 000 in 2019. 

Overall 
assessment 

The number of exchanges rapidly increased after 2015, reaching and 
surpassing the target ahead of schedule in 2019. 

For  the  specific  objective  to  facilitate  and  support  judicial  training,  one  main  result 
indicator measures the justice programme’s performance: the number and percentage of 
members  of  the  judiciary  and  judicial  staff  that  participated  in  training  activities,  staff 
exchanges, study visits, workshops and seminars funded by the justice programme. The 
number of participants is measured yearly. 

Table  10  -  Result  indicator:  the  number  and  percentage  of  members  of  the  judiciary  and  judicial 
staff  that  participated  in  training  activities,  staff  exchanges,  study  visits,  workshops  and  seminars 
funded by the justice programme. 

Target 

Baseline 

Results 

The  number  of  participants  per  annum  is  equal  to  16  000  for  the 
justice programme. 

In 2011, the number of yearly participants was 6 681. 

The  number  of  participants  almost  doubled  in  comparison  to  the 
baseline  to  reach  12  333  in  2014.  An  increase  in  number  of 
participants  was  also  recorded  the  following  year  with  16  723  in 
2015. However, the number of participants fell in 2016 to 13 930. In 

27 These obstacles were mentioned in several project reports and shown also in the Justice Scorecards. 

22 

 
 
 
                                                           
Overall 
assessment 

2018,  the  number  of  participants  increased  again  to  reach  18  358. 
This positive development did not endure as, according to the latest 
data, the number of participants fell again to 10 799 in 2020.28 

The  results  presented  should  be  interpreted  taking  into  account  the 
following  limitations.  Although  the  indicator  is  well  aligned  to  the 
specific objective of judicial training, the data collected directly from 
projects  funded  through  the  justice  programme  are  not  exhaustive 
and do not give a detailed picture of the number and percentage. 

Though  the  results  should  account  for  issues  of  data  unavailability, 
the justice programme only met its goal in 2015 and 2018. 

For  the  specific  objective  to  facilitate  effective  access  to  justice  for  all,  including  to 
promote  and  support  the  rights  of  victims  of  crime,  while  respecting  the  rights  of  the 
defence,  performance  of  the  justice  programme  is  measured  by  two  main  result 
indicators.  One  is  the  number  of  hits  on  the  e-justice  portal  to  track  the  justice 
programme’s  performance.29  The  other  is  the  number  of  victim  support  organisations 
with national coverage to track implementation of Directive 2012/29/EU. 

Table 11 Result indicator one: number of hits on the e-justice portal 

Target 

Baseline 

Results 

Overall 
assessment 

The number of hits increases by 20% per annum. 

In 2012, the total number of hits was 630 000. 

Over  the  years,30  the  number  of  hits  on  the  e-justice  portal  was:  
1 136 849 hits in 2014,  
1 751 180 hits in 2015, 
1 884 600 hits in 2016, 
2 690 574 hits in 2017,  
2 962 558 hits in 2018 and 
4 343 547 hits in 2019.  
This  corresponds  to  an increase of  over 20 % every  year, except  in 
2016 when it rose by 8 % and by only 10% in 2018. 
The target was for the hits on the e-justice portal to rise by 20% each 
year, therefore to reach 2 708 885 by 2020. This target was achieved 
ahead of schedule in 2018. 

Table  12  -  Result  indicator  two:  number  of  victim  support  organisations  with  national  coverage 
(implementation of Directive 2012/29/EU) 

Target 

By  2020,  there  is  at  least  one  victim  support  organisation,  which 
fulfils the quality standards and indicators set  by Member States or 
developed  individually  by  the  organisation,  in  each  Member  State. 
There were a total of 27 victim support organisations in 2020. 

28 European Commission, key monitoring indicators of the justice programme (link). 

29 Source of data: DG Justice. 
30  As  of  2017,  the  data  are  obtained  from  a  new  tool  for  website  statistics  tracking,  with  a  different 
calculation method. The targets for 2014-2016 were adjusted to the new calculation method. 

23 

 
 
 
                                                           
Baseline 

Results 

In 2012, there were in total 10 victim support organisations. 

in  2012, 

the  baseline 

the  number  of  victim  support 
From 
organisations began to increase. In 2014, it  doubled to reach 20. One 
year later, it rose to 22 and remained at that level in 2015 and 2016. 
In  2017,  the  number  of  victim  support  organisations  reached  the 
benchmark of 27 and remained at that level until 2019.  

Overall 
assessment 

The  target  number  of  victim  support  organisations  was  reached 
ahead of the deadline already in 2017. 

Last  but  not  least,  the  performance  of  the  justice  programme  in  terms  of  achieving  the 
specific  objective  to  support  initiatives  in  the  field  of  drug  prevention  policy  is 
measured  by  two  main  result  indicators.  The  first  is  the  number  of  new  psychoactive 
substances  assessed  (including  through  testing,  if  necessary)  to  enable  the  EU  or  the 
Member  States  to  take  appropriate  action  to  protect  consumers,  depending  on  the  type 
and level of risk that they may pose when consumed by humans. The second indicator is 
the percentage of problem opioid users in drug treatment. 

Table  13  -  Result  indicator  one:  the  number  of  new  psychoactive  substances  assessed  (including 
through testing, if necessary) to enable the EU or the Member States to take appropriate action to 
protect consumers,  depending on the type and level of risk that they  may pose when consumed  by 
humans 

Target 

Baseline 

Results 

Overall 
assessment 

By  2020,  the  number  of  new  psychoactive  substances  assessed  is 
equal to 95. 

In  2012,  the  number  of  new  psychoactive  substances  assessed  was 
68. 

Having  peaked  at  100  newly  assessed  psychoactive  substances  in 
2015,  the  number  of  new  psychoactive  substances  decreased 
significantly  over  the  course  of  the  years.  In  2016,  there  were  66, 
down  to  52  in  2017,  53  in  2018  and  53  in  2019  new  psychoactive 
substances assessed. 

The target was not achieved by 2019. Although data for 2020 is still 
missing,  the  constant  decrease  in  the  number  of  new  psychoactive 
substances assessed strongly suggests that the target was not reached 
in  2020  either.  However,  this  result  should  be  interpreted  with 
caution,  as  the  decrease  in  new  psychoactive  substances  is  not 
necessarily a negative result. The reporting and assessment of fewer 
substances may suggest that fewer illicit substances are available on 
the  market.  However,  the  2021  programme  statement  reported  how 
the  substances  assessed  in  recent  years  are  more  dangerous  than 
those assessed before. In other words, when the target was set, more 
substances  were  reported  and  detected.  The  situation  has  since 
changed, with a steady fall in the number of substances reported. 

Table 14 - Result indicator two: percentage of problem opioid users in drug treatment 

Target 

By 2020, the percentage of problem opioid users in drug treatment is 
at a level of 60%. 

24 

 
 
 
Baseline 

Results 

Overall 
assessment 

In 2011, 50% of problem opioid users were in drug treatment. 

Between  2014  and  2017  the  percentage  remained  steady  at  50%, 
before  decreasing  slightly  in  2018  to  48%.  In  2019,  the  percentage 
increased again to its baseline level of 50%. 

The target was not reached over the entire period for which data are 
available. Similar to result indicator one, the steady level of problem 
opioid  users  in  drug  treatment  strongly  suggests  that  the  target  was 
not met in 2020 either.  

4.  METHOD 

Short description of methodology 

The  evaluation  process  was  supported  by  an  external  study  by  Fondazione  Giacomo 
Brodolini  Srl  SB  and  Ernst  &  Young31.  The  criteria  used  for  the  evaluation  include: 
relevance,  effectiveness,  efficiency,  coherence,  equity,  scope  of  simplification  of  the 
programme  and  EU  added  value.  The  methodological  approach  was  aligned  with  the 
questions for the evaluation criteria. Annex 3 provides a comprehensive overview of the 
evaluation criteria and the evaluation questions.  

In  order  to  answer  these  evaluation  questions,  the  external  study  applied  a  mixed 
methods approach. This means the evaluation integrates and compares quantitative and 
qualitative  approaches,  data  collection,  data  analysis  and  interpretation.  The  data 
collected  was  then  used  to  perform  a  series  of  quantitative  and  qualitative  analyses. 
Quantitative analyses helped provide insights, both at programme level and project level. 
At  programme  level,  quantitative  methods  assess  for  example  how  and  where  the 
programme  used  the  funding  available.  At  project  level,  quantitative  analyses  took  a 
granular view of the results of individual projects. To carry out the qualitative analysis, 
automated  text  mining  techniques  reinforced  with  machine  learning  algorithms 
were used to interpret efficiently the large volumes of data on individual projects.  

The study chose this mixed methods approach to strengthen the reliability of data and the 
findings and recommendations. To be specific, the study triangulated multiple sources of 
information, to broaden and deepen the understanding of the processes through which the 
justice  programme  achieves  its  results.  The  external  study  also  scrutinised  how  the 
environment,  in  which  the  justice  programme  is  implemented,  affects  the  results  it 
produces.  

The methodological approach taken by the study is divided into three main tasks: 

  Task one: in-depth desk research, including the extraction, collection and analysis 
of monitoring information. The aim is to provide an overview of the state of play, 
the performance and main achievements of the justice programme. 

  Task  two:  two  main  activities,  one  is,  field  research,  including  interviews  with 
beneficiaries,  stakeholders  and  European  Commission  officials,  was  conducted. 
The  field  research  also  encompassed  focus  groups,  an  e-survey  to  programme 
committee members and an additional documentary review. The other activity was 

31 Evaluation of the justice programme 2014-2020 Final Report, not published 

25 

 
                                                           
to  analyse  a  sample  of  projects  and  services  funded  by  the  justice  programme. 
Both activities aim to provide more nuanced and clearer insights into key features 
and  achievements  of  a  limited  set  of  projects.  At  project  level,  the  following 
aspects were scrutinised: type of action, link with the specific objectives, outcomes 
and results and their contribution to the key evaluation criteria. 

  Task three: drafting  case studies and answering the  evaluation questions. Based 
on  the  information  and  findings  obtained  from  the  two  steps  above  and  their 
triangulation,  the  study  launched  eight  case  studies,  focusing  on  a  set  of 
operational and procedural aspects covering the justice programme. Building on the 
findings  from  the  case  studies,  the  study  answered  the  evaluation  questions  and 
made  suggestions  for  improvement.  Annex  3  provides  an  overview  of  the  case 
study topics. 

Annex 3 also provides, an overview of the documents, reviewed at project level to 
conduct the desk research for task one and two. 

The field research related to tasks two and three focused on the following stakeholders: 

  beneficiaries of funding provided by the programme, including national, regional 
and local authorities in all participating countries, international organisations, EU-
level umbrella organisations, NGOs, academic and research institutions, networks 
and representatives of target groups. In some instances, potential beneficiaries were 
also consulted. 

  Commission staff, such as programme managers and policy officers; 

 

the justice programme committee members representing EU Member States. 

The study chose a semi-structured design for the interviews, which were conducted via 
video conferencing.  The questions  were  tailored  to  the category of stakeholders, taking 
into account their different contribution to the evaluation questions. As a first step, five 
scoping interviews with DG JUST officials were organised to obtain more in-depth data. 
In  addition,  and  to  supplement  the  answers  to  the  evaluation  questions,  a  total  of  28 
interviews  was  conducted.  These  28  interviews  were  organised  with  28  project 
beneficiaries, covering both action and operation grants and procurement contracts.  

Furthermore, four focus groups were organised to deepen and validate the findings from 
the interviews. Three focus groups were held with a sub-sample of project beneficiaries 
and  one  with  DG  Justice  project  officers.  The  topics  of  the  focus  groups  with 
beneficiaries were: 

  Focus group one was on case study one: Recurring beneficiaries and continuations 

of projects. 

  Focus  group  two  was  on  case  study  three:  why  are  some  institutional  bodies 
reluctant to apply? Group two also provided insights into case study seven: are the 
beneficiaries selected the best fit to help the target groups? 

  Focus  group  three  was  on  case  study  8:  how  has  gender  mainstreaming  been 
promoted  within  the  funded  intervention  and  what  where  the  strengths  of  your 
approach? 

The  focus  group  with  project  officers  aimed  to  collect  additional  and  complementary 
information, focusing on management and procedural aspect of project implementation. 

26 

 
The  study  also  included  two  online  surveys  to  complete  the  field  research.  First,  the 
programme  committee  members  e-survey.  The  goal  of  the  survey  was  to  collect 
comprehensive and specific information on the application process, the main features and 
results of projects and activities funded by the justice programme, and to capture insights 
and expert views.  The survey results  enabled the study  to  gain  an understanding of the 
performance  of  the  justice  programme  against  the  evaluation  criteria.  In  total  21 
responses were collected, covering 17 countries (AL, BG, CY, EE, FI, GR, HR, IT, LT, 
LU,  MT,  PT,  RO,  SE,  SI,  SK).  In  addition  to  the  programme  committee  members  e-
survey, the study included a second survey targeting 200 project beneficiaries to answer  
the question raised in case study four: how do EU projects compare in terms of costs and 
benefits  to  similar  actions  funded  by  other  sources?  The  second  survey  elicited  11 
responses. 

Limitations and robustness of findings 

While  overall  the  methodology  enabled  the  study  to  draw  feasible  conclusions  to  the 
evaluation  questions,  conducting  the  evaluation  study  presented  some  challenges  that 
resulted in some gaps and limitations. 

These limitations were: 

1.  The  coverage,  completeness  and  consistency  of  the  data  used  to  monitor  the 
justice programme. In particular, the variety of sources of data was a challenge, as 
a  result  of  the  change  in  the  project  information  base  and  the  migration  to  e-
grants  in  the  middle  of  the  programming  period.  This  led  to  differences  in  the 
completeness and consistency of the data used to monitor the justice programme. 
The quality and coverage of data prior to 2016 was also uneven at times. 

2.  The lack of a fully-fledged monitoring framework, complete with clearly defined 
and measurable monitoring indicators, benchmarks and targets against which to 
assess the performance of the programme. 

3.  The limited response rate to the e-surveys conducted for the data collection work. 

4.  Data  extraction  proved  to  be  problematic  in  some  instances.  To  be  specific,  the 
study encountered challenges when using automatic data extraction techniques to 
extract semi- structured information, such as monitoring information reported in 
the annexes to the technical reports. Due to differences in format, the feasibility 
of using readily available information needs to be put into perspective.  

To address these limitations and challenges, the study took the following measures: 

1.  To  address  gaps  and  inconsistencies  in  the  monitoring  data,  the  study  used 
additional sources of data. The study compared to this end different data sources and 
merged them into one database. In particular, ongoing interviews with coordinators 
of  sampled  projects  and  European  Commission  officials  proved  useful  to  plug  the 
data gaps. Furthermore, the procurement sector of DG JUST was contacted to obtain 
data. 
new 

comprehensive 

more 

and 

2.  To remedy the difficulties, related to the monitoring indicators, the indicators for the 
study followed, where possible, the approach by the interim evaluation. The analysis 
of  current  indicators  built  on  official  and  approved  figures  from  2018  with 
additional manual data cleaning carried out to increase the coverage of data before 

27 

 
 
2016.  

3.  To remedy the low response rates in surveys, the response time was extended. 

5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS 

This section outlines the results of the analysis against the seven evaluation criteria: 
effectiveness; efficiency; relevance; coherence, complementarity and synergies; EU 
added value; equity and scope for simplification. 

Effectiveness  

The evidence collected suggests that the justice programme has overall been effective in 
terms of both the type of activities funded, and the corresponding project outcomes.32 The 
physical  and  financial  implementation  of  the  programme  has  been  satisfactory, 
despite visible differences between specific objectives. In terms of the indicators defined 
in  the  legal  base  of  the  justice  programme,  the  analysis  confirmed  that  several  targets 
have  been  achieved,  some  even  ahead  of  schedule.  This  is  the  case  for  the  following 
indicators: 

i. 

ii. 

iii. 
iv. 

cumulative number of legal professionals receiving training33 on EU law or law 
of another Member State; 
number  of  exchanges  of  information  in  the  European  Criminal  Records 
Information System; 
number of hits on the e-justice portal; 
number of victim support organisations with national coverage. 

By contrast, the goal defined for the indicator tracking the average time of the surrender 
procedure  under  the  European  Arrest  Warrant  was  not  reached.  Although  the  justice 
programme promotes implementation of the European Arrest Warrant, it should be taken 
into account that the EU Member States are not legally obliged to report on the average 
time of the surrender procedure. Moreover, identifying the average time of the surrender 
procedure may depend, in some cases, on additional requests for information at country 
level. These are external factors outside the scope of the justice programme. The targets 
set by indicators tracking the specific objective on drug prevention policy were also not 
met (the number of new psychoactive substances assessed and the percentage of problem 
opioid users that are in drug treatment). Similar to the surrender procedure, the number of 
new psychoactive substances assessed and the share of opioid users undergoing treatment 
can vary from  year to  year based on several variables that do not depend on the  justice 
programme.  These  variables  include  the  macroeconomic  conditions  and  the  rate  of 
development of new psychoactive substances. For the other indicators, which do not have 
a baseline for comparison, the analysis did not reach a concrete assessment as the targets 
or benchmarks are not defined. 

In a similar vein, the lack of high quality data rendered the evaluation of some indicators 
rather  difficult.  This  was  the  case  especially  for  indicators  with  a  qualitative  emphasis 

32 See Section four and the annex for a summary of the field and desk research undertaken. 
33 The indicator also includes professionals who receive training not through the justice programme, but via 
other sources and initiatives.  

28 

 
 
 
 
 
                                                           
such as the level of knowledge of Union law and policies in the groups participating in 
activities  funded  by  the  programme  compared  to  the  entire  target  group  and  the 
participants'  assessment  of  the  activities  in  which  they  participated  and  of  their 
(expected)  sustainability.  It  is  also  worth  emphasising  that  a  number  of  indicators 
described for the specific objectives are not necessarily  programme-specific as they are 
reported  in  the  general  budget  of  the  European  Union.  Achieving  these  objectives  can 
depend  on  multiple  external  factors,  including  the  COVID-19  pandemic,  which  was 
reported  as  the  most  influential  external  factor.  In  fact,  the  pandemic  required  re-
designing or cancelling some activities entirely.  

The uneven pace at which Member States transpose and enforce EU law and the refugee 
crisis  in  relation  to  the  specific  objective  access  to  justice  for  all  are  also  important 
external  factors  that  had  an  impact  on  the  effectiveness  of  the  justice  programme.  The 
refugee crisis had a negative impact on the effectiveness of the justice programme, as it 
led EU Member States to divert their priorities from the programme’s specific objectives 
to addressing the influx of refugees and the consequent judicial obligations.  

In addition to these external factors, internal factors were also identified as having an 
impact on the justice programme’s effectiveness. These factors are predominantly linked 
to  consortia’s  capacity  to  comply  with  the  standards  of  the  application  process  to 
participate  successfully  in  the  justice  programme.  A  key  factor  here  is  the  discrepancy 
between  applicants’  capacity  to  present  a  well  written  proposal  and  their  capacity  to 
deliver  the  results  they  had  described  in  their  proposals.  This  discrepancy  can  be 
explained  by  the  technical  expertise  needed  to  comply  with  the  standards  set  in  the 
application process. 

to 

to 

up 

90% 

grants 

action 

Another  important  internal  factor  is  the  level  of  the  co-funding  rate  under  the  justice 
programme. Given programme’s general objective to establish a European area of justice 
for  all,  it  should  be  taken  into  consideration  that  the  co-funding  represents  a  financial 
obstacle to small and grassroots organisations. Organisations operating  at this level lack 
the  financial  resources,  needed  to  participate  in  the  programme.  However,  the 
interviewees  welcomed  the  European  Commission’s  decision  to  increase  the  EU 
contribution 
costs.34 
In line with the results for the indicators, the programme committee members surveyed 
also  confirmed  the  effectiveness  of  the  programme.  The  majority  viewed  the  justice 
programme’s  activities  as  highly  effective  and  sustainable.  80%  of  respondents  (16) 
answered that it performed in terms of meeting its objectives ‘to a high extent’ and 15% 
(3)  ‘to  a  very  high  extent’.35  One  respondent  answered:  ‘I  do  not  know’.  The  28 
consulted  project  beneficiaries  seconded  that  view  and  reported  that  their  projects 
indeed  did  achieve  or  are  achieving  the  expected  results.  In  particular,  activities 
carried  out  under  the  specific  objective  of  judicial  training  were  perceived  very 
positively.  This  statement  is  supported  by  the  allocation  of  financial  resources.  The 
analysis of technical reports of funded projects in conjunction with the interviews results 
showed that there is a suitable allocation of funds to projects to obtain specific objectives, 
with the exception of the budget for drug prevention policy.36  

project 

total 

of 

34  Interview feedback: two representatives from project beneficiaries. 
35 Source: Survey to programme committee members (20 replies) 
36 Interviews with five European Commission policy officers confirmed the effectiveness of funded actions 
and their contribution to the achievement of specific objectives. 

29 

 
 
 
 
                                                           
On  this  specific  objective,  a  mismatch  between  the  funding  available  and  the  scope  of 
prevention  measures  should  be  put  into  perspective.  It  is  worth  noting  that  the  justice 
programme focuses primarily on aspects of judicial cooperation for this specific objective 
and thereby complements other funding instruments, such as the Internal Security Fund 
or the health for growth programme. 

In  terms  of  the  type  of  activities,  the  evidence  contained  in  the  project  documents  and 
data  from  the  interim  evaluation  show  that  projects  across  specific  objectives  plan 
different types of activities,37 and not just those related to their core objective. In fact, 
the majority of projects analysed between 2016 and 2020 (85%, or 264 projects) included 
more  than  one  type  of  activity.  Although  the  presence  of  multiple  activities  within  the 
same project is not an indication of effectiveness per se, it suggests that projects covered 
multiple  types  of  activities  in  order  to  meet  the  objectives  and  needs  of  the  justice 
programme from different angles. By addressing project-related issues through different 
activities,  the  potential  effectiveness  of  interventions  is  increased  as  this  minimises  the 
risk that projects  omit  critical  aspects.  A key factor  here  is  the flexibility  of the  justice 
programme, which gives scope to tailor project activities to emerging needs. The analysis 
indicates that projects funded are generally fit for purpose.  The survey of applicants 
shows 
the  European 
Commission’s  communication  on  funding  activities  positively.  Beneficiaries  reported 
that the quality of communication activities was sufficient, clear, intelligible and targeted 
to the right audience.38 By contrast, the evaluation of the effectiveness of communication 
activities  through  funded  projects  is  more  difficult  as  data  available  lack  a  baseline  for 
comparison and therefore do not allow to draw robust conclusions.  

that  both,  successful  and  unsuccessful  applicants  view 

Efficiency 

The majority of beneficiaries, over 60% of consulted beneficiaries, agreed that the justice 
programme  is  efficient  as  the  benefits  gained  by  participating  in  the  programme  are 
higher  than  the  costs  incurred.  However,  stakeholders  also  underlined  that  it  is  very 
demanding  to  obtain  the  expertise  and  administrative  capacity,  required  to  be  able  to 
receive funding under the justice programme.39  

The  COVID-19  pandemic  is  an  important  external  factor  that  had  an  impact  on  the 
justice  programme’s  efficiency.  The  obstacles 
the  pandemic  created  rendered 
implementation  of  funded  projects  difficult  as  activities  had  to  be  re-designed  or 
interrupted.40  In  turn,  interviewees  stated  that  the  cost-benefit  ratio  of  measures 
worsened. What is more, in comparison with alternative sources of funding, both national 
and  international,  the  justice  programme  is  less  accessible  due  to  the  requirements  for 
applicants. This finding  should be put into perspective since the European Commission 

37  Project  documents  include  primarily  technical  reports  of  implemented  projects.  For  a  comprehensive 
overview please consult the annex.  
38 Interview feedback: six representatives from project beneficiaries. 
39  In  the  interim  evaluation  of  the  justice  programme,  the  proposal  preparation  with  34  %  and  the 
accounting  and  reporting  with  (45%)  accorded  together  for  79%  of  the  costs  not  related  to  project 
implementation. Six interviewed beneficiaries also underlined the demanding administrative burden. 

40 Feedback from four beneficiaries. 

30 

 
  
                                                           
took  action  in  2014  to  increase  the  programme’s  efficiency  by  streamlining  the  three 
predecessor funding instruments41 into one set of application rules and procedures.  

Streamlining the application rules and procedures was received as a welcome change in 
the  management  of  the  justice  programme  as  confirmed  by  the  interim  evaluation.  The 
new funding rules brought in with the justice programme also facilitated a more efficient 
use of the available budget as excess resources allocated to a specific objective could be 
rerouted  to  another  objective  with  a  higher  spending  capacity.  As  a  result,  resources 
linked  to  underperforming  calls  for  proposals,  thus  calls  with  low  quality  applications, 
would be reallocated to calls for proposals with a higher number of quality applications. 
Although  the  outlined  measures  improved  the  use  of  the  available  budget,  the  justice 
programme  still  suffers  from  underspending  on  projects.  Within  a  sample  of  116 
projects  completed  between  2015  and  2018,  102  projects  (88%)  requested  a  final  EU 
contribution  lower  than  the  amount  initially  requested  for  a  total  of  EUR  5  495213,21 
over  the  4  years.  Underspending  is  due  to  financial  needs  being  overstated  when 
budgeting,  making  savings  over  the  course  of  the  project  implementation  or  in  some 
cases by partners leaving the consortium.  

Another  issue  that  emerged  from  the  analysis,  is  the  duration  and  structure  of  the 
programme  management  cycle.42  Before  2018,  irregularities  in  the  timing  of  the 
adoption  of  annual  work  programmes,  which  resulted  in  significant  delays  in  the 
publication and launch of calls for proposals, made it very difficult for applicants to plan 
and  prepare  to  participate  in  the  justice  programme.  This  is  especially  true  for 
organisations  that  operate  on  a  programme-based  approach.  However,  the  European 
Commission made considerable efforts to speed up the project management cycle, which 
led to a higher regularity and predictability in the adoption of annual work programmes 
and the subsequent launch of calls for proposals as of 2018. 

The accessibility of information on calls for proposals and on the justice programme as a 
whole  has  also  improved,  which  facilitated  planning  for  applicants.43  Despite  visible 
improvements  in  the  Commission’s  management  of  the  justice  programme,  some 
applicants,  as  stated  by  five  beneficiaries,  still  find  the  time  it  takes  to  submit  project 
proposals too ambitious and the time between submission and selection of proposals too 
long.  In  particular,  these  interviewees  underscored  that  the  long  waiting  time  to 
potentially receive funding puts the relevance of their project proposals at risk. 

In this context, the framework partnership agreements brought in with the launch of the 
justice  programme  in  2014,  are  an  important  financial  instrument  that  should  make 
project planning more efficient. The importance and advantage of framework partnership 
agreements  are  illustrated  by  the  yearly  pre-financing  that  beneficiaries  receive  as 
confirmed  by  two  interviewed  operating  grant  beneficiaries.  By  having  a  secure  yearly 
contribution from the European Commission, recipients of operating grants can plan their 
projects more efficiently. However, submitting a yearly application under the umbrella of 
the framework partnership agreement is a loss in efficiency for the justice programme, as 
part  of  the  resources  of  an  operating  grant  is  used  to  refinance  itself.  In  addition,  the 

41 The  criminal  justice  programme, the  civil justice  programme and the  drug  prevention and  information 
programme. 
42 Finding based on interviews with 5 beneficiaries. 

43 Finding based on interviews with three beneficiaries. 

31 

 
                                                           
yearly  application  puts  a  considerable  administrative  burden  on  the  beneficiaries  of  the 
operating grants and also on the European Commission.  

In conclusion, a high share of beneficiaries believes that, despite the remaining obstacles, 
the balance between costs and benefits under the justice programme is better than under 
alternative sources of funding. In other words, responses from beneficiaries indicate that 
the comparatively higher costs of participating in the justice programme are outweighed 
by even greater benefits.  

Relevance 

Overall,  the  evaluation  confirmed  that  the  justice  programme  has  remained  highly 
relevant and aligned with the needs of selected target groups over the years. One of the 
key  features  of  the  programme  has  been  the  ability  to  adapt  and  adjust  its  priorities  in 
light  of  emerging  needs.  For  instance,  following  a  string  of  terrorist  attacks  in  Europe, 
two  calls  for  proposals  were  issued  to  combat  the  radicalisation  of  inmates  in  prisons. 
The  programme  achieves  this  flexibility  thanks  to  its  unique  structure,  which 
encompasses  the  scope  of  its  three  predecessor  programmes.  The  general  and  specific 
objectives allowed the European Commission to adapt the justice programme to evolving 
needs within the EU, especially in terms of judicial cooperation.  

In general, the needs identified at the time of the justice programme’s adoption are still 
current and relevant, in particular the general objective of further developing a European 
area of justice based on mutual recognition and mutual trust. While there is insufficient 
data  to  be  able  to  draw  a  general  conclusion  on  the  funded  activities’  alignment  with 
stakeholder  needs,  evidence  collected  through  exchanges  with  stakeholders  strongly 
suggest that the needs covered by the programme remain relevant to stakeholders and are 
likely  to  remain  relevant  in  the  near  future.44  This  finding  from  the  interviews  is  also 
confirmed  by  the  analysis  of  the  23  sampled  action  and  operating  grants,  which  also 
indicates  that  project  objectives  and  activities  are  considered  still  relevant  to  the 
needs of the stakeholders targeted. A reason for the continued relevance mentioned by 
project  coordinators  of  the  sampled  projects  is  that  the  partnerships  forged  by 
participating in the justice programme add value in terms of networking, mutual learning, 
and paving the way for future collaboration.  

All these factors result in  an  increased capacity  of stakeholders. However, stakeholders 
also  identified  the  need  for  intensified  exchanges  between  them  and  the  European 
Commission  to  ensure  that  the  justice  programme  is  more  responsive  and  attuned  to 
stakeholders’  invariably  changing  needs.  To  this  end,  regular  needs  assessments  could 
feed into annual work programmes. The project coordinators consulted also stressed that 
the continuation of one funded project through multiple projects on the same topic is in 
itself  evidence  of  the  programme’s  relevance  to  stakeholders.  In  turn,  the  analysis 
showed  that  the  target  groups  that  received  funding  under  the  justice  programme  are 
relevant  to  reach  programme’s  general  objective.  Specifically,  the  analysis  of  the  23 
sampled  projects  shows  that  projects  mostly  involve  professionals,  including  judges, 
judicial  staff,  lawyers,  notaries  and  court  staff.  Although  efforts  have  been  made  to 
expand  the  target  groups  of  projects  to  include,  for  instance,  prison  staff,  there  is  still 

44 Interview feedback: nine representatives from project beneficiaries. 

32 

 
 
                                                           
room  for  improvement  to  include  even  more  relevant  target  groups.  A  key  area  for 
improvement here is tapping into the broader community of civil society organisations. 
They have become more involved in the justice programme but there is scope to increase 
their involvement, especially in the area of victims’ rights. 

Coherence, Complementarity and Synergies  

The  justice  programme  has  a  good  level  of  coherence  and  complementarity  with  other 
EU  funds,  programmes  and  initiatives.  Especially  in  the  case  of  judicial  training,  the 
decision  to  merging  of  the  predecessor  programmes  into  the  justice  programme  has 
increased  the  coherence  with  other  EU  initiatives,  such  as  the  rights,  equality  and 
citizenship programme. By the same token, this fosters the coherence between different 
training objectives, which reduced the likelihood of duplications in scope and in funding. 
There is  a strong coherence and complementarity  of the specific objective  on access  to 
justice  for  all  with  the  Connecting  Europe  Facility  telecommunications  programme, 
which also contributed to the development of e-justice portal and e-Codex.  

Complementarity  with  national  projects  and  coherence  with  international  obligations  is 
also high. Complementarity with national projects is achieved by the transnational design 
of activities funded under the justice programme, which ensures that activities funded by 
the justice programme do not replace or replicate national activities. 

The  justice  programme  is  also  coherent  with  international  obligations,  such  as  the 
UN 2030 Sustainable Development Agenda. Some general principles of the UN agenda 
can be linked to objectives and areas targeted by the justice programme. For example, the 
UN objective of ensuring peace and security is linked to the creation of just and inclusive 
societies that provide equal access to justice, effective rule of law, as well as transparent 
and  effective  judicial  institutions.  The  EU  is  also  party  to  The  Hague  Conference  on 
Private  International  Law and pursues its international  action in relation to  civil  justice 
mainly  through  this  international  organisation.  As  a  result,  the  justice  programme 
promotes the harmonisation of international private law. 
By  contrast,  there  are  some  overlaps  in  the  justice  programme  at  the  project  level  as  a 
natural consequence of its broad objectives and target  groups, but these do not produce 
duplications or incoherence. The risk of duplication is minimised by the structure of the 
justice programme and the formulation of its general  and specific objectives. These are 
flexible  enough  to  tailor  funding  to  emerging  needs,  but  also  sufficiently  distinct  to 
ensure that specific objectives do not overlap.  

Last  but  not  least,  64%  of  33  surveyed  beneficiaries  indicated  that  they  found  and 
capitalised  on  synergies  of  the  justice  programme  with  projects  funded  by  other  EU 
initiatives,  such  as  the  rights,  equality  and  citizenship  programme,  as  well  as  with 
projects under the justice programme. However, there could be more exchanges between 
different beneficiaries to increase synergies. 

Thus, in summary, the current analysis confirms the findings from the interim evaluation. 
The  justice  programme  achieves  a  high  level  of  coherence,  complementarity  and 
synergies, both internal and external.  

EU added value 

The analysis fully confirms the high added value of the justice programme. This finding 
is  inter  alia  based  on  the  interview  feedback  from  eight  project  beneficiaries  and  four 

33 

 
 
 
 
European  Commission  officials.  In  particular,  EU  added  value  is  evident  through  the 
promotion  of  transnational  projects  to  tackle  cross-border  issues.  EU  added  value 
obtained  through  the  justice  programme  is  also  very  high  in  the  way  it  increases 
coherence  and  uniformity  in  the  application  of  EU  law,  which  is  obtained  through 
training and information exchanges provided by funded projects. High EU added value is 
created by the provision of crucial financial resources to fund activities in key areas that 
are  not  necessarily  high  on  the  agenda  of  individual  Member  States.  In  other  words, 
individual  actions  by  Member  States  would  not  be  able  to  achieve  the  same  results  in 
terms of size and scope as the justice programme at EU level as underlined by 53 % of 
the  21  surveyed  programme  committee  members.  The  unique  value  of  the  justice 
programme consists on this aspect in the fact that it would be difficult for Member States 
to  allocate an  equal  amount of resources to  projects  in  the  areas covered  by the  justice 
programme.  
Against  this  backdrop,  the  conclusions  drawn  in  the  interim  evaluation  of  the  justice 
programme remain valid: limiting existing EU action in this area would have negative 
effects on the overall capacity of both, national and European actors, to address issues in 
the areas covered by the programme or to implement the types of activities funded by the 
programme.45  Hence,  the  most  likely  consequences  of  reducing  or  withdrawing  EU 
involvement  under  the  justice  programme  would  be  significant  difficulties  in  the 
development  of  transnational  projects.  These  would  lead  to  less  coherence  across 
Member States in terms of understanding and implementation of EU law.  

In  addition,  a  sharply  reduced  number  of  judicial  practitioners  participating  in 
networking  and  transnational  training  activities  with  colleagues  from  other  Member 
States would be expected. As a result, this would undermine the adoption of innovative 
legislative  frameworks  or  cutting-edge  approaches  to  tackle  the  needs  of  vulnerable 
groups  and  to  develop  new  tools  and  practices.  In  other  words,  it  would be  difficult  to 
achieve the objectives of an EU-wide strategy.  

Equity 

In line with the results from the interim evaluation, it is clear that resources and support 
of the justice programme are not distributed equally across the different target groups, 
across the EU Member States or across the types of beneficiaries. On the macro level, 
there  is  a  contrast  between  high  application  and  funding  rates  from  countries  like 
Belgium,  Italy,  Spain,  Germany,  the  Netherlands  and  France  and  comparatively  low 
application  and  funding  rates  from  countries  like  Latvia,  Poland,  Croatia,  Slovakia  and 
Cyprus.  The  analysis  of  three  interviews  with  beneficiaries  from  these  less  represented 
countries yielded that the lower application and participation rates can be attributed to the 
limited  demand  for  support  and  the  lower  organisational  and  financial  capacity  of  the 
organisations, especially civil society organisations. What is more, the level of specificity 
of the justice programme’s annual work programmes and calls for proposals may be too 
specific  for  some  applicants.  Two  interviewed  beneficiaries  from  Sweden  and  Finland 
suggested that the limited participation from organisations based in their countries can be 
attributed  to  easier  access  to  abundant  national  funding  tied  to  a  lower  administrative 
burden, and to issues encountered in the recognition of their comparatively higher salary 
levels. Concerning the participation rate of organisations based in Belgium, it should be 
noted  that  the  high  concentration  of  resources  is  mainly  allocated  to  umbrella 

45 Interview feedback: five representatives from beneficiaries. 

34 

 
                                                           
organisations,  which  mostly  have  headquarters  in  Brussels  for  proximity  to  the  EU 
institutions.  Recent  interviews  with  beneficiaries  found  that  the  role  of  network 
organisations  is  seen  positively  as  they  support  and  enable  the  participation  of  less 
experienced  organisations  in  calls  for  proposals  for  action  grants.  Hence,  the  funding 
directed to these organisations is not entirely implemented in Belgium. 

The distribution of the main target groups per specific objective has evolved over the 
programming  period.46  For  example,  the  number  of  target  groups  has  increased  to  12 
according to the project documents available from 2016 to 2020 in comparison to 10 in 
2014-2016.  Also,  in  contrast  to  the  period  2014-2016,  the  justice  programme  primarily 
involved  legal  professionals  and  was  more  focused  on  reaching  citizens  in  general 
between 2016 and 2020. Before 2016, the justice programme rather involved researchers. 
This  change  reflects  the  adjustment  made  by  the  European  Commission  to  prioritise 
funding that has a direct impact on practitioners over funding for research that might not 
reach the intended audience, as discussed in interviews with EU officials.47  

However,  as  the  monitoring  and  data  collection  approaches  of  the  justice  programme 
need to be improved, it is very likely that the target groups reached through funding are 
not  as  diverse  as  they  could  be.  This  argument  is  also  supported  by  the  finding  on  the 
potential  room  for  improvement  to  step  up  dialogue  with  stakeholders.  More  dialogue 
would  help  identify  potential  blind  spots  in  the  portfolio  of  target  groups.  This  finding 
has  already  been  acted  upon  as  it  is  explicitly  mentioned  in  Regulation  2021/693 
establishing the justice programme for 2021-2027, that civil society organisations should 
be  more  involved  as  a  target  group  by  interventions,  especially  in  the  field  of  victims’ 
rights. 

The  themes  of  gender  mainstreaming,  promoting  the  rights  of  the  child  and  promoting 
the  rights  of  people  with  disabilities,  were  rightly  treated  as  cross-cutting  topics  in  the 
funded projects. On gender mainstreaming, the importance of gender equality and non-
discrimination to build a true European area of justice is already included in the preamble 
as  well  as  in  Article  14  of  the  justice  programme  Regulation.  48  In  the  programming 
documents, the principle of equality between women and men is constantly referred to in 
the  annual  work  programmes  between  2014  and  2018.  Gender  mainstreaming  is 
mentioned and framed in the annual work programmes of 2019 and 2020, as an aspect to 
promote when implementing all activities funded under the justice programme.  

Against this backdrop, the analysis concluded that the justice programme does not focus 
specifically  on  gender  aspects,  although  some  calls  for  proposals  include  dedicated 
explanations for candidates on how gender can be taken into account when outlining the 
project  implementation.  Nevertheless,  the  beneficiaries  interviewed  49  stated  they  had 
made conscious efforts to respect gender equality and non-discrimination in all stages of 
the  project  cycle.  The  way  in  which  beneficiaries  presented  their  approach  to  gender 
mainstreaming and contributed to gender equality showed that applicants still struggle to 
understand and apply these concepts.  Hence, the  justice  programme  has  the capacity to 
raise awareness of the need to promote gender equality and gender mainstreaming among 
applicants  and  beneficiaries,  and  there  is  potential  to  improve  the  promotion  of  gender 

46 However, differences can be partly explained by the different size of sources used to carry out the text 
mining exercise.  
47 Interview feedback: Three interviewed EC officials. 
48 Regulation (EU) No 1382/2013. 
49 Interview feedback: seven beneficiaries. 

35 

 
                                                           
equality  and  gender  mainstreaming  at  project  level.  To  tap  this  potential,  it  would  be 
useful  to  look  at  producing  additional  guidance  to  applicants  and  making  a  closer 
alignment of funding with gender mainstreaming. 

As  for  children’s  rights,  annual  work  programmes  enshrine  in  a  similar  fashion  to 
gender  equality  the  need  to  support  the  protection  of  the  rights  of  the  child.  Calls  for 
proposals under the purview of the justice programme contain mainstreaming clauses, 
where  warranted,  which  ensure  that  funded  projects  also  seek  to  promote  equality 
between women and men and the rights of the child. In addition, respect for the rights of 
the  child  is  enhanced  by  requirement  for  organisations  applying  for  funding  and  their 
partners  working  directly  or  having  contact  with  children  to  provide  the  European 
Commission with a description of their child protection policy. 

Evidence at the project level collected through interviews with beneficiaries revealed that 
many  project  consortia  found  it  difficult  to  include  the  rights  of  the  child  when 
addressing judicial matters.50 These difficulties were in interpreting the relation between 
the project objectives and children. Interviewees indicated that they deemed children not 
relevant  or  made  no  distinction  for  minors.51  Similar  to  gender  mainstreaming,  some 
interviewees struggled to describe how their projects would cover aspects relating to the 
rights of the child.52 Thus, to improve coverage of children’s rights in project design and 
implementation, the provision of additional guidance to applicants could clarify how and 
when to include children. 

Last but definitely not least, the justice programme enshrines provisions linked to Article 
10  of  the  Treaty  on  the  Functioning  of  the  European  Union,  which  requires  that  the 
definition and implementation of EU policies and activities aim to combat, among others, 
discrimination  based  on  disability.  Furthermore,  with  the  ratification  of  the  UN 
Convention on the Rights of Persons with Disabilities, the EU has a legal obligation to 
mainstream  disability  matters  into  a  wide  range  of  EU  policies.  In  comparison  to  the 
rather advanced work to mainstream gender equality and children’s rights, the rights of 
people  with  a  disability  do  not  receive  not  as  much  attention  in  projects  funded  by  the 
justice  programme.  Beneficiaries  revealed  in  interviews  that  they  found  it  difficult  to 
include  the  rights  of  people  with  disabilities  when  covering  judicial  matters  that  were 
valid  to  all  people.53  Some interviewees struggled to  describe how their projects  would 
cover the rights of people with disabilities.54 These observations are  also underlined by 
two European Commission officials, who indicated in interviews that projects under the 
justice  programme  are  not  always  well  suited  to  cover  the  rights  of  people  with 
disabilities. This can result in attempts from beneficiaries to justify in their applications a 
link to priorities under the call for proposals that does not necessarily exist. 

Scope for simplification 

Concerning the scope for simplification, the European Commission made major changes 
to  review,  speed  up  and  simplify  the  process  over  the  2014-2020  programming  period. 
Although, the current management model is considered to be suitable, and the project 

50 Interview feedback: three beneficiaries. 
51 Interview feedback: three beneficiaries. 
52  Interview  feedback:  two  beneficiaries  mentioned  it  was  difficult  for  them  to  justify  how  their  project 
would address gender mainstreaming (other than working with a gender balanced team), the promotion of 
rights of the child and the promotion of the rights of people with disabilities. 
53 Interview feedback: four beneficiaries. 
54 Interview feedback: two beneficiaries. 

36 

 
                                                           
beneficiaries have positively welcomed the changes, they also identified issues and room 
for further improvement.  

results-based 

reporting  procedures, 

According to the analysis, the scope for simplification encompasses the need for simpler 
application  and 
improved  monitoring  and 
management  tools  to  collect  more  systematically  information  on  achieved  outputs, 
results and targets, and help for organisations with difficulties working only in English. 
However, it should be mentioned that the perceived burden and complexity varies among 
project  beneficiaries.  Organisations  with  a  larger  capacity  and  greater  experience  in 
applying,  coordinating,  and  participating  in  projects,  find  it  much  easier  and  smoother 
than smaller organisations and first-time applicants. For newcomers, procedures, changes 
in  reporting  templates  and  the  lack  of  swift  responses  to  queries  can  hamper  the 
application process. 

Lastly, despite improvements to the monitoring and management tools, there appears 
to be room for further improvement. In particular there is scope to improve and adapt the 
application portal to the types of beneficiaries supported by the justice programme and to 
support  first-time  applicants  and  beneficiaries  in  using  the  tool  without  the  risk  of 
delaying  their  activities.  The  collection  of  output  data  could  also  be  improved  to 
improve the quality and consistency of information provided by project beneficiaries on 
the results of the project.  

CONCLUSIONS 

As mentioned in the introduction, this first report of the ex- post evaluation of the justice 
programme does not evaluate the long-term impact of funded projects. The conclusions 
and  recommendations  presented  in  this  section  cover  only  the  activities  that  can 
reasonably be evaluated since the justice programme was launched.  

The  previous  section  has  shown,  in  the  answers  to  the  evaluation  questions,  that  so  far 
the justice programme has delivered results on all evaluation criteria.  

In terms of the effectiveness of the justice programme, the current analysis confirmed the 
result of the interim evaluation. The activities funded through action and operating grants 
as well as procurement activities achieve the results they are expected to achieve. In other 
words,  the  available  budget  is  used  to  achieve  results  that  are  well  aligned  with  the 
general and specific objective of the justice programme. A key finding that emerged from 
the  analysis  is  that  activities  promoting  the  specific  objective  on  judicial  training  are 
especially  well  received.  Though  an  overall  positive  assessment  of  the  justice 
programme’s  effectiveness  is  justified,  there  is  room  for  improvement  as  the  justice 
programme  does  not  perform  equally  well  on  every  specific  objective.  In  fact,  the 
specific  objective  drug  prevention  policy  is  allocated  a  significant  lower  amount  of 
funding than the other specific objectives, even though it receives numerous applications. 
This results in a very low award rate and requires the European Commission to be very 
strict in evaluating the proposals, such that only a few of the most relevant projects can 
be funded.  

However,  this  criticism  needs  to  be  put  into  perspective  as  DG  Justice  addressed  this 
problem  by  moving  the  objective  of  initiatives  in  the  field  of  drug  prevention  policy 
outside the remit of the justice programme for the next programming period, as suggested 
in  the interim  evaluation. Concerning  interaction with the target groups, the analysis 
shed  light  on  the  need  to  create  a  more  sophisticated  monitoring  system,  which  would 
facilitate data collection in a reliable and systematic way. This is of major importance as 

37 

 
a  better  monitoring  system  would  enable  the  Commission  to  obtain  more  in-depth 
insights about target groups, and thereby help to increase the programme’s effectiveness. 
Fortunately,  the  European  Commission  covered  this  problem  in  its  proposal  leading  to 
Regulation  2021/693  providing  the  legal  base  for  the  justice  programme  between  2021 
and 2027. The Regulation envisions that indicators will be consolidated and simplified. It 
will also improve the calculation of indicators as their calculation method will be based 
solely on hard data.  

In  addition,  in  terms  of  implementing  the  justice  programme  and  the  application 
procedure, the analysis identified the need to provide additional guidance to applicants in 
order  to  promote  the  quality  of  writing  in  proposals.  Additional  guidance  is  needed  as 
there  is  a  discrepancy  between  the  capacity  of  applicants  to  submit  a  well  written 
proposal that meets all application standards, and the capacity of applicants to deliver the 
results  they  describe  in  their  proposals.  Broadly  speaking,  communication  activities 
promoting  the  justice  programme  are  effective.  However,  some  beneficiaries  suggested 
that a swifter reaction to their doubts and questions during the application period would 
immensely facilitate the process.  

Key points in a nutshell  

  There  is  a  need  for  structured  data  collection  tools  to  understand  how  target  groups 

perceive the quality of projects funded by the justice programme. 

  The  application  process  is  still  challenging  as  the  indication  of  project  outputs  in 
proposals remains burdensome for some applicants, and is a factor potentially excluding 
relevant organisations from participating 

The  justice  programme  also  made  signification  progress  in  terms  of  its  efficiency 
compared to its predecessor programmes. The stakeholders consulted confirmed the cost-
efficiency of the programme. Although they underlined that the application procedure is 
very  demanding  and  costly,  they  equally  stressed  that  the  benefits  provided  by 
participating in the programme is even more rewarding. This finding is in line with the 
positive feedback on streamlining the application process.  

However, there are also some aspects on which more critical views were expressed. For 
example,  applicants  repeatedly  stated  that  despite  visible  improvements  in  the 
management cycle, the timeframe for submitting project proposals is still too ambitious 
and  the  time  between  proposal  submission  and  selection  is  too  long.  Another  area  of 
potential  improvement  in  the  duration  of  the  application  period  is  the  provision  of 
guidelines and information sessions to help applicants handle the administrative burden 
involved in making an application. This could also help encourage organisations, put off 
from applying due to the requirements to participate in the justice programme. Last but 
not  least,  despite  the  increase  in  flexibility  of  the  overall  budget  of  the  justice 
programme,  underspending  at  project  level  remains  an  issue.  The  reason  for  the 
continued  underspending  of  resources  is  mainly  linked  to  beneficiaries  over  budgeting 
projects. Thus, there is a need for additional guidance on budgeting.  

38 

 
 
Key points in a nutshell 

  Applying  and  receiving  funding  under  the  justice  programme  is  very  demanding  and 
costly. However, the benefits of participating in the programme outweigh the associated 
costs.  

  Underspending is still an issue for beneficiaries, who struggle with accurate budgeting.  

The  analysis  showed  that  the  programme  and  its  projects  achieve  a  very  high  level  of 
relevance. It concludes that project objectives and activities are relevant to the needs 
of  targeted  stakeholders.  In  turn,  the  programme’s  target  groups  are  relevant  for  the 
achievement of the programmes’ specific objectives. Here the evaluation found few areas 
for  improvement.  These  include  engaging  in  a  more  systematic  and  open  manner  with 
stakeholders  to  better  align  the  activities  and  specific  objectives  outlined  in  the  annual 
work programmes with the needs of the programme’s target groups. There is also a need 
to  step  up  the  current  and  successful  work  to  expand  the  programme’s  target  group  to 
include more stakeholders relevant to the justice programme. In particular, it could more 
actively target civil society organisations working on victims’ rights. 

Key points in a nutshell 

  Despite a high degree of relevance of the projects funded, regular needs assessments and 
more  frequent  exchanges  with  stakeholders  are  needed  to  even  better  define  the  annual 
priorities and to align them with the needs of target groups. 

  On  victims’  rights,  civil  society  organisations  could  be  more  actively  involved  in  the 

programme. 

The  current  analysis  shows  that  the  justice  programme  has  a  good  level  of  coherence 
and  complementarity  with  other  EU  instruments,  programmes  and  actions.  It 
underscores  that,  especially  on  judicial  training,  the  merge  of  the  predecessor 
programmes  into  the  justice  programme  increased  coherence  with  other  EU  initiatives. 
The  same  applies  to  several  training  objectives,  which  has  reduced  the  likelihood  of 
duplications.  This  result  confirms  the  findings  outlined  in  the  interim  evaluation. 
However, exchange of expertise and best practices between beneficiaries could be more 
actively  promoted  to  increase  synergies  between  funded  projects.  Concerning  potential 
overlaps  between  projects  under  the  justice  programme,  the  broad  scope  of  the 
programme  does  result  in  some  overlaps,  by  design.  Although  the  justice  programme’s 
structure  provides  enough  flexibility  to  minimise  the  risk  of  duplication,  potential 
overlaps should be closely monitored to maintain the high level of complementarity.  

Key points in a nutshell 

  By design, the justice programme minimises the risk of duplications in funding. However, 

the design also results in overlaps at project level. 

  There  is  a  need  to  monitor  projects  more  closely  to  forestall  overlaps  between  projects 

under the justice programme. 

In terms of the EU added value of the justice programme, the analysis fully confirms the 

39 

 
 
 
 
high added value of funded projects. In particular, high EU added value is achieved by 
promoting  transnational  projects  to  tackle cross-border issues  as  well as the increase of 
coherence and uniformity in the application of EU law. Key  activities that generate the 
high EU added value of the justice programme are trainings and information exchanges 
in funded projects. High EU added value was also confirmed by programme committee 
members  of  the  justice  programme,  who  asserted  the  programme  is  an  important 
instrument  as  national  initiative  are  not  able  to  achieve  the  same  results.  Due  to  the 
substantial EU added value of the justice programme, the analysis concluded that, at the 
current stage of programme implementation, there are no specific area for improvement. 

Key points in a nutshell 

  The inherent “European” and transnational dimension of the Programme is at the core 
of  the  EU  added  value  of  the  justice  programme,  especially  with  regard  to  judicial 
training 

In  terms  of  the  equity  of  the  justice  programme,  the  current  evaluation  reiterates  the 
finding from the interim evaluation that resources and support provided through the 
justice programme are not distributed in a balanced manner across different target 
groups, EU Member States and types of beneficiaries. At the macro level, there is a 
discrepancy  between  high  application  rates  from  countries  like  Belgium,  Italy,  Spain, 
Germany,  the  Netherlands  and  France  and  comparatively  low  application  rates  from 
countries  like  Latvia,  Poland,  Croatia,  Slovakia  and  Cyprus.  These  result  in  lower 
participation  rates  from  these  countries  in  funding  under  the  justice  programme.  In 
addition,  the  analysis  identified  the  need  for  capacity  building  in  underrepresented 
countries,  which  renders  their  participation  in  the  justice  programme  difficult.  At  an 
intermediate level, the analysis revealed that overall the programme’s target groups have 
become  more  diverse  and  include  a  wider  variety  of  justice  professionals.  But  civil 
society organisations should be more involved in projects funded in the field of victims’ 
rights. This finding dovetails with the need for capacity building in the above countries. 
Civil  society  organisations  from  underrepresented  countries  might  need  additional 
support to build the necessary capacity to apply to the justice programme.  

At  the  micro  level,  more  robust  and  systematic  monitoring  data  would  allow  to  obtain 
deeper  insights  into  the  target  group’s  composition.  This  could  be  facilitated  by 
collecting data that are disaggregated inter alia by sex, age and disability status. In turn, 
the insights gathered at project level could feed into the programming at macro level to 
increase  the  funding’s  impact  on  an  even  more  diverse  group  of  stakeholders.  This 
finding is also related to the promotion of gender equality, children’s rights and the rights 
of people with  a disability  as part of the  funded  projects.  More robust data would  help 
identify areas, in which the rights of women, men, girls, boys and people with a disability 
are not as well covered as they could be. 

In terms of the cross-sectional themes of gender mainstreaming, promoting the rights 
of  the  child  and  promoting  the  rights  of  people  with  disabilities,  the  analysis 
concluded  that  applicants  struggle  to  understand  all  these  concepts.  As  a  result,  some 
projects  funded  through  the  justice  programme  do  not  live  up  to  their  potential  in 
promoting  gender  equality,  the  rights  of  the  child  and  the  rights  of  people  with 
disabilities.  Although  all  cross-sectional  themes  require  further  attention,  there  are 
differences  in  the degree to  which the  programme  covers these themes at  project  level. 

40 

 
 
 
Gender equality and children’s rights receive dedicated attention in funded projects, but 
the rights of people with disabilities are not as prominently featured within projects. As 
interviewees’ responses suggest, this is very likely to be due to the finding that projects 
run under the justice programme are not always suitable to address the rights of people 
with disabilities. 

Key points in a nutshell 

  There  is  a  need  for  further  action  to  better  integrate  cross-cutting  topics  such  as  gender 
mainstreaming, the rights of the child and the rights of people with disabilities in projects.  
  An  improved  approach  on  tracking  project  data  is  needed  to  increase  the  impact  of  the 

justice programme on a more diverse range of target groups. 

  The geographical allocation of funds across Member States needs to be improved. 

Concerning  the  scope  for  simplification,  the  European  Commission  made  significant 
changes to review, speed up and simplify the process over the 2014-2020 programming 
period.  Although  the  current  management  model  is  considered  to  be  suitable,  and 
project  beneficiaries  positively  welcomed  the  changes,  they  also  identified  issues  and 
room for further improvement.  

According to the analysis, the scope is for simplification to meet the need for a simpler 
application  and  results-based  reporting  procedures,  for  improved  monitoring  and  for 
management  tools  to  collect  more  systematically  information  on  achieved  outputs, 
results and targets, as well as help for organisations with difficulties in working only in 
English.  However,  the  perceived  burden  and  complexity  varies  among  project 
beneficiaries.  Organisations  with  more  capacity  and  greater  experience  in  applying, 
coordinating, and participating in projects, find it much easier and smoother than smaller 
applicants. 
organisations 
Last,  despite  improvements  to  the  monitoring  and  management  tools,  there  is  still 
room for improvement, especially in terms of adapting the application portal to the types 
of  beneficiaries  supported  by  the  justice  programme  or  supporting  first-time  applicants 
and beneficiaries to use the tool without incurring delays in their activities.  

first-time 

and 

Key points in a nutshell 

  Additional guidance is needed to help more applicants follow the application procedures. 
  The  reporting  procedures  should  be  simplified  by  bringing  in  standardised  cost  options 

and results-based management. 

41 

 
 
Annex 1: Procedural information 

1.  LEAD DG, DeCIDE PLANNING/CWP REFERENCES 

Lead Directorate-General: Directorate-General for Justice and Consumers (DG JUST) 

Decide planning reference: PLAN/2020/9645 - JUST - Report to the EP and the Council 
assessing the 2014-2020 justice programme  

ORGANISATION AND TIMING 

The evaluation was carried out between March 2021 and September 2021. 

An  external  contractor,  the  Fondazione  Giacomo  Brodolini  Srl  SB  and  Ernst  &  Young 
Consulting  BV/SRL,  was  commissioned  under  the  framework  contract  for  evaluation, 
impact assessment and  related services (JUST/2020/PR/03/0001-01) to carry out an ex-
post  evaluation  of  the  justice  programme  (2014-2020).  However,  the  current  initiative 
primarily  aims  to  meet  the  Commission’s  legal  reporting  obligations  under  Regulation 
(EU) No 1382/2013. Since, it is premature to analyse the long-term effects of the justice 
programme  in  2021  (too  many  projects,  around  30%,  run  throughout  2021),  this  report 
will  focus  on  providing  a  first  assessment  based  on  data  available  by  2021  of  the 
programme’s achievements. It will take inspiration in the better regulation methodology 
for  evaluations  applied  for  the  programme’s  mid-term  report.  The  assessment  will, 
however, not be fully aligned with better regulation guidelines. On this basis, no roadmap 
and public consultation would be required. A study will underpin the assessment and the 
process  will  be  steered  by  a  steering  group  (formed  by  DG  units  implementing  the 
programme with SG associated), but no formal inter-service steering group would be set 
up. The evaluation of the long-term effects of the 2014-2020 justice programme will be 
done  alongside  the  mid-term  evaluation  of  the  succeeding  financial  programme  2021-
2027. This be will fully follow the Better Regulation Guidelines. 

Against  this  backdrop,  no  Inter-service  Steering  Group  was  established.  However, 
several DG JUST policy units (JUST 01, JUST 03, JUST A1, JUST B1, JUST B2, JUST 
C3, JUST D2) as well as colleagues from DG Home, DG  COMP  were involved in  the 
work and contributed to the evaluation. 

Timeframe for the evaluation: 

November 2020 

March 2021 

Appointment of colleagues in policy units 
to follow the evaluation and preparation of 
the terms of reference for the request for 
services 
Contract with external evaluators starts 

42 

 
 
 
April 2021 
June 2021 

September 2021 
October 2021 

Kick-off meeting of the evaluation 
Inception report 
Interim report and the 2nd meeting 

Final report and the 3rd meeting 
Preparation  of  the  report  to  the  European 
Parliament  and  the  Council  and  of  the 
accompanying Staff Working Document 

2.  EXCEPTIONS TO THE BETTER REGULATION GUIDELINES 

As this ‘first-stage’ report provides only an initial assessment of the justice programme’s 
achievements,  it  is  not  fully  aligned  with  the  better  regulation  guidelines.  In  particular, 
while taking guidance from the better regulation process for the evaluation methodology 
to  be  applied,  no  roadmap,  no  public  consultation  and  no  formal  inter-service  steering 
group were created at this stage. The ‘better regulation’ requirements will be followed in 
full for the ‘second-stage’ report as the benefits would be more pertinent when preparing 
the second part of the ex post evaluation. 

3.  CONSULTATION OF THE RSB (IF APPLICABLE) 

N/A 

4.  EVIDENCE, SOURCES AND QUALITY 

The  evaluation  was  based  on  evidence  from  different  sources.  The  complete  set  of 
documents that were consulted for this evaluation is listed below: 

CATEGORY 

AUTHOR 

GENERAL DOCUMENTS 

GENERAL DOCUMENTS 

- 

- 

GENERAL DOCUMENTS 

 Prof. Julia 
Laffranque, 
Judge, European 
Court of Human 
Rights 

YEA
R 

2012 

2012 

2014 

GENERAL DOCUMENTS 

CEPEJ 

2016 

DOCUMENT 

the 

the  Treaty  on 

Consolidated  version  of  the  Treaty  on  European 
Union (TEU) 
Consolidated  version  of 
Functioning of the European Union (TFEU) 
Article  6  of  the  European  Convention  On  Human 
Rights  (Convention)  and  Article  47  Of  The 
European  Union  (EU)  Charter  On  Fundamental 
Rights  (Charter):  Mutual  Relation,  Scope,  And 
Interpretation 
European Judicial Systems:  Efficiency and quality 
of justice 

GENERAL DOCUMENTS 

GENERAL DOCUMENTS 

GENERAL DOCUMENTS 

GENERAL DOCUMENTS 

GENERAL DOCUMENTS 

GENERAL DOCUMENTS 

European 
Commission 
European 
Commission 
European 
Commission 
European 
Commission 
European 
Commission 
European 

2020  European Judicial Training 2020 

2019  European Judicial Training 2019 

2018  European Judicial Training 2018 

2017  European Judicial Training 2017 

2016  European Judicial Training 2016 

2016  European Judicial Training 2015 

43 

 
 
 
 
GENERAL DOCUMENTS 

GENERAL DOCUMENTS 

GENERAL DOCUMENTS 

GENERAL DOCUMENTS 

GENERAL DOCUMENTS 

GENERAL DOCUMENTS 

GENERAL DOCUMENTS 

GENERAL DOCUMENTS 

GENERAL DOCUMENTS 

POLICY DOCUMENTS 

POLICY DOCUMENTS 

POLICY DOCUMENTS 

POLICY DOCUMENTS 

POLICY DOCUMENTS 

POLICY DOCUMENTS 

POLICY DOCUMENTS 

POLICY DOCUMENTS 

POLICY DOCUMENTS  

POLICY DOCUMENTS  

Commission 
European 
Commission 
European 
Commission 
EU  Agency  for 
Fundamental 
Rights  
EU  Agency  for 
Fundamental 
Rights  
EU Agency for 
Fundamental 
Rights  
European 
institute 
for 
Gender Equality  
European 
institute for 
Gender Equality  

European 
Commission 

European 
Commission 

European 
Commission  and 
the Parliament 
European 
Commission  
United Nations 
European 
Commission 

European 
Commission 

European 
Commission 

European 
Commission 

European 
Commission 

European 
Commission 

European 
Commission 

POLICY DOCUMENTS  

European 
Commission  

2014 

POLICY DOCUMENTS  

European 
Commission  

2014 

44 

2016  European Judicial Training 2014 

2011  European Union Judicial Training Strategy 

2020 

Fundamental Rights Report 2020 

2020 

Ten  years  on:  Unlocking 
Potential 

the  Charter’s  Full 

2017 

Fundamental Rights Report 2017 

2020 

Gender  Equality  Index  2020:  Key  Finding  for  the 
European Union - Report 

2017 

2019 

2020 

Gender  Equality  Index  2017:  Measuring  gender 
equality  in  the  European  Union  2005-2015  – 
Report 
COM(2019)  400  -  Draft  Budget  of  the  European 
Union  for  the  financial  year  2020.  Working 
document  Part  I  –  Programme  Statements  of 
operational expenditures 
COM(2020) 300  - - Draft Budget of the  European 
Union  for  the  financial  year  2021.  Working 
document  Part  I  –  Programme  Statements  of 
operational expenditures 

2014 

The  EU  Justice  Agenda  for  2020  -  Strengthening 
Trust, Mobility and Growth within the Union 

2014 

10 Priorities for 2015-19 

2015  Agenda 2030 for Sustainable Development  

2016  DG Justice Strategic Plan 2016-2020 

2017
-
2019 
2014
-
2016 
2014
-
2019 
2014
-
2018 

2011 

2011 

DG Justice Annual Activity Reports  

DG Justice Annual Activity Reports  

DG Justice Management Plans 

DG Justice Programme Statements 

- 

accompanying 

Commission  staff  working  paper  – 
Impact 
Assessment 
the  document 
the  European 
Proposal  for  a  Regulation  of 
Parliament  and  of  the  Council  establishing  for  the 
Period  to  2020  the  Justice  Programme.  Impact 
Assessment’. SEC(2011) 1364 final 
Commission  Staff  Working  Paper  on  Consumer 
Empowerment in the EU, SEC(2011)469 final 
Communication  from 
the 
to 
European  Parliament,  the  Council,  the  European 
Economic 
the 
Committee of the Regions, the EU Justice Agenda 
for  2020  -  Strengthening  Trust,  Mobility  and 
Growth  within  the  Union,  Strasbourg,  11.3.2014, 
COM(2014) 144 final 
Communication  From  the  Commission  to  the 
European  Parliament,  the  Council,  the  European 
Economic  And  Social  Committee  And 
the 
Committee  of  the  Regions  the  EU  Justice  Agenda 

social  Committee 

the  Commission 

and 

and 

 
POLICY DOCUMENTS  

European 
Commission 

2016 

for  2020  -  Strengthening  Trust,  Mobility  and 
Growth  within  the  Union,  Strasbourg,  11.3.2014, 
COM(2014) 144 final 
Communication  from 
the 
European  Parliament,  the  Council,  the  European 
Central  Bank,  the  European  Economic  and  Social 
Committee  and  the  Committee  of  the  Regions  — 
The 2016 EU Justice Scoreboard, 

the  Commission 

to 

POLICY DOCUMENTS 

POLICY DOCUMENTS 

POLICY DOCUMENTS 

POLICY DOCUMENTS 

POLICY DOCUMENTS  

POLICY DOCUMENTS  

POLICY DOCUMENTS 

POLICY DOCUMENTS  

POLICY DOCUMENTS 

POLICY DOCUMENTS 

POLICY DOCUMENTS 

POLICY DOCUMENTS  

European 
Commission 
European 
Commission 
European 
Commission 
European 
Commission 
European 
Commission 
European 
Commission 
European 
Commission 
European 
Commission 
European 
Commission 
European 
Commission 
European 
Commission 
 European 
for 
institute 
Gender Equality 

POLICY DOCUMENTS  

European 
Commission 

2017 

POLICY DOCUMENTS  

European 
Commission 

2017 

POLICY DOCUMENTS  

European 
Commission  

2017 

2020  The 2020 EU Justice Scoreboard 

2019  The 2019 EU Justice Scoreboard 

2018  The 2018 EU Justice Scoreboard 

2017  The 2017 EU Justice Scoreboard 

2016  The 2015 EU Justice Scoreboard 

2016  The 2014 EU Justice Scoreboard 

2020  EU Citizenship Report 2020 

2016 

Effective  Roma 
Member States 

integration  measures 

in 

the 

2016 

Strategic Engagement for Gender Equality 

2019  Countering Racism and Xenophobia in the EU 

2010  European Disability Strategy 2010-2020 

2017 

staff 

from 

working 

the  Commission 

the  Communication 

European  institute  for  Gender  Equality,  Gender 
Equality Index 2017: Measuring gender equality in 
the European Union 
Commission 
document: 
Comprehensive Assessment of EU Security Policy, 
accompanying 
the 
Commission  to  the  European  Parliament,  the 
European Council and the Council: Ninth progress 
report  -towards  an  effective  and  genuine  Security 
Union, {COM(2017) 407 final}. 
the 
Communication  from 
European  Parliament,  the  Council,  the  European 
Central  Bank,  the  European  Economic  and  Social 
Committee  and  the  Committee  of  the  Regions  — 
The 2017 EU Justice Scoreboard, COM(2017) 167 
final 
Commission  staff  working  document  on 
the 
Application  of  the  EU  Charter  of  Fundamental 
Rights  in  2016,  accompanying  Communication 
from the Commission to the European Parliament, 
the  Council,  the  European  Economic  and  Social 
Committee  and  the  Committee  of  the  Regions  on 
the  2016  Report  on  the  Application  of  the  EU 
Charter  of  Fundamental  Rights,  {COM(2017)  239 
final}, 18 May 2017 

to 

POLICY DOCUMENTS  

POLICY DOCUMENTS 

POLICY DOCUMENTS 

POLICY DOCUMENTS 

European  Union 
Agency 
for 
Fundamental 
Rights  
European 
Council 
European 
Council 

European 
Commission 

2017 

FRA, Fundamental Rights Report 2017  

European  Council  (2019)  A  new  strategic  agenda 
for the EU 2019-2024 
European  Council,  Draft  Strategic  Guidelines  in 
the field of Justice and Home Affairs 
European  Commission  –  Directorate-General  for 
Justice  and  Consumers  (2020),  European  Judicial 
Training  2020,  Luxembourg,  Publications  Office 
of the European Union 

2019 

- 

2020 

45 

 
POLICY DOCUMENTS 

POLICY DOCUMENTS 

PROJECT DOCUMENTS 

PROJECT DOCUMENTS 

PROJECT DOCUMENTS 

PROJECT DOCUMENTS 

PROJECT DOCUMENTS 

PROJECT DOCUMENTS 

PROJECT DOCUMENTS 

REGULATIONS 

REGULATIONS 

REGULATIONS 

European 
Commission 

European 
Commission 

European 
Commission 

European 
Commission 
European 
Commission 

European 
Commission 

European 
Commission 

European 
Commission 

European 
Commission 

European 
Parliament and 
the Council 

European 
Parliament  and 
the Council 

European 
Parliament  and 
the Council  

2020 

2020 

2014 

European  Commission,  A  Counter-Terrorism 
Agenda  for  the  EU:  Anticipate,  Prevent,  Protect, 
Respond – COM(2020) 795 final 
European  Commission,  EU  Agenda  and  Action 
Plan on Drugs 2021-2025 (COM(2020) 606 final) 
Annex  to  the  Commission  Implementing  Decision 
concerning  the  adoption  of  the  work  programme 
for 2014 and the financing for the implementation 
of the Justice Programme C(2014) 2556 final 

2015  Annual Monitoring Report Justice 2014 

2015  Annual Monitoring Report Justice 2014 

Annual Monitoring Reports REC  

Annual work programmes REC 

impact 
Commission  staff  working  document 
the  proposal  for  a 
assessment  accompanying 
the  rights  and  values 
Regulation  establishing 
programme  proposal  for  a  Regulation  establishing 
the  justice  programme  Proposal  for  a  Regulation 
establishing 
the  creative  Europe  programme, 
Brussels, 30.5.2018, SWD(2018) 290 final 
European Commission  staff  working document on 
the  Evaluation  of  the  2011  European  judicial 
training strategy. SWD(2019) 381 
Regulation  (EU)  1381/2013  of 
the  European 
Parliament  and  of  the  Council  of  17  December 
and 
2013 
Citizenship  Programme  for  the  period  2014  to 
2020 
Regulation  (EU)  No  1382/2013  of  the  European 
Parliament  and  of  the  Council  of  17  December 
2013  establishing  a  Justice  Programme  for  the 
period 2014 to 2020 
Regulation 
the  European 
Parliament  and  of  the  Council  of  28  April  2021 
establishing  the  Justice  Programme  and  repealing 
Regulation (EU) No 1382/2013 

a  Rights,  Equality 

(EU)  2021/693  of 

establishing 

2014
-
2016 
2014
-
2017 

2018 

2019 

2013 

2013 

2021 

46 

 
 
 
 
Annex 2: Stakeholder consultation 

Consultations with the main stakeholders of the justice programme were conducted over 
several stages: 

two web-based survey targeting the following stakeholder groups:  

 
  project applicants (including unsuccessful applicants ); 
  project beneficiaries;  
  EU26 Member State programme committee members; 
  other relevant stakeholders contacted directly by the Commission; 

The first survey targeting programme committee members was launched though an online tool 
(Qualtrics) on June 23rd and it ran until July 14 2021. The survey was kept open four weeks 
longer than originally planned, to provide more time to answer the questions and increase the 
response rate. It received 21 responses. 

The second survey ran from 28 June to 9th July and aimed to collect information to feed 
Case Study 4. The survey was sent to 200 project beneficiaries, and elicited 11 responses. 

The three focus groups to test and validate preliminary findings and conclusions were conducted 
between the 1 and 16 July 2021. 

  33  interviews  with  5  Commission  officials  and  28  selected  projects  beneficiaries  and 
representatives of target groups were  conducted between 1 July 2021 and 16July 2021. 
The  interviews  had  a  twofold  objective:  1.  Collecting  additional  information  (from  the 
survey and the desk research) to feed into the analysis alongside the evaluation criteria, 
filling gaps (especially in terms of quantification of costs and benefits) and going more 
in-depth on specific aspects, such as identification of best practices; 2. Gathering insight 
and input to draw conclusions and recommendations on how to improve the design and 
functioning of the programme.  

The interviews took the form of semi-structured phone interviews. The questions were tailored to 
the different categories of target stakeholders, and their different contribution to the evaluation 
questions. 

A  broad  public  consultation  undertaken  by  the  Commission  in  the  area  of  "values  and 
mobility" was available online in 23 official EU languages for a mandatory period between 10 
January 2018 and 9 March 2018. The purpose of this consultation was to collect the views of all 
interested parties on how to make the most of every euro of the EU budget. Consultations have 
taken place in the context of evaluations of existing EU financial programmes covering several 
policy  areas,  including  the  justice  programme.  In  total,  the  public  consultation  received  1839 
replies  from  all  over  Europe.  The  respondents  had  experience  with  the  following  EU 
programmes:  1.  Europe  for  Citizens  Programme  and  /or  2.  Rights,  Equality  and  Citizenship 
Programme and /or 3. Creative Europe Programme and /or 4. Justice Programme. The results of 
this  public  consultation  have  shown  that,  according  to  the  majority  of  respondents,  "promote 
European  identity  and  common  values",  as  well  as  "promote  rights  and  equality",  are 
important common policy challenges to be addressed in each of the four programmes. "Support 

47 

 
 
 
active  citizenship,  democratic  participation  in  society  and  the  rule  of  law"  and  "promote 
social  inclusion  and  fairness"  appear  to  be  important  challenges  to  be  addressed  in  the 
concerned  programmes  as  well.  Instead,  "support  innovation",  "foster  European  cultural 
diversity  and  cultural  heritage",  "promote  European  identity  and  common  values"  are 
considered  as  policies  that  fully  or  fairly  well  address  the  challenges  by  half  or  more  of  the 
respondents.  Around  80%  the  respondents  agree  that  these  programmes  add  value  to  a  large 
extent or to a fairly good extent to what Member States could achieve at national, regional and/or 
local  levels.  The  main  obstacles  identified  by  the  respondents,  that  could  prevent  the  current 
programmes/funds from achieving their objectives, are very similar regardless of the programme 
concerned:  "lack  of  budget  of  the  programmes  to  satisfy  demand",  "insufficient  support 
provided  to  small-scale  stakeholders"  and  "lack  of  support  to  first-time  applicants"  are 
identified  as  the  main  three  obstacles.  Finally,  the  respondents  agree  that  "the  use  of  more 
simplified  application  forms",  "facilitating  structured  network  and  partnerships", 
"facilitating  funding  for  actions  cutting  across  the  sectors  of  action",  as  well  as  "better 
coordination between different programmes/funds", are the main steps to be taken to simplify 
and reduce the administrative burdens for beneficiaries. 

48 

 
  
 
 
 
Annex 3: Methods and analytical models 

The following methodological tools were used:   

1. Data collection; 

Desk research of relevant documents (at programme, policy and project level); 

The following documents were reviewed: 

For Task one: 

 

files with extracts of key monitoring data, 

  monitoring data derived from the interim evaluation, 

  grant  agreements  and  project  proposals  (covering  projects  funded  from  2016 

onwards) 

 

technical  periodic  reports  (final  reports),  including  monitoring  annexes  and 
financial reports (only for finalised projects funded from 2016 onwards). 

For task two, additional project documents such as: 

  amendments  of  grant  agreements  (notarised):  providing  information  on  possible 

changes intervened after the signature of the agreement, 

  proposal evaluation forms, in particular evaluation summary reports, which provide 
information  on  the  process  of  evaluating  the  project  proposal  (scores,  ranking, 
qualitative assessment), 

 

technical report and annexes 

  evaluation  of  the  final  technical  reports  (by  policy  officers)  providing  a  final 

assessment of the project , 

final financial statements 

list of final deliverables 

list of milestones. 

 

 

 

The table below quantifies the documents reviewed. 

Table 16 Distribution and coverage of project documents reviewed 

Type of document 

Coverage/stakeholders 

Monitoring files from the 
Interim Evaluation 
Monitoring files by DG JUST 
Grant agreements/proposals 

Number of 
documents. 
1 

Steps involved 

Task 1 
Monitoring 
Task 1 
Monitoring 
Task 1 

Projects funded in 2014-2016 

Projects funded from 2016 onwards 
Projects funded from 2016 onwards 

7 
336 

49 

 
 
 
 
Technical reports 

Finalised projects, funded from 2016 
onwards 

171 

Additional project docs 
(e.g. amendments and evaluation 
summary) 

Sampled projects 

Action grants and 
operating grants: 245 
Procurements: 23 

Monitoring 
Task 2 In-
depth analysis 
Task 1 
Monitoring 
Task 2 In-
depth analysis 
Task 2 In-
depth analysis 
Task 3 Case 
studies 

The field research was based on two web-based surveys with stakeholders, i.e. (unsuccessful) 
project applicants, project beneficiaries, programme committee members, Commission staff and 
additional stakeholders, 33 interviews with selected project beneficiaries and representatives of 
target groups and with Commission officials. Three focus groups meetings were held to test 
and validate preliminary findings and conclusions. 

Eight case studies of projects were connected to the specific objectives, see more detailed 
overview below. 

Analysis of the results of a broad public consultation undertaken by the Commission between 
10 January and 8 March 2018 in the framework of the proposal for the post 2020 Multiannual 
Financial Framework in the area of European values and mobility.  

2. Data analysis 

Quantitative data analysis: in addition to monitoring and survey data, descriptive statistics 
were used to analyse effectiveness, efficiency and EU added value of the programme; self-
reported counterfactual analysis was used to analyse effectiveness and added value; an 
exploratory data analysis was used to analyse effectiveness and EU added value; a cost-benefit 
analysis to analyse efficiency and social network analysis to analyse effectiveness and equity. 

Qualitative data analysis: a text mining methodology was used. 

Based on this methodology, output indicators ( monitoring data at project level) and results 
indicators (based on Eurobarometer and other sources) were used. The analysis of the 
performance at the project level was mainly based on the answers to the survey and interviews.  

The following paragraphs provide more detailed information on data collection and analysis. 

Task 1 - Extraction, categorisation, and aggregation of monitoring data 

For this task, the following steps were taken.  

Creation of the project list for action grants and operating grants  

The information retrieved from monitoring files was cleaned, systematised, and transferred to a 
dataset specifically created for this study. Since information available from monitoring files and 
project documents provided by DG JUST mostly covered 2016-2020 projects and applications, 
resulting in the dataset still having gaps and lacking information on older projects, the dataset 
was integrated with information from the interim evaluation. Similarly, information on applicants 
and participants covered mostly 2016 onwards. Information was completed from different 
sources: e.g. information on the type of organisation for applicants – which was missing – was 

50 

 
 
extracted from the list of participants and linked to the applicants that also participated in the 
granted project.  

Additional information was obtained from: 

• 
New extractions on projects and applications provided by DG JUST – both updating the 
list and adding new types of information from proposals (e.g. on the names, country and type of 
beneficiaries) and reports (e.g. on dissemination activities implemented). The new data were 
either added to the existing dataset or used to update it.  

Annex 3 to the technical reports, which were first extracted automatically and then 

• 
manually cleaned and systematised to reduce noise and loss of data.  

The lack of detail is also reported for grants awarded under calls from other DGs (COMP, EMPL, 
HOME), which provide far less information than grants managed by DG JUST.  

Creation of the project list for procurement contracts 

In parallel to grants, the research team compiled a list of procurement contracts. The list was 
drawn up using information provided by DG JUST, which was combined to extract only the 
unique records and the corresponding specific objective. 

The list was revised by DG JUST, cleaning the list of contracts under the justice programme, and 
linking them to the type of activities.  

Extraction of key qualitative texts/unstructured information: 

In order to extract data on qualitative descriptors, the research team adopted a mix of qualitative 
data analysis approaches and tools (word search, categorisation, text mining and semantic tools). 
The starting point for these approaches entailed first identifying the document sources and the 
exact text fields within these sources to be screened. Next, the team manually extracted 
(copy/pasted) the relevant text fields into an Excel sheet on which the various qualitative analysis 
tools could then be implemented. Based on the qualitative information extracted, the team carried 
out the following analyses: 

Text mining/semantic tool 

The team analysed the documents using Named Entity Recognition (NER) tools. NER is a 
subtask of information extraction that seeks to locate named entities mentioned in unstructured 
text and classify them into pre-defined categories such as person names, organisations, locations, 
medical codes, time expressions, quantities, monetary values percentages. For the project 
proposal analysis, these entities are words that are relevant to the content of the call. The research 
team focused on: target groups, dissemination activities and needs.  

Screening of qualitative information and categorisation 

For a number of descriptors, the research team adopted a manual categorisation approach, by 
screening specific text fields and categorising the information for each project based on a drop-
down list of categories.  

Mixed methods (word search + categorisation): 

51 

 
For a number of descriptors, the team took a mixed approach to qualitative analysis. First, the 
relevant information (when available) was identified through a simple word search, it was then 
manually extracted. This approach was followed for the following descriptors: 

Number of interactions (likes, shares, retweets, comments) generated by online 

• 
awareness raising, information and dissemination activities 

Number of fans and followers acquired through online awareness-raising, information 

• 
and dissemination activities 

Creation of a list of indicators and descriptors: 

The monitoring data collected via the previous steps were then categorised and aggregated to 
support a narrative on the key topics, as follows.  

• 

• 

• 

• 

Indicators and descriptors related to applicants/participants  

Indicators and descriptors related to the project budget allocation and distribution  

Indicators and descriptors related to the type of activity funded  

Indicators and descriptors related to target groups/stakeholders  

Indicators and descriptors related to other features (such as geographical coverage, 

• 
project duration) 

Indicators and descriptors related to the performance of the programme/the evaluation 

• 
criteria 

Task 2 - Additional collection and systematisation of information  

Following the collection, extraction, and systematisation of project-related data and documents 
received from DG JUST as described in the previous section, the research team collected 
additional data as additional input and systematised all information collected to feed into the 
response to the evaluation questions and the drafting of this report. 

This included additional desk research on documents mentioned in Section 1.2.1., and both desk 
and field research on a sample of projects.  

A total of 23 projects – both action (19) and operating grants (4) – was sampled with the aim of 
carrying on a qualitative analysis based on comments and insights from key stakeholders on the 
implementation of the justice programme. The projects were sampled bearing in mind 
geographical distribution, the specific objective, and the financial amount.  

Project beneficiaries were selected using a non-statistical ‘stratified’ sampling method: The 
sample was then integrated into the interim report, discussed, and approved by DG JUST. The 
overall aim of this sample of projects was to be representative, to the largest extent possible, of 
the overall project ‘population’  with a breakdown by specific objective and Member State. 

Sampling of action grants: 

• 

• 

proportional to the amount and form of funding per country/specific objective; 

comprehensive, i.e. they include projects with the following characteristics: 

52 

 
• 

• 

• 

they cover all specific objectives; 

they cover all forms of support; 

they cover different years of programme implementation. 

These criteria ensured that the sample of projects identified was representative of how the 
programme was implemented, in terms of  its specific objectives. 

In order for this distribution to be representative not only in terms of specific objectives and 
forms of support, but also geographically, the team also paid attention to the nationality of the 
beneficiary. Thus, the sample of projects ensured that Member States representation (in terms of 
beneficiary nationalities) reflected the overall geographical distribution of projects (in terms of 
participation in the calls and the success rate for action grants and operating grants awarded).  

Sampling of operating grants 

The sample of operating grants was selected following criteria such as the overall financial 
allocation to a given organisation/network under each specific objective, the relevance of the 
organisation to the objectives of the programmes and the specific objective. The final selection 
was made on a case-by-case basis, in agreement with DG JUST. 

Sampling of procurement contracts: 

The selection of the procurement contracts was based on the relevance of the contracted service 
to the objectives of the programmes and to the information needs of the European Commission. 
When selecting the sample contracts, the team sought to cover different types of procurement 
activities.  

Task 3 Drafting case studies and answering the evaluation questions 

Case studies 

The evaluation envisaged drafting eight case studies. The main purpose of the case studies is to 
provide solid, triangulated evidence, to feed into the formulation of replies to evaluation 
questions as well as to suggest improvements, with a focus on the specific issues mentioned in 
the tender specifications. The case studies include a methodological approach, an analysis of the 
relevance of the objective to the current needs (contextual analysis), descriptive statistics on the 
type of projects, actions and beneficiaries related to the objective, and the analysis of the results 
of all the activities funded and procured by the Commission and of a sample of action and 
operating grants. 

In addition to data and information collected under task 1 and 2, the case studies envisaged 
additional field and desk research and in particular: 

• 

• 

further in-depth analysis of sampled projects’ documents  

an online survey to all sampled projects (for Case Study 4) 

three focus groups, including beneficiaries from projects that were not included in the 

• 
sample (For Case study 1, 3 and 7, 8). 

53 

 
The table below provides an overview of the focus of the eight case studies and shows the 
evaluation questions they refer to. 

Overview of the eight case studies 

Case study 

Case study 1 

Recurring  beneficiaries 
continuous projects 

and 

Case study 2 

the  quality  of  selection/ 
Is 
evaluation  of  project  proposals 
robust? 
70-points 
the 
threshold pertinent? 

Is 

Description 

Relevant EQs 

analysis 

The 
and 
disadvantages  of  financing  continuations  of  projects 
or projects implemented by recurring beneficiaries.  

investigates 

benefits 

the 

EQ  2.  How  fit  for  purpose  were 
the  actions  funded  under 
the 
programme  to  the  objectives  of 
the programme?  

The objective of the analysis is to verify whether the 
selection  and  evaluation  processes  are  effective  in 
selecting quality project proposals. 

EQ 3. What factors influenced the 
Sub-
achievement 
question:  Has 
selection 
procedure been effective?  

observed? 
the 

Case study 3 

What are the reasons  for which 
some potential beneficiaries are 
reluctant to apply? 

The  aim  is  to  understand  what  the  main  reasons  for 
not  applying  are.  The  analysis  also  focuses  on 
beneficiaries  that  apply  to  understand  what  are  the 
drivers for participation. 

the  evaluation  will  explore 
More  specifically, 
administrative  burden  as  a  possible  reason  for  not 
applying.  The  analysis  controlled  for  factors  such  as 
awareness, 
language 
barriers and relevance.  

communication, 

capacity, 

Case study 4 

How do EU projects compare in 
to 
terms  of 
similar  actions  funded  by  other 
sources? 

costs/benefits 

The  objective  of  the  analysis  is  to  conclude  whether 
the EU projects are comparable in terms of costs and 
benefits  to  similar  ones,  which  are  funded  by  other 
sources. Furthermore, if EU projects are more costly, 
the  analysis  will  explore  the  reasons  behind  the 
difference in costs. 

EQ  4.  How  effective  have  been 
the  communication  activities  in 
informing the potential applicants 
about  upcoming  calls  and 
in 
increasing the visibility of funded 
projects?  

EQ  9.  How  relevant  were  the 
actions and results achieved to the 
needs 
different 
of 
stakeholders? 

the 

EQ  6.  To  what  extent  has  the 
intervention been cost-effective?  

Case study 5 

Understanding 
underspending in projects 

the  scale  of 

The  objective  of  the  analysis  is  to  compare  the 
planned costs with the actual costs per project. Thus, 
to  understand  the  reasons  behind  the  recurrent 
underspending  and  to  identify  solutions  tackling  the 
underspending. 

EQ  6.  To  what  extent  has  the 
intervention been cost-effective?  

Case study 6 

Changes  in  management  of  the 
programmes by DG JUST 

The objective is to scrutinise how the management of 
the justice programme evolved over the programming 
period  2014-2020.  In  addition,  it  will  explore  the 
effect of changes in management on the beneficiaries. 

EQ7. Has the management of the 
programme  by  the  Commission 
become more efficient?  

Case study 7 

Are  the  beneficiaries  selected 
the  best  fit  to  help  the  target 
groups 
of 
socioeconomic impact? 

terms 

in 

The objective of the analysis is to understand whether 
the  justice  programmes  funds  the  most  relevant 
organisations.  The  case  study  assesses  which  is  the 
most  effective  combination  of  actors  within  a 
consortium  and  how  essential  is  the  role  of  project 
management organisations. 

EQ  9.  How  relevant  were  the 
actions and results achieved to the 
needs 
different 
stakeholders?  

the 

of 

Case study 8 

of 

Evaluation 
Gender 
Mainstreaming  at  the  level  of 
programme activities 

The  analysis  aims  to  evaluate  gender  mainstreaming 
at  the  level  of  project  activities  by  looking  at  how 
gender  has  been  mainstreamed  across  the  different 
projects and stages of the process cycle.  

54 

15.  How 

EQ 
gender 
mainstreaming  been  promoted  in 
the projects funded?  

has 

 
 
Overview of the evaluation criteria and questions 

Evaluation Criteria 

Evaluation questions 

1.  EQ1. To what extent have the objectives been achieved considering the set indicators? 

i)  What progress has been achieved so far by the programmes in meeting indicators listed in 

the respective Regulations? 

2.  EQ2.  How  fit  for  purpose  were  the  actions  funded  under  the  programme  to  the  objectives  of  the 

Effectiveness 

programme? 

3.  EQ3. What factors influenced the achievements observed?  

4.  EQ4: - How effective were the communication activities as to inform the potential applicants about 

upcoming calls and in increasing the visibility of funded projects? 

5.  EQ5. Have the projects delivered the results envisaged in the applications? 

6.  EQ6: To what extent have the interventions been cost-effective? 
7.  EQ 7 - Has the management of the programme by the Commission become more efficient? 
8.  EQ8: How well do the (original) objectives (still) correspond to the needs within the EU? 

9.  EQ9: How relevant were the actions and results achieved to the needs of the different stakeholders? 
10.  EQ10: How relevant for achieving the objectives of the Programme were the groups targeted by the 

intervention? 

Efficiency 

Relevance 

Coherence, 
Complementarity, 
Synergies 

11.  EQ11:  To  what  extent  are  these  interventions  coherent/  complementary  with  other  EU  and/or 
national  policies  and  funding  programmes  that  have  similar  objectives  and  Union  bodies'  work? 
Are the conclusions on coherence and complementarity in the interim evaluation still valid? 

EU added value 

Equity 

12.  EQ12: What is the added value resulting from the EU intervention(s), compared to what could be 

achieved by Member States? 

13.  EQ13:  What  would  be  the  most  likely  consequences  of  limiting  the  level  of  the  existing  EU 

intervention or completely stopping/withdrawing from it? 

14.  EQ14:  How  fairly  are  the  different  activities  distributed  across  the  different  target  groups,  and 

participating countries? 

15.  EQ15: How has gender mainstreaming been promoted within the funded interventions? 

i)  Was  gender  mainstreaming  included  in  the  programming  and  implementation  of  the 

programme, including calls for proposals? 

ii)  How was the principle of gender mainstreaming applied in practice by beneficiaries? 

16.  EQ16: How have the rights of the child been promoted within the funded interventions? 

17.  EQ17:  How  have  the  rights  of  people  with  disabilities  been  promoted  within  the  funded 

interventions? 

Scope for 
simplification 

18.  EQ18: How can the programme management, with a focus in particular on the grant management, 
be further simplified to alleviate the administrative burden of the Commission and of the applicants 
and the beneficiaries? 

55 

 
 
 
 
 
 
Annex 4: Results of the case studies 

Case study 

Case study 1 

Recurring beneficiaries and continuous projects 

Case study 2 

Is  the  quality  of  selection/  evaluation of  project 
proposals  robust?  Is  the  70-points  threshold 
pertinent? 

Case study 3 

What  are  the  reasons  for  which  some  potential 
beneficiaries are reluctant to apply? 

Case study 4 

How  do  EU  projects  compare  in  terms  of 
costs/benefits to similar actions funded by other 
sources? 

Case study 5 

Understanding  the  scale  of  underspending  in 
projects 

Result 

 
 

52.6% of partners involved in a project were recurring beneficiaries 
The  highest  share  of  grant  requests  was  submitted  by  recurring 
beneficiaries (46.43%), 

  On  average  continuous  projects  received  the  highest  score  in 
evaluations. Recurring applicants proved more successful in obtaining 
a high score than one time applicants. 

  There  were  no  significant  differences  in  the  characteristics  of 
projects obtaining 70 to 100 points.   
  On  average,  REC  projects  scored    higher  evaluation  scores  than 
projects under the justice programme.  
  Despite  identifying  a  number  of  aspects  related  to  the  evaluation 
procedures  to  improve  in  the  future,  overall,  the  evaluation  criteria  and 
appraisal accurately capture the quality of the applications/projects.   
  The  quality  of  selection/evaluation of  project  proposals  is  robust  and 
the 70 points threshold is pertinent.   

  The  geographical  coverage  of  countries  from  countries  like  Latvia, 
Poland, Croatia, Slovakia and Cyprus remains lower.  
  There is no evidence of potential applicants not being informed of 
programme  calls  due  to  low  coverage/quality  of  communication  and 
dissemination activities.   
  The main factors for not applying (mainly public institutions):  

low capacity and lack of experience in applying for and implementing 
their  perceived  complexity  and 

1. 
EU-  funded  projects  (vis  à  vis 
competitiveness of applications, including language);   

the  requirement  to  secure  co-financing  can  be  an  obstacle  for  both 

2. 
public institutions and non-profit organisations.  

3.  The  short  time  span  between  the  call’s  launch  and  the  deadlines, 
especially for public entities  

  Despite  being  quite  burdensome  in  the  application  procedures,  the 
justice  programme  is  found  to  generate  much  greater  benefits  than 
other  non-EU  initiatives,  despite  the  initial  costs  and  ongoing  work 
needed to participate.  
  Nonetheless,  potentially  valuable  organisations  from  civil  society 
(especially  smaller  ones)  are  often  locked  out  of  participating  in  EU 
programmes, and are left with alternative funding instruments that are less 
focused on creating EU added value, as the sole alternatives.  

  Underspending  of  EU  resources  seems  rather  limited under  the 
programme.  
justice 

  The key drivers appear to be  both internal and external factors, mainly 
linked to the difficulty in implementing and/or completing the activities as 
planned, namely:  

56 

 
 
 
 
Case study 

Result 

Case study 6 

Changes  in  management  of  the  programmes  by 
DG JUST 

Case study 7 

Are the beneficiaries selected best fit to help the 
target groups in terms of socioeconomic impact? 

Case study 8 

Evaluation of gender mainstreaming at the level 
of programme activities 

1.  unexpected events (e.g. the COVID-19 pandemic);  

adoption  of  a  careful  project  management  approach  and  expenditure 

2. 
choices by project beneficiaries;  

3.  overestimation of costs in proposals and then identifying savings over 
the course of project implementation.  

  Administrative  complexities  or  potential  rigidities  in  programme 
procedures do not seem to be the cause of project underspending. 

The programme’s management cycle was improved in recent years, at least in 
the perception of recurring beneficiaries:  

 
adopting  the  annual  work  programmes  in  the  year  before  they  are 
implemented  facilitated  an  earlier  launch  of  calls  and  thus  the  completion 
year;  
during 
of 

process 

same 

the 

the 

 
agreements 

the  length  of  the  appraisal  phase  and  the  preparation  of  grant 
shortened;  

been 

also 

has 

 

 

the  predictability 

of 

calls 

for 

proposals 

has 

improved; 

digitalisation of application and reporting procedures.  

However, the length of the application process is one of the main problems 
encountered by beneficiaries, and this has not yet been addressed.    

Another factor that makes the application process difficult for beneficiaries is 
the very technical language used in the application form. 

  This  evaluation  could  not  identify  clear  evidence  that  projects  are 
awarded  to  organisations  that  are  not  relevant  to  the  objectives  set  (at 
level).  
programme, 

project 

call, 

and 

  Beneficiaries  –  especially  public  authorities  –  are  more  likely  to  use 
the  services  of  external  consultancies  or  partner  with  specialised 
organisations to submit applications and receive support in implementation. 
se.  
However 

negative 

factor 

this 

per 

not 

is 

a 

  With  all  due  the  caveats,  effectiveness  is  typically  lower  in  projects 
implemented  by  consortia  formed  of  two  types  of  beneficiaries,  including 
one    secondary  or  higher  education  entity.  It  is  higher  in  projects 
partners.   
implemented 

three 

more 

by 

or 

  Difficulties  encountered  by  beneficiaries  in  the  application  and 
terms  of  partnership  development  and 
implementation  phase 
coordination  have  a  significant  impact  on  the  potential  of  the  project  to 
achieve its outputs and generate medium and long-term effects. 

in 

 

Projects  financed  under  the  justice  programme  rarely  cover  gender 
aspects in their design and implementation 

  As  a  result,  gender-sensitive  needs  assessments  are seldom carried 
out  and  this  is  the  key  reason  for  which  there  is  still  significant  scope  to 
improve  gender  mainstreaming.   A  second  reason  is  the  less  detailed 
presentation  of  expectations 
terms  of  gender  mainstreaming  at 
programme level. This second aspect is particularly  relevant to the justice 
programme.  

in 

57 

 
 
 
 
 
 
 
 
 
Case study 

Result 

  Overall,  there  is  a  high  level  of  awareness  of  the  importance  of 
promoting  gender  equality,  but  this  awareness  does  not  translate  into 
activities.  
gender 

sensitive/transformative 

  The gender-related measures taken are often limited to ensuring equal 
participation in project activities (e.g. training) or teams.   

58