COMMISSION STAFF WORKING DOCUMENT Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS on the interim evaluation of the implementation of the Justice Programme 2014-2020
Date: 2018-06-29
Institutions: Directorate-General for Justice and Consumers, European Commission
Subjects: EU programme, European judicial area, access to the courts, fight against crime, fundamental rights, judicial cooperation in civil matters in the EU, judicial cooperation in criminal matters in the EU, mutual recognition principle, rights of the defence, rule of law

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 29.6.2018  SWD(2018) 356 final  COMMISSION STAFF WORKING DOCUMENT Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS on the interim evaluation of the implementation of the Justice Programme 2014-2020 {COM(2018) 507 final} - {SWD(2018) 357 final}  
 
 
Table of contents 

1. 

2. 

3. 

INTRODUCTION ................................................................................................................................ 3 

BACKGROUND TO THE INITIATIVE ............................................................................................. 4 

IMPLEMENTATION / STATE OF PLAY ........................................................................................ 10 

4.  METHOD ........................................................................................................................................... 17 

5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS .......................................... 20 

6. 

CONCLUSIONS AND WAY FORWARD ........................................................................................ 25 

ANNEX 1: PROCEDURAL INFORMATION ............................................................................................ 28 

ANNEX 2: STAKEHOLDERS CONSULTATION .................................................................................... 31 

ANNEX 3: METHODS ................................................................................................................................ 33 

 
 
 
1. 

INTRODUCTION 

Purpose and scope of the evaluation 

This staff working document sets out the results of the interim evaluation of the Justice 
Programme for the period 2014-2020.  

The evaluation was carried out under the provisions of Article 14(2)(b) of the European 
Parliament  and  Council  Regulation  establishing  the  Justice  Programme  for  the  period 
2014-20201.  

It aimed mainly to assess the Programme’s outputs and results compared to its objectives 
and to assess qualitative and quantitative aspects of its implementation. It also assessed 
whether the current Programme is on track to achieving its objectives and the extent to 
which  the  recommendations  of  the  previous  2007-2013  ex-post  evaluation  have  been 
followed.  The  ex-post  evaluation  carried  out  in  2015  concerned  the  three  predecessor 
programmes  which  were  merged  in  the  current  Justice  Programme  (namely  the  Civil 
justice  programme,  the  Criminal  justice  programme  and  the  Drug  prevention  and 
information programme)2. 

The  current  interim  evaluation  informed  the  Report  from  the  Commission  to  the 
European  Parliament,  the  Council  of  the  European  Union,  the  European  Economic  and 
Social  Committee  and  the  Committee  of  Regions  which  this  document  accompanies.  It 
also  helped  the  reflection  on  the  Programme's  future  by  providing  evidence-based 
information.  

Under  the  Regulation,  the  Commission  is  requested  to  present  an  ex-post  evaluation 
report for the Justice Programme by 31 December 2021. 

The Programme was assessed based on the following main evaluation criteria: 

1.  Relevance:  whether  and  to  what  extent  the  Justice  Programme  addresses  needs  and 
problems of the target groups identified in the 2011 Impact Assessment3 and in the legal 
basis of the Programme (as well as emergent needs related to the creation of a European 
Area of Justice) and whether its objectives are still relevant for the needs and problems of 
the beneficiaries; 

2. Efficiency: whether and to what extent the costs of the Programme were proportionate 
given the benefits achieved and which parameters/factors participated in these results; 

3.  Effectiveness:  whether  and  to  what  extent  the  Justice  Programme  has  achieved  its 
general  objective,  as  well  as  its  four  specific  objectives,  and  which  are  the  factors  that 
have contributed to these achievements; 

1  Regulation  (EU)  No 1382/2013  of  the  European  Parliament  and  of  the  Council  of  17 December  2013  establishing  a  Justice 
Programme  for  the  period  2014  to  2020  (OJ  L  354  of  28.12.2013),  available  online  at:  http://eur-lex.europa.eu/legal-
content/EN/TXT/PDF/?uri=CELEX:32013R1381&from=EN. 
2 Ex-post evaluation of five programmes implemented under the 2007-2013 financial perspective – Final Report. Specific programme 
evaluation: 
here 
Justice, 
http://ec.europa.eu/justice/grants1/files/expost_evaluations_reports_2007_2013/civil_justice_programme.pdf  .  Ex-post  evaluation  of 
five programmes implemented under the 2007-2013 financial perspective - specific programme evaluation: Criminal Justice Support 
Programme, 
here 
http://ec.europa.eu/justice/grants1/files/expost_evaluations_reports_2007_2013/criminal_justice_programme.pdf.  Ex-post  evaluation 
of  five  programmes  implemented  under  the  2007-2013  financial  perspective  Specific  programme  evaluation:  Drug  Prevention  and 
Information 
here 
Programme, 
http://ec.europa.eu/justice/grants1/files/expost_evaluations_reports_2007_2013/dpip_programme.pdf . 
3 SEC (2011) 1364 Final. 

available 

available 

available 

Civil 

2 

 
 
                                                            
4. Coherence/Complementarity/Synergies: whether and to what extent the Programme 
is  coherent  with  other  interventions  at  the  EU  and  international  level,  such  as  with  the 
predecessor  EU  programmes  in  the  field4,  with  activities  supported  by  other  Union 
instruments  and,  in  general,  with  the  European  priorities  in  the  fields  covered  by  the 
Programme; 

5. EU added-value: to what extent the effects from the EU action  are additional to the 
value that would have resulted from action at the national level only; 

6.  Equity:  whether  and  to  what  extent  the  Justice  Programme  has  distributed  the 
available  resources  fairly  among  beneficiaries  in  different  Member  States,  took  into 
consideration the needs of target  groups,  promoted gender mainstreaming, the rights of 
the child and the rights of people with disabilities5; 

7. Scope for simplification: whether and to what extent the management of the Justice 
Programme could be further simplified. 

Geographically,  the  Programme  is  open  to  all  EU  Member  States6,  but  also  to  the 
European Free Trade Association States that are party to the European Economic Area, 
candidate countries, potential candidates and countries acceding to the Union, provided 
that  they  conclude  an  agreement  with  the  Union  laying  down  the  details  of  their 
respective participation in the programme. Albania joined the Programme as of 2017.  

The reference period for this interim evaluation is the first half time of the Programme’s 
implementation from 2014 to mid-20177. 

2.  BACKGROUND TO THE INITIATIVE 

The  Treaty  on  the  Functioning  of  the  European  Union  provides  for  the  creation  of  a 
European  area  of  freedom,  security  and  justice  based  on  mutual  recognition  of  judicial 
decisions and mutual trust among Member States, in which persons are free to move and 
can  rely on the respect of fundamental rights  as well as  of common principles, such as 
non-discrimination, gender equality, effective access to justice for all, the rule of law and 
well-functioning independent judicial systems.  
These  ambitious  goals,  set  by  the  Treaty,  have  also  been  reaffirmed  by  the  European 
Council in the Stockholm Programme8. The achievement of a Europe of law and justice 
is one of the political priorities of the EU and the 2014-2020 Justice Programme is one of 
the EU instruments that contribute achieving this objective.  

The  Programme  was  designed  to  overcome  the  obstacles  in  the  functioning  of  an 
effective  European  Area  of  Justice9  and  to  encourage  national  judicial  systems  to  have 
faith in each other’s standards of fairness and justice, as prerequisite for the realisation of 
a  European  Area  of  Justice.  The  Programme  also  contributes  directly  to  the  Juncker 

4 As already mentioned, the Justice Programme has replaced three programmes that were in force during the 2007-2013 Multi-annual 
Financial Framework. 
5  In  this  context,  “equity”  does  not  mean  necessarily  equality  (i.e.  that  target  groups  and  Member  States  receive  the  same  level  of 
support  in  terms  of  projects  funded  and  resources  allocated).  Instead,  the  concept  of  equity  relates  to  the  allocation  of  resources 
mirroring  the  different  intensity  of,  and  urgency  for,  support  of  each  target  group/Member  States  rather  than  for  example  their 
capacity for advocacy or in accessing the Programme funds/managing projects. 
6 The United Kingdom and Denmark do not participate in the Programme. 
7 Given its relatively early stage of implementation, the 2017 Annual Working Programme has been analysed in terms of design and 
structure, not in terms of execution. 
8 OJ C 115, 4.5.2010, p. 1. 
9 Such as, in particular, insufficient judicial cooperation in civil and criminal matters, wide differences in national procedures and laws 
across the EU, lack of trust from EU citizens in the functioning of judicial systems of other Member States, lack of  awareness and 
difficulties in the exercise of EU citizenship rights by people, limited improvements in preventing and combating drug-related crimes. 

3 

 
 
 
                                                            
priority aimed at establishing an area of justice built on respect of fundamental rights and 
mutual trust, and digital single market. 

Description of the programme 

The general objective of the Justice Programme is to contribute to the development of a 
European area of justice based on mutual recognition and mutual trust, in particular by 
promoting judicial cooperation in civil and criminal matters.  

In order to do so, the Justice Programme provides for four specific objectives which aim 
to: 

  facilitate and support judicial cooperation in civil and criminal matters. Through this 

objective, the Programme supports activities that contribute to: 

 

 

 

the  effective  and  coherent  application  of  the  EU  acquis  relating  to  judicial 
cooperation  in  civil  and  criminal  matters  also  by  building-up  and/or 
improving data collection and statistics on the application of the EU acquis; 

the  enforcement  of  EU  instruments  and  judicial  decisions,  in  particular 
resulting from cross-border disputes;  

the improvement of the exchange of information among professionals in order 
to enhance the operational cooperation and mutual trust in the EU.  

Under this specific objective, the Programme supports the participation of the EU to 
the  Hague  Conference  of  Private  International  Law10  and  activities  related  to  the 
setting  up  and  strengthening  of  national  networks  under  the  European  Judicial 
Network  in  civil  and  commercial  areas.  Moreover,  specific  calls  for  proposals  to 
prevent  radicalisation  leading  to  terrorism  and  violent  extremism  have  also  been 
launched in the wake of terrorism episodes in the EU. In the context of this specific 
objective,  the  EU  carries-out  its  cooperation  with  the  Council  of  Europe  on  two 
particular issues: the SPACE report on prison statistics and the setting up of the EU 
network of prison monitoring bodies in the Member States. 

 
support  and  promote  judicial  training  of  professionals  such  as  judges, 
prosecutors,  notaries,  prison  staff  and  lawyers  on  civil  and  criminal  law  EU 
instruments, fundamental rights, judicial ethics and the rule of law, including language 
training  on  legal  terminology,  with  a  view  to  fostering  a  common  legal  and  judicial 
culture.  Judicial  training  can  involve  “basic”  components,  such  linguistic  skills  and 
terminology,  and  more  specialised  aspects,  such  as  seminars  on  specific  aspects  of 
both  civil  and  criminal  law,  e-learning  and  exchanges  of  staff  and  experience.11  For 
example, specialised training is made available in the field of competition law, which 
typically requires a significant amount of specialisation. The Programme supports, in 
particular, the training of the members of the judiciary and judicial staff, but also other 
legal practitioners associated with the judiciary. It also funds the development of tools 
for training providers and co-finances the expenditure associated with the work of the 
European Judicial Training Network.  
Certain  areas  represent  priorities  for  the  training  activities:  EU  civil,  criminal  and 
fundamental  rights  law,  legal  systems  of  the  Member  States,  judicial  ethics  and  the 
rule of law, knowledge of cross-border IT tools and linguistic skills.  

10  The  EU  is  party  to  The  Hague  Conference  on  Private  International  Law  since  in  2007.  The  purpose  of  this  international 
intergovernmental organisation is to work for the progressive unification of the rules of private international law in the participating 
countries. Most of the conventions developed by The Hague Conference fall within exclusive or partial external competence of the 
EU and are part of the EU acquis. 
11 European Commission, Directorate-General for Justice and Consumers, European Judicial Training 2016. Available online at the 
following address: http://ec.europa.eu/justice/criminal/files/final_report_2015_en.pdf. 

4 

 
                                                            
Moreover, the Programme finances projects which aim at achieving the objectives of 
the  European  e-Justice  Strategy  2014-201812  and  that  contribute  to  the  effective  and 
coherent application of EU criminal law in the areas of the rights of persons suspected 
or accused of crime (procedural rights priority) and of the rights of victims of crime; 

  facilitate effective access to justice for all, including to promote and support the rights 
of victims of crime as well as the procedural rights of suspects and accused persons in 
criminal  proceedings.  Actions  financed  in  this  area  by  the  Programme  aim  at 
providing  EU  citizens  with  effective  remedies  in  case  of  violation  of  EU  law, 
especially  where  national  procedures  are  too  difficult  for  citizens  to  be  enabled.  In 
particular,  the  Programme  aims  to  promote  the  use  of  other  types  of  remedies  and 
non-remedies developed in the EU that can provide a quick, efficient and less costly 
solution to disputes, as supported, for example, by the e-Justice Portal.13 It also aims 
to  encourage  a  close  cooperation  between  national  authorities  or  administrative 
bodies, which is particularly important for the effectiveness of certain EU rights.  

  promote  initiatives  in  the  field  of  drug  policy  as  regards  judicial  cooperation  and 
crime  prevention  aspects14.  In  this  field,  the  Justice  Programme  promotes  initiatives 
which  focus  on  judicial  cooperation  and  crime  prevention.  The  activities  under  the 
Programme  mainly  aim  to  promote  practical  application  of  drug-related  research,  to 
support  civil  society  organisations  and  key  stakeholders  and  to  expand  the  knowledge 
base  and  develop  innovative  methods  of  addressing  the  phenomenon  of  new 
psychoactive substances.  

Budget 

The  initial  total  Justice  programme's  budget  for  the  period  2014-2020  is  EUR 
377 604 000.  According  to  available  sources,  the  total  requested  and  committed  EU 
contribution still not reached the amount planned, despite the improvements made 
in 2016 (see Table 1). In general, most resources allocated to action grants and operating 
grants  have  been  committed  (with  a  commitment  rate  of  close  to  90%),  but  the 
commitment rate for procurement activities was much lower, hovering around 60%-70%. 

Table 1: Total amount planned and committed in the Justice Programme 2014-2016 

Budget year 

Annual amount 
planned (in euro) 

Annual amount 
committed (in euro) 

Ratio 
committed/planned 

2014 

2015 

2016 

45 812 000.00 

47 226 649.00 

50 250 000.00 

36 671 240.16  

39 675 719.11  

47 535 032.81  

Total 2014-2016 

143 288 649.00 

123 881 992.08 

80.05% 

84.01% 

94.60% 

86.46% 

12  Council  of  the  European  Union,  Multiannual  European  e-Justice  Action  Plan  2014-2018,  2014/C  182/02,  available  here 
https://publications.europa.eu/en/publication-detail/-/publication/2a2f84b9-f3a4-11e3-831f-01aa75ed71a1/language-en.  
13 Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and 
the  Committee  of  the  Regions,  The  EU  Justice  Agenda  for  2020  -  Strengthening  Trust,  Mobility  and  Growth  within  the  Union, 
Strasbourg, 11.3.2014, COM(2014) 144 final. 
14  Insofar  as  this  kind  of  initiatives  are  not  covered  by  the  Internal  Security  Fund  for  financial  support  for  police  cooperation, 
preventing 
https://ec.europa.eu/home-
affairs/financing/fundings/security-and-safeguarding-liberties/internal-security-fund-police_en  ),  or  by  the  Health  for  Growth 
Programme (for more info, see https://ec.europa.eu/health/funding/programme/2014-2020_en ). 

crisis  management 

(for  more 

combating 

crime 

info, 

and 

and 

see 

5 

 
 
 
 
 
                                                            
Sources: Annual monitoring reports (reports on the implementation of the Annual working programmes and data extracted from 
Sygma for 2016) 

In  order  to  allow  the  achievement  of  its  objectives,  the  Programme  identifies  a  wide 
range  of  activities  to  be  implemented.  In  particular,  according  to  Article  6  of  the 
Regulation  establishing  the  Justice  Programme  for  the  period  2014  to  2020,  the 
Programme can finance: 

  Analytical  activities,  in  particular:  collection  of  data  and  statistics;  development  of 
common  methodologies  and,  where  appropriate,  indicators  or  benchmarks;  studies, 
researches, analyses  and  surveys;  evaluations; elaboration and  publication  of  guides, 
reports  and  educational  material;  workshops,  seminars,  experts  meetings  and 
conferences;  

  Mutual learning,  cooperation, awareness  raising and dissemination activities, in 
particular:  identification  of,  and  exchanges  concerning  good  practices,  innovative 
approaches  and  experiences;  organisation  of  peer  reviews  and  mutual  learning; 
organisation  of  conferences,  seminars,  information  campaigns;  compilation  and 
publication  of  materials  to  disseminate  information  about  the  Programme  and  its 
results;  development,  operation  and  maintenance  of  systems  and  tools,  using 
information  and  communication  technologies,  including  the  further  development  of 
the European e-Justice portal as a tool to improve citizens' access to justice;  

  Training  activities,  for  instance:  staff  exchanges,  workshops,  seminars,  train-the-
trainer events, including language training on legal terminology, and the development 
of  online  training  tools  or  other  training  modules  for  members  of  the  judiciary  and 
judicial staff; 

  Actions  to  support  the  main  actors  whose  activities  contribute 

the 
implementation  of  the  objectives  of  the  Programme,  such  as:  support  for  Member 
States  in  the  implementation  of  Union  law  and  policies;  support  for  key  European 
actors and European-level networks, including in the field of judicial training; support 
for  networking  activities  at  European  level  among  specialised  bodies  and  entities  as 
well as national, regional and local authorities and non-governmental organisations15. 

to 

The main stakeholders  eligible for support are: European networks active in the area of 
judicial  cooperation;  justice  professionals;  public  or  private  organisations  (usually  non-
profit-oriented,  but  in  particular  cases  also  profit-oriented  organisations16);  national, 
regional  and  local  authorities  in  EU  Member  States;  non-governmental  organisations; 
national  judicial  schools  of  individual  Member  States;  universities  and  research 
institutions  as  well  as  international  organisations  active  in  the  areas  covered  by  the 
Programme.  

In terms of target groups, intended as the groups that can benefit directly or indirectly 
from the Programme, potentially all EU citizens are included, since the Programme aims 
at creating a European Area of Justice where all citizens are aware of and can exercise 
their rights. However, in particular, the Justice Programme covers: judicial practitioners, 
public 
international  organisations,  non-governmental 
organisations  and  other  research  entities;  companies  undergoing  insolvency  or  pre-
insolvency  proceedings;  citizens  accused  or  victim  of  a  crime,  families,  consumers, 
minors,  groups  in  need  of  specialised  legal  protection  and  subjects  at  risk  of 

authorities,  universities, 

15 To see the types of activities funded by the Justice Programme in 2014, 2015 and 2016, see the Mid-term evaluation of the Justice 
Programme 2014-2020, Ernst & Young Financial-Business Advisors, Final Report, April 2018, p. 71-72. 
16 Bodies and entities which are profit-oriented can be funded only in conjunction with non-profit or public organisations. 

6 

 
                                                            
radicalisation. Additionally, specific calls support entrepreneurs in the single market and 
national drug awareness and prevention agencies17. 

The Justice Programme  has  three main funding mechanisms:  action grants,  operating 
grants and procurement actions18.  

  Action  grants  are  addressed  to  civil  society  organizations,  to  Member  States 

authorities and universities;  

  Operating grants ("support to Networks") fund mainly European networks active in 
the  area  of  facilitating  and  supporting  judicial  cooperation  in  civil  and/or  criminal 
matters that have signed Framework Partnership Agreements with the Commission;  

  Procurement  actions  ("Commission  initiatives")  fund  mostly  conferences,  expert 
meetings,  seminars,  awareness-raising  activities  and  studies,  but  also  translation 
services and IT projects. 

The Programme intervention logic is outlined in the Figure 1. 

External factors could influence the Programme's outcomes and make challenging to 
identify changes as a result of its intervention. In particular, the economic crisis has led to 
a  lack  of  interest  on  the  part  of  public  sector  stakeholders  in  the  areas  covered  by  the 
Programme  (particularly  in  the  field  of  access  to  justice)  and,  therefore,  to  a  related 
reduction of national funding for the justice systems. This is more accentuated in certain 
Member  States  where  the  political climate  is  less  supportive19.  Also  the  uneven  pace at 
which Member States incorporate the EU acquis in their national legislations is a factor 
influencing the performance of the Programme20.  

Baseline and points of comparison 

17 For more information, see below p. 15 and see also section 5.4 on "Target groups and beneficiaries" in the Interim report, p. 77 et 
seq. 
18  For  more  information  on  the  number of  action  grants  and  operating  grants  awarded  per year and  on  the  number of  procurement 
activities, see section 5 "Implementation state of play" in the Interim report, in particular p. 69 et seq. 
19 Mid-term evaluation of the Justice Programme 2014-2020, Ernst & Young Financial-Business Advisors, Final Report, April 2018, 
published. 
20 Ibid. 

7 

 
 
 
                                                            
The  interim  evaluation  assessed  the  Justice  Programme  performance  starting  from  the 
situation  described  in  the  2011  Impact  Assessment21  and  in  the  ex-post  evaluation  of 
2007-2013, concerning the three predecessors programmes (Civil justice programme, the 
Criminal  justice  programme  and  the  Drug  prevention  and  information  programme), 
carried out in 201522.  

These documents contain the baselines and main points of comparison for measuring the 
Programme’s achievements during the reporting period.  

The  baseline  analysis  showed  that,  if  no  changes  had  been  made  between  the  2007-13 
and 2014-20 programming period, the three previous programmes would have continued 
to  be  successfully  implemented,  but  at  a  reduced  potential  in  particular  due  to:  1.  The 
lack of flexibility in the funding instruments which did not reflect the pace of change and 
reform in this policy area; 2. The fragmentation of funding that reduced the capacity of 
the  programmes  to  deliver  results  in  horizontal  and  cross-cutting  issues;  and  3.  The 
elevated  number  of  different  funding  instruments  which  increased  the  administrative 
burden. 

Given these difficulties, the Impact Assessment suggested the option to consolidate the 
three programmes in the current Justice Programme. 

in 

This  possibility  was  assessed 
relevance/scope,  effectiveness, 
terms  of 
complementarity,  European  added  value,  efficiency  and  potential  for  simplification. 
Since the option to proceed with only one programme was undertaken, the findings of the 
Impact  Assessment,  together  with  the  recommendations  of  the  ex-post  evaluations  of 
2007-2013,  provide  a  useful  baseline  to  verify  whether  the  current  Justice  Programme 
achieved its expected goals. 

its 

Follow-up on the ex-post evaluation report of the period 2007-2013 

All  the  ex-post  evaluation  reports  concerning  the  previous  funding  programmes 
confirmed  their  overall  effectiveness  and  highlighted  that  their  specific  objectives  and 
priorities  were  largely  specific,  attainable  and  realistic,  but  not  always  measurable. 
Indicators,  allowing  the  measuring  of  progress  toward  the  attainment  of  the  specific 
objectives, were not included.  

Moreover, the reports pointed out the need to make improvements on other issues which 
could have increased the impact, added value, effectiveness and efficiency in delivering 
result  of  the  programmes.  In  particular,  the  reports  have  identified  the  following  needs 
common to the three programmes:  

1.  Better  definition  of  the  priorities  in  order  to  ensure  that  they  can  be  adequately 
achieved within an earmarked budget;  

2.  Realistic  assessments  of  project  risks  and  better  risk  mitigation  strategies 
throughout the projects duration;  

3. Increase focus on assessment of impacts at all levels and not merely on outputs, as 
regards monitoring and evaluation;  

4.  Explore  ways  of  enhancing  the  uptake  of  project  outputs,  results  and  best 
practices by other organisations, including in other Member States;  

5. More dissemination/use of results and outputs of the funded activities.  

21  European  Commission,  'Commission  Staff  Working  Paper  –  Impact  Assessment  -  Accompanying  the  document  Proposal  for  a 
Regulation  of  the  European  Parliament  and  of  the  Council  establishing  for  the  Period  2014-2020  the  Justice  Programme.  Impact 
Assessment’. SEC (2011) 1364 Final. 
22 Ex-post evaluation of five programmes implemented under the 2007-2013 financial perspective – Final Report, ibid. 

8 

 
                                                            
The  ex-post  evaluation  reports  of  the  period  2007-2013  have  also  highlighted  the 
following problems: 1. The dilution of funds amongst many small-scale projects with 
limited  impact  and  EU  dimension;  2.  No  balanced  geographical  spread  among  the 
organisations which receive funding; 3. The complex and bureaucratic procedures for 
the applicants; 4. The high administrative burden on the Commission and an increase 
of  the  length  of  procedures  due  to  the  multiplication  of  procedures  for  the  different 
programmes.  

These issues were taken into consideration and were either integrated into the Regulation 
establishing  the  current  Justice  Programme  and  in  the  Annual  Work  Programmes  or 
translated  into  technical  implementation  measures  and  included  in  the  Guides  for 
applicants to projects calls. 

In particular, the merger of the three previous programmes has provided a significant 
positive impact on the identified problems of scope, effectiveness, fragmentation and 
efficiency  (see  further  details  in  section  5  “Analysis  and  answer  to  the  evaluation 
questions”).  

3. 

IMPLEMENTATION / STATE OF PLAY 

Programme management  

The  Justice  Programme  is  implemented  via  direct  centralised  management  by  the 
Commission.  This  was  meant  to  ensure  a  close  relationship  between  the  Programme 
management  and  EU  policy-making  and  to  contribute  to  the  alignment  of  budget 
implementation  with  EU  policy  priorities  in  line  with  the  "budget  for  result"  approach. 
This  management  mode  allows  the  Commission  to  tailor  funded  activities  to  policy 
priorities  and  policy  needs  and  to  target  directly  the  relevant  groups  of  stakeholders. 
According  to  the  evaluation  carried  out,  it  also  allows  a  close  contact  with  the 
programmes' beneficiaries and better knowledge of the needs on the ground. Indeed, all 
beneficiaries interviewed, who also participated in predecessor programmes, agreed that 
the  current  Justice  Programme  represented  an  improvement  in  terms  of  capacity  to 
engage  with  European  Commission  officials  (i.e.  having  a  reliable  point  of  contact  to 
answer  their  questions  and  help  them  address  any  project-related  issues  that  may  arise 
during the implementation phase)23.  

State of play  

The  purpose  of  this  section  is  to  present  the  state  of  play  of  the  Programme’s 
implementation  and  its  key  initiatives  in  the  period  2014-2017.  It  will  also  provide  the 
qualitative and quantitative results achieved in the same period, in general terms and for 
each specific objective. 

23  Although,  the  beneficiaries  interviewed  were  not  specifically  asked  to  describe  their  experience  with  other  kinds  of  Programme 
management. 

9 

 
 
                                                            
In  order  to  measure  the  progress  from  the  baseline  situation,  a  series  of  Programme 
indicators  have  been  devised  for  the  current  Justice  Programme.  The  introduction  of  a 
system  of  indicators  has  proved  to  be  adequate  for  measuring  the  achievements  of  the 
Programme since the selected indicators show if the targets are close to being achieved or 
have  already  been  achieved  and  this  contributes  to  better  focus  the  Programme's 
outcomes.  

  General  objective:  The  development  of  a  European  area  of  justice  based  on 

mutual recognition and mutual trust 

The  Regulation  establishing  the  Justice  Programme  does  not  provide  for  any  indicator 
concerning  its  general  objective.  However,  the  Commission  reports  via  the  Programme 
Statements, accompanying the Draft General Budget of the EU24, use the percentage of 
legal  practitioners  trained  (not  only  through  this  Programme)  as  the  main  indicator  for 
measuring the Justice Programme’s impact, as shown in Table 2. 

Table 2: Result indicator, target and actual value 

Result indicator 

1. Cumulative number of legal professionals 
receiving  training  (not  only  through  the 
Programme)  on  EU  law  or  law  of  another 
Member  State, 
including  Civil  Justice, 
Criminal Justice and Fundamental Rights 

Baseline 
2013 

2016 

Targets 2020 

239 000 

638 000 

700 000 legal 
practitioners by 2020 

Source: Annual reports on European Judicial Training  

The  2020  target  was  almost  reached  already  in  201725.  The  graphic  below  (Figure  2) 
shows  clearly  that  the  number  of  legal  practitioners  trained  has  constantly  increased 
between 2013 and 2016.  

Figure 2: number of legal practitioners trained 

800000

700000

600000

500000

400000

300000

200000

100000

0

2013 (baseline)

2014

2015

2016

2020 (target)

24 COM (2017) 400 - May 2017, Draft Budget of the European Union 2017. Information on indicators related to the general objectives 
of the Programmes and indicators related to the specific objectives of the Programmes are provided by this document. 
25  Moreover,  gaps  in  data  described  in  the  training  reports  lead  to  surmise  that  the  number  of  trained  judicial  staff  is  somewhat 
underreported. 

10 

 
 
 
 
 
 
 
 
                                                            
Concerning  the  Specific  objectives  of  the  Programme,  the  following  indicators  have 
been used to evaluate its performance: 

  Specific objective 1: Judicial cooperation in civil and criminal matters 

The  Programme's  performance  under  this  specific  objective  is  measured  through  two 
main  indicators  (see  Table  3):  the  average  time  of  the  surrender  procedure  under  the 
European  Arrest  Warrant  in  cases  where  the  person  consents  to  the  surrender  and  the 
number  of  exchanges  of  information  in  the  European  Criminal  Records  Information 
System.  The  European  Arrest  Warrant  is  the  most  important  EU  legal  instrument 
developed  in  the  area  of  mutual  recognition  of  judicial  decisions.  It  consists  in  a 
simplified  cross-border  judicial  surrender  procedure  for  the  purpose  of  prosecuting  or 
executing a custodial sentence or detention order (a warrant issued by one EU Member 
State's judicial authority is valid in the entire territory of the EU)26. 
The European Criminal Records Information System is a database established to improve 
the  exchange  of  information  on  criminal  records  throughout  the  EU.  All  EU  Member 
States are currently connected to this system27.  
Concerning  the  first  indicator,  the  data  appears  to  be  highly  influenced  by  exogenous 
factors (such as the different degree of incorporation of EU directives at national level, as 
well  as  the  different  degree  of  judicial  system  reforms  within  EU  Member  States  that 
have  a  significant  impact  on  the  level  of  enforcement  of  the  EU  acquis28)  that  make 
estimating the contribution of the Justice Programme to the realization of this indicator 
difficult. Concerning the second indicator, after a rapid increase post the 2012 baseline, 
the number of exchanges appears to have stabilised and it is likely that this objective will 
be reached.  

Table 3: Result indicators, target and actual values 

Result indicators 

Baseline 2013 

2015-2017 

Targets 2020 

(time  between 

1. Average time of the surrender 
procedure 
the 
arrest  and  the  decision  on  the 
surrender  of  the  person  sought) 
the  European  Arrest 
under 
Warrant 
the 
person consents to the surrender 

in  cases  where 

14-20  days  (according  to  DG 
Justice 
of 
national  reports  of  Member 
States to the Council)  

aggregations 

14 days (2015) 

10 days by 2020 

2.  Number  of  exchanges  of 
the  European 
information 
Criminal  Records 
Information 
System  

in 

300 000 exchanges (2012) 

2 571 991 (2017) 

3 500 000 
exchanges by 
2020 

  Specific objective 2: Judicial training  

The  Programme's  performance  concerning  this  specific  objective  is  measured  through 
one main indicator: the number and percentage of members of the judiciary and judicial 

26 The European arrest warrant has been operational since 1 January 2004. It has replaced the extradition procedure that used to exist 
between EU countries. At present, a person who has committed a serious crime in a Member State, but who lives in another can be 
returned  to  the  first  country  to  face  justice  quickly  and  with  little  administrative  burden.  For  more  info,  see  Council  Framework 
Decision of 13 June 2002 on the European arrest warrant and the surrender procedures between Member States (2002/584/JHA). 
27 This database ensures that information on convictions is exchanged between EU Member States in a uniform, fast and compatible 
way, provides judges and prosecutors with easy access to comprehensive information on the criminal history of persons concerned, 
including in which EU Member States that person has previously been convicted and removes the possibility for offenders to escape 
convictions by moving from one Member State to another. On 28 June 2017, the European Commission published its first report on 
the use of this database by EU countries. 
28 These obstacles were mentioned by several projects and shown also by the "Justice Scorecards". 

11 

 
 
 
 
                                                            
staff that participated in training activities, staff exchanges, study visits, workshops and 
seminars funded by the Programme.  

This  indicator  appears  well-adapted  to  the  goals  of  the  specific  objective  and  of  the 
Programme, as it helps track both the overall contribution of the EU funding instruments 
of the European Commission and the specific contribution of the Justice Programme. It is 
clearly measurable and it is showing that, looking at the entire Commission, the goal of 
training 20 000 practitioners yearly by 2020 has been already achieved (see Table 4). The 
drop in 2016 below this threshold is likely misleading, as data from the European Social 
Fund-funded judicial training is currently missing29. As ca. 3 000 legal practitioners were 
trained in 2015 with European Social Fund funding, it is quite likely that once this data 
are reported, the objective will have been achieved for 2016 as well. 

Table 4: Result indicator, target and actual value 

Result indicators 

Baseline 2011 

2015 

2016 

Targets 2020 

1.  Number  and  percentage  of 
members  of  the  judiciary  and 
judicial staff that participated in 
staff 
training 
exchanges, 
visits, 
workshops and seminars funded 
by the Programme 

activities, 
study 

For the whole Commission: 
8 639 in 2011  

25 680 

18 444 

20 000 by 2020  

(for the whole  

Commission) 

For  DG 
Consumers: 6 681  

Justice 

and 

16 723 

13 930 

  Specific objective 3: Access to justice 

The  Programme's  performance  concerning  this  specific  objective  is  measured  through 
two main indicators: the number of hits on the e-justice portal and the number of Victim 
Support Organisations with national coverage (see Table 5). The e-justice portal provides 
information  on  justice  systems  and  improves  and  facilitates  the  access  to  justice 
throughout the EU, in 23 languages30. The number of Victim Support Organisations is an 
important  indicator  to  assess  the  implementation  of  Directive  2012/29/EU  establishing 
minimum  standards  for  the  rights,  support  and  protection  of  victims  of  crime.  This 
directive  contains  important  procedural  provisions  for  victims  of  crime  regarding,  in 
particular, their right to be heard and have access to support services (i.e. Victim Support 
Organisations). 

Concerning  the  first  indicator,  it  has  already  exceeded  the  stated  goals  for  2020. 
Concerning the second indicator, it seems likely that it will be reached, though it partly 
depends  on  whether  Member  States/Victim  Support  Organisations  will  implement 
quality  standards,  which  is  partially  out  of  the  control  of  the  Justice  Programme  and 
could  be  influenced  by  exogenous  factors,  such  as  the  uneven  pace  at  which  Member 
States implement and enforce the EU acquis or other political priorities or emergencies in 
the  Member  States  (in  particular  the  economic  crises  has  weakened  the  civil  society 
engagement for this specific objective). 

Table 5: Result indicators, target and actual values 

Result indicators 

Baseline 2013 

2017 

Targets 2020 

1.  Number  of  hits  on  the  e-justice 
portal31  

630 000 (2012) 

2 690 574  

+ 20% per year 

29 For more info on the European Social Fund, see http://ec.europa.eu/social/main.jsp?catId=325&langId=en. 
30 More info available at the following link: https://e-justice.europa.eu/home. 
31 Source of data: DG Justice and Consumers. 

12 

 
 
 
 
 
 
 
                                                            
2.  Number  of  Victim  Support 
Organisations with national coverage 
(implementation 
Directive 
2012/29/EU) 

of 

10 Victim Support 

 Organisations 

22  Victim  Support 
Organisations  

(
2
0
1
2
) 

Victim 

28 
Support 
Organisations  –  at  least  one 
Victim Support Organisation 
each  Member  State 
in 
fulfilling 
quality 
the 
standards/indicators  set  by 
Member States or developed 
individually  by  the  Victim 
Support  Organisations  (by 
2020) 

  Specific objective 4: Drugs policy 

The  Programme's  performance  concerning  this  specific  objective  is  measured  through 
two  main  indicators:  the  number  of  new  psychoactive  substances  assessed  (including 
through testing, if necessary) to enable the EU or the Member States to take appropriate 
action to protect consumers, depending on the type and level of risk that they may pose 
when consumed by humans and the % of problem opioid users that are in drug treatment.  

Given the stability observed over time of indicator 2 (which has remained at the 50% for 
the  last  five  years),  it  seems  difficult  that  this  percentage  could  reach  60%  in  2020. 
Concerning  the  first  indicator,  the  reverse  is  true,  as  figures  appear  to  fluctuate 
significantly. The 2020 yearly goal was reached in 2015, but in 2016 the number returned 
near  to  the  2012  baseline  (see  Table  6).  This  indicator  can  be  influenced  by  different 
exogenous  factors,  such  as  macroeconomic  conditions,  rate  of  development  of  new 
psychoactive substances, etc. 

Table 6: Result indicator, target and actual value 

Result indicators 

Baseline  

2014/2015/2016  Targets 

68 (2012) 

100 (2015) 

95 (by 2020) 

66 (2016) 

2020 

testing, 

through 

1. Number of new psychoactive substances 
assessed  (including 
if 
necessary) to enable the EU or the Member 
States  to  take  appropriate  action  to  protect 
consumers, depending on the type and level 
of risk that they may pose when consumed 
by humans 

2.  %  of  problem  opioid  users  that  are  in 
drug treatment 

50% (2011) 

50%  (2014, 2015 
and 2016) 

60% (by 2020) 

Moreover, a series of Programme-related indicators are also mentioned in the legal basis 
of  the  Justice  Programme32.  They  provide  a  basis  for  monitoring  and  evaluating  the 
extent  to  which  each  of  the  Programme's  specific  objectives  has  been  achieved.  These 
are:  1.  The  number  of  persons  reached  by  awareness-raising  activities  funded  by  the 
Programme; 2. The improvement in the level of knowledge of Union law and policies in 
the  groups  participating  in  activities  funded  by  the  Programme  compared  to  the  entire 
target group; 3. The number of cases, activities and outputs of cross-border cooperation, 
including  cooperation  by  means  of  information  technology  tools  and  procedures 
established  at  Union  level;  4.  Participants'  assessment  of  the  activities  in  which  they 
participated  and  of  their  (expected)  sustainability;  and  5.The  geographical  coverage  of 
the activities funded by the Programme. 

32 Art. 15 of the Regulation (EU) 1382/2013 of 17 December 2013 establishing a Justice Programme for the period 2014 to 2020. 

13 

 
 
 
 
 
                                                            
Programme indicators 1 and 3 relate to the outputs and outcomes of funded activities 
and can be calculated through the final reports of the projects. At the moment, there are 
not  sufficient  available  data  to  come  to  consolidated  conclusion  on  these  indicators. 
Programme indicators 2 and 4 relate to the perception of target groups concerning the 
activities they participated in and the level of knowledge they acquired. The indicator 4 is 
not measurable at this stage, instead for the indicator 2, no surveys on the perception of 
target  groups  were  deployed  and,  therefore,  it  cannot  be  measured.  Programme 
Indicator  5  speaks  to  the  equity  in  distribution  of  activities  funded  by  the  Programme 
amongst  beneficiaries  in  the  26  participating  Member  States.  For  projects  financed  by 
means of applications connected to the Annual Working Programmes of 2014 and 2015, 
beneficiaries from all Member States benefitted in some capacity (either as coordinators 
or  as  participants),  while  for  2016  no  beneficiaries  from  Estonia received  funding.  The 
goal  is  to  ensure  100%  coverage  through  2020 and  it  has  been  largely respected  so  far 
(Member States coverage of participating organisations in 2016 was of 96%). However, 
this indicator does not capture two essential elements connected to the equity dimensions, 
namely:  (a)  the  unfulfilled  ‘demand’  of  applications  (i.e.  ratio  of  beneficiaries  / 
applications for each Member State) and (b) the distribution of programme resources. 
In general, there is no baseline available for all these indicators, as they relate directly to 
the  specific  actions  of  the  Justice  Programme,  and  there  are  also  no  defined  targets, 
except for the geographical coverage. 

Other data, collected during the interim evaluation, show the current state of play of the 
Programme’s implementation and its key initiatives in the period 2014-2017. 

As regards the type of target groups, the Justice Programme focuses mainly on research 
institutions/universities, victims, young people and families (see Figure 3). 

Figure 3: Text mining of target groups of the Justice Programme, by Specific objective 

Source:  Evaluation  team’s  elaboration  based  on  the  text  mining  exercise  conducted  on  Grant  Application  Forms  of  the  Justice 
Programme 

In  relation  to  the  types  of  partnership,  the  data  collected  in  the  interim  evaluation 
suggest that the Programme is successful in attracting more transnational partnerships33. 
Indeed, on average  the number of participating organisation per project is equal to  six-
eight, a slight increase compared to the average of four-five observed in the predecessor 
programmes.  Within  the  Justice  Programme  network  there  are  292  organisations 
connected out of 336 organisation reported in the database, i.e. 45 are isolated, meaning 
that  only  one  organisation  out  of  five  within  the  Justice  Programme  collaboration 

33  Interim  evaluation  of  the  Justice  Programme  2014-2020,  Ernst  &  Young  Financial-Business  Advisors,  Final  Report,  April  2018, 
published. 

14 

 
 
                                                            
network  is  isolated.  Public  bodies  (mainly  judicial  training  organisations)  and  higher 
education  institutions  (which  also  provide  judicial  training)  are  particularly  well 
represented within the networks of the Justice Programme, reflecting the judicial training 
priority of the Programme. 

The  evaluation  showed  that  one  of  the  key  vectors  through  which  the  Programme 
supports  the  development  of  an  area  of  justice  in  Europe  is  by  catalysing  the 
development  of  transnational  partnerships.  Transnational  partnerships  are  essential 
contributors  to  the  effectiveness  of  the  Programme.  According  to  the  78%  of 
respondents, this has increased their capacity to implement the projects, 70% reported an 
increase in sustainability of results and 75% mentioned increased capacities. 

Moreover, these partnerships appear to be quite robust, with many interlinkages amongst 
participants34.  

In addition, 50 organisations participating in the Justice Programme are also active in the 
Rights,  Equality  and  Citizenship  Programme.  It  thus  appears  that  the  Programme  has 
facilitated the growth of durable European networks in the area of justice policy and that 
structured  networks  of  collaborations  exist  also  in  relation  to  other  EU  funding 
programmes. 

However, taking into account the involvement of partner organisations, it appears that 
the bulk of participants to the Programme come only from few Member States and, 
therefore,  a  balanced  geographical  spread  has  still  to  be  achieved  (as  was  the  case 
with the predecessor programmes). For example, about 25% of all the beneficiaries come 
from either Italy or Belgium and about half of the partner organisations come from five 
countries  and  the  remaining  50%  are  organisations  from  the  remaining  21  Member 
States.  Consequently,  this  affected  the  allocation  of  funds.  For  example,  the  largest 
share  of  available  resources  for  action  grants  is  attributed  to  Italian  organisations, 
followed by organisations in the Netherlands, France, Belgium and Germany. Instead, the 
vast  majority  of  resources  available  for  Annual  Work  Programmes  allocated  to  main 
actors via operating grants is distributed to organisations based in Belgium35.  

With regard to the kind of projects/activities funded, the two specific objectives with 
the highest number of awarded grants are the ones on "Judicial Training" and "Access to 
Justice".  Through  them,  the  Programme  mainly  financed  grants  focused  on  training 
activities,  but  also  analytical  activities  and  actions  to  support  main  actors.  Instead,  in 
relation  to  the  implementation  of  procurements,  the  Programme  finances  activities 
focused mostly on the specific objective "Judicial Cooperation", followed by the specific 
objective  on  "Access  to  Justice".  In  2014,  most  funded  activities  concerned  the 
organisation  of  conferences  and  studies,  whereas  in  2015  activities  focused  more  on 
Information Technology interventions.  
The  awarding  rate  of  calls  for  proposal  (in  2014  and  201536),  of  almost  all  specific 
objectives, ranged between around 16% and 47% . With respect to the specific objectives 
"Judicial Cooperation" and "Judicial Training", almost half of the applications submitted 
were awarded; instead, in the field of drug policies, where very few calls were launched 
in the period covered by the 2014-2016 Annual Work Programmes (5%), the demand for 
funding outstripped the supply significantly. 

34 See the results of the Social Network analysis of the networks of beneficiaries within the Programme in the Interim report, ibid, p. 
56. 
35 This is due to the fact that operating grants resources are made available to organisations representing European networks, which 
tend to be headquartered in Belgium (thanks to proximity with other EU institutions ). 
36 This information is not available for 2016 yet. 

15 

 
                                                            
In  2016,  a  drop  in  the  grant  applications  received,  coinciding  with  the  rollout  of  the 
Participant Portal37, was registered. This drop in applications notwithstanding, in general, 
the number of awarded projects remained  stable across the four specific objectives and 
even increased significantly in the case of judicial cooperation projects. The drop also led 
to an increase of the awarding rate.  

4.  METHOD 

Short description of methodology 

The  evaluation  process  was  supported  by  an  external  evaluator  (Ernst  &  Young 
Financial-Business Advisors)38, under the Commission's responsibility. The criteria used 
for the evaluation include: relevance, effectiveness, efficiency, coherence, equity, scope 
of simplification of the Programme and EU added value (see Table 7).  

Table 7: Evaluation criteria and corresponding evaluation questions 

Evaluation Criteria 

Evaluation questions 

1.  EQ1. To what extent have the objectives been achieved considering the set indicators?  

i)  What progress has been achieved so far by the Justice Programme in meeting indicators 

listed in Article 15 of the Justice Regulation  
Is the Justice Programme meeting its general objective at the Programme level? 

ii) 

2.  EQ2.  How  adequate  were  the  actions  funded  under  the  Programme  to  the  objectives  of  the 

Programme? 

3.  EQ3. What factors influenced the achievements observed?  

i)  Have  any  specific  factors  favoured  or  deterred  the  achievement  of  the  SOs  of  the 

Programme?  

ii)  Have any unintended effects influenced the effectiveness of the Programme? 

4.  EQ4:  How  have  the  eligibility  criteria  influenced  the  formation  of  partnerships  and  the  scope  of 

proposals? 

Effectiveness 

5.  EQ5.  Compared  to  the  2007-2013  six  predecessor  programmes,  how  did  the  Justice  Programme 
perform  in  terms  of  better  policy  targeting,  and  targeting  of  the  right  groups  of  beneficiaries, 
Programme management and economies of scale?  

i)  How has the consolidation of predecessor programmes influenced the management of the 
Justice  Programme  in  terms  reduction  of  administrative  burden  and  the  achievement  of 
economies of scale?  

ii)  Has  the  consolidation  of  predecessor  programmes  resulted  in  better  policy  targeting,  in 
terms  of  a  reduction  of  dispersion/proliferation  of  calls  with  similar  or  overlapping 
objectives?  

iii)  Have funded beneficiaries been able to enhance their capacity to support relevant target 

groups? 

6.  EQ6: How effective have been the  communication activities in informing the potential applicants 

about upcoming calls and in increasing the visibility of funded projects? 

7.  EQ7. Will the results of the implemented actions be sustainable in the long-term?  

i)  Do funded actions achieve results that last beyond the duration of individual projects?  
ii)  Do partnerships among beneficiaries of the Justice and REC Programme endure beyond 

participation in the individual calls? 

8.  EQ8: Which are the costs and benefits of the intervention? 
9.  EQ9: To what extent has the intervention been cost-effective? 
10.  EQ10: What factors influenced the efficiency with which the achievements observed were attained? 
11.  EQ11:  Is  there  any  scope  for  using  alternative  implementing  measures  other  than  action  grants 

and operating grants, e.g. innovative financial instruments? 

12.  EQ12: How relevant were the actions funded under the Justice and REC Programme to the needs 

of the different stakeholders? 

i)  Did  the  Programmes  address  stakeholders’  needs,  as  identified  in  the  IL  of  the 

Programmes?  

ii)  Has  the  Justice  Programme  provided  relevant  support  to  beneficiaries,  in  terms  of 

Efficiency 

Relevance 

37  See  ‘The  Justice  Programme  -  implementation  in  2016  &  2017’,  Directorate-General  for  Justice  and  Consumers,  2017.  The 
Participant  Portal  is  an  electronic  platform  used  to  manage  the  applications  received  for  calls  published  in  the  framework  of  the 
Justice  Programme.  According  to  the  evaluation  results,  the  new  Participant  Portal  is  not  adapted  to  typical  Justice  Programme 
applicants  (such  as  training  organisations),  as  the  Portal  was  initially  designed  with  research  institutions  and  voluminous  grants  in 
mind. 
38  Interim  evaluation  of  the  Justice  Programme  2014-2020,  Ernst  &  Young  Financial-Business  Advisors,  Final  Report,  April  2018, 
published. 

16 

 
 
                                                            
Evaluation Criteria 

Evaluation questions 

capacity to address the needs of target groups? 

Coherence, 
Complementarity, 
Synergies 

EU added value 

Equity 

13.  EQ13: How well do the (original) objectives (still) correspond to the needs within the EU? 
14.  EQ14: How relevant is the EU intervention to EU citizens? 
15.  EQ15: How relevant were the groups targeted by the intervention?  
16.  EQ16: To what extent is the Justice Programme coherent / complementary with other EU and/or 
national policies and funding Programmes that have similar objectives to the Union bodies' work 
(external coherence)? 

17.  EQ17: What synergies exist within the Programme and with other EU Programmes? 
18.  EQ18: To  what  extent  the  different  actions  and  interventions  lead  to  a  coherent  approach  within 

the Programme (internal coherence)? 

19.  EQ19:  To  what  extent  is  the  intervention  coherent  with  international  obligations,  including  the 

2030 Agenda for sustainable development? 

20.  EQ20: What is the additional value resulting from the EU intervention(s), compared to what could 

be achieved by Member States? 

21.  EQ21: To what extent do the issues addressed by the intervention continue to require action at EU 

level? 

22.  EQ22:  What  would  be  the  most  likely  consequences  of  limiting  the  level  of  the  existing  EU 

intervention or completely stopping/withdrawing from it?  

i)  How have policy areas relevant to the Justice Programme evolved since 2014, and how 

does this affect the added value of the Programme?  

ii)  How has the Justice Programme influenced national policies in its respective fields? 

23.  EQ23:  How  much  EU  added  value  resulted  from  national  projects  compared  to  transnational 

projects?  

i)  Did any national projects overlap with, and address the same needs as, projects funded by 

the Justice Programme? 

ii)  Did projects funded under the Justice Programme foster the development of effective and 

durable cross-border networks? 

24.  EQ24: How fairly are the different activities distributed across the different target groups and EU 

Member States? 

i)  Were resources of the Justice and Programme allocated with the differentiated needs of 

target groups from different Member States in mind? 

ii)  Were  resources  allocated  also  in  consideration  of  the  different  needs  of  beneficiaries 

across Member States? 

25.  EQ25: How has gender mainstreaming been promoted? 

i)  Was  gender  mainstreaming  (intended  as  a  cross-cutting  theme)  included  in  the 
programming  and  implementation  of  the  Justice  Programme,  including  calls  for 
proposals?  

ii)  How was the principle of gender mainstreaming applied in practice by beneficiaries? 

26.  EQ26: How have the rights of the child been promoted? 
27.  EQ27: How have the rights of people with disabilities been promoted? 

Scope for 
simplification 

28.  EQ28: How can the Programme management with focus in particular on the grant management, 
be further simplified to alleviate administrative burden of the Commission and of the applicants? 
29.  EQ29. Is there any scope for further simplification through changes in the management mode of 

the Programme? 

The  evaluation  deployed  a  series  of  quantitative  and  qualitative  tools,  which  were 
devised  to  provide  detailed  responses  to  specific  evaluation  questions  associated  with 
each  evaluation  criteria.  The  evaluation  methodology  relied  on  a  Mixed  Methods 
approach  which  integrates  and  compares  quantitative  and  qualitative  approaches,  data 
collection,  data  analysis  and  interpretation  in  order  to  strengthen  the  reliability  of  data, 
the  validity  of  the  findings  and  recommendations  by  triangulating  multiple  sources  of 
information. 

The analysis covered the call for proposals and procurement activities financed in 2014, 
2015  and  2016  Annual  Work  Programmes.  The  2017  Work  Programme  on  the  other 
hand  has  been  analysed  chiefly  in  terms  of  design  and  structure,  not  in  terms  of 
execution.  

One key method used to analyse the Programme was  desk research, which focused on 
available  documents  at  programme,  policy  and  project  level.  The  desk  research  was 

17 

 
 
complemented by fieldwork, which was leveraged to collect relevant data and input from 
stakeholders.  This  helped  to  fill  knowledge  gaps  and  validate  information  which  was 
retrieved during the course of the desk research. The following fieldwork activities were 
performed:  1.  A  web-based  survey  was  launched  to  gauge  and  compare  the  views  of 
relevant  stakeholders  (i.e.  project  applicants,  project  beneficiaries,  Programme 
Committee  Members  and  additional  relevant  stakeholders);  2.  Some  interviews  were 
conducted  with  project  beneficiaries  and  Commission  officials;  3.  Finally,  a  webinar 
was  held,  which  provided  a  chance  to  discuss  preliminary  findings  with  a  number  of 
beneficiaries.  

The  collected  data  was  then  used  to  perform  a  series  of  quantitative  and  qualitative 
analyses.  On  the  quantitative  side,  the  analysis  performed  helped  provide  perspective 
both at the Programme level, assessing for instance how and where the Programme has 
deployed the available financial resources, and at the project level, giving a granular view 
of quantitative results indicators provided by individual projects. Also, a Social Network 
Analysis  was  performed.  This  analysis  helped  achieve  a  “bird’s-eye”  view  of  the 
collaborations  and  transnational  networks  that  were  developed  as  a  result  of  projects 
financed  by  the  2014-2020  Justice  Programme.  Finally,  concerning  the  qualitative 
analysis,  automated  text  mining  techniques  reinforced  with  machine  learning 
algorithms  were  deployed  to  help  sift  through  the  large  amounts  of  data  available  on 
individual projects. This was done in light of the need to give structure and find meaning 
within  a  large  and  mostly  unstructured  dataset  (namely  project  applications  and  final 
reports),  so  as  to  create  taxonomies  of  the  information  present  in  free  text  fields. 
Crucially, text mining helped in the analysis to find commonalities and correlations in the 
data that might not been found had solely manual techniques been deployed. 

Robustness and limitations of findings 

The  evaluation’s  findings  and  conclusions  are  deemed  as  credible  and  consistent 
according  to  the  methodology.  There  were,  however,  some  challenges  which  by  their 
nature could not always be mitigated.  

First of all, only 8 projects had both monitoring and survey data available which means 
that no statistical inference has proved feasible and that the triangulation of quantitative 
data at the project level has been limited. Nevertheless, the analysis of a series of output 
indicators  allowed  to  have  a  description  of  the  actions’  features  as  well  as  progress  in 
terms of target groups reached and activities carried out.  

Moreover, another challenge was that in evaluations of EU legislation “it can be difficult 
to identify a robust counterfactual situation (i.e. what the situation would be if EU laws 
had not been adopted)”, which made purely quantitative analysis problematic39.  

In view of these limitations, a counterfactual analysis, built on qualitative data, has been 
considered  to  be  the  more  suitable  method.  The  analysis  of  direct  questions  on 
behavioural change assumes that the respondents are able to reflect on their behaviour in 
hypothetical,  counterfactual  situations  and  that  they  are  telling  the  truth  to  the  best  of 
their knowledge. However, as respondents have an interest in the continuation of public 
support,  they  might  be  tempted  to  over-emphasize  the  merits  thereof  by  answering 
strategically40.  To  address  the  positive  bias  in  answers  by  beneficiaries  and  the  self-

39 European Commission, Better Regulation Toolbox, p. 344, https://ec.europa.eu/info/sites/info/files/better-regulation-toolbox_0.pdf.  
40 In interpreting the results, it should be considered that there  may be two sources of bias to the answers: 1.  Strategic behaviour 
(beneficiaries may tend to overestimate the effect of the treatment to further legitimise their receipt of funding, while applicants may 
tend to say to be well off also in absence of funding – also to qualify for the next round); 2. Estimation bias: both beneficiaries and 
applicants  don’t  know  exactly  what  the  counterfactual  situation  would  be  like,  so  answers  are  based  on  their  educated  guess.  This 

18 

 
                                                            
selection bias in answers from applicants who have not received funding, a similar set of 
questions was asked to applicants having had their proposal rejected. As the features of 
all  respondents  (both  successful  and  unsuccessful  applicants)  are  known,  this  helped 
triangulate findings gathered from beneficiaries.  

In addition, even though the four case studies proved to be a useful tool to get a better 
picture  of  the  Programme’s  activities,  their  representativeness  of  the  Programme  as  a 
whole was somewhat limited.  

Despite  these  limitations,  the  evaluation  carried  out  is  supported  by  a  robust  evidence 
base. 

5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS 

The  Justice  Programme  is  performing  generally  well  at  midterm  with  regard  to  its 
specific 
relevance, 
coherence/complementarity/synergies  and  EU  added  value,  but  should  do  better  in 
particular in terms of equity41.  

effectiveness, 

objectives, 

efficiency, 

terms 

of 

in 

5.1 Effectiveness 

The  analysis  of  indicators  has  shown  that  significant  progress  has  been  achieved  on 
several  fronts  since  some  targets  are  close  to  being  achieved  and  others  have  already 
been  achieved.  Therefore,  the  Justice  Programme  contributes  positively  to  the 
general  objective  of  the  Programme  through  the  progress  under  its  specific 
objectives.  
The  Programme-specific  indicators  are  adequate  to  monitor  progress  towards  the 
objectives  of  the  Programme, but sometimes they  are  difficult  to  measure due  to a 
lack of adequate tools (e.g. there is no satisfaction survey to help gauge the perception of 
participants that took part in training activities) or to the influence of exogenous factors 
(such  as  macroeconomic  conditions,  rate  of  development  of  new  psychoactive 
substances).  Moreover,  some  indicators,  concerning  for  instance  the  geographical 
coverage of the Programme, do not capture certain underlying dynamics, such as the fact 
that the Programme, so far, has been dominated by beneficiaries from a small number of 
Member States. 
Overall,  the  2014-2020  Justice  Programme  is  broadly  perceived  by  applicants, 
beneficiaries  and  stakeholders  as  an  improvement  compared  to  the  three 
predecessor  programmes  (namely  the  Civil  justice  programme,  the  Criminal  justice 
programme and the Drug prevention and information programme), both in terms of better 
policy targeting and better involvement of the right stakeholder groups. A better targeting 
is  observed  in  the  sense  that  calls  tend  to  tackle  specific  issues  which  are  considered 
priorities42.  The  merging  of  predecessor  programmes  has  been  particularly  effective  in 
the field of European judicial training since this has reduced the overlaps and enhanced 
the capacity of training providers by reducing the complications associated  with having 
to deliver training across multiple programmes with different rules. 
Given  the  wide  scope  of  the  general  objective  of  the  Programme,  a  large  number  of 
external factors influenced its effectiveness, such as the migration crisis (concerning, in 
particular, the specific objective on “access to justice”) and, in general, the uneven pace 

applies especially to applicants who can base their judgment only on the knowledge they acquired in the application phase with no 
actual exposure to the implementation of the measures. 
41 For more information and evidence, see section 6 "Responses to evaluation questions" in the Interim report, p. 94 et seq. 
42  However,  it  should  be  noted  that  better targeting  makes  sense  as  long as  there is  enough  demand.  If  the  targeting  is  excessively 
specific  the  risk  is  that  of  receiving  a  small  number  of  high  quality  proposals,  with  the  risk  of  not  allocating  the  entire  available 
budget. 

19 

 
 
                                                                                                                                                                                 
at  which  Member  States  transpose  and  enforce  the  EU  acquis.  This,  however,  did  not 
undermine  its  intervention  logic  and  its  operational  flexibility:  the  annual  work 
programmes  can  be  easily  adapted  to  emerging  needs  in  the  area  of  justice  (see  more 
under “Relevance”). 
Finally,  compared  to  the  2007-2013  period,  the  sustainability  of  projects  (with  their 
results and outputs43) beyond the end of their life-cycle has become an increasingly 
important  factor  to  consider  in  the  evaluation  process.  However,  projects  focused  on 
creating tools/outputs might have a more difficult sustainability landscape since, in these 
cases, sustainability depends on whether the organisations that implemented the projects 
can receive sufficient resources to maintain the tools once the EU funding is over. 

5.2 Efficiency 

The Justice Programme has been cost-effective relative to the actions it has funded 
so far. Indeed, the results of the evaluation have shown that the beneficiaries’ perception 
of the efficiency of the Programme is positive. 60% of respondents to the relevant survey 
question consider the benefits of the intervention greater than costs. This is true for the 
Programme as a whole, but, in particular, for the specific objective on judicial training 
where more projects than expected have reported to have carried out training events and 
activities and more people than expected have been involved in the activities, both within 
the categories foreseen and beyond. The increase is quite significant also in terms of the 
types of modules of training provided (live events/remote training/tools and guidelines). 
Moreover, the average expenditure on judicial training is on average lower than expected 
for the projects identified. 
A key achievement of the Programme, compared to its predecessors, has been the lower 
burden on beneficiaries in terms of time and financial resources. Nevertheless, there 
is  still  room  for  improvement  in  terms  of  easing  requirements  and  obligations  to  make 
the  Programme  even  more  efficient  in  its  implementation  (see  more  under  “Scope  for 
simplification”).  
According to beneficiaries,  the  current instruments (action grants, operating grants 
and  procurement  activities)  are  adequate  for  the  needs  of  the  Programme  and, 
therefore,  using  alternative  and  innovative  funding  instruments  is  not  necessary. 
However, the efficiency of their implementation should continue to be improved, notably 
with regard to procurement actions, to ensure that the allocated funding is indeed used. 
Indeed, since the forward planning for procurement actions occurs, usually, many months 
before  the  action  itself,  this  leads  to  a  tendency  to  overestimate  procurement  needs  (to 
ensure that sufficient budget is allocated) and to a subsequent low implementation rate of 
the  allocated  funding.  The  efficiency  of  such  actions  must  be  increased  by  ensuring  a 
greater  degree  of  connection  between  the  planning  of  procurement  actions  and  their 
implementation. 

5.3 Relevance 

According  to  all  interviewed  beneficiaries,  the  Programme  is  highly  relevant  to 
address  the  needs  of  selected  target  groups.  Indeed,  one  of  the  key  features  of  the 
Programme  has  been  its  ability  to  adapt  and  modify  the  priorities  in  light  of  emerging 
needs.  For  instance,  following  a  string  of  terrorist  attacks  in  Europe,  two  calls  were 

43Sustainability  of  results  and  outputs,  achieved  through  the  implementation  of  activities  that  generate  increased  awareness, 
knowledge and skills and through the development  of tools that can be used also after the project ends, by beneficiaries and target 
groups alike, (including transferred to other organisations). 

20 

 
 
 
                                                            
 
issued to combat the radicalisation of inmates in prison. This can be achieved thanks to 
the unique structure of the Justice Programme, as the Programme has been devised with 
broad  specific  objectives,  due  to  the  fact  that  it  has  aggregated  the  three  programmes 
from the previous programming period. The general and specific objectives allowed the 
Commission  to  adapt  the  Programme  to  evolving  needs  within  the  EU,  especially  in 
terms  of  the  judicial  cooperation.  However,  the  specific  objective  concerning  the 
initiatives in the field of drug policies is sometimes difficult to reconcile with other 
Programme priorities, such as judicial cooperation and access to justice, as drug-related 
prevention policies tend to be broader in scope.  
In general, however, the needs identified at the time of the Programme’s adoption are 
still  actual  and  relevant,  in  particular  the  general  objective  of  further  developing  a 
European Area of Justice based on mutual recognition and mutual trust. 
With respect to stakeholder needs, there is still scope to further increase the relevance of 
the  Programme  through  systematic  analyses  per  main  stakeholder  type,  as  well  as  per 
specific  objective  and  Member  States,  so  as  to  ensure  that  priorities  related  to  each 
specific  objective  for  each  call,  as  set  out  in  the  annual  work  programmes,  are  in  line 
with the current key needs of the stakeholders. Moreover, the Programme could include 
additional  target  groups  that  are  relevant  to  the  achievement  of  its  general  objective. 
These  groups  include  staff  in  regulatory  agencies,  young  judicial  professionals  and 
students and judicial professionals in candidate countries and countries within the scope 
of the European Neighbourhood policy. This is however currently not possible under the 
legal basis of the Justice Programme.  

5.4 Coherence, Complementarity, Synergies 

The Programme presents a good level of coherence and complementarity with other 
EU instruments, programmes and actions (for instance, there is a very high coherence 
with  the  European  Agenda  for  Justice  202044)  and  the  risk  of  duplications  or 
incoherence is very low. Especially in the case of judicial training, the merging of the 
predecessor  programmes  has  increased  the  coherence  with  other  EU  initiatives  and 
among  different  training  objectives  and  reduced  the  possibility  of  duplications,  both  in 
scope  and  in  funding.  A  strong  coherence and  complementarity  in  the  access to  justice 
specific  objective  exist  with  Connecting  Europe  Facility  Telecommunications 
Programme that also contributed to the development of e-Justice Portal and e-Codex.  
Some exceptional overlaps in terms of objectives, target groups and actions still exist as a 
natural  consequence  of  the  broad  objectives  and  target  groups  covered  by  the 
Programme,  ranging  from  judicial  cooperation  in  civil  and  criminal  matters  (thus 
essentially the full spectrum of judicial activity) to drug policy and judicial training, as 
well as since it targets all EU citizens. 
In any case, there is still potential for strengthening the synergies with other EU funding 
programmes and initiatives. This is the case, for instance, of the drugs policy area, where 
the coordination with the Health for Growth Programme could be increased. 
According  to  interviewed  stakeholders,  the  coherence  with  national  policies  and 
initiatives  with  similar  objectives  and/or  targeting  the  same  areas  is  high.  The 
Programme fills the gaps left by national actions, while existing national projects and 
initiatives  complement  the  Programme  rather  than  being  in  contrast  with  it  or  a  mere 
duplication. Indeed,  when  both  the  Justice  Programme  and  national  initiatives  have  the 
same  (or  similar)  objectives  and  target  groups,  there  are  always  differences  between 

44 The objectives and addressed areas of the European Agenda for Justice for 2020 are very much in line with the Justice Programme, 
especially  in  the  area  of  judicial  cooperation,  even  if  the  Agenda  prioritises  areas  such  as  terrorism  and  cybercrime  that  are  less 
present under the Justice Programme. 

21 

 
 
                                                            
them,  in  terms  of  size  of  geographical  scope  (with  enlarged  scope  in  terms  of  target 
groups), available resources and number of projects funded. 
The  Justice  Programme  is  coherent  also  with  international  obligations,  such as  the 
UN 2030 Sustainable Development Agenda. Indeed, some general principles of the UN 
agenda can be linked to objectives and areas targeted by the Justice Programme. This is 
the case of the UN objective of ensuring peace and security, to be achieved through just 
and inclusive societies that provide equal access to justice, effective rule of law, as well 
as  transparent  and  effective  judicial  institutions.  All  these  elements  can  be  found  as 
integrative parts of the Justice Programme, both in terms of access to justice and judicial 
training.  Moreover,  the  EU  is  party  to  The  Hague  Conference  on  Private  International 
Law  and  pursues  its  international  action  in  relation  to  civil  justice  mainly  through  this 
international organisation. 

5.5 EU added value 

All  evidence  collected  confirms  the  high  added  value  of  the  Programme 
intervention,  which  is  considered  de  facto  instrumental  for  a  good  and  effective 
achievement of objectives in the area of justice.  
The results of the evaluation  show that the survey respondents  agree in saying that not 
only the funded activities would have not been possible without the EU intervention, but 
also confirmed that the same results would have not been achieved with Member States 
intervention only, in terms of creation of partnerships, realisation of outputs, funding of 
innovative actions, sustainability of results, and, especially, implementation of projects of 
transnational  size  and  scale.  Indeed,  the  EU  added  value  of  the  Justice  Programme  is 
evident, above all, in the promotion of transnational projects with a European dimension 
to tackle cross-border issues and in the provision of financial resources to fund activities 
in key areas that are not necessarily high on the agenda of Member States, due to lack of 
political will (according to stakeholders, this is particularly true for the specific objective 
on access to justice45).  
According to beneficiaries interviewed, the Programme can influence and align, at least 
to a moderate extent, national actions in the targeted areas. 
The  Programme  makes  also  beneficiaries  able  to  work  with  partners  in  other  Member 
States,  a  possibility  that  increased  their  knowledge  and  understanding  of  the  issues 
covered  by  the  Programme,  widened  their  approach  and  range  of  skills,  and  provided 
them with access to good practice examples and tools developed in other Member States. 
As already mentioned above, partnerships are a key element for the success of the Justice 
Programme and are present in the majority of projects. 
The  lack  of  national  funding  is  one  of  the  main  reasons  why  the  Programmes’  funded 
activities  would  not  have  been  possible  through  single  Member  States  action.  This  is 
particularly true for the initiatives in the field of drug policies, where national initiatives 
are  often  missing  a  crucial  transnational  dimension  that  is  pivotal  in  this  field,  and  for 
EU-level judicial  training  that,  usually,  is  not provided  at  national  level.  Moreover,  the 
Justice  Programme  ensures  the  continued  existence  of  European  networks  such  as  the 
European Judicial Training Network. 
The perceived importance of the Programme can be explained also by the high number of 
projects awarded after just three years, compared to the three predecessor programmes46 
and  the  increase  in  the  number  of  yearly  applications  received  from  potential 

45 See Interim report, ibid. 
46 The Civil Justice Programme, the Criminal Justice Programme and the Drug Prevention and Information Programme awarded 806 
projects over seven years (2007-2013). The Justice Programme awarded 418 just in the three-year period 2014-2016. 

22 

 
 
                                                            
beneficiaries  indicates  that  the  Programme  remains  a  key  source  of  funding  across  the 
EU.  
Moreover, judicial training figures more prominently in the intervention logic of the 
Justice Programme, compared to the predecessor programmes. This is an important step 
forward  for  the  EU  added  value  of  the  Programme  since  judicial  training  is  central  to 
build mutual trust, improve cooperation between judicial authorities and practitioners in 
the Member States and increase the coherence in the application of the EU legislation. 
The results of the evaluation show that the  issues and areas addressed by the Justice 
Programme  would require further action and involvement at  EU level. Indeed, the 
demand of EU action in these fields and the fact that the number of applications to the 
Programme  are  still  higher  than  the  number  of  awarded  grants  demonstrate  a  clear 
interest in the priorities addressed by the Programme.  

5.6 Equity 

The promotion of the cross-cutting priorities of gender equality, rights of child and 
rights of people with disabilities is highly significant for the Justice Programme and 
it’s enshrined in its legal basis.  In particular, the principles of gender and child rights 
mainstreaming  are  evaluated  during  the  evaluation  process  under  the  quality  of 
proposals. The issue of gender mainstreaming is specifically mentioned in the “part B” of 
the  application  form.  However,  the  evaluation  showed  that,  in  terms  of  projects 
developed,  gender  issues  and  equality  are  hardly  a  major  theme  in  most  of  them. 
Reference  to  women  and  gender  equality  are  nonetheless  found  in  five-six  projects, 
which,  still not  focusing  directly  on  the  matter,  take  into  account  such  element  in  their 
implementation.  
The Justice Programme  supports the rights  of the  child both in the programming  phase 
(design of the calls for proposals) and by means of the activities of projects selected for 
funding.  The  respect  of  the  rights  of  the  child  is  therefore  enhanced  by  the  fact  that 
organisations applying for funding (and any of their partners), which will work directly 
with  children  during  the  project  implementation,  must  provide  the  Commission  with  a 
description of  their  child  protection  policy.  Moreover,  calls  for  proposals  funded  under 
the Justice Programme contain topics that are relevant to the subject. The results of the 
evaluation  show  that  the  majority  of  respondents  agree  to  further  mainstream  and 
promote the rights of the child in the Programme. 
Concerning  the  priority  of  rights  of  people  with  disabilities,  it  seems  more  in  the 
background compared to the previous two. Indeed, among the projects funded, only one 
directly  targeted  at  people  with  disabilities,  under  the  specific  objective  access  to 
justice47. In terms of the need to further promote the rights of people with disability in the 
Programme,  stakeholders  have  quite  heterogeneous  opinions,  but  all  categories  tend  to 
think there is at least a “moderate” need to do it. 
To  understand  how  the  Programme  promotes  equity  through  the  funded  activities, 
participants'  data  broken  down  by  sex,  disability  status  or  age,  as  required  by  the 
Regulation, shall be collected. This is however not yet done. 
Finally,  as  already  mentioned,  in  the  future,  the  Programme  should  try  to  distribute  its 
resources  in a  more balanced manner across  the  different target groups of beneficiaries 
and Member States. 

5.7 Scope for simplification 

47  Project  “Enhancing  Procedural  Rights  of  Persons  with  Intellectual  and/or  Psychiatric  Impairments  in  Criminal  Proceedings: 
Exploring the Need for Actions”. 

23 

 
 
 
                                                            
The  evaluation  did  not  identify  clear  scope  for  further  simplifications  concerning 
the  management  mode  of  the  Programme.  The  current  direct  management  mode 
appears adequate given the size of the Programme.  
However,  notwithstanding  the  improvements  made  with  the  current  Programme,  both 
Commission  officials  and  beneficiaries  interviewed  expressed  critical  opinions  on  the 
implementation process, with specific regard to budget management and reporting duties. 
Among  the  reasons  provided,  beneficiaries  indicate  that  the  financial  reporting  is  too 
detailed and inflexible when compared to the ones applied within other EU Programmes 
(e.g. Horizon 2020 and Erasmus+). 
In terms of administrative burden, nearly 70% of beneficiaries and applicants  perceived 
as still burdensome the proposal drafting, the provisions of necessary administrative and 
financial  information  to  participate  in  the  calls  and  the  monitoring  and  reporting 
requirements.  In  general,  beneficiaries  agreed  that  the  application  and  reporting 
procedures were complex and long, but recognised that it was probably hard to simplify 
them further, given the need to ensure accountability of European financial resources. 
According to beneficiaries, the duration of projects financed by the Programme could be 
extended,  especially  with  regards  to  action  grants.  In  particular,  an  extension  to  three 
years was considered ideal, as it would be a timeframe that is aligned with the average 
duration  of  scientific  partnerships  between  higher  education  institutions.  Moreover, 
beneficiaries mentioned that also operating grants could be extended to cover at least two 
years  (instead  of  one)  in  order  to  reduce  the  administrative  burden,  in  terms  of 
application and reporting. However, the annual operating grants allow the Commission to 
have oversight of the activities of the funded organisations and allow for more flexibility 
to  adapt  the  work  of  beneficiaries  to  the  emerging  needs  in  their  respective  fields  of 
expertise. 
Another key issue, according to small civil society organisations, is the difficulty to find 
the  co-financing  amounts  required;  therefore,  small  non-governmental  organisations 
could be more supported by the Justice Programme48.  
Even  if,  at  first,  the  rollout  of  the  new  Participant  Portal  created  some  difficulties, 
currently stakeholders consider the submission of proposals via the Participant Portal an 
improvement,  compared  to  the  previous  IT  system,  since  the  number  of  documents 
required  for  the  eligibility  check  has  decreased  and,  therefore,  also  the  related 
administrative burden. In addition, once registered, these documents will not be required 
any  more,  only  in  case  of  changes.  However,  since  the  Participant  Portal  has  been 
designed  with  research  institutions  and  voluminous  grants  in  mind,  in  its  current  form, 
the  Portal  appears  ill-adapted  to  typical  Justice  Programme  applicants.  There  is, 
therefore, scope for some improvements in this regard.  
Finally,  monitoring  requirements  and  indicators,  both  at  Programme  and  project  level, 
could be simplified and streamlined. 

6.  CONCLUSIONS AND WAY FORWARD 

Without  the  Programme's  support,  there  is  broad  agreement  among  beneficiaries,  legal 
professionals  and  Commission  officials  that  a  lot  of  activities,  above  all  cross-border 
activities  (such  as  the  judicial  training  and  the  judicial  cooperation  in  both  civil  and 
criminal  matter)  would  not  be  provided  by  the  Member  States  and  the  existence  of 
relevant  European  networks,  such  as  the  European  Judicial  Training  Network  (a  key 
player in the field of judicial training), would be at risk. Indeed, the areas covered by the 

48 It is to be noticed that the average amount of grants has been increased to the current level following the ex-post evaluation of the 
predecessor  programmes  that,  as  already  said  above,  found  that  the  dilution  of  funds  amongst  many  small-scale  projects  had  had 
limited impact and EU dimension. 

24 

 
 
                                                            
Justice  Programme  are  often  not  prioritised  at  national  level  and  this  makes  the 
Programme unique since its positive impacts are not likely to be felt by other means or 
by  single  Member  States'  actions.  In  this  sense,  the  EU  added  value  of  the  Justice 
programme  is  undeniable  and  this  is  true  for  both  its  general  objective  and  specific 
objectives.  The  activities  financed  help  to  tackle  cross-border  issues  by  creating 
partnerships, networking opportunities  and occasions to build trust among judicial staff 
of different Member States. This fosters the acquisition of durable legal knowledge and 
competencies.    Moreover,  the  broader  scope  of  the  current  Justice  Programme  makes 
possible to identify each year the adequate priorities and to adapt them to emerging needs 
and threats. 

The Programme's interim evaluation confirms its relevance for: 

  contributing to upholding EU values such as the rule of law, the independence of the 

judiciary, the effectiveness of the justice system and its equal access to all; 

  supporting  Member  States  to  achieve  more  effective  justice  systems,  where  legal 
practitioners know how and when to apply EU acquis, where cross-border cooperation 
is thriving and the rights of victims and persons accused of crime are respected; 

  supporting  to  collect  comparative  information  on  the  efficiency,  quality  and 

independence of national justice systems and national rule of law safeguards. 

Some  new  priorities  emerged  and  improvement  could  be  made  in  specific  areas  of  the 
Programme.  In  particular,  the  basin  of  potential  recipients  of  the  actions  could  be 
expanded  to  increase  its  relevance  and  the  achievement  of  its  general  objective. 
Moreover, the geographic balance is still not good overall, since the Justice Programme 
is dominated by beneficiaries from only a small number of Member States. 

Regarding  the  efficiency  of  the  Programme,  the  administrative  burden  on  beneficiaries 
has been reduced but there is still room for improvements to make the Programme even 
more efficient in its implementation. 

The simplification carried out, thanks to the merger of the three previous programmes, 
has  produced  improvements  in  the  quality  of  the  evaluation  and  monitoring  process. 
However, the development of workable indicators, allowing for precise monitoring of the 
quality and the impact of projects, should be better taken into consideration. 

The Programme’s current structure works well by reducing the risks of duplications and 
allowing  the  Programme  to  be  sufficiently  flexible  to  adapt  to  emerging  needs,  even 
though  coordination  and  information  exchange  between  EU  programmes  and  projects 
could  be  strengthened  in  order  to  guarantee  a more  coherent  and efficient  allocation  of 
resources according to the most relevant priorities.  

At  EU  level,  the  Justice  Programme  is  coherent  and  complementary  with  other  EU 
instruments,  programmes  and  actions.  However,  in  the  future,  there  could  be  scope  for 
further centralisation in terms of Judicial Training for legal practitioners49. 

Lessons learned and way forward for the next programming period50:  

49 However, it should be noted that each Directorate-General is responsible for the enforcement of the legislation it is in charge of and 
the centralisation of training on all EU topics would require additional financial resources. 
For additional information on the conclusions, see section 7 "Conclusions" in the Interim report, p. 161 et seq. 
50 For more information, see section 8 "Recommendations" in the Interim report, p. 167 et seq. 

25 

 
 
                                                            
Effectiveness  

  The  Commission  will  consider  the  possibility  to  separate  the  specific  objective  on  drug  policy 

from the Justice Programme and move it to a dedicated fund addressing drug policies. 

Efficiency  

  The  Programme  should  increase  its  capacity  to  make  an  efficient  use  of  resources  allocated  to 
procurement  actions,  by  re-assessing  the  way  these  actions  are  planned.  For  example,  needs 
assessments  should  be  performed  to  gauge  the  amount  of  resources  required  to  finance 
procurement actions.  

Relevance 

  The  relevance  of  the  Programme  could  be  improved  by  engaging  in  a  more  systematic  manner 
with  stakeholders  concerning  the  definition  of  the  annual  work  programmes.  This  could  be 
achieved, for instance, by means of open public consultations or yearly surveys. The Commission 
will also consider the possibility to target additional groups that are currently outside the scope of 
the Programme. 

Coherence, Complementarity, Synergies 

  The  Commission  will  consider  the  possibility  to  ensure  that  projects  with  similar  objectives  are 
aware  of  each  other  and  can  build  on  their  achievements.  Commission  could  also  help  projects 
funded across specific objectives to get in touch with each other. 

EU added value 

  The Commission will consider the possibility to set up analytical activities and needs assessments 
that identify systematically the critical conditions for the successful implementation of tools and 
policies at national level before the launch of calls for projects. 

Equity 

  The  Commission  will  consider  to  instruct  the  applicants  more  concretely on  how  to  include  the 
gender mainstreaming and the promotion of the rights of the child and of people with disabilities 
in  the  different  stages  of  a  project.  The  Programme  should  also  stress  more  the  importance  of 
collecting data disaggregated by age and sex. The Commission will also consider to what extent 
the  participation  of  non-governmental  organisations  (for  which  co-financing  might  represent  an 
issue), should be supported. Actions should also be taken to foster a more equitable distribution of 
resources  among  organisations  in  different  Member  States  and  to  foster  development  of 
Framework Partnership Agreements with a broader set of organisations.  

Scope for simplification 

  The Commission will consider to provide more guidance to the applicants in the template Part B. 
Each item should be better explained and complemented with examples. In addition, the Guide for 
Evaluators could be published, to make the applicants aware of the evaluation methodology, and 
the Participant Portal should be more adapted to cater to the needs of typical Justice Programme 
applicants.  Moreover,  indicators  contained  in  the  Regulation  should  be  separated  conceptually 
into  output  and  outcome/impact  indicators,  indicate  how  the  Programme  contributes  to  the 
fulfilment of said indicators and identify the main data sources for each indicator as well as their 
availability. Finally, the monitoring system should envisage the possibility for the beneficiary to 
update the values of the indicators.  

26 

 
 
 
 
 
 
 
1.  LEAD DG, DECIDE PLANNING/CWP REFERENCES 

ANNEX 1: PROCEDURAL INFORMATION 

Lead Directorate-General: Directorate-General for Justice and Consumers (DG JUST) 

Decide planning reference:  

2.  ORGANISATION AND TIMING 

The evaluation was carried out between October 2017 and April 2018. 

An external contractor, Ernst & Young Financial-Business Advisors, was commissioned 
under the Framework Contract for evaluation, Impact Assessment and Related Services 
(JUST/2016/JCOO/FW/JU04/0178)  to  carry  out  a  interim  evaluation  of  the  Justice 
programme (2014-2020). The final report was completed in April 2018. An Inter-service 
Steering  Group  led  by  DG  JUST  was  set  up  specifically  for  this  evaluation.  The 
representatives of the following Commission services were members of the group:  

Secretariat  General,  Directorate-General 
for 
International  Cooperation  and  Development,  Legal  Service,  Directorate-General  for 
Communications Networks, Content and Technology, Directorate-General for Migration 
and  Home  Affairs,  Directorate-General 
for  Neighbourhood  and  Enlargement 
Negotiations, Directorate-General for Health and Food Safety, and European Anti-Fraud 
Office.  

for  Budget,  Directorate-General 

Chronology of the interim evaluation:  

Date 

 Meetings of Inter-service Steering Group 

January 2017 

Kick-off  meeting  of  the  mid-term  evaluation  Inter-Service 
Steering group 

December 2017 

Second  meeting  of  the  mid-term  evaluation  Inter-Service 
Steering group to discuss the interim report 

April 2018 

Third  meeting  of  the  mid-term  evaluation  Inter-Service 
Steering  group  to  discuss  the  draft  final  report  and  ask  for 
several improvements 

3.  EXCEPTIONS TO THE BETTER REGULATION GUIDELINES 

A public consultation was not carried out specifically under this evaluation, but under the 
Impact Assessment for the post-2020 Multiannual Financial Framework.  

4.  CONSULTATION OF THE RSB (IF APPLICABLE) 

N/A 

5.  EVIDENCE, SOURCES AND QUALITY 

The  evaluation  was  based  on  evidence  from  different  sources.  The  complete  set  of 
documents that were consulted for this evaluation is listed below: 

Legal basis of the Justice Programme: 

27 

 
 
 
 
  Arts. 81(1) and 82(1) TFUE; 

  Regulation (EU) No 1382/2013 of the European Parliament and of The Council of 17 

December 2013 establishing a Justice Programme for the period 2014 to 2020. 

Commission Implementing Decisions 2014, 2015, 2016: 

  Commission  Implementing  Decision  C(2014)2556  concerning  the  adoption    of  the 
work  programme  for  2014  and  the  financing  for  the  implementation  of  the  Justice 
Programme; 

  Commission  Implementing  Decision  C(2015)1997  concerning  the  adoption    of  the 
work  programme  for  2015  and  the  financing  for  the  implementation  of  the  Justice 
Programme; 

  Commission  Implementing  Decision  C(2016)1677  concerning  the  adoption    of  the 
work  programme  for  2016  and  the  financing  for  the  implementation  of  the  Justice 
Programme. 

EU documents: 

  Council  conclusions  on  the  European  Judicial  Network  in  civil  and  commercial 

matters (15349/16) on 8 and 9 December 2016; 

  Conclusions  of  the  European  Council  on  26-27  June  2014  in  the  area  of  freedom, 

security and justice (including on judicial cooperation in civil matters) 

  The European Agenda for Justice for 2020, adopted in 2014;  

  The European Agenda for Security, adopted in 2015; 

  The 2030 Sustainable Development Agenda;  

  The “Juncker Priorities” for 2015-2019; 

  Conclusions  of  the  European  Council  on  26-27  June  2014  in  the  area  of  freedom, 

security and justice; 

  Multiannual European e-Justice Action Plan 2014-2018.      

Justice Programme documentation: 

  Ex-post evaluation report of Civil Justice Programme (2007-2013) 

http://eur-lex.europa.eu/legal-content/en/TXT/?uri=CELEX:52017DC0059  

  Ex-post evaluation report of Criminal Justice Programme (2007-2013) 

http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52017DC0115  

  Ex-post  evaluation  report  of  Drugs  Prevention  and  Information  Programme  (2007-

2013) 

  The mid-term evaluation report of Civil Justice Programme, 2011 

http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2011:0351:FIN:EN:PDF  

  The mid-term evaluation report of Criminal Justice Programme, 2011 

http://ec.europa.eu/justice/funding/jpen/interim_evaluation_report_2011_en.pdf  

  The  mid-term  evaluation  report  of  Drugs  Prevention  and  Information  Programme, 

2011 

http://eur-lex.europa.eu/legal 
content/EN/TXT/PDF/?uri=CELEX:52011DC0246&qid=1467122450426&from=EN 

28 

 
  Policy-specific  reports  published  by  DG  Justice  and  Consumers,  such  as  the  annual 

report on European judicial training  
https://e-justice.europa.eu/fileDownload.do?id=5799faaa-b4ce-498c-aeb1-bba4b81d900e, 
https://e-justice.europa.eu/fileDownload.do?id=ae7d84cc-2bd0-4eb3-9c31-94b753a20bdb 

  EU Justice Scoreboard COM (2017) 167 final  

https://ec.europa.eu/info/strategy/justice-and-fundamental-rights/effective-justice/eu-
justice-scoreboard_en  

29 

 
ANNEX 2: STAKEHOLDERS CONSULTATION 

Consultations  with  the  main  stakeholders  of  the  ‘Justice  Programme’  were  conducted 
over several steps: 

  A web-based survey targeting the following stakeholder groups:  

o  projects applicants (also unsuccessful applicants )51; 

o  projects beneficiaries52;  

o  EU26 Member State Programme Committee Members53; 

o  Other relevant stakeholders involved directly by the Commission; 

This consultation was launched by the external consultant via the EU Survey tool and 
was open from 23 December 2017 until 6 March 2018. It received 132 responses54. 

  33 interviews with 12 Commission officials and 21 selected projects beneficiaries and 
representatives of target groups were performed between the 8th of January 2018 and 
the 26th  of  March 2018. Interviews had  a twofold objective: 1. Collecting additional 
information (from the survey and the desk research) to feed the analysis alongside the 
evaluation  criteria,  filling  gaps  (especially  in  terms  of  quantification  of  costs  and 
benefits) and going more in-depth with specific aspects, such as the identification of 
to  draw  conclusions  and 
best  practices;  3.  Gathering 
recommendations on how to improve the design and functioning of the Programmes.  

insight  and 

input 

Interviews consisted of semi-structured phone interviews. Questions were customised 
according to the different categories of stakeholders targeted, taking into account their 
different contribution to the evaluation questions; 

  1  webinar,  organized  in  March  2018,  to  discuss  preliminary  findings  with  few 
selected project beneficiaries. The webinar aimed at assessing at a higher level results 
achieved  by  the  Programme,  testing  and  discussing  preliminary  findings  and 
conclusions, also identifying key success factors and areas for improvement in view of 
the design of operational recommendations. It was organised using Zoom (an online 
tool); 

  A  broad  public  consultation  undertook  by  the  Commission  in  the  area  of  "values 
and  mobility"  was  available  online  in  23  official  EU  languages  for  a  mandatory 
period between 10 January 2018 and 9 March 2018. The purpose of this consultation 
was  to  collect  the  views  of  all  interested  parties  on  how  to  make  the  most  of  every 
euro of the EU budget. Consultations have taken place in the context of evaluations of 
existing EU financial programmes covering several policy areas, including the Justice 
Programme.  In  total,  the  public  consultation  received  1839  replies  from  all  over 
Europe.  The  respondents  had  experience  with  the  following  EU  programmes:  1. 
Europe  for  Citizens  Programme  and  /or  2.  Rights,  Equality  and  Citizenship 
Programme and /or 3. Creative Europe Programme and /or 4. Justice Programme. The 
results  of  this  public  consultation  have  shown  that,  according  to  the  majority  of 
respondents,  "promote  European  identity  and  common  values",  as  well  as 

51 From 16 Member States. 
52 From 16 Member States. 
53 From 8 Member States. 
54  For  more  information  on  the  outcomes  of  the  web  survey,  see  Annex  2  "Analysis  of  surveys  and  quantitative  analysis"  in  the 
Interim report, ibid, p. 172 et seq. 

30 

 
 
                                                            
"promote  rights  and  equality",  are  important  common  policy  challenges  to  be 
addressed in each of the four programmes. "Support active citizenship, democratic 
participation  in  society  and  the  rule  of  law"  and  "promote  social  inclusion  and 
fairness"  appear  to  be  important  challenges  to  be  addressed  in  the  concerned 
programmes  as  well.  Instead,  "support  innovation",  "foster  European  cultural 
diversity  and  cultural  heritage",  "promote  European  identity  and  common 
values"  are considered as  policies  that  fully  or  fairly  well  address  the  challenges  by 
half  or  more  of  the  respondents.  Around  80%  the  respondents  agree  that  these 
programmes  add  value  to  a  large  extent  or  to  a  fairly  good  extent  to  what  Member 
States  could  achieve  at  national,  regional  and/or  local  levels.  The  main  obstacles 
identified by the respondents, that could prevent the current programmes/funds from 
achieving  their  objectives,  are  very  similar  regardless  of  the  programme  concerned: 
"lack  of  budget  of  the  programmes  to  satisfy  demand",  "insufficient  support 
provided  to  small-scale  stakeholders"  and  "lack  of  support  to  first-time 
applicants" are identified as the main three obstacles. Finally, the respondents agree 
that  "the  use  of  more  simplified  application  forms",  "facilitating  structured 
network  and  partnerships",  "facilitating  funding  for  actions  cutting  across  the 
sectors  of  action",  as  well  as  "better  coordination  between  different 
programmes/funds",  are  the  main  steps  to  be  taken  to  simplify  and  reduce  the 
administrative burdens for beneficiaries. 

31 

 
 
 
 
 
 
ANNEX 3: METHODS 

The following methodological tools have been used:   

1. Data collection55; 

  desk research of relevant documents (at programme, policy and project level); 

  field  research  relied  on  a  web-based  survey  with  relevant  stakeholders,  i.e. 
(unsuccessful)  project  applicants,  project  beneficiaries,  Programme  Committee 
members, Commission staff  and additional relevant stakeholders, 33 interviews with 
21  selected  project  beneficiaries  and  representatives  of  target  groups  and  12  with 
Commission  officials  and  one  webinar  to  test  and  discuss  preliminary  findings  and 
conclusions with few selected project beneficiaries. 

  four case studies of projects connected to the specific objectives56; 

  analysis of the results of a broad public consultation undertaken by the Commission 
between 10 January and 8 March 2018 in the framework of the proposal for the post 
2020  Multiannual  Financial  Framework  in  the  area  of  "European  values  and 
mobility".  

2. Data analysis: 

  quantitative data analysis57:  in  addition to  monitoring and survey data,  descriptive 
statistics was used to analyse the effectiveness, efficiency and EU added value of the 
Programme; self-reported counterfactual analysis have been used for the analysis of 
its  effectiveness  and  added  value58;  exploratory  data  analysis  and  econometric 
modelling were used to analyse its effectiveness and EU added value59; cost-benefit 
analysis  to  analyse  its  efficiency60  and  Social  Network  Analysis  to  analyse  its 
effectiveness and equity61; 

  qualitative data analysis62: a "text mining" methodology was used63. 

Based  on  this  methodology,  output  indicators  (as  monitoring  data  at  project  level)  and 
results indicators (based on Eurobarometer and other sources) were used. The analysis of 
the performance at the project level has mainly relied on the answers to the survey.  

55 For more information, see section 4.1.1 "Data collection tools and activities" in the Interim report, p. 49 et seq. 
56 For more information, see the Annexe 4 of the Interim report, ibid. 
57 For more information, see section 4.2.1 "Quantitative data analysis" in the Interim report, p. 51 et seq. 
58 A counterfactual analysis is a tool that can be used to distil changes that are caused by a specific policy intervention from changes 
that would have occurred even in absence of the given intervention (see Interim report, ibid). 
59  Different  tools  of  econometric  analysis  have  been  employed  in  order  to  examine  the  results  of  the  self-reported  counterfactual 
questions, as well as to test the responses of different stakeholders to the main question of the survey linked to effectiveness/relevance 
(see Interim report, ibid). 
60  The  Cost-Benefit  analysis  is  as  an  economic  tool  applied  to  public  decision−making  to  quantify  the  advantages  (in  terms  of 
benefits) and disadvantages (in terms of costs) associated with a particular project or policy (see Interim report, ibid). 
61  The  Social  Network  analysis  enabled  to  better  understand  the  explicit  and  implicit  links  and  interrelations  among  the  different 
beneficiaries and hence the value of the partnership approach that is a distinguishing feature of the (see Interim report, ibid). 
62 For more information, see section 4.2.2 "Qualitative data analysis through text mining" in the Interim report, p. 58 et seq. 
63  Text  mining  is  an  analytical  methodology  that  has  the  aim  of  structuring  large  amounts  of  unstructured  texts  into  categories  or 
clusters of information by broad themes (e.g. project objectives, weaknesses) (see Interim report, ibid). 

32