EUROPEAN COMMISSION Brussels, 22.3.2022 SWD(2022) 64 final COMMISSION STAFF WORKING DOCUMENT



EVALUATION

Accompanying the document



REPORT OF THE EUROEAN COMMISSION TO THE EUROEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS assessing the implementation and achievements of the 2014-2020 justice programme {COM(2022) 121 final} - {SWD(2022) 69 final}



Table of contents 1.Introduction Purpose and scope 2.Background to the intervention Description of the intervention and its objectives Budget Baseline and points of comparison 3.Implementation 4.Method Short description of methodology Limitations and robustness of findings 5.Analysis and answers to the evaluation questions Effectiveness Efficiency Relevance Coherence, Complementarity and Synergies EU added value Equity Scope for simplification

Conclusions Annex 3: Methods and analytical models Glossary Term or acronym Meaning or definition DG JUST Directorate-General for Justice and Consumers EU European Union AG Action grant OG Operating grant HES Higher secondary education OTH Other non-profit organisation PRC Private company REC Research organisation PUB Public Institution AWP Annual work programme JCOO Judicial Cooperation JTRA Judicial Training JACC Access to Justice JDRU Drug prevention policy MMF Multi Annual Financial Framework 1.Introduction Purpose and scope This staff working document accompanies the report from the European Commission to the European Parliament, the Council, the Economic and Social Committee and the Committee of the Regions in accordance with the obligation specified in Article 14.2(c) of the legal base of the justice programme. The Regulation states that the European Commission shall provide the European Parliament and the Council with an ex-post evaluation report of the programme by 31 December 2021. The report shall assess the long-term impact and the sustainability of the effects of the programme, with a view to informing a decision on a subsequent programme. In 2021, however, a considerable number of projects, around 30 %, are still ongoing and many completed ones have just produced the first outputs. This situation is also exacerbated by the COVID-19 pandemic, which has resulted in the timeframe several projects being extended. For these reasons, and in order to have a meaningful evaluation of the long-term results or impacts of the programme, the ex post evaluation will be carried out in two parts. This staff working document accompanying the report to the European Parliament, the Council, the Economic and Social Committee and the Committee of the Regions represents the first part of the evaluation. It is based on the currently available data, provides an overview of the funding distribution and assesses the achievements of the justice programme so far. While the new justice programme 2021-2027 was already adopted, based amongst others on the results of the interim evaluation of the justice programme 2014-2020, this first part of the ex-post evaluation of the previous programme will be of added value in order to shed light on potential areas for improvement in the implementation phase of the new justice programme. The second part of the evaluation will be carried out at a later stage, once all final data will be available and in concomitance with the interim evaluation of the succeeding justice programme (2021-2027). This second part will assess the long-term impacts and sustainability of the effects of the justice programme and will provide recommendations for the new MFF after 2027, as appropriate. This two-step approach will enable the Commission to conduct an in-depth evaluation of the entire programming period, while being fully transparent on the rationale for splitting the reports. Against this backdrop, the first part, covering available data from 2014 to 2020 will provide a good overview of the funding distribution and achievements of the programmes so far. By contrast, the second report will provide further insights into the effectiveness and long-term impacts of the programme. In terms of geographical coverage, the justice programme is open to all EU Member States with the exception of Denmark and the United Kingdom. In addition to the EU Member States, it is open to the European Free Trade Association States party to the European Economic Area, candidate countries, potential candidates and countries acceding to the EU, provided that they conclude an agreement with the Union laying down the details of their respective participation in the programme. Albania joined the programme in 2017 and Montenegro in 2018. 2.Background to the intervention Description of the intervention and its objectives In accordance with the Treaty on the Functioning of the European Union, which provides for the creation of a European area of freedom, security and justice based on mutual recognition of judicial decisions and mutual trust among Member States, the justice programme was set up to overcome obstacles to the functioning of an effective European area of justice. The justice programme was also designed to encourage national judicial systems to build trust in each other’s standards of fairness and justice, as this is a prerequisite to achieve a European area of justice. More specifically, the justice programme as set out in in Article 4 of its legal base is to contribute to the development of a European area of justice:

(a)facilitating and supporting judicial cooperation in civil and criminal matters; (b)supporting and promoting judicial training, including language training on legal terminology, with a view to fostering a common legal and judicial culture; (c)facilitating effective access to justice for all, including promoting and supporting the rights of victims of crime, while respecting the rights of the defence; and (d)supporting initiatives in the field of drug policy as regards judicial cooperation and crime prevention aspects closely linked to the general objective of the programme, insofar as they are not covered by the Internal Security Fund (ISF) for financial support for police cooperation, preventing and combating crime, and crisis management or by the Health for Growth Programme. This vision to create a European area of justice is to enable everyone in the EU to fully enjoy their full right to the freedom of movement, the respect for their fundamental rights and respect for common principles, such as non-discrimination, gender equality, access to justice for all, the rule of law and well-functioning independent judicial systems. The justice programme also has links, potential synergies, and is complementary to other relevant initiatives of the European Union (EU) and other international organisations, such as the United Nations (UN). These include: (a)The European Agenda for Justice for 2020, adopted in 2014; (b)The European Agenda for Security, adopted in 2015; (c)The 2030 Sustainable Development Agenda; (d)The Charter of Fundamental Rights, which came into effect in 2000, and is linked to the Convention for the Protection of Human Rights and Fundamental Freedoms; (e)The “Juncker Priorities” for 2015-2019; (f)The Von der Leyen's Commission priorities for 2019-2024. These documents and initiatives have similar objectives, particularly regarding issues of effectiveness of judicial remedies, judicial cooperation, and drug prevention policy. Description of the programme

With the aim of meeting the above specific objectives of the justice programme, the following paragraphs outline the programme’s main building blocks. Judicial cooperation in civil and criminal matters To achieve its specific objective to facilitate and support judicial cooperation in civil and criminal matters, the justice programme offers financial support for activities that promote the effective and coherent application of the EU law by building-up and improving data collection and statistics on the application of EU law. The programme fosters judicial cooperation in civil and criminal matters by funding activities that contribute to the better enforcement of EU law and judicial decisions, in particular resulting from cross-border disputes. This includes, Directive 2011/99 on the European Protection Order and Framework Decision 2002/584/JHA on the European arrest warrant in criminal matters. Regulation (EC) No 861/2007 establishing a European small claims procedure and Regulation (EU) No 1215/2012 on jurisdiction and the recognition and enforcement of judgments in civil and commercial matters are also relevant pieces of EU law in civil matters. EU legislation on judicial cooperation in civil matters also covers multiple aspects of family law (parental responsibility, parental child abduction, maintenance, matrimonial property rights and succession, to give a few examples). The justice programme contributes by funding improvements to the exchange of information among professionals in order to boost operational cooperation and mutual trust in the EU. Under this specific objective, the justice programme facilitates the EU’s participation in The Hague Conference of Private International Law and action at the national level to set up and strengthen networks in the framework of the European Judicial Network in civil and commercial matters. Last but not least, this specific objective encompasses calls for proposals, launched in the wake of terrorist attacks in the EU, to prevent radicalisation in prisons leading to terrorism and violent extremism. In this context, the European Commission cooperates closely with the Council of Europe on two specific initiatives: the SPACE report on prison statistics and on setting up the EU network of prison monitoring bodies in the Member States.



Judicial training Under the specific objective to facilitate and support judicial training, activities funded by the justice programme cover the training of professionals such as judges, prosecutors, court staff, bailiffs, lawyers, notaries, prison staff, probation officers, mediators, and legal interpreters. The topics of trainings courses are wide and varied, including EU civil and criminal law, fundamental rights, judicial ethics and the rule of law. Activities funded under this specific objective are also designed to provide language training on legal terminology to foster a common legal and judicial culture. In addition to measures that aim to create a shared understanding of legal terminology, the justice programme provides funding for seminars on specific aspects of civil and criminal law, e-learning and exchanges of staff and experience. For example, it covers specialised training in the field of competition law, which typically requires a significant amount of specialisation. Support for judicial training provided by the justice programme is strongly linked to the European judicial training strategy, which was adopted in 2011. In conjunction with this training strategy, the European Judicial Training Network is a key structure funded by the justice programme to reach more practitioners from across the Member States and to promote trust in the judicial systems of different Member States. The European Judicial Training Network is an essential structure for judicial training. It is also referenced in Article 6.2 of the legal base of the justice programme as a recipient of an operating grant to co-finance expenditure on its permanent work programme. In summary, by providing the training opportunities outlined above, the justice programme aims to help create a common legal and judicial culture and foster mutual trust of legal practitioners in the judicial systems of EU Member States. Access to justice for all The remit of the specific objective to facilitate effective access to justice for all, is divided in two sub- objectives. One key topic is promoting and supporting the rights of victims of crime. The other is procedural rights, which ensure respect for the right of defence. To this end, the aim of the projects financed is to provide EU citizens with effective remedies if EU law is breached. In particular, where national procedures prove too complex. In other words, the measures are designed to promote the use of remedies and non-remedies developed in the EU that can provide quick, efficient and less costly solutions to disputes, for example through the e-Justice portal. Access to justice is also an enabling right for victims of crime to be able to enforce their rights and seek redress. When the Treaty of Lisbon came into effect, a specific reference to access to justice was also made in the Treaty on the Functioning of the EU, which states that ’the Union shall facilitate access to justice, in particular through the principle of mutual recognition of judicial and extrajudicial decisions in civil matters’. Initiatives in the field of drug policy This specific objective covers the promotion of initiatives in the field of drug prevention policy, which mainly focus on judicial cooperation and crime prevention. Thus, the main aim of activities funded by the justice programme is to foster the practical application of drug-related research. Activities under this objective are also designed to help civil society organisations and key stakeholders expand the knowledge base and develop innovative methods to address the phenomenon of new psychoactive substances. These aims are also pursued by the European agenda on security. Under this agenda, the EU should continue to support action taken by the Member States to counter both the demand and the supply of illicit drugs, including prevention work. In the context of drug prevention policy, through its focus on judicial cooperation and crime prevention, the justice programme complements other EU initiatives such as the Internal Security Fund, which tackles the sale, transport, import, and export of illegal drugs. The justice programme complements the health for growth programme, which focuses on actions to reduce the health-related damage caused by drugs. Budget The initial total budget for the justice programme for 2014-2020 was EUR 377 604 000. The justice programme has three main funding mechanisms: action grants, operating grants and procurement. ·action grants are mainly provided to civil society organisations, Member State authorities and universities; ·operating grants (support to networks) fund mainly European networks active in the following areas: judicial training as set out in the programme’s legal base, judicial cooperation and access to justice; ·procurement actions (Commission initiatives) fund mostly conferences, seminars, studies, surveys, awareness-raising activities, but also specific IT projects and the EU’s membership in The Hague Conference of Private International Law. In order to achieve its objectives, the justice programme can support a wide range of activities. In particular, under Article 6 of the Regulation establishing the justice programme for the period 2014 to 2020, the programme can finance: ·analytical activities, in particular: the collection of data and statistics; the development of common methods and, where appropriate, indicators or benchmarks; studies, research, analyses and surveys; evaluations; drafting and publishing guides, reports and educational material; workshops, seminars, expert meetings and conferences.

·mutual learning, cooperation, awareness raising and dissemination activities, in particular: identifying, and exchanging concerning good practice, innovative approaches and experiences; organising of peer reviews and mutual learning; organising of conferences, seminars, information campaigns; compilation and publishing of materials to publicise about the justice programme and its results; developing, operating and maintaining systems and tools, using information and communication technologies, including developing of the European e-justice portal as a tool to improve public access to justice. ·training, for instance staff exchanges, workshops, seminars, train-the-trainer events, language training on legal terminology, and the development of online training tools or other training modules for members of the judiciary and judicial staff. Measures to support the main actors that implement the programme’s objectives, such as: support for Member States in implementing EU law and policies; support for key European actors and European-level networks, including in the field of judicial training; support for networking at EU level among specialised bodies and entities as well as national, regional and local authorities and non-governmental organisations. The stakeholders eligible for support can be either public or private organisations (usually non-profit-oriented), established in one of the countries participating in the programme, or international organisations (Council of Europe). In terms of target groups, defined as the groups that can receive support directly or indirectly from the justice programme, potentially all people in the EU are included. This broad target group is based on the overarching objective of the justice programme to create a European area of justice, where everybody is aware of and can exercise their rights. More specifically, the programme covers support for judicial practitioners, public authorities, universities, international organisations, non-governmental organisations and other research groups; companies going through insolvency or pre-insolvency proceedings, citizens accused or victims of a crime, families, consumers, minors, groups in need of specialised legal protection and people at risk of radicalisation. The figure overleaf summarises the intervention logic of the justice programme by illustrating causal mechanisms to an effective area of justice. Baseline and points of comparison The 2011 impact assessment used as part of the interim evaluation of the justice programme serves as a baseline and a point of comparison. The document outlines a ‘baseline scenario’ against which the policy options for the justice programme were benchmarked. The document also describes a series of expected outcomes, connected to the development of the justice programme. The baseline analysis showed that, if no changes had been made between the 2007-13 and the 2014-20 programming periods, the programme would have continued to be successfully implemented, but it would not have reached its full potential. In this context, it is worth underlining that before its launch in 2014, the justice programme was preceded by three different funding instruments, which each covered the thematic areas now covered by one programme. Before launching the justice programme, the Commission identified the following issues as potentially critical: ·the lack of flexibility in the funding instruments would have been an obstacle, given the pace of change and reform in the area of justice; ·the fragmentation of funding, especially where funding is needed to support horizontal and cross-cutting issues, could have reduced the capacity of the predecessor programmes to deliver results; ·having multiple funding programmes with different rules and procedures, in the context of rising numbers of applicants, could have increased the administrative burden, potentially resulting in delays in procedures for the beneficiaries. Given these bottlenecks and difficulties, the criminal justice programme, the civil justice programme and the drug prevention and information programme were merged into the justice programme. The 2011 impact assessment used in the interim evaluation also assessed the effect of the consolidation in terms of relevance and scope, effectiveness, complementarity, EU added value, efficiency and potential for simplification. ·In terms of the relevance and scope of the consolidated programme, the impact assessment found that funding would be available for all of the areas and activities covered by predecessor programmes. In addition, the expectation was that it would reduce fragmentation and overlaps, as policies with close links to each other, such as civil and criminal justice, were streamlined under the same programme. The aim of streamlining predecessor programmes into the justice programme was also to reduce the potential for overlap with other funding instruments. In terms of flexibility, the broader scope of the justice programme was designed to progressively integrate new actions that better support justice policies as the policy environment continuously evolves. ·In terms of effectiveness, the new comprehensive justice programme was expected to promote the concentration of funds and the development of stronger partnerships. The applicants would coordinate their action in broader partnerships with more funding available. It was also expected to lead to a more balanced geographical spread among beneficiaries as a result of larger partnerships. ·In terms of efficiency, the expectation was that the justice programme would bring about improvements by simplifying the funding procedures for applicants and beneficiaries. In particular, merging the predecessor programmes under a broader justice programme aimed to put an end to the practice of beneficiaries oscillating between the previous three funding programmes based on their individual fit to them. Last but not least, the aim was to improve efficiency by reducing the number of calls for proposals, which also became more thematically focused. ·In terms of complementarity, it was expected that merging the three predecessor programmes would ensure a coordinated approach to identifying annual priorities. This approach had to be applied also vis-à-vis programmes in other EU policy areas with links to justice and rights. Coordination during the annual programming process would make sure that funding is not duplicated and that funds are used in a complementary way, in the best public interest.

·With a view to reducing the administrative burden, reducing the three annual work programmes to one and reducing the number of calls for proposals were expected to yield considerable improvements. Harmonising the management procedures would lighten the administrative burden for applicants and beneficiaries. It would also enable the Commission to halve the number of staff members assigned to managing calls for proposals. In addition to the findings of the impact assessment of the interim evaluation, all ex-post evaluation reports on the previous funding programmes confirmed that overall they were effective and highlighted that their specific objectives and priorities were on the whole specific, attainable and realistic, but not always measurable. In particular, it identified the following problems common to the three programmes: 1. the funds were diluted and spread across many small-scale projects with limited impact and EU dimension; 2. no balanced geographical spread among the organisations, receiving funding;



3. complex and bureaucratic procedures for applicants; 4. high administrative burden on the European Commission and longer procedures due to the multiplication of procedures for the different programmes. In summary, the baseline and point of comparison of the justice programme, is characterised by a fragmented funding landscape, which put heavy administrative burden and complex procedures in place. As a result, the Commission took measures to make EU funding more accessible and to promote more targeted action to reach its policy goals. 3.Implementation Programme management The European Commission implements the justice programme via direct management. This approach was taken in order to ensure a close relationship between the programme management and EU policy-making. This management mode also enables the European Commission to tailor funded activities to policy priorities and policy needs and thereby to directly target specific groups of stakeholders. It also facilitates close contacts with the programme's beneficiaries and results in better knowledge of the needs on the ground. State of play This section presents the state of play of the programme’s implementation on the basis of the information available at the time of writing. As explained in the introduction, a significant number of projects, around 30%, are still ongoing, and therefore the ex post evaluation will be carried out in two parts. The first part, based on the currently available data, is prepared in 2021 and assesses the achievements of the justice programme so far. The second part of the evaluation will be carried out at a later stage, once all final data will be available and in concomitance with the interim evaluation of the succeeding justice programme (2021-2027). This second part will assess the long-term impacts and sustainability of the effects of the justice programme and will provide recommendations for the new MFF after 2027, as appropriate. Annual work programmes The annual work programmes of the justice programme set out goals to pursue, by funding the measures. In other words, the annual work programmes outline how the specific objectives of the justice programmes are to be met through targeted activities. The annual work programmes will also seek to adapt to emerging needs so that beneficiaries can adapt to any changes in the policy environment of the justice programme. In 2014, the first annual work programme was overall balanced. The specific objectives of judicial cooperation, access to justice and judicial training received approximately the same level of funding. The beneficiary identified by the legal base of the justice programme, the European Judicial Training Network, received an operating grant of EUR 7 880 000 the highest amount of funding in that year. Action grants for work to meet the specific objective on judicial training had a rather broad focus on family and succession matters, as well as on criminal law. Calls for proposals on access to justice for all covered the expansion of the e-justice portal and work on procedural and victims’ rights. Action grants under the specific objective on judicial cooperation in civil and criminal matters focused on family matters such as divorce, legal separation, parental custody, and succession. Last but not least, the drafting of new legislation on new psychoactive substances and support to implement the EU drug strategy were financed through procurement and action grants in the field of drug prevention policy.

The 2015 annual work programme underscored the importance of the specific objective of access to justice for all. In total there were six calls for proposals focusing on this specific objective. In parallel, action grants promoting the specific objective on judicial training were aligned with the topics pursued in 2014 under the specific objectives of judicial cooperation and access to justice. Action grants for judicial training focused on training judges on legal instruments in family matters and successions. Other training courses were focused on international debt recovery in civil matters and commercial matters. On criminal matters, training sessions focused on procedural rights in criminal proceedings. In addition, judicial training was promoted through the operating grant to the European Judicial Training Network with a budget of EUR 8 800 000. The European Judicial Training Network supported implementation of EU legislation in commercial matters, for example by working on the Commission recommendation for a new approach to business failure and insolvency. The Network also supported action on criminal matters to implement Framework Decision 2002/584/JHA on the European Arrest Warrant. In 2016, the budget reserved for funding the work of the European Judicial Training Network grew to EUR 9 500 000, underlining the importance of the network. The 2016 annual work programme also attributed major importance to the specific objective of access to justice for all through procurement, which funded the evaluation of existing criminal justice instruments and the preparation of possible new tools. For the specific objective on judicial cooperation, funding supported the management, analytical work and information activities of the European Judicial Network in civil and commercial matters. Procurement funding supported the maintenance and upgrading of the European Criminal Records Information System. The specific objective on judicial training was a salient aspect of the annual work programme in 2017. The funding for the European Judicial Training Network grew and amounted to EUR 10 200 000. With a view to judicial cooperation, family matters, succession and commercial law enforcement remained key topics in civil matters. On criminal matters, the focus was on implementation and practical application of mutual recognition instruments, such as the European Arrest Warrant, the transfer of prisoners and the European Supervision Order. For the specific objective on drug prevention policy, the focus remained on identifying new psychoactive substances. In 2018, the European Judicial Network in Civil and Commercial Matters and the European Judicial Training Network were high on the agenda of the annual work programme. Support for the European Judicial Network in Civil and Commercial Matters was designed to improve implementation and case-handling under civil justice legislation. Training courses tackled gaps in cross-border training for court staff, bailiffs and prison staff. Another priority on the training front was to facilitate access to training and instructing training providers. By contrast, there was a reduction in funding for the specific objective on access to justice for all and for judicial cooperation, except for support for the European Judicial Network in Civil and Commercial Matters. In 2019, there were continued reductions in the budget allocated to the specific objectives of judicial cooperation and access to justice. However, activities continued to promote cooperation between national networks in the field of civil justice under the umbrella of the European Judicial Network in Civil and Commercial Matters. The 2019 annual work programme provided funding to ensure that Member States create national networks to boost judicial cooperation. In addition, the funding was designed to consolidate existing structures. On judicial training, the level of funding remained relatively stable. The continued high importance of judicial training can be partially attributed to the fact that judicial training became a crucial part of the European Commission’s work programme, under the priorities ’An Area of Justice and Fundamental Rights based on Mutual Trust” and “a Union of democratic change’. Judicial training also claimed a central role with the European e-justice portal expanded by rolling out the European Training Platform. In 2020, the last annual work programme under this programming period, there was a particular focus on digital technologies and their impact on the goals of the justice programme. In the area of judicial cooperation in criminal matters, the focus was on access to electronic evidence as a means to counter criminal activities. Under access to justice, funding was targeted to promote new initiatives to implement the European e-justice strategy for 2019-2023. As in the previous year, maintaining and upgrading of the e-justice portal remained an important topic. This included creating synergies between the work on e-justice and the Connecting Europe Facility programme with the aim of contributing to the creation of the digital single market. Digital technologies also influenced work under the specific objective of judicial training. The potential impact of frontier technologies such as artificial intelligence and machine learning on law enforcement, amplified the importance of judicial training. Therefore, the focus was on training judicial staff on the use of such technologies and on tools to counter digital threats. Technological developments also affected implementation of the programme in the field of drug prevention policy, in particular in combating the online drugs trade. The developments described above are reflected in the tables below, which show the budget for each annual work programme and provide an overview of funding across the specific objectives for the entire programming period. Budget Year Amount planned in the annual work programme € Annual change % 2014 € 45 812 000 - 2015 € 48 051 000 4,9% 2016 € 52 250 000 8,7% 2017 € 52 631 000 0,7% 2018 € 45 949 000 -12,7% 2019 € 43 675 000 -4,9% 2020 € 45 603 000 4,4% Total € 333 971 000 Source: annual work programmes 2014-2020 The overall budget increased steadily up to 2017, when it reached a peak of EUR 52.631.000. It fell afterwards to reach approximately the same level as in 2014. The budget decreased as funds of the justice programme were reallocated to the Civil Protection Mechanism (CPM - COM(2017)772) and to provide eu-LISA with the necessary financial resources to develop the European Criminal Records System for third country nationals.

Specific

Objective 2014 2015 2016 2017 2018 2019 2020 Planned Planned Planned Planned Planned Planned Planned Judicial Cooperation 14 228 14 415 14 570 15 789 12 000 11 600 11 661 Judicial Training 14 825 16 515 16 900 18 048 17 870 17 570 17 570 Access to Justice 13 775 13 296 16 268 15 662 13 330 11 630 13 373 Initiatives in the field of drugs policy 3 004 3 000 2 512 3 123 2 749 2 875 2 999 Total 45 812 48 051 52 250 52 631 45 949 43 675 45 603 Source: annual work programmes 2014-2020 The distribution of funds across specific objectives shows that every year, the specific objective on judicial training received a large share of the financial resources available. In total, 36,12% of the budget or (EUR 119 908 000) was allocated to judicial training. Access to justice was the second most funded specific objective, with EUR 99 364 000 and 29,94% of the budget. This was closely followed by judicial cooperation, with 28,45% of the budget, (EUR 94 418 000). Initiatives in the field of drugs policy had a significantly lower budget. Only 6,11 % of the budget, (EUR 20 281 000) was earmarked for this specific objective. The distribution of funds is set out in the allocation of funds to specific objectives in the Annex to the Regulation 1382 establishing the justice programme 2014-2020. Specific objective Action grants Operating grants Procurement Other Amount planned € Share of total Judicial Cooperation € 49 089 600 €9 258 400 € 35 833 000 € 237000,00 € 94 418 000 28,45% Access to Justice € 48 454 400 € 14 473 170 € 36 436 430 - € 99 364 000 29,94% Judicial Training € 44 493 000 € 69 380 000 € 6 035 000 - € 119 908 000 36,12% Drugs €16 781 000 - € 3 500 000 - € 20 281 000 6,11% Total € 158 818 000 € 93 111 570 € 818 044 30 237 000 € 333 971 000 100% Source: annual work programmes 2014-2020 Calls for proposals In the period covered by the 2014-2020 annual work programmes, 118 calls for proposals were launched for all types of funding under the justice programme: action grants, operating grants and procurement. Concerning the specific objectives of the justice programme, a large share of the calls for proposals (40 in total) were to support judicial cooperation in civil and criminal matters. The second highest number of calls (36 in total), were to support the specific objective on effective access to justice for all, including promoting and supporting the rights of victims of crime, and respecting the rights of defence. The third highest number of calls were for the specific objective to facilitate and support judicial training (26 calls for proposals). Finally, 16 calls for proposals were run to support initiatives in the field of drug prevention policy. The distribution of calls over the entire programming period shows that the yearly number of calls peaked in the initial years of the programme, before dropping as a result of the merge in 2016 to manage the justice programme more efficiently. Specific objective Year Judicial cooperation Access to justice Judicial training Drugs policy Total 2014 7 8 3 2 20 2015 5 6 4 5 20 2016 7 5 4 1 17 2017 6 5 4 2 17 2018 6 4 4 2 16 2019 4 4 3 2 13 2020 5 4 4 2 15 Total 40 36 26 16 118 Source: annual work programmes 2014-2020

Action grants and operating grants The table overleaf presents the number of projects per specific objective financed between 2014 and 2020. A large share of action and operating grants is allocated to the specific objective on access to justice for all. The second largest share of projects provided funding to advance judicial training. Third in terms of number of projects is judicial cooperation in civil and criminal matters. The first three specific objectives have an equal allocation in terms of the projects related to them. By contrast, with only 32 projects over the entire programming period, the fewest projects were financed under the specific objective of drug prevention policy. Table 5: Number of funded projects per year (action grants and operating grants) SOs 2014 2015 2016 2017 2018 2019 2020 TOT Judicial Cooperation 28 16 29 29 25 16 19 162 Access to Justice 40 33 28 32 22 12 14 181 Judicial Training 37 32 32 30 25 22 16 194 Drugs 7 6 5 5 4 5 - 32 Total. 112 87 94 96 76 55 49 569 Source: Data for 2014-2015 are retrieved from the interim evaluation of the justice programme. The projects funded for 2016-2020 were reconstructed using the project database In terms of the type of activities, implemented through action and operating grants, the figure below depicts the distribution over the entire programming period. Source: Data for 2014-2015 are retrieved from the interim evaluation of the justice programme. The projects funded for 2016-2020 were reconstructed using the project database. A large share of projects implemented were analytical and monitoring activities, followed by awareness-raising, information and dissemination and training activities. Procurement Over the whole programming period 2014-2020, a total of EUR 81,7 million was earmarked for procurement in the 2014-2020 annual work programmes. The largest share of the budget for procurement was allocated to the specific objective on judicial training (36%), followed by access to justice (30%), judicial cooperation (28%) and initiatives in the field of drug prevention policy (6%). SOs 2014 2015 2016 2017 2018 2019 2020 Judicial Cooperation 32 36 22 26 40 24 21 Access to Justice 1 2 1 1 2 - - Judicial Training 28 23 19 31 32 45 28 Drugs 2 1 - - - - - Total. 63 62 42 58 74 69 49 Information communication technology tools and other IT-related activities were the most common types of procurement activities contracted over 2014-2020. These IT-related activities were followed up by mutual learning initiatives. By contrast, training activities were the least most common activity contracted, as shown in Figure 3.

Source: Detail on procurement data provided by DG JUST. Although training accounted for the fewest in absolute number of projects, their average size was the largest in terms of the budget spent per procurement contract. By contrast, mutual learning activities, which rank second highest in absolute numbers, had the lowest average budget. Source: Detail on procurement data provided by DG JUST. Target groups In terms of the target groups reached per specific objective through funding under the justice programme, text mining of project documents (such as final reports of funded projects) yielded the result shown in the figures below.



Figure 5: Target groups per specific objective (2014-2016 left; 2016-2020 right) The figure below summarises the distribution of funding to key groups of beneficiaries. Source: Interim evaluation of the justice programme (for 2014-2015), detail on project monitoring data provided by DG JUST (for 2016-2020). According to the data, there is rather strong prevalence of higher secondary education institutions (HES) and research organisations (REC) between 2014 and 2018. Only in 2015, public institutions (PUB), non-profit organisations (OTH) and private companies (PRC) participated more frequently in funding under the justice programme. Although higher secondary education institutions and research organisations still had a relatively high share of the budget until 2018, the amount distributed to them decreased steadily as of 2017. Beginning in 2018, public institutions, non-profit organisations and private companies together received over 50% of the available budget. Also as of 2018, the share of the budget attributed to public institutions and non-profit organisations rose to, reach over 60 % of funds in 2020. Distribution of funds and applications

In terms of geographical distribution of resources, as for predecessor programmes, participation in the justice programme was concentrated in a number of Member States (e.g. Italy, Belgium, Spain, Germany, the Netherlands and France). Organisations from these Member States have consistently remained the lead beneficiaries, despite changes in the numbers of applications over the years. In contrast, there is a relatively limited demand for support from countries like Latvia, Poland, Croatia, Slovakia and Cyprus. In fact, when looking at the selected project beneficiaries, it emerges that not all Member States participated in an equal way in the programme: about 22% of all beneficiaries come from either Italy or Belgium. In absolute terms, Spain (157), the Netherlands (120), France (113) and Germany (110) are the other countries with over 100 organisations selected for grants under the justice programme. The figure below summarises the described geographical allocation of funds. Figure 7 : Number of applications, and grants (action and operating) awarded by country (2014-2020) Source: Data for 2014-2015 are taken from the interim evaluation of the justice programme. The projects funded for 2016-2020 are taken from the project database. Population data Eurostat Similar to the actual distribution of funds, the same EU countries that received most funds, are also the most frequent applicants. Looking at the whole period, Italy (889 applicants, 17.8%) and Spain (503, 10%) had the most applicant organisations, comprising combined over 25% of all applicants. Belgium (295), France (274), Germany (284) and Romania (272) also had over 250 organisations applying to the justice programme between 2014 and 2020. The overleaf figure provides an overview of the number of the applications and funds awarded over the programming period. Source: Data for 2014-2015 are taken from the interim evaluation of the justice programme. Data on the projects funded for 2016-2020 are taken the project database. Overall, the number of applications fell over the course of the programming period to reach the lowest point in 2019 with 143 applications. The number of applications started increasing again in 2020. The fall in applications resulted in a similar reduction in terms of grants awarded to projects. The number of selected projects fell from 112 in 2014 to 48 in 2020, the lowest number over the whole programming period. Progress in indicators In order to measure progress against the baseline situation, the Commission created a series of programme indicators for the justice programme 2014-2020. This section presents, the results available for each indicator, against the 2020 target, the baseline, the milestone and the actual results achieved. The results are either based on the analysis carried out or on official sources, including information reported in the draft general budget of the EU for the 2020 and 2021 financial years. Indicator tracking progress on the general objective The Regulation establishing the justice programme does not stipulate an indicator to track its general objective. However, the European Commission’s programme statements, which accompany the draft general budget of the EU, refer to the percentage of legal practitioners trained as the main indicator for measuring the justice programme’s impact. The indicator is defined as: the cumulative number of legal professionals receiving training, not only through the justice programme, on EU law or law of another Member State in the area of civil justice, criminal justice and fundamental rights. Target 700 000 legal practitioners will have received training by 2020. Baseline 239 000 legal practitioners were trained in 2013. Results By 2014, 371 000 practitioners had received training. In 2016, the cumulative number of trainees had increased to approximately 640 000. According to the 2020 programme statement, 820 199 legal practitioners were trained by 2017, of which 180 000 practitioners had been trained in 2017 alone. In 2018, the cumulative number of trained legal practitioners surpassed one million. Overall assessment The target has been reached ahead of schedule. Result indicators measuring progress towards the specific objectives

For the specific objective to support and facilitate judicial cooperation in civil and criminal matters, the justice programme's performance is measured by tracking two main indicators: the average time of the surrender procedure under the European Arrest Warrant in cases where the person consents to the surrender, and the number of exchanges of information in the European Criminal Records Information System. Target The average time is equal to 10 days by 2020. Baseline According to DG Justice’s analysis of Member States reports to the Council, the average time was 14-20 days in 2013. Results In 2014, the average time was 19,4 days, falling to 16 days by 2015. In 2017, the average time fell further to 15 days but increased again to 16,4 days in 2018. For subsequent years, no data are available. Overall assessment Two main factors impede the accuracy of this result indicator. First, data are difficult to collect. Second, data that is available is susceptible to influence by external factors, such as the different degree of incorporation of EU directives at national level, as well as the different degree of judicial system reforms within EU Member States that have a significant impact on the level of enforcement of EU law. In turn, these factors make it very difficult to estimate to which extent the justice programme contributes to a change in the average number of days. On the basis of the current data, the target for this result indicator was not reached. Target The number of exchanges of information in the European Criminal Records Information System is equal to 3 500 000. Baseline In 2012, 300 000 exchanges were recorded. Results In 2014, there were 1 250 000 exchanges, increasing to 1 978 000 in 2016. This increasing trend continued with 2 963 789 exchanges in 2018 and 3 700 000 in 2019. Overall assessment The number of exchanges rapidly increased after 2015, reaching and surpassing the target ahead of schedule in 2019. For the specific objective to facilitate and support judicial training, one main result indicator measures the justice programme’s performance: the number and percentage of members of the judiciary and judicial staff that participated in training activities, staff exchanges, study visits, workshops and seminars funded by the justice programme. The number of participants is measured yearly. Target The number of participants per annum is equal to 16 000 for the justice programme. Baseline In 2011, the number of yearly participants was 6 681. Results The number of participants almost doubled in comparison to the baseline to reach 12 333 in 2014. An increase in number of participants was also recorded the following year with 16 723 in 2015. However, the number of participants fell in 2016 to 13 930. In 2018, the number of participants increased again to reach 18 358. This positive development did not endure as, according to the latest data, the number of participants fell again to 10 799 in 2020. Overall assessment The results presented should be interpreted taking into account the following limitations. Although the indicator is well aligned to the specific objective of judicial training, the data collected directly from projects funded through the justice programme are not exhaustive and do not give a detailed picture of the number and percentage. Though the results should account for issues of data unavailability, the justice programme only met its goal in 2015 and 2018. For the specific objective to facilitate effective access to justice for all, including to promote and support the rights of victims of crime, while respecting the rights of the defence, performance of the justice programme is measured by two main result indicators. One is the number of hits on the e-justice portal to track the justice programme’s performance. The other is the number of victim support organisations with national coverage to track implementation of Directive 2012/29/EU. Target The number of hits increases by 20% per annum. Baseline In 2012, the total number of hits was 630 000. Results Over the years, the number of hits on the e-justice portal was:

1 136 849 hits in 2014, 1 751 180 hits in 2015, 1 884 600 hits in 2016, 2 690 574 hits in 2017, 2 962 558 hits in 2018 and 4 343 547 hits in 2019. This corresponds to an increase of over 20 % every year, except in 2016 when it rose by 8 % and by only 10% in 2018. Overall assessment The target was for the hits on the e-justice portal to rise by 20% each year, therefore to reach 2 708 885 by 2020. This target was achieved ahead of schedule in 2018. Target By 2020, there is at least one victim support organisation, which fulfils the quality standards and indicators set by Member States or developed individually by the organisation, in each Member State. There were a total of 27 victim support organisations in 2020. Baseline In 2012, there were in total 10 victim support organisations. Results From the baseline in 2012, the number of victim support organisations began to increase. In 2014, it doubled to reach 20. One year later, it rose to 22 and remained at that level in 2015 and 2016. In 2017, the number of victim support organisations reached the benchmark of 27 and remained at that level until 2019. Overall assessment The target number of victim support organisations was reached ahead of the deadline already in 2017.



Last but not least, the performance of the justice programme in terms of achieving the specific objective to support initiatives in the field of drug prevention policy is measured by two main result indicators. The first is the number of new psychoactive substances assessed (including through testing, if necessary) to enable the EU or the Member States to take appropriate action to protect consumers, depending on the type and level of risk that they may pose when consumed by humans. The second indicator is the percentage of problem opioid users in drug treatment. Target By 2020, the number of new psychoactive substances assessed is equal to 95. Baseline In 2012, the number of new psychoactive substances assessed was 68. Results Having peaked at 100 newly assessed psychoactive substances in 2015, the number of new psychoactive substances decreased significantly over the course of the years. In 2016, there were 66, down to 52 in 2017, 53 in 2018 and 53 in 2019 new psychoactive substances assessed. Overall assessment The target was not achieved by 2019. Although data for 2020 is still missing, the constant decrease in the number of new psychoactive substances assessed strongly suggests that the target was not reached in 2020 either. However, this result should be interpreted with caution, as the decrease in new psychoactive substances is not necessarily a negative result. The reporting and assessment of fewer substances may suggest that fewer illicit substances are available on the market. However, the 2021 programme statement reported how the substances assessed in recent years are more dangerous than those assessed before. In other words, when the target was set, more substances were reported and detected. The situation has since changed, with a steady fall in the number of substances reported.

Target By 2020, the percentage of problem opioid users in drug treatment is at a level of 60%. Baseline In 2011, 50% of problem opioid users were in drug treatment. Results Between 2014 and 2017 the percentage remained steady at 50%, before decreasing slightly in 2018 to 48%. In 2019, the percentage increased again to its baseline level of 50%. Overall assessment The target was not reached over the entire period for which data are available. Similar to result indicator one, the steady level of problem opioid users in drug treatment strongly suggests that the target was not met in 2020 either. 4.Method Short description of methodology The evaluation process was supported by an external study by Fondazione Giacomo Brodolini Srl SB and Ernst & Young. The criteria used for the evaluation include: relevance, effectiveness, efficiency, coherence, equity, scope of simplification of the programme and EU added value. The methodological approach was aligned with the questions for the evaluation criteria. Annex 3 provides a comprehensive overview of the evaluation criteria and the evaluation questions. In order to answer these evaluation questions, the external study applied a mixed methods approach. This means the evaluation integrates and compares quantitative and qualitative approaches, data collection, data analysis and interpretation. The data collected was then used to perform a series of quantitative and qualitative analyses. Quantitative analyses helped provide insights, both at programme level and project level. At programme level, quantitative methods assess for example how and where the programme used the funding available. At project level, quantitative analyses took a granular view of the results of individual projects. To carry out the qualitative analysis, automated text mining techniques reinforced with machine learning algorithms were used to interpret efficiently the large volumes of data on individual projects. The study chose this mixed methods approach to strengthen the reliability of data and the findings and recommendations. To be specific, the study triangulated multiple sources of information, to broaden and deepen the understanding of the processes through which the justice programme achieves its results. The external study also scrutinised how the environment, in which the justice programme is implemented, affects the results it produces. The methodological approach taken by the study is divided into three main tasks: ·Task one: in-depth desk research, including the extraction, collection and analysis of monitoring information. The aim is to provide an overview of the state of play, the performance and main achievements of the justice programme. ·Task two: two main activities, one is, field research, including interviews with beneficiaries, stakeholders and European Commission officials, was conducted. The field research also encompassed focus groups, an e-survey to programme committee members and an additional documentary review. The other activity was to analyse a sample of projects and services funded by the justice programme. Both activities aim to provide more nuanced and clearer insights into key features and achievements of a limited set of projects. At project level, the following aspects were scrutinised: type of action, link with the specific objectives, outcomes and results and their contribution to the key evaluation criteria. ·Task three: drafting case studies and answering the evaluation questions. Based on the information and findings obtained from the two steps above and their triangulation, the study launched eight case studies, focusing on a set of operational and procedural aspects covering the justice programme. Building on the findings from the case studies, the study answered the evaluation questions and made suggestions for improvement. Annex 3 provides an overview of the case study topics. Annex 3 also provides, an overview of the documents, reviewed at project level to conduct the desk research for task one and two. The field research related to tasks two and three focused on the following stakeholders: ·beneficiaries of funding provided by the programme, including national, regional and local authorities in all participating countries, international organisations, EU-level umbrella organisations, NGOs, academic and research institutions, networks and representatives of target groups. In some instances, potential beneficiaries were also consulted. ·Commission staff, such as programme managers and policy officers; ·the justice programme committee members representing EU Member States. The study chose a semi-structured design for the interviews, which were conducted via video conferencing. The questions were tailored to the category of stakeholders, taking into account their different contribution to the evaluation questions. As a first step, five scoping interviews with DG JUST officials were organised to obtain more in-depth data. In addition, and to supplement the answers to the evaluation questions, a total of 28 interviews was conducted. These 28 interviews were organised with 28 project beneficiaries, covering both action and operation grants and procurement contracts. Furthermore, four focus groups were organised to deepen and validate the findings from the interviews. Three focus groups were held with a sub-sample of project beneficiaries and one with DG Justice project officers. The topics of the focus groups with beneficiaries were: ·Focus group one was on case study one: Recurring beneficiaries and continuations of projects.

·Focus group two was on case study three: why are some institutional bodies reluctant to apply? Group two also provided insights into case study seven: are the beneficiaries selected the best fit to help the target groups? ·Focus group three was on case study 8: how has gender mainstreaming been promoted within the funded intervention and what where the strengths of your approach? The focus group with project officers aimed to collect additional and complementary information, focusing on management and procedural aspect of project implementation. The study also included two online surveys to complete the field research. First, the programme committee members e-survey. The goal of the survey was to collect comprehensive and specific information on the application process, the main features and results of projects and activities funded by the justice programme, and to capture insights and expert views. The survey results enabled the study to gain an understanding of the performance of the justice programme against the evaluation criteria. In total 21 responses were collected, covering 17 countries (AL, BG, CY, EE, FI, GR, HR, IT, LT, LU, MT, PT, RO, SE, SI, SK). In addition to the programme committee members e-survey, the study included a second survey targeting 200 project beneficiaries to answer the question raised in case study four: how do EU projects compare in terms of costs and benefits to similar actions funded by other sources? The second survey elicited 11 responses. Limitations and robustness of findings While overall the methodology enabled the study to draw feasible conclusions to the evaluation questions, conducting the evaluation study presented some challenges that resulted in some gaps and limitations. These limitations were: 1.The coverage, completeness and consistency of the data used to monitor the justice programme. In particular, the variety of sources of data was a challenge, as a result of the change in the project information base and the migration to e-grants in the middle of the programming period. This led to differences in the completeness and consistency of the data used to monitor the justice programme. The quality and coverage of data prior to 2016 was also uneven at times. 2.The lack of a fully-fledged monitoring framework, complete with clearly defined and measurable monitoring indicators, benchmarks and targets against which to assess the performance of the programme. 3.The limited response rate to the e-surveys conducted for the data collection work. 4.Data extraction proved to be problematic in some instances. To be specific, the study encountered challenges when using automatic data extraction techniques to extract semi- structured information, such as monitoring information reported in the annexes to the technical reports. Due to differences in format, the feasibility of using readily available information needs to be put into perspective. To address these limitations and challenges, the study took the following measures: 1.To address gaps and inconsistencies in the monitoring data, the study used additional sources of data. The study compared to this end different data sources and merged them into one database. In particular, ongoing interviews with coordinators of sampled projects and European Commission officials proved useful to plug the data gaps. Furthermore, the procurement sector of DG JUST was contacted to obtain new and more comprehensive data.

2.To remedy the difficulties, related to the monitoring indicators, the indicators for the study followed, where possible, the approach by the interim evaluation. The analysis of current indicators built on official and approved figures from 2018 with additional manual data cleaning carried out to increase the coverage of data before 2016.

3.To remedy the low response rates in surveys, the response time was extended. 5.Analysis and answers to the evaluation questions This section outlines the results of the analysis against the seven evaluation criteria: effectiveness; efficiency; relevance; coherence, complementarity and synergies; EU added value; equity and scope for simplification. Effectiveness The evidence collected suggests that the justice programme has overall been effective in terms of both the type of activities funded, and the corresponding project outcomes. The physical and financial implementation of the programme has been satisfactory, despite visible differences between specific objectives. In terms of the indicators defined in the legal base of the justice programme, the analysis confirmed that several targets have been achieved, some even ahead of schedule. This is the case for the following indicators:

I.cumulative number of legal professionals receiving training on EU law or law of another Member State; II.number of exchanges of information in the European Criminal Records Information System; III.number of hits on the e-justice portal; IV.number of victim support organisations with national coverage. By contrast, the goal defined for the indicator tracking the average time of the surrender procedure under the European Arrest Warrant was not reached. Although the justice programme promotes implementation of the European Arrest Warrant, it should be taken into account that the EU Member States are not legally obliged to report on the average time of the surrender procedure. Moreover, identifying the average time of the surrender procedure may depend, in some cases, on additional requests for information at country level. These are external factors outside the scope of the justice programme. The targets set by indicators tracking the specific objective on drug prevention policy were also not met (the number of new psychoactive substances assessed and the percentage of problem opioid users that are in drug treatment). Similar to the surrender procedure, the number of new psychoactive substances assessed and the share of opioid users undergoing treatment can vary from year to year based on several variables that do not depend on the justice programme. These variables include the macroeconomic conditions and the rate of development of new psychoactive substances. For the other indicators, which do not have a baseline for comparison, the analysis did not reach a concrete assessment as the targets or benchmarks are not defined. In a similar vein, the lack of high quality data rendered the evaluation of some indicators rather difficult. This was the case especially for indicators with a qualitative emphasis such as the level of knowledge of Union law and policies in the groups participating in activities funded by the programme compared to the entire target group and the participants' assessment of the activities in which they participated and of their (expected) sustainability. It is also worth emphasising that a number of indicators described for the specific objectives are not necessarily programme-specific as they are reported in the general budget of the European Union. Achieving these objectives can depend on multiple external factors, including the COVID-19 pandemic, which was reported as the most influential external factor. In fact, the pandemic required re-designing or cancelling some activities entirely. The uneven pace at which Member States transpose and enforce EU law and the refugee crisis in relation to the specific objective access to justice for all are also important external factors that had an impact on the effectiveness of the justice programme. The refugee crisis had a negative impact on the effectiveness of the justice programme, as it led EU Member States to divert their priorities from the programme’s specific objectives to addressing the influx of refugees and the consequent judicial obligations. In addition to these external factors, internal factors were also identified as having an impact on the justice programme’s effectiveness. These factors are predominantly linked to consortia’s capacity to comply with the standards of the application process to participate successfully in the justice programme. A key factor here is the discrepancy between applicants’ capacity to present a well written proposal and their capacity to deliver the results they had described in their proposals. This discrepancy can be explained by the technical expertise needed to comply with the standards set in the application process. Another important internal factor is the level of the co-funding rate under the justice programme. Given programme’s general objective to establish a European area of justice for all, it should be taken into consideration that the co-funding represents a financial obstacle to small and grassroots organisations. Organisations operating at this level lack the financial resources, needed to participate in the programme. However, the interviewees welcomed the European Commission’s decision to increase the EU contribution to action grants up to 90% of total project costs.

In line with the results for the indicators, the programme committee members surveyed also confirmed the effectiveness of the programme. The majority viewed the justice programme’s activities as highly effective and sustainable. 80% of respondents (16) answered that it performed in terms of meeting its objectives ‘to a high extent’ and 15% (3) ‘to a very high extent’. One respondent answered: ‘I do not know’. The 28 consulted project beneficiaries seconded that view and reported that their projects indeed did achieve or are achieving the expected results. In particular, activities carried out under the specific objective of judicial training were perceived very positively. This statement is supported by the allocation of financial resources. The analysis of technical reports of funded projects in conjunction with the interviews results showed that there is a suitable allocation of funds to projects to obtain specific objectives, with the exception of the budget for drug prevention policy. On this specific objective, a mismatch between the funding available and the scope of prevention measures should be put into perspective. It is worth noting that the justice programme focuses primarily on aspects of judicial cooperation for this specific objective and thereby complements other funding instruments, such as the Internal Security Fund or the health for growth programme.

In terms of the type of activities, the evidence contained in the project documents and data from the interim evaluation show that projects across specific objectives plan different types of activities, and not just those related to their core objective. In fact, the majority of projects analysed between 2016 and 2020 (85%, or 264 projects) included more than one type of activity. Although the presence of multiple activities within the same project is not an indication of effectiveness per se, it suggests that projects covered multiple types of activities in order to meet the objectives and needs of the justice programme from different angles. By addressing project-related issues through different activities, the potential effectiveness of interventions is increased as this minimises the risk that projects omit critical aspects. A key factor here is the flexibility of the justice programme, which gives scope to tailor project activities to emerging needs. The analysis indicates that projects funded are generally fit for purpose. The survey of applicants shows that both, successful and unsuccessful applicants view the European Commission’s communication on funding activities positively. Beneficiaries reported that the quality of communication activities was sufficient, clear, intelligible and targeted to the right audience. By contrast, the evaluation of the effectiveness of communication activities through funded projects is more difficult as data available lack a baseline for comparison and therefore do not allow to draw robust conclusions. Efficiency The majority of beneficiaries, over 60% of consulted beneficiaries, agreed that the justice programme is efficient as the benefits gained by participating in the programme are higher than the costs incurred. However, stakeholders also underlined that it is very demanding to obtain the expertise and administrative capacity, required to be able to receive funding under the justice programme. The COVID-19 pandemic is an important external factor that had an impact on the justice programme’s efficiency. The obstacles the pandemic created rendered implementation of funded projects difficult as activities had to be re-designed or interrupted. In turn, interviewees stated that the cost-benefit ratio of measures worsened. What is more, in comparison with alternative sources of funding, both national and international, the justice programme is less accessible due to the requirements for applicants. This finding should be put into perspective since the European Commission took action in 2014 to increase the programme’s efficiency by streamlining the three predecessor funding instruments into one set of application rules and procedures. Streamlining the application rules and procedures was received as a welcome change in the management of the justice programme as confirmed by the interim evaluation. The new funding rules brought in with the justice programme also facilitated a more efficient use of the available budget as excess resources allocated to a specific objective could be rerouted to another objective with a higher spending capacity. As a result, resources linked to underperforming calls for proposals, thus calls with low quality applications, would be reallocated to calls for proposals with a higher number of quality applications. Although the outlined measures improved the use of the available budget, the justice programme still suffers from underspending on projects. Within a sample of 116 projects completed between 2015 and 2018, 102 projects (88%) requested a final EU contribution lower than the amount initially requested for a total of EUR 5 495213,21 over the 4 years. Underspending is due to financial needs being overstated when budgeting, making savings over the course of the project implementation or in some cases by partners leaving the consortium.

Another issue that emerged from the analysis, is the duration and structure of the programme management cycle. Before 2018, irregularities in the timing of the adoption of annual work programmes, which resulted in significant delays in the publication and launch of calls for proposals, made it very difficult for applicants to plan and prepare to participate in the justice programme. This is especially true for organisations that operate on a programme-based approach. However, the European Commission made considerable efforts to speed up the project management cycle, which led to a higher regularity and predictability in the adoption of annual work programmes and the subsequent launch of calls for proposals as of 2018. The accessibility of information on calls for proposals and on the justice programme as a whole has also improved, which facilitated planning for applicants. Despite visible improvements in the Commission’s management of the justice programme, some applicants, as stated by five beneficiaries, still find the time it takes to submit project proposals too ambitious and the time between submission and selection of proposals too long. In particular, these interviewees underscored that the long waiting time to potentially receive funding puts the relevance of their project proposals at risk. In this context, the framework partnership agreements brought in with the launch of the justice programme in 2014, are an important financial instrument that should make project planning more efficient. The importance and advantage of framework partnership agreements are illustrated by the yearly pre-financing that beneficiaries receive as confirmed by two interviewed operating grant beneficiaries. By having a secure yearly contribution from the European Commission, recipients of operating grants can plan their projects more efficiently. However, submitting a yearly application under the umbrella of the framework partnership agreement is a loss in efficiency for the justice programme, as part of the resources of an operating grant is used to refinance itself. In addition, the yearly application puts a considerable administrative burden on the beneficiaries of the operating grants and also on the European Commission. In conclusion, a high share of beneficiaries believes that, despite the remaining obstacles, the balance between costs and benefits under the justice programme is better than under alternative sources of funding. In other words, responses from beneficiaries indicate that the comparatively higher costs of participating in the justice programme are outweighed by even greater benefits. Relevance Overall, the evaluation confirmed that the justice programme has remained highly relevant and aligned with the needs of selected target groups over the years. One of the key features of the programme has been the ability to adapt and adjust its priorities in light of emerging needs. For instance, following a string of terrorist attacks in Europe, two calls for proposals were issued to combat the radicalisation of inmates in prisons. The programme achieves this flexibility thanks to its unique structure, which encompasses the scope of its three predecessor programmes. The general and specific objectives allowed the European Commission to adapt the justice programme to evolving needs within the EU, especially in terms of judicial cooperation. In general, the needs identified at the time of the justice programme’s adoption are still current and relevant, in particular the general objective of further developing a European area of justice based on mutual recognition and mutual trust. While there is insufficient data to be able to draw a general conclusion on the funded activities’ alignment with stakeholder needs, evidence collected through exchanges with stakeholders strongly suggest that the needs covered by the programme remain relevant to stakeholders and are likely to remain relevant in the near future. This finding from the interviews is also confirmed by the analysis of the 23 sampled action and operating grants, which also indicates that project objectives and activities are considered still relevant to the needs of the stakeholders targeted. A reason for the continued relevance mentioned by project coordinators of the sampled projects is that the partnerships forged by participating in the justice programme add value in terms of networking, mutual learning, and paving the way for future collaboration. All these factors result in an increased capacity of stakeholders. However, stakeholders also identified the need for intensified exchanges between them and the European Commission to ensure that the justice programme is more responsive and attuned to stakeholders’ invariably changing needs. To this end, regular needs assessments could feed into annual work programmes. The project coordinators consulted also stressed that the continuation of one funded project through multiple projects on the same topic is in itself evidence of the programme’s relevance to stakeholders. In turn, the analysis showed that the target groups that received funding under the justice programme are relevant to reach programme’s general objective. Specifically, the analysis of the 23 sampled projects shows that projects mostly involve professionals, including judges, judicial staff, lawyers, notaries and court staff. Although efforts have been made to expand the target groups of projects to include, for instance, prison staff, there is still room for improvement to include even more relevant target groups. A key area for improvement here is tapping into the broader community of civil society organisations. They have become more involved in the justice programme but there is scope to increase their involvement, especially in the area of victims’ rights. Coherence, Complementarity and Synergies The justice programme has a good level of coherence and complementarity with other EU funds, programmes and initiatives. Especially in the case of judicial training, the decision to merging of the predecessor programmes into the justice programme has increased the coherence with other EU initiatives, such as the rights, equality and citizenship programme. By the same token, this fosters the coherence between different training objectives, which reduced the likelihood of duplications in scope and in funding. There is a strong coherence and complementarity of the specific objective on access to justice for all with the Connecting Europe Facility telecommunications programme, which also contributed to the development of e-justice portal and e-Codex. Complementarity with national projects and coherence with international obligations is also high. Complementarity with national projects is achieved by the transnational design of activities funded under the justice programme, which ensures that activities funded by the justice programme do not replace or replicate national activities. The justice programme is also coherent with international obligations, such as the UN 2030 Sustainable Development Agenda. Some general principles of the UN agenda can be linked to objectives and areas targeted by the justice programme. For example, the UN objective of ensuring peace and security is linked to the creation of just and inclusive societies that provide equal access to justice, effective rule of law, as well as transparent and effective judicial institutions. The EU is also party to The Hague Conference on Private International Law and pursues its international action in relation to civil justice mainly through this international organisation. As a result, the justice programme promotes the harmonisation of international private law. By contrast, there are some overlaps in the justice programme at the project level as a natural consequence of its broad objectives and target groups, but these do not produce duplications or incoherence. The risk of duplication is minimised by the structure of the justice programme and the formulation of its general and specific objectives. These are flexible enough to tailor funding to emerging needs, but also sufficiently distinct to ensure that specific objectives do not overlap. Last but not least, 64% of 33 surveyed beneficiaries indicated that they found and capitalised on synergies of the justice programme with projects funded by other EU initiatives, such as the rights, equality and citizenship programme, as well as with projects under the justice programme. However, there could be more exchanges between different beneficiaries to increase synergies. Thus, in summary, the current analysis confirms the findings from the interim evaluation. The justice programme achieves a high level of coherence, complementarity and synergies, both internal and external. EU added value

The analysis fully confirms the high added value of the justice programme. This finding is inter alia based on the interview feedback from eight project beneficiaries and four European Commission officials. In particular, EU added value is evident through the promotion of transnational projects to tackle cross-border issues. EU added value obtained through the justice programme is also very high in the way it increases coherence and uniformity in the application of EU law, which is obtained through training and information exchanges provided by funded projects. High EU added value is created by the provision of crucial financial resources to fund activities in key areas that are not necessarily high on the agenda of individual Member States. In other words, individual actions by Member States would not be able to achieve the same results in terms of size and scope as the justice programme at EU level as underlined by 53 % of the 21 surveyed programme committee members. The unique value of the justice programme consists on this aspect in the fact that it would be difficult for Member States to allocate an equal amount of resources to projects in the areas covered by the justice programme. Against this backdrop, the conclusions drawn in the interim evaluation of the justice programme remain valid: limiting existing EU action in this area would have negative effects on the overall capacity of both, national and European actors, to address issues in the areas covered by the programme or to implement the types of activities funded by the programme. Hence, the most likely consequences of reducing or withdrawing EU involvement under the justice programme would be significant difficulties in the development of transnational projects. These would lead to less coherence across Member States in terms of understanding and implementation of EU law. In addition, a sharply reduced number of judicial practitioners participating in networking and transnational training activities with colleagues from other Member States would be expected. As a result, this would undermine the adoption of innovative legislative frameworks or cutting-edge approaches to tackle the needs of vulnerable groups and to develop new tools and practices. In other words, it would be difficult to achieve the objectives of an EU-wide strategy. Equity In line with the results from the interim evaluation, it is clear that resources and support of the justice programme are not distributed equally across the different target groups, across the EU Member States or across the types of beneficiaries. On the macro level, there is a contrast between high application and funding rates from countries like Belgium, Italy, Spain, Germany, the Netherlands and France and comparatively low application and funding rates from countries like Latvia, Poland, Croatia, Slovakia and Cyprus. The analysis of three interviews with beneficiaries from these less represented countries yielded that the lower application and participation rates can be attributed to the limited demand for support and the lower organisational and financial capacity of the organisations, especially civil society organisations. What is more, the level of specificity of the justice programme’s annual work programmes and calls for proposals may be too specific for some applicants. Two interviewed beneficiaries from Sweden and Finland suggested that the limited participation from organisations based in their countries can be attributed to easier access to abundant national funding tied to a lower administrative burden, and to issues encountered in the recognition of their comparatively higher salary levels. Concerning the participation rate of organisations based in Belgium, it should be noted that the high concentration of resources is mainly allocated to umbrella organisations, which mostly have headquarters in Brussels for proximity to the EU institutions. Recent interviews with beneficiaries found that the role of network organisations is seen positively as they support and enable the participation of less experienced organisations in calls for proposals for action grants. Hence, the funding directed to these organisations is not entirely implemented in Belgium. The distribution of the main target groups per specific objective has evolved over the programming period. For example, the number of target groups has increased to 12 according to the project documents available from 2016 to 2020 in comparison to 10 in 2014-2016. Also, in contrast to the period 2014-2016, the justice programme primarily involved legal professionals and was more focused on reaching citizens in general between 2016 and 2020. Before 2016, the justice programme rather involved researchers. This change reflects the adjustment made by the European Commission to prioritise funding that has a direct impact on practitioners over funding for research that might not reach the intended audience, as discussed in interviews with EU officials. However, as the monitoring and data collection approaches of the justice programme need to be improved, it is very likely that the target groups reached through funding are not as diverse as they could be. This argument is also supported by the finding on the potential room for improvement to step up dialogue with stakeholders. More dialogue would help identify potential blind spots in the portfolio of target groups. This finding has already been acted upon as it is explicitly mentioned in Regulation 2021/693 establishing the justice programme for 2021-2027, that civil society organisations should be more involved as a target group by interventions, especially in the field of victims’ rights. The themes of gender mainstreaming, promoting the rights of the child and promoting the rights of people with disabilities, were rightly treated as cross-cutting topics in the funded projects. On gender mainstreaming, the importance of gender equality and non-discrimination to build a true European area of justice is already included in the preamble as well as in Article 14 of the justice programme Regulation. In the programming documents, the principle of equality between women and men is constantly referred to in the annual work programmes between 2014 and 2018. Gender mainstreaming is mentioned and framed in the annual work programmes of 2019 and 2020, as an aspect to promote when implementing all activities funded under the justice programme. Against this backdrop, the analysis concluded that the justice programme does not focus specifically on gender aspects, although some calls for proposals include dedicated explanations for candidates on how gender can be taken into account when outlining the project implementation. Nevertheless, the beneficiaries interviewed stated they had made conscious efforts to respect gender equality and non-discrimination in all stages of the project cycle. The way in which beneficiaries presented their approach to gender mainstreaming and contributed to gender equality showed that applicants still struggle to understand and apply these concepts. Hence, the justice programme has the capacity to raise awareness of the need to promote gender equality and gender mainstreaming among applicants and beneficiaries, and there is potential to improve the promotion of gender equality and gender mainstreaming at project level. To tap this potential, it would be useful to look at producing additional guidance to applicants and making a closer alignment of funding with gender mainstreaming. As for children’s rights, annual work programmes enshrine in a similar fashion to gender equality the need to support the protection of the rights of the child. Calls for proposals under the purview of the justice programme contain mainstreaming clauses, where warranted, which ensure that funded projects also seek to promote equality between women and men and the rights of the child. In addition, respect for the rights of the child is enhanced by requirement for organisations applying for funding and their partners working directly or having contact with children to provide the European Commission with a description of their child protection policy. Evidence at the project level collected through interviews with beneficiaries revealed that many project consortia found it difficult to include the rights of the child when addressing judicial matters. These difficulties were in interpreting the relation between the project objectives and children. Interviewees indicated that they deemed children not relevant or made no distinction for minors. Similar to gender mainstreaming, some interviewees struggled to describe how their projects would cover aspects relating to the rights of the child. Thus, to improve coverage of children’s rights in project design and implementation, the provision of additional guidance to applicants could clarify how and when to include children. Last but definitely not least, the justice programme enshrines provisions linked to Article 10 of the Treaty on the Functioning of the European Union, which requires that the definition and implementation of EU policies and activities aim to combat, among others, discrimination based on disability. Furthermore, with the ratification of the UN Convention on the Rights of Persons with Disabilities, the EU has a legal obligation to mainstream disability matters into a wide range of EU policies. In comparison to the rather advanced work to mainstream gender equality and children’s rights, the rights of people with a disability do not receive not as much attention in projects funded by the justice programme. Beneficiaries revealed in interviews that they found it difficult to include the rights of people with disabilities when covering judicial matters that were valid to all people. Some interviewees struggled to describe how their projects would cover the rights of people with disabilities. These observations are also underlined by two European Commission officials, who indicated in interviews that projects under the justice programme are not always well suited to cover the rights of people with disabilities. This can result in attempts from beneficiaries to justify in their applications a link to priorities under the call for proposals that does not necessarily exist. Scope for simplification Concerning the scope for simplification, the European Commission made major changes to review, speed up and simplify the process over the 2014-2020 programming period. Although, the current management model is considered to be suitable, and the project beneficiaries have positively welcomed the changes, they also identified issues and room for further improvement. According to the analysis, the scope for simplification encompasses the need for simpler application and results-based reporting procedures, improved monitoring and management tools to collect more systematically information on achieved outputs, results and targets, and help for organisations with difficulties working only in English. However, it should be mentioned that the perceived burden and complexity varies among project beneficiaries. Organisations with a larger capacity and greater experience in applying, coordinating, and participating in projects, find it much easier and smoother than smaller organisations and first-time applicants. For newcomers, procedures, changes in reporting templates and the lack of swift responses to queries can hamper the application process. Lastly, despite improvements to the monitoring and management tools, there appears to be room for further improvement. In particular there is scope to improve and adapt the application portal to the types of beneficiaries supported by the justice programme and to support first-time applicants and beneficiaries in using the tool without the risk of delaying their activities. The collection of output data could also be improved to improve the quality and consistency of information provided by project beneficiaries on the results of the project. Conclusions As mentioned in the introduction, this first report of the ex- post evaluation of the justice programme does not evaluate the long-term impact of funded projects. The conclusions and recommendations presented in this section cover only the activities that can reasonably be evaluated since the justice programme was launched. The previous section has shown, in the answers to the evaluation questions, that so far the justice programme has delivered results on all evaluation criteria. In terms of the effectiveness of the justice programme, the current analysis confirmed the result of the interim evaluation. The activities funded through action and operating grants as well as procurement activities achieve the results they are expected to achieve. In other words, the available budget is used to achieve results that are well aligned with the general and specific objective of the justice programme. A key finding that emerged from the analysis is that activities promoting the specific objective on judicial training are especially well received. Though an overall positive assessment of the justice programme’s effectiveness is justified, there is room for improvement as the justice programme does not perform equally well on every specific objective. In fact, the specific objective drug prevention policy is allocated a significant lower amount of funding than the other specific objectives, even though it receives numerous applications. This results in a very low award rate and requires the European Commission to be very strict in evaluating the proposals, such that only a few of the most relevant projects can be funded.

However, this criticism needs to be put into perspective as DG Justice addressed this problem by moving the objective of initiatives in the field of drug prevention policy outside the remit of the justice programme for the next programming period, as suggested in the interim evaluation. Concerning interaction with the target groups, the analysis shed light on the need to create a more sophisticated monitoring system, which would facilitate data collection in a reliable and systematic way. This is of major importance as a better monitoring system would enable the Commission to obtain more in-depth insights about target groups, and thereby help to increase the programme’s effectiveness. Fortunately, the European Commission covered this problem in its proposal leading to Regulation 2021/693 providing the legal base for the justice programme between 2021 and 2027. The Regulation envisions that indicators will be consolidated and simplified. It will also improve the calculation of indicators as their calculation method will be based solely on hard data. In addition, in terms of implementing the justice programme and the application procedure, the analysis identified the need to provide additional guidance to applicants in order to promote the quality of writing in proposals. Additional guidance is needed as there is a discrepancy between the capacity of applicants to submit a well written proposal that meets all application standards, and the capacity of applicants to deliver the results they describe in their proposals. Broadly speaking, communication activities promoting the justice programme are effective. However, some beneficiaries suggested that a swifter reaction to their doubts and questions during the application period would immensely facilitate the process. Key points in a nutshell ·There is a need for structured data collection tools to understand how target groups perceive the quality of projects funded by the justice programme. ·The application process is still challenging as the indication of project outputs in proposals remains burdensome for some applicants, and is a factor potentially excluding relevant organisations from participating The justice programme also made signification progress in terms of its efficiency compared to its predecessor programmes. The stakeholders consulted confirmed the cost-efficiency of the programme. Although they underlined that the application procedure is very demanding and costly, they equally stressed that the benefits provided by participating in the programme is even more rewarding. This finding is in line with the positive feedback on streamlining the application process. However, there are also some aspects on which more critical views were expressed. For example, applicants repeatedly stated that despite visible improvements in the management cycle, the timeframe for submitting project proposals is still too ambitious and the time between proposal submission and selection is too long. Another area of potential improvement in the duration of the application period is the provision of guidelines and information sessions to help applicants handle the administrative burden involved in making an application. This could also help encourage organisations, put off from applying due to the requirements to participate in the justice programme. Last but not least, despite the increase in flexibility of the overall budget of the justice programme, underspending at project level remains an issue. The reason for the continued underspending of resources is mainly linked to beneficiaries over budgeting projects. Thus, there is a need for additional guidance on budgeting. Key points in a nutshell ·Applying and receiving funding under the justice programme is very demanding and costly. However, the benefits of participating in the programme outweigh the associated costs. ·Underspending is still an issue for beneficiaries, who struggle with accurate budgeting.



The analysis showed that the programme and its projects achieve a very high level of relevance. It concludes that project objectives and activities are relevant to the needs of targeted stakeholders. In turn, the programme’s target groups are relevant for the achievement of the programmes’ specific objectives. Here the evaluation found few areas for improvement. These include engaging in a more systematic and open manner with stakeholders to better align the activities and specific objectives outlined in the annual work programmes with the needs of the programme’s target groups. There is also a need to step up the current and successful work to expand the programme’s target group to include more stakeholders relevant to the justice programme. In particular, it could more actively target civil society organisations working on victims’ rights. Key points in a nutshell ·Despite a high degree of relevance of the projects funded, regular needs assessments and more frequent exchanges with stakeholders are needed to even better define the annual priorities and to align them with the needs of target groups. ·On victims’ rights, civil society organisations could be more actively involved in the programme.



The current analysis shows that the justice programme has a good level of coherence and complementarity with other EU instruments, programmes and actions. It underscores that, especially on judicial training, the merge of the predecessor programmes into the justice prog