Table of contents 1.Introduction Purpose and scope 2.Background to the intervention Description of the intervention and its objectives Baseline and points of comparison 3.Implementation / state of Play Description of the current situation 4.Method Short description of methodology Limitations and robustness of findings 5.Analysis of the evaluation criteria Effectiveness Efficiency Coherence Relevance EU added value 6.Conclusions Effectiveness

Efficiency Coherence Relevance EU added value 7.Developments made during the last 2 years of the 2014-2020 programming period and the outlook for 2021-2027 Annex 1: Procedural information Annex 2: Stakeholder consultation Annex 3: Methods and analytical models Annex 4: Inforegio report “Analysis and assessment of Services and Performance of JASPERS initiative 2014-2018” Glossary Term or acronym Meaning or definition CEF Connecting Europe Facility CF Cohesion Fund CPR Common Provisions Regulation, Regulation (EU) 1303/2013 DG MOVE Directorate General for Mobility and Transport DG NEAR Directorate General for European Neighbourhood Policy and Enlargement Negotiations DG REGIO Directorate General of Regional and Urban Policy EC European Commission EIB European Investment Bank ERDF European Regional Development Fund ICT Information and communication technology IPA Instrument for Pre-accession Assistance Fund IQR Independent quality review JASPERS Joint Assistance to Support Projects in European Regions MoU Memorandum of Understanding MS Member State PSA Post-submission appraisal SGA Specific Grant Agreement SWD Staff Working Document 1.Introduction Purpose and scope The Joint Assistance to Support Projects in European Regions (JASPERS) is an initiative of the European Commission (EC) and the European Investment Bank (EIB). It provides independent and free-of-charge technical advisory services to all Member States and pre-accession countries in all sectors relevant for EU funded investments. JASPERS was created in 2005 and launched in 2006. Since then, it has grown into one of the largest initiatives for technical assistance supported by the EC. In line with Article 2(4) of the 2014-2020 framework partnership agreement for the management of the JASPERS technical assistance facility concluded between the EIB and the EC, which regulated the functioning of JASPERS in the 2014-2020 programming period, the parties agreed to undertake a mid-term evaluation of the implementation of the agreement. This evaluation aims to assess the services provided by JASPERS in 2014-2018 and how they contributed to the development of high-quality, mature infrastructure projects as well as to the technical capacity of managing authorities and beneficiaries to prepare such projects. The previous evaluation commissioned by the EC on JASPERS was finalised in 2012 and covered 2006-2011. The current evaluation is intended to inform decision-makers at European and national level and to provide useful insights for steering the future organisation of and demand for the services concerned. The evaluation covers all JASPERS’ activities during the 2014-2018 period in all countries that made use of its advisory services (i.e. Member States and pre-accession countries). It follows the five evaluation criteria defined by the EC’s better regulation guidelines: effectiveness, efficiency, coherence, relevance and EU added value.

The main analytical sources of the staff working document are the Inforegio report ‘Analysis and assessment of JASPERS’ services and performance 2014-2018’ and the final report of the external study ‘Mid-term evaluation of the JASPERS initiative in 2014-2018’. The findings of the European Court of Auditors special report on JASPERS, published in 2018, have also been considered in the scope of this evaluation, as have the results from the previous evaluation of the JASPERS initiativecompleted in 2012. The findings and conclusions of this evaluation are drawn from the cross analysis of these documentary sources. 2.Background to the intervention Description of the intervention and its objectives The legal documents establishing the functioning of JASPERS (i.e. the Memorandum of Understanding for 2007-2013, and the framework partnership agreement for 2014-2020) set out JASPERS’ overall objective, which was generally and widely stated as contributing to the improvement of the quality of investments supported by EU funds. The idea behind JASPERS was first described in a concept paper issued by the EC in 2005. At the time, the provision of technical advisory services was justified by: -the increased financial resources available to fund certain types of investment, which received relatively small amounts of funding in the past; -a relative lack of experience and capacity in the new Member States and acceding countries to develop large-scale projects. Consequently, JASPERS’ general aim, as defined by the 2006 MoU, has been to support cohesion policy implementation by increasing the resources available for preparing projects, and by improving the quality of technical advice available to project promoters, thus contributing to the preparation of high quality projects. The expected impact was a more rapid absorption of European Regional Development Fund (ERDF) and Cohesion Fund (CF) budgets by Member States and acceding countries finally contributing to economic growth and environmental development. In 2011, JASPERS conducted a needs analysis in preparation of the 2014-2020 period, based on inputs from the Member States and the EC. The analysis showed that the types of service provided by JASPERS would continue to be relevant for at least seven Member States that had 'high' priority need for support in project preparation in at least one of the sectors examined. Therefore, in the 2014-2020 programming period, JASPERS’ general objective continued to be improving the quality of investments supported by EU funds. However, due in part to the new regulatory framework for approval of major projects, its scope was extended significantly in terms of Member States covered, the types of service provided and the EU funds targeted.



Figure 1. JASPERS beneficiary countries Source: JASPERS website (15.06.2020) In the 2014-2020 programming period, JASPERS assistance covered advisory services for potential investments supported by the following EU funding instruments (i.e. mandates): -ERDF and CF: mandate of the Directorate-General of Regional and Urban Policy (DG REGIO); -Connecting Europe Facility (CEF), since 2015: mandate of the Directorate-General for Mobility and Transport (DG MOVE); -Instrument for Pre-accession Assistance Fund (IPA II), since 2016: mandate of the Directorate-General for European Neighbourhood Policy and Enlargement Negotiations (DG NEAR). From a procedural and operational perspective, in the 2014-2020 programming period, JASPERS activities were regulated by the FPA signed in 2014 between the EC and the EIB for the ESIF mandate. This was subsequently amended in 2015 to include the CEF mandate, and in 2016 for the IPA II mandate. The FPA is based on the broader Financial and Administrative Framework Agreement (FAFA) signed by the EC and the EIB in 2014. The two documents fix the overall roles and responsibilities of JASPERS as well as the roles and responsibilities of other parties involved in managing and implementing JASPERS.

Since its creation, JASPERS was managed by the EIB and supervised by a Steering Committee of representatives of its partner institutions, the EC and the EIB. In accordance with the FPA, the role of the Steering Committee was to: a.review and decide on all significant policy issues related to the operation of JASPERS including, but not limited to, eligibility of a new country for JASPERS support; joining of JASPERS by a new partner; issues involving national authorities at a high level; communication policy; b.review and approve significant operational outputs and provide guidance on any significant operational issues arising; c.at the request of any of the partners, review the implementation of JASPERS with a view to proposing improvements to its supervision or management d.review and approve the annual reports and annual financial statements of JASPERS, as well as any other publications reporting on the operations of JASPERS. Further on, annual or multiannual specific grant agreements (SGAs) were signed between the EIB and the three DGs governing the mandates, as follows: -The ESIF mandate was based on annual SGAs signed between DG REGIO and the EIB, which defined the scope, priorities, and conditions of implementation and the annual resources allocated to JASPERS; -The CEF mandate was based on multiannual SGAs, signed between DG MOVE and the EIB; -The IPA II mandate was based on a multiannual SGA, signed between DG NEAR and the EIB. The SGAs reiterate the roles and responsibilities of JASPERS and further specify the types of services that JASPERS can provide under the different mandates. Further to this, regular Steering Committees with the participation of the EC and JASPERS, provide guidance on the development of JASPERS. JASPERS was organised in seven divisions. Five of them are thematic (for roads; rail, air, and maritime; water and wastewater; smart development; and energy and solid waste), the remaining ones being the Independent Quality Review (IQR division) and the Networking and Competence Centre (NCC division). In addition, JASPERS included a Quality Management Unit. Over the 2014-2020 programming period, JASPERS has operated its advisory services on a demand basis, in response to specific requests made by Member States, with the subsequent agreement of the EC. Each task implemented by JASPERS is referred to as an ‘assignment’. JASPERS formalised its work on assignments in rolling country action plans maintained throughout the period. According to JASPERS’ quality manual, the country action plans included working arrangements signed with the beneficiary countries and a list of projects and areas where JASPERS was likely to be asked for support. These plans were updated in line with requests for support received from the beneficiary countries. As defined in the FPA, the services provided by JASPERS were: a)Technical advisory services (relevant for all mandates, available from 2006); b)Review function (relevant for the ESIF mandate, introduced in the 2014-2020 programming period); c)Capacity building and horizontal and strategic support (relevant for all mandates, available from 2006). The specific objectives for each type of service were developed and explained in the quality manual produced by JASPERS. a)The technical advisory services were designed primarily to improve the projects intended for EU financing, indirectly contributing to their timely approval. The advisory services focused on the project preparation phase, but they could also be available during project implementation, with a view of achieving a more rapid absorption of EU funds. These services were provided for all three mandates.

For the ESIF mandate, technical advisory services concerned both major and non-major projects as defined by the Common Provisions Regulation (CPR) (EU) 1303/2013, . In the case of the CEF and IPA II mandates, the distinction between major and non-major projects did not apply. b)The review function was relevant solely for the ESIF mandate. It depended only in part on the demand from Member States as JASPERS received a clear mandate from the EC to review all new major projects (except of phased projects under Article 103 of the CPR). This review was linked to the EC’s assessment of major projects in the 2014-2020 programing period carried out under Articles 101 and 102 CPR. The review services were designed to help Member States and the EC assess whether or not major projects fulfil the quality regulatory criteria required for EU funding on a sound, economic and technical basis, by providing clear statements on a major project’s feasibility and economic viability. For the 2014-2020 programming period, as proposed by the EC, legislators provided a new system for appraising and approving new major projects funded under EU cohesion policy. It suggested two alternative systems. 1. In line with Article 102(1) CPR (notification procedure), Member States could notify a selected major project to the EC, accompanying it with an independent quality review (IQR) report performed by independent experts. 2.Under Article 102(2) CPR (submission procedure), Member States had the possibility to submit the major project application, together with the full project documentation, directly to the EC for assessment and approval, . This is similar to the procedure applied during the 2007-2013 programming period. Unlike in the 2007-2013 procedures, in the 2014-2020 programming period, the EC had the possibility when assessing a major project to request a post-submission appraisal (PSA) report from independent experts. Under the notification procedure (Article 102(1)), Member States submitted all project documentation directly to JASPERS’ IQR division for assessment. JASPERS IQR then issued its opinion as to whether or not the project met the required quality criteria, and confirmed the project’s feasibility and economic viability. The IQR report included an assessment covering all the requirements and was sent to the Member State upon completion. If the IQR conclusions were positive, the Member State notified the project application, together with the IQR report, to the EC for approval. Under the notification procedure the Member States needed to wait until JASPERS’ assessment was complete and the project application was submitted to the EC before being able to certify expenditure. With the entry into force of the Omnibus Regulation (EU, Euratom) 2018/1046, even projects subject to the notification procedure could have had certified expenditure as soon as JASPERS’ IQR division confirmed the project documentation and application were complete. Under the submission procedure (Article 102(2)), Member States submitted the full project documentation to the EC, together with the application for funding, since, in this case, it was the EC that was responsible for assessing the quality of the project submitted. Upon receiving the project documentation and the application for EU financing, the EC asked JASPERS’ IQR division to perform a PSA and to assess all aspects of the project, except for the alignment of the project with the operational programme; State aid rules; environmental issues; and the completeness of the project application (which were assessed directly by the EC). As regards climate change aspects, the PSA report assessed the project’s alignment with climate-proofing requirements established in a Memorandum of Understanding agreed by DG REGIO and DG CLIMA in 2015. JASPERS’ IQR division issued its opinion on project quality in the form of a PSA report sent directly to the EC. As this review had a more limited scope than the IQR report, a PSA report was less comprehensive; it distinguished only between critical and non-critical issues identified for project quality and documentation for the topics covered. Information provided in the PSA report was not the only source the EC used to assess a major project. Member States could certify expenditure for major projects submitted under Article 102(2) as soon as the EC had determined the completeness of project documentation and application in SFC2014. c)Capacity building and horizontal strategy support was available for all three mandates. The aim was to strengthen Member States’ capacity to identify, prepare and implement projects, and to provide support in preparing national/regional strategies for the implementation of EU cohesion policy. From an operational perspective, the process of assignment creation and implementation of assignments for capacity building was similar to that for the technical advisory function. In the context of this type of service, in 2012 JASPERS established the JASPERS networking platform. The platform complemented JASPERS’ project advisory operations by implementing knowledge sharing and capacity building activities, as well as fostering dissemination of good practices and exchange of experiences among all Member States, pre-accession countries and other JASPERS stakeholders. Its website included information on JASPERS’ past and upcoming events, annual programmes from 2012/2013 to the present, and a library of JASPERS’ guidance and capacity building documents. Figure 2 showcases (in a simplified manner) the intervention logic of JASPERS covering all types of services, i.e. technical advisory services to projects, horizontal and strategy support, capacity building, and review services (IQR and PSA).

Figure 2. Intervention logic of the JASPERS initiative for the 2014-2020 period Source: Mid-term evaluation of the JASPERS initiative in 2014-2020 Since the 2007-2013 programming period, the performance of JASPERS services has been measured through key performance indicators (KPI), such as numbers of completions, cumulative investment volumes of approved projects supported by JASPERS, and EC approval times for major projects. The KPIs were complemented for the 2014-2020 period by a result management framework with four dimensions (institutional; customer; financial & risk; internal processes; learning & people) and the European Foundation for Quality Management (EFQM) model. Baseline and points of comparison The baseline for this evaluation is the 2007-2013 period, when JASPERS was operating with a smaller quantitative scope and geographical coverage. The legal provisions regarding major projects have also evolved between the two programming periods. The baseline was chosen to ensure comparability and consistency with regard to the concept of ‘programming period’ and the number of years covered by the evaluation. The previous evaluation of the JASPERS initiative, completed in 2012, informs the points of comparison. The detailed numerical information for both programming periods is available in Tables A.3.1 – A.4.6 of Annex 4 to the SWD. This section provides the comparison point used in the analysis included in Section 5 and informs the findings under the different evaluation criteria used to assess the performance of the instrument over the 2014-2018 period. The points of comparison are used primarily for the effectiveness and relevance criteria. A proxy analysis was performed for the efficiency criterion. It refers to a new type of service (i.e. the review service) introduced in the 2014-2020 period and the analysis of the overtime trend of human and financial resources. The coherence and EU added value criteria are not influenced by the points of comparison as the analysis is self-standing in relation to the 2014-2018 period of the evaluation. Over the period 2007-2013, the EC invested EUR 132 million in the JASPERS initiative, representing 71% of the total costs. The remaining 29% was covered primarily by the EIB with in-kind support (i.e. for additional technical expertise, logistics of the operating offices, etc.). During this period, JASPERS contributed to mobilising more than EUR 32 billion ERDF and Cohesion Fund resources by assisting 383 major projects which were subsequently approved by the EC . As mentioned earlier, during the 2007-2013 period, JASPERS’ work consisted of solely the ESIF mandate for DG REGIO. JASPERS completed around 790 assignments for EU cohesion policy projects and six assignments for IPA-funded projects. 82% of these activities were related to the advisory function for preparing major and non-major projects and, to a lesser extent, for project implementation. The remaining activities consisted primarily of horizontal and strategic support. The water and wastewater treatment sector was predominant in JASPERS’ portfolio, accounting for one third of all assignments for the period, followed by the knowledge economy, energy and waste management combined. Capacity building activities and horizontal and strategic support focused largely on the same sectors as well as on multi-sector assignments. As regards the distribution of assignments between countries, 76% of JASPERS’ activities in 2007-2013 were concentrated in five Member States: Poland, Romania, Bulgaria, Czechia and Hungary. The remaining assignments were distributed across nine other Member States and three IPA countries. The highest number of assignments (160 accounting for 20% of all assignments of the period) was completed in 2010 (the fourth year of the programming period). Moreover, by the fifth year of the period (i.e. the year equivalent to the reference end-date for the evaluation of the 2014-2020 period), JASPERS had carried out 68% of all assignments completed in 2007-2013. 3.Implementation / state of play Description of the current situation During the 2014-2018 period, the EC planned to invest close to EUR 150 million in JASPERS, representing around 81% of the total costs of the initiative (EUR 175 million in total). The remaining amount was to be covered primarily by the EIB. The technical advisory support provided by JASPERS helped to mobilise almost EUR 23 billion of ERDF and Cohesion Fund resources through 142 major projects assisted during preparation and which were adopted or approved by the EC. A further EUR 2.2 billion from the CEF cohesion budget was mobilised, with the projects assisted by JASPERS and approved for CEF funding under the CEF1.1 mandate, and EUR 82 million for 10 IPA projects assisted and approved for IPA funding by the end of 2018. Over this reference period, JASPERS’ portfolio included 1096 assignments for all mandates and services. Figure 3 illustrates the distribution of these assignments by mandate and type. Figure 3. JASPERS’ portfolio by mandate and assignment type, 2014-2018 Note: CB = capacity building, HSS = horizontal and strategic advice ; data covers 1096 assignments Source: Inforegio report “Analysis and assessment of JASPERS’ Services and Performance 2014-2018”, JASPERS’ JADE database, December 2018

The ESIF mandate was the largest in JASPERS’ portfolio, with close to 960 assignments, followed by the IPA mandate with around 100 assignments. JASPERS’ traditional function of technical advisory services – primarily project preparation for the three mandates – represented almost two-thirds of its portfolio. The activities for capacity building and horizontal and strategic support mostly focused on cross-cutting topics relevant for different sectors (e.g. environmental requirements, cost benefit analysis, climate proofing), and country-level issues. They were linked primarily to the ESIF and the IPA mandates. Finally, the review function was specific to the ESIF mandate and included project assessment services provided to Member States (for IQR assessments – 60 assignments) and to the Commission (for PSA assessments – 150 assignments). All new major projects submitted by Member States in the period evaluated were reviewed by JASPERS IQR division using one of the two procedures mentioned above. Figure 4. Distribution of JASPERS’ assignments by sector, 2014-2018 Note: CB = capacity building, HSS = horizontal and strategic advice ; data covers 1096 assignments. Source: Inforegio report “Analysis and assessment of JASPERS’ Services and Performance 2014-2018”, JASPERS’ JADE database, December 2018 Assignments for the rail, air and maritime transport sector accounted for the largest share of JASPERS’ portfolio (33%). The majority of these assignments (70%) were related to technical advisory services for project preparation. The water and wastewater treatment sector was the second largest, with a substantial number of assignments for technical advisory services. The multi-sector assignments related primarily to capacity-building activities, including technical support papers, workshops and training (both multi-country and country-specific) and were on topics of general relevance, such as cost-benefit analysis, climate adaptation and mitigation, and environmental requirements.



Figure 5. Distribution of JASPERS’ assignments by country, 2014-2018 Note: CB = capacity building, HSS = horizontal and strategic advice ; data covers 1096 assignments Source: Inforegio report “Analysis and assessment of JASPERS’ Services and Performance 2014-2018”, JASPERS’ JADE database, December 2018 Until end-2018, JASPERS initiated assignments in 21 Member States and 5 IPA countries. The countries with the largest number of assignments were Poland, Romania, and Croatia, accounting together for 52% of all assignments in JASPERS’ portfolio. Poland also stood out for the relatively large number of assignments for the review assignments related to major projects. As regards timing, 76% of assignments were started in 2014 and the following years, while the remaining 24% were started before 2014, in preparation for the 2014-2020 programming period. By December 2018, 51% of assignments had been completed and 9% administratively closed, while the rest were still ongoing. Figure 6. Ongoing assignments by main type by year, 2014-2018

Note: CB = capacity building, HSS = horizontal and strategic advice ; data covers 1066 assignments Source: Inforegio report “Analysis and assessment of JASPERS’ Services and Performance 2014-2018”, JASPERS’ JADE database, December 2018 The yearly data for ongoing assignments indicate that, already by the first year of the 2014-2020 programming period, JASPERS’ portfolio included more than 300 assignments for technical advisory, capacity building and horizontal and strategic support services. The assignments for the review function were first requested in 2015 and increased over time as project preparation was completed and projects were submitted for EU co-financing. The number of ongoing assignments increased steadily each year until 2017 and decreased for the first time in 2018. By end-December 2018, there were 435 assignments still ongoing. Figure 7. Completed assignments by main type by year, 2014-2018 Note: CB = capacity building, HSS = horizontal and strategic advice ; data covers 535 assignments Source: Inforegio report “Evaluation of JASPERS’ Services and Performance 2014-2018”, JASPERS’ JADE database, December 2018 The pace of assignment completion accelerated significantly from 2017. For the first 2 years, JASPERS completed assignments relating only to technical advisory or capacity building and horizontal/strategic support services. From 2016, it also reported completing assignments linked to the review function (IQR and PSA). State of implementation – ESIF mandate The financial resources mobilised for JASPERS’ services under the ESIF mandate over the 2014-2018 period amounted to EUR 175 million, representing 84% of the ESIF mandate budget for 2014-2020. The EC contribution covered 81% of the total cost of these services (EUR 141 million). Considering the overall EC financial contribution to JASPERS’ services for the three mandates over the 2014-2018 period, the ESIF mandate provided the main share of the budget (97%). JASPERS’ portfolio for the ESIF mandate in the 2014-2020 period up until December 2018 included close to 960 assignments. These were primarily for technical assistance for project preparation and, to a much lesser extent, implementation (59% combined), followed by horizontal activities and capacity building (23%) and its new review function (18%). In terms of the distribution of the ESIF assignments across countries, 57% were carried out in three main beneficiary countries (Romania, Croatia, and Poland), 5% were multi-country assignments, and the rest were distributed across 19 other Member States. In the previous period, 59% of JASPERS’ activities for this mandate were similarly concentrated in three main beneficiary countries (Poland, Romania, Bulgaria), with the rest of activities distributed across 11 other Member States. State of implementation – CEF mandate EUR 1.4 million was allocated for JASPERS technical assistance services under the CEF1.1 mandate in 2015-2018, of which the EC contribution covered 90%. According to the final annual report of the CEF1.1 mandate, JASPERS carried out 31 assignments in support of the preparation of 29 CEF projects , with a large majority of them in the rail sector. At country level, the initial agreement for the first CEF pilot (CEF1.1) was that JASPERS would support projects chosen from the list of 58 projects in seven cohesion countries included in Annex I to the Special Grant Agreement. The actual number of projects supported was determined by the budget available for the JASPERS assistance pilot for the CEF1.1 mandate. State of implementation – IPA mandate

The resources allocated under the IPA mandate amounted to just over EUR 4 million in 2015-2018. This was 64% of the total IPA budget allocated . JASPERS’ portfolio for IPA included 102 assignments up to end-December 2018. Of these, 38% had been completed, 50% were still ongoing, and the rest had been administratively closed. The IPA assignments were not only for investment projects, but they were also used to help national authorities in guiding and supervising contracts for technical assistance. As regards the horizontal and strategic support for the IPA mandate, 80% of the 20 assignments in the portfolio at end-2018 had been completed or were still ongoing in Montenegro, Republic of North Macedonia and Serbia. Half of these were on workshops and advisory support on cost-benefit analysis and on assistance with national strategies and master plans. The remaining assignments had a variety of objectives, such as preparing a project pipeline and analysis and advice at sector level. More information on the state of implementation of each JASPERS’ mandate can be found in Section 5 of Annex 4 to this SWD. 4.Method Short description of methodology This evaluation is based on an extensive analysis of documentary sources and qualitative and quantitative data on the activities performed by the JASPERS services during the 2014-2018 period. The documentary sources include the strategic agreements governing the functioning of JASPERS, the documents produced by JASPERS in relation to the assignments, activity reports for the CEF mandates, annual JASPERS activity reports, country action plans, and documents produced by the EC. The quantitative data were collected from JASPERS’ dataset of assignments (the JADE database), the EC’s database which includes information on the major projects (SFC2014), EC databases with information on CEF and IPA projects, and JASPERS’ annual implementation reports. The qualitative data were collected via an online targeted consultation, a seminar organised in Brussels in September 2019 with relevant stakeholders, and interviews with MA representatives and EC officials. The analytical comparisons in this evaluation cover the following aspects: -For the comparisons between the 2007-2013 and 2014-2020 programming periods, the first 5 years are used as the basis for the calculations, as this is the number of years covered by the current mid-term evaluation. -Projects assisted by JASPERS and those not assisted are compared using econometric models in order to estimate JASPERS’ contribution to project preparation, the subsequent performance of the assisted projects in the approval process for EU funding and later in implementation. Limitations and robustness of findings This SWD relies only to a limited extent on the final report produced by the external evaluators commissioned by the EC to assess JASPERS. The data of the evaluation report, on which this SWD is based, relate to the description of JASPERS services, the six country case studies and the qualitative data collected from stakeholders. The evidence compiled in the final report was not considered sufficiently robust and representative, and so the conclusions were rather general in nature. It was therefore decided to carry out a new round of data collection and analysis in order to reach conclusions that are backed by sufficiently strong evidence and representative of all the assignments undertaken. This new and complete analysis is annexed to the SWD as the Inforegio report ‘Analysis and assessment of JASPERS’ Services and Performance 2014-2018’ – Annex 4. This analysis is used to inform all the findings related to the quantitative analysis. References throughout the document indicate the different sources of information where the detailed analysis can be consulted. Most of the findings of this evaluation can be considered robust as they are based on evidence collected mostly for the relevant population as a whole. Tables A.2.1.1, A.2.1.2, and A.2.1.3 of Annex 4 present a complete picture of the information that was collected and used for the analysis under each of the evaluation criteria, making a link between the investigated topic, the source of information and the percentage of the relevant population covered by the collected data. As shown in these tables, in the majority of cases the entire population was covered by the collected data, which ensures the reliability of the findings put forward by the evaluation. In some instances, the findings are affected by the following limitations. a)The objectives of the intervention were stated generally in the original documents, a weakness which was also documented by the Court of Auditors in the Special Report on JASPERS (2018). The specific objectives that were linked to each type of service provided by JASPERS were listed in the Quality Manual drafted by the JASPERS services, but no operational objectives were defined. b)At the end of 2018, it was planned to update the monitoring system in place to better reflect the performance of JASPERS services, but this update was scheduled for 2019 (outside the period covered by this evaluation).

c)The needs analysis performed before the start of the 2014-2020 period used mostly qualitative data collected directly from the stakeholders concerned (the Member States and the EC). d)Because of the need for the EC to carry out a new analysis and data collection, there was limited time for deepening the analysis of the need for JASPERS’ services and of the objectives of the intervention. e)The analysis of JASPERS’ advisory for non-major projects remains partial because of the lack of follow-up data on what happened to these projects once JASPERS’ advisory was completed. JASPERS does not collect this type of data, and only two of the six Member States sampled for the evaluation responded to the EC’s data requests. f)As regards the efficiency of the time allocation for JASPERS’ activities, the SWD analyses only the assignments completed in 2017-2019. Before 2017, data on the actual person-days the JASPERS experts spent on each assignment were not recorded. A more complete dataset for annual actual work per assignment over the whole period would have been needed for a full estimation of the scope for a more optimal allocation of resources. g)The analysis of the effectiveness of the IQR review service is also limited because of a lack of data. The IQR process was less formal than the PSA assessment made for the EC, in the sense that much of the information relating to the IQR review of projects was conveyed by email from JASPERS to the national authorities, as such in a non-structured form. As a result, it was not possible to carry out a systematic analysis of the effectiveness and coherence of this service relative to the advisory services (i.e. analysis of the issues signalled during IQR review versus the issues signalled during technical advisory). h)The analysis of EU added-value is limited to the interpretation of qualitative data. The information used to assess this criterion was collected in an online targeted consultation addressed to stakeholders. However, the opinions collected from stakeholders make reference more to the general added-value of the JASPERS services as compared to similar services available, rather than to the perceived EU added-value. Given the limitations listed above, a number of proxy analyseswere undertaken in order to assess the effectiveness, efficiency and relevance of the intervention. The details of these are presented in the corresponding sections on the evaluation criteria in Chapter 5 of the SWD. A comparative analysis was also made between projects assisted by JASPERS and those not assisted in order to judge the performance of the intervention. More details relating to the data sources and methods used are listed in Section 4 of Annex 4 to the SWD. 5.Analysis of the evaluation criteria The evaluation carried out in the current SWD follows the five evaluation criteria listed by the better regulation guidelines. It was decided to present the analysis by criterion without referring to individual evaluation questions. This decision was justified by the fact that the SWD relies primarily on the analysis performed internally by the EC. Because of the limitations indicated in the previous chapter, the SWD relies only to a limited extent on the external evaluation report. The analysis by evaluation criterion is mainly focused on the ESIF mandate. The ESIF mandate encompasses almost 90% of the assignments in the reference period and accounts for 97% of the EC’s financial contribution to the functioning of JASPERS in 2014-2018. The CEF mandate was initiated during the evaluation. The evaluation was a pilot initiative aimed at testing the relevance of the technical-assistance services proposed, so the evaluation findings are therefore limited. For the IPA mandate, most of the assignments were still ongoing at the end of 2018, and therefore the evaluation findings are based on a limited number of completed assignments. Effectiveness The analysis of effectiveness aims at assessing how successful the intervention has been in achieving, or progressing towards, its objectives. The main objective of JASPERS was to help Member States prepare large investment projects (mainly major projects) to meet EU standards and comply with EU regulatory requirements. The advisory function of JASPERS was intended to contribute to the preparation of good quality projects, which could then be proposed by Member States and succeed in a timely way to receive EU funding. Ultimately, well prepared projects should reach the stage of implementation as quickly as possible, contributing to achievement of cohesion policy goals. Under the review function, JASPERS provided an independent expert assessment of major projects presented for EU financing. Under the horizontal advice and capacity building function, JASPERS: (i) assisted MS in preparing strategies and national plans for investments supported by EU funds; (ii) helped to identify viable project pipelines for operational programmes; and (iii) helped to strengthen national technical capacity in preparing and implementing projects. To assess effectiveness, an analysis was carried out based on the following indicators: -the amount of EU resources mobilised and the number of major projects approved compared to the previous programming period; -the number of issues signalled on the quality of the projects submitted for funding – i.e. the rate of interruptions in the approval of major projects; -compliance with the regulatory deadlines (only for review services);

-the development of technical capacity at Member State level, understood as a reduction in the requests for advisory services between the two programming periods. On the ESIF mandate, JASPERS’ portfolio of assignments for technical advisory in 2014-2018 included 414 assignments for major projects. Of these assignments, 178 were completed by the end of 2018 and an additional 48 were completed in 2019. Of the 178 major projects covered by the completed assignments, 158 were adopted by the EC by October 2019. The adopted projects assisted by JASPERS involved a total EU funding of EUR 23.9 billion (ERDF and Cohesion Fund combined), representing 65% of EU resources mobilised for major projects adopted by the EC during the reference period of the evaluation. Compared with the first 5 years of 2007-2013, in 2014-2018 JASPERS completed fewer advisory assignments for major projects in the period under review (178 compared to 343 in the first 5 years of the previous programming period). Despite this smaller number of assignments, JASPERS helped mobilise 22% more EU resources in 2014-2018 than in 2007-2013. The variations in the number of assignments between Member States and sectors is detailed on page 35 of Annex 4 of the SWD. On the rates of interruption, there has been a significant reduction (from 82% to 46%) in interruptions of major projects assisted by JASPERS advisory services during approval procedures compared to the previous programming period. Comparing projects assisted and not assisted by JASPERS advisory services, the rate of interruption for assisted projects is 48% as against 43% for the non-assisted ones. However, the higher rate of interruption for projects assisted by JASPERS was not attributable to any failings of JASPERS advisory services, but rather to: (i) the features of the projects concerned (such as a higher level of complexity, which underpinned the need to seek external advice); and (ii) country-specific factors (such as the level of technical administrative capacity). For example, projects assisted by JASPERS advisory tended to have lower rates of interruption than non-assisted projects for all sectors, except for the roads sector. For the water and waste water sector, all projects submitted had been assisted by JASPERS advisory, and therefore a comparison was not possible. Within the group of assisted projects across sectors, the rate of interruption tended to be higher for energy and solid waste projects and for water and wastewater projects. When comparing rates of interruption for assisted and non-assisted projects at country level, it was found that the projects assisted by JASPERS advisory tended to have zero or lower rates of interruption in 4 countries (Austria, Czechia, Portugal, Slovakia), similar rates of interruption for 3 countries (Spain, Italy, Poland), and a higher rate for only 1 country (Greece). Table A4.6 of Annex 4 to the SWD has more detail on this. An analysis was made of documents produced when providing technical advisory services. This analysis showed that JASPERS’ experts signalled that changes were needed in the documentation in over 57% of the projects that later received ‘observations’ from the EC. This suggests that, even though the JASPERS’ experts included significant issues and comments in their reports, in some cases the issues were not resolved by the Member State before submitting the project to the EC. There are many possible reasons why the comments by the JASPERS experts were not fully resolved by the Member States. One possible explanation is that the experts’ message on issues requiring further attention might not have been sufficiently strong for the Member State to understand that changes were needed. Another possible reason is that there may have been time constraints on the side of the Member States for submitting the project. Yet another possible reason is that political decisions could have been taken not to address the issues signalled by JASPERS at that point. There were also cases where major projects were interrupted and received observations from the EC, even though JASPERS advisory provided a positive assessment of the project, with no suggestions for improvement. This is the case, for instance, as regards the assessment of environmental issues, indicators, and project descriptions, for approximately 90% of the major projects interrupted for issues on these topics. By comparison, and as noted above, in the 2007-2013 programming period, the overall rate of interruption of major projects assisted by JASPERS advisory was much higher (82% of all projects submitted), with similar rates of interruption for both assisted and non-assisted projects. Because the extent of the reduction in interruptions between these two periods is roughly the same for both assisted and non-assisted projects, this improvement can be seen as a consequence of the changes in the administrative set-up for assessing and approving projects instituted by the EC for 2014-2020, the latter implicitly integrating the involvement of JASPERS services for the review of all major projects. Finally, on interruptions, an analysis was carried out to assess whether the probability that a project was not interrupted was linked to the stage in which JASPERS’ advisory support was requested for project preparation. This analysis took explicit account of other factors such as project size, sector and country-specific factors. The analysis covered 95% of all new major projects approved by the EC under the submission procedure (entailing the PSA review) up until October 2019. The analysis indicates that major projects were more likely not to be interrupted during the approval/adoption process if they had been supported by JASPERS advisory during the preparation phase. This effect is stronger and more statistically significant when JASPERS advisory provided assistance from the very beginning of the preparation of the project – or at least from half way through its preparation – rather than at a later stage. In addition, JASPERS’ advisory support was positively correlated to a shorter duration of the preparation of the project relative to the overall project lifecycle, but only when the support was requested at the very start of the preparation stage. At the same time, the evaluation found that later involvement of JASPERS advisory in project preparation might have been correlated with a longer duration of project preparation. For non-major projects, the effectiveness of JASPERS’ advisory cannot be established, because data on the progress of projects to approval and implementation is not available. However, preliminary evidence for a third of completed assignments indicates an approval rate no higher than 60%. The detailed analysis underpinning the above findings can be consulted in Chapter 5.2 (Technical assistance for major projects) of Annex 4 to the SWD. On the review functions, the two strands of work were different in their scope and purpose as set out in the two bullet points below. ·Under Article 102(1) (notification procedure, entailing the IQR review), the full documentation of the major project was assessed by JASPERS before submission to the EC. The EC then received the IQR report issued by JASPERS and a notification template filled in by the Member State. ·Under Article 102(2) (submission procedure, entailing the PSA review), the full documentation of the major project was submitted to the EC. The choice of one of the two procedures for assessing the major projects was in the remit of the Member State. An important difference between the two procedures is that the Member States were permitted to make changes to the project documentation during the drafting of the IQR report. This was not the case for the submission procedure entailing the PSA report (where changes were possible only based on an official letter of interruption sent by the EC). Implicitly, depending on whether the Member State took on the advice of JASPERS, the IQR process had the potential to lead to faster processing and a higher approval rate of the major projects, and therefore a faster start to implementation on the ground. Moreover, the median total duration of the process for assessing and approving major projects notified under Article 102(1) (notification procedure, entailing the IQR review) was 7 months, of which the median duration of the IQR assessment was 5 months. By contrast, the median total duration of the assessment and approval process for major projects submitted under Article 102(2) (submission procedure, entailing the PSA review) was 3 months for projects not interrupted and 6 months for those interrupted. In this process, JASPERS was highly effective in delivering timely PSA reports. Compared to 2007-2013, there was a significant reduction in the overall time taken by the EC to assess and approve major projects in 2014-2018. This reduction was greater for the submission procedure with PSA review (50%) than for the notification procedure with IQR review (13%). By comparison, in 2007-2013, the average duration of the approval process for a major project was around 9 months, and the minimum duration recorded was 6 months. Details on the ability of JASPERS IQR services to signal issues related to project quality can be consulted in Chapter 5.2 (Review function) of Annex 4 to the SWD. For capacity building and horizontal and strategic advice, JASPERS’ portfolio included 220 assignments, 75% of which were for horizontal and strategic advice and 25% for capacity building.

The objective of the capacity-building assignments was: (i) to provide training, workshops and online fora on topics relevant for preparing and implementing projects; and (ii) to help disseminate guides and studies. The training and events organised for capacity building could have been either multi-country or tailored to the specific needs of an individual country. In the period analysed, most assignments for capacity building were multi-country assignments, and the most common topics of training/workshops were climate change, environmental requirements and state aid. During the 2014-2018 evaluation period, most activities for capacity building were completed and achieved their objective of delivering training to the Member States. However, these activities did not prove sufficient to trigger a visible knock-on effect on the technical capacity of Member States that would allow them to effectively address issues in their preparation of projects such as: (i) cost-benefit analysis; (ii) climate change; and (iii) environmental requirements. It is helpful to compare the main issues addressed by capacity-building activities completed up until end-2018 on the one hand with the quality gaps identified in project preparation during the advisory work of JASPERS and during the assessment of major projects by the EC on the other. Two main issues were identified in this comparison, both of which are addressed in the paragraphs below. Firstly, despite the relatively frequent focus of capacity-building activities on cost-benefit analyses, climate change and the environment, shortcomings in these aspects of project applications were still among the most frequent reasons for interruptions issued by the EC in the approval process of major projects. This suggests that the capacity-building activities organised on these topics did not succeed in having a significant effect on the technical capacity of Member States to address these issues themselves without support. Secondly, the documentation of advisory work and the EC’s assessments of major projects showed that there were still many projects in need of advisory support on topics such as feasibility studies, risk analysis, and sensitivity analysis. Capacity building is not that frequent in these three areas. During a consultation, stakeholders suggested that capacity building is better strengthened with hands-on experience while working directly with JASPERS on the projects, rather than through training and workshops. While, in principle, there might be merit in this argument, there is little evidence that this hands-on experience is reflected in better preparation of projects across sectors and countries. This may be due to high turnover of staff in national authorities’ administrations, a phenomenon also referred to in the ECA report. It may also be because JASPERS often interacted with consultants contracted by Member States to prepare project documentation, and therefore the learning effect for the staff of national authorities was limited. Moreover, the evaluation highlighted that countries’ reliance on JASPERS advisory assistance for project preparation has not significantly decreased compared to the previous period. This suggests there has only been a modest capacity increase in project preparation at national level between the two periods . A pragmatic approach implemented by JASPERS for technical capacity building was to ‘train the trainers’. This focused training on local trainers who could subsequently disseminate the know-how they had acquired to larger audiences on many occasions within and across countries. This approach was used by JASPERS for some capacity-building assignments over the period, but it had not yet been mainstreamed at the level of the portfolio, within the timeframe of this evaluation. On horizontal and strategic support for capacity building, these assignments were highly diverse in their specific objectives (including support for national strategies on specific issues, river basin flood management plans or integrated urban development plans). For this reason, it was far more difficult to assess the overall effectiveness of the support. An analysis of 10 sampled capacity-building assignments indicated that JASPERS’ support helped compensate for weaker local technical capacity to guide and supervise tasks on specialised topics of policy relevance. However, this finding cannot be generalised for all 166 training assignments for horizontal and strategic support from the reference period of the evaluation. The detailed analysis underpinning the findings from above can be consulted in Chapter 5.2 (Capacity building and horizontal and strategic advice) of Annex 4 to the SWD. On the CEF mandate, most of the projects supported by JASPERS during its first pilot mandate were successful in CEF competitions up to August 2019. Seven of the investment projects assisted by JASPERS at the application stage for CEF competitions had also been assisted by JASPERS technical advisory in their preparation in the 2007-2013 programming period under the ESIF mandate. Because of its pilot nature, a comparative analysis of the effectiveness of JASPERS’ intervention under the CEF mandate compared to a previous existing baseline cannot be undertaken. It is also not possible to conduct a comparative analysis of the effectiveness of the intervention for the ESIF mandate. Nevertheless, the objectives stated in the strategic documents have been followed, with JASPERS providing technical assistance to prepare eligible projects, with the intention of them being submitted in response to CEF calls. On the IPA mandate, by the end of 2018, only 18% of JASPERS’ assignments resulted in IPA projects being approved for EU funding (17 projects). Overall, by the end of 2018, 38% of the assignments had been completed, 50% were still ongoing, and the rest had been administratively closed. Based on an analysis of a sample of six assignments, there is an indication that the effectiveness of JASPERS’ advisory support was limited in cases where the requests for support arrived late in the project preparation cycle. For these late-arriving projects, it can be concluded that JASPERS advisory could have made a difference to project quality if its support had been requested earlier in the project-preparation process. When JASPERS support was requested in the final stages of preparation (as was the case for three completed assignments by 2018), despite JASPERS’ critical assessment of project quality, no substantial changes were made by the national authorities concerned. Finally, there is a finding that is linked to the consultation activities and valid for all mandates. This finding is that JASPERS had a significant, and even decisive, effect on project technical specifications and documentation. A large majority of stakeholders (80% of 210 respondents) rated highly JASPERS’ assistance in aligning projects with EU and national policies and in achieving faster project preparation and approval. The state of play of the JASPERS initiative was assessed by the ECA in a performance audit report published in 2018. The audit covered the first 10 years (2006-2016) of JASPERS’ activities and four Member States: Croatia, Malta, Poland, and Romania. On the effectiveness of JASPERS’ activities, the ECA auditors concluded that these activities contributed to faster project approval and had a positive effect on the quality of projects and their documentation. For the implementation of projects assisted by JASPERS, the ECA auditors found that, while JASPERS’ comprehensive assistance could generally not have an impact on the financial implementation of EU funds, the projects assisted were found to be less frequently affected by legal and regularity errors during the compliance audit exercises carried out by ECA in 2014-2015. Efficiency The evaluation of efficiency aims at assessing the relationship between the resources used by an intervention and the changes generated by that intervention (the costs and benefits). Because data on the actual costs incurred by the work on individual assignments is limited (see Section 4), the analysis considers JASPERS efficiency by looking at: -the administrative closure of assignments; -the work intensity distributed across assignments; -the correlation between human resources, financial resources and the number of assignments. The analysis below covers only the ESIF mandate, which provided 97% of the total financial resources allocated to JASPERS during the 2014-2020 programming period. Overall, more than 90% of JASPERS’ assignments were completed in the period 2014-2018, with 9% being administratively closed (i.e. assignments cancelled before completion), usually long time after being initiated. Half of those administratively closed assignments were concentrated in three countries. A quarter of the cancelled assignments had a duration of more than 50 months, highlighting the long time taken before closure. Moreover, based on the analysis of country plans, it can be concluded that many of the administratively closed assignments had no initial expected date for completion or, if they had such a date, this date was much later than set out in the initial plan. On the reasons for administrative closure, the evaluation identified three main reasons: (i) an agreement between JASPERS and the country concerned to combine the assignment with other ongoing assignments; (ii) closure of the assignment because of a change in context (for example, projects prepared for the ESIF but eventually submitted for CEF); and (iii) a decision by the country concerned to remove the assignment from its plan because it was deemed no longer feasible.

On the intensity of work in relation to the duration of assignments, the evaluation concludes that there is little correlation between these two variables in most cases. This means that assignments that last longer tend to have less intensity of work. For technical advisory support, this can be explained by the fact that the advice during various steps in the preparation and implementation of the project depends on a number of factors (time needed to receive the reply from the Member State, contractual arrangements with external consultants at Member State level, etc.). This finding also holds true for review services. However, it is important to differentiate the two strands, as the PSA services had a high concentration of short assignments (with most being finalised within a month), while the IQR services had a wider range of durations of assignments (up to 10 months). PSA assignments remained the most work intensive. Comparing the resources allocated over the two periods, it is clear that JASPERS has grown significantly, increasing from 56 people in 2007 to 140 professional and support staff in 2018. This reflects adaptation to the demand for JASPERS’ services, which grew and became more diversified in the 2014-2020 period. Moreover, the change in the structure of the budget indicates a reorientation in JASPERS’ staff policy. In the period 2014-2018, the initiative relied on its own staff to a larger extent, as opposed to the more extensive use of external consultancy services in the previous period. This means that most of JASPERS’ resources were invested in its people, with the share of staff costs increasing from 82% to 91% between the two periods. Figure 8. Human and financial resources, and assignments over time, ESIF mandate Source: Inforegio report Analysis and assessment of JASPERS’ Services and Performance 2014-2018, based on data reported in JASPERS’ annual reports 2007-2018. The left panel in Figure 8 illustrates the number of annual full-time equivalent workers (FTEs) supported in JASPERS and the annual monetary contribution by the EC over the period 2011-2018. These data indicate that, starting in 2014, there was a continuous increase in staff every year over the period, while the EC’s annual contribution increased at the same time. Compared to 2013, the number of JASPERS’ personnel in 2018 was 55% larger and the EC’s annual contribution was 65% larger. For ongoing assignments, there has been an almost constant, linear increase each year until 2017, while staff numbers continued to increase throughout the period. The increase in the annual number of assignments up until 2017 can be explained by both the rate at which assignments were accepted and their median duration of roughly 2 years. The completion or closure of assignments mostly began to increase in 2016. However, the number of ongoing assignments is an imperfect proxy for the actual work involved. This is because, as explained earlier, the work intensity per assignment varied significantly, and work intensity was not very high for most assignments other than those involving the PSA review. This preliminary evidence therefore indicates a need for further in-depth analysis of the topic to help identify possibilities to optimise the allocation of work and rationalise costs. In addition, the ECA audit also identified scope for improvement in a number of aspects in the administrative set-up and approach of JASPERS’ activities. These included: -the role and responsibilities of the main stakeholders, which the ECA believes should be defined better by calibrating more precisely the working arrangements between JASPERS and the Member States for all types of service (including capacity-building activities and horizontal and strategic support); -the need to implement a better system for monitoring the efficiency of JASPERS’ operations. The detailed analysis underpinning the above findings can be consulted in Chapter 5.3 of Annex 4 to the SWD. Coherence The analysis of the coherence criterion was undertaken from both an internal and external perspective. The internal perspective refers to both: (i) the coherence of the different types of service provided by JASPERS with each other; and (ii) the coherence of JASPERS’ services with EU priorities. The external perspective assesses coordination between JASPERS and other advisory services at EU level providing similar technical assistance to Member States. The evaluation finds that JASPERS’ services were coherent with EU requirements set by cohesion policy legislation and the EU-level objectives in the area of climate, the environment and competition. The services helped to guide project preparation and supported the assessment of projects in line with these objectives. Applying for EU co-financing of major projects required the Member States to ensure their projects: (i) were aligned with both cohesion policy objectives and the EU-level objectives of mitigating/adapting to climate change; (ii) accounted for their environmental impacts; and (iii) met all regulatory requirements related to state aid . As a consequence, all JASPERS services were organised so as to align the advisory and review work with these EU priorities. In the course of the advisory work, depending on the scope of the assignment requested by Member States, JASPERS provided an assessment (or, at a minimum, a description) of the project’s alignment with: (i) EU policy objectives and priorities; and (ii) EU requirements on the environment, climate change, and competition. In this process, JASPERS identified changes needed in project documentation to ensure this alignment, or made suggestions for improvement. The evaluation found that JASPERS provided advice on climate and environmental issues on 54-60% of the projects assisted, while JASPERS addressed state aid issues in about a quarter of these projects. From an internal coherence perspective, while acting independently during the course of providing the services, JASPERS review services and JASPERS technical advisory were found to be generally coherent with each other in the way they applied EU requirements to project preparation on the one hand and to project assessment on the other. Some scope for further alignment remained, although this was limited to around 10% or less of projects and concerned minimum regulatory requirements for project quality. The evaluation finds that for a large majority of projects (above 90%), there was full coherence between the two services. Firstly, for projects with no critical issues identified in the PSA report, there were cases for which JASPERS advisory had raised significant issues during project preparation and the Member State addressed these issues before submitting the project to the EC. These issues were therefore no longer present when the project reached the PSA assessment. The two outputs (of the technical advisory and of the review services) were therefore coherent even if they were different. Secondly, on many other projects on which the PSA report did raise critical issues, JASPERS advisory also signalled significant issues during project preparation, but these issues had not been addressed by the Member State by the time the project reached the EC. In these cases, the two outputs were also coherent, as the outstanding critical issues were due to the Member State failing to heed JASPERS’ warnings on the shortcomings of the application.

On external coherence, the evaluation explored the possibility of assessing the coherence and synergies between JASPERS services and other EC initiatives for technical assistance. Under the 2014-2020 multiannual financial framework, there were several centrally managed initiatives for technical assistance, most of them focused on capacity building and project preparation. These initiatives include: the European Investment Advisory Hub (EIAH); European Local Energy Assistance (ELENA); and the European Energy Efficiency Fund (EEEF). JASPERS was one of the largest of these initiatives, with 30% of the total cumulated budget of more than EUR 700 million for centrally managed technical assistance. The evaluation concludes that, until the end of 2018, the potential for synergies between JASPERS and other technical assistance initiatives were only partly exploited. JASPERS’ Quality Manual mentions the possibility for JASPERS to cooperate in sharing expertise with other technical assistance services. In 2015, following the publication of the European Fund for Strategic Investments (EFSI) Regulation , JASPERS assessed the scope for cooperation with the EIAH and even set up procedures to promote this cooperation. During the reference period of this evaluation, JASPERS cooperated with the EIAH on three assignments to construct and equip regional hospitals in Romania. The detailed analysis underpinning the above findings can be consulted in Chapter 5.4 of Annex 4 to the SWD. Relevance Because of the limitations described in Section 4 of this SWD, the relevance of intervention is assessed mainly in terms of its scope as compared with the previous programming period. The SWD only examines the connection of JASPERS’ work with identified needs in a limited way. More specifically, the needs analysis performed to inform the 2014-2020 working arrangements was based mostly on direct feedback from the Member States, which were also the direct beneficiaries of the technical-assistance services provided by JASPERS. The main approach for evaluating the relevance criterion was to assess the structure of JASPERS’ portfolio compared with the previous programming period. For the ESIF mandate, JASPERS remained relevant for project preparation in the 2014-2020 period, and provided more services for more countries than it did in 2007-2013. In addition, JASPERS became relevant for the new function of quality reviewing ESIF major projects. Similarly, JASPERS was relevant for the objectives set out in the CEF and IPA mandates, which called for the provision of technical assistance for projects that would be eligible for EU funding. During the programming period, JASPERS’ assistance became available for all Member States and IPA countries and was mandated by three EC services (DG REGIO, DG MOVE, and DG NEAR). Its services included: (i) technical advice for the preparation of major and non-major projects; (ii) capacity-building activities and horizontal and strategic support; and (iii) a new review function for projects supported by cohesion policy. For the first 5 years of the period, JASPERS registered around 1 100 assignments in 26 countries. By comparison, in the previous period 2007-2013, JASPERS provided technical assistance for 14 Member States and IPA countries through just under 800 completed assignments. Most of JASPERS’ activities for the ESIF mandate were concentrated in 3 main beneficiary countries (Romania, Croatia and Poland) – 2 of which (Romania and Poland) were consulted for the needs assessment to prepare the 2014-2020 period and indicated high needs of support from JASPERS services. When looking at the distribution of the ESIF assignments at country level, 57% of these assignments were carried out in 3 main beneficiary countries (Romania, Croatia, and Poland), 5% were multi-country assignments, and the rest were distributed across 19 other Member States. By comparison, during the previous period 2007-2013, 59% of JASPERS’ activities for this mandate were also concentrated in 3 main beneficiary countries (Poland, Romania, Bulgaria), with the rest of the activities distributed across 11 more Member States. JASPERS’ new review function was highly relevant, because it assessed the quality of all new major projects submitted or notified to the EC under the CPR legislative framework for the period 2014-2020 involving either Article 102(1) entailing the IQR review or Article 102(2) entailing the PSA review. The evaluation finds that 71% of new major projects were assessed under the submission procedure (with PSA report) during 2014-2018. On training support to build administrative capacity, the share of training activities for multi-country capacity building on topics of high relevance for all countries increased. Nevertheless, horizontal and strategic support tailored to country-specific needs accounted for most of the support. Based on JADE data, the evaluation finds that JASPERS carried out 220 assignments for capacity building and horizontal and strategic support up to December 2018, 75% of which were for horizontal and strategic advice. By comparison, in 2007-2013 JASPERS completed close to 140 such assignments, 96% of which were for horizontal and strategic advice. The relative share of assignments for capacity building, therefore, increased from 4% to 25% between the two periods. These assignments can be tailored to the needs of a specific country or can be organised on a multi-country basis for topics of high relevance to several or even all Member States. For capacity-building training, in contrast to the previous period when almost all assignments were country specific, there was an increase in 2014-2020 in the frequency of multi-country assignments. The most frequent topics for these training activities were climate change, the environment and state aid. The evaluation finds that the demand for country-specific capacity-building activities was not high during the period examined. Only 19 country-specific assignments for capacity building had been completed or were ongoing at the end of December 2018 in 10 Member States. More than 60% of these training assignments were devoted to climate-change adaptation and the environment, with the remainder mostly focused on horizontal aspects, planning and implementation. Finally, the evaluation finds that JASPERS’ support was relevant for meeting cohesion policy objectives, with JASPERS’ general aim of supporting Member States during the project-preparation process being maintained across the last two programming periods. In practical terms, Member States first sent requests for assistance to JASPERS, which then communicated these requests to the EC in order to receive the green light to proceed, which was generally done by tacit agreement. This approach tended to ensure that Member States’ needs for support were addressed, but it did not mean that there was a strategic analysis or prioritisation of requests. As highlighted by ECA auditors, JASPERS has not targeted its activities sufficiently, as it has not decided on clear priority areas for providing assistance to Member States, but has simply responded to demands as they arose. EU added value This section includes the main findings from the analysis of EU added value of the JASPERS initiative. First, it should be noted that the nature of the support concerned implicitly embeds the European dimension, as the technical assistance services provided are supported financially by EU institutions (EC and EIB). The evaluation of EU added value relies primarily on the feedback received from stakeholders responding to an online targeted consultation during summer/autumn 2019, in which stakeholders’ views primarily referred to the general added value of JASPERS. Of the respondents to this consultation, 75% were either project promoters/beneficiaries or managing authorities, while the remaining 25% were consultancies, companies, EC officials, and other types of institutions at national level. As they had first-hand experience with JASPERS services, the project promoters and managing authorities were able to provide informed opinions on the topics related to services provided by JASPERS. This distribution of respondents to the consultation inherently includes a possible bias in assessing involvement with JASPERS services, as project promoters and managing authorities were directly benefiting from the free-of-charge support provided by JASPERS.

The information obtained from stakeholders indicates that JASPERS services were appreciated because of the extensive methodological and sectoral technical expertise that it provided, thanks to the experience JASPERS gained from working with many Member States. According to stakeholders, this expertise cannot easily be replaced as it is not readily available on local markets. In addition, a large majority of stakeholders considered that, relative to other advisory services, JASPERS’ added value was higher especially in: (i) its support for aligning projects with EU priorities; (ii) the potential for knowledge transfer among Member States; and (iii) its hands-on experience. Different opinions were expressed on aspects such as flexibility in responding to needs and expertise on administrative and procedural issues. A large majority of managing authorities and project beneficiaries/promoters were of the view that JASPERS’ services will still be needed in the future, even if EC approval of major or strategic projects was no longer required. According to this majority of managing authorities, JASPERS support is necessary for aligning national strategies with EU sectoral policy priorities. Finally, from an operational perspective, stakeholders also stressed that the use of JASPERS services does not require public procurement procedures, and that this was a distinct advantage. However, willingness to pay for these services from their own technical-assistance budgets is rather low. The synopsis of the stakeholder consultation is included in Annex 2 of the SWD. Another aspect that is relevant to JASPERS’ EU added value lies in the organisation of its review function. According to the regulatory framework in place, Member States were able to set up individual arrangements for the independent quality review of the project documentation before they submitted their major project to the EC under Article 102(1) of the CPR. The creation of the IQR division in JASPERS, which provides review services available to all project promoters across Member States, reduced the administrative burden for Member States in the area of independent quality reviews. 6.Conclusions The purpose of this evaluation is to assess: (i) the implementation of the services provided by JASPERS; (ii) the contribution of JASPERS services to the development of high-quality, mature infrastructure projects; and (iii) the contribution of JASPERS to enhancing the technical capacity of managing authorities and beneficiaries in preparing such projects. The evaluation covers all JASPERS activities during the 2014-2018 period across all countries that made use of its advisory services (i.e. Member States and pre-accession countries). The JASPERS initiative is assessed against the five evaluation criteria decided on by the EC’s better regulation guidelines: effectiveness, efficiency, coherence, relevance and EU added value. As explained in the previous sections, the effectiveness, efficiency and relevance evaluation criteria were assessed via proxy analysis because of limitations in the definition of objectives and the design of the data monitoring system. The broad definition of JASPERS’ general objective constrained the evaluation, as this definition does not provide an indication of measurable outcomes that could inform the evaluation of JASPERS’ effectiveness and relevance. On the monitoring system, the available data on the exact amount of time worked on assignments started to be recorded only in 2017, while the evaluation covered the period up to the end of 2018. The limited timeframe of available data allows only for having initial reflections on the efficiency criteria. Overall, the evaluation finds that JASPERS’ activities expanded both in scope and volume during 2014-2018 compared to the previous period. On scope, although the ESIF mandate represented the largest share (close to 90%) of the overall portfolio in numbers of assignments, JASPERS’ services were supported over the period by two more EC services, as DG MOVE and DG NEAR joined DG REGIO in financing the initiative for CEF and IPA projects. JASPERS’ services were available for all Member States and IPA countries. JASPERS’ role was significantly strengthened by the EC through the creation of a new function for the quality review of major new ESIF projects. On volume, in the first 5 years of the 2014-2020 programming period, JASPERS received more requests for services than in the entire period 2007-2013. Effectiveness Much of JASPERS’ activities consisted of helping Member States to prepare major projects to obtain EU funding in a timely way. Of the 226 assignments for technical assistance completed up to end-2018, 158 (70%) resulted in major projects being adopted by the EC by October 2019. These accounted for a total financing from the ERDF and Cohesion Fund of EUR 23.9 billion, 65% of all funding going to major projects over the period. This is the same percentage as in the first 5 years of the 2007-2011 period (when more major projects but of a smaller size were financed). The timing of the involvement of JASPERS’ advisory in project preparation for major projects had a significant effect on the effectiveness of this involvement. Firstly, the evidence indicates that, when compared to non-assisted projects, the projects assisted by JASPERS advisory had better chances of being adopted without interruptions for quality issues. On the other hand, there is no compelling evidence that JASPERS’ support reduced either the number of issues that the EC raised or the duration of the interruptions. Nevertheless, early involvement of JASPERS advisory services in the project cycle was associated with a shorter duration of the core project preparation relative to the overall project lifecycle. The evaluation found that JASPERS advisory collected limited data on what happened to non-major projects it assisted once the advisory work was completed. This restricted the possibility of assessing the effectiveness of JASPERS’ support. The evidence suggests that JASPERS was relatively effective in flagging quality issues in the PSA assessments, which were then included by the EC in its letters to Member States setting out the reasons for requesting improvements (i.e. interrupting projects). While a large majority of activities for capacity building were completed, and achieved their objective of delivering training to the Member States, there is little evidence that they triggered a visible ripple effect in reducing Member State reliance on JASPERS’ support, which should, in principle, be the ultimate objective. The demand for these services increased rather than declined between 2007-2013 and 2014-2020. Issues relating to cost benefit analysis, climate change and the environment, on which training and workshops focused, continued to feature prominently in EC interruption letters. Other issues that also featured extensively in EC interruption letters included feasibility studies and risk and sensitivity analysis, neither of which were much covered in capacity-building activities. However, the evaluation found examples of a pragmatic approach to capacity building, in the form of JASPERS’: (i) ‘training the trainers’ activities at local level as a source of continuing knowledge dissemination; or (ii) coordinating training on specific issues across countries. These activities have the potential to be promoted more widely (as is now recognised by JASPERS management) as part of the strategic planning of capacity-building activities, with the long-term aim of reducing reliance on JASPERS’ assistance for preparing and implementing projects. Efficiency When assessing efficiency, it is difficult to set a suitable benchmark against which indicators of the time taken to complete assignments (or the staff effort/work effort involved) can be measured. In other words, although it might be possible to calculate the indicators concerned, it is still difficult to interpret these indicators and determine whether or not they are higher than they should be. This is a problem that is common to assessing the efficiency of cohesion policy interventions in general. The duration of assignments is a potential indicator of efficiency, but it does not necessarily reflect the amount of time taken, data on which have only been recorded since January 2017. The duration of an assignment is also linked to: (i) the initial quality of documentation; and (ii) the scale of revisions needed to meet minimum quality standards. The analysis of JASPERS’ efficiency was thus limited by the short timeframe of the available data on work per assignment. Before 2017, the monitoring framework at the level of individual assignments provided mostly descriptive information (i.e. without any quantitative data) of the assignment itself. Work intensity, defined as the days worked relative to the days available over the period of the assignment, amounted to 30% on average per assignment for technical assistance on major projects. In practice, there was little correlation identified between work intensity and the duration of assignments. Preliminary evidence on developments over time indicates a need for further in-depth reflection on possibilities for optimising the allocation of work effort and rationalising costs. However, it should be stressed that the main added value of JASPERS lies in the technical expertise and accumulated experience of its experts, and that ownership of projects and timely availability of project documentation remain the responsibility of project promoters.

Although JASPERS services were generally found to be efficient in completing the assignments initiated during the 2014-2018 period, the evaluation concludes that the planning and management of administratively closed assignments could be improved. Coherence During the 2014-2018 period, JASPERS’ services were coherent with: (i) EU requirements set by cohesion policy legislation; (ii) the EU-level objectives for climate and environment; and (iii) regulatory requirements for competition. The evidence compiled by the evaluation also suggests a high degree of internal coherence between the different independent JASPERS divisions. For over 90% of major projects examined, the technical advisory support provided by JASPERS to Member States in preparing projects was consistent with the conclusions of JASPERS IQR services involved in reviewing the major projects submitted to the EC. However, for the period examined, limited evidence was found of JASPERS cooperating with other technical-assistance initiatives of the EC, although the possibility of sharing experience was mentioned in its Quality Manual. Relevance JASPERS’ advisory services remained relevant in the 2014-2020 period, as reflected by the over 400 assignments requested by Member States up to December 2018. By October 2019, 70% of these assignments had resulted in major projects adopted by the EC – projects which mobilised almost EUR 24 billion worth of investments from the ERDF and Cohesion Fund. In addition, for the 2014-2020 programming period, JASPERS provided a new kind of support: reviewing the quality of new major projects submitted for EU funding, as required by the regulations, with the aim of ensuring consistency in the assessment and reducing the time taken for it. JASPERS’ capacity-building activities and provision of horizontal and strategic support continued in the 2014-2020 period. Most of the requests for support were horizontal rather than strategic in nature and occurred just before or just after the beginning of the 2014-2020 period as Member States sought to prepare for this new period. The evaluation finds that the needs analysis developed for 2014-2020 did not take a sufficiently critical view on the possible bias of the stakeholders involved in the analysis, who were also potential direct beneficiaries of JASPERS’ services. JASPERS’ activities continued to be demand-driven, assignments being initiated at the request of Member States or pre-accession countries. This left little room for the strategic planning of these activities, and for deploying resources where they could produce the most added-value in terms of the management and implementation of EU funding. EU added value The consultation carried out indicates that JASPERS services have been appreciated by stakeholders, because of their technical expertise and EU-wide experience. However, another reason stakeholders appreciated JASPERS was because accessing its support did not entail a public procurement process and it was provided free of charge. JASPERS’ added value was seen as being especially high in: (i) its support for helping to ensure that projects were aligned with EU priorities; (ii) the potential for knowledge transfer among Member States; and (iii) its hands-on experience in countries. Moreover, stakeholders considered that JASPERS services will also be needed in the future, not only because of the technical expertise brought by the experts and their EU-wide knowledge, but equally because of their ability to help align national strategies with EU priorities. To conclude, JASPERS’ EU added value, as indicated by the stakeholders engaged with JASPERS support, primarily lies in the technical expertise brought by its experts. These experts have EU-wide experience acquired while working in many countries and with the EC, and would be difficult to replace. Overall, in relation to the period 2014-2018, the evaluation reaches the following four conclusions. -JASPERS technical advisory services contributed to a faster and better preparation of projects leading to a faster mobilisation of EU funds. -JASPERS’ general objective has been broadly achieved together with all of the specific objectives, with the exception of the transfer of technical knowledge to Member States in view of reducing their reliance on JASPERS advisory services. -the evaluation aimed to address all the evaluation criteria in accordance with the Better Regulation guidelines, it relied on several proxy analysis due to various limitations; in particular the limitation concerning the availability of data needed for assessing the effectiveness and efficiency of JASPERS services was significant (more structured performance monitoring at the level of assignments only started in 2017, and was further strengthened only after 2018).

-There is scope for a more structured approach to the planning and monitoring of assignments, with: (i) well-defined and targeted objectives; (ii) clear and realistic timelines; and (iii) milestones that enable progress in implementation to be monitored over time. 7.Developments made during the last 2 years of the 2014-2020 programming period and the outlook for 2021-2027 Because the evaluation coincides with the preparation of the new mandate for JASPERS that will cover technical assistance during the 2021-2027 programming period, a short presentation of the developments that took place during 2019-2020 is worthwhile. The following bullet points contain this short presentation. -To improve effectiveness and internal coherence, JASPERS and the EC continued their efforts to strengthen the coordination and early warning system on issues flagged by the advisory services. To this end, regular meetings were organised between the EC and JASPERS on the one hand, and between JASPERS IQR and JASPERS advisory, on the other. In addition, the IQR process became more formal in 2019-2020, following the example of the submission procedure. -The demand for country-specific capacity-building activities has increased since 2018, partly because of the need to prepare the 2021-2027 programmes. -The review-services function will no longer exist for the 2021-2027 programming period, as the new legislation proposed a new assessment system for projects of strategic importance, and this assessment is to be carried out by the Member States. -The EC is currently developing new working arrangements, which are planned to take the form of trilateral work programmes, negotiated for Member States that plan to use JASPERS regularly. These programmes: (i) will set clear objectives for JASPERS’ support at country level; (ii) will decide on priority areas for the support; and (iii) are expected to lead to better prioritisation of assignments. -Following the ECA performance audit of JASPERS, the EC drew up an action plan with measures to be implemented starting in 2018. By January 2019, the following measures had been implemented fully or were in the process of being finalised: (i) measures relating to clarification of the roles and responsibilities of the main stakeholders; and (ii) measures relating to the working arrangements of the tripartite cooperation between the EC, JASPERS and the Member States. In line with the audit recommendations, the EC has requested improvements to the monitoring system. A new set of performance indicators and JASPERS has reported regularly on these indicators since late 2018. -On external coherence, in 2019 it was decided that, for the next mandate, JASPERS should be included in the future InvestEU Advisory Hub. This hub will be a single point of entry for EU-funded technical-assistance initiatives. Currently, the EC is working to implement this decision by adjusting the JASPERS governance system to the context of the Advisory Hub. There was a reorganisation of JASPERS inside the EIB to accommodate these changes. -Following the CEF pilot experience with JASPERS’ assistance, DG MOVE has decided to support a renewed mandate for JASPERS’ services for CEF projects in the programming period 2021-2027. This renewed mandate has a significantly larger budget (EUR 30 million for the period 2021-2024), and there is therefore increased potential for more extensive assistance to projects in all Member States. DG MOVE will delegate its budget to DG REGIO to promote cross-fertilisation, coordination and synergies between the cohesion policy programmes and the CEF for advisory services of the very same nature on transport projects (including on the trans-European transport network (TEN-T)) in all Member States. The developments in 2019-2020 have already addressed most of the points for improvement highlighted by this evaluation. The DG REGIO negotiating team for JASPERS’ services for the 2021-2027 programming period has been regularly informed of the progress of the evaluation. The evaluation’s findings and conclusions have been discussed and taken into account by DG REGIO as appropriate. DG REGIO has given particular attention to the more strategic approach of the assistance to be provided by JASPERS (the country-specific trilateral work programmes between the EC, the Member States and JASPERS as explained above). Two of the main aspects still to be discussed for the JASPERS services in the 2021-2027 period concern: (i) the monitoring framework for which the conclusions of the evaluation will be taken into account; and (ii) knowledge transfer (capacity building) to the Member States. The proactive approach implemented in the last 2 years of the 2014-2020 programming period provides reassurance that the development of the JASPERS initiative and its development will be closely scrutinised and monitored by the parties involved.



Annex 1: Procedural information 1.Lead DG, Decide Planning/CWP references Directorate General Regional and Urban Policy, Decide Planning: PLAN/2017/2128. 2.Organisation and timing

Tender procedure launched December 2017 Contract signed July 2018 First Steering Group meeting November 2018 Last Steering Group meeting March 2021 Number of Steering Group meetings 4 Participating DGs (in addition to DG REGIO) CLIMA: Climate Action ENV: Environment MOVE: Mobility and Transport SG: Secretary General NEAR: Neighbourhood and Enlargement ECFIN: Economic and Financial Affairs 3.Exceptions to the better regulation guidelines Due to the specialised topic of the evaluation, the open public consultation was replaced by a targeted consultation of relevant stakeholders. The targeted consultation was organised online, during July – September 2019. 4.Consultation of the RSB (if applicable) The draft staff working document (SWD) was presented at the Regulatory Scrutiny Board (RSB) meeting on 19 May 2021. A preliminary upstream meeting between DG REGIO and the Regulatory Scrutiny Board took place in July 2018. 5.Evidence, sources and quality See section 4 of the Staff Working Document. 6.Consultation of the Regulatory Scrutiny Board The draft SWD was presented at the Regulatory Scrutiny Board (RSB) meeting on 19 May 2021 and the RSB issued a negative opinion on 21 May 2021. The RSB comments were addressed in this final document as indicated in the following table. RSB comment Reply 1. The report does not articulate what should have been achieved by the time of the evaluation. It does not sufficiently establish what developments can be attributed to JASPERS as compared to other factors. The general aim of the JASPERS initiative, as defined by the 2006 Memorandum of Understanding, has been to support the implementation of Cohesion Policy by increasing the resources available for project preparation and by improving the quality of technical advice available to project promoters, thus contributing to the preparation of high quality projects. In the 2014-2020 programming period, whose first five years are addressed by this evaluation, the general objective of the JASPERS initiative continued to be the improvement of the quality of investments supported by EU Funds. The evaluation finds that the JASPERS general objective has been broadly achieved, as regards all of its specific objectives with the exception of the transfer of technical knowledge to MS in view of reducing their reliance on JASPERS advisory service. This has been included in the conclusions of the evaluation. With respect to the causal links, the evaluation includes several econometric analysis developed by REGIO and detailed in section 4.3 of Annex 4 of the SWD. For analyses requiring the separation of JASPERS’ contribution from other factors for a given result, the regression analysis was used. This is the case, for example, in the analysis of the correlation between JASPERS’ assistance and the duration of preparation of projects. Nevertheless, the contribution of JASPERS cannot be dissociated in all cases from the other factors that might influence the analysis such as the legal framework in place or the administrative set-up of the major projects approval process. 2. The report does not adequately assess the EU added value and relevance of JASPERS services. It does not examine the potential of JASPERS conflicting objectives and the risk of crowding out capacity building in Member States, and does not draw clear policy lessons from JASPERS’ failure to meet its national capacity building objective. JASPERS’ EU added value, as indicated by the stakeholders engaged with JASPERS support, primarily lies in the technical expertise brought by its experts who have EU-wide experience acquired while working in many countries and with the European Commission. This has been highlighted in the conclusions of the SWD. The evaluation analyses the relevance perspective of JASPERS during the 2014-2018 period. The needs analysis made by JASPERS before the 2014-2020 period is briefly described in page 4 of the SWD. Because of limitations described in section 4 of the SWD, the relevance of the intervention is assessed mainly in terms of its scope as compared with the previous programming period, while the connection with identified needs is examined in a limited way. The matter of technical capacity building in MS has been thoroughly analysed in the SWD. The conclusion was that during the 2014-2018 evaluation period, while a large majority of activities for capacity building were completed and achieved their objective of delivering training to the Member States, they did not prove sufficient to trigger a visible ripple effect on the technical capacity of Member States that would allow to address issues such as cost-benefit analysis, climate change and environmental requirements in project preparation. A detailed analysis of the findings that led to this conclusion is presented in section 5 – Effectiveness of the SWD and in section 5.2 of Annex 4 to the SWD. The evaluation, however, found examples of a pragmatic approach to capacity building, in the form of ‘training the trainers’ at local level as a source of continuing knowledge dissemination or coordinating training on specific issues across countries. This is highlighted in the conclusions of the SWD. Furthermore JASPERS’ assistance is seen as relevant in the current and future context of the green and digital transition and increased funding opportunities for MS, for which the preparation of mature and high quality projects is fundamental. At the EC level, the need for technical assistance services linked to the future project pipeline has been recognized through the creation of the InvestEU Advisory Hub in which JASPERS will be embedded. 3. The efficiency analysis is not clear enough on observed costs and benefits of JASPERS. The potential for efficiency improvement is not clearly set out. JASPERS